Singular Value Decomposition

Generalization of Diagonalizing

Let $A\in {\mathbb{R}}^{m\times n}$ We try to find a basis $V=\left[\begin{array}{cccc}{v}_1& v_2& \dots & v_n\end{array}\right]\in {\mathbb{R}}^{n\times n}$ and $U=\left[\begin{array}{cccc}{u}_1& u_2& \dots & u_n\end{array}\right]\in {\mathbb{R}}^{m\times m}$ $A{A}^{\top }=U\Lambda U^{\top }$ $A^{\top }A=V{\Lambda }^{\prime }{V}^{\top }$ (the eigenvalues are the same because of Proposition 9.12.15)

Definition

NOTE

$A=U\Sigma V^{\top }$

${\Sigma }_{ij}=0$ for $i\ne j$ ${\Sigma }_{ij}\ge 0\forall i\in \{1,\dots ,min\{m,n\}\}$ ${\Sigma }_{11}\ge {\Sigma }_{22}\ge \cdots \ge {\Sigma }_{kk}$

$V=A^{\top }U{\Sigma }^{-1}$ ($V$ can not be chosen freely, it is determined by the chosen eigenvalues and eigenvectors of $A$)

Theorem 9.3.3

NOTE

Every matrix $A\in {\mathbb{R}}^{m\times n}$ has a singular value decomposition.

Important

In other words: Every linear transformation is diagonal (only scaling on some axis) when viewed in the bases of the singular vectors.