Spectral Theorem

GPT5.2

Certain “nice” linear operators can be understood by decomposing the space into spectral components (generalized eigenspaces), so the operator acts like multiplication by a scalar on those components.

Theorem

NOTE

Any symmetric matrix $A\in {\mathbb{R}}^{n\times n}$ has $n$ real eigenvalues and an orthonormal basis of ${\mathbb{R}}^n$ consisting of its eigenvectors.

Proof (Induction) There are $k$ orthonormal eigenvectors von $A$ Base case $k=1$: if $\lambda \in \mathbb{C}$ an eigenvalue of $A$, then $\lambda \in \mathbb{R}$ (we can normalize this $\lambda$): Let $v\in {\mathbb{C}}^n$ be an eigenvector of $\lambda$. We have $Av=\lambda v$.

Hermitian conjugate

Let $A^{\ast }={\bar{A}}^{\top }$ with $\bar{A}=\left[\begin{array}{c}{\bar{A}}_{i,j}{\forall }_{i,j}\end{array}\right]\in {\mathbb{C}}^{m\times n}$

$(AB{)}^{\ast }=B^{\ast }{A}^{\ast }$

Link to original

Since $A$ is real symmetric we have $A^{\ast }=A$. Thus, $\bar{\lambda }\Vert v{\Vert }^2=\bar{\lambda }{v}^{\ast }v=(\lambda v{)}^{\ast }v=(Av{)}^{\ast }v=v^{\ast }{A}^{\ast }v=v^{\ast }Av=v^{\ast }\lambda v=\lambda \Vert v{\Vert }^2$ Since $v\ne 0$, then $\lambda =\bar{\lambda }$. $⟹\lambda \in \mathbb{R}$

Induction steps $k\to k+1$ $\exists k$ orthonormal eigenvectors $v_1,\dots ,v_k$ with real eigenvalues ${\lambda }_1,\dots {\lambda }_k$ Let $u_{k+1},\dots ,u_n$ a orthonormal basis of $span(v_1,\dots ,v_k{)}^{\perp }$ Define $V=\left[\begin{array}{cccccc}{v}_1& \dots & v_k& u_{k+1}& \dots & u_n\end{array}\right]$ $V$ is orthogonal $⟹V^{\top }V=I⟺V^{\top }=V^{-1}$ Let $B=V^{\top }AV\in {\mathbb{R}}^{n\times n}$ ($B$ is symmetric) $B=\left[\begin{array}{cc}{\Lambda }_k& F\\ G& C\end{array}\right]$ $G_{ij}=u_i^{\top }{\lambda }_j{v}_j={\lambda }_j{u}_i^{\top }{v}_j=0$ $F_{ij}=$ $C_{ij}=u_i^{\top }A{u}_j=u_i^{\top }{A}^{\top }{u}_j=(A{u}_i{)}^{\top }{u}_j=u_j^{\top }A{u}_i=C_{ji}$ $C$ is symmetric $⟹C$ has a real eigenvalue ${\lambda }_{k+1}$ with eigenvector $y\in {\mathbb{R}}^{n-k}$ Let $w=\left(\begin{array}{c}0\\ y\end{array}\right)\in {\mathbb{R}}^n$ $Bw=\left[\begin{array}{cc}{\Lambda }_k& 0\\ 0& C\end{array}\right]\left(\begin{array}{c}0\\ y\end{array}\right)=\left(\begin{array}{c}0\\ Cy\end{array}\right)={\lambda }_{k+1}\left(\begin{array}{c}0\\ y\end{array}\right)={\lambda }_{k+1}w$ $⟹$ $w$ is eigenvector of ${\lambda }_{k+1}$ (under matrix $B$)

Define $v_{k+1}=Vw⟺V^{\top }{v}_{k+1}=w$ $B=V^{\top }AV⟺A=VB{V}^{\top }$ $⟹A{v}_{k+1}=VB{V}^{\top }{v}_{k+1}=VBw={\lambda }_{k+1}Vw={\lambda }_{k+1}{v}_{k+1}$ $v_{k+1}\in span(u_{k+1},\dots ,u_n)⟹v_i^{\top }{u}_{k+1}=0\forall i=1\dots k$ Finally we normalize $v_{k+1}$ such that $\Vert v_{k+1}\Vert =1$ Now we proved that there is always $k+1$ real orthonormal eigenvectors

Corollary 9.2.2

For any symmetric matrix $A$, there exists an orthogonal matrix $V\in {\mathbb{R}}^{n\times n}$ such that $A=V\Lambda V^{\top }$

Corollary 9.2.4

The rank of a real symmetric matrix A is the number of non-zero eigenvalues (counting repetitions)

Proposition 9.2.10 (Rayleigh Quotient)

Given a symmetric matrix $A\in {\mathbb{R}}^{n\times n}$ the Rayleigh Quotient defined for $x\in {\mathbb{R}}^n\setminus \{0\}$ is

$$R(x)=\frac{x^{\top }Ax}{x^{\top }x}$$

NOTE

The Rayleigh Quotient gives the corresponding eigenvalue of a eigenvector

How do we get this quotient

we have $Av=\lambda v$. $v$ is given, and we want to find $\lambda$. However, we cannot divide $v$ on both sides, because vector division is not defined. We multiply both side by $v^{\top }$ first $v^{\top }Av=v^{\top }\lambda v$ $v^{\top }Av=\lambda v^{\top }v$ $\lambda =\frac{v^{\top }Av}{v^{\top }v}$

Warning

Even if $v$ is not an eigenvector, the quotient is still defined, but does not generally give an eigenvalue.

In fact there is ${\lambda }_{min}\le R(x)\le {\lambda }_{max}$ for all non-zero $x$

Lemma 9.2.7

Let $A\in {\mathbb{R}}^{n\times n}$ be a symmetric matrix and ${\lambda }_1\ne {\lambda }_2\in \mathbb{R}$ be two distinct eigenvalues of $A$ with corresponding eigenvectors $v_1,v_2$. Then $v_1$ and $v_2$ are orthogonal.