Orthonormal Bases

Vector $q_1,\dots ,q_n\in {\mathbb{R}}^m$ are orthonormal (orthogonal and normalized) if:

• $\Vert q_1\Vert =1\forall i=1\dots n$
• $q_i^{\top }{q}_j=0\forall i\ne j$

Let the columns of $Q$ be $q_1,\dots ,q_n$: $Q^{\top }Q$ matrix with entries ${\delta }_{ij}=q_i^{\top }{q}_j$ (the Kronecker delta) $Q^{\top }Q=I$

Title

Note that $Q$ may not be a square matrix. Hence, it is not necessarily the case that $Q{Q}^{\top }=I$

$q_1,\dots ,q_n$ are orthogonal $⟹$ $q_1,\dots ,q_n$ are linearly independent (see Fact 2)

Example

Let $Q=\left[\begin{array}{cc}1& 0\\ 0& \frac{1}{3}\\ 0& \frac{2}{3}\\ 0& \frac{2}{3}\end{array}\right]$ $Q^{\top }Q=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$

Definition: Orthogonal matrix

A square matrix $Q\in {\mathbb{R}}^{n\times n}$ is orthogonal if $Q^{\top }Q=I$ $⟹Q^{\top }=Q^{-1}$ and $Q{Q}^{\top }=I$ the columns of $Q$ form an orthonormal basis for ${\mathbb{R}}^n$.

Properties

Proposition 6.3.6

Orthogonal matrices preserve norm and inner product of vectors

This means

$$\Vert Qx\Vert =\Vert x\Vert \forall x\in {\mathbb{R}}^n$$ $$(Qx{)}^{\top }(Qy)=x^{\top }y\forall x,y\in {\mathbb{R}}^n$$

Let $S$ be a subspace of ${\mathbb{R}}^m$ and $q_1,\dots ,q_n$ form an orthonormal basis of $S$ Then,

• $S=span(q_1,\dots q_n)=\left\{{\sum }_{i=1}^n{\lambda }_i{q}_i|\forall i{\lambda }_i\in \mathbb{R}\right\}$
• The projection matrix $P$ to $S$ is given by $Q{Q}^{\top }$
  • $P=Q(Q^{\top }Q{)}^{-1}{Q}^{\top }=Q{Q}^{\top }$

Construction of Orthonormal Bases

Task

Given $a_1,\dots ,a_n\in {\mathbb{R}}^m$ are basis of a subspace $S_n=span(a_1,\dots ,a_n)$ Find a orthonormal basis of $S_n$

Gram-Schmidt Process

Idea

• $q_1=\frac{a_1}{\Vert a_1\Vert }$
• $q_2$ is $a_2$ subtract $a_2$ projection onto $C(a_1)$
• For $k\ge 3$
  • $a_k$ subtract $a_k$ projection onto $span(q_1,\dots ,q_{k-1})$

Algorithm

• $q_1=\frac{a_1}{\Vert a_1\Vert }$
• For $k\ge 2$:
  • $q_k^{\prime }=a_k-{\sum }_{j=1}^{k-1}(a_k^{\top }{q}_j)q_i$
  • $q_k=\frac{q_k^{\prime }}{\Vert q_k^{\prime }\Vert }$

Proof of correctness