Linear AlgebraSymmetric Matrices and Quadratic Forms1 of 4
Symmetric Matrices and Quadratic Forms · 1

The Spectral Theorem

Diagonalizing a matrix as A=PDP−1A = PDP^{-1} turns it into simple stretching along its eigenvector axes, and we want to know when those axes sit at right angles so we never have to invert PP.

1

A stretching matrix at a 45∘45^\circ angle

We want to see what the matrix B=[2112]B = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}, which mixes xx and yy coordinates together, has in common with the simple diagonal stretching matrix A=[3001]A = \begin{bmatrix} \eone{3} & 0 \\ 0 & \etwo{1} \end{bmatrix}.

A stretching matrix in disguise
A=[3001]A10=  ?B=[2112]B10=  ?\displaystyle \begin{alignedat}{2} A &= \begin{bmatrix} \eone{3} & 0 \\ 0 & \etwo{1} \end{bmatrix} &\qquad A^{10} &= \;? \\[10pt] B &= \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} &\qquad B^{10} &= \;? \end{alignedat}
A=[3001]A = \begin{bmatrix} \htmlClass{lc-green}{3} & 0 \\ 0 & \htmlClass{lc-pink}{1} \end{bmatrix}
[10]\htmlClass{lc-green}{\begin{bmatrix} 1 \\ 0 \end{bmatrix}}
λ1=3\htmlClass{lc-green}{\lambda_1} = \htmlClass{lc-green}{3}
[01]\htmlClass{lc-pink}{\begin{bmatrix} 0 \\ 1 \end{bmatrix}}
λ2=1\htmlClass{lc-pink}{\lambda_2} = \htmlClass{lc-pink}{1}
B=[2112]B = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}
v1=[11]\htmlClass{lc-green}{v_1} = \htmlClass{lc-green}{\begin{bmatrix} 1 \\ 1 \end{bmatrix}}
λ1=3\htmlClass{lc-green}{\lambda_1} = \htmlClass{lc-green}{3}
v2=[−11]\htmlClass{lc-pink}{v_2} = \htmlClass{lc-pink}{\begin{bmatrix} -1 \\ 1 \end{bmatrix}}
λ2=1\htmlClass{lc-pink}{\lambda_2} = \htmlClass{lc-pink}{1}

Testing the two 45∘45^\circ diagonal directions shows that BB keeps both vectors on their own lines:

B[11]⏟v1=[33]=3⏟λ1[11],B[−11]⏟v2=[−11]=1⏟λ2[−11]\displaystyle \begin{aligned} B\eone{\underbrace{\begin{bmatrix} 1 \\ 1 \end{bmatrix}}_{v_1}} &= \begin{bmatrix} 3 \\ 3 \end{bmatrix} = \underbrace{\eone{3}}_{\eone{\lambda_1}}\eone{\begin{bmatrix} 1 \\ 1 \end{bmatrix}}, \\ B\etwo{\underbrace{\begin{bmatrix} -1 \\ 1 \end{bmatrix}}_{v_2}} &= \begin{bmatrix} -1 \\ 1 \end{bmatrix} = \underbrace{\etwo{1}}_{\etwo{\lambda_2}}\etwo{\begin{bmatrix} -1 \\ 1 \end{bmatrix}} \end{aligned}

So BB performs the exact same stretch by 3\eone{3} and 1\etwo{1} as the diagonal matrix AA, just along perpendicular axes tilted by 45∘45^\circ.

2

The Spectral Theorem: why symmetric matrices have perpendicular eigenvectors

For most matrices, the eigenvectors in A=PDP−1A = PDP^{-1} meet at a slanted angle, so we have to compute P−1P^{-1} to find a vector's coordinates. The Spectral Theorem tells us that the matrices with a full set of perpendicular eigenvectors are the symmetric matrices: matrices that equal their own transpose, AT=AA^T = A.

The Spectral Theorem: why symmetric matrices have perpendicular eigenvectors
B=[2112]B = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}

When we divide perpendicular eigenvectors by their lengths to make them unit vectors and put them into the columns of an orthogonal matrix QQ, its inverse is simply its transpose (Q−1=QTQ^{-1} = Q^T, so QTQ^T has those same unit eigenvectors as its rows):

AT=A⏟symmetric matrix  ⟺  A\displaystyle \underbrace{A^T = A}_{\text{symmetric matrix}} \iff A =Q⏟orthonormaleigenvectorsΛ⏟realeigenvaluesQT⏟Q−1\displaystyle = \underbrace{Q}_{\substack{\text{orthonormal} \\ \text{eigenvectors}}} \underbrace{\Lambda}_{\substack{\text{real} \\ \text{eigenvalues}}} \underbrace{Q^T}_{Q^{-1}}

To see why different eigenvalues of a symmetric matrix always give perpendicular eigenvectors, use the transpose rule for dot products, (Au)⋅v=(Au)Tv=uTATv(A\eone{u}) \cdot \etwo{v} = (A\eone{u})^T \etwo{v} = \eone{u}^T A^T \etwo{v}. Because AT=AA^T = A, moving a symmetric matrix from one vector to the other inside a dot product changes nothing:

(Au)⋅v\displaystyle (A\eone{u}) \cdot \etwo{v} =u⋅(Av)\displaystyle = \eone{u} \cdot (A\etwo{v})

Using this on our eigenvectors v1\eone{v_1} (with λ1=3\eone{\lambda_1} = \eone{3}) and v2\etwo{v_2} (with λ2=1\etwo{\lambda_2} = \etwo{1}) forces their dot product to be zero:

3(v1⋅v2)⏟(Bv1)⋅v2\displaystyle \underbrace{\eone{3}(\eone{v_1} \cdot \etwo{v_2})}_{(B\eone{v_1}) \cdot \etwo{v_2}} =1(v1⋅v2)⏟v1⋅(Bv2)  ⟹  v1⋅v2\displaystyle = \underbrace{\etwo{1}(\eone{v_1} \cdot \etwo{v_2})}_{\eone{v_1} \cdot (B\etwo{v_2})} \implies \eone{v_1} \cdot \etwo{v_2} =0\displaystyle = 0

On a 90∘90^\circ grid of unit axes, a vector's coordinate along each axis is just its dot-product shadow on that axis, whereas on a slanted grid, moving along one axis changes the shadow on the other.

Dot-product shadows add right back to the vector when the unit axes meet at a right angle, and miss when the axes tilt because one axis casts a shadow onto the other.
3

Orthogonal diagonalization step by step

Now let's factor B=[2112]B = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} into B=QΛQTB = Q\Lambda Q^T and see how the three matrices act on an input vector x\vecx{x}.

Orthogonal diagonalization step by step
v1=[11]\htmlClass{lc-green}{v_1} = \htmlClass{lc-green}{\begin{bmatrix} 1 \\ 1 \end{bmatrix}}
v2=[−11]\htmlClass{lc-pink}{v_2} = \htmlClass{lc-pink}{\begin{bmatrix} -1 \\ 1 \end{bmatrix}}
B=[2112]=QΛQTB = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} = Q\Lambda Q^T
Q=  ?Q = \;?
Text solution
Divide each perpendicular eigenvector by its length to make unit vectors So that the dot products don't multiply coordinates by the eigenvector length 12+12=2\sqrt{1^2 + 1^2} = \sqrt{2}, we divide v1\eone{v_1} and v2\etwo{v_2} by 2\sqrt{2} to give them length 11.
u1=v1∥v1∥=[1212],u2=v2∥v2∥=[−1212]\displaystyle \begin{aligned} \eone{u_1} &= \frac{\eone{v_1}}{\|\eone{v_1}\|} = \eone{\begin{bmatrix} \tfrac{1}{\sqrt{2}} \\ \tfrac{1}{\sqrt{2}} \end{bmatrix}}, \\ \etwo{u_2} &= \frac{\etwo{v_2}}{\|\etwo{v_2}\|} = \etwo{\begin{bmatrix} -\tfrac{1}{\sqrt{2}} \\ \tfrac{1}{\sqrt{2}} \end{bmatrix}} \end{aligned}
Put the unit eigenvectors into the columns of QQ, their transposes into the rows of QTQ^T, and the eigenvalues into Λ\Lambda Putting each column of QQ in the same spot as its eigenvalue in Λ\Lambda finishes the factorization without inverting a matrix.
B=[12−121212]⏟Q: rebuild in standard space[3001]⏟Λ: stretch[1212−1212]⏟QT: measure projections\displaystyle \begin{aligned} B = {} & \underbrace{\begin{bmatrix} \eone{\tfrac{1}{\sqrt{2}}} & \etwo{-\tfrac{1}{\sqrt{2}}} \\ \eone{\tfrac{1}{\sqrt{2}}} & \etwo{\tfrac{1}{\sqrt{2}}} \end{bmatrix}}_{Q\text{: rebuild in standard space}} \underbrace{\begin{bmatrix} \eone{3} & 0 \\ 0 & \etwo{1} \end{bmatrix}}_{\Lambda\text{: stretch}} \\ & \underbrace{\begin{bmatrix} \eone{\tfrac{1}{\sqrt{2}}} & \eone{\tfrac{1}{\sqrt{2}}} \\ \etwo{-\tfrac{1}{\sqrt{2}}} & \etwo{\tfrac{1}{\sqrt{2}}} \end{bmatrix}}_{Q^T\text{: measure projections}} \end{aligned}

Always remember to divide the eigenvectors by their lengths before writing QQ: perpendicular columns that don't have length 11 won't give Q−1=QTQ^{-1} = Q^T.

Because u1\eone{u_1} and u2\etwo{u_2} have length 11, QTQ^T and QQ purely rotate by 45∘45^\circ without changing any lengths, leaving all the stretching to the diagonal matrix Λ\Lambda in the middle.

Because the unit eigenvectors have length 1, the outer steps are pure 45 degree rotations and the diagonal matrix in the middle does all the stretching.
4

Repeated eigenvalues and Gram-Schmidt in an eigenspace plane

When a symmetric matrix has a repeated eigenvalue, the Spectral Theorem guarantees we still get a full set of eigenvectors. For a double eigenvalue, we get a whole 2D plane of eigenvectors rather than a single line. That whole plane is automatically 90∘90^\circ to eigenvectors from other eigenvalues, but the two basis vectors we pick inside the plane usually meet at an angle other than 90∘90^\circ. Let's orthogonally diagonalize a 3×33 \times 3 symmetric matrix with a repeated eigenvalue:

A=[211121112]\displaystyle A = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix}
Repeated eigenvalues and Gram-Schmidt in an eigenspace plane
A=[211121112]=QΛQT\displaystyle A = \begin{bmatrix} 2 & 1 & 1 \\ 1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix} = Q\Lambda Q^T
Q\displaystyle Q =  ?\displaystyle = \;?

Because every vector in an eigenspace plane gets stretched by the same number λ\lambda, any combination of eigenvectors in that plane stays in the plane and is still an eigenvector. When our two basis vectors w1\etwo{w_1} and w2\ethree{w_2} in the plane aren't perpendicular (w1⋅w2≠0\etwo{w_1} \cdot \ethree{w_2} \miss{\neq 0}), we keep v2=w1\etwo{v_2} = \etwo{w_1} and use Gram-Schmidt to subtract the projection of w2\ethree{w_2} onto v2\etwo{v_2}:

v3\displaystyle \ethree{v_3} =w2−w2⋅v2v2⋅v2v2⏟projection of w2 onto v2\displaystyle = \ethree{w_2} - \underbrace{\frac{\ethree{w_2} \cdot \etwo{v_2}}{\etwo{v_2} \cdot \etwo{v_2}}\etwo{v_2}}_{\text{projection of } \ethree{w_2} \text{ onto } \etwo{v_2}}
Text solution
Factor the characteristic polynomial to find the eigenvalues Solving det⁡(A−λI)=0\det(A - \lambda I) = 0 gives one single eigenvalue and one repeated eigenvalue.
−(λ−4)(λ−1)2\displaystyle -(\lambda - 4)(\lambda - 1)^2 =0  ⟹  λ1\displaystyle = 0 \implies \eone{\lambda_1} =4,λ2\displaystyle = \eone{4}, \quad \etwo{\lambda_2} =1,λ3\displaystyle = \etwo{1}, \quad \ethree{\lambda_3} =1\displaystyle = \ethree{1}
Find the eigenvector for λ1=4\eone{\lambda_1} = \eone{4} and two basis eigenvectors in the λ=1\lambda = 1 plane Solving (A−4I)v=0(A - 4I)v = 0 gives v1=[111]\eone{v_1} = \eone{\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}}, while (A−1I)v=0(A - 1I)v = 0 simplifies to the single plane equation x1+x2+x3=0\eigsp{x_1 + x_2 + x_3 = 0}; setting the free variables (x2,x3)(x_2, x_3) to (1,0)(1, 0) and (0,1)(0, 1) gives two eigenvectors in that plane that meet at 60∘60^\circ.
v1=[111],w1=[−110],w2=[−101](w1⋅w2=1≠0)\displaystyle \begin{aligned} & \eone{v_1} = \eone{\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}}, \qquad \etwo{w_1} = \etwo{\begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}}, \\ & \ethree{w_2} = \ethree{\begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix}} \qquad (\etwo{w_1} \cdot \ethree{w_2} = \miss{1 \neq 0}) \end{aligned}
Subtract the projection of w2\ethree{w_2} onto v2=w1\etwo{v_2} = \etwo{w_1} to get a perpendicular eigenvector in the plane Keeping v2=[−110]\etwo{v_2} = \etwo{\begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}}, Gram-Schmidt subtracts the part of w2\ethree{w_2} along v2\etwo{v_2}, and multiplying by 22 clears the fractions before we divide by the length.
v3=w2−w2⋅v2v2⋅v2v2=[−101]−12[−110]=[−12−121]→×2[−1−12]\displaystyle \begin{aligned} \ethree{v_3} &= \ethree{w_2} - \frac{\ethree{w_2} \cdot \etwo{v_2}}{\etwo{v_2} \cdot \etwo{v_2}}\etwo{v_2} \\ &= \ethree{\begin{bmatrix} -1 \\ 0 \\ 1 \end{bmatrix}} - \frac{1}{2}\etwo{\begin{bmatrix} -1 \\ 1 \\ 0 \end{bmatrix}} \\ &= \ethree{\begin{bmatrix} -\tfrac{1}{2} \\ -\tfrac{1}{2} \\ 1 \end{bmatrix}} \xrightarrow{\times 2} \ethree{\begin{bmatrix} -1 \\ -1 \\ 2 \end{bmatrix}} \end{aligned}
Divide v1,v2,v3\eone{v_1}, \etwo{v_2}, \ethree{v_3} by their lengths to build QQ and Λ\Lambda Dividing v1,v2,v3\eone{v_1}, \etwo{v_2}, \ethree{v_3} by their lengths 3\sqrt{3}, 2\sqrt{2}, and 6\sqrt{6} gives the three orthonormal columns of QQ.
Q=[13−12−161312−1613026],Λ=[400010001]\displaystyle \begin{aligned} Q &= \begin{bmatrix} \eone{\tfrac{1}{\sqrt{3}}} & \etwo{-\tfrac{1}{\sqrt{2}}} & \ethree{-\tfrac{1}{\sqrt{6}}} \\ \eone{\tfrac{1}{\sqrt{3}}} & \etwo{\tfrac{1}{\sqrt{2}}} & \ethree{-\tfrac{1}{\sqrt{6}}} \\ \eone{\tfrac{1}{\sqrt{3}}} & \etwo{0} & \ethree{\tfrac{2}{\sqrt{6}}} \end{bmatrix}, \\ \Lambda &= \begin{bmatrix} \eone{4} & 0 & 0 \\ 0 & \etwo{1} & 0 \\ 0 & 0 & \ethree{1} \end{bmatrix} \end{aligned}

So v3=[−1−12]\ethree{v_3} = \ethree{\begin{bmatrix} -1 \\ -1 \\ 2 \end{bmatrix}} stays inside the λ=1\lambda = 1 plane (−1−1+2=0-1 - 1 + 2 = 0) and sits at 90∘90^\circ to both v2\etwo{v_2} in the plane and v1\eone{v_1} perpendicular to the plane.

Because every vector in the λ=1\lambda = 1 plane is an eigenvector and is already 90∘90^\circ to v1\eone{v_1}, subtracting the projection onto v2\etwo{v_2} keeps v3\ethree{v_3} in the plane while making all three eigenvectors perpendicular.

Every direction in the eigenspace plane is an eigenvector perpendicular to the first one, and subtracting its projection onto the second leaves the third along the same perpendicular line in the plane, unless it already lies along the second.
5

Spectral decomposition into rank-1 projections

Multiplying out the columns of QQ and the rows of QTQ^T rewrites A=QΛQTA = Q\Lambda Q^T as a sum of projection matrices, breaking a symmetric matrix into separate stretches along each eigenvector line.

Remember that while a row-times-column product uiTui=1u_i^T u_i = 1 is a single number, a column-times-row product uiuiTu_i u_i^T of a unit vector uiu_i is an n×nn \times n rank-1 matrix that projects vectors onto the line along uiu_i.

Spectral decomposition into rank-1 projections
QQ
u1\htmlClass{lc-green}{u_1}
u2\htmlClass{lc-pink}{u_2}
Λ\Lambda
λ1\htmlClass{lc-green}{\lambda_1}
00
00
λ2\htmlClass{lc-pink}{\lambda_2}
QTQ^T
u1T\htmlClass{lc-green}{u_1^T}
u2T\htmlClass{lc-pink}{u_2^T}
x\htmlClass{lc-blue}{x}
=  ?= \;?

Multiplying each rank-1 projection matrix uiuiTu_i u_i^T by its eigenvalue λi\lambda_i gives the spectral decomposition of AA:

A=QΛQT=λ1u1u1T⏟rank-1 projectiononto u1+λ2u2u2T⏟rank-1 projectiononto u2+⋯+λnununT⏟rank-1 projectiononto un\displaystyle \begin{aligned} A &= Q\Lambda Q^T \\ &= \eone{\lambda_1} \underbrace{\eone{u_1 u_1^T}}_{\substack{\text{rank-1 projection} \\ \text{onto } \eone{u_1}}} + \etwo{\lambda_2} \underbrace{\etwo{u_2 u_2^T}}_{\substack{\text{rank-1 projection} \\ \text{onto } \etwo{u_2}}} \\ &\quad + \cdots + \lambda_n \underbrace{u_n u_n^T}_{\substack{\text{rank-1 projection} \\ \text{onto } u_n}} \end{aligned}
Text solution
Multiply each unit eigenvector column uiu_i by its transpose row uiTu_i^T to build the rank-1 projection matrices For B=[2112]B = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}, multiplying column-by-row for u1=[1212]\eone{u_1} = \eone{\begin{bmatrix} \tfrac{1}{\sqrt{2}} \\ \tfrac{1}{\sqrt{2}} \end{bmatrix}} and u2=[−1212]\etwo{u_2} = \etwo{\begin{bmatrix} -\tfrac{1}{\sqrt{2}} \\ \tfrac{1}{\sqrt{2}} \end{bmatrix}} gives the two projection matrices onto the 45∘45^\circ axes.
u1u1T=[1212][1212]=[12121212],u2u2T=[−1212][−1212]=[12−12−1212]\displaystyle \begin{aligned} \eone{u_1 u_1^T} &= \eone{\begin{bmatrix} \tfrac{1}{\sqrt{2}} \\ \tfrac{1}{\sqrt{2}} \end{bmatrix}}\eone{\begin{bmatrix} \tfrac{1}{\sqrt{2}} & \tfrac{1}{\sqrt{2}} \end{bmatrix}} \\ &= \eone{\begin{bmatrix} \tfrac{1}{2} & \tfrac{1}{2} \\ \tfrac{1}{2} & \tfrac{1}{2} \end{bmatrix}}, \\ \etwo{u_2 u_2^T} &= \etwo{\begin{bmatrix} -\tfrac{1}{\sqrt{2}} \\ \tfrac{1}{\sqrt{2}} \end{bmatrix}}\etwo{\begin{bmatrix} -\tfrac{1}{\sqrt{2}} & \tfrac{1}{\sqrt{2}} \end{bmatrix}} \\ &= \etwo{\begin{bmatrix} \tfrac{1}{2} & -\tfrac{1}{2} \\ -\tfrac{1}{2} & \tfrac{1}{2} \end{bmatrix}} \end{aligned}
Multiply each projection matrix by its eigenvalue and add them together Multiplying the first projection by λ1=3\eone{\lambda_1} = \eone{3} and the second by λ2=1\etwo{\lambda_2} = \etwo{1} rebuilds BB.
B=3u1u1T+1u2u2T=3[12121212]+1[12−12−1212]=[32+1232−1232−1232+12]=[2112]\displaystyle \begin{aligned} B &= \eone{3} \eone{u_1 u_1^T} + \etwo{1} \etwo{u_2 u_2^T} \\ &= \eone{3}\eone{\begin{bmatrix} \tfrac{1}{2} & \tfrac{1}{2} \\ \tfrac{1}{2} & \tfrac{1}{2} \end{bmatrix}} + \etwo{1}\etwo{\begin{bmatrix} \tfrac{1}{2} & -\tfrac{1}{2} \\ -\tfrac{1}{2} & \tfrac{1}{2} \end{bmatrix}} \\ &= \begin{bmatrix} \eone{\tfrac{3}{2}} + \etwo{\tfrac{1}{2}} & \eone{\tfrac{3}{2}} - \etwo{\tfrac{1}{2}} \\ \eone{\tfrac{3}{2}} - \etwo{\tfrac{1}{2}} & \eone{\tfrac{3}{2}} + \etwo{\tfrac{1}{2}} \end{bmatrix} \\ &= \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \end{aligned}

So multiplying any vector x\vecx{x} by BB projects x\vecx{x} onto each unit eigenvector and stretches each piece by its eigenvalue: Bx=λ1(u1⋅x)u1+λ2(u2⋅x)u2B\vecx{x} = \eone{\lambda_1}(\eone{u_1} \cdot \vecx{x})\eone{u_1} + \etwo{\lambda_2}(\etwo{u_2} \cdot \vecx{x})\etwo{u_2}.

Key concepts

To decide how to handle any problem on the Spectral Theorem, start by checking whether the matrix equals its own transpose, AT=AA^T = A:

Symmetry test (AT=AA^T = A)

A real matrix can be orthogonally diagonalized as A=QΛQTA = Q\Lambda Q^T if and only if its entries match across the main diagonal, like B=[2112]B = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}. If AT≠AA^T \neq A, such as [1423]\begin{bmatrix} 1 & 4 \\ 2 & 3 \end{bmatrix}, you can't find a full set of perpendicular eigenvectors, so at best you can use ordinary diagonalization PDP−1PDP^{-1} instead of QΛQTQ\Lambda Q^T.

Different eigenvalues (λ1≠λ2\eone{\lambda_1} \neq \etwo{\lambda_2}), divide by length directly and skip Gram-Schmidt

Eigenvectors from different eigenvalues of a symmetric matrix are already perpendicular (v1⋅v2=0\eone{v_1} \cdot \etwo{v_2} = 0, like [11]⋅[−11]=0\eone{\begin{bmatrix} 1 \\ 1 \end{bmatrix}} \cdot \etwo{\begin{bmatrix} -1 \\ 1 \end{bmatrix}} = 0). You don't need Gram-Schmidt here, and if you already know all but one eigenvector of a symmetric matrix, you can find the last eigenvector directly by setting its dot product with each of the others to zero, without solving (A−λI)v=0(A - \lambda I)v = 0. Just divide each eigenvector by its length ∥vi∥\|v_i\| to get the unit columns uiu_i of QQ.

Repeated eigenvalues (λ2=λ3\etwo{\lambda_2} = \ethree{\lambda_3}), use Gram-Schmidt inside that eigenspace only

When an eigenvalue repeats, (A−λI)v=0(A - \lambda I)v = 0 gives a plane of eigenvectors (such as x1+x2+x3=0\eigsp{x_1 + x_2 + x_3 = 0}) that is already 90∘90^\circ to eigenvectors from other eigenvalues, but two basis vectors w1,w2\etwo{w_1}, \ethree{w_2} in the plane usually meet at an angle (w1⋅w2≠0\etwo{w_1} \cdot \ethree{w_2} \miss{\neq 0}). Keep v2=w1\etwo{v_2} = \etwo{w_1}, replace w2\ethree{w_2} with v3=w2−w2⋅v2v2⋅v2v2\ethree{v_3} = \ethree{w_2} - \frac{\ethree{w_2} \cdot \etwo{v_2}}{\etwo{v_2} \cdot \etwo{v_2}}\etwo{v_2}, and then divide all three eigenvectors by their lengths.

Spectral decomposition (A=λ1u1u1T+⋯+λnununTA = \eone{\lambda_1} \eone{u_1 u_1^T} + \cdots + \lambda_n u_n u_n^T)

To rebuild AA from its eigenvalues and unit eigenvectors, or to find Ax=∑λi(ui⋅x)uiA\vecx{x} = \sum \lambda_i(u_i \cdot \vecx{x})u_i without finding AA, multiply each rank-1 projection uiuiTu_i u_i^T by λi\lambda_i and add them up (for BB, 3[12121212]+1[12−12−1212]\eone{3}\eone{\begin{bmatrix} \tfrac{1}{2} & \tfrac{1}{2} \\ \tfrac{1}{2} & \tfrac{1}{2} \end{bmatrix}} + \etwo{1}\etwo{\begin{bmatrix} \tfrac{1}{2} & -\tfrac{1}{2} \\ -\tfrac{1}{2} & \tfrac{1}{2} \end{bmatrix}}). Any term with λi=0\lambda_i = 0 is zero and drops out.

6

Practice

1

The matrix A=[06k63060−3]A = \begin{bmatrix} 0 & 6 & k \\ 6 & 3 & 0 \\ 6 & 0 & -3 \end{bmatrix} can be orthogonally diagonalized as A=QΛQTA = Q\Lambda Q^T. If v1=[221]\eone{v_1} = \eone{\begin{bmatrix} 2 \\ 2 \\ 1 \end{bmatrix}} is an eigenvector of AA with eigenvalue λ1=9\eone{\lambda_1} = \eone{9} and v2=[1c2]\etwo{v_2} = \etwo{\begin{bmatrix} 1 \\ c \\ 2 \end{bmatrix}} is an eigenvector with eigenvalue λ2=0\etwo{\lambda_2} = \etwo{0}, find kk and cc without multiplying by AA.

What has to be true about the entries of AA across the main diagonal when A=QΛQTA = Q\Lambda Q^T, and what is the dot product of eigenvectors when λ1≠λ2\eone{\lambda_1} \neq \etwo{\lambda_2}?

By the Spectral Theorem, AA can be written as QΛQTQ\Lambda Q^T if and only if AT=AA^T = A. Matching the top-right entry with the bottom-left entry gives:
k\displaystyle k =6\displaystyle = 6
Since λ1=9\eone{\lambda_1} = \eone{9} and λ2=0\etwo{\lambda_2} = \etwo{0} are different eigenvalues of a symmetric matrix, their eigenvectors are perpendicular, so v1⋅v2=0\eone{v_1} \cdot \etwo{v_2} = 0 (watch out: don't set v1⋅v2\eone{v_1} \cdot \etwo{v_2} equal to the eigenvalue 9\eone{9}, which gives c=52c = \tfrac{5}{2}, and don't drop the minus sign when solving 4+2c=04 + 2c = 0, which gives c=2c = 2):
v1⋅v2\displaystyle \eone{v_1} \cdot \etwo{v_2} =2(1)+2(c)+1(2)\displaystyle = \eone{2}(\etwo{1}) + \eone{2}(\etwo{c}) + \eone{1}(\etwo{2}) =4+2c\displaystyle = 4 + 2c =0\displaystyle = 0
k\displaystyle k =6,c\displaystyle = 6, \qquad c =−2\displaystyle = -2
2

Orthogonally diagonalize A=[4221]A = \begin{bmatrix} 4 & 2 \\ 2 & 1 \end{bmatrix} as A=QΛQTA = Q\Lambda Q^T.

Find the eigenvalues from det⁡(A−λI)=0\det(A - \lambda I) = 0 and one eigenvector for each, then check what you need to do with their lengths before putting them into QQ.

Find the eigenvalues from the characteristic equation:
det⁡(A−λI)=(4−λ)(1−λ)−4=λ2−5λ=λ(λ−5)=0  ⟹  λ1=5,λ2=0\displaystyle \begin{aligned} \det(A - \lambda I) &= (4 - \lambda)(1 - \lambda) - 4 \\ &= \lambda^2 - 5\lambda \\ &= \lambda(\lambda - 5) = 0 \\ &\implies \eone{\lambda_1} = \eone{5}, \quad \etwo{\lambda_2} = \etwo{0} \end{aligned}
Find an eigenvector for λ1=5\eone{\lambda_1} = \eone{5} and a perpendicular eigenvector for λ2=0\etwo{\lambda_2} = \etwo{0}:
(A−5I)v1=[−122−4][x1x2]=[00]  ⟹  v1=[21],v2=[−12]\displaystyle \begin{aligned} (A - 5I)\eone{v_1} &= \begin{bmatrix} -1 & 2 \\ 2 & -4 \end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} \\ &= \begin{bmatrix} 0 \\ 0 \end{bmatrix} \\ &\implies \eone{v_1} = \eone{\begin{bmatrix} 2 \\ 1 \end{bmatrix}}, \\ \etwo{v_2} &= \etwo{\begin{bmatrix} -1 \\ 2 \end{bmatrix}} \end{aligned}
Divide both eigenvectors by their length 22+12=5\sqrt{2^2 + 1^2} = \sqrt{5} to make the columns of QQ unit vectors (putting v1\eone{v_1} and v2\etwo{v_2} into QQ without dividing by 5\sqrt{5} gives Q=[2−112]Q = \begin{bmatrix} 2 & -1 \\ 1 & 2 \end{bmatrix}, which makes QΛQTQ\Lambda Q^T five times too large):
Q=[25−151525],Λ=[5000]\displaystyle Q = \begin{bmatrix} \eone{\tfrac{2}{\sqrt{5}}} & \etwo{-\tfrac{1}{\sqrt{5}}} \\ \eone{\tfrac{1}{\sqrt{5}}} & \etwo{\tfrac{2}{\sqrt{5}}} \end{bmatrix}, \qquad \Lambda = \begin{bmatrix} \eone{5} & 0 \\ 0 & \etwo{0} \end{bmatrix}
3

A 2×22 \times 2 symmetric matrix AA has eigenvalue λ1=10\eone{\lambda_1} = \eone{10} with eigenvector v1=[12]\eone{v_1} = \eone{\begin{bmatrix} 1 \\ 2 \end{bmatrix}}, and its second eigenvalue is λ2=−5\etwo{\lambda_2} = \etwo{-5}. Find the two rank-1 projection matrices u1u1T\eone{u_1 u_1^T} and u2u2T\etwo{u_2 u_2^T}, and use the spectral decomposition A=λ1u1u1T+λ2u2u2TA = \eone{\lambda_1} \eone{u_1 u_1^T} + \etwo{\lambda_2} \etwo{u_2 u_2^T} to find AA.

Use v1⋅v2=0\eone{v_1} \cdot \etwo{v_2} = 0 to find a second eigenvector v2\etwo{v_2} for λ2=−5\etwo{\lambda_2} = \etwo{-5}, and divide both eigenvectors by their lengths before multiplying column-by-row.

Because AA is symmetric and λ1≠λ2\eone{\lambda_1} \neq \etwo{\lambda_2}, a second eigenvector perpendicular to v1=[12]\eone{v_1} = \eone{\begin{bmatrix} 1 \\ 2 \end{bmatrix}} is v2=[−21]\etwo{v_2} = \etwo{\begin{bmatrix} -2 \\ 1 \end{bmatrix}}. Divide both by their length 12+22=5\sqrt{1^2 + 2^2} = \sqrt{5}:
u1=[1525],u2=[−2515]\displaystyle \eone{u_1} = \eone{\begin{bmatrix} \tfrac{1}{\sqrt{5}} \\ \tfrac{2}{\sqrt{5}} \end{bmatrix}}, \qquad \etwo{u_2} = \etwo{\begin{bmatrix} -\tfrac{2}{\sqrt{5}} \\ \tfrac{1}{\sqrt{5}} \end{bmatrix}}
Multiply each unit column by its transpose row to get the rank-1 projection matrices (be sure to divide by 5\sqrt{5} before multiplying: computing v1v1T\eone{v_1 v_1^T} and v2v2T\etwo{v_2 v_2^T} without dividing by 5\sqrt{5} leaves out the 15\tfrac{1}{5} and gives A=[−10303035]A = \begin{bmatrix} -10 & 30 \\ 30 & 35 \end{bmatrix}):
u1u1T=[1525][1525]=[15252545],u2u2T=[−2515][−2515]=[45−25−2515]\displaystyle \begin{aligned} \eone{u_1 u_1^T} &= \eone{\begin{bmatrix} \tfrac{1}{\sqrt{5}} \\ \tfrac{2}{\sqrt{5}} \end{bmatrix}}\eone{\begin{bmatrix} \tfrac{1}{\sqrt{5}} & \tfrac{2}{\sqrt{5}} \end{bmatrix}} \\ &= \eone{\begin{bmatrix} \tfrac{1}{5} & \tfrac{2}{5} \\ \tfrac{2}{5} & \tfrac{4}{5} \end{bmatrix}}, \\ \etwo{u_2 u_2^T} &= \etwo{\begin{bmatrix} -\tfrac{2}{\sqrt{5}} \\ \tfrac{1}{\sqrt{5}} \end{bmatrix}}\etwo{\begin{bmatrix} -\tfrac{2}{\sqrt{5}} & \tfrac{1}{\sqrt{5}} \end{bmatrix}} \\ &= \etwo{\begin{bmatrix} \tfrac{4}{5} & -\tfrac{2}{5} \\ -\tfrac{2}{5} & \tfrac{1}{5} \end{bmatrix}} \end{aligned}
Multiply each projection matrix by its eigenvalue λ1=10\eone{\lambda_1} = \eone{10} and λ2=−5\etwo{\lambda_2} = \etwo{-5} and add them (watch the negative sign on λ2=−5\etwo{\lambda_2} = \etwo{-5}; adding +5u2u2T+5 \etwo{u_2 u_2^T} instead of subtracting gives [6229]\begin{bmatrix} 6 & 2 \\ 2 & 9 \end{bmatrix}):
A=10[15252545]−5[45−25−2515]=[2448]−[4−2−21]=[−2667]\displaystyle \begin{aligned} A &= \eone{10}\eone{\begin{bmatrix} \tfrac{1}{5} & \tfrac{2}{5} \\ \tfrac{2}{5} & \tfrac{4}{5} \end{bmatrix}} - \etwo{5}\etwo{\begin{bmatrix} \tfrac{4}{5} & -\tfrac{2}{5} \\ -\tfrac{2}{5} & \tfrac{1}{5} \end{bmatrix}} \\ &= \eone{\begin{bmatrix} 2 & 4 \\ 4 & 8 \end{bmatrix}} - \etwo{\begin{bmatrix} 4 & -2 \\ -2 & 1 \end{bmatrix}} \\ &= \begin{bmatrix} -2 & 6 \\ 6 & 7 \end{bmatrix} \end{aligned}
4

Orthogonally diagonalize the symmetric matrix A=[31−113−1−1−13]A = \begin{bmatrix} 3 & 1 & -1 \\ 1 & 3 & -1 \\ -1 & -1 & 3 \end{bmatrix}, which has characteristic equation −(λ−5)(λ−2)2=0-(\lambda - 5)(\lambda - 2)^2 = 0.

Solve (A−2I)v=0(A - 2I)v = 0 to find the plane equation for the repeated eigenvalue λ=2\lambda = 2; after picking two basis vectors w1w_1 and w2w_2 in that plane using the free variables, check w1⋅w2w_1 \cdot w_2 before building QQ.

For λ1=5\eone{\lambda_1} = \eone{5}, solving (A−5I)v=0(A - 5I)v = 0 gives v1=[11−1]\eone{v_1} = \eone{\begin{bmatrix} 1 \\ 1 \\ -1 \end{bmatrix}}. For the repeated eigenvalue λ=2\lambda = 2, (A−2I)v=0(A - 2I)v = 0 simplifies to the single plane equation x1+x2−x3=0\eigsp{x_1 + x_2 - x_3 = 0}, where setting (x1,x2)(x_1, x_2) to (1,0)(1, 0) and (0,1)(0, 1) gives two basis eigenvectors:
w1=[101],w2=[011]\displaystyle \etwo{w_1} = \etwo{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}}, \qquad \ethree{w_2} = \ethree{\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}}
Because w1⋅w2=1≠0\etwo{w_1} \cdot \ethree{w_2} = \miss{1 \neq 0}, keep v2=w1\etwo{v_2} = \etwo{w_1} and use Gram-Schmidt inside the λ=2\lambda = 2 plane (don't divide w2\ethree{w_2} by its length and put it straight into the third column of QQ as [01212]\begin{bmatrix} 0 \\ \tfrac{1}{\sqrt{2}} \\ \tfrac{1}{\sqrt{2}} \end{bmatrix}, because w1\etwo{w_1} and w2\ethree{w_2} meet at 60∘60^\circ, not 90∘90^\circ):
v3=w2−w2⋅v2v2⋅v2v2=[011]−12[101]=[−12112]→×2[−121]\displaystyle \begin{aligned} \ethree{v_3} &= \ethree{w_2} - \frac{\ethree{w_2} \cdot \etwo{v_2}}{\etwo{v_2} \cdot \etwo{v_2}}\etwo{v_2} \\ &= \ethree{\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}} - \frac{1}{2}\etwo{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}} \\ &= \ethree{\begin{bmatrix} -\tfrac{1}{2} \\ 1 \\ \tfrac{1}{2} \end{bmatrix}} \xrightarrow{\times 2} \ethree{\begin{bmatrix} -1 \\ 2 \\ 1 \end{bmatrix}} \end{aligned}
Divide v1,v2,v3\eone{v_1}, \etwo{v_2}, \ethree{v_3} by their lengths 3\sqrt{3}, 2\sqrt{2}, and 6\sqrt{6} to build QQ:
Q=[1312−1613026−131216],Λ=[500020002]\displaystyle \begin{aligned} Q &= \begin{bmatrix} \eone{\tfrac{1}{\sqrt{3}}} & \etwo{\tfrac{1}{\sqrt{2}}} & \ethree{-\tfrac{1}{\sqrt{6}}} \\ \eone{\tfrac{1}{\sqrt{3}}} & \etwo{0} & \ethree{\tfrac{2}{\sqrt{6}}} \\ \eone{-\tfrac{1}{\sqrt{3}}} & \etwo{\tfrac{1}{\sqrt{2}}} & \ethree{\tfrac{1}{\sqrt{6}}} \end{bmatrix}, \\ \Lambda &= \begin{bmatrix} \eone{5} & 0 & 0 \\ 0 & \etwo{2} & 0 \\ 0 & 0 & \ethree{2} \end{bmatrix} \end{aligned}
5

A 3×33 \times 3 symmetric matrix AA has eigenvalue λ1=6\eone{\lambda_1} = \eone{6} with eigenvector v1=[1−10]\eone{v_1} = \eone{\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}}, eigenvalue λ2=3\etwo{\lambda_2} = \etwo{3} with eigenvector v2=[111]\etwo{v_2} = \etwo{\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}}, and eigenvalue λ3=0\ethree{\lambda_3} = \ethree{0}. Without finding the entries of AA, find a unit eigenvector u3\ethree{u_3} for λ3=0\ethree{\lambda_3} = \ethree{0}, and compute AxA\vecx{x} for x=[51−2]\vecx{x} = \vecx{\begin{bmatrix} 5 \\ 1 \\ -2 \end{bmatrix}} using projections onto the orthonormal eigenvectors.

Since λ3=0\ethree{\lambda_3} = \ethree{0} is different from λ1=6\eone{\lambda_1} = \eone{6} and λ2=3\etwo{\lambda_2} = \etwo{3}, use v1⋅v3=0\eone{v_1} \cdot \ethree{v_3} = 0 and v2⋅v3=0\etwo{v_2} \cdot \ethree{v_3} = 0 to find v3\ethree{v_3}, then write AxA\vecx{x} as a sum of projections λi(ui⋅x)ui\lambda_i(u_i \cdot \vecx{x})u_i.

Because λ1,λ2,λ3\eone{\lambda_1}, \etwo{\lambda_2}, \ethree{\lambda_3} are different eigenvalues of a symmetric matrix, v3\ethree{v_3} has to be perpendicular to both v1\eone{v_1} and v2\etwo{v_2}:
{v1⋅v3=x1−x2=0v2⋅v3=x1+x2+x3=0  ⟹  v3=[11−2],u3=v3∥v3∥=[1616−26]\displaystyle \begin{aligned} & \begin{cases} \eone{v_1} \cdot \ethree{v_3} = x_1 - x_2 = 0 \\ \etwo{v_2} \cdot \ethree{v_3} = x_1 + x_2 + x_3 = 0 \end{cases} \\ &\implies \ethree{v_3} = \ethree{\begin{bmatrix} 1 \\ 1 \\ -2 \end{bmatrix}}, \\ & \ethree{u_3} = \frac{\ethree{v_3}}{\|\ethree{v_3}\|} = \ethree{\begin{bmatrix} \tfrac{1}{\sqrt{6}} \\ \tfrac{1}{\sqrt{6}} \\ -\tfrac{2}{\sqrt{6}} \end{bmatrix}} \end{aligned}
Divide v1\eone{v_1} and v2\etwo{v_2} by their lengths to get unit vectors u1=12[1−10]\eone{u_1} = \eone{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}} and u2=13[111]\etwo{u_2} = \etwo{\frac{1}{\sqrt{3}}\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}}, then project x=[51−2]\vecx{x} = \vecx{\begin{bmatrix} 5 \\ 1 \\ -2 \end{bmatrix}} onto each (don't use v1\eone{v_1} and v2\etwo{v_2} without dividing by their lengths in 6(v1⋅x)v1+3(v2⋅x)v2\eone{6}(\eone{v_1} \cdot \vecx{x})\eone{v_1} + \etwo{3}(\etwo{v_2} \cdot \vecx{x})\etwo{v_2}, which leaves out dividing by ∥v1∥2=2\|\eone{v_1}\|^2 = 2 and ∥v2∥2=3\|\etwo{v_2}\|^2 = 3 and gives the wrong vector [36−1212]\begin{bmatrix} 36 \\ -12 \\ 12 \end{bmatrix}):
u1⋅x=5−1+02=42,u2⋅x=5+1−23=43\displaystyle \begin{aligned} \eone{u_1} \cdot \vecx{x} &= \frac{5 - 1 + 0}{\sqrt{2}} = \frac{4}{\sqrt{2}}, \\ \etwo{u_2} \cdot \vecx{x} &= \frac{5 + 1 - 2}{\sqrt{3}} = \frac{4}{\sqrt{3}} \end{aligned}
Since λ3=0\ethree{\lambda_3} = \ethree{0} makes the u3\ethree{u_3} term zero, add the first two stretched projections:
Ax=6(u1⋅x)u1+3(u2⋅x)u2+0(u3⋅x)u3=6(42)12[1−10]+3(43)13[111]=12[1−10]+4[111]=[16−84]\displaystyle \begin{aligned} A\vecx{x} &= \eone{6}(\eone{u_1} \cdot \vecx{x})\eone{u_1} + \etwo{3}(\etwo{u_2} \cdot \vecx{x})\etwo{u_2} \\ &\quad + \ethree{0}(\ethree{u_3} \cdot \vecx{x})\ethree{u_3} \\ &= \eone{6}\left(\frac{4}{\sqrt{2}}\right)\eone{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}} \\ &\quad + \etwo{3}\left(\frac{4}{\sqrt{3}}\right)\etwo{\frac{1}{\sqrt{3}}\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}} \\ &= 12\eone{\begin{bmatrix} 1 \\ -1 \\ 0 \end{bmatrix}} + 4\etwo{\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}} \\ &= \begin{bmatrix} 16 \\ -8 \\ 4 \end{bmatrix} \end{aligned}