Linear AlgebraSymmetric Matrices and Quadratic Forms1 of 4
Symmetric Matrices and Quadratic Forms · 1
The Spectral Theorem
Diagonalizing a matrix as A=PDP−1 turns it into simple stretching along its eigenvector axes, and we want to know when those axes sit at right angles so we never have to invert P.
1
A stretching matrix at a 45∘ angle
We want to see what the matrix B=[2112], which mixes x and y coordinates together, has in common with the simple diagonal stretching matrix A=[3001].
A stretching matrix in disguise
AB=[3001]=[2112]A10B10=?=?
Testing the two 45∘ diagonal directions shows that B keeps both vectors on their own lines:
So B performs the exact same stretch by 3 and 1 as the diagonal matrix A, just along perpendicular axes tilted by 45∘.
2
The Spectral Theorem: why symmetric matrices have perpendicular eigenvectors
For most matrices, the eigenvectors in A=PDP−1 meet at a slanted angle, so we have to compute P−1 to find a vector's coordinates. The Spectral Theorem tells us that the matrices with a full set of perpendicular eigenvectors are the symmetric matrices: matrices that equal their own transpose, AT=A.
The Spectral Theorem: why symmetric matrices have perpendicular eigenvectors
When we divide perpendicular eigenvectors by their lengths to make them unit vectors and put them into the columns of an orthogonal matrix Q, its inverse is simply its transpose (Q−1=QT, so QT has those same unit eigenvectors as its rows):
To see why different eigenvalues of a symmetric matrix always give perpendicular eigenvectors, use the transpose rule for dot products, (Au)⋅v=(Au)Tv=uTATv. Because AT=A, moving a symmetric matrix from one vector to the other inside a dot product changes nothing:
(Au)⋅v=u⋅(Av)
Using this on our eigenvectors v1 (with λ1=3) and v2 (with λ2=1) forces their dot product to be zero:
On a 90∘ grid of unit axes, a vector's coordinate along each axis is just its dot-product shadow on that axis, whereas on a slanted grid, moving along one axis changes the shadow on the other.
Dot-product shadows add right back to the vector when the unit axes meet at a right angle, and miss when the axes tilt because one axis casts a shadow onto the other.
3
Orthogonal diagonalization step by step
Now let's factor B=[2112] into B=QΛQT and see how the three matrices act on an input vector x.
Orthogonal diagonalization step by step
Text solution
Divide each perpendicular eigenvector by its length to make unit vectors So that the dot products don't multiply coordinates by the eigenvector length 12+12=2, we divide v1 and v2 by 2 to give them length 1.
Put the unit eigenvectors into the columns of Q, their transposes into the rows of QT, and the eigenvalues into Λ Putting each column of Q in the same spot as its eigenvalue in Λ finishes the factorization without inverting a matrix.
B=Q: rebuild in standard space[2121−2121]Λ: stretch[3001]QT: measure projections[21−212121]
Always remember to divide the eigenvectors by their lengths before writing Q: perpendicular columns that don't have length 1 won't give Q−1=QT.
Because u1 and u2 have length 1,QT and Q purely rotate by 45∘ without changing any lengths, leaving all the stretching to the diagonal matrix Λ in the middle.
Because the unit eigenvectors have length 1, the outer steps are pure 45 degree rotations and the diagonal matrix in the middle does all the stretching.
4
Repeated eigenvalues and Gram-Schmidt in an eigenspace plane
When a symmetric matrix has a repeated eigenvalue, the Spectral Theorem guarantees we still get a full set of eigenvectors. For a double eigenvalue, we get a whole 2D plane of eigenvectors rather than a single line. That whole plane is automatically 90∘ to eigenvectors from other eigenvalues, but the two basis vectors we pick inside the plane usually meet at an angle other than 90∘. Let's orthogonally diagonalize a 3×3 symmetric matrix with a repeated eigenvalue:
A=211121112
Repeated eigenvalues and Gram-Schmidt in an eigenspace plane
A=211121112=QΛQT
Q=?
Because every vector in an eigenspace plane gets stretched by the same number λ, any combination of eigenvectors in that plane stays in the plane and is still an eigenvector. When our two basis vectors w1 and w2 in the plane aren't perpendicular (w1⋅w2=0), we keep v2=w1 and use Gram-Schmidt to subtract the projection of w2 onto v2:
v3=w2−projection of w2 onto v2v2⋅v2w2⋅v2v2
Text solution
Factor the characteristic polynomial to find the eigenvalues Solving det(A−λI)=0 gives one single eigenvalue and one repeated eigenvalue.
−(λ−4)(λ−1)2=0⟹λ1=4,λ2=1,λ3=1
Find the eigenvector for λ1=4 and two basis eigenvectors in the λ=1 plane Solving (A−4I)v=0 gives v1=111, while (A−1I)v=0 simplifies to the single plane equation x1+x2+x3=0; setting the free variables (x2,x3) to (1,0) and (0,1) gives two eigenvectors in that plane that meet at 60∘.
Subtract the projection of w2 onto v2=w1 to get a perpendicular eigenvector in the plane Keeping v2=−110, Gram-Schmidt subtracts the part of w2 along v2, and multiplying by 2 clears the fractions before we divide by the length.
So v3=−1−12 stays inside the λ=1 plane (−1−1+2=0) and sits at 90∘ to both v2 in the plane and v1 perpendicular to the plane.
Because every vector in the λ=1 plane is an eigenvector and is already 90∘ to v1, subtracting the projection onto v2 keeps v3 in the plane while making all three eigenvectors perpendicular.
Every direction in the eigenspace plane is an eigenvector perpendicular to the first one, and subtracting its projection onto the second leaves the third along the same perpendicular line in the plane, unless it already lies along the second.
5
Spectral decomposition into rank-1 projections
Multiplying out the columns of Q and the rows of QT rewrites A=QΛQT as a sum of projection matrices, breaking a symmetric matrix into separate stretches along each eigenvector line.
Remember that while a row-times-column product uiTui=1 is a single number, a column-times-row product uiuiT of a unit vector ui is an n×n rank-1 matrix that projects vectors onto the line along ui.
Spectral decomposition into rank-1 projections
Multiplying each rank-1 projection matrix uiuiT by its eigenvalue λi gives the spectral decomposition of A:
Multiply each unit eigenvector column ui by its transpose row uiT to build the rank-1 projection matrices For B=[2112], multiplying column-by-row for u1=[2121] and u2=[−2121] gives the two projection matrices onto the 45∘ axes.
So multiplying any vector x by B projects x onto each unit eigenvector and stretches each piece by its eigenvalue: Bx=λ1(u1⋅x)u1+λ2(u2⋅x)u2.
Key concepts
To decide how to handle any problem on the Spectral Theorem, start by checking whether the matrix equals its own transpose, AT=A:
Symmetry test (AT=A)
A real matrix can be orthogonally diagonalized as A=QΛQT if and only if its entries match across the main diagonal, like B=[2112]. If AT=A, such as [1243], you can't find a full set of perpendicular eigenvectors, so at best you can use ordinary diagonalization PDP−1 instead of QΛQT.
Different eigenvalues (λ1=λ2), divide by length directly and skip Gram-Schmidt
Eigenvectors from different eigenvalues of a symmetric matrix are already perpendicular (v1⋅v2=0, like [11]⋅[−11]=0). You don't need Gram-Schmidt here, and if you already know all but one eigenvector of a symmetric matrix, you can find the last eigenvector directly by setting its dot product with each of the others to zero, without solving (A−λI)v=0. Just divide each eigenvector by its length ∥vi∥ to get the unit columns ui of Q.
Repeated eigenvalues (λ2=λ3), use Gram-Schmidt inside that eigenspace only
When an eigenvalue repeats, (A−λI)v=0 gives a plane of eigenvectors (such as x1+x2+x3=0) that is already 90∘ to eigenvectors from other eigenvalues, but two basis vectors w1,w2 in the plane usually meet at an angle (w1⋅w2=0). Keep v2=w1, replace w2 with v3=w2−v2⋅v2w2⋅v2v2, and then divide all three eigenvectors by their lengths.
To rebuild A from its eigenvalues and unit eigenvectors, or to find Ax=∑λi(ui⋅x)ui without finding A, multiply each rank-1 projection uiuiT by λi and add them up (for B,3[21212121]+1[21−21−2121]). Any term with λi=0 is zero and drops out.
6
Practice
1
The matrix A=066630k0−3 can be orthogonally diagonalized as A=QΛQT. If v1=221 is an eigenvector of A with eigenvalue λ1=9 and v2=1c2 is an eigenvector with eigenvalue λ2=0, find k and c without multiplying by A.
What has to be true about the entries of A across the main diagonal when A=QΛQT, and what is the dot product of eigenvectors when λ1=λ2?
By the Spectral Theorem, A can be written as QΛQT if and only if AT=A. Matching the top-right entry with the bottom-left entry gives:
k=6
Since λ1=9 and λ2=0 are different eigenvalues of a symmetric matrix, their eigenvectors are perpendicular, so v1⋅v2=0 (watch out: don't set v1⋅v2 equal to the eigenvalue 9, which gives c=25, and don't drop the minus sign when solving 4+2c=0, which gives c=2):
v1⋅v2=2(1)+2(c)+1(2)=4+2c=0
k=6,c=−2
2
Orthogonally diagonalize A=[4221] as A=QΛQT.
Find the eigenvalues from det(A−λI)=0 and one eigenvector for each, then check what you need to do with their lengths before putting them into Q.
Find the eigenvalues from the characteristic equation:
Divide both eigenvectors by their length 22+12=5 to make the columns of Q unit vectors (putting v1 and v2 into Q without dividing by 5 gives Q=[21−12], which makes QΛQT five times too large):
Q=[5251−5152],Λ=[5000]
3
A 2×2 symmetric matrix A has eigenvalue λ1=10 with eigenvector v1=[12], and its second eigenvalue is λ2=−5. Find the two rank-1 projection matrices u1u1T and u2u2T, and use the spectral decomposition A=λ1u1u1T+λ2u2u2T to find A.
Use v1⋅v2=0 to find a second eigenvector v2 for λ2=−5, and divide both eigenvectors by their lengths before multiplying column-by-row.
Because A is symmetric and λ1=λ2, a second eigenvector perpendicular to v1=[12] is v2=[−21]. Divide both by their length 12+22=5:
u1=[5152],u2=[−5251]
Multiply each unit column by its transpose row to get the rank-1 projection matrices (be sure to divide by 5 before multiplying: computing v1v1T and v2v2T without dividing by 5 leaves out the 51 and gives A=[−10303035]):
Multiply each projection matrix by its eigenvalue λ1=10 and λ2=−5 and add them (watch the negative sign on λ2=−5; adding +5u2u2T instead of subtracting gives [6229]):
Orthogonally diagonalize the symmetric matrix A=31−113−1−1−13, which has characteristic equation −(λ−5)(λ−2)2=0.
Solve (A−2I)v=0 to find the plane equation for the repeated eigenvalue λ=2; after picking two basis vectors w1 and w2 in that plane using the free variables, check w1⋅w2 before building Q.
For λ1=5, solving (A−5I)v=0 gives v1=11−1. For the repeated eigenvalue λ=2,(A−2I)v=0 simplifies to the single plane equation x1+x2−x3=0, where setting (x1,x2) to (1,0) and (0,1) gives two basis eigenvectors:
w1=101,w2=011
Because w1⋅w2=1=0, keep v2=w1 and use Gram-Schmidt inside the λ=2 plane (don't divide w2 by its length and put it straight into the third column of Q as 02121, because w1 and w2 meet at 60∘, not 90∘):
A 3×3 symmetric matrix A has eigenvalue λ1=6 with eigenvector v1=1−10, eigenvalue λ2=3 with eigenvector v2=111, and eigenvalue λ3=0. Without finding the entries of A, find a unit eigenvector u3 for λ3=0, and compute Ax for x=51−2 using projections onto the orthonormal eigenvectors.
Don't use v1 and v2 without dividing by their lengths in 6(v1⋅x)v1+3(v2⋅x)v2, which leaves out dividing by ∥v1∥2=2 and ∥v2∥2=3 and gives the wrong vector 36−1212.
Since λ3=0 is different from λ1=6 and λ2=3, use v1⋅v3=0 and v2⋅v3=0 to find v3, then write Ax as a sum of projections λi(ui⋅x)ui.
Because λ1,λ2,λ3 are different eigenvalues of a symmetric matrix, v3 has to be perpendicular to both v1 and v2:
Divide v1 and v2 by their lengths to get unit vectors u1=211−10 and u2=31111, then project x=51−2 onto each (don't use v1 and v2 without dividing by their lengths in 6(v1⋅x)v1+3(v2⋅x)v2, which leaves out dividing by ∥v1∥2=2 and ∥v2∥2=3 and gives the wrong vector 36−1212):
u1⋅xu2⋅x=25−1+0=24,=35+1−2=34
Since λ3=0 makes the u3 term zero, add the first two stretched projections: