Linear AlgebraThe Singular Value Decomposition1 of 6
The Singular Value Decomposition · 1
The SVD: Rotate, Stretch, Rotate
We want to break any matrix into a simple stretch along perpendicular axes, even when the matrix twists space or isn't square.
1
Why Eigenvectors Aren't Enough
When a square matrix A is symmetric, the Spectral Theorem breaks it into perpendicular stretches along its unit eigenvectors qi:
A=QΛQT
Once a matrix is no longer symmetric, like A=[22−12], or no longer square, eigenvectors stop giving us perpendicular axes.
Why Eigenvectors Aren't Enough
Eigenvectors fall short in three ways:
Tilted axes: A non-symmetric matrix can have eigenvectors that meet at non-right angles (A=XΛX−1).
Rotations: A matrix that twists the plane, such as A=[22−12] with λ=2±2i, rotates every vector off its line and has no real eigenvectors at all.
Rectangular matrices: An m×n matrix maps inputs in Rn to outputs in Rm, so an output vector cannot even lie on the same line as its input vector.
2
Circle to Ellipse: Perpendicular Inputs to Perpendicular Outputs
Instead of asking for directions that stay on their own line (Ax=λx), we look for perpendicular unit vectors v1 and v2 in the input space that land on perpendicular unit vectors u1 and u2 in the output space.
Circle to Ellipse: Perpendicular Inputs to Perpendicular Outputs
Every invertible 2×2 matrix maps the unit circle to an ellipse whose major and minor axes are perpendicular. If we call the unit vectors along those semi-axes u1 and u2 and the semi-axis lengths σ1 and σ2, the unit inputs v1 and v2 that land on the ends of the semi-axes are also perpendicular to each other:
Av1=σ1u1,Av2=σ2u2
Almost every perpendicular pair on the unit circle lands at a non-right angle on the ellipse. The outputs are perpendicular only when the inputs line up with v1 and v2, landing on the ellipse's longest and shortest semi-axes σ1u1 and σ2u2.
Two perpendicular arrows on the input circle stay perpendicular on the output ellipse only when they land on its longest and shortest axes.
3
Rotate, Stretch, Rotate: A=UΣVT
The Singular Value Decomposition (SVD) breaks the action of A on any input vector x into three steps: rotate the input axes to the standard axes, stretch along the standard axes, and rotate to the output axes.
When we put the orthonormal vectors v1,v2 as the columns of an orthogonal matrix V, its inverse is its transpose (V−1=VT). Multiplying VTx takes the dot product of each row viT with x to find how much of x goes along v1 and v2. Strictly, VT is a reflection rather than a rotation when v2 is clockwise from v1, and the same goes for U with u1 and u2; a reflection still keeps every length and right angle.
Rotate, Stretch, Rotate: A = UΣVᵀ
Putting Av1=σ1u1 and Av2=σ2u2 together as the columns of AV=UΣ and multiplying on the right by VT gives the Singular Value Decomposition:
Even when A tilts and shears the plane, Ax is just a simple stretch by σ1 and σ2 along the standard axes once VT rotates v1 and v2 onto the standard axes and U rotates them onto u1 and u2.
Every matrix transformation is just a rotation of the input projections onto the standard axes, a stretch along those axes, and a final rotation onto the output axes.
4
Why v1 and v2 Are Perpendicular: ATA
To find the perpendicular input directions vi and the singular values σi of a matrix A, we need a symmetric matrix that keeps each vi on its own line without rotating it to ui.
Why v₁ and v₂ Are Perpendicular: AᵀA
Applying A and then AT cancels the output rotation because UTU=I:
ATA=(VΣTUT)(UΣVT)=VΣ2VT⟹(ATA)vi=σi2vi
Because ATA is always symmetric, it always has perpendicular unit eigenvectors v1,v2, and A sends them to perpendicular outputs: Av1⋅Av2=v1T(ATA)v2=σ22(v1⋅v2)=0. Notice that the eigenvalues of ATA are σi2, so we take their square roots to get the singular values σi.
Text solution
Find σi and vi from the symmetric matrix ATAFor A=[22−12], multiplying ATA gives eigenvalues σ12=9 and σ22=4 with orthonormal eigenvectors v1 and v2.
Compute the output directions ui from Avi=σiuiApplying A to each input vector vi and dividing by σi gives the matching unit semi-axis directions u1 and u2.
Eigenvalues λi measure stretch only along directions that stay on their own line, while singular values σi measure the true longest and shortest stretches of the unit circle even when those directions rotate.
Singular Values vs. Eigenvalues
For a shear, eigenvalues miss the stretch because the directions of greatest and least stretch rotate off their starting lines. Only the horizontal eigenvector line stays put with λ=1, while the singular values σ1 and σ2 give the true longest and shortest semi-axes of the sheared ellipse.
As a shear stretches the unit circle into a longer ellipse, its only eigenvalue stays at 1 on the horizontal axis while the singular values track the longest and shortest stretches.
When A is symmetric (A=QΛQT), its eigenvectors qi are already perpendicular. If every eigenvalue is positive and they're listed from largest to smallest, A=QΛQT is already the SVD with U=V=Q and Σ=Λ. If an eigenvalue is negative, we can turn QΛQT into an SVD without computing ATA:
Text solution
Take absolute values for the singular valuesFor A=Q[400−3]QT, the semi-axis lengths are the absolute values σi=∣λi∣.
σ1=4,σ2=∣−3∣=3⟹Σ=[4003]
Flip the output vector sign when λi<0Stretching q2 by −3 is the same as stretching by +3 in the opposite direction −q2, so v2=q2 and u2=−q2.
Aq2=−3q2=3(−q2)⟹V=[q1q2],U=[q1−q2]
6
Area and the Determinant from the SVD
For any square matrix A, the absolute value of the determinant ∣det(A)∣ tells us how much A scales area in 2D or volume in nD.
Area and the Determinant from the SVD
Because the rotations VT and U keep area the same (∣det(U)∣=∣det(VT)∣=1), all the area scaling comes from the diagonal stretch Σ:
For A=[22−12], the unit square with sides v1 and v2 becomes a 3×2 rectangle of area σ1σ2=6, matching ∣det(A)∣=∣(2)(2)−(−1)(2)∣=6.
7
The SVD for Rectangular Matrices
When A is an m×n matrix mapping Rn to Rm, the factorization A=UΣVT still works: U is m×m in the output space, VT is n×n in the input space, and Σ has the same m×n rectangular shape as A.
The SVD for Rectangular Matrices
m×nA=?×?U?×?Σ?×?VT
2×3[∙∙∙∙∙∙]3×2∙∙∙∙∙∙
m×nA=m×morthonormal u1,…,umin output space RmUm×nsame shape as AΣn×northonormal v1,…,vnin input space RnVT
v1
v2
A
σ1u1
σ2u2
u3
Wide matrix (2×3, mapping R3→R2): Σ=[σ100σ200] has a third column of zeros, which multiplies the v3 coordinate by 0 and flattens the 3D unit sphere onto a filled 2D ellipse.
Tall matrix (3×2, mapping R2→R3): Σ=σ1000σ20 has a bottom row of zeros, placing the 2D ellipse in R3 along u1 and u2 while u3 points perpendicular to the plane of the ellipse.
8
Zero Singular Values, Rank, and the Four Subspaces
Any matrix, square or rectangular, crushes an input direction vi to the zero vector whenever its singular value is σi=0. The number of nonzero singular values σ1≥σ2≥⋯≥σr>0 is the rank r of the matrix.
Zero Singular Values, Rank, and the Four Subspaces
AAv2=U[5000]VT=?
v1
v2
When we replace Σ=[3002] from our earlier example with Σ=[5000], we get the rank-1 matrix A=[2412], which flattens the unit circle into a 1D line segment along u1:
Text solution
Follow the direction that survives (i=1)Because σ1=5>0,A stretches v1 to 5u1, so v1 is an orthonormal basis for the Row Space C(AT) and u1 is an orthonormal basis for the Column Space C(A).
Av1=5u1⟹Row Space C(AT):{v1=51[21]},Column Space C(A):{u1=51[12]}
Follow the crushed and unreached directions (i=2)Because σ2=0,v2 gets crushed to zero and spans the Null Space N(A), while u2 is perpendicular to the column space and spans the Left Null Space N(AT).
Av2=0⟹Null Space N(A):{v2=51[−12]},Left Null Space N(AT):{u2=51[−21]}
For any m×n matrix of rank r, splitting the columns of V and U right after the r nonzero singular values gives orthonormal bases for all four fundamental subspaces at once:
V=[Row Space C(AT)Avi=σiuiv1…vrNull Space N(A)Avi=0vr+1…vn],U=[Column Space C(A)reachable outputsu1…urLeft Null Space N(AT)unreachable outputsur+1…um]
Moving the input vector x along v2 doesn't change Ax because the entire Null Space part gets crushed to zero. Only the Row Space part along v1 survives, stretched by σ1 onto the Column Space line along u1.
Only the Row Space part of the input survives to set the output on the Column Space line, while its Null Space part is crushed to zero.
9
What the SVD Lets Us Do
Because every matrix factors into a rotation, a stretch along the axes, and a second rotation, we can compress matrices by dropping tiny σi, solve or best-approximate non-invertible systems by dividing only by the nonzero σi, and pick out the main directions of variation in data.
What the SVD Unlocks
A=UrotateΣstretchVTrotate
v1
v2
Key concepts
Every m×n matrix A factors as A=UΣVT, turning perpendicular unit inputs vi in Rn into perpendicular unit outputs ui in Rm with Avi=σiui and singular values σ1≥σ2≥⋯≥σr>0. In UΣVTx, first VT takes dot products vi⋅x to rotate onto the standard axes, then Σ stretches the first r coordinates by σi and sets the rest to zero, and finally U rotates the result onto ui.
How you find the SVD depends on what you already have:
When A is symmetric and you already have A=QΛQT, you don't need ATA
Take the singular values directly as σi=∣λi∣ sorted from largest to smallest, set vi=qi, and flip the output sign ui=−qi whenever λi<0 (for example, Aq=−7q=7(−q) gives σ=7,v=q, and u=−q).
When A is non-symmetric or rectangular, use ATA=VΣTΣVT
Find the eigenvalues σi2 and orthonormal eigenvectors vi of ATA, take square roots σi=σi2 in decreasing order, and compute each output vector from ui=σi1Avi (for A=[22−12],ATA=[8225] has eigenvalues 9 and 4, giving σ1=3 and σ2=2).
When A is 2×2 and one singular value is already known, use ∣det(A)∣=σ1σ2
Because rotations keep area the same, the product of the singular values equals ∣det(A)∣ (for A=[2032] with σ1=4,∣det(A)∣=4 gives σ2=1 without computing ATA).
The shape and rank of Σ show what the matrix does geometrically: U is m×m,Σ is m×n, and VT is n×n. The number of nonzero singular values is rank(A)=r, which splits the columns of V into orthonormal bases for the Row Space {v1,…,vr} and Null Space {vr+1,…,vn}, and splits the columns of U into orthonormal bases for the Column Space {u1,…,ur} and Left Null Space {ur+1,…,um}.
10
Practice
1
A 2×2 matrix A has the SVD A=UΣVT with u1=21[11],u2=21[−11],Σ=[6002],v1=51[34], and v2=51[−43]. (a) When A maps the unit circle to an ellipse, what are the ends of the ellipse's major and minor semi-axes, and which unit input vectors land on them? (b) Follow the input vector x=[211] step by step through VTx,ΣVTx, and UΣVTx to find Ax.
The semi-axes point along the output vectors u1 and u2 (be careful not to stretch the input vectors v1 and v2 into σ1v1=51[1824] and σ2v2=51[−86]).
(a) Which vectors point along the ellipse in the output space and get stretched by σ1 and σ2? (b) The rows of VT are v1T and v2T, so take their dot products with x first.
(a) From Avi=σiui, the semi-axes point along the output vectors u1 and u2 (be careful not to stretch the input vectors v1 and v2 into σ1v1=51[1824] and σ2v2=51[−86]). The end of the major semi-axis is
σ1u1=6(21[11])=[3232],which comes from v1=51[34]
and the end of the minor semi-axis is
σ2u2=2(21[−11])=[−22],which comes from v2=51[−43]
(b) First, put v1T and v2T as the rows of VT (forgetting to transpose V and multiplying Vx instead would give 51[−3841] and lead to [−312−5732]):
For A=[122−2], the symmetric matrix ATA=[5−2−28] has eigenvalues 9 and 4 with unit eigenvectors 51[1−2] and 51[21]. Find the singular values σ1 and σ2, the orthonormal output vectors u1 and u2, and write the full SVD A=UΣVT.
Take the square roots of the eigenvalues (don't leave σ1=9 and σ2=4 without taking the square root).
The eigenvalues of ATA are σi2, not σi. Once you have σi and vi, use Avi=σiui to solve for ui.
From (ATA)vi=σi2vi with σ1≥σ2, take the square roots of the eigenvalues (don't leave σ1=9 and σ2=4 without taking the square root):
A symmetric matrix A has the orthogonal diagonalization A=QΛQT with Q=[q1q2]=51[34−43] and Λ=[200−7]. Why is QΛQT not an SVD of A, and what are U,Σ, and VT in a valid SVD A=UΣVT?
We have to both take the absolute value and reorder the diagonal (simply taking the absolute value in place gives Σ=[2007], which is out of order).
Check both rules for Σ: can a singular value be negative, and in what order must σ1 and σ2 appear? When you swap their order, remember to swap the corresponding vectors too.
QΛQT is not an SVD because Λ has a negative entry (−7) and the diagonal entries are not ordered from largest to smallest in absolute value (2<∣−7∣).
Since σi=∣λi∣ with σ1≥σ2, we have to both take the absolute value and reorder the diagonal to σ1=7 and σ2=2 (simply taking the absolute value in place gives Σ=[2007], which is out of order, and reordering Σ without swapping q2 into the first column and flipping the sign of u1 will not equal A):
Σ=[7002]
For σ1=∣−7∣=7, the equation Aq2=−7q2=7(−q2) gives v1=q2=51[−43] and flips the sign of the output vector to u1=−q2=51[4−3]. For σ2=2>0, we keep v2=u2=q1=51[34]. This gives:
U=51[4−334],Σ=[7002],VT=51[−4334]
4
Consider the matrix A=[2032]. (a) What are the eigenvalues of A, and why don't they equal the singular values of A? (b) When A acts on the unit circle, the longest semi-axis of the resulting ellipse has length σ1=4. Without computing ATA, find the shortest semi-axis length σ2 and the area of the rectangle that A makes from the unit square with sides v1 and v2.
(a) How does the 3 in the top-right corner affect the directions of greatest and least stretch? (b) How does ∣det(A)∣ connect to σ1 and σ2?
(a) Because A is upper triangular, its eigenvalues are λ1=2 and λ2=2. They do not equal the singular values because A shears the plane, rotating the directions of greatest and least stretch off their starting lines instead of keeping them along fixed eigenvector lines.
(b) Using ∣det(A)∣=∣(2)(2)−(3)(0)∣=4 and ∣det(A)∣=σ1σ2 (don't set σ2 equal to the eigenvalue λ2=2, which would give an area of 8):
4=4σ2⟹σ2=1
The unit square with sides v1 and v2 gets rotated and stretched into a 4×1 rectangle of area σ1σ2=4.
5
Suppose A is a 2×4 matrix with singular values σ1=5 and σ2=3, and B is a 4×2 matrix with the same singular values σ1=5 and σ2=3. (a) Give the dimensions of U,Σ, and VT for A and for B, and write out the full matrix Σ for each. (b) Explain geometrically what the zero columns in A's Σ do to the unit sphere in R4, and what the zero rows in B's Σ mean for the ellipse that B makes in R4.
Match U (m×m), Σ (m×n), and VT (n×n) to the dimensions of each matrix (don't swap the sizes of U and VT).
(a) Match U to the output dimension, VT to the input dimension, and Σ to the exact shape of the matrix. (b) Follow what gets multiplied by 0 in ΣVTx vs. what gets extra zeros added to it before U acts.
(a) Match U (m×m), Σ (m×n), and VT (n×n) to the dimensions of each matrix (don't write Σ as a 2×2 matrix [5003] by leaving off the zero columns or rows, and don't swap the sizes of U and VT):
For A (2×4): U is 2×2,VT is 4×4, and Σ is 2×4:
ΣA=[50030000]
For B (4×2): U is 4×4,VT is 2×2, and Σ is 4×2:
ΣB=50000300
(b) In A, the two zero columns of ΣA multiply the third and fourth rotated input coordinates by 0, crushing the input directions v3 and v4 in R4 to zero and flattening the 4D unit sphere onto a filled 2D ellipse in R2. In B, the two zero rows at the bottom of ΣB put zeros in the third and fourth coordinates before U acts, so B maps the 2D unit circle to a 2D ellipse in R4 spanned by u1 and u2, while u3 and u4 are unit vectors in R4 perpendicular to the plane of that ellipse.
6
A 3×3 matrix A has the SVD A=UΣVT, where U=[u1u2u3] and V=[v1v2v3] are 3×3 orthogonal matrices and Σ=400000000. (a) What is the rank of A, what geometric shape does A make from the 3D unit sphere, and which columns of U and V form orthonormal bases for each of the Four Fundamental Subspaces of A? (b) For b=12u1+cu2, for what value of c does Ax=b have a solution, and for that c, what are all vectors x (written in terms of v1,v2,v3) that satisfy Ax=b?
(a) Count the nonzero singular values to find the rank r, then split V (inputs) and U (outputs) after the r-th column. (b) Write x=x1v1+x2v2+x3v3 and apply A to each term.
(a) A has 1 nonzero singular value (σ1=4), so rank(A)=1. Because A crushes v2 and v3 to 0 and stretches v1 to 4u1, it flattens the 3D unit sphere into a 1D line segment along u1. Splitting the input columns of V and output columns of U at r=1 (don't swap vi and ui between the input and output subspaces) gives the orthonormal bases:
Row Space C(AT):{v1}
Null Space N(A):{v2,v3}
Column Space C(A):{u1}
Left Null Space N(AT):{u2,u3}
(b) Because C(A) is spanned by u1 and u2 lies in the Left Null Space, b=12u1+cu2 is in the column space if and only if c=0. Writing x=x1v1+x2v2+x3v3 and applying A gives:
Ax=x1(4u1)+x2(0)+x3(0)=4x1u1=12u1⟹x1=3
Since v2 and v3 get crushed to 0,x2 and x3 can be any real numbers (don't stop at the single vector x=3v1 and forget the Null Space vectors), giving all solutions: