Linear AlgebraThe Singular Value Decomposition2 of 6
The Singular Value Decomposition · 2

Computing the SVD from A Transpose A

We know that any matrix factors into a rotation or reflection, a stretch, and a second rotation or reflection as A=UΣVTA = U\Sigma V^T, and now we need a way to find the matrices UU, Σ\Sigma, and VV from the numbers in AA.

1

Finding VV and Σ\Sigma from ATAA^T A

To build the SVD, we first want to find the perpendicular unit vectors viv_i in the input space and the singular values σi\sigma_i that tell us how much AA stretches them. We cannot solve Avi=σiuiAv_i = \sigma_i u_i directly like a normal eigenvalue problem because each equation contains two different unknown vectors, viv_i and uiu_i, and AA might not even be square. Taking a round trip through AA and then ATA^T brings viv_i back onto itself and gets uiu_i out of the equation.

Finding V and Sigma from A Transpose A
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}
A\htmlClass{lc-violet}{A}
σ1u1\htmlClass{lc-green}{\sigma_1 u_1}
σ2u2\htmlClass{lc-pink}{\sigma_2 u_2}
U, Σ, V=  ?U,\ \Sigma,\ V = \;?

While AA sends each perpendicular input axis viv_i to σiui\sigma_i u_i on the output ellipse, ATA^T sends uiu_i back to σivi\sigma_i v_i, stretching viv_i by σi\sigma_i a second time:

(ATA)vi\displaystyle (A^T A)v_i =σi2⏟λivi\displaystyle = \underbrace{\sigma_i^2}_{\lambda_i} v_i

No matter what shape AA has, ATAA^T A is always square, symmetric, and positive semidefinite. Its eigenvalues λi\lambda_i are guaranteed to be real and never negative, so we take their square roots in decreasing order to get the singular values and divide the perpendicular eigenvectors of ATAA^T A by their lengths to get the unit columns of VV:

σi\displaystyle \sigma_i =λi,λ1\displaystyle = \sqrt{\lambda_i}, \qquad \first{\lambda_1} ≥λ2\displaystyle \ge \second{\lambda_2} ≥…\displaystyle \ge \dots ≥0\displaystyle \ge 0

As vv turns around the unit circle, the round-trip vector ATAvA^T A v lines up with vv only at the two perpendicular directions v1\first{v_1} and v2\second{v_2} where the output length ∥Av∥\|Av\| reaches its maximum σ1\first{\sigma_1} and minimum σ2\second{\sigma_2}. At those two directions, ATAA^T A scales vv by σ12\first{\sigma_1^2} and σ22\second{\sigma_2^2}.

Interactive c1
2

Computing the SVD of a Square Matrix

Once ATAA^T A gives us VV and Σ\Sigma, we still need the output directions uiu_i to fill UU. Let's find the full SVD A=UΣVTA = U\Sigma V^T for a square matrix where every singular value is nonzero:

A=[3045]\displaystyle A = \begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}
Computing the SVD of a Square Matrix
A=[3045]=U Σ VT\displaystyle A = \begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix} = U\,\Sigma\,V^T
U, Σ, V\displaystyle U,\ \Sigma,\ V =  ?\displaystyle = \;?
Text solution
Eigenvalues of ATAA^T AMultiply ATA^T by AA and solve det⁡(ATA−λI)=0\det(A^T A - \lambda I) = 0, listing the eigenvalues from largest to smallest:
ATA=[3405][3045]=[25202025]det⁡(ATA−λI)=(25−λ)2−400=0  ⟹  λ1=45,λ2=5\displaystyle \begin{aligned} & A^T A = \begin{bmatrix} 3 & 4 \\ 0 & 5 \end{bmatrix}\begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix} \\ &\quad = \begin{bmatrix} 25 & 20 \\ 20 & 25 \end{bmatrix} \\ & \det(A^T A - \lambda I) = (25 - \lambda)^2 \\ &\quad - 400 = 0 \\ &\quad \implies \first{\lambda_1 = 45}, \second{\lambda_2 = 5} \end{aligned}
Square roots of λi\lambda_i for Σ\SigmaTake the square root of each eigenvalue to put the singular values on the diagonal of Σ\Sigma:
σ1=45=35,σ2=5  ⟹  Σ=[35005]\displaystyle \first{\sigma_1 = \sqrt{45} = 3\sqrt{5}}, \quad \second{\sigma_2 = \sqrt{5}} \implies \Sigma = \begin{bmatrix} \first{3\sqrt{5}} & 0 \\ 0 & \second{\sqrt{5}} \end{bmatrix}
Unit eigenvectors of ATAA^T A for VVFind an eigenvector of ATAA^T A for each λi\lambda_i and divide by its length 2\sqrt{2} so the columns of VV are unit vectors:
v1=12[11],v2=12[1−1]  ⟹  V=[121212−12]\displaystyle \begin{aligned} & \first{v_1 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}, \quad \second{v_2 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \end{bmatrix}} \\ & \implies V = \begin{bmatrix} \first{\tfrac{1}{\sqrt{2}}} & \second{\tfrac{1}{\sqrt{2}}} \\[6pt] \first{\tfrac{1}{\sqrt{2}}} & \second{-\tfrac{1}{\sqrt{2}}} \end{bmatrix} \end{aligned}
Multiply viv_i by AA and divide by σi\sigma_i for UURewrite Avi=σiuiAv_i = \sigma_i u_i as ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i} to find where AA sends each viv_i and scale it to length 11:
u1=Av1σ1=135[3045](12[11])=1310[39]=110[13]u2=Av2σ2=15[3045](12[1−1])=110[3−1]\displaystyle \begin{aligned} & \first{u_1} = \frac{A\first{v_1}}{\first{\sigma_1}} \\ & = \frac{1}{\first{3\sqrt{5}}}\begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}\left(\first{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}\right) \\ & = \frac{1}{3\sqrt{10}}\begin{bmatrix} 3 \\ 9 \end{bmatrix} = \first{\frac{1}{\sqrt{10}}\begin{bmatrix} 1 \\ 3 \end{bmatrix}} \\ & \second{u_2} = \frac{A\second{v_2}}{\second{\sigma_2}} \\ & = \frac{1}{\second{\sqrt{5}}}\begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}\left(\second{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \end{bmatrix}}\right) \\ & = \second{\frac{1}{\sqrt{10}}\begin{bmatrix} 3 \\ -1 \end{bmatrix}} \end{aligned}
Transpose VV and put together A=UΣVTA = U\Sigma V^TPut u1\first{u_1} and u2\second{u_2} into the columns of UU and transpose VV at the end:
[3045]=[110310310−110]⏟U[35005]⏟Σ[121212−12]⏟VT\displaystyle \begin{aligned} & \begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix} \\ &\quad = \underbrace{\begin{bmatrix} \first{\tfrac{1}{\sqrt{10}}} & \second{\tfrac{3}{\sqrt{10}}} \\[6pt] \first{\tfrac{3}{\sqrt{10}}} & \second{-\tfrac{1}{\sqrt{10}}} \end{bmatrix}}_{U} \underbrace{\begin{bmatrix} \first{3\sqrt{5}} & 0 \\ 0 & \second{\sqrt{5}} \end{bmatrix}}_{\Sigma} \\ &\qquad \underbrace{\begin{bmatrix} \first{\tfrac{1}{\sqrt{2}}} & \first{\tfrac{1}{\sqrt{2}}} \\[6pt] \second{\tfrac{1}{\sqrt{2}}} & \second{-\tfrac{1}{\sqrt{2}}} \end{bmatrix}}_{V^T} \end{aligned}

Here VTV^T and UU both have determinant −1-1, so AA reflects the input plane by VTV^T, stretches the axes by 35\first{3\sqrt{5}} and 5\second{\sqrt{5}}, and reflects them onto u1\first{u_1} and u2\second{u_2} by UU.

3

When a Singular Value Is Zero

The formula ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i} works whenever σi\sigma_i is nonzero, but it does not work if a singular value is zero because we cannot divide by zero. Here is a matrix with identical columns that squashes the entire 2D unit circle flat onto a 1D line:

A=[3344]\displaystyle A = \begin{bmatrix} 3 & 3 \\ 4 & 4 \end{bmatrix}
When a Singular Value Is Zero
A=[3344]A = \begin{bmatrix} 3 & 3 \\ 4 & 4 \end{bmatrix}
UU
Σ\Sigma
VV
Text solution
Eigenvalues of ATAA^T A and Σ\SigmaSince ATAA^T A has trace 5050 and determinant 00, its eigenvalues are 50\first{50} and 0\second{0}, giving one positive singular value and one zero singular value:
ATA=[3434][3344]=[25252525]  ⟹  λ1=50,λ2=0σ1=50=52,σ2=0  ⟹  Σ=[52000]\displaystyle \begin{aligned} & A^T A = \begin{bmatrix} 3 & 4 \\ 3 & 4 \end{bmatrix}\begin{bmatrix} 3 & 3 \\ 4 & 4 \end{bmatrix} \\ &\quad = \begin{bmatrix} 25 & 25 \\ 25 & 25 \end{bmatrix} \\ &\quad \implies \first{\lambda_1 = 50}, \second{\lambda_2 = 0} \\ & \first{\sigma_1} = \first{\sqrt{50} = 5\sqrt{2}}, \\ & \second{\sigma_2 = 0} \implies \Sigma = \begin{bmatrix} \first{5\sqrt{2}} & 0 \\ 0 & \second{0} \end{bmatrix} \end{aligned}
Unit eigenvectors of ATAA^T A for VVDivide the eigenvectors for λ1=50\first{\lambda_1 = 50} and λ2=0\second{\lambda_2 = 0} by their lengths to get unit vectors, noting that v2\second{v_2} is in the null space of AA because Av2=[00]A\second{v_2} = \lost{\begin{bmatrix} 0 \\ 0 \end{bmatrix}}:
v1=12[11],v2=12[1−1]\displaystyle \first{v_1 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}, \quad \second{v_2 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \end{bmatrix}}
Find u1\first{u_1} from Av1σ1\tfrac{A\first{v_1}}{\first{\sigma_1}}Use σ1=52\first{\sigma_1 = 5\sqrt{2}} to find the unit vector along the 1D output line:
u1=Av1σ1=152[3344](12[11])=110[68]=15[34]\displaystyle \first{u_1} = \frac{A\first{v_1}}{\first{\sigma_1}} = \frac{1}{\first{5\sqrt{2}}}\begin{bmatrix} 3 & 3 \\ 4 & 4 \end{bmatrix}\left(\first{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}\right) = \frac{1}{10}\begin{bmatrix} 6 \\ 8 \end{bmatrix} = \first{\frac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix}}
Choose u2\second{u_2} perpendicular to u1\first{u_1}Because Av2σ2\tfrac{A\second{v_2}}{\second{\sigma_2}} gives 00\lost{\tfrac{0}{0}}, swap the entries of u1\first{u_1} and flip one sign to get a unit vector perpendicular to u1\first{u_1}:
u2=15[−43]  ⟹  [3344]=[35−454535][52000][121212−12]\displaystyle \begin{aligned} & \second{u_2 = \frac{1}{5}\begin{bmatrix} -4 \\ 3 \end{bmatrix}} \\ \implies & \begin{bmatrix} 3 & 3 \\ 4 & 4 \end{bmatrix} \\ & \quad = \begin{bmatrix} \first{\tfrac{3}{5}} & \second{-\tfrac{4}{5}} \\[6pt] \first{\tfrac{4}{5}} & \second{\tfrac{3}{5}} \end{bmatrix}\begin{bmatrix} \first{5\sqrt{2}} & 0 \\ 0 & \second{0} \end{bmatrix} \\ & \qquad \begin{bmatrix} \first{\tfrac{1}{\sqrt{2}}} & \first{\tfrac{1}{\sqrt{2}}} \\[6pt] \second{\tfrac{1}{\sqrt{2}}} & \second{-\tfrac{1}{\sqrt{2}}} \end{bmatrix} \end{aligned}

Because σ2=0\second{\sigma_2 = 0} multiplies u2\second{u_2} by zero in UΣVTU\Sigma V^T, every output lands along u1\first{u_1}, and either plus or minus u2\second{u_2} gives the exact same product AA.

4

Computing the SVD of a Rectangular Matrix

What if AA is not square, so the input and output spaces have different dimensions? For an m×nm \times n matrix AA, the input matrix VV is n×nn \times n, the output matrix UU is m×mm \times m, and Σ\Sigma has the same m×nm \times n shape as AA.

AA
3×23 \times 2
==
UU
u1\htmlClass{lc-green}{u_1}
u2\htmlClass{lc-pink}{u_2}
u3\htmlClass{lc-violet}{u_3}
3×33 \times 3
Σ\Sigma
σ1\htmlClass{lc-green}{\sigma_1}
00
00
σ2\htmlClass{lc-pink}{\sigma_2}
00
00
3×23 \times 2
VTV^T
v1T\htmlClass{lc-green}{v_1^T}
v2T\htmlClass{lc-pink}{v_2^T}
2×22 \times 2

For this 3×23 \times 2 matrix, ATAA^T A is 2×22 \times 2 and gives only two vv vectors, while UU needs three perpendicular columns in R3\mathbb{R}^3:

A=[111−111]\displaystyle A = \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}
Computing the SVD of a Rectangular Matrix
A=[111−111]A = \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}
3×23 \times 2
R2\mathbb{R}^2
A\htmlClass{lc-violet}{A}
R3\mathbb{R}^3
AA
==
UU
Σ\Sigma
VTV^T
3×23 \times 2
?×?\htmlClass{lc-mute}{{?} \times {?}}
?×?\htmlClass{lc-mute}{{?} \times {?}}
?×?\htmlClass{lc-mute}{{?} \times {?}}
Text solution
Eigenvalues of ATAA^T A and the 3×23 \times 2 matrix Σ\SigmaCompute the 2×22 \times 2 matrix ATAA^T A, put the square roots of its eigenvalues on the diagonal of Σ\Sigma, and fill the extra bottom row with zeros:
ATA=[1111−11][111−111]=[3113]  ⟹  λ1=4,λ2=2σ1=2,σ2=2  ⟹  Σ=[200200]\displaystyle \begin{aligned} & A^T A \\ &\quad = \begin{bmatrix} 1 & 1 & 1 \\ 1 & -1 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix} \\ &\quad = \begin{bmatrix} 3 & 1 \\ 1 & 3 \end{bmatrix} \\ &\quad \implies \first{\lambda_1 = 4}, \second{\lambda_2 = 2} \\ & \first{\sigma_1} = \first{2}, \\ & \second{\sigma_2 = \sqrt{2}} \implies \Sigma \\ &\quad = \begin{bmatrix} \first{2} & 0 \\ 0 & \second{\sqrt{2}} \\ 0 & 0 \end{bmatrix} \end{aligned}
Unit eigenvectors of ATAA^T A for VVDivide the eigenvectors of ATAA^T A by their lengths to fill the 2×22 \times 2 matrix VV:
v1=12[11],v2=12[1−1]\displaystyle \first{v_1 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}, \quad \second{v_2 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \end{bmatrix}}
Find u1\first{u_1} and u2\second{u_2} from Aviσi\tfrac{Av_i}{\sigma_i}Divide Av1A\first{v_1} by σ1\first{\sigma_1} and Av2A\second{v_2} by σ2\second{\sigma_2} to get the two axes of the flat 2D ellipse inside R3\mathbb{R}^3:
u1=Av1σ1=12[111−111](12[11])=122[202]=12[101]u2=Av2σ2=12[111−111](12[1−1])=12[020]=[010]\displaystyle \begin{aligned} \first{u_1} &= \frac{A\first{v_1}}{\first{\sigma_1}} \\ &= \frac{1}{\first{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}\left(\first{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}\right) \\ &= \frac{1}{2\sqrt{2}}\begin{bmatrix} 2 \\ 0 \\ 2 \end{bmatrix} = \first{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}} \\ \second{u_2} &= \frac{A\second{v_2}}{\second{\sigma_2}} \\ &= \frac{1}{\second{\sqrt{2}}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix}\left(\second{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \end{bmatrix}}\right) \\ &= \frac{1}{2}\begin{bmatrix} 0 \\ 2 \\ 0 \end{bmatrix} = \second{\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}} \end{aligned}
Find u3\third{u_3} perpendicular to both u1\first{u_1} and u2\second{u_2}Pick a unit vector u3\third{u_3} with u1Tu3=0\first{u_1^T} \third{u_3} = 0 and u2Tu3=0\second{u_2^T} \third{u_3} = 0 to finish the 3×33 \times 3 matrix UU:
u3=12[10−1]  ⟹  [111−111]=[12012010120−12][200200][121212−12]\displaystyle \begin{aligned} & \third{u_3 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}} \\ \implies & \begin{bmatrix} 1 & 1 \\ 1 & -1 \\ 1 & 1 \end{bmatrix} \\ & = \begin{bmatrix} \first{\tfrac{1}{\sqrt{2}}} & \second{0} & \third{\tfrac{1}{\sqrt{2}}} \\[6pt] \first{0} & \second{1} & \third{0} \\[6pt] \first{\tfrac{1}{\sqrt{2}}} & \second{0} & \third{-\tfrac{1}{\sqrt{2}}} \end{bmatrix} \\ & \quad \begin{bmatrix} \first{2} & 0 \\ 0 & \second{\sqrt{2}} \\ 0 & 0 \end{bmatrix}\begin{bmatrix} \first{\tfrac{1}{\sqrt{2}}} & \first{\tfrac{1}{\sqrt{2}}} \\[6pt] \second{\tfrac{1}{\sqrt{2}}} & \second{-\tfrac{1}{\sqrt{2}}} \end{bmatrix} \end{aligned}

Here u1\first{u_1} and u2\second{u_2} span the 2D plane of the output ellipse in R3\mathbb{R}^3, while u3\third{u_3} points perpendicular to that plane and gets multiplied by the bottom row of zeros in Σ\Sigma.

When u1\first{u_1} and u2\second{u_2} have zeros in their entries, you can often see right away what u3\third{u_3} needs to be so the dot products are zero, or you can always solve the equations u1Tx=0\first{u_1^T} \third{x} = 0 and u2Tx=0\second{u_2^T} \third{x} = 0 (or run Gram-Schmidt). If AA is wide instead of tall, such as 2×32 \times 3, then ATAA^T A is 3×33 \times 3 with at least one zero eigenvalue, which gives the third column v3\third{v_3} directly as an eigenvector in the null space of AA.

As you move u3\third{u_3} around the 3D unit sphere, its projections onto u1\first{u_1} and u2\second{u_2} drop to zero only when u3\third{u_3} points straight out of the plane of the ellipse. That gives three perpendicular unit vectors to form the 3D coordinate system UU.

A third unit vector completes a square 3D corner only when it points straight out of the ellipse's plane, at right angles to the first two.
5

The SVD and the Four Fundamental Subspaces

The vectors we found when σi=0\lost{\sigma_i = 0} or when AA was rectangular are not just extra columns to fill out the matrices. Splitting VV and UU right after the r\kept{r} nonzero singular values gives an orthonormal basis for each of the four fundamental subspaces of AA.

The SVD and the Four Fundamental Subspaces
Am×n=U Σ VTΣ=[σ1⋱σr0⋱]\displaystyle \begin{aligned} \kern0.6em\underset{\mathclap{m \times n}}{A} &= U\,\Sigma\,V^T \\[4pt] \Sigma &= \begin{bmatrix} \kept{\sigma_1} & & & & \\ & \mathclap{\kept{\ddots}} & & & \\ & & \kept{\sigma_r} & & \\ & & & \lost{0} & \\ & & & & \lost{\ddots} \end{bmatrix} \end{aligned}
rank⁡(A)\displaystyle \operatorname{rank}(A) =  ?\displaystyle = \;?

Because the orthogonal matrices UU and VTV^T are rotations, possibly combined with a reflection, that keep every dimension intact, all the squashing in A=UΣVTA = U\Sigma V^T happens inside Σ\Sigma. If Σ\Sigma has r\kept{r} nonzero singular values on its diagonal, then r\kept{r} dimensions survive:

rank(A)\displaystyle \text{rank}(A) =r(σ1≥⋯≥σr>0)\displaystyle = \kept{r} \qquad (\kept{\sigma_1 \ge \dots \ge \sigma_r > 0})
V=V =
r\htmlClass{lc-blue}{r}
v1v_1
⋯\cdots
vrv_r
Row Space\htmlClass{lc-blue}{\text{Row Space}}
C(AT)\htmlClass{lc-blue}{C(A^T)}
n−rn - r
vr+1v_{r+1}
⋯\cdots
vnv_n
Null Space\htmlClass{lc-orange}{\text{Null Space}}
N(A)\htmlClass{lc-orange}{N(A)}
U=U =
r\htmlClass{lc-blue}{r}
u1u_1
⋯\cdots
uru_r
Column Space\htmlClass{lc-blue}{\text{Column Space}}
C(A)\htmlClass{lc-blue}{C(A)}
m−rm - r
ur+1u_{r+1}
⋯\cdots
umu_m
Left Null Space\htmlClass{lc-orange}{\text{Left Null Space}}
N(AT)\htmlClass{lc-orange}{N(A^T)}

Splitting VV and UU after column r\kept{r} separates the directions that survive from the directions that get squashed to zero or missed by AA:

V=[ v1,…,vr⏟row space C(AT)  ∣  vr+1,…,vn⏟null space N(A) ],U=[ u1,…,ur⏟column space C(A)  ∣  ur+1,…,um⏟left null space N(AT) ]\displaystyle \begin{aligned} V &= \Big[\,\underbrace{v_1, \dots, v_r}_{\text{row space } \kept{C(A^T)}} \;\Big|\; \underbrace{v_{r+1}, \dots, v_n}_{\text{null space } \lost{N(A)}}\,\Big], \\ U &= \Big[\,\underbrace{u_1, \dots, u_r}_{\text{column space } \kept{C(A)}} \;\Big|\; \underbrace{u_{r+1}, \dots, u_m}_{\text{left null space } \lost{N(A^T)}}\,\Big] \end{aligned}

For the rank-1 matrix A=[3344]A = \begin{bmatrix} 3 & 3 \\ 4 & 4 \end{bmatrix}, moving the input x\vecx{x} along the null space direction v2\second{v_2} does not change AxA\vecx{x} at all, because AA squashes v2\second{v_2} to zero. Only the part of x\vecx{x} along the row space direction v1\first{v_1} survives: it gets stretched by σ1\first{\sigma_1} along the column space direction u1\first{u_1}, and never reaches the perpendicular left null space direction u2\second{u_2}.

Only an input’s row-space component stays to get stretched along the column space, while its null-space component gets squashed to zero and nothing ever reaches the left null space.

Key concepts

To factor an m×nm \times n matrix into A=UΣVTA = U\Sigma V^T, begin with the round-trip matrix ATAA^T A. Put its eigenvalues λ1≥λ2≥⋯≥0\first{\lambda_1} \ge \second{\lambda_2} \ge \dots \ge 0 in decreasing order: their square roots σi=λi\sigma_i = \sqrt{\lambda_i} go on the diagonal of the m×nm \times n matrix Σ\Sigma (filled with zeros to match the shape of AA), and the unit eigenvectors v1,…,vnv_1, \dots, v_n fill the columns of the n×nn \times n matrix VV.

How you find each column uiu_i of the m×mm \times m matrix UU depends on whether it has a nonzero singular value:

When σi>0\kept{\sigma_i > 0}, use ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i}

Compute ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i}, like u1=135Av1\first{u_1} = \frac{1}{\first{3\sqrt{5}}}A\first{v_1}. Do not find uiu_i by solving AATAA^T separately when σi>0\kept{\sigma_i > 0}, because choosing eigenvector signs on their own can flip a sign so that Avi=−σiuiAv_i = \wrong{-}\sigma_i u_i.

When σi=0\lost{\sigma_i = 0} or UU needs more columns than VV (m>nm > n), use perpendicularity

You cannot use Aviσi\tfrac{Av_i}{\sigma_i} (which either divides 0\lost{0} by 0\lost{0} or has no matching viv_i). Instead, pick unit vectors perpendicular to the uu columns you already have, by swapping entries and flipping one sign in 2D (15[34]→15[−43]\first{\frac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix}} \to \second{\frac{1}{5}\begin{bmatrix} -4 \\ 3 \end{bmatrix}}) or solving uiTx=0u_i^T x = 0 in 3D.

Once UU, Σ\Sigma, and VV are built, the number of nonzero singular values r\kept{r} is rank(A)\text{rank}(A). Splitting VV and UU after column r\kept{r} gives you orthonormal bases for all four fundamental subspaces: v1,…,vrv_1, \dots, v_r for the row space C(AT)\kept{C(A^T)}, vr+1,…,vnv_{r+1}, \dots, v_n for the null space N(A)\lost{N(A)}, u1,…,uru_1, \dots, u_r for the column space C(A)\kept{C(A)}, and ur+1,…,umu_{r+1}, \dots, u_m for the left null space N(AT)\lost{N(A^T)}.

6

Practice

1

For A=[122−2]A = \begin{bmatrix} 1 & 2 \\ 2 & -2 \end{bmatrix}, the matrix ATA=[5−2−28]A^T A = \begin{bmatrix} 5 & -2 \\ -2 & 8 \end{bmatrix} gives singular values σ1=3\first{\sigma_1 = 3}, σ2=2\second{\sigma_2 = 2} and unit eigenvectors v1=15[1−2]\first{v_1 = \frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ -2 \end{bmatrix}}, v2=15[21]\second{v_2 = \frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}}. A student solves AATAA^T separately and picks u1=15[1−2]\first{u_1 = \frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ -2 \end{bmatrix}} and u2=15[21]\second{u_2 = \frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}}. Use ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i} to check whether these signs work and write the correct matrix UU.

Multiply AA by v1\first{v_1} first: does Av1A\first{v_1} point in the same direction as the student's u1\first{u_1}, or the opposite direction?

Using ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i} on v1\first{v_1} and v2\second{v_2} gives:
u1=Av13=135[122−2][1−2]=135[−36]=15[−12]\displaystyle \first{u_1} = \frac{A\first{v_1}}{\first{3}} = \frac{1}{3\sqrt{5}}\begin{bmatrix} 1 & 2 \\ 2 & -2 \end{bmatrix}\first{\begin{bmatrix} 1 \\ -2 \end{bmatrix}} = \frac{1}{3\sqrt{5}}\begin{bmatrix} -3 \\ 6 \end{bmatrix} = \first{\frac{1}{\sqrt{5}}\begin{bmatrix} -1 \\ 2 \end{bmatrix}}
u2=Av22=125[122−2][21]=125[42]=15[21]\displaystyle \second{u_2} = \frac{A\second{v_2}}{\second{2}} = \frac{1}{2\sqrt{5}}\begin{bmatrix} 1 & 2 \\ 2 & -2 \end{bmatrix}\second{\begin{bmatrix} 2 \\ 1 \end{bmatrix}} = \frac{1}{2\sqrt{5}}\begin{bmatrix} 4 \\ 2 \end{bmatrix} = \second{\frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}}
(Warning: Because AA is symmetric, AATAA^T equals ATAA^T A, which makes it tempting to copy VV directly into U=[1525−2515]U = \begin{bmatrix} \tfrac{1}{\sqrt{5}} & \tfrac{2}{\sqrt{5}} \\[6pt] -\tfrac{2}{\sqrt{5}} & \tfrac{1}{\sqrt{5}} \end{bmatrix}; that gives Av1=−σ1u1A\first{v_1} = \wrong{-}\first{\sigma_1 u_1} because the student's u1\first{u_1} has the wrong sign.) The correct matrix is:
U=[−15252515]\displaystyle U = \begin{bmatrix} \first{-\tfrac{1}{\sqrt{5}}} & \second{\tfrac{2}{\sqrt{5}}} \\[6pt] \first{\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix}
2

Compute the full SVD A=UΣVTA = U\Sigma V^T of the matrix A=[22−12]A = \begin{bmatrix} 2 & 2 \\ -1 & 2 \end{bmatrix}.

Start by computing ATAA^T A and listing its eigenvalues λ1,λ2\first{\lambda_1}, \second{\lambda_2} in decreasing order before finding the unit eigenvectors v1,v2\first{v_1}, \second{v_2}.

Compute ATAA^T A and find its eigenvalues:
ATA=[2−122][22−12]=[5228]  ⟹  λ1=9,λ2=4\displaystyle A^T A = \begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix}\begin{bmatrix} 2 & 2 \\ -1 & 2 \end{bmatrix} = \begin{bmatrix} 5 & 2 \\ 2 & 8 \end{bmatrix} \implies \first{\lambda_1 = 9}, \quad \second{\lambda_2 = 4}
Take square roots to get σ1=3\first{\sigma_1 = 3} and σ2=2\second{\sigma_2 = 2}, so Σ=[3002]\Sigma = \begin{bmatrix} \first{3} & 0 \\ 0 & \second{2} \end{bmatrix}. (Warning: Don't put the eigenvalues λ1=9,λ2=4\first{\lambda_1 = 9}, \second{\lambda_2 = 4} directly into Σ=[9004]\Sigma = \begin{bmatrix} 9 & 0 \\ 0 & 4 \end{bmatrix} without taking their square roots.) The unit eigenvectors of ATAA^T A are:
v1=15[12],v2=15[−21]  ⟹  V=[15−252515]\displaystyle \first{v_1 = \frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ 2 \end{bmatrix}}, \quad \second{v_2 = \frac{1}{\sqrt{5}}\begin{bmatrix} -2 \\ 1 \end{bmatrix}} \implies V = \begin{bmatrix} \first{\tfrac{1}{\sqrt{5}}} & \second{-\tfrac{2}{\sqrt{5}}} \\[6pt] \first{\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix}
Using ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i} on v1\first{v_1} and v2\second{v_2} gives:
u1=Av13=15[21],u2=Av22=15[−12]  ⟹  U=[25−151525]\displaystyle \first{u_1} = \frac{A\first{v_1}}{\first{3}} = \first{\frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}}, \quad \second{u_2} = \frac{A\second{v_2}}{\second{2}} = \second{\frac{1}{\sqrt{5}}\begin{bmatrix} -1 \\ 2 \end{bmatrix}} \implies U = \begin{bmatrix} \first{\tfrac{2}{\sqrt{5}}} & \second{-\tfrac{1}{\sqrt{5}}} \\[6pt] \first{\tfrac{1}{\sqrt{5}}} & \second{\tfrac{2}{\sqrt{5}}} \end{bmatrix}
A=[25−151525][3002][1525−2515]\displaystyle A = \begin{bmatrix} \first{\tfrac{2}{\sqrt{5}}} & \second{-\tfrac{1}{\sqrt{5}}} \\[6pt] \first{\tfrac{1}{\sqrt{5}}} & \second{\tfrac{2}{\sqrt{5}}} \end{bmatrix}\begin{bmatrix} \first{3} & 0 \\ 0 & \second{2} \end{bmatrix}\begin{bmatrix} \first{\tfrac{1}{\sqrt{5}}} & \first{\tfrac{2}{\sqrt{5}}} \\[6pt] \second{-\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix}
3

Compute the full SVD A=UΣVTA = U\Sigma V^T of the matrix A=[1−22−4]A = \begin{bmatrix} 1 & -2 \\ 2 & -4 \end{bmatrix}.

One eigenvalue of ATAA^T A is zero. How do you find u2\second{u_2} once you have u1=Av1σ1\first{u_1} = \tfrac{A\first{v_1}}{\first{\sigma_1}}?

Compute ATAA^T A and its eigenvalues:
ATA=[12−2−4][1−22−4]=[5−10−1020]  ⟹  λ1=25,λ2=0\displaystyle A^T A = \begin{bmatrix} 1 & 2 \\ -2 & -4 \end{bmatrix}\begin{bmatrix} 1 & -2 \\ 2 & -4 \end{bmatrix} = \begin{bmatrix} 5 & -10 \\ -10 & 20 \end{bmatrix} \implies \first{\lambda_1 = 25}, \quad \second{\lambda_2 = 0}
So σ1=5\first{\sigma_1 = 5}, σ2=0\second{\sigma_2 = 0}, and Σ=[5000]\Sigma = \begin{bmatrix} \first{5} & 0 \\ 0 & \second{0} \end{bmatrix}. The unit eigenvectors of ATAA^T A are:
v1=15[1−2],v2=15[21]  ⟹  V=[1525−2515]\displaystyle \first{v_1 = \frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ -2 \end{bmatrix}}, \quad \second{v_2 = \frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}} \implies V = \begin{bmatrix} \first{\tfrac{1}{\sqrt{5}}} & \second{\tfrac{2}{\sqrt{5}}} \\[6pt] \first{-\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix}
Compute u1=Av15=15[12]\first{u_1} = \tfrac{A\first{v_1}}{\first{5}} = \first{\frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ 2 \end{bmatrix}}, and swap entries with one sign flip to get a perpendicular unit vector u2=15[−21]\second{u_2 = \frac{1}{\sqrt{5}}\begin{bmatrix} -2 \\ 1 \end{bmatrix}} (or its negative). Now transpose VV to put together A=UΣVTA = U\Sigma V^T (Warning: Leaving VV untransposed puts v1\first{v_1} in the first column instead of the first row of VTV^T, which multiplies out to [1224]\begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix} instead of AA):
A=[15−252515][5000][15−252515]\displaystyle A = \begin{bmatrix} \first{\tfrac{1}{\sqrt{5}}} & \second{-\tfrac{2}{\sqrt{5}}} \\[6pt] \first{\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix}\begin{bmatrix} \first{5} & 0 \\ 0 & \second{0} \end{bmatrix}\begin{bmatrix} \first{\tfrac{1}{\sqrt{5}}} & \first{-\tfrac{2}{\sqrt{5}}} \\[6pt] \second{\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix}
4

Compute the full SVD A=UΣVTA = U\Sigma V^T of the 3×23 \times 2 matrix A=[101101]A = \begin{bmatrix} 1 & 0 \\ 1 & 1 \\ 0 & 1 \end{bmatrix}.

Compare the shapes of AA, UU, Σ\Sigma, and VTV^T first: how many columns does UU need after you find u1\first{u_1} and u2\second{u_2} from v1\first{v_1} and v2\second{v_2}?

The 2×22 \times 2 matrix ATAA^T A is:
ATA=[2112]  ⟹  λ1=3,λ2=1  ⟹  σ1=3,σ2=1\displaystyle A^T A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \implies \first{\lambda_1 = 3}, \quad \second{\lambda_2 = 1} \implies \first{\sigma_1 = \sqrt{3}}, \quad \second{\sigma_2 = 1}
Filling the extra bottom row of Σ\Sigma with zeros to match the 3×23 \times 2 shape of AA gives Σ=[300100]\Sigma = \begin{bmatrix} \first{\sqrt{3}} & 0 \\ 0 & \second{1} \\ 0 & 0 \end{bmatrix}. The unit eigenvectors of ATAA^T A are v1=12[11]\first{v_1 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}} and v2=12[1−1]\second{v_2 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ -1 \end{bmatrix}}, so V=[121212−12]V = \begin{bmatrix} \first{\tfrac{1}{\sqrt{2}}} & \second{\tfrac{1}{\sqrt{2}}} \\[6pt] \first{\tfrac{1}{\sqrt{2}}} & \second{-\tfrac{1}{\sqrt{2}}} \end{bmatrix}. Next, find u1\first{u_1} and u2\second{u_2} using ui=Aviσiu_i = \tfrac{Av_i}{\sigma_i} (Warning: Remember to combine both 13\frac{1}{\first{\sqrt{3}}} from σ1\first{\sigma_1} and 12\frac{1}{\sqrt{2}} from v1\first{v_1} into 16\frac{1}{\sqrt{6}}, rather than writing 12[121]\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix} or 13[121]\frac{1}{\sqrt{3}}\begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}):
u1=Av13=16[121],u2=Av21=12[10−1]\displaystyle \first{u_1} = \frac{A\first{v_1}}{\first{\sqrt{3}}} = \first{\frac{1}{\sqrt{6}}\begin{bmatrix} 1 \\ 2 \\ 1 \end{bmatrix}}, \quad \second{u_2} = \frac{A\second{v_2}}{\second{1}} = \second{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 0 \\ -1 \end{bmatrix}}
(Warning: Don't stop at two columns and leave UU as 3×23 \times 2; a full SVD needs a 3×33 \times 3 orthogonal matrix UU.) Solving u1Tx=0\first{u_1^T} \third{x} = 0 and u2Tx=0\second{u_2^T} \third{x} = 0 gives the third unit vector u3=13[1−11]\third{u_3 = \frac{1}{\sqrt{3}}\begin{bmatrix} 1 \\ -1 \\ 1 \end{bmatrix}} (or its negative), so:
A=[161213260−1316−1213][300100][121212−12]\displaystyle \begin{aligned} & A = \begin{bmatrix} \first{\tfrac{1}{\sqrt{6}}} & \second{\tfrac{1}{\sqrt{2}}} & \third{\tfrac{1}{\sqrt{3}}} \\[6pt] \first{\tfrac{2}{\sqrt{6}}} & \second{0} & \third{-\tfrac{1}{\sqrt{3}}} \\[6pt] \first{\tfrac{1}{\sqrt{6}}} & \second{-\tfrac{1}{\sqrt{2}}} & \third{\tfrac{1}{\sqrt{3}}} \end{bmatrix} \\ & \qquad \begin{bmatrix} \first{\sqrt{3}} & 0 \\ 0 & \second{1} \\ 0 & 0 \end{bmatrix}\begin{bmatrix} \first{\tfrac{1}{\sqrt{2}}} & \first{\tfrac{1}{\sqrt{2}}} \\[6pt] \second{\tfrac{1}{\sqrt{2}}} & \second{-\tfrac{1}{\sqrt{2}}} \end{bmatrix} \end{aligned}
5

Find the full SVD A=UΣVTA = U\Sigma V^T of the 2×32 \times 3 matrix A=[200011]A = \begin{bmatrix} 2 & 0 & 0 \\ 0 & 1 & 1 \end{bmatrix}, and use UU, Σ\Sigma, and VV to state the rank of AA and orthonormal bases for all four fundamental subspaces.

Since AA is 2×32 \times 3, ATAA^T A is 3×33 \times 3 and gives three vv vectors. Split the columns of VV and UU right after the last nonzero singular value.

Compute the 3×33 \times 3 matrix ATAA^T A and find its eigenvalues:
ATA=[400011011]  ⟹  λ1=4,λ2=2,λ3=0\displaystyle A^T A = \begin{bmatrix} 4 & 0 & 0 \\ 0 & 1 & 1 \\ 0 & 1 & 1 \end{bmatrix} \implies \first{\lambda_1 = 4}, \quad \second{\lambda_2 = 2}, \quad \third{\lambda_3 = 0}
Taking square roots gives σ1=2\first{\sigma_1 = 2}, σ2=2\second{\sigma_2 = \sqrt{2}}, σ3=0\third{\sigma_3 = 0}, and the 2×32 \times 3 matrix Σ=[200020]\Sigma = \begin{bmatrix} \first{2} & 0 & 0 \\ 0 & \second{\sqrt{2}} & 0 \end{bmatrix}. The unit eigenvectors of ATAA^T A are:
v1=[100],v2=12[011],v3=12[01−1]  ⟹  V=[10001212012−12]\displaystyle \begin{aligned} & \first{v_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}}, \quad \second{v_2 = \frac{1}{\sqrt{2}}\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}}, \\ & \third{v_3 = \frac{1}{\sqrt{2}}\begin{bmatrix} 0 \\ 1 \\ -1 \end{bmatrix}} \\ & \implies V = \begin{bmatrix} \first{1} & \second{0} & \third{0} \\[6pt] \first{0} & \second{\tfrac{1}{\sqrt{2}}} & \third{\tfrac{1}{\sqrt{2}}} \\[6pt] \first{0} & \second{\tfrac{1}{\sqrt{2}}} & \third{-\tfrac{1}{\sqrt{2}}} \end{bmatrix} \end{aligned}
Multiplying v1\first{v_1} and v2\second{v_2} by AA and dividing by σi\sigma_i gives u1=Av12=[10]\first{u_1} = \frac{A\first{v_1}}{\first{2}} = \first{\begin{bmatrix} 1 \\ 0 \end{bmatrix}} and u2=Av22=[01]\second{u_2} = \frac{A\second{v_2}}{\second{\sqrt{2}}} = \second{\begin{bmatrix} 0 \\ 1 \end{bmatrix}}, which already fills the 2×22 \times 2 matrix U=[1001]U = \begin{bmatrix} \first{1} & \second{0} \\ \first{0} & \second{1} \end{bmatrix}, so:
A=[1001][200020][10001212012−12]\displaystyle A = \begin{bmatrix} \first{1} & \second{0} \\ \first{0} & \second{1} \end{bmatrix}\begin{bmatrix} \first{2} & 0 & 0 \\ 0 & \second{\sqrt{2}} & 0 \end{bmatrix}\begin{bmatrix} \first{1} & \first{0} & \first{0} \\[6pt] \second{0} & \second{\tfrac{1}{\sqrt{2}}} & \second{\tfrac{1}{\sqrt{2}}} \\[6pt] \third{0} & \third{\tfrac{1}{\sqrt{2}}} & \third{-\tfrac{1}{\sqrt{2}}} \end{bmatrix}
(Warning: Don't swap the roles of UU and VV or put v3\third{v_3} into the left null space. The viv_i vectors are in the input space R3\mathbb{R}^3 and split into the row space and null space, while the uiu_i vectors are in the output space R2\mathbb{R}^2 and split into the column space and left null space.) Because there are r=2\kept{r} = 2 nonzero singular values:
rank(A)=2\text{rank}(A) = 2
Row space C(AT)\kept{C(A^T)} orthonormal basis: {v1,v2}\{\first{v_1}, \second{v_2}\}
Null space N(A)\lost{N(A)} orthonormal basis: {v3}\{\third{v_3}\}
Column space C(A)\kept{C(A)} orthonormal basis: {u1,u2}\{\first{u_1}, \second{u_2}\}
Left null space N(AT)\lost{N(A^T)}: {0}\lost{\{0\}}