Linear AlgebraThe Singular Value Decomposition1 of 6
The Singular Value Decomposition · 1

The SVD: Rotate, Stretch, Rotate

We want to break any matrix into a simple stretch along perpendicular axes, even when the matrix twists space or isn't square.

1

Why Eigenvectors Aren't Enough

When a square matrix AA is symmetric, the Spectral Theorem breaks it into perpendicular stretches along its unit eigenvectors qiq_i:

A\displaystyle A =QΛQT\displaystyle = Q\Lambda Q^T

Once a matrix is no longer symmetric, like A=[2−122]A = \begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix}, or no longer square, eigenvectors stop giving us perpendicular axes.

Why Eigenvectors Aren't Enough
A=[2−122]A = \begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix}
ı^\htmlClass{lc-green}{\hat{\imath}}
ȷ^\htmlClass{lc-pink}{\hat{\jmath}}

Eigenvectors fall short in three ways:

2

Circle to Ellipse: Perpendicular Inputs to Perpendicular Outputs

Instead of asking for directions that stay on their own line (Ax=λxA\vecx{x} = \lambda \vecx{x}), we look for perpendicular unit vectors v1\first{v_1} and v2\second{v_2} in the input space that land on perpendicular unit vectors u1\first{u_1} and u2\second{u_2} in the output space.

Circle to Ellipse: Perpendicular Inputs to Perpendicular Outputs
AA
?\large ?

Every invertible 2×22 \times 2 matrix maps the unit circle to an ellipse whose major and minor axes are perpendicular. If we call the unit vectors along those semi-axes u1\first{u_1} and u2\second{u_2} and the semi-axis lengths σ1\first{\sigma_1} and σ2\second{\sigma_2}, the unit inputs v1\first{v_1} and v2\second{v_2} that land on the ends of the semi-axes are also perpendicular to each other:

Av1\displaystyle A\first{v_1} =σ1u1,Av2\displaystyle = \first{\sigma_1} \first{u_1}, \qquad A\second{v_2} =σ2u2\displaystyle = \second{\sigma_2} \second{u_2}
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}
AA
σ1u1\htmlClass{lc-green}{\sigma_1 u_1}
σ2u2\htmlClass{lc-pink}{\sigma_2 u_2}

Almost every perpendicular pair on the unit circle lands at a non-right angle on the ellipse. The outputs are perpendicular only when the inputs line up with v1\first{v_1} and v2\second{v_2}, landing on the ellipse's longest and shortest semi-axes σ1u1\first{\sigma_1} \first{u_1} and σ2u2\second{\sigma_2} \second{u_2}.

Two perpendicular arrows on the input circle stay perpendicular on the output ellipse only when they land on its longest and shortest axes.
3

Rotate, Stretch, Rotate: A=UΣVTA = U\Sigma V^T

The Singular Value Decomposition (SVD) breaks the action of AA on any input vector x\vecx{x} into three steps: rotate the input axes to the standard axes, stretch along the standard axes, and rotate to the output axes.

When we put the orthonormal vectors v1,v2\first{v_1}, \second{v_2} as the columns of an orthogonal matrix VV, its inverse is its transpose (V−1=VTV^{-1} = V^T). Multiplying VTxV^T \vecx{x} takes the dot product of each row viTv_i^T with x\vecx{x} to find how much of x\vecx{x} goes along v1\first{v_1} and v2\second{v_2}. Strictly, VTV^T is a reflection rather than a rotation when v2\second{v_2} is clockwise from v1\first{v_1}, and the same goes for UU with u1\first{u_1} and u2\second{u_2}; a reflection still keeps every length and right angle.

Rotate, Stretch, Rotate: A = UΣVᵀ
Av1=σ1u1Av2=σ2u2Ax=  ?\begin{aligned} A\htmlClass{lc-green}{v_1} &= \htmlClass{lc-green}{\sigma_1 u_1} \\ A\htmlClass{lc-pink}{v_2} &= \htmlClass{lc-pink}{\sigma_2 u_2} \\ A\htmlClass{lc-blue}{x} &= \;? \end{aligned}
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}
x\htmlClass{lc-blue}{x}

Putting Av1=σ1u1A\first{v_1} = \first{\sigma_1} \first{u_1} and Av2=σ2u2A\second{v_2} = \second{\sigma_2} \second{u_2} together as the columns of AV=UΣAV = U\Sigma and multiplying on the right by VTV^T gives the Singular Value Decomposition:

A=U⏟rotate tou1,u2Σ⏟stretch bysingular valuesσ1≥σ2≥0VT⏟rotate fromv1,v2=[∣∣u1u2∣∣][σ100σ2][—v1T——v2T—]\displaystyle \begin{aligned} A &= \underbrace{U}_{\substack{\text{rotate to} \\ \first{u_1}, \second{u_2}}} \underbrace{\Sigma}_{\substack{\text{stretch by} \\ \text{singular values} \\ \first{\sigma_1} \ge \second{\sigma_2} \ge 0}} \underbrace{V^T}_{\substack{\text{rotate from} \\ \first{v_1}, \second{v_2}}} \\ &= \begin{bmatrix} \first{|} & \second{|} \\ \first{u_1} & \second{u_2} \\ \first{|} & \second{|} \end{bmatrix} \begin{bmatrix} \first{\sigma_1} & 0 \\ 0 & \second{\sigma_2} \end{bmatrix} \begin{bmatrix} \first{\text{---}} & \first{v_1^T} & \first{\text{---}} \\ \second{\text{---}} & \second{v_2^T} & \second{\text{---}} \end{bmatrix} \end{aligned}

Here Σ\Sigma (capital Greek sigma) is the diagonal matrix of stretches, not a summation sign, and in UΣVTxU\Sigma V^T \vecx{x} the matrices act on x\vecx{x} from right to left:

x→  VT  [v1⋅xv2⋅x]→  Σ  [σ1(v1⋅x)σ2(v2⋅x)]→  U  σ1(v1⋅x)u1+σ2(v2⋅x)u2=Ax\displaystyle \begin{aligned} \vecx{x} &\xrightarrow{\;V^T\;} \begin{bmatrix} \first{v_1} \cdot \vecx{x} \\ \second{v_2} \cdot \vecx{x} \end{bmatrix} \xrightarrow{\;\Sigma\;} \begin{bmatrix} \first{\sigma_1}(\first{v_1} \cdot \vecx{x}) \\ \second{\sigma_2}(\second{v_2} \cdot \vecx{x}) \end{bmatrix} \\ &\xrightarrow{\;U\;} \first{\sigma_1}(\first{v_1} \cdot \vecx{x})\first{u_1} + \second{\sigma_2}(\second{v_2} \cdot \vecx{x})\second{u_2} = A\vecx{x} \end{aligned}

Even when AA tilts and shears the plane, AxA\vecx{x} is just a simple stretch by σ1\first{\sigma_1} and σ2\second{\sigma_2} along the standard axes once VTV^T rotates v1\first{v_1} and v2\second{v_2} onto the standard axes and UU rotates them onto u1\first{u_1} and u2\second{u_2}.

Every matrix transformation is just a rotation of the input projections onto the standard axes, a stretch along those axes, and a final rotation onto the output axes.
4

Why v1v_1 and v2v_2 Are Perpendicular: ATAA^T A

To find the perpendicular input directions viv_i and the singular values σi\sigma_i of a matrix AA, we need a symmetric matrix that keeps each viv_i on its own line without rotating it to uiu_i.

Why v₁ and v₂ Are Perpendicular: AᵀA
(ATA) v1=  ?(A^{T}A)\,\htmlClass{lc-green}{v_1} = \;?
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}
AA
ATA^{T}
σ1u1\htmlClass{lc-green}{\sigma_1 u_1}
σ2u2\htmlClass{lc-pink}{\sigma_2 u_2}

Applying AA and then ATA^T cancels the output rotation because UTU=IU^T U = I:

ATA\displaystyle A^T A =(VΣTUT)(UΣVT)\displaystyle = (V\Sigma^T U^T)(U\Sigma V^T) =VΣ2VT  ⟹  (ATA)vi\displaystyle = V\Sigma^2 V^T \implies (A^T A)v_i =σi2vi\displaystyle = \sigma_i^2 v_i

Because ATAA^T A is always symmetric, it always has perpendicular unit eigenvectors v1,v2\first{v_1}, \second{v_2}, and AA sends them to perpendicular outputs: Av1⋅Av2=v1T(ATA)v2=σ22 (v1⋅v2)=0A\first{v_1} \cdot A\second{v_2} = \first{v_1^T} (A^T A) \second{v_2} = \second{\sigma_2^2}\,(\first{v_1} \cdot \second{v_2}) = 0. Notice that the eigenvalues of ATAA^T A are σi2\sigma_i^2, so we take their square roots to get the singular values σi\sigma_i.

Text solution
Find σi\sigma_i and viv_i from the symmetric matrix ATAA^T AFor A=[2−122]A = \begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix}, multiplying ATAA^T A gives eigenvalues σ12=9\first{\sigma_1^2} = \first{9} and σ22=4\second{\sigma_2^2} = \second{4} with orthonormal eigenvectors v1\first{v_1} and v2\second{v_2}.
ATA=[22−12][2−122]=[8225]  ⟹  {σ1=9=3,v1=15[21]σ2=4=2,v2=15[−12]\displaystyle \begin{aligned} A^T A &= \begin{bmatrix} 2 & 2 \\ -1 & 2 \end{bmatrix}\begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix} \\ &= \begin{bmatrix} 8 & 2 \\ 2 & 5 \end{bmatrix} \\ &\implies \begin{cases} \first{\sigma_1} = \sqrt{9} = \first{3}, \\ \first{v_1} = \first{\dfrac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}} \\[8pt] \second{\sigma_2} = \sqrt{4} = \second{2}, \\ \second{v_2} = \second{\dfrac{1}{\sqrt{5}}\begin{bmatrix} -1 \\ 2 \end{bmatrix}} \end{cases} \end{aligned}
Compute the output directions uiu_i from Avi=σiuiAv_i = \sigma_i u_iApplying AA to each input vector viv_i and dividing by σi\sigma_i gives the matching unit semi-axis directions u1\first{u_1} and u2\second{u_2}.
u1=1σ1Av1=13⋅15[36]=15[12],u2=1σ2Av2=12⋅15[−42]=15[−21]\displaystyle \begin{aligned} \first{u_1} &= \frac{1}{\first{\sigma_1}}A\first{v_1} = \frac{1}{\first{3}}\cdot\frac{1}{\sqrt{5}}\begin{bmatrix} 3 \\ 6 \end{bmatrix} \\ &= \first{\frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ 2 \end{bmatrix}}, \\ \second{u_2} &= \frac{1}{\second{\sigma_2}}A\second{v_2} = \frac{1}{\second{2}}\cdot\frac{1}{\sqrt{5}}\begin{bmatrix} -4 \\ 2 \end{bmatrix} \\ &= \second{\frac{1}{\sqrt{5}}\begin{bmatrix} -2 \\ 1 \end{bmatrix}} \end{aligned}
Put together A=UΣVTA = U\Sigma V^TPutting u1,u2\first{u_1}, \second{u_2} as the columns of UU, σ1,σ2\first{\sigma_1}, \second{\sigma_2} on the diagonal of Σ\Sigma, and v1T,v2T\first{v_1^T}, \second{v_2^T} as the rows of VTV^T finishes the decomposition.
A=UΣVT=[15−252515][3002][2515−1525]\displaystyle \begin{aligned} A &= U\Sigma V^T \\ &= \begin{bmatrix} \first{\tfrac{1}{\sqrt{5}}} & \second{-\tfrac{2}{\sqrt{5}}} \\ \first{\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix} \begin{bmatrix} \first{3} & 0 \\ 0 & \second{2} \end{bmatrix} \\ &\qquad \begin{bmatrix} \first{\tfrac{2}{\sqrt{5}}} & \first{\tfrac{1}{\sqrt{5}}} \\ \second{-\tfrac{1}{\sqrt{5}}} & \second{\tfrac{2}{\sqrt{5}}} \end{bmatrix} \end{aligned}
5

Singular Values vs. Eigenvalues

Eigenvalues λi\lambda_i measure stretch only along directions that stay on their own line, while singular values σi\sigma_i measure the true longest and shortest stretches of the unit circle even when those directions rotate.

Singular Values vs. Eigenvalues
σλ[2−122]??\begin{array}{c|cc} & \sigma & \lambda \\ \hline \begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix} & ? & ? \end{array}
σ1\htmlClass{lc-green}{\sigma_1}
σ2\htmlClass{lc-pink}{\sigma_2}

For a shear, eigenvalues miss the stretch because the directions of greatest and least stretch rotate off their starting lines. Only the horizontal eigenvector line stays put with λ=1\lambda = 1, while the singular values σ1\first{\sigma_1} and σ2\second{\sigma_2} give the true longest and shortest semi-axes of the sheared ellipse.

As a shear stretches the unit circle into a longer ellipse, its only eigenvalue stays at 1 on the horizontal axis while the singular values track the longest and shortest stretches.

When AA is symmetric (A=QΛQTA = Q\Lambda Q^T), its eigenvectors qiq_i are already perpendicular. If every eigenvalue is positive and they're listed from largest to smallest, A=QΛQTA = Q\Lambda Q^T is already the SVD with U=V=QU = V = Q and Σ=Λ\Sigma = \Lambda. If an eigenvalue is negative, we can turn QΛQTQ\Lambda Q^T into an SVD without computing ATAA^T A:

Text solution
Take absolute values for the singular valuesFor A=Q[400−3]QTA = Q\begin{bmatrix} \first{4} & 0 \\ 0 & \flag{-3} \end{bmatrix}Q^T, the semi-axis lengths are the absolute values σi=∣λi∣\sigma_i = |\lambda_i|.
σ1=4,σ2=∣−3∣=3  ⟹  Σ=[4003]\displaystyle \first{\sigma_1} = \first{4}, \qquad \second{\sigma_2} = |\flag{-3}| = \second{3} \implies \Sigma = \begin{bmatrix} \first{4} & 0 \\ 0 & \second{3} \end{bmatrix}
Flip the output vector sign when λi<0\lambda_i < 0Stretching q2\second{q_2} by −3\flag{-3} is the same as stretching by +3\second{+3} in the opposite direction −q2\second{-q_2}, so v2=q2\second{v_2} = \second{q_2} and u2=−q2\second{u_2} = \second{-q_2}.
Aq2=−3q2=3(−q2)  ⟹  V=[q1q2],U=[q1−q2]\displaystyle A\second{q_2} = \flag{-3}\second{q_2} = \second{3}(\second{-q_2}) \implies V = \begin{bmatrix} \first{q_1} & \second{q_2} \end{bmatrix}, \quad U = \begin{bmatrix} \first{q_1} & \second{-q_2} \end{bmatrix}
6

Area and the Determinant from the SVD

For any square matrix AA, the absolute value of the determinant ∣det⁡(A)∣\area{|\det(A)|} tells us how much AA scales area in 2D or volume in nnD.

Area and the Determinant from the SVD
A=UΣVT∣det⁡(A)∣=  ?\begin{aligned} A &= U\Sigma V^{T} \\ \htmlClass{lc-blue}{|\det(A)|} &= \;? \end{aligned}
1\htmlClass{lc-blue}{1}
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}
1\htmlClass{lc-blue}{1}
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}
VTV^T
1\htmlClass{lc-blue}{1}
Σ\Sigma
σ1σ2\htmlClass{lc-green}{\sigma_1}\htmlClass{lc-pink}{\sigma_2}
σ1\htmlClass{lc-green}{\sigma_1}
σ2\htmlClass{lc-pink}{\sigma_2}
UU
σ1σ2\htmlClass{lc-green}{\sigma_1}\htmlClass{lc-pink}{\sigma_2}
σ1u1\htmlClass{lc-green}{\sigma_1 u_1}
σ2u2\htmlClass{lc-pink}{\sigma_2 u_2}

Because the rotations VTV^T and UU keep area the same (∣det⁡(U)∣=∣det⁡(VT)∣=1|\det(U)| = |\det(V^T)| = 1), all the area scaling comes from the diagonal stretch Σ\Sigma:

∣det⁡(A)∣\displaystyle \area{|\det(A)|} =∣det⁡(U)∣⋅det⁡(Σ)⋅∣det⁡(VT)∣\displaystyle = |\det(U)| \cdot \det(\Sigma) \cdot |\det(V^T)| =σ1σ2⋯σn\displaystyle = \first{\sigma_1} \second{\sigma_2} \cdots \sigma_n =∣λ1λ2⋯λn∣\displaystyle = |\lambda_1 \lambda_2 \cdots \lambda_n|

For A=[2−122]A = \begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix}, the unit square with sides v1\first{v_1} and v2\second{v_2} becomes a 3×2\first{3} \times \second{2} rectangle of area σ1σ2=6\first{\sigma_1} \second{\sigma_2} = 6, matching ∣det⁡(A)∣=∣(2)(2)−(−1)(2)∣=6\area{|\det(A)|} = |(2)(2) - (-1)(2)| = 6.

7

The SVD for Rectangular Matrices

When AA is an m×nm \times n matrix mapping Rn\mathbb{R}^n to Rm\mathbb{R}^m, the factorization A=UΣVTA = U\Sigma V^T still works: UU is m×mm \times m in the output space, VTV^T is n×nn \times n in the input space, and Σ\Sigma has the same m×nm \times n rectangular shape as AA.

The SVD for Rectangular Matrices
Am×n  \displaystyle \underset{\htmlClass{lc-mute}{m \times n}}{A} \;=  U?×?Σ?×?VT?×?\displaystyle =\; \underset{\htmlClass{lc-mute}{? \times ?}}{U} \quad \underset{\htmlClass{lc-mute}{? \times ?}}{\Sigma} \quad \underset{\htmlClass{lc-mute}{? \times ?}}{V^{T}}
[∙∙∙∙∙∙]2×3[∙∙∙∙∙∙]3×2\displaystyle \underset{\htmlClass{lc-mute}{2 \times 3}}{\htmlClass{lc-mute}{\begin{bmatrix} \bullet & \bullet & \bullet \\ \bullet & \bullet & \bullet \end{bmatrix}}} \qquad \underset{\htmlClass{lc-mute}{3 \times 2}}{\htmlClass{lc-mute}{\begin{bmatrix} \bullet & \bullet \\ \bullet & \bullet \\ \bullet & \bullet \end{bmatrix}}}
Am×n=Um×morthonormal u1,…,umin output space RmΣm×nsame shape as AVTn×northonormal v1,…,vnin input space Rn\displaystyle \begin{aligned} \underset{m \times n}{A} &= \underset{\substack{m \times m \\ \text{orthonormal } \first{u_1}, \dots, u_m \\ \text{in output space }\mathbb{R}^m}}{U} \\ &\qquad \underset{\substack{m \times n \\ \text{same shape as }A}}{\Sigma} \\ &\qquad \underset{\substack{n \times n \\ \text{orthonormal } \first{v_1}, \dots, v_n \\ \text{in input space }\mathbb{R}^n}}{V^T} \end{aligned}
A ⁣:2×3,Σ=[σ1000σ20]A\colon 2 \times 3, \quad \Sigma = \begin{bmatrix} \htmlClass{lc-green}{\sigma_1} & 0 & \htmlClass{lc-orange}{0} \\ 0 & \htmlClass{lc-pink}{\sigma_2} & \htmlClass{lc-orange}{0} \end{bmatrix}
AA
σ1u1\htmlClass{lc-green}{\sigma_1 u_1}
σ2u2\htmlClass{lc-pink}{\sigma_2 u_2}
Av3=0A\htmlClass{lc-violet}{v_3} = \htmlClass{lc-orange}{0}
A ⁣:3×2,Σ=[σ100σ200]A\colon 3 \times 2, \quad \Sigma = \begin{bmatrix} \htmlClass{lc-green}{\sigma_1} & 0 \\ 0 & \htmlClass{lc-pink}{\sigma_2} \\ \htmlClass{lc-orange}{0} & \htmlClass{lc-orange}{0} \end{bmatrix}
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}
AA
8

Zero Singular Values, Rank, and the Four Subspaces

Any matrix, square or rectangular, crushes an input direction viv_i to the zero vector whenever its singular value is σi=0\sigma_i = \lost{0}. The number of nonzero singular values σ1≥σ2≥⋯≥σr>0\first{\sigma_1} \ge \second{\sigma_2} \ge \dots \ge \sigma_r > 0 is the rank rr of the matrix.

Zero Singular Values, Rank, and the Four Subspaces
A=U[5000]VTAv2=  ?\begin{aligned} A &= U\begin{bmatrix} \htmlClass{lc-green}{5} & 0 \\ 0 & \htmlClass{lc-orange}{0} \end{bmatrix}V^{T} \\ A\htmlClass{lc-pink}{v_2} &= \;? \end{aligned}
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}

When we replace Σ=[3002]\Sigma = \begin{bmatrix} \first{3} & 0 \\ 0 & \second{2} \end{bmatrix} from our earlier example with Σ=[5000]\Sigma = \begin{bmatrix} \first{5} & 0 \\ 0 & \lost{0} \end{bmatrix}, we get the rank-1 matrix A=[2142]A = \begin{bmatrix} 2 & 1 \\ 4 & 2 \end{bmatrix}, which flattens the unit circle into a 1D line segment along u1\first{u_1}:

Text solution
Follow the direction that survives (i=1i = 1)Because σ1=5>0\first{\sigma_1} = \first{5} > 0, AA stretches v1\first{v_1} to 5u1\first{5u_1}, so v1\first{v_1} is an orthonormal basis for the Row Space C(AT)\kept{C(A^T)} and u1\first{u_1} is an orthonormal basis for the Column Space C(A)\kept{C(A)}.
Av1=5u1  ⟹  Row Space C(AT):{v1=15[21]},Column Space C(A):{u1=15[12]}\displaystyle \begin{aligned} &A\first{v_1} = \first{5u_1} \implies && \\ &\text{Row Space } \kept{C(A^T)}: \\ &\qquad \left\{\first{v_1} = \first{\frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}}\right\}, \\ &\text{Column Space } \kept{C(A)}: \\ &\qquad \left\{\first{u_1} = \first{\frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ 2 \end{bmatrix}}\right\} \end{aligned}
Follow the crushed and unreached directions (i=2i = 2)Because σ2=0\second{\sigma_2} = \lost{0}, v2\second{v_2} gets crushed to zero and spans the Null Space N(A)\lost{N(A)}, while u2\second{u_2} is perpendicular to the column space and spans the Left Null Space N(AT)\lost{N(A^T)}.
Av2=0  ⟹  Null Space N(A):{v2=15[−12]},Left Null Space N(AT):{u2=15[−21]}\displaystyle \begin{aligned} &A\second{v_2} = \lost{0} \implies && \\ &\text{Null Space } \lost{N(A)}: \\ &\qquad \left\{\second{v_2} = \second{\frac{1}{\sqrt{5}}\begin{bmatrix} -1 \\ 2 \end{bmatrix}}\right\}, \\ &\text{Left Null Space } \lost{N(A^T)}: \\ &\qquad \left\{\second{u_2} = \second{\frac{1}{\sqrt{5}}\begin{bmatrix} -2 \\ 1 \end{bmatrix}}\right\} \end{aligned}

For any m×nm \times n matrix of rank rr, splitting the columns of VV and UU right after the rr nonzero singular values gives orthonormal bases for all four fundamental subspaces at once:

V=[v1  …  vr⏟Row Space C(AT)Avi=σiui  ∣  vr+1  …  vn⏟Null Space N(A)Avi=0],U=[u1  …  ur⏟Column Space C(A)reachable outputs  ∣  ur+1  …  um⏟Left Null Space N(AT)unreachable outputs]\displaystyle \begin{aligned} & V = \\ & \Big[\underbrace{\first{v_1} \;\dots\; v_r}_{\substack{\text{Row Space } \kept{C(A^T)} \\ Av_i = \sigma_i u_i}} \;\Big|\; \underbrace{v_{r+1} \;\dots\; v_n}_{\substack{\text{Null Space } \lost{N(A)} \\ Av_i = \lost{0}}}\Big], \\[6pt] & U = \\ & \Big[\underbrace{\first{u_1} \;\dots\; u_r}_{\substack{\text{Column Space } \kept{C(A)} \\ \text{reachable outputs}}} \;\Big|\; \underbrace{u_{r+1} \;\dots\; u_m}_{\substack{\text{Left Null Space } \lost{N(A^T)} \\ \text{unreachable outputs}}}\Big] \end{aligned}

Moving the input vector x\vecx{x} along v2\second{v_2} doesn't change AxA\vecx{x} because the entire Null Space part gets crushed to zero. Only the Row Space part along v1\first{v_1} survives, stretched by σ1\first{\sigma_1} onto the Column Space line along u1\first{u_1}.

Only the Row Space part of the input survives to set the output on the Column Space line, while its Null Space part is crushed to zero.
9

What the SVD Lets Us Do

Because every matrix factors into a rotation, a stretch along the axes, and a second rotation, we can compress matrices by dropping tiny σi\sigma_i, solve or best-approximate non-invertible systems by dividing only by the nonzero σi\sigma_i, and pick out the main directions of variation in data.

What the SVD Unlocks
A=UΣVTrotatestretchrotate\begin{array}{rccc} A = & U & \Sigma & V^{T} \\ & \small\text{rotate} & \small\text{stretch} & \small\text{rotate} \end{array}
v1\htmlClass{lc-green}{v_1}
v2\htmlClass{lc-pink}{v_2}

Key concepts

Every m×nm \times n matrix AA factors as A=UΣVTA = U\Sigma V^T, turning perpendicular unit inputs viv_i in Rn\mathbb{R}^n into perpendicular unit outputs uiu_i in Rm\mathbb{R}^m with Avi=σiuiAv_i = \sigma_i u_i and singular values σ1≥σ2≥⋯≥σr>0\first{\sigma_1} \ge \second{\sigma_2} \ge \dots \ge \sigma_r > 0. In UΣVTxU\Sigma V^T \vecx{x}, first VTV^T takes dot products vi⋅xv_i \cdot \vecx{x} to rotate onto the standard axes, then Σ\Sigma stretches the first rr coordinates by σi\sigma_i and sets the rest to zero, and finally UU rotates the result onto uiu_i.

How you find the SVD depends on what you already have:

When AA is symmetric and you already have A=QΛQTA = Q\Lambda Q^T, you don't need ATAA^T A

Take the singular values directly as σi=∣λi∣\sigma_i = |\lambda_i| sorted from largest to smallest, set vi=qiv_i = q_i, and flip the output sign ui=−qiu_i = -q_i whenever λi<0\lambda_i < 0 (for example, Aq=−7q=7(−q)Aq = \flag{-7}q = 7(-q) gives σ=7\sigma = 7, v=qv = q, and u=−qu = -q).

When AA is non-symmetric or rectangular, use ATA=VΣTΣVTA^T A = V\Sigma^T\Sigma V^T

Find the eigenvalues σi2\sigma_i^2 and orthonormal eigenvectors viv_i of ATAA^T A, take square roots σi=σi2\sigma_i = \sqrt{\sigma_i^2} in decreasing order, and compute each output vector from ui=1σiAviu_i = \frac{1}{\sigma_i}Av_i (for A=[2−122]A = \begin{bmatrix} 2 & -1 \\ 2 & 2 \end{bmatrix}, ATA=[8225]A^T A = \begin{bmatrix} 8 & 2 \\ 2 & 5 \end{bmatrix} has eigenvalues 9\first{9} and 4\second{4}, giving σ1=3\first{\sigma_1} = \first{3} and σ2=2\second{\sigma_2} = \second{2}).

When AA is 2×22 \times 2 and one singular value is already known, use ∣det⁡(A)∣=σ1σ2\area{|\det(A)|} = \first{\sigma_1} \second{\sigma_2}

Because rotations keep area the same, the product of the singular values equals ∣det⁡(A)∣\area{|\det(A)|} (for A=[2302]A = \begin{bmatrix} 2 & 3 \\ 0 & 2 \end{bmatrix} with σ1=4\first{\sigma_1} = \first{4}, ∣det⁡(A)∣=4\area{|\det(A)|} = 4 gives σ2=1\second{\sigma_2} = \second{1} without computing ATAA^T A).

The shape and rank of Σ\Sigma show what the matrix does geometrically: UU is m×mm \times m, Σ\Sigma is m×nm \times n, and VTV^T is n×nn \times n. The number of nonzero singular values is rank(A)=r\text{rank}(A) = r, which splits the columns of VV into orthonormal bases for the Row Space {v1,…,vr}\{\first{v_1}, \dots, v_r\} and Null Space {vr+1,…,vn}\{v_{r+1}, \dots, v_n\}, and splits the columns of UU into orthonormal bases for the Column Space {u1,…,ur}\{\first{u_1}, \dots, u_r\} and Left Null Space {ur+1,…,um}\{u_{r+1}, \dots, u_m\}.

10

Practice

1

A 2×22 \times 2 matrix AA has the SVD A=UΣVTA = U\Sigma V^T with u1=12[11]\first{u_1} = \first{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}, u2=12[−11]\second{u_2} = \second{\frac{1}{\sqrt{2}}\begin{bmatrix} -1 \\ 1 \end{bmatrix}}, Σ=[6002]\Sigma = \begin{bmatrix} \first{6} & 0 \\ 0 & \second{2} \end{bmatrix}, v1=15[34]\first{v_1} = \first{\frac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix}}, and v2=15[−43]\second{v_2} = \second{\frac{1}{5}\begin{bmatrix} -4 \\ 3 \end{bmatrix}}. (a) When AA maps the unit circle to an ellipse, what are the ends of the ellipse's major and minor semi-axes, and which unit input vectors land on them? (b) Follow the input vector x=[211]\vecx{x} = \vecx{\begin{bmatrix} 2 \\ 11 \end{bmatrix}} step by step through VTxV^T \vecx{x}, ΣVTx\Sigma V^T \vecx{x}, and UΣVTxU\Sigma V^T \vecx{x} to find AxA\vecx{x}.

(a) Which vectors point along the ellipse in the output space and get stretched by σ1\first{\sigma_1} and σ2\second{\sigma_2}? (b) The rows of VTV^T are v1T\first{v_1^T} and v2T\second{v_2^T}, so take their dot products with x\vecx{x} first.

(a) From Avi=σiuiAv_i = \sigma_i u_i, the semi-axes point along the output vectors u1\first{u_1} and u2\second{u_2} (be careful not to stretch the input vectors v1\first{v_1} and v2\second{v_2} into σ1v1=15[1824]\first{\sigma_1} \first{v_1} = \frac{1}{5}\begin{bmatrix} 18 \\ 24 \end{bmatrix} and σ2v2=15[−86]\second{\sigma_2} \second{v_2} = \frac{1}{5}\begin{bmatrix} -8 \\ 6 \end{bmatrix}). The end of the major semi-axis is
σ1u1=6(12[11])=[3232],which comes from v1=15[34]\displaystyle \first{\sigma_1} \first{u_1} = \first{6}\left(\first{\frac{1}{\sqrt{2}}\begin{bmatrix} 1 \\ 1 \end{bmatrix}}\right) = \first{\begin{bmatrix} 3\sqrt{2} \\ 3\sqrt{2} \end{bmatrix}}, \quad \text{which comes from } \first{v_1} = \first{\frac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix}}
and the end of the minor semi-axis is
σ2u2=2(12[−11])=[−22],which comes from v2=15[−43]\displaystyle \second{\sigma_2} \second{u_2} = \second{2}\left(\second{\frac{1}{\sqrt{2}}\begin{bmatrix} -1 \\ 1 \end{bmatrix}}\right) = \second{\begin{bmatrix} -\sqrt{2} \\ \sqrt{2} \end{bmatrix}}, \quad \text{which comes from } \second{v_2} = \second{\frac{1}{5}\begin{bmatrix} -4 \\ 3 \end{bmatrix}}
(b) First, put v1T\first{v_1^T} and v2T\second{v_2^T} as the rows of VTV^T (forgetting to transpose VV and multiplying VxV\vecx{x} instead would give 15[−3841]\frac{1}{5}\begin{bmatrix} -38 \\ 41 \end{bmatrix} and lead to [−312−7352]\begin{bmatrix} -31\sqrt{2} \\ -\frac{73}{5}\sqrt{2} \end{bmatrix}):
VTx=[v1⋅xv2⋅x]=[15(6+44)15(−8+33)]=[105]\displaystyle V^T \vecx{x} = \begin{bmatrix} \first{v_1} \cdot \vecx{x} \\ \second{v_2} \cdot \vecx{x} \end{bmatrix} = \begin{bmatrix} \frac{1}{5}(6 + 44) \\ \frac{1}{5}(-8 + 33) \end{bmatrix} = \begin{bmatrix} 10 \\ 5 \end{bmatrix}
Second, stretch by Σ\Sigma:
ΣVTx=[6002][105]=[6010]\displaystyle \Sigma V^T \vecx{x} = \begin{bmatrix} \first{6} & 0 \\ 0 & \second{2} \end{bmatrix}\begin{bmatrix} 10 \\ 5 \end{bmatrix} = \begin{bmatrix} 60 \\ 10 \end{bmatrix}
Third, rotate by UU:
Ax=UΣVTx=60u1+10u2=602[11]+102[−11]=[252352]\displaystyle \begin{aligned} A\vecx{x} &= U\Sigma V^T \vecx{x} = 60\first{u_1} + 10\second{u_2} \\ &= \frac{60}{\sqrt{2}}\first{\begin{bmatrix} 1 \\ 1 \end{bmatrix}} + \frac{10}{\sqrt{2}}\second{\begin{bmatrix} -1 \\ 1 \end{bmatrix}} = \begin{bmatrix} 25\sqrt{2} \\ 35\sqrt{2} \end{bmatrix} \end{aligned}
2

For A=[122−2]A = \begin{bmatrix} 1 & 2 \\ 2 & -2 \end{bmatrix}, the symmetric matrix ATA=[5−2−28]A^T A = \begin{bmatrix} 5 & -2 \\ -2 & 8 \end{bmatrix} has eigenvalues 99 and 44 with unit eigenvectors 15[1−2]\frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ -2 \end{bmatrix} and 15[21]\frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}. Find the singular values σ1\first{\sigma_1} and σ2\second{\sigma_2}, the orthonormal output vectors u1\first{u_1} and u2\second{u_2}, and write the full SVD A=UΣVTA = U\Sigma V^T.

The eigenvalues of ATAA^T A are σi2\sigma_i^2, not σi\sigma_i. Once you have σi\sigma_i and viv_i, use Avi=σiuiAv_i = \sigma_i u_i to solve for uiu_i.

From (ATA)vi=σi2vi(A^T A)v_i = \sigma_i^2 v_i with σ1≥σ2\first{\sigma_1} \ge \second{\sigma_2}, take the square roots of the eigenvalues (don't leave σ1=9\first{\sigma_1} = 9 and σ2=4\second{\sigma_2} = 4 without taking the square root):
σ1=9=3withv1=15[1−2],σ2=4=2withv2=15[21]\displaystyle \begin{aligned} \first{\sigma_1} &= \sqrt{9} = \first{3} \quad \text{with} \quad \first{v_1} = \first{\frac{1}{\sqrt{5}}\begin{bmatrix} 1 \\ -2 \end{bmatrix}}, \\ \second{\sigma_2} &= \sqrt{4} = \second{2} \quad \text{with} \quad \second{v_2} = \second{\frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}} \end{aligned}
Using Avi=σiuiAv_i = \sigma_i u_i gives:
Av1=15[−36]=3u1  ⟹  u1=15[−12]\displaystyle A\first{v_1} = \frac{1}{\sqrt{5}}\begin{bmatrix} -3 \\ 6 \end{bmatrix} = \first{3u_1} \implies \first{u_1} = \first{\frac{1}{\sqrt{5}}\begin{bmatrix} -1 \\ 2 \end{bmatrix}}
Av2=15[42]=2u2  ⟹  u2=15[21]\displaystyle A\second{v_2} = \frac{1}{\sqrt{5}}\begin{bmatrix} 4 \\ 2 \end{bmatrix} = \second{2u_2} \implies \second{u_2} = \second{\frac{1}{\sqrt{5}}\begin{bmatrix} 2 \\ 1 \end{bmatrix}}
Putting u1,u2\first{u_1}, \second{u_2} as the columns of UU and v1T,v2T\first{v_1^T}, \second{v_2^T} as the rows of VTV^T (be careful not to write v1,v2\first{v_1}, \second{v_2} as the columns of VTV^T):
A=UΣVT=[−15252515][3002][15−252515]\displaystyle \begin{aligned} A &= U\Sigma V^T \\ &= \begin{bmatrix} \first{-\tfrac{1}{\sqrt{5}}} & \second{\tfrac{2}{\sqrt{5}}} \\ \first{\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix}\begin{bmatrix} \first{3} & 0 \\ 0 & \second{2} \end{bmatrix} \\ &\qquad \begin{bmatrix} \first{\tfrac{1}{\sqrt{5}}} & \first{-\tfrac{2}{\sqrt{5}}} \\ \second{\tfrac{2}{\sqrt{5}}} & \second{\tfrac{1}{\sqrt{5}}} \end{bmatrix} \end{aligned}
3

A symmetric matrix AA has the orthogonal diagonalization A=QΛQTA = Q\Lambda Q^T with Q=[q1q2]=15[3−443]Q = \begin{bmatrix} q_1 & q_2 \end{bmatrix} = \frac{1}{5}\begin{bmatrix} 3 & -4 \\ 4 & 3 \end{bmatrix} and Λ=[200−7]\Lambda = \begin{bmatrix} 2 & 0 \\ 0 & -7 \end{bmatrix}. Why is QΛQTQ\Lambda Q^T not an SVD of AA, and what are UU, Σ\Sigma, and VTV^T in a valid SVD A=UΣVTA = U\Sigma V^T?

Check both rules for Σ\Sigma: can a singular value be negative, and in what order must σ1\first{\sigma_1} and σ2\second{\sigma_2} appear? When you swap their order, remember to swap the corresponding vectors too.

QΛQTQ\Lambda Q^T is not an SVD because Λ\Lambda has a negative entry (−7\flag{-7}) and the diagonal entries are not ordered from largest to smallest in absolute value (2<∣−7∣2 < |\flag{-7}|).
Since σi=∣λi∣\sigma_i = |\lambda_i| with σ1≥σ2\first{\sigma_1} \ge \second{\sigma_2}, we have to both take the absolute value and reorder the diagonal to σ1=7\first{\sigma_1} = \first{7} and σ2=2\second{\sigma_2} = \second{2} (simply taking the absolute value in place gives Σ=[2007]\Sigma = \begin{bmatrix} 2 & 0 \\ 0 & 7 \end{bmatrix}, which is out of order, and reordering Σ\Sigma without swapping q2\first{q_2} into the first column and flipping the sign of u1\first{u_1} will not equal AA):
Σ=[7002]\displaystyle \Sigma = \begin{bmatrix} \first{7} & 0 \\ 0 & \second{2} \end{bmatrix}
For σ1=∣−7∣=7\first{\sigma_1} = |\flag{-7}| = \first{7}, the equation Aq2=−7q2=7(−q2)A\first{q_2} = \flag{-7}\first{q_2} = \first{7}(\first{-q_2}) gives v1=q2=15[−43]\first{v_1} = \first{q_2} = \first{\frac{1}{5}\begin{bmatrix} -4 \\ 3 \end{bmatrix}} and flips the sign of the output vector to u1=−q2=15[4−3]\first{u_1} = \first{-q_2} = \first{\frac{1}{5}\begin{bmatrix} 4 \\ -3 \end{bmatrix}}. For σ2=2>0\second{\sigma_2} = \second{2} > 0, we keep v2=u2=q1=15[34]\second{v_2} = \second{u_2} = \second{q_1} = \second{\frac{1}{5}\begin{bmatrix} 3 \\ 4 \end{bmatrix}}. This gives:
U=15[43−34],Σ=[7002],VT=15[−4334]\displaystyle U = \frac{1}{5}\begin{bmatrix} \first{4} & \second{3} \\ \first{-3} & \second{4} \end{bmatrix}, \qquad \Sigma = \begin{bmatrix} \first{7} & 0 \\ 0 & \second{2} \end{bmatrix}, \qquad V^T = \frac{1}{5}\begin{bmatrix} \first{-4} & \first{3} \\ \second{3} & \second{4} \end{bmatrix}
4

Consider the matrix A=[2302]A = \begin{bmatrix} 2 & 3 \\ 0 & 2 \end{bmatrix}. (a) What are the eigenvalues of AA, and why don't they equal the singular values of AA? (b) When AA acts on the unit circle, the longest semi-axis of the resulting ellipse has length σ1=4\first{\sigma_1} = \first{4}. Without computing ATAA^T A, find the shortest semi-axis length σ2\second{\sigma_2} and the area of the rectangle that AA makes from the unit square with sides v1\first{v_1} and v2\second{v_2}.

(a) How does the 33 in the top-right corner affect the directions of greatest and least stretch? (b) How does ∣det⁡(A)∣\area{|\det(A)|} connect to σ1\first{\sigma_1} and σ2\second{\sigma_2}?

(a) Because AA is upper triangular, its eigenvalues are λ1=2\lambda_1 = 2 and λ2=2\lambda_2 = 2. They do not equal the singular values because AA shears the plane, rotating the directions of greatest and least stretch off their starting lines instead of keeping them along fixed eigenvector lines.
(b) Using ∣det⁡(A)∣=∣(2)(2)−(3)(0)∣=4\area{|\det(A)|} = |(2)(2) - (3)(0)| = 4 and ∣det⁡(A)∣=σ1σ2\area{|\det(A)|} = \first{\sigma_1} \second{\sigma_2} (don't set σ2\second{\sigma_2} equal to the eigenvalue λ2=2\lambda_2 = 2, which would give an area of 88):
4\displaystyle 4 =4σ2  ⟹  σ2\displaystyle = \first{4}\second{\sigma_2} \implies \second{\sigma_2} =1\displaystyle = \second{1}
The unit square with sides v1\first{v_1} and v2\second{v_2} gets rotated and stretched into a 4×1\first{4} \times \second{1} rectangle of area σ1σ2=4\first{\sigma_1} \second{\sigma_2} = 4.
5

Suppose AA is a 2×42 \times 4 matrix with singular values σ1=5\first{\sigma_1} = \first{5} and σ2=3\second{\sigma_2} = \second{3}, and BB is a 4×24 \times 2 matrix with the same singular values σ1=5\first{\sigma_1} = \first{5} and σ2=3\second{\sigma_2} = \second{3}. (a) Give the dimensions of UU, Σ\Sigma, and VTV^T for AA and for BB, and write out the full matrix Σ\Sigma for each. (b) Explain geometrically what the zero columns in AA's Σ\Sigma do to the unit sphere in R4\mathbb{R}^4, and what the zero rows in BB's Σ\Sigma mean for the ellipse that BB makes in R4\mathbb{R}^4.

(a) Match UU to the output dimension, VTV^T to the input dimension, and Σ\Sigma to the exact shape of the matrix. (b) Follow what gets multiplied by 00 in ΣVTx\Sigma V^T \vecx{x} vs. what gets extra zeros added to it before UU acts.

(a) Match UU (m×mm \times m), Σ\Sigma (m×nm \times n), and VTV^T (n×nn \times n) to the dimensions of each matrix (don't write Σ\Sigma as a 2×22 \times 2 matrix [5003]\begin{bmatrix} 5 & 0 \\ 0 & 3 \end{bmatrix} by leaving off the zero columns or rows, and don't swap the sizes of UU and VTV^T):
For AA (2×42 \times 4): UU is 2×22 \times 2, VTV^T is 4×44 \times 4, and Σ\Sigma is 2×42 \times 4:
ΣA=[50000300]\displaystyle \Sigma_A = \begin{bmatrix} \first{5} & 0 & \lost{0} & \lost{0} \\ 0 & \second{3} & \lost{0} & \lost{0} \end{bmatrix}
For BB (4×24 \times 2): UU is 4×44 \times 4, VTV^T is 2×22 \times 2, and Σ\Sigma is 4×24 \times 2:
ΣB=[50030000]\displaystyle \Sigma_B = \begin{bmatrix} \first{5} & 0 \\ 0 & \second{3} \\ \lost{0} & \lost{0} \\ \lost{0} & \lost{0} \end{bmatrix}
(b) In AA, the two zero columns of ΣA\Sigma_A multiply the third and fourth rotated input coordinates by 0\lost{0}, crushing the input directions v3\third{v_3} and v4\third{v_4} in R4\mathbb{R}^4 to zero and flattening the 4D unit sphere onto a filled 2D ellipse in R2\mathbb{R}^2. In BB, the two zero rows at the bottom of ΣB\Sigma_B put zeros in the third and fourth coordinates before UU acts, so BB maps the 2D unit circle to a 2D ellipse in R4\mathbb{R}^4 spanned by u1\first{u_1} and u2\second{u_2}, while u3\third{u_3} and u4\third{u_4} are unit vectors in R4\mathbb{R}^4 perpendicular to the plane of that ellipse.
6

A 3×33 \times 3 matrix AA has the SVD A=UΣVTA = U\Sigma V^T, where U=[u1u2u3]U = \begin{bmatrix} \first{u_1} & \second{u_2} & \third{u_3} \end{bmatrix} and V=[v1v2v3]V = \begin{bmatrix} \first{v_1} & \second{v_2} & \third{v_3} \end{bmatrix} are 3×33 \times 3 orthogonal matrices and Σ=[400000000]\Sigma = \begin{bmatrix} \first{4} & 0 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{bmatrix}. (a) What is the rank of AA, what geometric shape does AA make from the 3D unit sphere, and which columns of UU and VV form orthonormal bases for each of the Four Fundamental Subspaces of AA? (b) For b=12u1+cu2\vecx{b} = 12\first{u_1} + c\second{u_2}, for what value of cc does Ax=bA\vecx{x} = \vecx{b} have a solution, and for that cc, what are all vectors x\vecx{x} (written in terms of v1,v2,v3\first{v_1}, \second{v_2}, \third{v_3}) that satisfy Ax=bA\vecx{x} = \vecx{b}?

(a) Count the nonzero singular values to find the rank rr, then split VV (inputs) and UU (outputs) after the rr-th column. (b) Write x=x1v1+x2v2+x3v3\vecx{x} = x_1 \first{v_1} + x_2 \second{v_2} + x_3 \third{v_3} and apply AA to each term.

(a) AA has 11 nonzero singular value (σ1=4\first{\sigma_1} = \first{4}), so rank(A)=1\text{rank}(A) = 1. Because AA crushes v2\second{v_2} and v3\third{v_3} to 0\lost{0} and stretches v1\first{v_1} to 4u1\first{4u_1}, it flattens the 3D unit sphere into a 1D line segment along u1\first{u_1}. Splitting the input columns of VV and output columns of UU at r=1r = 1 (don't swap viv_i and uiu_i between the input and output subspaces) gives the orthonormal bases:
Row Space C(AT)\kept{C(A^T)}: {v1}\{\first{v_1}\}
Null Space N(A)\lost{N(A)}: {v2,v3}\{\second{v_2}, \third{v_3}\}
Column Space C(A)\kept{C(A)}: {u1}\{\first{u_1}\}
Left Null Space N(AT)\lost{N(A^T)}: {u2,u3}\{\second{u_2}, \third{u_3}\}
(b) Because C(A)\kept{C(A)} is spanned by u1\first{u_1} and u2\second{u_2} lies in the Left Null Space, b=12u1+cu2\vecx{b} = 12\first{u_1} + c\second{u_2} is in the column space if and only if c=0c = 0. Writing x=x1v1+x2v2+x3v3\vecx{x} = x_1 \first{v_1} + x_2 \second{v_2} + x_3 \third{v_3} and applying AA gives:
Ax\displaystyle A\vecx{x} =x1(4u1)+x2(0)+x3(0)\displaystyle = x_1(\first{4u_1}) + x_2(\lost{0}) + x_3(\lost{0}) =4x1u1\displaystyle = \first{4}x_1 \first{u_1} =12u1  ⟹  x1\displaystyle = 12\first{u_1} \implies x_1 =3\displaystyle = 3
Since v2\second{v_2} and v3\third{v_3} get crushed to 0\lost{0}, x2x_2 and x3x_3 can be any real numbers (don't stop at the single vector x=3v1\vecx{x} = 3\first{v_1} and forget the Null Space vectors), giving all solutions:
x\displaystyle \vecx{x} =3v1+x2v2+x3v3for any real numbers x2,x3\displaystyle = 3\first{v_1} + x_2 \second{v_2} + x_3 \third{v_3} \quad \text{for any real numbers } x_2, x_3