Linear AlgebraThe Singular Value Decomposition5 of 6
The Singular Value Decomposition · 5

PCA: Directions of Greatest Variance

When a dataset has many measurements for every point, we want to compress that cloud of data down to just one or two new directions without losing what tells the points apart.

Information Is Spread
square footage\htmlClass{lc-mute}{\text{square footage}}
bedrooms\htmlClass{lc-mute}{\text{bedrooms}}

To replace a 2D cloud of points (x1,x2)\data{(x_1, x_2)} with a single number per point, we project every point perpendicularly onto a line through the middle of the cloud. How much information survives depends on how spread out the projected points are along the line: projecting along the length of the cloud spreads them as far apart as possible, while projecting across the narrow width crowds them together.

The angle of the line decides how much of the cloud's spread stays on the line.
1

Measuring Spread with the Covariance Matrix

To find the direction of greatest variance without guessing angles, we need a formula for the variance of the projections along any unit vector uu, one that separates the fixed data cloud from the rotating direction uu.

The Covariance Matrix
square footage\htmlClass{lc-mute}{\text{square footage}}
bedrooms\htmlClass{lc-mute}{\text{bedrooms}}
uu
Spread(u)=  ?\text{Spread}(u) = \;?

First, shift the cloud so its average sits at (0,0)(0, 0) by subtracting (Avg x1,Avg x2)(\text{Avg } \data{x_1}, \text{Avg } \data{x_2}) from every point, and stack the NN centered points x(1),x(2),…,x(N)\data{x^{(1)}}, \data{x^{(2)}}, \dots, \data{x^{(N)}} as the rows of a centered data matrix X\data{X}.

When uu has length 11, the dot product x⋅u\data{x} \cdot u is the signed distance of a point's projection along the line through uu. Multiplying the matrix X\data{X} by uu takes that dot product for every row at once, giving the column vector Xu\data{X}u of all NN projections. Because the projections are centered at 00, their variance is the average of their squares, which we can write by taking the dot product of Xu\data{X}u with itself and using (Xu)T=uTXT(\data{X}u)^T = u^T \data{X}^T to pull the direction uu to the outside:

Spread along u=1N∑(x⋅u)2Average of squared projections=1N(Xu)T(Xu)Sum of squares as a dot product=uT(1NXTX)⏟Ccovariance matrixuSeparate data X from direction u\displaystyle \begin{aligned} & \text{Spread along } u \\[4pt] &\quad = \frac{1}{N} \sum (\data{x} \cdot u)^2 \\[4pt] &\quad \text{Average of squared projections} \\[4pt] &\quad = \frac{1}{N} (\data{X}u)^T (\data{X}u) \\[4pt] &\quad \text{Sum of squares as a dot product} \\[4pt] &\quad = u^T \underbrace{\left(\frac{1}{N} \data{X}^T \data{X}\right)}_{\substack{C \\ \text{covariance matrix}}} u \\ &\quad \text{Separate data } \data{X} \text{ from direction } u \end{aligned}

The matrix C=1NXTXC = \frac{1}{N} \data{X}^T \data{X} depends only on the centered data points, and for a 2D cloud its four entries measure the horizontal spread, the vertical spread, and how the two coordinates tilt together:

C=1NXTX=[Avg ofx12Avg ofx1x2Avg ofx1x2Avg ofx22]=[horizontalvariancecovariance(tilt)covariance(tilt)verticalvariance]\displaystyle \begin{aligned} & C = \frac{1}{N} \data{X}^T \data{X} \\[4pt] &\quad = \begin{bmatrix} \begin{gathered}\text{Avg of} \\ \data{x_1}^2\end{gathered} & \begin{gathered}\text{Avg of} \\ \data{x_1 x_2}\end{gathered} \\[16pt] \begin{gathered}\text{Avg of} \\ \data{x_1 x_2}\end{gathered} & \begin{gathered}\text{Avg of} \\ \data{x_2}^2\end{gathered} \end{bmatrix} \\[8pt] &\quad = \begin{bmatrix} \begin{gathered}\text{horizontal} \\ \text{variance}\end{gathered} & \begin{gathered}\text{covariance} \\ \text{(tilt)}\end{gathered} \\[16pt] \begin{gathered}\text{covariance} \\ \text{(tilt)}\end{gathered} & \begin{gathered}\text{vertical} \\ \text{variance}\end{gathered} \end{bmatrix} \end{aligned}

Stretching the cloud horizontally or vertically increases the diagonal entries, while tilting it up and right makes the off-diagonal covariance positive, tilting it down and right makes the covariance negative, and leaving it untilted makes the quadrants cancel to zero.

In a covariance matrix, the diagonal tracks horizontal and vertical spread, while the other two entries track how the data cloud tilts across the quadrants.
2

Principal Components from Eigenvectors

Now that the variance along any unit direction uu is written as the quadratic form Spread(u)=uTCu\text{Spread}(u) = u^T C u, we want to find the unit vector uu that makes uTCuu^T C u as large as possible.

Principal Components from Eigenvectors
x1x_1
x2x_2
uu
largestuTCu=  ?\begin{aligned} &\text{largest} \\ &u^T C u = \;? \end{aligned}

Even when a real data cloud is uneven and lopsided, its covariance matrix C=1NXTXC = \frac{1}{N} \data{X}^T \data{X} is always symmetric because the top-right entry (Avg of x1x2\text{Avg of } \data{x_1 x_2}) and bottom-left entry (Avg of x2x1\text{Avg of } \data{x_2 x_1}) are the exact same number. By the Spectral Theorem, any symmetric matrix CC has perpendicular unit eigenvectors v1\first{v_1} and v2\second{v_2} (with eigenvalues λ1≥λ2\first{\lambda_1} \ge \second{\lambda_2}), and pointing uu straight along v1\first{v_1} turns Cv1C \first{v_1} into λ1v1\first{\lambda_1} \first{v_1}:

Spread(v1)=v1TCv1=v1T(λ1v1)Since Cv1=λ1v1=λ1(v1Tv1)⏟1=λ1Since v1 is a unit vector\displaystyle \begin{aligned} & \text{Spread}(\first{v_1}) = \first{v_1^T} C \first{v_1} = \first{v_1^T} (\first{\lambda_1} \first{v_1}) \\[4pt] &\qquad \text{Since } C \first{v_1} = \first{\lambda_1} \first{v_1} \\[4pt] &\quad = \first{\lambda_1} \underbrace{(\first{v_1^T} \first{v_1})}_{1} = \first{\lambda_1} \\ &\qquad \text{Since } \first{v_1} \text{ is a unit vector} \end{aligned}

Pointing uu along the perpendicular unit eigenvector v2\second{v_2} gives Spread(v2)=v2TCv2=λ2\text{Spread}(\second{v_2}) = \second{v_2^T} C \second{v_2} = \second{\lambda_2}, and pointing uu in any direction in between gives a weighted average of λ1\first{\lambda_1} and λ2\second{\lambda_2}:

λ2⏟minimum variancealong v2 (Principal Component 2)\displaystyle \underbrace{\second{\lambda_2}}_{\substack{\text{minimum variance} \\ \text{along } \second{v_2} \text{ (Principal Component 2)}}} ≤uTCu\displaystyle \le \quad u^T C u \quad ≤λ1⏟maximum variancealong v1 (Principal Component 1)\displaystyle \le \underbrace{\first{\lambda_1}}_{\substack{\text{maximum variance} \\ \text{along } \first{v_1} \text{ (Principal Component 1)}}}

Try rotating the unit vector uu around the origin to see the variance uTCuu^T C u peak at λ1\first{\lambda_1} along v1\first{v_1} and drop to its minimum λ2\second{\lambda_2} along v2\second{v_2}, 90∘90^\circ away.

Even on a lopsided data cloud, the directions of widest and narrowest spread meet at a right angle, and every other direction has a spread somewhere in between.
3

Finding Principal Components Step by Step

Let's take N=4N = 4 data points in 2D: (1,2)\data{(1, 2)}, (3,4)\data{(3, 4)}, (5,4)\data{(5, 4)}, and (7,6)\data{(7, 6)}. We want to find the direction of greatest variance v1\first{v_1}, the share of total variance it captures, and the compressed 1D projections of the four points.

Finding Principal Components Step by Step
22
44
66
22
44
66
x1x_1
x2x_2
(1, 2)\htmlClass{lc-blue}{(1,\ 2)}
(3, 4)\htmlClass{lc-blue}{(3,\ 4)}
(5, 4)\htmlClass{lc-blue}{(5,\ 4)}
(7, 6)\htmlClass{lc-blue}{(7,\ 6)}
v1=  ?\htmlClass{lc-green}{v_1} = \;?
11
22
33
44
55
66
77
11
22
33
44
55
66
x1x_1
x2x_2
(4, 4)(4,\ 4)
−2-2
22
−2-2
22
x1x_1
x2x_2
v1≈[0.850.53]\htmlClass{lc-green}{v_1 \approx \begin{bmatrix} 0.85 \\ 0.53 \end{bmatrix}}
−3.61\htmlClass{lc-blue}{-3.61}
−0.85\htmlClass{lc-blue}{-0.85}
+0.85\htmlClass{lc-blue}{+0.85}
+3.61\htmlClass{lc-blue}{+3.61}
Text solution
Subtract the center (4,4)(4, 4) to form X\data{X}Find the average of each coordinate across the N=4N = 4 points and subtract it from every point so the cloud is centered at (0,0)(0, 0):
Avg x1=1+3+5+74=4,Avg x2=2+4+4+64=4X=[1−42−43−44−45−44−47−46−4]=[−3−2−101032]\displaystyle \begin{aligned} \text{Avg } \data{x_1} &= \frac{1 + 3 + 5 + 7}{4} = 4, \\[4pt] \text{Avg } \data{x_2} &= \frac{2 + 4 + 4 + 6}{4} = 4 \\[6pt] \data{X} &= \begin{bmatrix} 1 - 4 & 2 - 4 \\ 3 - 4 & 4 - 4 \\ 5 - 4 & 4 - 4 \\ 7 - 4 & 6 - 4 \end{bmatrix} \\ &= \data{\begin{bmatrix} -3 & -2 \\ -1 & 0 \\ 1 & 0 \\ 3 & 2 \end{bmatrix}} \end{aligned}
Multiply 14XTX\frac{1}{4} \data{X}^T \data{X} to get the covariance matrix CCAverage the squares and products of the centered coordinates across the four rows (you might also see this divided by N−1N - 1 instead of NN, which multiplies every eigenvalue by the same factor without changing the eigenvectors):
C=14XTX=[(−3)2+(−1)2+12+324(−3)(−2)+(−1)(0)+(1)(0)+(3)(2)4(−3)(−2)+(−1)(0)+(1)(0)+(3)(2)4(−2)2+02+02+224]=[5332]\displaystyle \begin{aligned} & C = \frac{1}{4} \data{X}^T \data{X} \\[6pt] &\quad = \begin{bmatrix} \frac{\begin{gathered}(-3)^2 \\ + (-1)^2 \\ + 1^2 \\ + 3^2\end{gathered}}{4} & \frac{\begin{gathered}(-3)(-2) \\ + (-1)(0) \\ + (1)(0) \\ + (3)(2)\end{gathered}}{4} \\[10pt] \frac{\begin{gathered}(-3)(-2) \\ + (-1)(0) \\ + (1)(0) \\ + (3)(2)\end{gathered}}{4} & \frac{\begin{gathered}(-2)^2 \\ + 0^2 \\ + 0^2 \\ + 2^2\end{gathered}}{4} \end{bmatrix} \\[8pt] &\quad = \begin{bmatrix} 5 & 3 \\ 3 & 2 \end{bmatrix} \end{aligned}
Solve for the eigenvalues λ1,λ2\first{\lambda_1}, \second{\lambda_2} and the share of varianceUse trace(C)=5+2=7\text{trace}(C) = 5 + 2 = 7 and det⁡(C)=(5)(2)−(3)(3)=1\det(C) = (5)(2) - (3)(3) = 1 to write the characteristic equation and compare λ1\first{\lambda_1} to the total variance λ1+λ2=trace(C)\first{\lambda_1} + \second{\lambda_2} = \text{trace}(C):
λ2−7λ+1=0λ1≈6.85,λ2≈0.15λ1λ1+λ2=6.857≈98%\displaystyle \begin{aligned} \lambda^2 - 7\lambda + 1 &= 0 \\[4pt] \first{\lambda_1} &\approx \first{6.85}, \qquad \second{\lambda_2} \approx \second{0.15} \\[4pt] \frac{\first{\lambda_1}}{\first{\lambda_1} + \second{\lambda_2}} &= \frac{\first{6.85}}{7} \approx 98\% \end{aligned}
Find the unit eigenvector v1\first{v_1} and project with Xv1\data{X} \first{v_1}Plug λ1≈6.85\first{\lambda_1} \approx \first{6.85} into Cv1=λ1v1C \first{v_1} = \first{\lambda_1} \first{v_1} to find the unit direction v1\first{v_1}, then multiply X\data{X} by v1\first{v_1} to get the four signed projection distances along that line:
v1≈[0.850.53],Xv1=[−3−2−101032][0.850.53]=[−3.61−0.85+0.85+3.61]\displaystyle \begin{aligned} \first{v_1} &\approx \first{\begin{bmatrix} 0.85 \\ 0.53 \end{bmatrix}}, \\[6pt] \data{X} \first{v_1} &= \data{\begin{bmatrix} -3 & -2 \\ -1 & 0 \\ 1 & 0 \\ 3 & 2 \end{bmatrix}} \first{\begin{bmatrix} 0.85 \\ 0.53 \end{bmatrix}} \\ &= \data{\begin{bmatrix} -3.61 \\ -0.85 \\ +0.85 \\ +3.61 \end{bmatrix}} \end{aligned}

So the four 2D points compress down to the single list of numbers −3.61,−0.85,+0.85,+3.61\data{-3.61}, \data{-0.85}, \data{+0.85}, \data{+3.61} along v1\first{v_1}, keeping 98%98\% of the total spread in the cloud.

4

PCA and the SVD

When we want to plot our compressed 1D projections as points back in the original space, or when we already have the Singular Value Decomposition X=UΣVT\data{X} = U \Sigma V^T of the centered data matrix, we can read off the principal components and the compressed data matrix directly from the SVD.

PCA and the SVD
−2-2
22
−2-2
22
x1x_1
x2x_2
v1\htmlClass{lc-green}{v_1}
Xv1\htmlClass{lc-blue}{X}\htmlClass{lc-green}{v_1}
=[−3.61−0.85+0.85+3.61]= \begin{bmatrix} \htmlClass{lc-blue}{-3.61} \\ \htmlClass{lc-blue}{-0.85} \\ \htmlClass{lc-blue}{+0.85} \\ \htmlClass{lc-blue}{+3.61} \end{bmatrix}
Xcompressed\htmlClass{lc-blue}{X_{\text{compressed}}}
=  ?= \;?

Because the right singular vectors v1,v2,…\first{v_1}, \second{v_2}, \dots in X=UΣVT\data{X} = U \Sigma V^T are the unit eigenvectors of XTX\data{X}^T \data{X} with eigenvalues σ12,σ22,…\first{\sigma_1^2}, \second{\sigma_2^2}, \dots, dividing XTX\data{X}^T \data{X} by NN to form C=1NXTXC = \frac{1}{N} \data{X}^T \data{X} leaves the eigenvectors untouched and divides each eigenvalue by NN:

v1,v2,…⏟principal directions(columns of V)λi=σi2N⏟variance along vifrom singular value σi\displaystyle \underbrace{\first{v_1}, \second{v_2}, \dots}_{\substack{\text{principal directions} \\ \text{(columns of } V\text{)}}} \qquad\qquad \underbrace{\lambda_i = \frac{\sigma_i^2}{N}}_{\substack{\text{variance along } v_i \\ \text{from singular value } \sigma_i}}
−2-2
22
−2-2
22
x1x_1
x2x_2
Xv1\htmlClass{lc-blue}{X}\htmlClass{lc-green}{v_1}
−3.61\htmlClass{lc-blue}{-3.61}
−0.85\htmlClass{lc-blue}{-0.85}
+0.85\htmlClass{lc-blue}{+0.85}
+3.61\htmlClass{lc-blue}{+3.61}
00
× v1T\times\, \htmlClass{lc-green}{v_1^T}
line through v1\htmlClass{lc-green}{\text{line through } v_1}
−3.61\htmlClass{lc-blue}{-3.61}
−0.85\htmlClass{lc-blue}{-0.85}
+0.85\htmlClass{lc-blue}{+0.85}
+3.61\htmlClass{lc-blue}{+3.61}
Xcompressed=(Xv1)v1T=σ1u1v1T\htmlClass{lc-blue}{X_{\text{compressed}}} = (\htmlClass{lc-blue}{X}\htmlClass{lc-green}{v_1})\htmlClass{lc-green}{v_1^T} = \htmlClass{lc-green}{\sigma_1 u_1 v_1^T}

Each entry in the column vector Xv1\data{X} \first{v_1} is a single number telling how far along v1\first{v_1} a point projects. Multiplying that column of distances by the row vector v1T\first{v_1^T} turns each projection back into a 2D point sitting on the line through v1\first{v_1}, which is the first term of the SVD:

Xcompressed=(Xv1)v1TPoints on the line through v1=(σ1u1)v1TSince Xv1=σ1u1 from X=UΣVT=σ1u1v1TRank-1 approximation of X\displaystyle \begin{aligned} & \data{X_{\text{compressed}}} = (\data{X} \first{v_1}) \first{v_1^T} \\[4pt] &\qquad \text{Points on the line through } \first{v_1} \\[4pt] &\quad = (\first{\sigma_1 u_1}) \first{v_1^T} \\[4pt] &\qquad \text{Since } \data{X} \first{v_1} = \first{\sigma_1 u_1} \\ &\qquad \text{ from } \data{X} = U \Sigma V^T \\[4pt] &\quad = \first{\sigma_1 u_1 v_1^T} \\ &\qquad \text{Rank-1 approximation of } \data{X} \end{aligned}

Keeping the top kk principal components in higher dimensions simply keeps the first kk rank-1 pieces of the SVD:

Xcompressed\displaystyle \data{X_{\text{compressed}}} =σ1u1v1T+σ2u2v2T+⋯+σkukvkT\displaystyle = \first{\sigma_1 u_1 v_1^T} + \second{\sigma_2 u_2 v_2^T} + \dots + \sigma_k u_k v_k^T

Key concepts

Every PCA problem starts by centering the data at (0,0)(0, 0): subtract (Avg x1,Avg x2)(\text{Avg } \data{x_1}, \text{Avg } \data{x_2}) from each point before stacking the NN points as the rows of X\data{X} (for (1,2),(3,4),(5,4),(7,6)\data{(1, 2)}, \data{(3, 4)}, \data{(5, 4)}, \data{(7, 6)} with average (4,4)(4, 4), the first row of X\data{X} is [−3−2]\data{\begin{bmatrix} -3 & -2 \end{bmatrix}}).

From the centered matrix X\data{X}, the covariance matrix C=1NXTXC = \frac{1}{N} \data{X}^T \data{X} records the horizontal variance Avg of x12\text{Avg of } \data{x_1}^2 and vertical variance Avg of x22\text{Avg of } \data{x_2}^2 on the diagonal, and the tilt Avg of x1x2\text{Avg of } \data{x_1 x_2} off the diagonal (in C=[5332]C = \begin{bmatrix} 5 & 3 \\ 3 & 2 \end{bmatrix}, the positive +3+3 means the cloud tilts up and right; a negative off-diagonal entry means it tilts down and right). For any unit vector uu, the variance along uu is the quadratic form uTCuu^T C u.

The principal components and extreme variances are the perpendicular unit eigenvectors and eigenvalues of the symmetric matrix CC. The largest eigenvalue λ1\first{\lambda_1} is the maximum variance (along Principal Component 1, v1\first{v_1}), and the smallest eigenvalue λ2\second{\lambda_2} is the minimum variance (along v2\second{v_2}). Because the total variance is λ1+λ2=trace(C)\first{\lambda_1} + \second{\lambda_2} = \text{trace}(C), Principal Component 1 captures the fraction λ1trace(C)\frac{\first{\lambda_1}}{\text{trace}(C)} of the total variance (for C=[5332]C = \begin{bmatrix} 5 & 3 \\ 3 & 2 \end{bmatrix}, λ1≈6.85\first{\lambda_1} \approx \first{6.85} out of trace(C)=7\text{trace}(C) = 7, or 98%98\%). To project and reconstruct, Xv1\data{X} \first{v_1} gives the column of NN signed 1D distances along the line, and Xcompressed=(Xv1)v1T\data{X_{\text{compressed}}} = (\data{X} \first{v_1}) \first{v_1^T} puts those points back into the original space along the line through v1\first{v_1}.

When not to form C=1NXTXC = \frac{1}{N} \data{X}^T \data{X}: if the centered matrix X\data{X} is already given in SVD form X=UΣVT\data{X} = U \Sigma V^T, skip multiplying out XTX\data{X}^T \data{X}. Read the principal directions viv_i directly from the columns of VV, compute their variances from λi=σi2N\lambda_i = \frac{\sigma_i^2}{N} (for N=10N = 10 and σ1=15\first{\sigma_1} = \first{15}, λ1=15210=22.5\first{\lambda_1} = \frac{\first{15}^2}{10} = \first{22.5}), and write the rank-kk compressed matrix as σ1u1v1T+⋯+σkukvkT\first{\sigma_1 u_1 v_1^T} + \dots + \sigma_k u_k v_k^T.

5

Practice

1

Given N=4N = 4 data points in 2D: (1,5)\data{(1, 5)}, (2,3)\data{(2, 3)}, (4,3)\data{(4, 3)}, and (5,1)\data{(5, 1)}. Center the data at (0,0)(0, 0) to form the centered data matrix X\data{X}, compute the covariance matrix C=14XTXC = \frac{1}{4} \data{X}^T \data{X}, and explain what the off-diagonal entry tells you about the cloud.

Find the average (Avg x1,Avg x2)(\text{Avg } \data{x_1}, \text{Avg } \data{x_2}) of the four points first. What must you do to each coordinate before stacking the rows into X\data{X}?

Find the center of the four points and subtract it from each coordinate to form X\data{X}:
Avg x1=1+2+4+54=3,Avg x2=5+3+3+14=3X=[1−35−32−33−34−33−35−31−3]=[−22−10102−2]\displaystyle \begin{aligned} \text{Avg } \data{x_1} &= \frac{1 + 2 + 4 + 5}{4} = 3, \\[4pt] \text{Avg } \data{x_2} &= \frac{5 + 3 + 3 + 1}{4} = 3 \\[6pt] \data{X} &= \begin{bmatrix} 1 - 3 & 5 - 3 \\ 2 - 3 & 3 - 3 \\ 4 - 3 & 3 - 3 \\ 5 - 3 & 1 - 3 \end{bmatrix} \\ &= \data{\begin{bmatrix} -2 & 2 \\ -1 & 0 \\ 1 & 0 \\ 2 & -2 \end{bmatrix}} \end{aligned}
(Be careful not to stack the uncentered points directly into X\data{X}. Skipping the centering step gives 14XTX=[11.57711]\frac{1}{4} \data{X}^T \data{X} = \begin{bmatrix} 11.5 & 7 \\ 7 & 11 \end{bmatrix} with a false positive off-diagonal entry +7+7 just because all four points sit in the first quadrant.)
Multiply 14XTX\frac{1}{4} \data{X}^T \data{X} (and remember to divide by N=4N = 4, or you will get [10−8−88]\begin{bmatrix} 10 & -8 \\ -8 & 8 \end{bmatrix}):
C=14XTX=[(−2)2+(−1)2+12+224(−2)(2)+(−1)(0)+(1)(0)+(2)(−2)4(−2)(2)+(−1)(0)+(1)(0)+(2)(−2)422+02+02+(−2)24]=[2.5−2−22]\displaystyle \begin{aligned} & C = \frac{1}{4} \data{X}^T \data{X} \\[6pt] & = \begin{bmatrix} \frac{\begin{gathered}(-2)^2 + (-1)^2 \\ + 1^2 + 2^2\end{gathered}}{4} & \frac{\begin{gathered}(-2)(2) \\ + (-1)(0) \\ + (1)(0) \\ + (2)(-2)\end{gathered}}{4} \\[10pt] \frac{\begin{gathered}(-2)(2) \\ + (-1)(0) \\ + (1)(0) \\ + (2)(-2)\end{gathered}}{4} & \frac{\begin{gathered}2^2 + 0^2 \\ + 0^2 + (-2)^2\end{gathered}}{4} \end{bmatrix} \\[8pt] & = \begin{bmatrix} 2.5 & -2 \\ -2 & 2 \end{bmatrix} \end{aligned}
The negative off-diagonal covariance (−2-2) means x1\data{x_1} and x2\data{x_2} move in opposite directions, so the cloud tilts down and to the right.
2

A centered 2D dataset has covariance matrix C=[5228]C = \begin{bmatrix} 5 & 2 \\ 2 & 8 \end{bmatrix}. Find the variance of the projections along the unit vector u=[0.60.8]u = \begin{bmatrix} 0.6 \\ 0.8 \end{bmatrix}, then find the maximum and minimum possible variances across all unit directions and the percentage of total variance captured by Principal Component 1.

Variance along a unit vector uu is the quadratic form uTCuu^T C u; which properties of CC give the extreme values of uTCuu^T C u and the total variance across all directions?

Evaluate the quadratic form uTCuu^T C u for the variance along uu:
uTCu=[0.60.8][5228][0.60.8]=[0.60.8][4.67.6]=8.84\displaystyle \begin{aligned} u^T C u &= \begin{bmatrix} 0.6 & 0.8 \end{bmatrix} \begin{bmatrix} 5 & 2 \\ 2 & 8 \end{bmatrix} \begin{bmatrix} 0.6 \\ 0.8 \end{bmatrix} \\[6pt] &= \begin{bmatrix} 0.6 & 0.8 \end{bmatrix} \begin{bmatrix} 4.6 \\ 7.6 \end{bmatrix} = 8.84 \end{aligned}
Use trace(C)=5+8=13\text{trace}(C) = 5 + 8 = 13 and det⁡(C)=(5)(8)−(2)(2)=36\det(C) = (5)(8) - (2)(2) = 36 to find the eigenvalues of CC (do not read the diagonal entries 88 and 55 as the maximum and minimum variances; that ignores the off-diagonal tilt 22, and 8.848.84 is already larger than 88):
λ2−13λ+36=(λ−9)(λ−4)=0λ1=9(maximum variance),λ2=4(minimum variance)\displaystyle \begin{aligned} & \lambda^2 - 13\lambda + 36 = (\lambda - \first{9})(\lambda - \second{4}) \\ &\quad = 0 \\[4pt] & \first{\lambda_1} = \first{9} \quad (\text{maximum variance}), \\ & \second{\lambda_2} = \second{4} \quad (\text{minimum variance}) \end{aligned}
Divide the top eigenvalue λ1=9\first{\lambda_1} = \first{9} (not the vertical variance 88, which gives 813≈61.5%\tfrac{8}{13} \approx 61.5\%) by the total variance trace(C)=5+8=13\text{trace}(C) = 5 + 8 = 13:
λ1trace(C)\displaystyle \frac{\first{\lambda_1}}{\text{trace}(C)} =913≈69.2%\displaystyle = \frac{\first{9}}{13} \approx 69.2\%
3

The centered data matrix X=[−11−2−5−10510112]\data{X} = \data{\begin{bmatrix} -11 & -2 \\ -5 & -10 \\ 5 & 10 \\ 11 & 2 \end{bmatrix}} has covariance matrix C=14XTX=[73363652]C = \frac{1}{4} \data{X}^T \data{X} = \begin{bmatrix} 73 & 36 \\ 36 & 52 \end{bmatrix} with eigenvalues λ1=100\first{\lambda_1} = \first{100} and λ2=25\second{\lambda_2} = \second{25}. Find the perpendicular unit eigenvectors v1\first{v_1} and v2\second{v_2}, compute the 1D projections Xv1\data{X} \first{v_1} along Principal Component 1, and reconstruct the compressed 2D points Xcompressed=(Xv1)v1T\data{X_{\text{compressed}}} = (\data{X} \first{v_1}) \first{v_1^T}.

Solve (C−100I)v1=0(C - \first{100}I)\first{v_1} = 0 and remember to scale v1\first{v_1} to length 11 before computing Xv1\data{X} \first{v_1}.

Solve (C−100I)v1=0(C - \first{100}I)\first{v_1} = 0 and (C−25I)v2=0(C - \second{25}I)\second{v_2} = 0, and divide by 42+32=5\sqrt{4^2 + 3^2} = 5 so both eigenvectors have unit length:
(C−100I)v1=[−273636−48]v1=[00]  ⟹  v1=15[43]=[0.80.6](C−25I)v2=[48363627]v2=[00]  ⟹  v2=15[−34]=[−0.60.8]\displaystyle \begin{aligned} & (C - \first{100}I)\first{v_1} = \begin{bmatrix} -27 & 36 \\ 36 & -48 \end{bmatrix} \first{v_1} \\ &\quad = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \\ &\quad \implies \quad \first{v_1} = \frac{1}{5} \first{\begin{bmatrix} 4 \\ 3 \end{bmatrix}} = \first{\begin{bmatrix} 0.8 \\ 0.6 \end{bmatrix}} \\[6pt] & (C - \second{25}I)\second{v_2} = \begin{bmatrix} 48 & 36 \\ 36 & 27 \end{bmatrix} \second{v_2} \\ &\quad = \begin{bmatrix} 0 \\ 0 \end{bmatrix} \\ &\quad \implies \quad \second{v_2} = \frac{1}{5} \second{\begin{bmatrix} -3 \\ 4 \end{bmatrix}} \\ &\quad = \second{\begin{bmatrix} -0.6 \\ 0.8 \end{bmatrix}} \end{aligned}
(Be sure to normalize v1\first{v_1} to length 11 before projecting. Using the unnormalized vector [43]\first{\begin{bmatrix} 4 \\ 3 \end{bmatrix}} scales Xv1\data{X} \first{v_1} by 55 to [−50−505050]\begin{bmatrix} -50 \\ -50 \\ 50 \\ 50 \end{bmatrix} and scales Xcompressed\data{X_{\text{compressed}}} by 2525 to [−200−150−200−150200150200150]\begin{bmatrix} -200 & -150 \\ -200 & -150 \\ 200 & 150 \\ 200 & 150 \end{bmatrix}.)
Multiply X\data{X} by v1\first{v_1} for the 1D projections, then multiply by v1T\first{v_1^T} for the reconstructed 2D points on the line through v1\first{v_1}:
Xv1=[−11−2−5−10510112][0.80.6]=[−10−101010]Xcompressed=(Xv1)v1T=[−10−101010][0.80.6]=[−8−6−8−68686]\displaystyle \begin{aligned} \data{X} \first{v_1} &= \data{\begin{bmatrix} -11 & -2 \\ -5 & -10 \\ 5 & 10 \\ 11 & 2 \end{bmatrix}} \first{\begin{bmatrix} 0.8 \\ 0.6 \end{bmatrix}} \\ &= \data{\begin{bmatrix} -10 \\ -10 \\ 10 \\ 10 \end{bmatrix}} \\[6pt] \data{X_{\text{compressed}}} &= (\data{X} \first{v_1}) \first{v_1^T} \\ &= \data{\begin{bmatrix} -10 \\ -10 \\ 10 \\ 10 \end{bmatrix}} \first{\begin{bmatrix} 0.8 & 0.6 \end{bmatrix}} \\ &= \data{\begin{bmatrix} -8 & -6 \\ -8 & -6 \\ 8 & 6 \\ 8 & 6 \end{bmatrix}} \end{aligned}
4

A centered data matrix X\data{X} for N=10N = 10 points in 3D has the Singular Value Decomposition X=UΣVT\data{X} = U \Sigma V^T with singular values σ1=15\first{\sigma_1} = \first{15}, σ2=4\second{\sigma_2} = \second{4}, σ3=3\third{\sigma_3} = \third{3} and V=[23−23132313−23132323]V = \begin{bmatrix} \tfrac{2}{3} & -\tfrac{2}{3} & \tfrac{1}{3} \\[3pt] \tfrac{2}{3} & \tfrac{1}{3} & -\tfrac{2}{3} \\[3pt] \tfrac{1}{3} & \tfrac{2}{3} & \tfrac{2}{3} \end{bmatrix}. Without computing XTX\data{X}^T \data{X}, find the three principal directions v1,v2,v3\first{v_1}, \second{v_2}, \third{v_3} and their variances λ1,λ2,λ3\first{\lambda_1}, \second{\lambda_2}, \third{\lambda_3}, the percentage of total variance captured by keeping the top two principal components, and the SVD expression for the rank-2 compressed data matrix.

How do the columns of VV and the singular values σi\sigma_i of X\data{X} relate to the eigenvectors and eigenvalues of C=1NXTXC = \frac{1}{N} \data{X}^T \data{X}?

Read the three principal directions directly from the columns of VV:
v1=[232313],v2=[−231323],v3=[13−2323]\displaystyle \first{v_1} = \first{\begin{bmatrix} \tfrac{2}{3} \\ \tfrac{2}{3} \\ \tfrac{1}{3} \end{bmatrix}}, \qquad \second{v_2} = \second{\begin{bmatrix} -\tfrac{2}{3} \\ \tfrac{1}{3} \\ \tfrac{2}{3} \end{bmatrix}}, \qquad \third{v_3} = \third{\begin{bmatrix} \tfrac{1}{3} \\ -\tfrac{2}{3} \\ \tfrac{2}{3} \end{bmatrix}}
Square each singular value and divide by N=10N = 10 to find the variances (do not forget to square σi\sigma_i, which would give 15+415+4+3≈86.4%\tfrac{\first{15}+\second{4}}{\first{15}+\second{4}+\third{3}} \approx 86.4\%, and do not forget to divide by N=10N = 10, which would give 225,16,9\first{225}, \second{16}, \third{9}):
λ1\displaystyle \first{\lambda_1} =15210\displaystyle = \frac{\first{15}^2}{10} =22.5,λ2\displaystyle = \first{22.5}, \qquad \second{\lambda_2} =4210\displaystyle = \frac{\second{4}^2}{10} =1.6,λ3\displaystyle = \second{1.6}, \qquad \third{\lambda_3} =3210\displaystyle = \frac{\third{3}^2}{10} =0.9\displaystyle = \third{0.9}
Sum the three eigenvalues for the total variance and find the share captured by the top two components:
λ1+λ2λ1+λ2+λ3\displaystyle \frac{\first{\lambda_1} + \second{\lambda_2}}{\first{\lambda_1} + \second{\lambda_2} + \third{\lambda_3}} =22.5+1.622.5+1.6+0.9\displaystyle = \frac{\first{22.5} + \second{1.6}}{\first{22.5} + \second{1.6} + \third{0.9}} =24.125\displaystyle = \frac{24.1}{25} =96.4%\displaystyle = 96.4\%
Keep the first two rank-1 terms of the SVD (using the singular values σ1=15\first{\sigma_1} = \first{15} and σ2=4\second{\sigma_2} = \second{4}, not the variances 22.5\first{22.5} and 1.6\second{1.6}):
Xcompressed\displaystyle \data{X_{\text{compressed}}} =σ1u1v1T+σ2u2v2T\displaystyle = \first{\sigma_1 u_1 v_1^T} + \second{\sigma_2 u_2 v_2^T} =15u1v1T+4u2v2T\displaystyle = \first{15 u_1 v_1^T} + \second{4 u_2 v_2^T}