Linear AlgebraOrthogonality and Projections4 of 9
Orthogonality and Projections · 4

Projection onto a Subspace

When a target vector b\target{b} lies outside the subspace spanned by our vectors, the equation Ax=bAx = \target{b} has no solution, so we look for the closest point p\proj{p} on the subspace instead.

1

The closest point on a subspace

Given two vectors a1\firstvec{a_1} and a2\secondvec{a_2}, their combinations form a flat plane: the column space of the matrix A=[a1a2]A = \begin{bmatrix} \firstvec{a_1} & \secondvec{a_2} \end{bmatrix}. When a target vector b\target{b} points off that plane, the projection p\proj{p} of b\target{b} onto the subspace is the point on the plane directly beneath b\target{b}.

The closest point on a subspace

To make the distance from p\proj{p} to b\target{b} as short as possible, the error vector from p\proj{p} to b\target{b} must meet the plane at a 90∘90^\circ angle:

b−p⏟error vector⊥plane\displaystyle \underbrace{\err{b - p}}_{\text{error vector}} \perp \plane{\text{plane}}
2

Setting up the dot-product equations

Because the projection p\proj{p} lies on the plane spanned by a1\firstvec{a_1} and a2\secondvec{a_2}, it is a combination p=x^1a1+x^2a2\proj{p} = \firstvec{\hat{x}_1} \firstvec{a_1} + \secondvec{\hat{x}_2} \secondvec{a_2}, and we want to find the two multipliers x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2}.

Remember, when we project onto a single line along a\firstvec{a}, making the error b−x^a\err{b - \hat{x}a} meet a\firstvec{a} at 90∘90^\circ means setting a⋅(b−x^a)=0\firstvec{a} \cdot (\err{b - \hat{x}a}) = 0, which solves in one step to give x^=a⋅ba⋅a\firstvec{\hat{x}} = \frac{\firstvec{a} \cdot \target{b}}{\firstvec{a} \cdot \firstvec{a}} and p=a⋅ba⋅aa\proj{p} = \frac{\firstvec{a} \cdot \target{b}}{\firstvec{a} \cdot \firstvec{a}}\firstvec{a}. For a plane spanned by two vectors a1\firstvec{a_1} and a2\secondvec{a_2}, we apply that same 90∘90^\circ dot-product rule to both vectors at once.

Setting up the dot-product equations
a\firstvec{a}
b\target{b}
p\proj{p}

The hat on x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2} reminds us that these multipliers give the closest point p=Ax^\proj{p} = A\hat{x} on the plane, not an exact solution to Ax=bAx = \target{b}. For the error vector b−p\err{b - p} to be perpendicular to the plane, its dot product with both a1\firstvec{a_1} and a2\secondvec{a_2} must be zero:

a1⋅(b−p)=0a2⋅(b−p)=0\displaystyle \begin{aligned} \firstvec{a_1} \cdot (\err{b - p}) &= 0 \\ \secondvec{a_2} \cdot (\err{b - p}) &= 0 \end{aligned}

Substituting p=x^1a1+x^2a2\proj{p} = \firstvec{\hat{x}_1} \firstvec{a_1} + \secondvec{\hat{x}_2} \secondvec{a_2} turns these two dot-product equations into a system of two linear equations for x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2}:

(a1⋅a1)x^1+(a1⋅a2)x^2=a1⋅b(a2⋅a1)x^1+(a2⋅a2)x^2=a2⋅b\displaystyle \begin{aligned} (\firstvec{a_1} \cdot \firstvec{a_1})\firstvec{\hat{x}_1} + (\firstvec{a_1} \cdot \secondvec{a_2})\secondvec{\hat{x}_2} &= \firstvec{a_1} \cdot \target{b} \\ (\secondvec{a_2} \cdot \firstvec{a_1})\firstvec{\hat{x}_1} + (\secondvec{a_2} \cdot \secondvec{a_2})\secondvec{\hat{x}_2} &= \secondvec{a_2} \cdot \target{b} \end{aligned}

Making the error vector perpendicular to just one basis vector still lets it tilt along the other; the error vector is shortest and perpendicular to the entire plane only at the single point p\proj{p} where its dot products with both a1\firstvec{a_1} and a2\secondvec{a_2} equal zero.

The error vector stands perpendicular to the whole plane, and reaches its shortest length, only where it meets both basis vectors at a right angle.

Now let's use these two equations to project b=[341]\target{b} = \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}} onto the plane spanned by a1=[101]\firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}} and a2=[011]\secondvec{a_2} = \secondvec{\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}}.

Projecting onto a plane: worked example
a1=[101],a2=[011]\displaystyle \firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}}, \quad \secondvec{a_2} = \secondvec{\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}}
b=[341],p=  ?\displaystyle \target{b} = \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}}, \quad \proj{p} = \;?
Text solution
Compute the five dot productsTake the dot products of a1\firstvec{a_1} and a2\secondvec{a_2} with each other for the left-hand side, and with b\target{b} for the right-hand side.
a1⋅a1=2,a1⋅a2=1,a2⋅a2=2,a1⋅b=4,a2⋅b=5\displaystyle \begin{aligned} \firstvec{a_1} \cdot \firstvec{a_1} &= 2, & \firstvec{a_1} \cdot \secondvec{a_2} &= 1, \\ && \secondvec{a_2} \cdot \secondvec{a_2} &= 2, \\ \firstvec{a_1} \cdot \target{b} &= 4, & \secondvec{a_2} \cdot \target{b} &= 5 \end{aligned}
Solve the linear system for the multipliers x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2}Plug the five dot products into our two equations and solve for x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2}.
2x^1+1x^2=41x^1+2x^2=5  ⟹  x^1=1,x^2=2\displaystyle \begin{aligned} 2\firstvec{\hat{x}_1} + 1\secondvec{\hat{x}_2} &= 4 \\ 1\firstvec{\hat{x}_1} + 2\secondvec{\hat{x}_2} &= 5 \end{aligned} \quad \implies \quad {\firstvec{\hat{x}_1} = 1, \quad \secondvec{\hat{x}_2} = 2}
Combine a1\firstvec{a_1} and a2\secondvec{a_2} using x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2} to find p\proj{p}Don't stop at the multipliers; use them to get the actual point on the plane.
p=x^1a1+x^2a2=1[101]+2[011]=[123]\displaystyle \proj{p} = {\firstvec{\hat{x}_1} \firstvec{a_1} + \secondvec{\hat{x}_2} \secondvec{a_2}} = {1\firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}} + 2\secondvec{\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}}} = \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}}
Subtract p\proj{p} from b\target{b} to check that the error is perpendicularTaking the dot product of the error vector b−p\err{b - p} with a1\firstvec{a_1} and a2\secondvec{a_2} confirms that it meets both vectors at 90∘90^\circ.
b−p=[341]−[123]=[22−2],a1⋅(b−p)=0,a2⋅(b−p)=0\displaystyle \begin{aligned} \err{b - p} &= \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}} - \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}} \\ &= \err{\begin{bmatrix} 2 \\ 2 \\ -2 \end{bmatrix}}, \\ \firstvec{a_1} \cdot (\err{b - p}) &= 0, \quad \secondvec{a_2} \cdot (\err{b - p}) = 0 \end{aligned}

So p=[123]\proj{p} = \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}} is the closest point on the plane to b=[341]\target{b} = \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}}, and the perpendicular error vector between them is b−p=[22−2]\err{b - p} = \err{\begin{bmatrix} 2 \\ 2 \\ -2 \end{bmatrix}}.

3

The normal equations in matrix form

We can write this whole system of dot products as a single matrix equation for the multiplier vector x^=[x^1x^2]\hat{x} = \begin{bmatrix} \firstvec{\hat{x}_1} \\ \secondvec{\hat{x}_2} \end{bmatrix}.

The normal equations in matrix form
(a1⋅a1)x^1+(a1⋅a2)x^2=a1⋅b(a2⋅a1)x^1+(a2⋅a2)x^2=a2⋅b\displaystyle \small\begin{aligned} (\firstvec{a_1} \cdot \firstvec{a_1})\firstvec{\hat{x}_1} + (\firstvec{a_1} \cdot \secondvec{a_2})\secondvec{\hat{x}_2} &= \firstvec{a_1} \cdot \target{b} \\ (\secondvec{a_2} \cdot \firstvec{a_1})\firstvec{\hat{x}_1} + (\secondvec{a_2} \cdot \secondvec{a_2})\secondvec{\hat{x}_2} &= \secondvec{a_2} \cdot \target{b} \end{aligned}
A=[a1a2]\displaystyle A = \begin{bmatrix} \firstvec{a_1} & \secondvec{a_2} \end{bmatrix}
(  ?  ) x^\displaystyle (\;?\;)\,\hat{x} =(  ?  )\displaystyle = (\;?\;)

Because a1\firstvec{a_1} and a2\secondvec{a_2} are the columns of AA and the rows of ATA^T, multiplying ATAA^T A and ATbA^T \target{b} takes every dot product in our system at once:

[a1⋅a1a1⋅a2a2⋅a1a2⋅a2]⏟ATA[x^1x^2]⏟x^=[a1⋅ba2⋅b]⏟ATb\displaystyle \underbrace{\begin{bmatrix} \firstvec{a_1} \cdot \firstvec{a_1} & \firstvec{a_1} \cdot \secondvec{a_2} \\ \secondvec{a_2} \cdot \firstvec{a_1} & \secondvec{a_2} \cdot \secondvec{a_2} \end{bmatrix}}_{A^T A} \underbrace{\begin{bmatrix} \firstvec{\hat{x}_1} \\ \secondvec{\hat{x}_2} \end{bmatrix}}_{\hat{x}} = \underbrace{\begin{bmatrix} \firstvec{a_1} \cdot \target{b} \\ \secondvec{a_2} \cdot \target{b} \end{bmatrix}}_{A^T \target{b}}

Writing this in matrix form gives the normal equations:

ATAx^=ATb⏟Normal Equations\displaystyle \underbrace{A^T A \hat{x} = A^T \target{b}}_{\text{Normal Equations}}
4

The projection matrix

The projection matrix PP is a single matrix that multiplies any target vector b\target{b} to give its projection p=Pb\proj{p} = P\target{b} on the subspace directly, without solving the system of equations again for each new vector.

The projection matrix

When the columns of AA are independent, the square dot-product matrix ATAA^T A is invertible. Solving the normal equations for x^\hat{x} and multiplying by AA to get p=Ax^\proj{p} = A\hat{x} combines both steps into PP:

x^=(ATA)−1ATbSolve normal equations for x^p=Ax^=A(ATA)−1AT⏟P bMultiply by A to get p\displaystyle \begin{aligned} \hat{x} &= (A^T A)^{-1} A^T \target{b} \\ & \text{Solve normal equations for }\hat{x} \\ \proj{p} = A\hat{x} &= \underbrace{A(A^T A)^{-1} A^T}_{P}\, \target{b} \\ & \text{Multiply by }A\text{ to get }\proj{p} \end{aligned}

Pulling out the matrix in front of b\target{b} gives the formula for the projection matrix onto the column space of AA:

P\displaystyle P =A(ATA)−1AT\displaystyle = A(A^T A)^{-1} A^T

Because AA is a tall rectangular matrix and not square, A−1A^{-1} does not exist, so never try to split (ATA)−1(A^T A)^{-1} into A−1(AT)−1A^{-1}(A^T)^{-1} to cancel the AA's. Only the square matrix ATAA^T A has an inverse, and it stays together as a single piece.

Let's find the 3×33 \times 3 projection matrix PP for our plane with A=[100111]A = \begin{bmatrix} \firstvec{1} & \secondvec{0} \\ \firstvec{0} & \secondvec{1} \\ \firstvec{1} & \secondvec{1} \end{bmatrix}. Remember that to invert a 2×22 \times 2 matrix [abcd]\begin{bmatrix} a & b \\ c & d \end{bmatrix}, we divide by the determinant ad−bcad - bc, swap the diagonal entries, and negate the off-diagonal entries: 1ad−bc[d−b−ca]\frac{1}{ad - bc}\begin{bmatrix} d & -b \\ -c & a \end{bmatrix}.

Building a projection matrix: worked example
A=[100111]\displaystyle A = \begin{bmatrix} \firstvec{1} & \secondvec{0} \\ \firstvec{0} & \secondvec{1} \\ \firstvec{1} & \secondvec{1} \end{bmatrix}
P\displaystyle P =A(ATA)−1AT\displaystyle = A(A^T A)^{-1} A^T =  ?\displaystyle = \;?
Text solution
Invert the 2×22 \times 2 dot-product matrix ATAA^T AUsing ATA=[2112]A^T A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} from earlier, the determinant is (2)(2)−(1)(1)=3(2)(2) - (1)(1) = 3.
(ATA)−1=13[2−1−12]\displaystyle (A^T A)^{-1} = \frac{1}{3}\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix}
Multiply AA, (ATA)−1(A^T A)^{-1}, and ATA^T togetherPulling out 13\frac{1}{3} and multiplying the first two matrices before multiplying by ATA^T gives the 3×33 \times 3 projection matrix PP.
P=13[100111][2−1−12][101011]=13[2−1−1211][101011]=13[2−11−121112]\displaystyle \begin{aligned} & P \\ &\quad = \frac{1}{3}\begin{bmatrix} \firstvec{1} & \secondvec{0} \\ \firstvec{0} & \secondvec{1} \\ \firstvec{1} & \secondvec{1} \end{bmatrix}\begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} \\ &\qquad \begin{bmatrix} \firstvec{1} & \firstvec{0} & \firstvec{1} \\ \secondvec{0} & \secondvec{1} & \secondvec{1} \end{bmatrix} \\ &\quad = \frac{1}{3}\begin{bmatrix} 2 & -1 \\ -1 & 2 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} \firstvec{1} & \firstvec{0} & \firstvec{1} \\ \secondvec{0} & \secondvec{1} & \secondvec{1} \end{bmatrix} \\ &\quad = \frac{1}{3}\begin{bmatrix} 2 & -1 & 1 \\ -1 & 2 & 1 \\ 1 & 1 & 2 \end{bmatrix} \end{aligned}
Multiply PP by any target vector b\target{b}Multiplying PP by our original target [341]\target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}} or by a new vector [003]\target{\begin{bmatrix} 0 \\ 0 \\ 3 \end{bmatrix}} gives each projection on the plane in one step.
P[341]=13[369]=[123],P[003]=[112]\displaystyle \begin{aligned} P\target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}} &= \frac{1}{3}\begin{bmatrix} 3 \\ 6 \\ 9 \end{bmatrix} = \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}}, \\ P\target{\begin{bmatrix} 0 \\ 0 \\ 3 \end{bmatrix}} &= \proj{\begin{bmatrix} 1 \\ 1 \\ 2 \end{bmatrix}} \end{aligned}

Once we have PP, projecting any vector in 3D space onto the plane takes just one multiplication, and multiplying by [003]\target{\begin{bmatrix} 0 \\ 0 \\ 3 \end{bmatrix}} picks out 33 times the third column of PP to give [112]\proj{\begin{bmatrix} 1 \\ 1 \\ 2 \end{bmatrix}}.

Now see what happens when we take a projection p=Pb\proj{p} = P\target{b} that is already on the plane and multiply it by PP a second time.

Why projecting twice changes nothing

When we multiply PP by PP, ATAA^T A ends up right next to its own inverse (ATA)−1(A^T A)^{-1} in the middle, and they cancel to the identity matrix II:

P2=(A(ATA)−1AT)(A(ATA)−1AT)=A(ATA)−1(ATA)(ATA)−1⏟IAT=A(ATA)−1AT=P\displaystyle \begin{aligned} P^2 &= \bigl(A(A^T A)^{-1} A^T\bigr)\bigl(A(A^T A)^{-1} A^T\bigr) \\ &= A(A^T A)^{-1}\underbrace{(A^T A)(A^T A)^{-1}}_{I} A^T \\ &= A(A^T A)^{-1} A^T = P \end{aligned}

Every target vector b\target{b} along the same perpendicular line has the exact same projection p=Pb\proj{p} = P\target{b} on the plane, and because p\proj{p} is already on the plane, multiplying by PP a second time keeps it at P2b=PbP^2 \target{b} = P\target{b}.

Every vector along the same perpendicular line lands on the exact same projection on the plane, and because that projection is already on the plane, projecting it a second time leaves it in place.
5

The shortcut for orthogonal vectors

When the basis vectors a1\firstvec{a_1} and a2\secondvec{a_2} are already perpendicular to each other (a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0), we can find the projection p\proj{p} without solving a system of equations or inverting a matrix.

The shortcut for orthogonal vectors

When a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0, the off-diagonal entries in ATAA^T A are zero, so each equation has only one unknown and we can solve for x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2} separately:

[a1⋅a100a2⋅a2][x^1x^2]=[a1⋅ba2⋅b]  ⟹  x^1=a1⋅ba1⋅a1,x^2=a2⋅ba2⋅a2\displaystyle \begin{aligned} &\begin{bmatrix} \firstvec{a_1} \cdot \firstvec{a_1} & 0 \\ 0 & \secondvec{a_2} \cdot \secondvec{a_2} \end{bmatrix} \begin{bmatrix} \firstvec{\hat{x}_1} \\ \secondvec{\hat{x}_2} \end{bmatrix} = \begin{bmatrix} \firstvec{a_1} \cdot \target{b} \\ \secondvec{a_2} \cdot \target{b} \end{bmatrix} \\ &\implies \quad \firstvec{\hat{x}_1} = \frac{\firstvec{a_1} \cdot \target{b}}{\firstvec{a_1} \cdot \firstvec{a_1}}, \quad \secondvec{\hat{x}_2} = \frac{\secondvec{a_2} \cdot \target{b}}{\secondvec{a_2} \cdot \secondvec{a_2}} \end{aligned}

Plugging these multipliers into p=x^1a1+x^2a2\proj{p} = \firstvec{\hat{x}_1} \firstvec{a_1} + \secondvec{\hat{x}_2} \secondvec{a_2} splits the plane projection into the sum of two separate 1D line projections:

p\displaystyle \proj{p} =a1⋅ba1⋅a1a1⏟1D projection onto a1+a2⋅ba2⋅a2a2⏟1D projection onto a2\displaystyle = \underbrace{\frac{\firstvec{a_1} \cdot \target{b}}{\firstvec{a_1} \cdot \firstvec{a_1}} \firstvec{a_1}}_{\text{1D projection onto }\firstvec{a_1}} + \underbrace{\frac{\secondvec{a_2} \cdot \target{b}}{\secondvec{a_2} \cdot \secondvec{a_2}} \secondvec{a_2}}_{\text{1D projection onto }\secondvec{a_2}}

When a1\firstvec{a_1} and a2\secondvec{a_2} are not perpendicular (a1⋅a2≠0\firstvec{a_1} \cdot \secondvec{a_2} \neq 0), they point partly in the same direction or partly in opposite directions, so adding the two separate 1D line projections misses p\proj{p}; only when a1\firstvec{a_1} and a2\secondvec{a_2} meet at 90∘90^\circ (a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0) do the two 1D projections add head-to-tail to land right on p\proj{p}.

The two separate 1D projections add up to the plane projection only when the two vectors on the plane meet at a right angle.

Let's project the same target vector b=[341]\target{b} = \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}} onto the same plane, now using the two perpendicular basis vectors a1=[101]\firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}} and a2=[−121]\secondvec{a_2} = \secondvec{\begin{bmatrix} -1 \\ 2 \\ 1 \end{bmatrix}}.

Projecting onto orthogonal vectors: worked example
b=[341]\displaystyle \target{b} = \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}}
a1=[101],a2=[−121]\displaystyle \firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}}, \quad \secondvec{a_2} = \secondvec{\begin{bmatrix} -1 \\ 2 \\ 1 \end{bmatrix}}
a1⋅a2\displaystyle \firstvec{a_1} \cdot \secondvec{a_2} =  ?,p\displaystyle = \;?, \quad \proj{p} =  ?\displaystyle = \;?
Text solution
Check that the basis vectors are perpendicularTaking the dot product a1⋅a2\firstvec{a_1} \cdot \secondvec{a_2} confirms that they meet at 90∘90^\circ.
a1⋅a2\displaystyle \firstvec{a_1} \cdot \secondvec{a_2} =(1)(−1)+(0)(2)+(1)(1)\displaystyle = (\firstvec{1})(\secondvec{-1}) + (\firstvec{0})(\secondvec{2}) + (\firstvec{1})(\secondvec{1}) =0\displaystyle = 0
Compute the 1D projection of b\target{b} onto each vector separatelyBecause a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0, we can use the single-line formula on each vector on its own.
x^1a1=a1⋅ba1⋅a1a1=42[101]=[202],x^2a2=a2⋅ba2⋅a2a2=66[−121]=[−121]\displaystyle \begin{aligned} \firstvec{\hat{x}_1 a_1} &= \frac{\firstvec{a_1} \cdot \target{b}}{\firstvec{a_1} \cdot \firstvec{a_1}} \firstvec{a_1} \\ &= \frac{4}{2}\firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}} = \firstvec{\begin{bmatrix} 2 \\ 0 \\ 2 \end{bmatrix}}, \\ \secondvec{\hat{x}_2 a_2} &= \frac{\secondvec{a_2} \cdot \target{b}}{\secondvec{a_2} \cdot \secondvec{a_2}} \secondvec{a_2} \\ &= \frac{6}{6}\secondvec{\begin{bmatrix} -1 \\ 2 \\ 1 \end{bmatrix}} = \secondvec{\begin{bmatrix} -1 \\ 2 \\ 1 \end{bmatrix}} \end{aligned}
Add the two 1D projections head-to-tailAdding these two perpendicular pieces gives the projection p\proj{p} on the plane.
p=[202]+[−121]=[123]\displaystyle \proj{p} = {\firstvec{\begin{bmatrix} 2 \\ 0 \\ 2 \end{bmatrix}} + \secondvec{\begin{bmatrix} -1 \\ 2 \\ 1 \end{bmatrix}}} = \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}}

We get the exact same closest point p=[123]\proj{p} = \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}} on the plane without solving a system or inverting a matrix. Better yet, when perpendicular basis vectors also have length 11 (a1⋅a1=1\firstvec{a_1} \cdot \firstvec{a_1} = 1 and a2⋅a2=1\secondvec{a_2} \cdot \secondvec{a_2} = 1, called an orthonormal basis), the denominators are 11 too and ATA=IA^T A = I.

Key concepts

Before setting up a system of equations to project b\target{b} onto the subspace spanned by a1\firstvec{a_1} and a2\secondvec{a_2}, check whether the vector or basis gives you a shortcut:

When b\target{b} is already on the subspace (Ax=bAx = \target{b} has a solution)

Do not set up dot-product equations at all. A vector already on the plane is its own projection (Pb=bP\target{b} = \target{b}, so P2b=PbP^2 \target{b} = P\target{b}).

When the basis vectors are perpendicular (a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0)

Do not solve a system of equations or invert ATAA^T A. The off-diagonal terms a1⋅a2\firstvec{a_1} \cdot \secondvec{a_2} are zero, so project b\target{b} onto each line separately and add the pieces:

p=a1⋅ba1⋅a1a1+a2⋅ba2⋅a2a2(for a1=[101], a2=[−121], b=[341]:p=42a1+66a2=[123])\displaystyle \begin{gathered} \proj{p} = \frac{\firstvec{a_1} \cdot \target{b}}{\firstvec{a_1} \cdot \firstvec{a_1}} \firstvec{a_1} + \frac{\secondvec{a_2} \cdot \target{b}}{\secondvec{a_2} \cdot \secondvec{a_2}} \secondvec{a_2} \\ \left(\begin{gathered}\text{for } \firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}},\, \secondvec{a_2} = \secondvec{\begin{bmatrix} -1 \\ 2 \\ 1 \end{bmatrix}},\, \\ \target{b} = \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}}: \\ \proj{p} = \frac{4}{2}\firstvec{a_1} + \frac{6}{6}\secondvec{a_2} = \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}}\end{gathered}\right) \end{gathered}
If the perpendicular vectors also have length 11 (a1⋅a1=1\firstvec{a_1} \cdot \firstvec{a_1} = 1 and a2⋅a2=1\secondvec{a_2} \cdot \secondvec{a_2} = 1), then ATA=IA^T A = I and P=AATP = AA^T.

When the basis vectors are not perpendicular (a1⋅a2≠0\firstvec{a_1} \cdot \secondvec{a_2} \neq 0) for a single target b\target{b}

Make the error b−p\err{b - p} meet both a1\firstvec{a_1} and a2\secondvec{a_2} at 90∘90^\circ by solving the normal equations ATAx^=ATbA^T A\hat{x} = A^T \target{b}, then remember to multiply by AA to get the point on the plane, p=Ax^=x^1a1+x^2a2\proj{p} = A\hat{x} = \firstvec{\hat{x}_1} \firstvec{a_1} + \secondvec{\hat{x}_2} \secondvec{a_2} (for a1=[101], a2=[011], b=[341]\firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 0 \\ 1 \end{bmatrix}},\, \secondvec{a_2} = \secondvec{\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}},\, \target{b} = \target{\begin{bmatrix} 3 \\ 4 \\ 1 \end{bmatrix}}, solving [2112][x^1x^2]=[45]\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\begin{bmatrix} \firstvec{\hat{x}_1} \\ \secondvec{\hat{x}_2} \end{bmatrix} = \begin{bmatrix} 4 \\ 5 \end{bmatrix} gives x^=[12]\hat{x} = \begin{bmatrix} 1 \\ 2 \end{bmatrix} and p=1a1+2a2=[123]\proj{p} = 1\firstvec{a_1} + 2\secondvec{a_2} = \proj{\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}}).

When projecting many vectors onto the same subspace

Combine both steps into the projection matrix

P=A(ATA)−1AT(never split (ATA)−1, since a tall matrix A has no inverse).\displaystyle \begin{gathered}P = A(A^T A)^{-1} A^T \\ (\text{never split } \\ (A^T A)^{-1}\text{, since a tall matrix } \\ A\text{ has no inverse}).\end{gathered}
For any b\target{b}, p=Pb\proj{p} = P\target{b} lies on the plane and the error b−p\err{b - p} is perpendicular to the plane, so projecting the error gives P(b−p)=Pb−P2b=[000]P(\err{b - p}) = P\target{b} - P^2 \target{b} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}.

6

Practice

1

Let the plane be spanned by a1=[110]\firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 1 \\ 0 \end{bmatrix}} and a2=[011]\secondvec{a_2} = \secondvec{\begin{bmatrix} 0 \\ 1 \\ 1 \end{bmatrix}}, and let the target vector be b=[234]\target{b} = \target{\begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix}}. Set up and solve the dot-product equations to find the multipliers x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2}, the projection vector p\proj{p} on the plane, and the error vector b−p\err{b - p}.

First find the five dot products a1⋅a1\firstvec{a_1} \cdot \firstvec{a_1}, a1⋅a2\firstvec{a_1} \cdot \secondvec{a_2}, a2⋅a2\secondvec{a_2} \cdot \secondvec{a_2}, a1⋅b\firstvec{a_1} \cdot \target{b}, and a2⋅b\secondvec{a_2} \cdot \target{b}; after solving for x^1\firstvec{\hat{x}_1} and x^2\secondvec{\hat{x}_2}, how do you combine a1\firstvec{a_1} and a2\secondvec{a_2} to get p\proj{p}?

The five dot products are a1⋅a1=2\firstvec{a_1} \cdot \firstvec{a_1} = 2, a1⋅a2=1\firstvec{a_1} \cdot \secondvec{a_2} = 1, a2⋅a2=2\secondvec{a_2} \cdot \secondvec{a_2} = 2, a1⋅b=5\firstvec{a_1} \cdot \target{b} = 5, and a2⋅b=7\secondvec{a_2} \cdot \target{b} = 7. (Watch out for a1⋅a2=1\firstvec{a_1} \cdot \secondvec{a_2} = 1: ignoring it and using the shortcut x^1=52,x^2=72\firstvec{\hat{x}_1} = \frac{5}{2}, \secondvec{\hat{x}_2} = \frac{7}{2} for perpendicular vectors gives the wrong point [52672]\begin{bmatrix} \tfrac{5}{2} \\ 6 \\ \tfrac{7}{2} \end{bmatrix}.) Solving the system of two equations
2x^1+x^2=5x^1+2x^2=7\displaystyle \begin{aligned} 2\firstvec{\hat{x}_1} + \secondvec{\hat{x}_2} &= 5 \\ \firstvec{\hat{x}_1} + 2\secondvec{\hat{x}_2} &= 7 \end{aligned}
gives the multipliers x^1=1\firstvec{\hat{x}_1} = 1 and x^2=3\secondvec{\hat{x}_2} = 3. Combining the vectors gives the projection p=1a1+3a2=[143]\proj{p} = 1\firstvec{a_1} + 3\secondvec{a_2} = \proj{\begin{bmatrix} 1 \\ 4 \\ 3 \end{bmatrix}}, and subtracting in the order b−p\err{b - p} (not p−bp - b, which flips the signs to [−11−1]\begin{bmatrix} -1 \\ 1 \\ -1 \end{bmatrix}) gives the error vector
b−p=[234]−[143]=[1−11].\displaystyle \err{b - p} = \target{\begin{bmatrix} 2 \\ 3 \\ 4 \end{bmatrix}} - \proj{\begin{bmatrix} 1 \\ 4 \\ 3 \end{bmatrix}} = \err{\begin{bmatrix} 1 \\ -1 \\ 1 \end{bmatrix}}.
2

A plane is spanned by a1=[102]\firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 0 \\ 2 \end{bmatrix}} and a2=[111]\secondvec{a_2} = \secondvec{\begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix}}, and the target vector is b=[524]\target{b} = \target{\begin{bmatrix} 5 \\ 2 \\ 4 \end{bmatrix}}. A student picks the point q=2a1+a2=[315]q = 2\firstvec{a_1} + \secondvec{a_2} = \begin{bmatrix} 3 \\ 1 \\ 5 \end{bmatrix} on the plane, checks that a1⋅(b−q)=0\firstvec{a_1} \cdot (\err{b - q}) = 0, and claims that qq is the projection of b\target{b} onto the plane. Is the student right? Explain why or why not.

Being perpendicular to a1\firstvec{a_1} is only half the test. What else must the error vector b−q\err{b - q} be perpendicular to?

No. The error vector is b−q=[21−1]\err{b - q} = \err{\begin{bmatrix} 2 \\ 1 \\ -1 \end{bmatrix}}. (Stopping after checking only a1⋅(b−q)=2+0−2=0\firstvec{a_1} \cdot (\err{b - q}) = 2 + 0 - 2 = 0 leads to the wrong answer "Yes," because an error vector that is perpendicular to just one vector on the plane can still tilt along the other.) Checking the second basis vector shows that b−q\err{b - q} is not perpendicular to a2\secondvec{a_2}:
a2⋅(b−q)\displaystyle \secondvec{a_2} \cdot (\err{b - q}) =(1)(2)+(1)(1)+(1)(−1)\displaystyle = (\secondvec{1})(\err{2}) + (\secondvec{1})(\err{1}) + (\secondvec{1})(\err{-1}) =2≠0.\displaystyle = 2 \neq 0.
3

Project b=[217]\target{b} = \target{\begin{bmatrix} 2 \\ 1 \\ 7 \end{bmatrix}} onto the plane spanned by a1=[122]\firstvec{a_1} = \firstvec{\begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}} and a2=[21−2]\secondvec{a_2} = \secondvec{\begin{bmatrix} 2 \\ 1 \\ -2 \end{bmatrix}} by first checking the dot product a1⋅a2\firstvec{a_1} \cdot \secondvec{a_2}.

Since a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0, how do the two 1D line projections onto a1\firstvec{a_1} and a2\secondvec{a_2} combine?

Because a1⋅a2=2+2−4=0\firstvec{a_1} \cdot \secondvec{a_2} = 2 + 2 - 4 = 0, the basis vectors are already perpendicular, so the projection is the sum of the two 1D line projections. Using a1⋅a1=9\firstvec{a_1} \cdot \firstvec{a_1} = 9 and a2⋅a2=9\secondvec{a_2} \cdot \secondvec{a_2} = 9 in the denominators (be careful not to divide by the lengths 9=3\sqrt{9} = 3, which triples the answer to [0918]\begin{bmatrix} 0 \\ 9 \\ 18 \end{bmatrix}), we get x^1=a1⋅ba1⋅a1=189=2\firstvec{\hat{x}_1} = \frac{\firstvec{a_1} \cdot \target{b}}{\firstvec{a_1} \cdot \firstvec{a_1}} = \frac{18}{9} = 2 and x^2=a2⋅ba2⋅a2=−99=−1\secondvec{\hat{x}_2} = \frac{\secondvec{a_2} \cdot \target{b}}{\secondvec{a_2} \cdot \secondvec{a_2}} = \frac{-9}{9} = -1 (watch the negative sign on x^2\secondvec{\hat{x}_2}; using +1+1 gives [452]\begin{bmatrix} 4 \\ 5 \\ 2 \end{bmatrix}). Adding the two pieces gives
p=2[122]−1[21−2]=[244]+[−2−12]=[036].\displaystyle \proj{p} = {2\firstvec{\begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix}} - 1\secondvec{\begin{bmatrix} 2 \\ 1 \\ -2 \end{bmatrix}}} = {\firstvec{\begin{bmatrix} 2 \\ 4 \\ 4 \end{bmatrix}} + \secondvec{\begin{bmatrix} -2 \\ -1 \\ 2 \end{bmatrix}}} = \proj{\begin{bmatrix} 0 \\ 3 \\ 6 \end{bmatrix}}.
4

Find the 3×33 \times 3 projection matrix P=A(ATA)−1ATP = A(A^T A)^{-1}A^T onto the column space of A=[110101]A = \begin{bmatrix} \firstvec{1} & \secondvec{1} \\ \firstvec{0} & \secondvec{1} \\ \firstvec{0} & \secondvec{1} \end{bmatrix}, and use PP to find the projection of b=[006]\target{b} = \target{\begin{bmatrix} 0 \\ 0 \\ 6 \end{bmatrix}}.

Start with the 2×22 \times 2 dot-product matrix ATAA^T A; after inverting it and multiplying A(ATA)−1ATA(A^T A)^{-1}A^T, which column of PP gets multiplied by 66?

The dot-product matrix is ATA=[1113]A^T A = \begin{bmatrix} 1 & 1 \\ 1 & 3 \end{bmatrix}, which has determinant (1)(3)−(1)(1)=2(1)(3) - (1)(1) = 2 and inverse (ATA)−1=12[3−1−11](A^T A)^{-1} = \frac{1}{2}\begin{bmatrix} 3 & -1 \\ -1 & 1 \end{bmatrix} (forgetting the 12\frac{1}{2} from the determinant doubles the answer to [066]\begin{bmatrix} 0 \\ 6 \\ 6 \end{bmatrix}, and forgetting to negate the off-diagonal 11's gives [633]\begin{bmatrix} 6 \\ 3 \\ 3 \end{bmatrix}). Multiplying A(ATA)−1ATA(A^T A)^{-1}A^T gives P=12[200011011]P = \frac{1}{2}\begin{bmatrix} 2 & 0 & 0 \\ 0 & 1 & 1 \\ 0 & 1 & 1 \end{bmatrix}, and multiplying PbP\target{b} (66 times the third column of PP) gives
p=[033].\displaystyle \proj{p} = \proj{\begin{bmatrix} 0 \\ 3 \\ 3 \end{bmatrix}}.
5

Suppose PP is the projection matrix onto a plane in 3D space, and for b=[422]\target{b} = \target{\begin{bmatrix} 4 \\ 2 \\ 2 \end{bmatrix}} you are told that its projection onto the plane is p=Pb=[330]\proj{p} = P\target{b} = \proj{\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix}}. Without knowing the plane's vectors or the entries of PP, find (a) P[330]P\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix} and (b) P[1−12]P\begin{bmatrix} 1 \\ -1 \\ 2 \end{bmatrix}.

Where is [330]\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix} relative to the plane, and how is [1−12]\begin{bmatrix} 1 \\ -1 \\ 2 \end{bmatrix} related to b\target{b} and p\proj{p}?

(a) Because [330]=Pb\proj{\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix}} = P\target{b} is already on the plane, projecting it a second time leaves it unchanged (P2b=PbP^2 \target{b} = P\target{b}):
P[330]=[330].\displaystyle P\proj{\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix}} = \proj{\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix}}.
(b) Notice that [1−12]=[422]−[330]=b−p\err{\begin{bmatrix} 1 \\ -1 \\ 2 \end{bmatrix}} = \target{\begin{bmatrix} 4 \\ 2 \\ 2 \end{bmatrix}} - \proj{\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix}} = \err{b - p} is the perpendicular error vector (do not confuse it with a vector that projects to [330]\proj{\begin{bmatrix} 3 \\ 3 \\ 0 \end{bmatrix}}). Multiplying the error vector by PP gives P(b−p)=Pb−P2b=p−pP(\err{b - p}) = P\target{b} - P^2 \target{b} = \proj{p} - \proj{p}:
P[1−12]=[000].\displaystyle P\err{\begin{bmatrix} 1 \\ -1 \\ 2 \end{bmatrix}} = \begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}.
6

Suppose the columns a1\firstvec{a_1} and a2\secondvec{a_2} of a 3×23 \times 2 matrix AA are perpendicular to each other and each has length 11 (an orthonormal basis). (a) What is the 2×22 \times 2 matrix ATAA^T A, and how does the projection matrix formula P=A(ATA)−1ATP = A(A^T A)^{-1}A^T simplify? (b) Why can't we rewrite P=A(ATA)−1ATP = A(A^T A)^{-1}A^T as AA−1(AT)−1AT=IAA^{-1}(A^T)^{-1}A^T = I for any 3×23 \times 2 matrix AA?

For (a), write out the four dot products in ATAA^T A when a1⋅a1=1\firstvec{a_1} \cdot \firstvec{a_1} = 1, a2⋅a2=1\secondvec{a_2} \cdot \secondvec{a_2} = 1, and a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0; for (b), what shape must a matrix have to have an inverse?

(a) Since a1⋅a1=1\firstvec{a_1} \cdot \firstvec{a_1} = 1, a2⋅a2=1\secondvec{a_2} \cdot \secondvec{a_2} = 1, and a1⋅a2=0\firstvec{a_1} \cdot \secondvec{a_2} = 0, the 2×22 \times 2 dot-product matrix is the identity matrix ATA=[1001]=IA^T A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = I, so the projection matrix simplifies to P=AI−1AT=AATP = AI^{-1}A^T = AA^T (be careful not to swap the order and assume the 3×33 \times 3 matrix AATAA^T equals II). (b) Because AA is 3×23 \times 2 and not square, A−1A^{-1} and (AT)−1(A^T)^{-1} do not exist, so (ATA)−1(A^T A)^{-1} cannot be split. Geometrically, PP projects 3D vectors onto a 2D plane, whereas the 3×33 \times 3 identity matrix II leaves every 3D vector off the plane unchanged.