When a target vector b lies outside the subspace spanned by our vectors, the equation Ax=b has no solution, so we look for the closest point p on the subspace instead.
1
The closest point on a subspace
Given two vectors a1 and a2, their combinations form a flat plane: the column space of the matrix A=[a1a2]. When a target vector b points off that plane, the projection p of b onto the subspace is the point on the plane directly beneath b.
The closest point on a subspace
To make the distance from p to b as short as possible, the error vector from p to b must meet the plane at a 90∘ angle:
error vectorb−p⊥plane
2
Setting up the dot-product equations
Because the projection p lies on the plane spanned by a1 and a2, it is a combination p=x^1a1+x^2a2, and we want to find the two multipliers x^1 and x^2.
Remember, when we project onto a single line along a, making the error b−x^a meet a at 90∘ means setting a⋅(b−x^a)=0, which solves in one step to give x^=a⋅aa⋅b and p=a⋅aa⋅ba. For a plane spanned by two vectors a1 and a2, we apply that same 90∘ dot-product rule to both vectors at once.
Setting up the dot-product equations
The hat on x^1 and x^2 reminds us that these multipliers give the closest point p=Ax^ on the plane, not an exact solution to Ax=b. For the error vector b−p to be perpendicular to the plane, its dot product with both a1 and a2 must be zero:
a1⋅(b−p)a2⋅(b−p)=0=0
Substituting p=x^1a1+x^2a2 turns these two dot-product equations into a system of two linear equations for x^1 and x^2:
Making the error vector perpendicular to just one basis vector still lets it tilt along the other; the error vector is shortest and perpendicular to the entire plane only at the single point p where its dot products with both a1 and a2 equal zero.
The error vector stands perpendicular to the whole plane, and reaches its shortest length, only where it meets both basis vectors at a right angle.
Now let's use these two equations to project b=341 onto the plane spanned by a1=101 and a2=011.
Projecting onto a plane: worked example
a1=101,a2=011
b=341,p=?
Text solution
Compute the five dot productsTake the dot products of a1 and a2 with each other for the left-hand side, and with b for the right-hand side.
a1⋅a1a1⋅b=2,=4,a1⋅a2a2⋅a2a2⋅b=1,=2,=5
Solve the linear system for the multipliers x^1 and x^2Plug the five dot products into our two equations and solve for x^1 and x^2.
2x^1+1x^21x^1+2x^2=4=5⟹x^1=1,x^2=2
Combine a1 and a2 using x^1 and x^2 to find pDon't stop at the multipliers; use them to get the actual point on the plane.
p=x^1a1+x^2a2=1101+2011=123
Subtract p from b to check that the error is perpendicularTaking the dot product of the error vector b−p with a1 and a2 confirms that it meets both vectors at 90∘.
Writing this in matrix form gives the normal equations:
Normal EquationsATAx^=ATb
4
The projection matrix
The projection matrix P is a single matrix that multiplies any target vector b to give its projection p=Pb on the subspace directly, without solving the system of equations again for each new vector.
The projection matrix
When the columns of A are independent, the square dot-product matrix ATA is invertible. Solving the normal equations for x^ and multiplying by A to get p=Ax^ combines both steps into P:
x^p=Ax^=(ATA)−1ATbSolve normal equations for x^=PA(ATA)−1ATbMultiply by A to get p
Pulling out the matrix in front of b gives the formula for the projection matrix onto the column space of A:
P=A(ATA)−1AT
Because A is a tall rectangular matrix and not square, A−1 does not exist, so never try to split (ATA)−1 into A−1(AT)−1 to cancel the A's. Only the square matrix ATA has an inverse, and it stays together as a single piece.
Let's find the 3×3 projection matrix P for our plane with A=101011. Remember that to invert a 2×2 matrix [acbd], we divide by the determinant ad−bc, swap the diagonal entries, and negate the off-diagonal entries: ad−bc1[d−c−ba].
Building a projection matrix: worked example
A=101011
P=A(ATA)−1AT=?
Text solution
Invert the 2×2 dot-product matrix ATAUsing ATA=[2112] from earlier, the determinant is (2)(2)−(1)(1)=3.
(ATA)−1=31[2−1−12]
Multiply A,(ATA)−1, and AT togetherPulling out 31 and multiplying the first two matrices before multiplying by AT gives the 3×3 projection matrix P.
Multiply P by any target vector bMultiplying P by our original target 341 or by a new vector 003 gives each projection on the plane in one step.
P341P003=31369=123,=112
Once we have P, projecting any vector in 3D space onto the plane takes just one multiplication, and multiplying by 003 picks out 3 times the third column of P to give 112.
Now see what happens when we take a projection p=Pb that is already on the plane and multiply it by P a second time.
Why projecting twice changes nothing
When we multiply P by P,ATA ends up right next to its own inverse (ATA)−1 in the middle, and they cancel to the identity matrix I:
Every target vector b along the same perpendicular line has the exact same projection p=Pb on the plane, and because p is already on the plane, multiplying by P a second time keeps it at P2b=Pb.
Every vector along the same perpendicular line lands on the exact same projection on the plane, and because that projection is already on the plane, projecting it a second time leaves it in place.
5
The shortcut for orthogonal vectors
When the basis vectors a1 and a2 are already perpendicular to each other (a1⋅a2=0), we can find the projection p without solving a system of equations or inverting a matrix.
The shortcut for orthogonal vectors
When a1⋅a2=0, the off-diagonal entries in ATA are zero, so each equation has only one unknown and we can solve for x^1 and x^2 separately:
When a1 and a2 are not perpendicular (a1⋅a2=0), they point partly in the same direction or partly in opposite directions, so adding the two separate 1D line projections misses p; only when a1 and a2 meet at 90∘ (a1⋅a2=0) do the two 1D projections add head-to-tail to land right on p.
The two separate 1D projections add up to the plane projection only when the two vectors on the plane meet at a right angle.
Let's project the same target vector b=341 onto the same plane, now using the two perpendicular basis vectors a1=101 and a2=−121.
Projecting onto orthogonal vectors: worked example
b=341
a1=101,a2=−121
a1⋅a2=?,p=?
Text solution
Check that the basis vectors are perpendicularTaking the dot product a1⋅a2 confirms that they meet at 90∘.
a1⋅a2=(1)(−1)+(0)(2)+(1)(1)=0
Compute the 1D projection of b onto each vector separatelyBecause a1⋅a2=0, we can use the single-line formula on each vector on its own.
Add the two 1D projections head-to-tailAdding these two perpendicular pieces gives the projection p on the plane.
p=202+−121=123
We get the exact same closest point p=123 on the plane without solving a system or inverting a matrix. Better yet, when perpendicular basis vectors also have length 1 (a1⋅a1=1 and a2⋅a2=1, called an orthonormal basis), the denominators are 1 too and ATA=I.
Key concepts
Before setting up a system of equations to project b onto the subspace spanned by a1 and a2, check whether the vector or basis gives you a shortcut:
When b is already on the subspace (Ax=b has a solution)
Do not set up dot-product equations at all. A vector already on the plane is its own projection (Pb=b, so P2b=Pb).
When the basis vectors are perpendicular (a1⋅a2=0)
Do not solve a system of equations or invert ATA. The off-diagonal terms a1⋅a2 are zero, so project b onto each line separately and add the pieces:
If the perpendicular vectors also have length 1 (a1⋅a1=1 and a2⋅a2=1), then ATA=I and P=AAT.
When the basis vectors are not perpendicular (a1⋅a2=0) for a single target b
Make the error b−p meet both a1 and a2 at 90∘ by solving the normal equations ATAx^=ATb, then remember to multiply by A to get the point on the plane, p=Ax^=x^1a1+x^2a2 (for a1=101,a2=011,b=341, solving [2112][x^1x^2]=[45] gives x^=[12] and p=1a1+2a2=123).
When projecting many vectors onto the same subspace
Combine both steps into the projection matrix
P=A(ATA)−1AT(never split (ATA)−1, since a tall matrix A has no inverse).
For any b,p=Pb lies on the plane and the error b−p is perpendicular to the plane, so projecting the error gives P(b−p)=Pb−P2b=000.
6
Practice
1
Let the plane be spanned by a1=110 and a2=011, and let the target vector be b=234. Set up and solve the dot-product equations to find the multipliers x^1 and x^2, the projection vector p on the plane, and the error vector b−p.
Watch out for a1⋅a2=1: ignoring it and using the shortcut x^1=25,x^2=27 for perpendicular vectors gives the wrong point 25627.
First find the five dot products a1⋅a1,a1⋅a2,a2⋅a2,a1⋅b, and a2⋅b; after solving for x^1 and x^2, how do you combine a1 and a2 to get p?
The five dot products are a1⋅a1=2,a1⋅a2=1,a2⋅a2=2,a1⋅b=5, and a2⋅b=7. (Watch out for a1⋅a2=1: ignoring it and using the shortcut x^1=25,x^2=27 for perpendicular vectors gives the wrong point 25627.) Solving the system of two equations
2x^1+x^2x^1+2x^2=5=7
gives the multipliers x^1=1 and x^2=3. Combining the vectors gives the projection p=1a1+3a2=143, and subtracting in the order b−p (not p−b, which flips the signs to −11−1) gives the error vector
b−p=234−143=1−11.
2
A plane is spanned by a1=102 and a2=111, and the target vector is b=524. A student picks the point q=2a1+a2=315 on the plane, checks that a1⋅(b−q)=0, and claims that q is the projection of b onto the plane. Is the student right? Explain why or why not.
Stopping after checking only a1⋅(b−q)=2+0−2=0 leads to the wrong answer "Yes," because an error vector that is perpendicular to just one vector on the plane can still tilt along the other.
Being perpendicular to a1 is only half the test. What else must the error vector b−q be perpendicular to?
No. The error vector is b−q=21−1. (Stopping after checking only a1⋅(b−q)=2+0−2=0 leads to the wrong answer "Yes," because an error vector that is perpendicular to just one vector on the plane can still tilt along the other.) Checking the second basis vector shows that b−q is not perpendicular to a2:
a2⋅(b−q)=(1)(2)+(1)(1)+(1)(−1)=2=0.
3
Project b=217 onto the plane spanned by a1=122 and a2=21−2 by first checking the dot product a1⋅a2.
Watch the negative sign on x^2; using +1 gives 452.
Since a1⋅a2=0, how do the two 1D line projections onto a1 and a2 combine?
Because a1⋅a2=2+2−4=0, the basis vectors are already perpendicular, so the projection is the sum of the two 1D line projections. Using a1⋅a1=9 and a2⋅a2=9 in the denominators (be careful not to divide by the lengths 9=3, which triples the answer to 0918), we get x^1=a1⋅a1a1⋅b=918=2 and x^2=a2⋅a2a2⋅b=9−9=−1 (watch the negative sign on x^2; using +1 gives 452). Adding the two pieces gives
p=2122−121−2=244+−2−12=036.
4
Find the 3×3 projection matrix P=A(ATA)−1AT onto the column space of A=100111, and use P to find the projection of b=006.
Start with the 2×2 dot-product matrix ATA; after inverting it and multiplying A(ATA)−1AT, which column of P gets multiplied by 6?
The dot-product matrix is ATA=[1113], which has determinant (1)(3)−(1)(1)=2 and inverse (ATA)−1=21[3−1−11] (forgetting the 21 from the determinant doubles the answer to 066, and forgetting to negate the off-diagonal 1's gives 633). Multiplying A(ATA)−1AT gives P=21200011011, and multiplying Pb (6 times the third column of P) gives
p=033.
5
Suppose P is the projection matrix onto a plane in 3D space, and for b=422 you are told that its projection onto the plane is p=Pb=330. Without knowing the plane's vectors or the entries of P, find (a) P330 and (b) P1−12.
Where is 330 relative to the plane, and how is 1−12 related to b and p?
(a) Because 330=Pb is already on the plane, projecting it a second time leaves it unchanged (P2b=Pb):
P330=330.
(b) Notice that 1−12=422−330=b−p is the perpendicular error vector (do not confuse it with a vector that projects to 330). Multiplying the error vector by P gives P(b−p)=Pb−P2b=p−p:
P1−12=000.
6
Suppose the columns a1 and a2 of a 3×2 matrix A are perpendicular to each other and each has length 1 (an orthonormal basis). (a) What is the 2×2 matrix ATA, and how does the projection matrix formula P=A(ATA)−1AT simplify? (b) Why can't we rewrite P=A(ATA)−1AT as AA−1(AT)−1AT=I for any 3×2 matrix A?
For (a), write out the four dot products in ATA when a1⋅a1=1,a2⋅a2=1, and a1⋅a2=0; for (b), what shape must a matrix have to have an inverse?
(a) Since a1⋅a1=1,a2⋅a2=1, and a1⋅a2=0, the 2×2 dot-product matrix is the identity matrix ATA=[1001]=I, so the projection matrix simplifies to P=AI−1AT=AAT (be careful not to swap the order and assume the 3×3 matrix AAT equals I). (b) Because A is 3×2 and not square, A−1 and (AT)−1 do not exist, so (ATA)−1 cannot be split. Geometrically, P projects 3D vectors onto a 2D plane, whereas the 3×3 identity matrix I leaves every 3D vector off the plane unchanged.