Linear AlgebraSymmetric Matrices and Quadratic Forms3 of 4
Symmetric Matrices and Quadratic Forms · 3
Positive Definite Matrices
In single-variable calculus, a positive number a makes f(x)=ax2 curve upward like a bowl with a single minimum at the origin. In multiple dimensions, we need a way to tell when a symmetric matrix A makes the quadratic form xTAx curve upward in every direction the same way.
1
Bowls, Angles, and Positive Definite Matrices
A positive definite matrix is a symmetric matrix A whose quadratic form xTAx is positive for every non-zero vector x, so its graph forms an upward-facing bowl with a single minimum at the origin.
Bowls, Angles, and Positive Definite Matrices
Rotating the coordinates to line up with the eigenvectors of A turns the quadratic form into a sum of squares weighted by the eigenvalues, which tells us the shape of the surface:
quadratic formxTAx=λ1u2+λ2v2
Positive definite (all λi>0)
Positive semi-definite (one λi=0, the other >0)
Indefinite (λi of opposite signs)
Grouping xTAx as xT(Ax) shows what positive definiteness means geometrically without solving for eigenvalues: it is the dot product between the starting vector x and the transformed vector Ax.
xTAx=dot product of x and AxxT(Ax)⟺angle(x,Ax)>0<90∘ for all x=0
As the unit vector x goes full circle around the origin for a positive definite matrix A, the transformed vector Ax stretches and tilts toward the steeper axis of the bowl, but stays within 90∘ of x, keeping the height xTAx positive everywhere.
Point the input vector in any direction around the circle, and the output vector stays within 90 degrees of it while the height on the bowl stays above zero.
2
Completing the Square and Pivots
Finding eigenvalues means solving a polynomial that quickly gets messy, so we want a way to test whether xTAx stays positive by looking directly at the entries of A.
Let's see how the diagonal terms and the off-diagonal cross-term pull against each other inside A=[24411] for the quadratic form 2x2+8xy+11y2, and how completing the square connects directly to row elimination.
Completing the Square and Pivots
Text solution
Split the cross-term into the symmetric matrixPut the x2 and y2 coefficients on the main diagonal and split the 8xy coefficient evenly into two 4's across the off-diagonal.
2x2+8xy+11y2=[xy][24411][xy]
Complete the square in xFactor 2 out of the x terms and rewrite x2+4xy using (x+2y)2, which uses up 2(4y2)=8y2 and leaves 3y2 behind.
Row reduce A to find the pivotsSubtract 2 times Row 1 from Row 2 to make the bottom-left entry zero, leaving the square multipliers 2 and 3 on the diagonal as the elimination pivots.
[24411]Row 2←Row 2−2Row 1[2043]
So both pivots 2 and 3 are positive, which means 2x2+8xy+11y2 is a sum of squares with positive multipliers. It is zero only when y=0 and x+2y=0, that is, only at x=y=0, so A is positive definite.
In the family 2x2+2bxy+11y2 with matrix [2bb11], making b larger pulls down the slice along the line x=−2by, where the height is just (11−2b2)y2, even while every entry in the matrix stays positive. The bowl flattens into a trough and bends into a saddle right when the second pivot 11−2b2 passes through zero.
Even when every number in the matrix stays positive, a bigger cross-term can pull the bowl down until the second pivot drops below zero and the surface turns into a saddle.
3
Upper-Left Determinants and the 3-by-3 Test
We want a way to check whether every pivot is positive, and even find the pivots themselves, directly from the original matrix without writing out row reduction step by step.
Now let's connect the pivots to the upper-left determinants and test the 3×3 matrix A=2424117279.
Upper-Left Determinants and the 3-by-3 Test
A=2424117279
D1,D2,D3p1,p2,p3=?=?
Each upper-left determinant Dk is the product of the first k pivots, so dividing consecutive determinants gives the pivots directly:
Eliminate down the first and second columnsSubtract 2 times Row 1 from Row 2 and 1 times Row 1 from Row 3 to clear the first column, then subtract Row 2 from Row 3 to put all three pivots on the diagonal.
Divide consecutive determinants to find the pivotsEach new determinant multiplies in the next pivot, so dividing consecutive determinants gives the pivots from elimination.
p1p2p3=D1=2,=D1D2=26=3,=D2D3=624=4
So all three upper-left determinants (and all three pivots 2,3, and 4) are positive, confirming that A is positive definite.
4
Square Roots and Cholesky Factorization
Every positive number can be written as a square r2 with r=0. For a positive definite matrix A, we want the matrix version of a square root: an upper-triangular matrix R with independent columns such that A=RTR.
Let's see why any product RTR turns xTAx into a squared length, and how to build the Cholesky factorizationA=RTR for A=[24411] right from its row-reduced form.
Square Roots and Cholesky Factorization
aA=[24411]=r2=RTR
R=?
When A=RTR, regrouping the transpose inside the quadratic form turns xTAx into the dot product of the vector Rx with itself:
xTAx=xT(RTR)x=(Rx)T(Rx)=squared length of Rx∥Rx∥2
Text solution
Move the positive pivots inside the squaresBecause the pivots 2 and 3 from 2(x+2y)2+3y2 are positive, take their square roots 2 and 3 and move them inside the squared terms.
Divide each reduced row by the square root of its pivotStarting from the row-reduced matrix [2043], divide Row 1 by 2 and Row 2 by 3 to get the upper-triangular Cholesky factor R.
So A=RTR, and xTAx=∥Rx∥2 is positive for every non-zero vector x.
For any product A=RTR, the surface xTAx=∥Rx∥2 stays an upward-facing bowl whenever the two columns of R lie on different lines through the origin, and only flattens into a trough when one column is a multiple of the other. Because a squared length is never negative, it can never bend downward into a saddle.
Multiplying a matrix by its transpose gives an upward bowl when its columns lie on different lines through the origin, and a flat trough when they lie on the same line, never a saddle.
5
Minimums, SVD, and the Six Equivalent Tests
In applications, positive definite matrices answer two big questions: whether a flat point of a multivariable function is a local minimum, and why ATA lets us factor any matrix into rotations and stretches in the Singular Value Decomposition (SVD).
Minimums, SVD, and the Six Equivalent Tests
H positive definite
H indefinite
Key concepts
A symmetric matrix A is positive definite if it passes any one of these five equivalent tests. Passing one guarantees all five:
Direction and quadratic form (xTAx>0 and angle(x,Ax)<90∘ for all x=0)
Use this right away when you are given a vector x and its output Ax instead of the entries of A. You do not need the matrix: if x=[2−1] and Ax=[14], the dot product xT(Ax)=2(1)+(−1)(4)=−2<0 already tells you the angle is past 90∘, so A cannot be positive definite.
Pivots and completing the square (all pivots pk>0)
Row elimination is completing the square in matrix form. For [24411]→[2043], the pivots 2 and 3 are the multipliers in 2(x+2y)2+3y2. Never judge a matrix by the signs of its entries alone: [2101011] has all positive entries, yet its second pivot is 11−50=−39<0 (a saddle), while a matrix with negative off-diagonal entries is still positive definite whenever its pivots stay positive.
Upper-left determinants (all Dk>0 from the top-left corner)
Best for 2×2 and 3×3 matrices, or when an unknown parameter k sits inside the matrix (like a Hessian matrix H at a critical point). Because D1=p1,D2=p1p2, and D3=p1p2p3, you can find every pivot without row reducing via pk=Dk−1Dk: for D1=2,D2=6,D3=24, the pivots are 2,26=3, and 624=4. Always check every upper-left determinant, because two negative pivots like 2,−3,−4 still multiply to D3=+24.
Square-root / Cholesky factor (A=RTR with independent columns, so xTAx=∥Rx∥2)
Once row elimination gives an upper-triangular matrix with positive pivots, divide each row by pk (turning [2043] into R=[20223]).
Eigenvalues (all λi>0)
Gives the steepness along the main axes of the bowl (and the symmetric square root R=QΛQT), though pivots and determinants are much faster for testing a matrix of numbers. If all λi are ≥0 with at least one 0, the surface is a positive semi-definite trough; if they have opposite signs, it is an indefinite saddle.
6
Practice
1
A symmetric matrix A sends the vector x=[2−1] to Ax=[14]. Without knowing the entries of A, determine whether the angle between x and Ax is less than or greater than 90∘, and whether A can be positive definite.
How does the quadratic form xTAx connect the vectors x and Ax through a dot product?
Take the dot product of x and Ax:
xTAx=xT(Ax)=[2−1][14]=2(1)+(−1)(4)=−2<0
(Watch the negative sign on −1: dropping it gives 2(1)+1(4)=6>0 and the opposite conclusion. Also note that even if one vector gave xTAx>0, a single vector cannot prove A is positive definite, whereas a single vector with xTAx≤0 immediately rules it out.) Because xTAx=−2<0, the angle between x and Ax is greater than 90∘, so A cannot be positive definite.
2
Write the symmetric matrix A for the quadratic form 3x2−12xy+17y2. Complete the square to write the quadratic form as a sum of squares, then use row elimination on A to find its pivots. Is A positive definite despite its negative off-diagonal entries?
How is the −12xy cross-term split across the off-diagonal entries of a symmetric matrix?
Split the −12xy coefficient evenly into two −6's on the off-diagonal (putting the full −12 in both off-diagonal spots is a common mistake, which gives [3−12−1217] and a negative second pivot −31):
A=[3−6−617]
Complete the square on 3x2−12xy+17y2, remembering that (x−2y)2 is multiplied by 3, so it uses up 3(4y2)=12y2 rather than just 4y2 (subtracting only 4y2 leaves 13y2 instead of 5y2):
Now add 2 times Row 1 to Row 2 to eliminate the bottom-left entry:
[3−6−617]Row 2←Row 2+2Row 1[30−65]
This gives pivots 3 and 5 matching the multipliers of the two squares; because both pivots are positive, A is positive definite.
3
Test whether A=3333573716 is positive definite by computing its upper-left determinants D1,D2, and D3, and use those determinants to find all three elimination pivots p1,p2, and p3 without row reducing.
Outline the 1×1,2×2, and 3×3 submatrices from the top-left corner, and use the relationship Dk=p1⋯pk.
Compute the three upper-left determinants from the top-left corner:
D1=3>0
D2=det[3335]=3(5)−3(3)=6>0
D3=3(80−49)−3(48−21)+3(21−15)=93−81+18=30>0
(Be careful to subtract the middle term −3(48−21)=−81 when expanding D3; adding it instead gives D3=192 and p3=32.) Since all three upper-left determinants are positive, A is positive definite. Divide consecutive determinants to get the pivots (do not list the determinants 3,6,30 themselves as the pivots):
p1p2p3=D1=3,=D1D2=36=2,=D2D3=630=5
4
Every entry of the symmetric matrix B=123216367 is positive, its first pivot is D1=1>0, and its full 3×3 determinant is D3=6>0. Is B positive definite? Find D2 and all three pivots of B to explain why.
Check the 2×2 upper-left determinant D2 sitting between D1 and D3, and see what its sign does to p2 and p3.
Compute the upper-left 2×2 determinant D2:
D2=det[1221]=1(1)−2(2)=−3<0
(Be careful not to skip D2 just because the entries of B,D1=1, and D3=6 are all positive.) Dividing consecutive determinants gives the three pivots, remembering to divide D3 by D2=−3, not by D1=1:
Because the two negative pivots p2=−3 and p3=−2 multiply to make D3=+6,B is not positive definite (it is indefinite).
5
Find the upper-triangular Cholesky factor R such that A=RTR for the positive definite matrix A=[46614], and use R to write xTAx as the squared length ∥Rx∥2.
Be sure to divide the off-diagonal 6 in Row 1 by 4=2 as well: leaving 6 alone gives [2065], and dividing by the pivot 4 instead of 4 gives [10231], neither of which gives RTR=A.
Row reduce A to find its upper-triangular form first, then divide each entire row by the square root of its pivot.
Subtract 23 times Row 1 from Row 2 to find the elimination pivots:
[46614]Row 2←Row 2−23Row 1[4065]
The pivots are 4 and 5. Divide all of Row 1 by 4=2 and Row 2 by 5 (be sure to divide the off-diagonal 6 in Row 1 by 4=2 as well: leaving 6 alone gives [2065], and dividing by the pivot 4 instead of 4 gives [10231], neither of which gives RTR=A):
R=[2035]
Using the rows of R, the quadratic form is the squared length:
xTAx=∥Rx∥2=(2x+3y)2+(5y)2
6
At a point where the slope is zero in every direction, a function has the Hessian matrix H=2k0k52022. For what values of k does the Hessian guarantee that the point is a local minimum? What shape does the surface of xTHx have when k=±6 and when ∣k∣>6?
Stopping after D2>0 gives the wider interval −10<k<10, where D3 is already negative for 6<k2<10.
For the Hessian to guarantee a local minimum, all three upper-left determinants D1,D2, and D3 of H must be positive at the same time.
Compute the three upper-left determinants of H:
D1=2>0
D2=det[2kk5]=10−k2
D3=2(10−4)−k(2k−0)+0=12−2k2
We need both D2=10−k2>0 and D3=12−2k2>0, which requires k2<6 (stopping after D2>0 gives the wider interval −10<k<10, where D3 is already negative for 6<k2<10). So the point is a local minimum for −6<k<6. When k=±6,D1>0,D2>0, and D3=0, so the third pivot is zero and the surface is a positive semi-definite trough. When ∣k∣>6, at least one pivot is negative and the surface is an indefinite saddle.