Fitting Models to Data: Features, Polynomials, and Regularization
When we have noisy data points, we want to find a formula that fits the overall pattern so we can predict y at new values of x.
1
Fitting a Straight Line with Least Squares
Suppose we want to fit a straight line y=w0+w1x to the three data points (1,1),(2,2), and (3,5) so we can predict y at new inputs like x=2.5 or x=4.
Plugging our three points into y=w0+w1x gives three equations for only two weights, w0 and w1. Here is how projecting onto the column space in 3D finds the closest line:
Fitting a Straight Line with Least Squares
Text solution
1. Stack the data points into feature columnsPlugging (1,1),(2,2), and (3,5) into w0(1)+w1x≈y gives one vector equation in 3D space.
w0Column 1111+w1Column 2123≈y125
2. Dot each column with the system to build ATAw=ATyFor the error vector to be perpendicular to the plane, its dot product with each column must be zero, so we dot Column 1 and Column 2 with every vector in the equation.
3. Double the first row and subtract to solve for the weightsSubtracting twice the first equation from the second leaves only w1, which we plug back in to find w0.
So the best-fit line is y=−34+2x, which comes as close as possible to all three points even though no straight line passes through all three.
Try moving the third data point to see how the 2D line and the 3D column-space projection change together:
Three points sit on one line in 2D when the data vector lies flat in the 3D plane, and when a point moves off the line, the best-fit line comes from the vector's perpendicular shadow on the plane.
2
Fitting Polynomial Curves
When our data points bend in a U-shape, a straight line cannot follow the pattern. We want to fit a parabola y=w0+w1x+w2x2 to the five noisy data points (−2,2),(−1,3),(0,1),(1,1), and (2,8).
Even though x2 makes the graph curve, we can use the exact same column projection because the weights w0,w1,w2 still just multiply columns of numbers that we calculate from x ahead of time.
Fitting Polynomial Curves
Text solution
1. Square the x values to build the third feature columnEvaluating 1,x, and x2 at the five inputs gives three feature columns in 5D space that we combine to get as close as possible to the target vector y.
2. Assemble the 3×3 dot-product grid ATA and target dots ATyDotting each of the three feature columns with every column and with y turns the five equations into a 3×3 system.
3. Solve the middle row directly and eliminate w0 between the outer rowsThe second row gives w1 right away, and subtracting twice the first row from the third row finds w2.
So the best-fit parabola is y=1+x+x2. Notice a handy shortcut: because the inputs x=−2,−1,0,1,2 are symmetric around 0, the odd column x is perpendicular to both even columns 1 and x2, putting zeros in the grid that let you solve for w1 on its own.
3
Custom Features and Multiple Inputs
Because each column is just a list of numbers evaluated at our data points, we can build columns out of waves like sin(x) and cos(x), exponentials like 2x, or separate input variables altogether.
Suppose we want to predict the price y (in $100k units) of four houses from two different inputs, house size x1 (in thousands of square feet) and number of bedrooms x2, by fitting the model y=w1x1+w2x2 to the four houses (1,1,4),(1,2,3),(2,1,4), and (2,2,7).
Custom Features and Multiple Inputs
Text solution
1. Place each input variable into its own columnThe four house sizes form Column 1, the bedroom counts form Column 2, and the prices form the target vector y in 4D space.
2. Dot the two input columns with each other and with yTaking the dot products in 4D space turns the four house equations into a 2×2 system of normal equations.
3. Subtract and add the two symmetric rowsSubtracting the second equation from the first gives the difference of the weights, and adding the two equations gives their sum.
So the fitted model is y=2x1+x2, which tells us that each thousand square feet adds $200k to the house price and each bedroom adds $100k.
4
Overfitting and Nearly Parallel Columns
If adding feature columns makes our model more flexible, what goes wrong if we put in a long list of features like 1,x,x2,x3,…,x10 all at once?
Overfitting is when a model has so many features that it passes right through the random noise in our data points while swinging wildly away from them, making its predictions at new points useless.
Overfitting and Nearly Parallel Columns
When we fit 5 independent polynomial columns (1,x,x2,x3,x4) to 5 noisy data points, the columns span all of 5D space, so y sits inside the column space with zero error. Because columns like x2,x3,x4 grow in nearly the same way for positive inputs like x=0.5,0.6,0.7,0.8, their column vectors point in almost the same direction. Reaching sideways to hit the noisy target y forces the model to use huge weights with opposite signs along those nearly parallel columns:
y=2cancel at the data points,explode away from them−150x+1200x2−1800x3+752x4
If we add a 6th feature column for only 5 data points, the columns become linearly dependent, so ATA is no longer invertible and the normal equations have infinitely many solutions.
Try nudging the target vector y across the narrow gap between two nearly parallel columns to see why the weights blow up and flip signs:
When two feature columns point in nearly the same direction, a tiny sideways shift in the data vector forces huge opposite steps along the columns and makes the fitted curve swing wildly past the end points.
5
Regularization and the Complete Recipe
We want to keep flexible features like x2,x3,x4 to capture curves, without letting nearly parallel columns end up with huge opposing weights like +1000 and −999.
Regularization (specifically Ridge or L2 regularization) keeps the weights small by adding extra penalty rows swi≈0 to the bottom of the system, pulling each weight toward zero with strength s.
Regularization and the Complete Recipe
Appending the penalty rows extends each column into its own new dimension while appending zeros to y:
w1123s0+w21490s≈12500
When we take dot products, the penalty rows contribute s(0)+0(s)=0 between different columns and 0 with y, and add λ=s2 only when a column is dotted with itself on the main diagonal:
(ATA+λI[1001])w=ATy,λ=s2
Here are the steps for the extreme case where two data columns are completely identical: w1[11]+w2[11]≈[22].
Text solution
1. Check the unregularized system (λ=0)Dotting each identical column with itself gives 2 and dotting with y gives 4, producing a non-invertible grid where both equations say the exact same thing.
[2222][w1w2]⟹w1+w2=[44]=2(infinitely many solutions,including 1000 and −998)
2. Add λ=1 down the main diagonal of ATARegularizing with λ=1 adds 1 to each diagonal entry while leaving the off-diagonal 2s and the right-hand side 4s unchanged.
Instead of canceling each other out at +1000 and −998, the two identical features share the weight equally at w1=0.8 and w2=0.8.
Try increasing the penalty s to tilt the two columns apart into their own dimensions and calm down the fitted curve:
Tilting each column into its own perpendicular dimension spreads nearly parallel columns apart, shrinking huge opposing weights into a smooth fit.
Key concepts
Every data-fitting problem follows one geometric recipe: choose your features to build the columns of A, take column dot products to project y at 90∘, and add λ to the diagonal if flexible features make the columns nearly parallel.
Choosing feature columns
Create one column of numbers per weight wi by plugging your data inputs into each term of your model. Use 111 for a constant intercept w0, powers like x2 when the data bends (turning x=−102 into x2=104), functions like 2x or sin(x) for exponential growth or waves, or separate columns for multiple inputs x1,x2. If a straight line or simple parabola already captures the trend with well-separated columns, stop there; don't add higher powers like x4 just to force the error at the noisy points to zero.
Least squares (ATAw=ATy)
Build the square grid ATA by dotting every feature column with every feature column, and build the right-hand side ATy by dotting every feature column with the target vector y.
Regularization ((ATA+λI)w=ATy)
When you have many flexible features or nearly parallel columns (which show up as huge weights of opposite signs like +1200x2−1800x3, or a non-invertible ATA when you have more features than points), add penalty rows swi≈0. In the normal equations, this simply adds λ=s2 to the main diagonal of ATA:
Fit a straight line y=w0+w1x to the three points (0,2),(1,3), and (2,10) by setting up the feature columns and solving ATAw=ATy, then use your line to predict y at x=2.5.
Be careful to set Col 1⋅Col 1=1+1+1=3 rather than 1, which is a common mistake that leads to w1=5.5,w0=−1.5 and a wrong prediction of 12.25.
What column vector multiplies the constant weight w0 across all three data points before you take the dot products?
Stacking the three points into columns gives Column 1 =111, Column 2 =012, and target y=2310.
Taking the dot products (be careful to set Col 1⋅Col 1=1+1+1=3 rather than 1, which is a common mistake that leads to w1=5.5,w0=−1.5 and a wrong prediction of 12.25):
[3335][w0w1]=[1523]
Subtracting the first equation from the second gives 2w1=8, so w1=4 and w0=1. The best-fit line is y=1+4x, which predicts y=1+4(2.5)=11.
2
Fit a parabola y=w0+w1x+w2x2 to the five data points (−2,12),(−1,0),(0,7),(1,−2), and (2,8) by constructing the three feature columns and solving the 3×3 normal equations.
Watch out when computing Col 3⋅Col 3: you must dot Column 3 with itself to get 42+12+02+12+42=34, not just sum the entries of Column 3 to get 10, which would give w2=−2.8 and w0=10.6.
Square each entry of the x column to form Column 3, and notice which column dot products cancel to 0 by symmetry.
Taking the dot products to build ATAw=ATy (watch out when computing Col 3⋅Col 3: you must dot Column 3 with itself to get 42+12+02+12+42=34, not just sum the entries of Column 3 to get 10, which would give w2=−2.8 and w0=10.6):
5010010010034w0w1w2=25−1078
The middle row gives 10w1=−10, so w1=−1. Doubling the first row gives 10w0+20w2=50, and subtracting that from 10w0+34w2=78 gives 14w2=28, so w2=2 and w0=1. The best-fit parabola is y=1−x+2x2.
3
Fit an exponential model y=w0+w12x to the three points (0,5),(1,2), and (2,10) by building the feature columns for 1 and 2x and solving ATAw=ATy.
Be sure to evaluate 20=1 in the first row of Column 2; writing 20=0 makes Column 2 024 and leads to w0=619,w1=45.
Evaluate 2x at x=0,1,2 first so Column 2 is just a vector of three plain numbers.
At x=0,1,2, the feature columns are Column 1 =111 and Column 2 =124 with target y=5210 (be sure to evaluate 20=1 in the first row of Column 2; writing 20=0 makes Column 2 024 and leads to w0=619,w1=45).
Taking the dot products gives:
[37721][w0w1]=[1749]
Tripling the first row gives 9w0+21w1=51; subtracting the second row gives 2w0=2, so w0=1 and w1=2, giving y=1+2(2x).
4
Four houses have size x1 (in thousands of square feet), bedroom count x2, and price y (in $100k units): (1,0,5),(0,1,2),(1,1,4), and (2,1,9). Fit the model y=w1x1+w2x2 and state what each weight means.
Watch the 0 in the first entry of Column 2 when computing Col 2⋅y=0(5)+1(2)+1(4)+1(9)=15; including the 5 gives 20 and leads to w1=37,w2=313.
Build Column 1 directly from the four values of x1 and Column 2 from the four values of x2.
Using Column 1 =1012, Column 2 =0111, and y=5249, the normal equations are (watch the 0 in the first entry of Column 2 when computing Col 2⋅y=0(5)+1(2)+1(4)+1(9)=15; including the 5 gives 20 and leads to w1=37,w2=313):
[6333][w1w2]=[2715]
Subtracting the second row from the first gives 3w1=12, so w1=4 and w2=1. The model is y=4x1+x2: each thousand square feet adds $400k and each bedroom adds $100k.
5
Suppose you have 4 noisy data points. Explain what happens in the 4D column space and to the fitting error when you fit a polynomial with 4 independent columns (1,x,x2,x3), why nearly parallel columns among them cause wild swings away from the points, and what happens to ATA if you add a 5th column x4.
How many independent vectors does it take to span all of 4D space, and what kind of steps must you take along two nearly parallel vectors to reach a target slightly off to the side?
Four independent columns span all of 4D space, so the target vector y sits inside the column space and the error at the 4 data points drops to zero (don't confuse zero error at the 4 noisy data points with a good fit overall; hitting the noise exactly is what causes overfitting). When feature columns point in nearly the same direction, reaching sideways to hit the noisy target y requires huge weights of opposite signs that cancel at the data points but grow huge away from them. Adding a 5th column in 4D space makes the columns linearly dependent, so ATA is no longer invertible and the normal equations have infinitely many solutions.
6
Consider the system w1[12]+w2[12]≈[714] with two identical feature columns. Write the extended column system with regularization penalty rows for s=2, find λ, and compare the solutions of the unregularized system (λ=0) and the regularized system (ATA+λI)w=ATy.
Be careful not to add s=2 instead of λ=s2=4, which gives [7557] and w1=w2=1235, and don't add 4 to the off-diagonal entries.
Append one penalty row for each weight, and check what dotting an extended column with itself adds to the main diagonal.
Appending the penalty rows for s=2 gives:
w11220+w21202≈71400,λ=s2=4
When λ=0, the normal equations are:
[5555][w1w2]=[3535]
Here ATA is not invertible, and any (w1,w2) with w1+w2=7 is a solution.
When λ=4, we add λ=4 to the main diagonal only (be careful not to add s=2 instead of λ=s2=4, which gives [7557] and w1=w2=1235, and don't add 4 to the off-diagonal entries):
[9559][w1w2]=[3535]
Adding and subtracting the equations gives 14(w1+w2)=70 and w1=w2, which has the single solution w1=2.5 and w2=2.5.