warming up your workspace

Why models overfit, and what regularization actually does about it

Here is the paradox at the center of machine learning: a model that fits your training data perfectly is often a bad model. Zero error on the data you have can mean high error on the data you do not, because the model stopped learning the pattern and started memorizing the noise. This is overfitting, and it is the single failure mode that every technique in the field, from dropout to weight decay to early stopping, is ultimately fighting. You can watch it happen, and watch it get fixed, with nothing more than a polynomial and twelve noisy points.

The one idea

Give a model more flexibility than the underlying pattern needs, and it will use the extra flexibility to chase the noise. Fit a straight line to points that came from a curve and you underfit, too rigid to follow the real shape. Fit a wiggly degree-11 polynomial to twelve points and you overfit, flexible enough to pass through every single point exactly, including the random measurement errors, which produces wild swings between the points where no data pins it down. The right model has just enough capacity to capture the signal and not enough to memorize the noise. Regularization is how you take a too-flexible model and discourage it from using all that flexibility, without having to guess the perfect size in advance.

Build overfitting

Sample a gentle sine wave with noise, then fit polynomials of increasing degree. Measure error on the training points and on the true underlying curve.

f = lambda x: np.sin(1.5*x)
x_train = np.linspace(-3, 3, 12)
y_train = f(x_train) + rng.normal(0, 0.25, 12)     # noisy observations

def fit(x, y, deg, lam=0.0):
    X = np.vander(x, deg+1, increasing=True)
    I = np.eye(deg+1); I[0,0] = 0                   # ridge penalty (skip the bias)
    return np.linalg.solve(X.T@X + lam*I, X.T@y)

Fit degree 1, 3, and 11, and compare training error to test error:

 degree  train RMSE  test RMSE
      1       0.752      0.704      <- underfit: too stiff for a curve
      3       0.364      0.301      <- just right: lowest test error
     11       0.000      0.437      <- overfit: perfect on train, worse on test

The degree-11 polynomial nails the training data, error exactly zero, it passes through all twelve points. And it is the worst of the three on data it did not see. That gap, zero training error but high test error, is the signature of overfitting. The degree-3 model, which cannot fit the noise, generalizes best. The lesson every practitioner learns: training error is not the score that matters.

Build the fix

You will not always know the right degree. Regularization lets you start with a flexible model and penalize complexity directly. Ridge regression (L2) adds a penalty proportional to the size of the weights, so the fit has to justify every bit of wiggle with real signal. The lam term does exactly that. Turn it up on the overfit degree-11 model:

  lambda  train RMSE  test RMSE    max|w|
     0.0       0.000      0.437       1.2      <- unregularized, overfit
     0.1       0.071      0.352       0.9      <- test error improved
     1.0       0.156      0.672       0.5      <- too much: now underfitting

Three details that matter:

  • Regularization trades a little training error for a lot less overfitting. At lambda = 0.1, training error rose from 0 to 0.07, but test error dropped from 0.437 to 0.352. You deliberately fit the training data worse to fit reality better.
  • The penalty shrinks the weights, max|w| falls as lambda rises. Smaller weights mean a smoother function that cannot swing wildly between points. That is the whole mechanism: big weights make sharp wiggles, so penalizing weight size penalizes wiggling.
  • There is a sweet spot, and past it you underfit again. At lambda = 1.0 the penalty is so strong the model is too smooth and test error climbs back up. Choosing lambda is itself done by testing on held-out data, which is what a validation set is for.

Where this shows up

Every serious model is regularized, and the degree-11 story scales straight up to networks with billions of parameters, which have more than enough capacity to memorize their training set outright. Weight decay is exactly the L2 penalty you just used. Dropout randomly disables neurons so the network cannot rely on memorized combinations. Early stopping halts training when validation error starts rising, before memorization sets in. Data augmentation and larger datasets fight the same fight from the data side. They are different implementations of one idea: prefer the simpler explanation, because it is the one that generalizes.

If you want to build these, weight decay, dropout, the train-validation-test discipline, into a real network and watch them work, that is the path the ai track on IWTLP builds from linear regression up.