warming up your workspace

PCA from scratch, finding the directions your data actually varies

Principal component analysis shows up everywhere: compressing hundred-dimensional data to two for a plot, removing redundant features before training a model, finding the "main axes" of variation in a dataset. It gets described with intimidating words, eigenvectors, orthogonal projections, explained variance, but the actual idea is something you already understand intuitively. If a cloud of data points is stretched out like a cigar, PCA finds the long axis of the cigar. That is it. Everything else is making that precise. Build it in R and watch it recover the axis of a dataset you constructed, then confirm it matches R's built-in tool exactly.

The one idea

Data with many features often really varies along just a few directions. Height and weight are two columns, but people mostly differ along one "bigger versus smaller" axis, the two are correlated. PCA finds those axes of maximum variation, in order. The first principal component is the single direction along which the data spreads out most. The second is the direction of most remaining spread, at right angles to the first. And so on. If most of the variation lives in the first couple of components, you can throw the rest away and keep nearly all the information in far fewer dimensions.

The precise version: the directions of maximum variance are the eigenvectors of the data's covariance matrix, and each eigenvalue tells you how much variance its direction captures.

Build it

Three steps: center the data, compute its covariance matrix, and eigen-decompose. Here is correlated 2D data (mostly along a diagonal) run through PCA by hand.

Xc <- scale(X, center = TRUE, scale = FALSE)   # subtract the mean of each column
C  <- cov(Xc)                                  # covariance matrix
e  <- eigen(C)                                 # eigenvectors = principal directions
vals <- e$values; vecs <- e$vectors            # eigenvalues = variance per direction

Three details that matter:

  • Centering is not optional. PCA measures variance around the mean, so you subtract each feature's mean first. Skip it and the first component points at where the data sits rather than how it varies, a completely different and usually useless answer.
  • The covariance matrix is where the correlations live. Its off-diagonal entries say how features move together; the eigen-decomposition finds new axes where those correlations vanish, so the components are uncorrelated by construction. That decorrelation is half of why PCA is useful.
  • Whether you scale features to unit variance first is a real modeling choice. If your columns are in different units, income in dollars, age in years, the large-numbered one dominates unless you scale. Here the features share a scale, so we center only.

Project down

The first eigenvector is your new primary axis. Project the data onto it to get a one-dimensional summary that keeps as much variation as any single direction can.

ord   <- order(vals, decreasing = TRUE)         # biggest variance first
scores <- Xc %*% vecs[, ord[1]]                 # coordinates along PC1

Proof: it finds the axis, and matches the standard tool

variance captured by each component:
  PC1: 97.9 %
  PC2: 2.1 %
first principal direction: -0.892 -0.452
variance of PC1 scores: 4.803 vs total: 4.905

The data was built to run mostly along a diagonal, and PCA found it: the first component captures 97.9 percent of all the variation, so collapsing this 2D data to that single axis loses barely 2 percent of the information. The variance of the projected scores, 4.80, is almost the entire total variance of 4.90, confirming the projection kept nearly everything. Cross-check against R's built-in prcomp:

prcomp PC1 direction: 0.892 0.452
prcomp variance %: 97.9 2.1

Identical direction and identical variance split. (The sign is flipped, which is expected and meaningless: a principal axis and its opposite describe the same line, so PCA never promises a particular sign.) Your covariance-and-eigenvector version is prcomp, just with the steps visible.

Where this shows up

PCA is the default first move on any high-dimensional dataset. It powers the 2D scatter plots that let you eyeball structure in data with dozens of features; it denoises by dropping the low-variance components that are usually just noise; it speeds up training by shrinking the feature count before a model ever sees the data. The same eigen-decomposition is the mathematical core of the singular value decomposition behind recommender systems and latent semantic analysis. And knowing it is a variance-maximizing projection is exactly what tells you its limits: PCA only sees linear structure, which is why nonlinear methods like t-SNE and UMAP exist for the cases it misses.

If you want to build PCA, the SVD it generalizes to, and the nonlinear methods that pick up where it stops, that is the path the machine-learning track on IWTLP takes in R, from the covariance matrix up.