p-values, explained by simulation not tables

The p-value is the most used and least understood number in science. People memorize "below 0.05 is significant" and reach for a t-table without knowing what the table is. But you can compute a p-value with no formula at all, just by shuffling your data a few thousand times. Do that once and the definition stops being abstract.

The one idea

You ran an experiment and saw an effect: the treatment group did better than the control by some amount. The honest question is: could this difference have happened by chance alone, if the treatment did nothing? The p-value answers exactly that: it is the probability of seeing a difference at least this big if there were no real effect. To measure it, we build the world where there is no effect, and see how often that world produces a result like ours.

Build the world of "no effect"

Here are two small groups. The treatment mean looks higher, but is it real or luck?

set.seed(42)
control   <- c(38, 42, 40, 37, 41, 39)
treatment <- c(45, 43, 48, 44, 46, 47)

observed <- mean(treatment) - mean(control)
observed
# [1] 6

A 6-point gap (the treatment mean is 45.5, the control mean 39.5). Now the key move. If the treatment truly did nothing, then the labels "control" and "treatment" are meaningless: any of these twelve numbers could have landed in either group. So we pool all twelve, reshuffle the labels, and recompute the difference, over and over. That gives us the distribution of differences we would see by chance alone.

combined <- c(control, treatment)
n <- length(control)

null_diffs <- replicate(10000, {
  shuffled <- sample(combined)                 # random relabeling
  mean(shuffled[(n + 1):12]) - mean(shuffled[1:n])
})

null_diffs is 10,000 differences from a world where the groups are interchangeable. It is centered on zero, because on average a random split has no difference. The spread tells us how big a gap chance can produce.

Read the p-value off the distribution

The p-value is simply: of those 10,000 chance differences, what fraction were as extreme as the 6 we actually observed?

p_value <- mean(abs(null_diffs) >= abs(observed))
p_value
# [1] 0.0019

About 0.0019, roughly 1 in 500. In ten thousand shuffles, only around nineteen produced a gap as large as ours by chance. So a difference this big is very unlikely if the treatment did nothing, which is what "statistically significant" means. We never opened a t-table; we built the null distribution ourselves and counted.

Why this is worth doing at least once

The simulation makes three things concrete that the formula hides. First, the null hypothesis is a real thing you can build, the world of "no effect" is just your data with the labels scrambled. Second, the p-value is a probability about the data, not about the truth: it does not say "there is a 0.2% chance the treatment is useless," it says "if the treatment were useless, data like this would be rare." Third, a small p-value is not a big effect; with enough data a trivial difference becomes significant, so significance and importance are different questions.

Real analyses often still use the formula-based t-test, because for large samples it agrees with the simulation and runs instantly. But the permutation test needs almost no assumptions (no bell curve required), and it is the version you can see. Once you have watched a p-value fall out of ten thousand shuffles, the number stops being a ritual threshold and becomes what it always was: a measurement of how surprised you should be. That build-it-to-understand-it approach to statistics is the whole point of the statistics in R track.