warming up your workspace

How a mixture of experts routes tokens, and why trillion-parameter models are cheap to run

The frontier models of 2026 keep announcing parameter counts that sound impossible to run: hundreds of billions, then trillions. And yet inference costs went down, not up. Kimi, DeepSeek, and the open Mixtral line all pulled the same trick. The model is enormous, but only a small slice of it runs for any given token. The mechanism is called a mixture of experts, and the part that makes it work is a tiny router. Build the router and the whole economic story falls out.

The one idea

A dense network runs every parameter on every token. A mixture of experts splits the big feed-forward layer into many smaller experts, and adds a router that, for each token, picks just a few experts to actually run. The other experts sit idle for that token. So the model can hold a huge number of parameters, the sum of all experts, while the compute per token stays fixed at "a couple of experts plus the router".

Total capacity is large. Active compute is small. That gap is the entire point.

Build the router and the layer

We will make eight experts, each a small linear transform, and a router that scores how well each expert fits a token. For each token we keep the top two experts, renormalize their weights with a softmax, and blend only those two outputs.

import numpy as np
rng = np.random.default_rng(0)

D, E, K = 16, 8, 2          # dimension, number of experts, experts used per token
experts = [rng.standard_normal((D, D)) * 0.1 for _ in range(E)]
router  = rng.standard_normal((D, E)) * 0.1

def softmax(x):
    x = x - x.max(-1, keepdims=True)
    e = np.exp(x); return e / e.sum(-1, keepdims=True)

def moe_layer(tokens):
    scores = tokens @ router                       # (T, E): fit of each expert
    topk = np.argsort(-scores, axis=1)[:, :K]      # keep the best K per token
    out = np.zeros_like(tokens)
    for t in range(tokens.shape[0]):
        ids = topk[t]
        w = softmax(scores[t, ids])                # weight only the chosen experts
        for j, e in enumerate(ids):
            out[t] += w[j] * (tokens[t] @ experts[e])   # only K experts run
    return out, topk

Three details that matter:

  • The router is just one small matrix. It costs almost nothing compared to an expert, yet it decides everything about where compute goes. Training it well is the hard part: a lazy router that always picks the same expert wastes all the others.
  • top-k is where the savings come from. With eight experts and K=2, each token runs a quarter of the expert parameters. Scale that to 256 experts choosing 8 and you touch about 3 percent of the model per token.
  • The softmax over the chosen experts is a weighted blend, not a hard switch. The token gets a mix of its two best specialists, which is where the name comes from.

Proof: watch the routing and count the compute

tokens = rng.standard_normal((5, D))
out, topk = moe_layer(tokens)

total  = sum(x.size for x in experts) + router.size
active = K * D * D + D * E
print("router sent each token to experts:", topk.tolist())
print(f"total expert params: {total:,}")
print(f"touched per token:   ~{active:,}  ({100*active/total:.0f}% of the model)")

Running it prints:

router sent each token to experts: [[4, 2], [3, 7], [7, 5], [2, 0], [5, 6]]
total expert params: 2,176
touched per token:   ~640  (29% of the model)

Every token went to a different pair of experts, chosen by the router, and each one used under a third of the parameters. In a real model with hundreds of experts that fraction drops into the low single digits. The model is huge in memory and small in compute, and now you can see exactly why: most of it did not run.

Where this shows up

This is the architecture behind most of the largest open and closed models shipping in 2026. It is also why serving them is a distributed-systems problem as much as a math problem: the experts get spread across GPUs, and the router's choices decide which machines light up for each token, so load balancing the router becomes a real engineering constraint. The trade is memory for compute, you pay to store every expert, but you only pay to run a few.

If you want to build the transformer these experts plug into, attention, the feed-forward blocks, and the training loop, that is the path the ai track on IWTLP walks from scratch.

Sources