GitHub's 25 ms problem: deciding what really needs to wait

A queue can hide in a healthy system

Imagine a small application that accepts changes to two independent customer records. It has several workers, but every update must acquire one shared lock. A slow update to the first record prevents an update to the second from starting. Adding another worker gives the waiting work another place to wait. It does not change the rule that only one update may proceed.

This is why a busy system can look puzzling when viewed only through resource charts. Some capacity may be available while response times keep growing. Before asking for a larger machine, it is worth asking where the program insists on taking turns. The answer can be a design choice rather than a shortage of hardware.

The news from GitHub

GitHub's 21 October engineering post describes a roughly 25 ms cross-region round trip constraining the migration of its busiest repositories. Its current repository-wide coordination can make unrelated writes wait together. The proposed move to per-reference quorums narrows that coordination so independent references can advance concurrently. GitHub presents this as part of an ongoing architecture transition, not a claim that every step of its final design has already shipped.

Source: GitHub Engineering

Build a model small enough to inspect

Here is an IWTLP thought experiment, not a benchmark of GitHub. Suppose three updates arrive together: two target main and one targets docs. Give each update an invented, fixed service time of 25 ms. With one shared queue ordered main, main, docs, the docs update finishes at 75 ms. With independent queues and sufficient workers, docs can finish at 25 ms while the two main updates remain ordered and finish at 25 and 50 ms.

The arithmetic is deliberately simple. There are no network retries, storage costs, multi-reference transactions or scheduling surprises in this model. Its job is to isolate one decision: whether work that does not share state should wait behind work that does. The improvement comes from changing the permitted overlap, not from pretending the individual operation became faster.

You can turn this into a short simulation. Represent each job with a resource name and a duration. Maintain the next available time for either one global resource or each named resource. For every job, record its start and finish. Print the resulting schedule. Even a few lines of sequential simulation can make a concurrency rule easier to reason about than a large diagram.

The difficult part is choosing the boundary

Now change the exercise. Add a job that must update both resources consistently. Independent queues are no longer the complete answer, because that job introduces a relationship the first model deliberately excluded. You must decide how to coordinate it and what other jobs may observe while it runs. This is the point where an attractive optimization becomes an engineering problem.

The same question appears in ordinary applications. Two actions might look unrelated in the interface while sharing a balance, an inventory limit or a unique identifier. Narrowing a lock without understanding that shared condition can make the system faster at producing incorrect results. Start by naming the condition that must remain true. Then identify the smallest set of operations that can violate it together.

A practical review can ask three questions. What state does this operation change? Which other operations depend on the order of that change? What evidence shows that the remaining work is independent? Write the answers next to the synchronization code. That explanation gives future maintainers a reason to preserve the boundary, and a warning when a new feature changes the assumptions.

Our reading of GitHub's update is a reminder to investigate waiting as carefully as computation. A small delay can become consequential when a design makes every task pay it in sequence. You do not need a large infrastructure team to explore that idea. Start with the queue in your own program, make its ordering rule visible, and ask which parts of that rule are actually required.