warming up your workspace

Same model, different workbench

GitHub’s benchmark view pairs agents running the same model. A sensor puzzle shows why the tools around the model can change the answer.

Two different instrument rigs hold matching luminous spheres.

Give two assistants the same bench

Set out two workbenches. On each put a copy of the same problem and an identical calculator. At the first bench there is a notebook, a labelled parts tray and a way to check a result. At the second there is one long roll of paper and a cupboard whose contents take a minute to find. If the people switch benches and their results change, you have learned something about the furniture. A comparison between coding agents can ask much the same question.

On 17 November, GitHub added a public preview that compares agent harnesses with the underlying model and reasoning effort held constant. The view includes matched configurations of Copilot CLI, Claude Code and Codex CLI. GitHub also added Terminal-Bench-Science 0.1, a benchmark of research tasks authored and reviewed by domain experts. These are two additions to the site. Seeing a science benchmark in the picker does not establish that a particular three-way comparison exists for every task.

Source: GitHub announcement

What the furniture does

A harness is the software around the model. Among other things, it can expose tools for opening files, running programs and observing results. Imagine asking two agents to find the first day a sensor drifted out of calibration. Both have access to the same reasoning model. One receives a clear error when it opens the wrong file. The other gets a long stream of unrelated output that obscures the error. The next decision each makes will be based on a different observation.

You can design a small classroom comparison around that difference. Prepare a folder with dated readings, a calibration sheet and one worked example. Before running either assistant, write down what a satisfactory answer must contain. A date alone is a poor test because a lucky guess can match it. Ask for the supporting readings and the unit conversion too. Keep a copy of each starting folder so the second assistant never inherits the first one's notes.

A one point lead can hide a lot

Suppose one setup finishes thirteen of twenty invented tasks and another finishes fourteen. Those totals fit a world in which both solved thirteen of the same tasks and the second solved one extra. They also fit a world in which they shared only seven successes. In that second world each setup routinely solves problems the other misses. A teacher choosing help for students with a particular recurring problem would want to see those pairs.

Look at cost with the same care. If a setup spends a little on every failed attempt and a lot on every success, simply averaging successful runs hides part of the bill. Record the total across all attempts, the number of attempts and the number that passed. From those three numbers you can calculate both average cost per attempt and cost per success. Each answers a different budgeting question.

Keep one failure on the page

A table of scores is a good starting point for choosing which runs to inspect. For our imaginary sensor task, the most instructive artifact might be the moment an assistant confused millivolts with volts. Save the reading it saw and the action it chose next. A student can use that little piece of evidence to improve a tool description, design a clearer test or decide that this job still needs a human checking the units.

Original source: GitHub