Docs > Benchmarks

Benchmarks

This page reports what Shift measures about its coding loop, the results so far with their dates and caveats, and the scripts/evals.py subcommands that run each set. It is a report rather than a procedure. Every number is a single run on the model and machine named, and the caveats are part of the result.

What is measured

Each task's receipt supplies the same fields: resolved or not, rounds, tool calls, files changed, test runs, tokens, wall time and a failure class. There is no other score; the point is to see which failure class dominates.

SWE-bench Verified

Slice runs use a 40-round cap and a token budget. A run that hits a cap is a recorded failure, not a retry.

Sonnet 5, September 8 and 9, 2026

The first graded batch, on September 8, 2026, ran easy-9 with Sonnet 5: 7 of 9 resolved. sphinx-10435 ended in exploration with no patch and sympy-15875 left an incomplete fix. Five runs were stopped by a token budget that counted cache reads at full weight; it now counts uncached prompt plus completion tokens.

The single pass over the whole slice, on September 9, 2026, graded by the official harness: 16 of 25 resolved, the 7 of 9 above plus 9 of 16 on rest-16 through the agentkernel backend. Two misses were harness defects. Both were fixed and rerun: sympy-23413 then resolved and django-15957 did not, which makes 17 of 25 with the fixes. A third, xarray-4695, has the correct one-line patch applied, but the harness's own pytest segfaults under amd64 emulation on this machine before any test runs, so it can only be graded on a native amd64 host. The other seven misses are model failures: five ended at the round cap still working and two completed with wrong fixes.

The seven misses on a local model

The Sonnet 5 comparison after the quality work could not run: the Anthropic key had reached its monthly usage limit until October 1.

Terminal-Bench and DeepSWE through Harbor

Both benchmarks ship in the Harbor task format, so one adapter, scripts/bench_agent.py, serves both: it installs the runtime into the task container, runs one print-mode turn in autopilot with every command allowed, commits whatever changed, and copies the session next to the receipt. On September 21, 2026, both pipelines completed end to end with the local model, and neither probe task was solved:

• Terminal-Bench html-js-filter, an XSS filter in Python: graded 0 at 40 rounds and again at 12 rounds under a 15-minute bound. The model probed BeautifulSoup's API and never wrote the file the verifier tests.
• DeepSWE anko-default-function-arguments, in Go: the verifier returned -1, could not grade. Harbor's oracle agent, which applies the reference solution, also got -1: the task image's Go toolchain is zero-byte files, as are the toolchains in the other non-Python images pulled on this machine. Until that is understood, the DeepSWE slice is five Python tasks.
• DeepSWE psd-tools-blend-range-api, in Python and validated with the oracle: the first graded Shift result on this benchmark, and it is 0. The 60-minute bound ended the turn at 21 rounds with one file changed and a 45-line patch; all 979 pass-to-pass tests still pass and none of the 45 fail-to-pass tests do.

These tasks are sized for hours on frontier models, so the local model's score is a check that the harness works, not a comparable number. The comparable run is the same slice on Sonnet 5, once the key's limit resets.

Dogfood

On September 10, 2026, the two dogfood tasks ran with ollama/qwen3.8:27b-mlx against clean commit 35eb811. Neither attempt delivered a completed, regression-clean patch. status-last-undo passed its hidden test at 38 rounds in 861.8 seconds, then exhausted its token budget with three failures in the final coding suite: its tests changed a shared fixture and included a wrong expected turn. run-list-source was stopped by the user with no edits, so it is ungraded. Both features were finished locally afterwards, which does not count as a model success.

Since that run, a hidden-test pass also requires the candidate's full make test to pass before a task counts as resolved, and the two tasks are regression cases that refuse to launch a model against a tree where they already pass. New dogfood attempts need an unsolved ticket.

Judge cases and compaction scoring

Two smaller sets sit beside these. evals/judge/cases.jsonl is a fixed set of autopilot decisions with the human's answer; scripts/evals.py judge --cases --model PROVIDER/MODEL replays it through a judge and fails on any disagreement, and --session NAME replays a session's own log instead. scripts/evals.py compaction --judge PROVIDER/MODEL adds a second column to the compaction scorer: whether each durable fact is recoverable from the summary, not only quoted in it. The judge results and their dates are on The judge.

Run the evals

Every set runs through print mode, with --allow-run seeding the allowlist and --receipt writing the per-task record, so nothing parses a transcript. Results land under evals/results/RUN_ID/, existing run names are refused, and a user interruption saves the partial patch without starting grading or another task.

What's next

• The judge for the judge case set and its agreement numbers.
• Tool policy for the autopilot mode the benchmark turns run in.
• Workflows for print mode in your own unattended runs.