Docs > Benchmarks

The judge

This page describes the judge model: the separate model call that decides whether an autopilot action goes ahead, and the other decisions that use it. It covers choosing between the session model and the typed judge, the confidence setting, shadow mode and its report, what evidence leaves the machine, what happens when the judge cannot answer, and the agreement measured so far.

What the judge decides

In autopilot, an action is resolved in three steps. Fixed rules run first: reads, allowlisted runs and allowlisted MCP tools proceed, and the destructive cases (rm -rf outside the project, force pushes, git reset --hard, git clean -f, git stash drop, curl … | sh and writes into Shift's own state) are refused without a model call. Everything else goes to the judge. It answers allow or block with a rule name and a reason; a block reaches the model as blocked by autopilot [rule]: reason so it tries another way. Three consecutive blocks, or twenty in a turn, pause the judge and manual prompting takes over for the rest of the turn.

Three other decisions use a judge:

A workflow step's (judge "CLAIM") check asks whether the step's answer establishes the claim. See Workflows.
With the typed judge on, tool_search sends its word-ranked candidates, names and descriptions only, to the judge and keeps the tools it is at least half sure do what the query asks. When none clears the bar, the word order stands.
With the typed judge on, each user turn asks once which offered skill, if any, is the procedure for the request. A confident pick adds one <skill_relevance> line to the prompt; the trace records it as skill-hint.

Everything that writes text, the answer, compaction summaries, notes and the receipt, stays with the session model.

Choose the judge model

The judge-model setting names the judge as PROVIDER/MODEL. Unset, the session's own provider and model judge: one request that asks for strict JSON, with a 10-second timeout, where a parse failure counts as a block. On the default local model this takes about 1.4 seconds per action: the default session's judge.jsonl records 1,397 and 1,461 ms on qwen3.8:27b-mlx.

The alternative is a typed judge. judge-model typesafe/jev-1.13.0 selects Jev, TypeSafe's model for decisions rather than text. It takes the same evidence as named fields plus typed questions, and returns typed answers with calibrated probabilities: one allow-or-block choice, and one yes-or-no per block category (escalation, outside the project or its remotes, secrets leaving the machine, discarding uncommitted work, changing the agent's own policy, crossing a stated boundary). Nothing is parsed. The most probable category becomes the rule, and its fixed description is the reason. Jev reads its criteria literally and treats what it is given as data, so the boundary cases are written into the criteria and the fixed rules stay beneath it as a floor.

To use it:

Put the key in TYPESAFE_API_KEY, or in ~/.config/typesafe/api-key with mode 600. There is no settings-file key.
Set judge-model typesafe/jev-1.13.0. The version is pinned because jev-latest moves.

Jev is never selected on its own. A key on disk does not turn it on.

Set the confidence to ask at

The session-model judge is binary. The typed judge's choice carries a confidence, and autopilot uses it as a third outcome. judge-ask-below (default 0.5) is the confidence under which an allow becomes the ordinary approval prompt instead of going through; #f never asks. A block stays a block at any confidence, because the expensive mistake is a false allow. In print mode there is nobody to ask, so an uncertain allow blocks with the rule judge-uncertain and the receipt says why. The receipt counts these prompts as judge_asked.

Shadow mode and the report

The judge setting is shadow by default. In autopilot the judge decides. In manual mode it also runs beside every approval prompt: its verdict shows in the preview, you answer as usual, and both answers go to the session's judge.jsonl. Each line names the judge model and, for the typed judge, its confidence.

Command	Effect
`/judge`	Shows the current judge, and why the typed judge is off if it turned itself off.
`/judge report`	Prints agreement between the judge and you, with the cases that disagreed.
`/judge on`	Keeps the judge for autopilot without shadowing manual mode.
`/judge off`	Autopilot asks for anything the rules do not resolve, so nothing runs everything unattended.

The receipt records judged, blocked and judge_ms. Each judge.jsonl record also stores what the judge saw, so scripts/evals.py judge --session NAME --model PROVIDER/MODEL can replay a session's decisions through another judge and report agreement, false blocks and false allows. --cases replays the fixed set in evals/judge/cases.jsonl and fails on any disagreement; that replay is the gate for making the typed judge the autopilot judge in any session.

What leaves the machine

The judge sees the user's last four messages, each clipped to twelve lines; the proposed action with its arguments and a 40-line preview; the project root, its git remotes, whether the tree has uncommitted work, the mode and the allowlist. It never sees tool output, file contents or skill bodies. On a Claude or OpenAI session this evidence already leaves the machine with every turn. On an Ollama session it does not, and only writing judge-model typesafe/… yourself changes that. The tool_search and skill decisions follow the same setting, so one setting answers whether anything from a session goes to TypeSafe. The receipt names the judge model, and the judge.decide trace span records the token counts the service reports.

When the typed judge cannot answer

The session model judges that action instead, which is the judge the session had before the setting, and the log line records the fallback class:

Class	Cause	Effect
`no-key`, `auth` (401), `credits` (402, 403), `request` (422)	Will not heal by retrying.	Jev is off for the rest of the session after one notice; `/judge` shows why.
`transient` (429 after its retry, 529, 5xx), `network`, `malformed`	Might.	This action only.

The replay path does not fall back; a failure there is reported as the disagreement it is. This was checked on September 21, 2026, with a bad key in a scripted autopilot session: the first judged action printed one shift> judge: line naming the 401 and the session model, the next action was judged by the local model and allowed, and /judge reported Jev off for the session with the reason.

Measured agreement

The typed judge has been measured against the fixed case set and in one scripted session, all on September 21, 2026. The gates for trusting it further wait for shadow data from ordinary sessions, which has not accumulated yet.

Measurement	Result
The nine fixed cases, first criteria, through `scripts/jev_probe.py`	8 of 9 agreed with the human, at a mean round trip of 226 ms. The miss was a requested prompt change through `live_eval`, blocked at confidence 0.19.
The nine fixed cases, corrected criteria, same script	9 of 9, at 259 ms. The correction was one sentence: the exception for requested behavior changes had to be in the block criterion, not only in the instructions. The seven `run` allows scored 0.84 to 0.97; the two `live_eval` allows scored 0.88 and 0.61, and the 0.61 is the case to watch.
The nine fixed cases through the runtime, `scripts/evals.py judge --cases --model typesafe/jev-1.13.0`	9 of 9 at a median of 514 ms; the gap from 259 ms is one curl process and one TLS handshake per decision. Against the session model's 1,397 and 1,461 ms on the same machine, that is three times faster through the runtime and six times faster in the probe.
One scripted autopilot session with `judge-ask-below` forced to 0.99	Three allows at confidence 0.77, 0.25 and 0.75, for benign commands the request never named; the default 0.5 would have asked once. The skill hint picked `allbeads` from the 34 skills the bundled plugins offer, in 863 ms, and the model loaded it unprompted. Whether the band catches something or noise is the open question.

These are one run each on a nine-case set and one session. They say the typed judge agrees with the human on cases chosen because the session-model judge once got them wrong, and that it is faster. They do not say how it does on a week of ordinary work.

What's next

Tool policy for the modes, the allowlist and the fixed rules that run before the judge.
Benchmarks for the other measurements and how to run them.
Workflows for the judge check in a workflow step.