A benchmark can carry one model’s name without showing which model finished every task. In a review of five Claude Opus 5.5 coding and workflow results, only Zapier’s AutomationBench explicitly barred fallback to an older model. Its 40.0% score counted safeguard interventions as failures. The other four rows permitted older models to take over specified sensitive work, but the published results do not show whether any handoff occurred.
That boundary affects what a score can describe. A fallback-permitted result may measure a safeguarded system rather than uninterrupted work by Opus 5.5. A no-fallback test makes model identity clearer, but counts a safeguard-stopped task as a failure. Neither rule alone predicts success on a buyer’s own work.
Five rows under one test of the labels
This review examines five rows in Anthropic’s September 22 launch table: Terminal-Bench 4.0, FrontierCode v1.1 Main, CursorBench 4.0, Terminal-Bench-Science 0.1 and AutomationBench. Using public documents available through September 23, it compares fallback rules, effort settings, graders and reporting parties. It does not classify Anthropic’s other benchmarks or claim to have rerun any test.
Anthropic’s table footnote says Opus 5.5 was evaluated with production safeguards. When they intervened, Opus 4.8 completed cybersecurity tasks; Opus 5 completed biology and frontier-model-development tasks. That rule applies to the four selected coding and science rows. Anthropic specifies an exception for AutomationBench: no fallback model, and an intervention meant failure. A permitted route is not evidence that a task actually took it.
Zapier’s clean boundary, uneven comparison
Zapier reports that max-effort Opus 5.5 completed 40.0% of AutomationBench’s official tasks. An agent works across simulated business apps, and fixed checks inspect the state it leaves behind. A task passes only when every scored requirement is met. The number measures strict workflow completion, not whether the agent wrote a convincing reply.
Anthropic’s table also lists Claude Fable 5.1 at 31.4%. Zapier’s leaderboard gives that entry a fuller name: Fable 5.1 with Opus 5 fallback. Zapier says Opus 5 handled 260 of 657 tasks after Fable’s safety classifier refused a step, and the score includes those fallback completions. The two percentages come from the same benchmark, but not the same model-identity rule.
Anthropic says counting interventions as failures lowered Opus 5.5’s score below what it would achieve in practice. That is a company counterfactual, not a measured fallback-enabled result presented here. Zapier’s released tasks cannot reproduce the official leaderboard figure exactly, either: the leaderboard uses a separate, harder private set.
What the other four scores leave open
For the four remaining rows, the safeguard footnote identifies the possible handoff, not its frequency. The checked public materials give no task-level record dividing successful work between Opus 5.5 and an older model. They also provide no matched Opus 5.5 runs with fallback switched off. The results therefore support neither a claim that fallback boosted those scores nor a claim that Opus 5.5 completed every task alone.
The effort setting behind the percentage
Model identity is only one comparison setting. On Terminal-Bench 4.0, Anthropic puts Opus 5.5 at 66.4% using xhigh effort beside GPT-6 Astra at 57.9% using high effort. The Astra number was reported by OpenAI. These are published scores with different effort levels and reporting parties, not a matched-effort run by one evaluator. Anthropic’s Opus 5 reproduction scored 52.3%, close to the 51.8% public Claude Code leaderboard result it cites.
Terminal-Bench-Science 0.1 supplies another baseline check. Anthropic reports 58.7% for Opus 5.5 and 29.0% for its Opus 5 reproduction. The benchmark owner reports Opus 5 at 30.0%, with three independent Claude Code trials on each of 70 tasks. Those close Opus 5 figures put Anthropic’s reproduction in context, but do not reveal any Opus 5.5 handoffs. Anthropic attributes its table’s 64.6% Astra figure to OpenAI.
Even Anthropic’s own presentation gives different Opus 5.5 scores at different effort levels. FrontierCode v1.1 Main shows 54.4% in the max-effort launch table and 54.6% at medium effort in the coding discussion. CursorBench 4.0 shows 57.8% in the table and 52.5% at medium effort in that discussion. A reader should not silently substitute a discussion score for a headline score, or assume that higher effort always produces the higher reported number.
The next disclosure that would settle the question
The tests grade different kinds of work. Zapier checks the final state of a simulated business. Cursor’s published account of an earlier benchmark version describes agentic graders for ambiguous coding tasks, while Anthropic says FrontierCode assesses whether code changes would be merged. These methods add context to each percentage; they do not provide a common scale for ranking all five rows.
What to watch is narrower than another headline score: task-level intervention and handoff records for the four fallback-permitted rows, then matched results with fallback disabled. Those would show whether older models contributed to the reported outcomes and by how much. Until then, the AutomationBench result has a clear no-fallback boundary; the other four have a stated route, not an observed count.
Reader comments
Newest comments first. Replies stay oldest first.