Artificial Analysis’ current coding-agent methodology settles the unit behind Astra’s 67-point result: it is GPT-6 Astra operating through Codex, not a score for the model in isolation. That makes it a practical product measurement—and a poor instrument for assigning credit between the model and the agent system surrounding it.
The score measures an agent at work
Artificial Analysis evaluates complete coding-agent variants across 326 tasks in DeepSWE, Terminal-Bench v2.1 and SWE-Atlas-QnA. Each task receives three attempts. The index averages those task-level pass@1 results across its components, while its cost, token-use and runtime figures come from the same benchmark coverage.
The listed configuration is Codex using GPT-6 Astra at max reasoning. It received a 67 index score: 67% on DeepSWE, 83% on Terminal-Bench v2.1 and 51% on SWE-Atlas-QnA. Artificial Analysis lists $4.72 in API cost, 26.8 minutes of active agent time and 4 million tokens per task for that configuration.
That framing is not a flaw in the index. It tells a customer how a named agent configuration performed, including its efficiency. Artificial Analysis defines runtime as active agent wall time, excluding environment startup, verifier or judge time, and other harness overhead; its cost measure is API token pricing, not the full cost of production deployment.
The surrounding system can change the outcome
A harness is the operating setup that gives a model its task context and working rules. In the Artificial Analysis comparisons, model labels sit inside native systems including Codex, Claude Code and Muse Code rather than one shared scaffold. Its 67-point Astra row is therefore a model-plus-harness result, just as the 68-point Muse Spark 1.3 row is a Muse Code result.
ARC Prize provides a concrete example of why that distinction matters. Its Astra results page identifies 12 harness configurations. The Standard harness lets the model retain notes it chooses to carry forward. The Provider Adapter preserves opaque reasoning state between requests and uses compaction for longer conversations, enabling reuse of earlier work.
The gap is striking, but it is not a measure of what the Provider Adapter alone added: the named harness and reasoning setting both changed. More broadly, no disclosed matched experiment runs Astra on the same tasks without a harness and with Codex or another named harness. Public evidence cannot yet put a percentage-point share of a result on the model and another share on its operating layer.
The protocol ledger is still thin
Across nine scoped public results, none discloses all nine protocol fields needed to reconstruct an agent run: scaffold, tools, state management, browser policy, retries, a time or token cap, exact model settings, task split and evaluator version. Evaluator version appears in six of nine results and task split in five, but state management and time or token caps appear in none.
The closest available comparisons still have limits
- ScreenSpot-Pro is the only disclosed no-tools result, making it the closest raw-model comparison in the set.
- OpenAI did not publish the Astra prompt, evaluator snapshot, sampling count or decoding configuration tied to its 92.7% ScreenSpot-Pro score.
- OSWorld discloses an evaluator snapshot, offline split and offline browser policy, but not Astra’s agent implementation, tools, state handling, retries or exact effort.
OSWorld’s partial disclosure is especially relevant because its public-evaluation expectation calls for agent implementations or monitoring evidence. It offers more visibility into the evaluation environment than many results, while leaving the tested Astra system itself insufficiently specified to reproduce its agent behavior.
Safety evidence answers a different question
OpenAI says it tested Astra in realistic browsing and professional computer environments and found it significantly less likely than GPT-5.6 Sol to take certain misaligned or potentially destructive actions, including unauthorized transactions, data loss, excessive access and circumvention of controls. That company-reported safety comparison is valuable evidence about behavior in those environments, but it does not specify a matched no-harness-versus-named-harness computer-use experiment.
Readers should watch for an evaluation that fixes the task split, model settings and budget, then publicly varies one harness component at a time. Until then, a 67-point Codex result is solid evidence about the tested Codex agent—not a clean answer to how much of the performance belongs to Astra alone.
Reader comments
Newest comments first. Replies stay oldest first.