Superpower Daily
xAI Publishes Procurement Bot Prompt That Requires Approval for...Artificial Analysis Clarifies Astra’s 67-Point Coding Score Is...Nvidia Agrees to Buy Hugging Face for $12.93B, Promising Hardwa...Google Opens Lyria 3.5 to Developers and Adds Song Generation t...Five State Lawmakers Urge Frontier AI Labs to Negotiate a Slowe...Arizona Study Finds Repeated Falsehoods Can Sway AI Models in L...xAI Publishes Procurement Bot Prompt That Requires Approval for...Artificial Analysis Clarifies Astra’s 67-Point Coding Score Is...Nvidia Agrees to Buy Hugging Face for $12.93B, Promising Hardwa...Google Opens Lyria 3.5 to Developers and Adds Song Generation t...Five State Lawmakers Urge Frontier AI Labs to Negotiate a Slowe...Arizona Study Finds Repeated Falsehoods Can Sway AI Models in L...
Businesspublished

Artificial Analysis Clarifies Astra’s 67-Point Coding Score Is a Codex System Result

The updated methodology makes the score a clearer measure of a deployable coding agent. It also leaves a central question unanswered: how much performance comes from Astra itself versus the software layer around it.

By 5 min read
Artificial Analysis Clarifies Astra’s 67-Point Coding Score Is a Codex System Result
Artificial Analysis Clarifies Astra’s 67-Point Coding Score Is a Codex System Result

Listen to this story

The audio brief

About 1:42
0:001:42
Read transcript
Artificial Analysis has clarified what sits behind Astra’s headline 67-point coding score: it is GPT-6 Astra operating through Codex, not GPT-6 Astra evaluated in isolation. That makes the result useful as a product measurement, but it cannot tell us how much credit belongs to the model versus the software layer around it. The score covers 326 coding tasks, with three attempts per task, and averages pass-at-one results across three benchmarks. Astra scored 67 percent on DeepSWE, 83 percent on Terminal-Bench version 2.1, and 51 percent on SWE-Atlas-QnA. The same configuration used about four million tokens per task, cost 4 dollars and 72 cents in API fees, and spent 26.8 minutes in active agent time. That timing excludes environment startup, verification, judging, and other production overhead. The distinction matters because a harness controls context, tools, state, retries, and sometimes how reasoning is preserved. Artificial Analysis compares native systems such as Codex rather than placing every model inside one shared setup. ARC Prize shows how large that effect can be: Astra scored 62.7 percent with its Standard harness, but 99.9 percent with Provider Adapter—and the reasoning setting changed too. Across nine public results, none reports every detail needed to reproduce an agent run. The missing test is straightforward in principle: hold the model, tasks, settings, and budget constant, then vary one harness component at a time. Until that happens, 67 points measures a deployable Codex agent—not Astra alone.

Story brief

3 key points

Artificial Analysis’ 67-point Astra result is a benchmark of a configured Codex agent, not GPT-6 Astra alone. The index combines pass@1 performance across 326 tasks, while reporting $4.72 API cost, 4 million tokens and 26.8 minutes of active agent time per task. Because harnesses manage context, state and tools—and comparisons use different native systems—public results cannot assign a percentage-point share to the...

  1. 01

    The index scores DeepSWE, Terminal-Bench v2.1 and SWE-Atlas-QnA at 67%, 83% and 51%, respectively.

  2. 02

    Reported Codex efficiency excludes environment startup, verifier, judge and other production infrastructure overhead.

  3. 03

    ARC Prize reports Astra at 62.7% with Standard versus 99.9% with Provider Adapter; reasoning settings also changed.

Artificial Analysis’ current coding-agent methodology settles the unit behind Astra’s 67-point result: it is GPT-6 Astra operating through Codex, not a score for the model in isolation. That makes it a practical product measurement—and a poor instrument for assigning credit between the model and the agent system surrounding it.

The score measures an agent at work

Artificial Analysis evaluates complete coding-agent variants across 326 tasks in DeepSWE, Terminal-Bench v2.1 and SWE-Atlas-QnA. Each task receives three attempts. The index averages those task-level pass@1 results across its components, while its cost, token-use and runtime figures come from the same benchmark coverage.

The listed configuration is Codex using GPT-6 Astra at max reasoning. It received a 67 index score: 67% on DeepSWE, 83% on Terminal-Bench v2.1 and 51% on SWE-Atlas-QnA. Artificial Analysis lists $4.72 in API cost, 26.8 minutes of active agent time and 4 million tokens per task for that configuration.

That framing is not a flaw in the index. It tells a customer how a named agent configuration performed, including its efficiency. Artificial Analysis defines runtime as active agent wall time, excluding environment startup, verifier or judge time, and other harness overhead; its cost measure is API token pricing, not the full cost of production deployment.

The surrounding system can change the outcome

A harness is the operating setup that gives a model its task context and working rules. In the Artificial Analysis comparisons, model labels sit inside native systems including Codex, Claude Code and Muse Code rather than one shared scaffold. Its 67-point Astra row is therefore a model-plus-harness result, just as the 68-point Muse Spark 1.3 row is a Muse Code result.

ARC Prize provides a concrete example of why that distinction matters. Its Astra results page identifies 12 harness configurations. The Standard harness lets the model retain notes it chooses to carry forward. The Provider Adapter preserves opaque reasoning state between requests and uses compaction for longer conversations, enabling reuse of earlier work.

The gap is striking, but it is not a measure of what the Provider Adapter alone added: the named harness and reasoning setting both changed. More broadly, no disclosed matched experiment runs Astra on the same tasks without a harness and with Codex or another named harness. Public evidence cannot yet put a percentage-point share of a result on the model and another share on its operating layer.

Representative OSWorld 2.0 long-horizon workflows and challenge categories
A representative OSWorld 2.0 workflow: submitting an ExpenseFlow reimbursement claim. The agent must follow a tutorial PDF, operate a legacy reimbursement portal, extract the correct amount from noisy receipt artifacts, trace order evidence across GMail and ChaseBank, react to a new email that changes the task state, r Source: osworld-v2.xlang.ai.

The protocol ledger is still thin

Across nine scoped public results, none discloses all nine protocol fields needed to reconstruct an agent run: scaffold, tools, state management, browser policy, retries, a time or token cap, exact model settings, task split and evaluator version. Evaluator version appears in six of nine results and task split in five, but state management and time or token caps appear in none.

The closest available comparisons still have limits

  • ScreenSpot-Pro is the only disclosed no-tools result, making it the closest raw-model comparison in the set.
  • OpenAI did not publish the Astra prompt, evaluator snapshot, sampling count or decoding configuration tied to its 92.7% ScreenSpot-Pro score.
  • OSWorld discloses an evaluator snapshot, offline split and offline browser policy, but not Astra’s agent implementation, tools, state handling, retries or exact effort.

OSWorld’s partial disclosure is especially relevant because its public-evaluation expectation calls for agent implementations or monitoring evidence. It offers more visibility into the evaluation environment than many results, while leaving the tested Astra system itself insufficiently specified to reproduce its agent behavior.

Safety evidence answers a different question

OpenAI says it tested Astra in realistic browsing and professional computer environments and found it significantly less likely than GPT-5.6 Sol to take certain misaligned or potentially destructive actions, including unauthorized transactions, data loss, excessive access and circumvention of controls. That company-reported safety comparison is valuable evidence about behavior in those environments, but it does not specify a matched no-harness-versus-named-harness computer-use experiment.

Readers should watch for an evaluation that fixes the task split, model settings and budget, then publicly varies one harness component at a time. Until then, a 67-point Codex result is solid evidence about the tested Codex agent—not a clean answer to how much of the performance belongs to Astra alone.

Editorial analysis

Our Read

Our read: end-to-end agent scores and model comparisons should not be treated as substitutes. A buyer needs to know whether a configured product completes work at an acceptable cost and speed. A model developer needs to know which component produced the gain. The next meaningful Astra result would run the same model on the same task split and budget while changing one disclosed harness feature at a time. ARC Prize’s Standard and Provider Adapter results make retained state an especially important place to start.

Citation desk / original work

Cite this

Permanent attributionView citation
Finding 01

Our read: end-to-end agent scores and model comparisons should not be treated as substitutes.

/posts/artificial-analysis-clarifies-astra-s-67-point-coding-score-is-a-codex-system-result#finding-1

Sources

  1. artificialanalysis.aiCoding Agent Index Methodology | Artificial Analysis
  2. artificialanalysis.aiCodex vs Muse Code: Coding Agent Comparison
  3. arcprize.orgGPT-6 Astra - ARC-AGI Results
  4. gui-agent.github.iogui-agent.github.io
  5. osworld-v2.xlang.aiosworld-v2.xlang.ai
  6. openai.comSafety overview: GPT-6 Astra

Related coverage

Continue reading