Loading page…
ARC Prize will label the two evaluation conditions separately after OpenAI’s context-management setup produced a near-perfect result that its shared interface did not reproduce.
Listen to this story
ARC Prize’s evaluation shows GPT-6 Astra’s ARC-AGI-3 result depends substantially on the interface: its controlled Standard harness score was 62.7%, versus 99.9% through OpenAI’s Provider Adapter. The adapter also cut aggregate elapsed time by roughly 3.66× and total tokens by 49% across shared runs, but cost more than $18,000 in either setup. ARC Prize will separate the results on its leaderboard, underscoring that tool and context design are part of the measured system—not evidence of general intelligence.
Standard-harness testing cost $26,098; the Provider Adapter run cost $18,817.
Across 167 shared game-reasoning pairs, Provider Adapter runs were 3.66× faster and used 49% fewer tokens.
Astra used fewer actions than the human baseline on 96% of completed levels, averaging 51.7% fewer actions.
GPT-6 Astra reached 99.9% on ARC-AGI-3’s Semi-Private set at high reasoning effort through a Provider Adapter harness. ARC Prize’s newly published evaluation puts a crucial label on that figure: Astra’s best score through its shared Standard harness was 62.7%.
The benchmark places agents in unfamiliar abstract, turn-based environments without explicit instructions. To complete them, agents must explore, build models from observations, identify goals, then plan and execute actions.
ARC Prize’s Standard harness gives models a provider-neutral interface and lets them carry visible notes forward. OpenAI’s Provider Adapter instead preserves opaque reasoning state between requests and uses compaction for longer conversations, letting Astra reuse prior work through its provider-designed context features.
At maximum reasoning effort, Astra scored 62.7% on the Semi-Private set at a reported cost of $26,098.
At high reasoning effort, Astra scored 99.9% under the Provider Adapter at a reported cost of $18,817.
The difference extended beyond completion. Across 167 game-reasoning pairs solved in both setups, Provider Adapter runs were about 3.66 times faster by aggregate recorded elapsed time and used 49% fewer total tokens, ARC Prize said.
ARC Prize observed Astra creating compact symbolic notation for objects, coordinates, rules, and unfinished plans. Under the Provider Adapter, it used fewer actions than the human baseline on 96% of completed levels and averaged 51.7% fewer actions per level.
In a separate PRO-LONG test with a code-execution sandbox, Astra created game-specific parsers, state models, search algorithms, planners, persistent notes, and small software libraries. ARC Prize cautioned that those runs measure a combined model-and-tools system, unlike its controlled human testing.
ARC Prize called Astra’s performance a frontier-model milestone, not proof of artificial general intelligence. ARC-AGI-3’s deterministic, closed-ended environments do not capture the real world’s complexity or open-endedness, and the group says it will report Standard and Provider Adapter results separately on its leaderboard.
OpenAI has supplied an early deployment example: it says Playco used Astra in its Playbot game-development environment to build three themed prototypes from one grey-box foundation, with 50% fewer manual fixes than its previous model. That is a company-reported result, not an independent measure of wider performance.
Editorial analysis
The striking result is not merely Astra’s 99.9% score. It is the scale of the gap between a common interface and a provider-optimized evaluation setup. ARC Prize’s decision to split its leaderboard makes that difference visible instead of treating the two numbers as interchangeable. The next useful signal is whether separately labeled results establish a consistent pattern across the leaderboard, and whether future benchmarks can test the kind of open-ended work ARC Prize says ARC-AGI-3 does not capture. For now, Astra’s result is strong evidence that the software around a model can be central to benchmark performance.
Loading discussion...
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.