GPT-6 Astra Hits 99.9% on ARC-AGI-3, but Scores 62.7% in a Shared Test
ARC Prize will label the two evaluation conditions separately after OpenAI’s context-management setup produced a near-perfect result that its shared interface did not reproduce.
Listen to this story
The audio brief
Story brief
3 key pointsARC Prize’s evaluation shows GPT-6 Astra’s ARC-AGI-3 result depends substantially on the interface: its controlled Standard harness score was 62.7%, versus 99.9% through OpenAI’s Provider Adapter. The adapter also cut aggregate elapsed time by roughly 3.66× and total tokens by 49% across shared runs, but cost more than $18,000 in either setup. ARC Prize will separate the results on its leaderboard, underscoring that...
- 01
Standard-harness testing cost $26,098; the Provider Adapter run cost $18,817.
- 02
Across 167 shared game-reasoning pairs, Provider Adapter runs were 3.66× faster and used 49% fewer tokens.
- 03
Astra used fewer actions than the human baseline on 96% of completed levels, averaging 51.7% fewer actions.
GPT-6 Astra reached 99.9% on ARC-AGI-3’s Semi-Private set at high reasoning effort through a Provider Adapter harness. ARC Prize’s newly published evaluation puts a crucial label on that figure: Astra’s best score through its shared Standard harness was 62.7%.
The benchmark places agents in unfamiliar abstract, turn-based environments without explicit instructions. To complete them, agents must explore, build models from observations, identify goals, then plan and execute actions.
The score changes with the interface
ARC Prize’s Standard harness gives models a provider-neutral interface and lets them carry visible notes forward. OpenAI’s Provider Adapter instead preserves opaque reasoning state between requests and uses compaction for longer conversations, letting Astra reuse prior work through its provider-designed context features.
Two harnesses, two reported outcomes
Standard harness
At maximum reasoning effort, Astra scored 62.7% on the Semi-Private set at a reported cost of $26,098.
Provider Adapter
At high reasoning effort, Astra scored 99.9% under the Provider Adapter at a reported cost of $18,817.
The difference extended beyond completion. Across 167 game-reasoning pairs solved in both setups, Provider Adapter runs were about 3.66 times faster by aggregate recorded elapsed time and used 49% fewer total tokens, ARC Prize said.
Astra turned game rules into working shorthand
ARC Prize observed Astra creating compact symbolic notation for objects, coordinates, rules, and unfinished plans. Under the Provider Adapter, it used fewer actions than the human baseline on 96% of completed levels and averaged 51.7% fewer actions per level.
In a separate PRO-LONG test with a code-execution sandbox, Astra created game-specific parsers, state models, search algorithms, planners, persistent notes, and small software libraries. ARC Prize cautioned that those runs measure a combined model-and-tools system, unlike its controlled human testing.
The next move is clearer labeling, not an AGI verdict
ARC Prize called Astra’s performance a frontier-model milestone, not proof of artificial general intelligence. ARC-AGI-3’s deterministic, closed-ended environments do not capture the real world’s complexity or open-endedness, and the group says it will report Standard and Provider Adapter results separately on its leaderboard.
OpenAI has supplied an early deployment example: it says Playco used Astra in its Playbot game-development environment to build three themed prototypes from one grey-box foundation, with 50% fewer manual fixes than its previous model. That is a company-reported result, not an independent measure of wider performance.
Editorial analysis
Our Read
The striking result is not merely Astra’s 99.9% score. It is the scale of the gap between a common interface and a provider-optimized evaluation setup. ARC Prize’s decision to split its leaderboard makes that difference visible instead of treating the two numbers as interchangeable. The next useful signal is whether separately labeled results establish a consistent pattern across the leaderboard, and whether future benchmarks can test the kind of open-ended work ARC Prize says ARC-AGI-3 does not capture. For now, Astra’s result is strong evidence that the software around a model can be central to benchmark performance.
Sources
- arcprize.orgOpenAI's GPT-6 Astra on ARC-AGI-3 | ARC Prize
- openai.comPlayco cut manual fixes 50% prototyping games with GPT-6 Astra
