Google publishes AI training results for MLPerf’s first coding-agent benchmark
The new test measures time to a fixed software-repair score—not raw processing speed. Google says overlapping generation and learning kept its training chips waiting less than 3% of the time.
The benchmark’s key result is not raw throughput but how efficiently a system alternates between generating coding attempts and training from their outcomes. Google reported that 640 Ironwood TPUs reached a 0.765 coding score against MLPerf’s 0.69 target, with training chips idle less than 3% of the time; its run took 18.02 minutes. The result gives infrastructure buyers a measure of post-training performance under a fixed quality bar, though scores depend on the benchmark’s constrained agent setup.
01
The benchmark uses pass@4: a validation problem counts as solved if the agent succeeds in at least one of four attempts.
02
Validation covers 251 problems and 1,004 attempts; agents are limited to 30 turns and a 65,536-token context.
03
Hidden test files are used to deter agents from gaming rewards by removing failing tests instead of fixing bugs.
Google Cloudpublished 10 AI training results on November 16, 2026, covering its Ironwood chips and NVIDIA Grace Blackwell GPUs. Among them: an 18.02-minute run on MLPerf’s first agentic reinforcement-learning benchmark. The new test measures how quickly a system can improve a coding agent to a fixed software-repair score, rather than simply process more data.
A benchmark built around repeated attempts
MLCommons outlined the test on September 24, ahead of the v6.1 results. Unlike pre-training, which builds a model’s foundation from large amounts of data, post-training refines an existing model for particular tasks. This workload uses Qwen3.5-397B, an already post-trained model, and aims to improve its software-engineering performance under limited working space and a fixed number of agent turns.
The agent receives an issue description and a software project inside an isolated Linux environment. It must produce a patch that solves the problem. Multiple attempts receive pass-or-fail rewards, which feed a training method called Group Relative Policy Optimization. That method compares rewards within groups of attempts and updates the model to favor more successful solutions.
This requires three connected components: model training, response generation and the environments where agents work. Hundreds of attempts can compete for processor time, memory and storage. MLCommons describes balancing training and generation as a central challenge: speeding up only one can move the bottleneck to the other.
Ironwood keeps generation and learning moving
Google’s November results put that coordination problem into numbers. Using 640 seventh-generation Ironwood Tensor Processing Units, or TPUs, the company reported a coding score of 0.765 against a target of 0.69. It ran code generation and learning simultaneously, keeping the training chips waiting less than 3% of the time.
Editorial illustration for Google publishes AI training results for MLPerf’s first coding-agent benchmark.
The same announcement covered pre-training and NVIDIA-based cloud systems, not just the new agent workload. Google reported:
8,192 Ironwood TPUs reached the pre-training benchmark’s target quality in 1.70 minutes.
A separate scaling measurement reached 96% efficiency at 4,096 Ironwood chips.
256 NVIDIA GB300 GPUs on A4X Max instances reached the target model’s benchmark quality in 31.96 minutes.
Google also submitted five agentic reinforcement-learning results using A4X and A4X Max instances. Its submissions were part of a larger round: MLCommons announced 250 test results across 153 systems, including 12 systems with at least 1,000 accelerators.
What the coding score actually measures
The coding benchmark uses pass@4: a problem counts as solved if the agent succeeds at least once in four attempts. The 0.69 target therefore means solving 69% of validation problems under that rule, not succeeding on 69% of individual attempts. Validation covers 251 problems, separate from the 700 training problems.
MLCommons chose this measure after observing substantial variation in solve rates, even with deterministic sampling. Tool calls, timing and batching can change an agent’s path through a task. A full validation requires 1,004 agent attempts, each potentially generating text, using tools and running tests. In the reference setup, that evaluation can cost several training steps.
The test also guards against reward hacking—getting a passing result without fixing the underlying bug. An agent might otherwise remove failing tests instead of repairing the code. Evaluation therefore uses test files the agent cannot see or modify. The benchmark allows up to 30 agent turns and a 65,536-token context, bounding how much work and information each attempt can use.
For infrastructure buyers, the published round now includes visualization tools for comparing systems and ranking results at similar scales. Those comparisons sit alongside a quality requirement: the benchmark clock measures time to the prescribed score, not processing speed alone.
Sources
mlcommons.orgMLPerf Training v6.1: First LLM Post-Training Benchmark
blog.googleNew AI training performance benchmarks for Google TPUs
globenewswire.comMLCommons Releases MLPerf Training v6.1 Results, Spotlighting Recent Advances in AI Training
Reader comments
Newest comments first. Replies stay oldest first.