Superpower Daily

Superpower Daily

Your reading space

Sign in
Reported OpenAI Code Points to Managed Agents as Agent Builder...Grove City Holds 310-Acre Data Center Plan, Exposing Limits of...Google’s Ironwood Takes an FP8 Cost Lead in New Serving Benchma...NHTSA Opens Cybercab Inquiry as AI Deployment Hits Infrastructu...MCP SDK Starter Guides Leave Authorization Outside the HTTP Set...Navana.ai Raises ₹40 Crore to Expand Voice AI for Regulated Fin...Reported OpenAI Code Points to Managed Agents as Agent Builder...Grove City Holds 310-Acre Data Center Plan, Exposing Limits of...Google’s Ironwood Takes an FP8 Cost Lead in New Serving Benchma...NHTSA Opens Cybercab Inquiry as AI Deployment Hits Infrastructu...MCP SDK Starter Guides Leave Authorization Outside the HTTP Set...Navana.ai Raises ₹40 Crore to Expand Voice AI for Regulated Fin...
Swipe

Google’s Ironwood Takes an FP8 Cost Lead in New Serving Benchmarks

The early result gives Google a defined economic win against Nvidia, while unfinished serving software and FP4 limits keep the wider contest open.

By 2 min read
Google’s Ironwood Takes an FP8 Cost Lead in New Serving Benchmarks
Google’s Ironwood Takes an FP8 Cost Lead in New Serving Benchmarks

Listen to this story

The audio brief

About 1:44
0:001:44
Read transcript
Google’s Ironwood TPU is showing an early cost advantage over Nvidia’s B200 and B300 for a specific kind of AI serving: FP8 precision, aggregated serving, and single-token prediction. At a target of 100 tokens per second for each user, the modeled cost was eighteen-point-one cents for Ironwood, versus twenty-two-point-two cents for B200 and twenty-seven-point-six cents for B300. In that setup, Google’s chip delivered up to fifty percent better performance per dollar. At a lower target of 20 tokens per second, Ironwood also produced 9,364 total tokens per second per chip, compared with 8,903 for B200 and 8,925 for B300. The strategic piece is the software bridge. Google is moving from TorchAX to TorchTPU, with planned support for vLLM and SGLang, so developers can use TPUs more directly from PyTorch. StableHLO, XLA, and Pallas handle the lower-level execution. Initial bring-up used Qwen3.5 397B in FP8, with Kimi K3 and GLM5.3 next. But this is still unfinished: the open-source release is expected around October 2026. And the benchmark does not settle the broader contest. Ironwood lacks native FP4 computation, where Nvidia leads in cited comparisons, despite FP4’s quality trade-off against FP8. Google, Inferact, RadixArk, and SemiAnalysis are still working on disaggregated serving, speculative decoding, KV-cache offloading, and agentic workloads. The key question is whether those missing software paths let Ironwood’s narrow FP8 cost lead generalize beyond this preview setup.

Story brief

3 key points

Google’s TPUv7 Ironwood shows a modeled serving-cost advantage over Nvidia’s B200 and B300 in one FP8, single-token setup: $0.181 versus $0.222 and $0.276 at 100 tokens per second per user. The more important strategic development is software: Google is replacing TorchAX with TorchTPU support for vLLM and SGLang, aiming to make TPUs easier to use from PyTorch. That path remains unfinished—open-source release is...

  1. 01

    At 20 tokens per second, Ironwood reached 9,364 total tokens per second per chip, versus 8,903 for B200 and 8,925 for B300.

  2. 02

    Ironwood lacks native FP4 computation, and Nvidia led cited FP4 comparisons despite FP4’s quality trade-off against FP8.

  3. 03

    TorchTPU began bring-up with Qwen3.5 397B in FP8; Kimi K3 and GLM5.3 are planned next.

New inference benchmark results put Google’s TPUv7 Ironwood ahead of Nvidia’s B200 and B300 on performance per dollar in a defined FP8 serving setup. The result gives Google an early economic case for making its custom AI chips more usable with open-model software beyond Google’s established TPU environment.

The comparison used aggregated serving with FP8 precision and single-token prediction. In that setup, Ironwood delivered up to 50% better performance per dollar than the tested B200 and B300 systems. It is a result for a particular precision, serving design and operating range, rather than a ruling on every inference workload.

A new bridge from PyTorch to TPUs

The benchmark arrives alongside Google’s planned TorchTPU backend for vLLM and SGLang, two software systems used to serve language models. TorchTPU is intended to let those systems use TPUs as native PyTorch devices, while StableHLO, XLA and Pallas handle lower-level execution. The approach replaces TorchAX, which translated PyTorch operations into JAX operations for TPU execution.

The rollout remains a work in progress

  • Google began TorchTPU bring-up with Qwen3.5 397B in FP8 and plans support for Kimi K3 and GLM5.3.
  • The stack is expected to leave private beta and become open source around October 2026.
Modeled cost at the same user-speed target
$0.181Ironwood

Modeled at 100 tokens per second per user in the cited configuration.

$0.222Nvidia B200

Modeled at the same user-speed target.

$0.276Nvidia B300

Modeled at the same user-speed target.

The lead narrows beyond the preview setup

At 20 tokens per second per user, Ironwood also reached 9,364 total tokens per second per chip, compared with 8,903 for B200 and 8,925 for B300 in the cited runs. But the external TPU serving stack still lacks a fully optimized disaggregated-serving path. GB200 and GB300 NVL72 systems are currently more competitive in those comparisons.

Ironwood also lacks native FP4 computation, leaving Nvidia ahead in the cited FP4 comparisons, where using FP4 involves a quality trade-off against FP8. Google, Inferact, RadixArk and SemiAnalysis are working on disaggregated serving, speculative decoding, KV-cache offloading and agentic workloads. Those efforts, if delivered, will determine how far this early FP8 result can travel.

Editorial analysis

Our Read

The important shift is the route to adoption, not one favorable benchmark point. TorchTPU is intended to reduce the difference between using a TPU and using familiar PyTorch-based serving software. That could make an accelerator evaluation less dependent on a specialized software stack. But buyers should separate today’s aggregated FP8 result from the unresolved disaggregated-serving comparison, where Nvidia’s rack systems are currently more competitive. The meaningful next evidence will be a like-for-like disaggregated comparison after the planned TPU optimizations are delivered.

Related coverageEnterprise Buyers Put Non-Nvidia Chip Evaluations Ahead of Nvidia’s Next GPUsInferenceX Ranks B300 First for Qwen3.5 at 15,815 Tokens per Second

Citation desk / original work

Cite this

Permanent attributionView citation
Finding 01

The important shift is the route to adoption, not one favorable benchmark point.

/posts/google-s-ironwood-takes-an-fp8-cost-lead-in-new-serving-benchmarks#finding-1

Loading discussion...

Related coverage

Continue reading