Loading page…
The early result gives Google a defined economic win against Nvidia, while unfinished serving software and FP4 limits keep the wider contest open.
Listen to this story
Google’s TPUv7 Ironwood shows a modeled serving-cost advantage over Nvidia’s B200 and B300 in one FP8, single-token setup: $0.181 versus $0.222 and $0.276 at 100 tokens per second per user. The more important strategic development is software: Google is replacing TorchAX with TorchTPU support for vLLM and SGLang, aiming to make TPUs easier to use from PyTorch.
At 20 tokens per second, Ironwood reached 9,364 total tokens per second per chip, versus 8,903 for B200 and 8,925 for B300.
Ironwood lacks native FP4 computation, and Nvidia led cited FP4 comparisons despite FP4’s quality trade-off against FP8.
TorchTPU began bring-up with Qwen3.5 397B in FP8; Kimi K3 and GLM5.3 are planned next.
New inference benchmark results put Google’s TPUv7 Ironwood ahead of Nvidia’s B200 and B300 on performance per dollar in a defined FP8 serving setup. The result gives Google an early economic case for making its custom AI chips more usable with open-model software beyond Google’s established TPU environment.
The comparison used aggregated serving with FP8 precision and single-token prediction. In that setup, Ironwood delivered up to 50% better performance per dollar than the tested B200 and B300 systems. It is a result for a particular precision, serving design and operating range, rather than a ruling on every inference workload.
The benchmark arrives alongside Google’s planned TorchTPU backend for vLLM and SGLang, two software systems used to serve language models. TorchTPU is intended to let those systems use TPUs as native PyTorch devices, while StableHLO, XLA and Pallas handle lower-level execution. The approach replaces TorchAX, which translated PyTorch operations into JAX operations for TPU execution.
Modeled at 100 tokens per second per user in the cited configuration.
Modeled at the same user-speed target.
Modeled at the same user-speed target.
At 20 tokens per second per user, Ironwood also reached 9,364 total tokens per second per chip, compared with 8,903 for B200 and 8,925 for B300 in the cited runs. But the external TPU serving stack still lacks a fully optimized disaggregated-serving path. GB200 and GB300 NVL72 systems are currently more competitive in those comparisons.
Ironwood also lacks native FP4 computation, leaving Nvidia ahead in the cited FP4 comparisons, where using FP4 involves a quality trade-off against FP8. Google, Inferact, RadixArk and SemiAnalysis are working on disaggregated serving, speculative decoding, KV-cache offloading and agentic workloads. Those efforts, if delivered, will determine how far this early FP8 result can travel.
Editorial analysis
The important shift is the route to adoption, not one favorable benchmark point. TorchTPU is intended to reduce the difference between using a TPU and using familiar PyTorch-based serving software. That could make an accelerator evaluation less dependent on a specialized software stack. But buyers should separate today’s aggregated FP8 result from the unresolved disaggregated-serving comparison, where Nvidia’s rack systems are currently more competitive. The meaningful next evidence will be a like-for-like disaggregated comparison after the planned TPU optimizations are delivered.
Citation desk / original work
The important shift is the route to adoption, not one favorable benchmark point.
/posts/google-s-ironwood-takes-an-fp8-cost-lead-in-new-serving-benchmarks#finding-1
Loading discussion...
Join the conversation
Explain which factor would decide the choice for you.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.