Loading page…
Co-developed with Meta, the stack promises minimal code changes and integrations with familiar training and serving tools. Its large-scale training results remain Google-reported.
Listen to this story
TorchTPU, released as open source by Google Cloud on October 20, 2026, gives PyTorch developers a path to train and serve models on Google TPUs without switching frameworks. Co-developed with Meta’s PyTorch team, it connects familiar PyTorch workflows to model libraries, serving systems and orchestration tools. Google reports strong results on a specific training setup, but those figures do not establish a performance advantage across workloads, and moving existing applications may still require code changes.
Transformers v5 can target TPUs with device_map="tpu"; Diffusers supports compilation and splitting large generative models across accelerators.
The vLLM TPU backend retains existing workers, schedulers and API servers, while SGLang can reuse cached model state across requests.
With TorchTitan, Google reports up to 52% Model Flops Utilization on Qwen 3 8B and stable, near-linear scaling across 8,192 Ironwood TPUs.
Developers using PyTorch to build and run AI models now have an open-source route onto Google’s Tensor Processing Units, or TPUs. Google Cloud released TorchTPU on GitHub on October 20, 2026. Co-developed with Meta’s PyTorch team, the stack promises to move existing workloads onto Google’s chips with minimal code changes, while retaining familiar tools.
TorchTPU is a software stack for running PyTorch workloads natively on TPUs, rather than a new model or chip. Google says it supports training, serving and deployment through standard PyTorch programming interfaces. The intended change is where the work runs, without requiring developers to replace the framework they use to write it.
The development path has two stages. Google says developers can debug models using eager mode, then use torch.compile to optimize execution for production. The company says that compilation step improves TPU use and reduces memory overhead. TorchTPU also supplies diagnostics and built-in support for distributing work across devices.
The migration promise needs precise wording. TechCrunch describes moving existing applications with a configuration change and no rewrite. Google’s own announcement uses the more qualified phrase “minimal code changes.” That leaves room for adaptation; it is not a blanket assurance that every existing workload will move unchanged.
At Meta, advancing the frontier of AI requires immense compute scale and deep hardware diversity. We cannot be constrained by a single hardware architecture.
Jana van Greunen, Director of PyTorch Engineering, Meta, in Google Cloud’s announcement
Google worked with Hugging Face to bring native TorchTPU execution into its libraries. In Transformers v5, Google says developers can select TPUs by passing device_map="tpu". Its Diffusers integration supports eager mode, compilation and splitting large generative models across multiple accelerators. Google also built dedicated pipelines to test the Transformers suite on TPU infrastructure every night.
The release also connects to tools that serve models and coordinate computing resources. These integrations address different parts of an AI system, rather than asking one library to handle the whole workflow:
For training, TorchTPU integrates with TorchTitan, PyTorch’s reference implementation for distributed training. Google says the combination uses compilation, specialized Pallas kernels and graph-level optimizations. TorchTitan supplies training recipes for large dense models, mixture-of-experts models and diffusion transformers, giving the release a training path alongside its model-serving integrations.
Google reports that this training path achieved up to 52% Model Flops Utilization, a measure of training compute use, on Qwen 3 8B. It also reports near-linear scaling and stability across as many as 8,192 Ironwood TPUs. Those are company-reported results for a named model and hardware setup, not a demonstrated performance advantage for every PyTorch workload.
Story updates
Developers serving AI models with vLLM or SGLang now have a native PyTorch route onto Google’s TPU chips. Google released the two backends on October 20, 2026, aiming to preserve familiar software while changing the hardware underneath. The release includes tools for handling the growing conversation histories that make long-running AI agents expensive to serve.
This is not the first way to run either engine on TPUs. Both already had backends based on JAX, another framework for numerical computing. Google says developers wanted to customize and debug their serving software in PyTorch instead. It built the new engines on TorchTPU, working with Inferact on vLLM and RadixArk on SGLang. The existing JAX versions will remain maintained and supported.
The vLLM project’s release post makes the compatibility promise concrete: most models supported by vLLM can run on the new backend without changes to their model code. Its command-line interface, OpenAI-compatible server, scheduler and cache manager stay the same. The backend installs as a plugin that vLLM discovers automatically, rather than replacing those upstream components.
Underneath, each TPU device runs its own PyTorch worker process. Model execution compiles through torch.compile into XLA, while specialized Pallas kernels—routines tuned for particular computations—handle operations such as attention and matrix multiplication. Google describes the same division across both engines: retain the framework’s serving machinery and put TPU-specific implementations below it. Most models use the standard upstream PyTorch definitions.
For agent workloads, the release addresses more than model execution. Agents repeatedly send conversations that grow with each turn. Prefix caching reuses the already-processed beginning of those conversations instead of computing it again. Both engines support this reuse, including for hybrid model architectures, and can move older cached state from fast chip memory into host memory.
That saved attention state is called the key-value, or KV, cache. Both engines use TPU Sync, an open-source C++ library, to move it. Google says the library reads device buffers directly, avoiding an extra staging copy in host memory. Transfers run outside Python while computation continues. The same machinery supports separating prompt processing from response generation across TPU hosts, even when those hosts divide the work differently.
The broader feature set includes speculative decoding, tool calling, structured output, and image and video inputs. Google also lists cache offloading to Managed Lustre storage. These features cover distinct pressures: accelerating response generation, connecting models to tools, and keeping reusable conversation state beyond the chip’s own memory.
Google says both engines already run at scale with preview customers. Its announcement also includes a statement from Meta’s PyTorch-TPU lead, Zongwei Zhou, confirming SGLang-TorchTPU is running in production at Meta. For performance, Google cites a lowest cost of 2.01 cents per million tokens in its optimized Qwen 3.5 InferenceX results. That is a company-reported benchmark result for the tested setup, not a universal serving cost.
Getting started with vLLM-TorchTPU requires Python 3.12 on a Cloud TPU virtual machine. Installation brings matching versions of vLLM and the required PyTorch and TPU packages. Ironwood, or TPU v7x, is the recommended hardware and is tested on every pull request; support for v6e, v5e and v5p remains experimental.
The first startup compiles graphs for the configured shapes and can take several minutes. Compiled artifacts can be cached for reuse after restarts. The project’s planned automated onboarding of new open models within 24 hours is still on its roadmap, alongside session-aware routing and diffusion-model serving—not a launch-day guarantee.
vllm.aiLoading discussion...
Join the conversation
Explain what else would make a switch worthwhile.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.