llm-d 1.0 adds traffic controls and safer upgrades for self-hosted AI

Teams running their own AI models have a new set of controls for deciding whose requests get served first—and keeping service running through upgrades. Contributors announced llm-d 1.0 on October 20, 2026, at PyTorch Conference North America, adding production-focused features to an open-source system that coordinates model serving across multiple machines.

Led by maintainers at Red Hat, IBM and Google, llm-d works with model-serving engines such as vLLM and SGLang. It adds routing and operational controls around those engines, using Kubernetes to coordinate the distributed service. It is infrastructure for operating models, not a new model.

Faster serving is not the new part

Earlier versions could already route requests toward reusable context, move cached model state out of accelerator memory, and separate prompt processing from response generation. IBM says version 1.0 advances their integration and operational maturity. The release adds a stable configuration API and controls for what maintainers call “day 2” operations: keeping a deployed service dependable.

• Traffic priorities: Flow control adds priority and fairness rules to help prevent bursty agent workloads from overwhelming services that need fast responses.
• Upgrade coordination: Router high availability and rolling-update controls are designed to keep traffic flowing. DisaggregatedSet coordinates updates across the separate pools that process prompts and generate responses.
• Background processing: A new asynchronous processor draws requests from queues such as Redis or Google Cloud Pub/Sub. Capacity checks, priority lanes and quotas let that work use spare resources alongside real-time requests.

When an agent competes with a person

An interactive coding assistant needs to answer promptly. An autonomous agent may send a burst of requests, stop to use a tool, then return for another round. Those repeated calls also offer savings: session-aware routing keeps successive turns close to servers holding their context, helping agents reuse previously processed code, documents and instructions.

The associated key-value, or KV, cache stores reusable model state. Rather than discard it when accelerator memory fills, llm-d can move it into host memory and storage. The router weighs cached context, server load and predicted response delay when selecting a destination.

The performance case has a specific baseline

Google’s release post describes a keynote demonstration with 1,000 sandboxed agents using one llm-d endpoint across vLLM, SGLang and mixed Google Cloud TPU and GPU fleets. Its quantified speed comparison comes from a separate coding-workload test.

Operators can inspect published configurations through Prism, the project’s open-source benchmark dashboard, and upload their own runs to compare throughput, latency and cost. Google recommends starting with prefix-aware routing before adding traffic controls, cache offloading and separate prompt-processing and generation pools. Teams can retain their existing model, serving engine and cluster, adopting only the components they need.