Nvidia Dynamo’s Agentic Speedups Target the Work Around the Model

NVIDIA Dynamo’s agentic-serving optimization work treats the router, request plane and cache-ownership ledger as performance-critical infrastructure—not background plumbing. In AgentX tests, batching cache-related routing work raised median output throughput 22.2% at concurrency 512, while a new ownership design cut replay time by more than a fifth on both vLLM and SGLang backends.

The catalyst is the shape of an agent session. Long, overlapping prefixes and thousands of live requests make Dynamo’s routing work grow with the number and length of stored prefixes, rather than simply with generated tokens. Repeated turns also send largely identical prompts and stream responses token by token, multiplying serialization, copying and metrics costs.

The router was built for a smaller live state

Dynamo’s first response was to shrink the work done for each routing decision. It removed redundant suffix invalidation and batched KV matching, registration, ownership and terminal dereferences. The reported 22.2% median throughput gain came from that group of changes, not from a faster model kernel.

Later router work addressed operations that had become costly as live state grew. Bucketed expiry pruning replaced a sweep across everything tracked and improved high-churn AgentX throughput 13.7%. Compressed prompt paths reduced frontend CPU 35.3% in the reported workload, while delta-only suffix cleanup processed only changed state rather than recomputing it.

Ownership moved out of individual cache blocks

The more structural Dynamo change concerns who is allowed to free cached state. Each block must remain available while requests still use it, but should not stay pinned after they finish. Dynamo moved from shared block chains to arena-level ownership counts and then backend-specific request leases, which track ownership at a coarser unit. Those leases reduced AgentX replay time 23.71% for vLLM and 22.02% for SGLang while lowering peak memory.

The frontend became part of the throughput budget

Dynamo then targeted work that occurs on every request or every streamed token. MessagePack payloads improved throughput 8.1% and reduced average time to first token 9.7% in an AgentX test. Static logging filters, which removed a shared span-matcher lock, raised reported frontend throughput from 932 to 1,133 requests per second.

SGLang’s parallel AgentX work reaches the same conclusion through different failure modes. Passing context length as a runtime scalar avoided repeated length-specialized compilation, improving output throughput 26.75% at concurrency 384 and reducing mean time to first token 36.25%. Its cache-aware data-parallel routing keeps reusable prefix state near the rank that holds it, while adding cache balance to prevent affinity from concentrating work on one worker.

SGLang also separates cache state by what can be recovered. Its HiCache offload moves the expensive full-attention KV cache across the bus, rebuilds the short sliding-window tail, and checkpoints recurrent state that cannot be reconstructed. FlashInfer GDN checkpoints increased throughput from 47,771 to 53,004 tokens per second per GPU at a 92.4% cache-hit rate.

Fast serving still has to preserve the right state

The staging result shows why these changes cannot be judged by throughput alone. In heterogeneous prefill-and-decode deployments, a prefix hit can leave the sender transferring only an uncached remainder while the receiver still needs a complete cache at precise offsets. SGLang’s earlier behavior could place that state incorrectly; the revised staging path fixed a correctness failure, not merely a capacity problem.

The published improvements come from specified tests and configurations, including AgentX results at stated concurrency levels. Open work spans several paths: Dynamo has a proposed change to flush detokenization metrics once per response rather than updating counters for every streamed chunk. SGLang continues resource-lease scheduling and a data-parallel graph-metadata fix for speculative decoding, while also listing heterogeneous-disaggregation work on multi-pool support, unified-KV HiSparse state and preserving a prefill-owned token when decode ends without visible content.