Loading page…
The reported gains come from changing how serving systems route, retain and stream state under concurrency. The next challenge is whether these AgentX-specific improvements hold across production mixes and unfinished scheduling work.
Listen to this story
NVIDIA’s Dynamo work shows that agentic inference bottlenecks increasingly sit in routing, cache ownership, staging, serialization, and frontend telemetry rather than model kernels. Changes across Dynamo and SGLang produced double-digit gains in throughput, latency, replay time, and memory use, but also exposed a cache-staging correctness failure that required fixing token placement. The practical takeaway for serving teams is to optimize the full request lifecycle—and validate cache state and answer correctness at
Dynamo’s batched cache operations raised median output throughput 22.2% at AgentX concurrency 512.
Arena-level ownership and request leases cut replay time 23.71% on vLLM and 22.02% on SGLang.
SGLang’s staging fix raised correct needles from 2/128 to 128/128 in a 127,500-token shared-prefix test.
NVIDIA Dynamo’s agentic-serving optimization work treats the router, request plane and cache-ownership ledger as performance-critical infrastructure—not background plumbing. In AgentX tests, batching cache-related routing work raised median output throughput 22.2% at concurrency 512, while a new ownership design cut replay time by more than a fifth on both vLLM and SGLang backends.
The catalyst is the shape of an agent session. Long, overlapping prefixes and thousands of live requests make Dynamo’s routing work grow with the number and length of stored prefixes, rather than simply with generated tokens. Repeated turns also send largely identical prompts and stream responses token by token, multiplying serialization, copying and metrics costs.
Dynamo’s first response was to shrink the work done for each routing decision. It removed redundant suffix invalidation and batched KV matching, registration, ownership and terminal dereferences. The reported 22.2% median throughput gain came from that group of changes, not from a faster model kernel.
Later router work addressed operations that had become costly as live state grew. Bucketed expiry pruning replaced a sweep across everything tracked and improved high-churn AgentX throughput 13.7%. Compressed prompt paths reduced frontend CPU 35.3% in the reported workload, while delta-only suffix cleanup processed only changed state rather than recomputing it.
The more structural Dynamo change concerns who is allowed to free cached state. Each block must remain available while requests still use it, but should not stay pinned after they finish. Dynamo moved from shared block chains to arena-level ownership counts and then backend-specific request leases, which track ownership at a coarser unit. Those leases reduced AgentX replay time 23.71% for vLLM and 22.02% for SGLang while lowering peak memory.
Dynamo then targeted work that occurs on every request or every streamed token. MessagePack payloads improved throughput 8.1% and reduced average time to first token 9.7% in an AgentX test. Static logging filters, which removed a shared span-matcher lock, raised reported frontend throughput from 932 to 1,133 requests per second.
SGLang’s parallel AgentX work reaches the same conclusion through different failure modes. Passing context length as a runtime scalar avoided repeated length-specialized compilation, improving output throughput 26.75% at concurrency 384 and reducing mean time to first token 36.25%. Its cache-aware data-parallel routing keeps reusable prefix state near the rank that holds it, while adding cache balance to prevent affinity from concentrating work on one worker.
SGLang also separates cache state by what can be recovered. Its HiCache offload moves the expensive full-attention KV cache across the bus, rebuilds the short sliding-window tail, and checkpoints recurrent state that cannot be reconstructed. FlashInfer GDN checkpoints increased throughput from 47,771 to 53,004 tokens per second per GPU at a 92.4% cache-hit rate.
The staging result shows why these changes cannot be judged by throughput alone. In heterogeneous prefill-and-decode deployments, a prefix hit can leave the sender transferring only an uncached remainder while the receiver still needs a complete cache at precise offsets. SGLang’s earlier behavior could place that state incorrectly; the revised staging path fixed a correctness failure, not merely a capacity problem.
The published improvements come from specified tests and configurations, including AgentX results at stated concurrency levels. Open work spans several paths: Dynamo has a proposed change to flush detokenization metrics once per response rather than updating counters for every streamed chunk. SGLang continues resource-lease scheduling and a data-parallel graph-metadata fix for speculative decoding, while also listing heterogeneous-disaggregation work on multi-pool support, unified-KV HiSparse state and preserving a prefill-owned token when decode ends without visible content.
Loading discussion...
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.