LMCache Reworks the Cache Plumbing That Can Stall Long-Running Agents

Long-running AI agents do not just demand more model compute. They create a traffic problem around the model: many requests may need to load large, overlapping pieces of attention state from a shared cache at once. LMCache’s latest agentic-inference work changes that loading path so a full cache pool does not automatically become a deadlock.

LMCache is an open-source layer beneath inference engines such as vLLM. It stores reusable key-value, or KV, cache chunks by prefix hash across CPU memory, local NVMe drives and remote stores. That cache lets a later request, including one on another instance, reuse state written by an earlier request instead of reconstructing it.

The old multiprocess loading path reserved space for an entire cache load before it began. With many requests carrying contexts above 100,000 tokens, requests could exhaust the pool while each waited for its own full reservation. None could advance far enough to release capacity.

The fix is to reserve cache capacity one chunk at a time. Loads can then interleave: a request takes room for a chunk, progresses, and allows the shared pool to drain rather than claiming its whole footprint upfront. It is a scheduling change, not a claim that the underlying model runs faster.

The project also targets cache movement that is unnecessary or administratively expensive. For DeepSeek-V4 hybrid groups, LMCache stores only the useful portions, which InferenceX says cut storage per token by nearly 20 times. Sliding-window prefetch similarly loads only the active window, not state that will never be read.

A native transfer call for each object group is meant to remove repeated Python lock handoffs across staging copies and kernel launches. The saving is in coordination work that grows with the number of pieces being moved, rather than the number of bytes alone.

Two other paths the work opens

• AMD Instinct: A Triton block-sparse attention backend can supply CacheBlend’s non-prefix reuse when ROCm is detected or CUDA-only FlashInfer is unavailable.
• Long-prefix CPU offload: With decode context parallelism, each rank holds only a strided slice of KV state. LMCache now gathers those shards before saving and redistributes them when loading, restoring usable CPU cache hits for those prefixes.

AMD and LMCache also published prebuilt gfx942 and gfx950 wheels through a GitHub release rather than PyPI, where a standard LMCache install remains the CUDA build. InferenceX reports that all 56 KV-transfer kernel tests passed on an MI350X.

Two AgentX-specific changes remain open, including a hybrid lock-accounting fix. InferenceX says the bug can let one request release another request’s read locks on shared sliding-window or recurrent-state chunks. Under sustained, memory-pressured Kimi-K3 runs with DRAM offload, it produced warnings, corrupt generations and eventually GPU crashes after eviction began.

The disclosed measurements demonstrate that these changes can prevent a particular reservation deadlock, reduce specific hybrid-cache storage, and enable tested AMD and context-parallel paths. They do not establish a single throughput figure across varied production workloads, models or cache backends. For operators, the practical question is whether their own mix resembles the long, shared-prefix, memory-pressured sessions that expose both the gains and the lock-accounting failure.