Snowflake Open-Sources Semi-Persistence With Sub-Second Single-GPU vLLM Swaps

Idle models can give back GPU memory only if they can return before the next request waits too long. Snowflake says its newly open-sourced Semi-Persistence cut single-GPU sleep-and-wake cycles to 214–801 milliseconds in internal tests, making faster model swapping available to vLLM-based serving stacks.

Semi-Persistence keeps a long-lived copy of model weights in a pinned CPU-memory pool while GPU copies are discarded and restored as demand changes. Snowflake released the software through ArcticInference/semi_persistence, alongside an experimental orchestrator and dashboard for scheduling many model instances.

Avoiding the slow path on every swap

The target is a tradeoff in vLLM, the inference engine Semi-Persistence currently wraps. Its Level 1 sleep mode copies weights from GPU to CPU before freeing the GPU, which makes waking faster but adds a full transfer when the model sleeps. Level 2 releases GPU memory immediately, then reloads weights from storage before service resumes.

Snowflake instead separates a reusable model skeleton, meaning its structure and runtime state, from the weights that define a specialization. It caches the skeleton once and retains weights in vLLM’s native format, so a model can be restored without copying weights back from a GPU or loading them again from storage.

Using the whole node to restore weights

The restore path shards weights across the node, transfers those shards in parallel over PCIe, and uses NVLink to assemble them on target GPUs. The system also supports asynchronous loading, eviction, migration, consolidation, and request pausing, so data movement can overlap with request processing.

Its reservation system divides GPU memory into logical slots ranging from fractions of a GPU to several GPUs. An instance without an appropriate slot waits; when one is freed, it is assigned to a waiting instance. That coordination lets instances make independent moves without overcommitting GPU memory.

Fast results on one tested configuration

Across 2B- to 397B-parameter models, Snowflake reports a 5.6×–19.9× reduction in end-to-end sleep-and-wake latency against vLLM Level 2 under otherwise identical configuration. The internal benchmark used an AWS p5en.48xlarge with 192 vCPUs, 2 TiB of memory, and eight H200 GPUs.

Snowflake also reports restoring trillion-parameter models in 32.8 seconds, compared with vLLM cold starts of 13.3–15.5 minutes. The figures are Snowflake-reported results from its tested system.