NVIDIA Publishes Confidential AI Test With Under 5% Per-Token Overhead

NVIDIA says it kept confidential AI inference within 1.2% to 4.3% of baseline per-token latency on an eight-GPU B200 system, a result that makes privacy protections look less costly for one demanding model-serving setup.

The company published the benchmark as enterprises and regulated users consider running sensitive prompts, proprietary model context and other data through AI systems. NVIDIA Confidential Computing uses memory-encrypted confidential virtual machines, confidential GPUs and encrypted NVLink connections to protect workloads while they are processed. NVIDIA’s result is a company measurement, not an independent benchmark.

One configuration, held constant

NVIDIA compared confidential-computing-on and -off runs while holding the model, hardware, framework, sequence lengths, parallelism and concurrency constant. The test used DeepSeek-R1-0528-NVFP4 with TensorRT-LLM on one DGX B200 system, with eight B200 GPUs. Requests had 32,000-token inputs and 1,000-token outputs; concurrency ranged from one to 16 requests.

The result required changes below the model

The numbers do not mean confidential mode is a simple switch. NVIDIA says secure execution changes how data moves, how software measures kernel speed and how GPUs communicate. Without adaptations, those differences can introduce overhead or lead the runtime to make poorer choices.

• For host-to-device transfers, TensorRT-LLM selects pageable memory on affected paths rather than always using pinned memory, because protected virtual-machine memory changes the usual transfer behavior.
• For device-to-host copies, it moves repeated token and sampling-data readback to an asynchronous worker so protected copies do not block the main decode scheduler.
• For kernel autotuning, it uses the GPU global timer in confidential mode after NVIDIA found CUDA-event timestamps unstable in the tested configuration.
• For multi-GPU communication, software must account for the absence of NVLink SHARP multicast in the tested B200 confidential-computing setup.

A useful result with a practical boundary

The workload was deliberately shaped to expose confidential-computing costs: a long prompt, extended generation and low enough concurrency that encryption-related stalls had fewer chances to disappear behind other requests. That makes it a meaningful stress test, but not a prediction for every model, traffic mix or hardware configuration.

For infrastructure teams, the more useful takeaway is methodological. NVIDIA recommends measuring the same production-target workload with confidential computing enabled and disabled, rather than treating its range as a universal penalty. The benchmark shows the privacy-performance tradeoff can be narrow, but only when security configuration and inference engineering are treated as one deployment problem.