Heidi Cuts ASR GPUs From 16 to 4 With CUDA MPS, Holding Sub-Second Latency

AWS, Nvidia and Heidi Health have published a speech-recognition serving design that cut Heidi’s required GPU instances from 16 to four while maintaining sub-second transcription latency. The collaborators report 92.1 requests per second per GPU, up from roughly 62 under default time-slicing at acceptable latency.

The target is automatic speech recognition, or ASR, where strict latency targets can strand capacity on expensive accelerators. A request for the Parakeet TDT 0.6B V2 model typically consumes only about 15% to 20% of an Nvidia L40S GPU’s compute capacity. Default CUDA time-slicing gives processes sequential access, leaving that unused capacity unavailable to another request.

One GPU context, several active requests

The deployment uses Amazon EC2 g6e.4xlarge and g7e.4xlarge instances with 48 GB Nvidia L40S GPUs. A FastAPI gateway decodes audio and sends it over gRPC to Triton Inference Server, which dispatches requests across MPS-partitioned model instances.

For transcription, Triton can hold requests for up to 50 milliseconds to assemble preferred batches of four, eight or 16. For diarization, sequence batching preserves each recording’s state as clients send 15-second audio chunks. Each Triton instance maps to an MPS partition, connecting request batching with GPU sharing.

The stack also splits model execution: the compute-heavy Conformer encoder uses ONNX Runtime with TensorRT, while the TDT decoder remains in PyTorch CUDA. The collaborators say directly calling the model’s forward method, instead of its higher-level transcription method, removes about 50 milliseconds of framework overhead per request.

MPS introduces a specific operational risk. The collaborators say a decoder trying to recapture a CUDA graph for an unseen tensor shape can be corrupted by sibling MPS instances and crash. Their implementation pre-warms expected shapes, uses eager execution outside that range, and serializes loading so several 600-million-parameter processes do not exhaust GPU memory at startup.

Heidi says it processes more than 2.4 million clinical consultations weekly across 190 countries, placing the configuration in a production setting. The reported result is built around Parakeet TDT 0.6B V2, L40S GPUs and a sub-second target. MPS trades the hard physical isolation of MIG for shared-context concurrent execution, so deployments with different models or isolation needs face a distinct choice.

The accompanying open-source repository supports either an all-in-one container or a two-container Docker Compose deployment, separating GPU inference from the CPU gateway. Its value is a reproducible serving recipe, not a universal capacity promise.