AWS says Deepgram processed 11–70% more audio per dollar on Trainium2

Deepgram can process more recorded speech for the same compute budget after adapting its Nova-3 transcription model to Amazon’s Trainium2 chips, according to an AWS engineering account published October 19. AWS measured 11–70% more audio per dollar on offline batch workloads. Getting there took months of work on the model and its serving software.

First, get the whole model onto the chip

The project targeted jobs such as contact-center analytics and medical transcription, where completed recordings can be processed in batches. Deepgram supplied Nova-3 in standard PyTorch code, the framework used to write and run the model. Its engineers then worked with AWS’s Forward Deployed Engineering team to adapt it for Trainium2.

The team used TorchNeuron, AWS’s native PyTorch integration, to run the model before compiling it for the accelerator. Initial checks looked for operations falling back to the CPU. On the compiled path, every model operation ran on Trainium2, with no CPU fallbacks detected.

They compiled the encoder and decoder—the model’s two main processing stages—separately for fixed input shapes. Accuracy checks accompanied the changes: individual layers were compared against CPU outputs, while the full model’s word error rate had to remain aligned with Deepgram’s reference values and target.

Reusing weights instead of reloading them

A working port was only the baseline. The team replaced several layers with kernels: low-level routines that carry out particular calculations on the chip. Prebuilt routines from AWS’s Neuron Kernel Interface library supplied the starting point. Profiling tools then identified where custom code could do better.

One bottleneck involved repeatedly fetching the same model weights from off-chip memory. The replacement routine loaded each block of weights once and reused it across multiple blocks of input. AWS reports that this halved latency for the affected feed-forward calculation at a batch size of 96, compared with the reference kernel.

Another calculation ran faster after being moved to a different processing engine within Trainium2. A separately written convolution kernel also removed a compilation barrier that had prevented the model from running as a single graph above batch size 32. The team tested batch sizes from one to 128 to find the highest throughput.

Keeping Python out of the serving path

Deepgram’s production serving software is written in Rust, so compiling the model did not finish the integration. The team built a lightweight runtime that coordinated replicas through a shared queue. A C software layer loaded and executed the compiled model directly through AWS’s Neuron Runtime API, without Python or PyTorch in the serving path.

That change addressed a specific source of overhead. Nova-3 makes a series of small decoder calls for each audio chunk, leaving the host computer’s dispatch work as a bottleneck at low batch sizes. Removing Python’s interpreter overhead produced the largest throughput improvement there, though gains persisted at the best-performing batch size.

The Rust runtime reproduced the reference token sequences exactly at every tested batch size, according to AWS. Each model replica fit on one physical Trainium2 core, allowing eight replicas to process inputs concurrently rather than grouping physical cores into the chip’s default, larger logical units.

The economic result is workload-specific, not a general ranking of chips. AWS calculated audio per dollar using instance prices at the time of evaluation. It did not name Deepgram’s previous compute options or publish absolute audio throughput. Those omissions limit direct comparisons for buyers assessing their own transcription systems.