Prime Intellect adds faster AI training kernels, claiming 34% higher throughput

Prime Intellect has added training support to DeepSeek’s Mega MoE GPU software, targeting time lost while chips exchange data. In its October 29 release, the company reports up to 34% higher training throughput in Prime-RL. The larger headline figure—up to 3.11 times faster—measures MoE kernels, the GPU routines handling part of that work.

Two speed gains, two different comparisons

Prime Mega MoE builds on a forward-only implementation from DeepSeek’s DeepGEMM collection. Prime Intellect added an overlapping backward pass, the part of training that works backward through a model to calculate updates. Combining both directions lets Prime replace standard MoE layers with custom GPU routines inside Prime-RL.

The company compares its kernels with a default PyTorch-based implementation, reporting a peak speedup of 3.11 times. Its end-to-end comparison instead switches Mega MoE on and off within Prime-RL, producing up to 1.34 times the throughput. These are different measurements against different baselines: the kernel result is not a claim that an entire training run finishes three times faster.

Up to 3.11×MoE kernels

Company-reported kernel speedup against the default PyTorch-based implementation.

Up to 1.34×End-to-end training

Company-reported throughput gain against the Prime-RL baseline, equivalent to 34% higher throughput.

Less waiting between experts

A mixture-of-experts model, or MoE, sends each token—a small unit of input—to only a subset of its specialized components, called experts. Those experts sit on different GPUs. Each layer must send tokens to the right devices, run the expert calculations and return the results. Training requires communication in both the forward and backward passes.

With separate operations, the GPU can spend time waiting between data transfers and calculations. DeepSeek’s fused kernel combines token routing, expert computation and result assembly in one invocation. While one group of tokens travels between devices, the GPU processes another group that has already arrived. Prime’s addition brings that overlapping approach to the backward pass.

Fusion also changes how temporary data is handled. Rather than creating every intermediate tensor separately, the kernel can use GPU-only workspace buffers. DeepSeek’s implementation handles computation and communication directly, avoiding round trips from the GPU back to the CPU. The optimization therefore addresses both idle time and the overhead of coordinating separate operations.

The hardware and workload behind the numbers

Prime’s full-step benchmarks used eight H200 GPUs per node. Qwen3-30B-A3B ran across four nodes, while Qwen3-235B-A22B used 16. Both used sequences of 16,384 tokens and FP8 precision. Experts were distributed across the eight GPUs within each node, with the wider training workload spread across all GPUs.

The tests used real data and real token routing, not a synthetic routing pattern. Prime ran 20 steps and averaged steps six through 20, excluding the first five for compilation warmup. Its separate per-layer test used one eight-H200 node, 128 experts and eight selected experts per token. That narrower test used identical random routing for both implementations.

Hardware support extends beyond that measured setup. Prime Mega MoE supports bfloat16, plus blockwise FP8 on Hopper and MXFP8 on Blackwell. The distinction is important: Blackwell support describes where the software can run, while the published full-step benchmark setup uses H200s. Those measurements should not be read as a Blackwell speed result.