Cohere Publishes One-Kernel Server for North Mini Code, Claims 1.58× vLLM Speedup

Cohere has published a serving engine for North Mini Code that replaces the usual chain of GPU operations during text generation with one persistent CUDA kernel. The company says the design reaches 292 tokens per second at batch size 1 on an H100, versus 185 tokens per second for vLLM.

The release is a serving-system change, not a new model. Cohere says the implementation is available on GitHub and supports continuous batching, paged attention, ragged sequence lengths, tool calling and an OpenAI-compatible endpoint. Those are the features that let an optimization designed around a single decode step operate as a model server rather than only as a benchmark demonstration.

The delay is between operations

Cohere says low-batch text decoding is limited more by moving model weights through GPU memory than by raw arithmetic. North Mini Code has 30 billion total parameters but activates 3.3 billion per token. In BF16, each decode step streams about 6.6 GB of weights and roughly 0.5 GB of key-value cache at an 8K context.

Conventional inference software typically launches separate kernels for operations such as normalization, projections, attention and mixture-of-experts routing. At each boundary, work can stall while GPU processors wait for other processors to finish and for the next kernel to start. Cohere argues that those pauses are especially costly when each generated token requires a relatively small amount of computation after large memory transfers.

One resident program, many small tasks

The megakernel keeps one threadblock, a working group of GPU threads, resident on each streaming multiprocessor for the whole decode step. Rather than asking the GPU driver to launch a fresh program for every operation, each threadblock reads a host-prepared list of small tasks from memory. Counters in global memory record when a task’s inputs are ready.

What the scheduling change is meant to avoid

• Wave quantization: a conventional operation can leave processors idle when its final group of work is smaller than the GPU’s full set of processors. The persistent design can assign a ready task to an available processor.
• False dependencies: a task can begin when its own input is ready instead of waiting for every related task in a whole GPU-wide operation to finish.
• Unused memory bandwidth: immutable weights can begin loading before a task’s activation input has arrived, according to Cohere’s description of its prefetching approach.

A hand-built approach to a specialized server

Cohere says its implementation is a single CUDA file built from ordinary tiled matrix multiplications and paged attention, not a compiler-generated system or new programming model. Each operation uses a shared calling convention, so different kinds of work can run back to back in one persistent kernel. Prior megakernel work has largely taken one of two paths: automatically generated kernels or standalone batch-size-one decode demonstrations.

The speed figures are specific to Cohere’s setup. The company reports end-to-end gains of 1.25× to 1.41× over vLLM on a single H100 in BF16. It says the advantage persists across batch sizes and to 256K context without measurable accuracy loss. The results test the serving architecture but do not establish the same gain on other models or hardware.

For operators running North Mini Code, the immediate proposition is straightforward: retain familiar server capabilities while removing some scheduling barriers inside each generated token. The unresolved question is portability. Cohere has shown that a hand-built persistent kernel can serve this model with real request features; whether that recipe transfers cleanly across the wider variety of model architectures and deployments is still to be tested.