Ant Group and vLLM release Humming to simplify faster, lower-memory AI inference

Developers running lower-precision AI models have a new open-source option for handling different number formats and GPUs without maintaining a separate implementation for every combination. Ant Group and the vLLM community released Humming on October 21, a library of GPU computation routines designed to turn reduced memory use into faster inference.

Inference is the work of running a model. Quantization reduces the precision of its stored weights and intermediate calculations, cutting memory use and data movement. But smaller numbers alone do not guarantee faster responses. The kernels—the GPU routines doing the calculations—must efficiently handle the chosen format, hardware and workload. Humming targets that execution layer rather than introducing a new model.

One description, many execution paths

The engineering problem grows beyond choosing four-bit or eight-bit numbers. Quantization tools also differ in how they arrange weights and represent scaling information. Each combination can require its own preprocessing, kernel selection, framework connections and tests. Optimizations then need to be carried across those separate implementations, adding maintenance work as formats and hardware multiply.

Humming converts supported exports into a common representation while preserving each scheme’s numerical rules. It describes weight formats, scales and other parameters separately from execution choices. Activations—the intermediate values produced as a model runs—have their own settings. Supported combinations cover both conventional dense models and mixture-of-experts models, which activate only a subset of their expert components for each token.

Underneath, reusable CUDA components handle scheduling, moving data, converting low-precision values and performing calculations. Just-in-time compilation assembles the required pieces and removes unused branches. Adding a format or GPU architecture can therefore mean extending the relevant components rather than building another independent implementation. Improvements to a shared component can benefit other compatible configurations.

Shared code still gets specialized

A common architecture does not mean one fixed kernel. Humming combines the quantization settings with matrix dimensions, the target GPU and computation options. Device-specific rules select execution settings by default, and developers can refine them. It can prepare multiple kernels for different input sizes, choose among them at runtime and cache compiled kernels for reuse.

That flexibility addresses a specific performance wrinkle: small and large batches can favor different GPU execution paths. Humming accommodates both within the same component system. The developers also say it compiles kernels in parallel during model warmup. In their tests, compiling and loading the input-size variants for one layer’s matrix multiplication took about two to four seconds.

Hardware coverage has boundaries

The release supports NVIDIA GPU architectures from Turing onward, plus T-Head’s CUDA-compatible Zhenwu 810E. Support does not make every numerical format equally fast on every device. The announcement notes that B300 GPUs can execute eight-bit integer matrix multiplication through older instructions, but at substantially lower performance than their peak computing capabilities.

Ant Group and the vLLM community say Humming outperforms Marlin, FlashInfer and DeepGEMM implementations across many representative workloads. Their kernel benchmarks time warmed-up matrix multiplications independently, using a cold GPU cache. Those are component measurements, not a universal multiplier for complete model serving. The announcement links to the public Humming repository for developers evaluating the release.