White Circle Open-Sources Halo for Distributed Hugging Face Training

White Circle has open-sourced Halo, a distributed-training framework meant to let teams scale Hugging Face model training without changing their model definitions or checkpoint formats. The company says Halo delivers 2.3 to 2.8 times the throughput of stock TRL on the same hardware, but that result comes from its own benchmarks.

The release addresses a practical divide in model training. White Circle says standard Hugging Face tooling can become difficult to use once training no longer fits on one GPU, while larger systems such as Megatron require teams to rebuild models in a separate codebase and convert checkpoints. Halo instead adds distributed behavior around existing Hugging Face models.

Keep the model format, change the scale

Halo trainers build on Hugging Face or TRL trainers, according to White Circle, and the resulting weights remain standard SafeTensors files loadable through from_pretrained. The company says a new model family can be added with roughly 100 lines of wrapper code rather than a full reimplementation.

2.3–2.8×Reported throughput

White Circle reports this throughput range on the same hardware, alongside lower peak memory use.

Four ways to split the work

The framework offers four types of parallelism, configured in YAML: expert, context, tensor and expert-tensor. They target different constraints, from distributing the separate experts in a mixture-of-experts model to splitting a long input sequence or a matrix computation across GPUs.

• Expert parallelism places different model experts on different GPUs.
• Context parallelism divides an input sequence across GPUs while coordinating attention.
• Tensor parallelism divides matrix work across GPUs; expert-tensor parallelism uses that approach within an expert.

The speed claim has defined limits

White Circle’s comparison uses gpt-oss-20b on B300 hardware with 4,096-token sequences and gradient checkpointing disabled. Its post also says the advantage varies by workload: at a 256,000-token context length, Halo’s reported lead over TRL narrowed to 1.3 times. Those configurations make the published figures a useful starting point, not a guarantee for every training job.

A deliberately narrower stack

Halo includes fused kernels, a bf16 AdamW optimizer, LoRA support and an asynchronous reinforcement-learning loop. It does not include pipeline parallelism, which White Circle says it left out because sequential stages can leave GPUs idle while waiting for earlier stages to finish.

Halo’s appeal is therefore less about replacing every training system than avoiding a disruptive move away from Hugging Face. The decisive test is whether its native-format approach and company-reported gains hold up across the models and workloads teams actually run.