Meta says Prometheus’s completed first phase topped 85% useful AI training time

Meta says its Prometheus supercluster spent more than 85% of its Muse Wonder pre-training runs making useful progress, despite the risk that one faulty host could stop a large job. In an October 19 engineering disclosure, the company said the Ohio system’s first phase is complete—and explained the spare hardware and recovery systems behind those results.

Prometheus already connects hundreds of thousands of GPUs across multiple buildings in New Albany, Ohio. Meta is working toward roughly one million GPUs operating together, rather than saying that full configuration is finished. The completed phase trained its Muse foundational models, including Muse Wonder.

The production measure Meta highlights is goodput: useful training progress after disruption and recovery take their toll. It reported more than 90% goodput for Muse Wonder’s reinforcement-learning runs, which further train a model using feedback. These are Meta’s own operating figures, not an independently established ranking against other superclusters.

Eight spare GPUs inside each 72-GPU group

The move to NVIDIA’s NVL72 architecture creates a larger group of tightly connected chips—and a larger group exposed to a fault. Meta’s earlier NVL8 arrangement linked eight GPUs in that group. NVL72 raises the count to 72, multiplying the potential reach of a failure ninefold while providing its bandwidth advantage.

Meta allocates 64 GPUs from each 72-GPU group to training and keeps eight as local spares. That arrangement fits training configurations that work most efficiently with power-of-two group sizes. If an active GPU fails, a spare replaces it and the job restarts from its latest saved state, without waiting for a physical repair.

Whole-rack failures need another reserve. Meta holds roughly 2% additional racks for those events. In its example, a training run using 32,768 active GPUs requires provisioning 523 NVL72 racks containing 37,656 GPUs. Installed capacity therefore includes chips held ready for replacement, not just chips advancing the model at that moment.

>85%Muse Wonder pre-training

Meta’s reported goodput during pre-training runs.

>90%Muse Wonder reinforcement learning

Meta’s reported goodput during reinforcement-learning runs.

Recover the affected work, not always the whole job

Spare chips solve only part of the problem. Meta uses two recovery paths, depending on how far a fault spreads. The distinction determines whether unaffected parts of the training system can keep working or must return to a saved point.

• For a fault isolated to one model replica, Parallelism-Aware Fault Tolerance takes that replica offline. The rest repeat the affected step and continue training while the damaged portion restarts on spare hosts and catches up.
• For a wider fault, the whole job restarts from a checkpoint—a saved training state. Meta saves that state asynchronously across storage tiers, using changed-data saves and memory staging to limit the disruption from checkpointing itself.

The hardware work also crosses company boundaries. Meta describes embedded NVIDIA teams sharing live operating data and jointly tracing system halts to their causes. Meta handles scheduling and infrastructure; NVIDIA works on firmware, signal diagnostics and memory handling. During Muse Wonder training, Meta measured about 47,000 GPU-hours between interruptions. Later changes, including firmware fixes, raised that figure above 54,000 GPU-hours.

Keeping distant GPUs working together

Prometheus’s network must support tightly coordinated training across buildings. A dedicated Ethernet back-end network carries GPU-to-GPU traffic separately from the standard data-center network. Within a building, each network fabric connects up to 20,736 GPUs. An aggregation network links those building-level systems across distances from two kilometers to more than 100 kilometers.

Meta places communication that needs low delay and high bandwidth within a model replica. More delay-tolerant traffic moves between replicas. Its infrastructure teams and Meta Superintelligence Labs jointly tune that division, traffic patterns and software so GPUs spend less time waiting for the network. That coordination becomes especially important when physical distance limits how quickly signals can travel.

The hardware is still evolving alongside that network. Prometheus currently uses NVIDIA GB200 and GB300 platforms, with a Vera Rubin NVL72-based platform expected soon. Those additions and the roughly million-GPU target are forward plans. The concrete operational result disclosed now is a completed first phase that has already carried model-training runs.