PyTorch 2.15 replaces its default GPU communication backend and adds hang diagnostics

PyTorch 2.15 now routes NVIDIA GPU training jobs using NCCL through a rewritten communication backend by default. The release pairs that switch with a recorder for stalled operations, finer control over training precision, and decoding improvements for NVIDIA Blackwell and Apple Silicon. It also ends support for Python 3.10.

The release contains 3,117 commits from 583 contributors since version 2.14. Its central communication change moves beyond the optional preview in that earlier version: every process group configured with nccl now takes the rewritten path, rather than requiring developers to select it.

A stalled training run gets a record

NCCL handles data exchanges between NVIDIA GPUs during distributed training. A single stalled GPU can halt the run. PyTorch’s new flight recorder writes diagnostic information to disk when a collective—a coordinated operation across training processes—hangs. A timeout on any process triggers recorder dumps across all of them, capturing what each was doing.

The switch is not irreversible. Setting TORCH_DIST_USE_NCCL2=0 or selecting nccl-legacy restores the previous implementation. PyTorch labels the rewritten backend’s API unstable. For jobs that reconfigure their GPU group, an optional shrink-or-grow fast path lets surviving processes skip a full bootstrap; that feature requires NCCL 2.29 or newer.

Another addition gives developers a different way to move data. A new Python transport interface in torch.distributed uses NIXL as its first backend. It connects to a peer through a store without requiring a process group or collective operation. Transfers can run on a CUDA stream and be captured in CUDA graphs.

FSDP2, PyTorch’s system for sharding a model across GPUs, gains more selective precision controls. Sensitive parameters, including mixture-of-experts router gates and normalization weights, can remain in 32-bit precision while the rest trains in BF16, a 16-bit format. The implementation retains one parameter-gathering operation per fully_shard unit. A full gradient-accumulation step can also be captured in one CUDA graph.

Decoding improvements target specific hardware

On NVIDIA Blackwell, the release targets language-model decoding in the four-bit NVFP4 format. Its NVGEMM matrix-multiplication backend folds a dequantization scaling step into the main computation, removing a separate kernel launch for each layer. It also allows a following computation to overlap with the end of the matrix multiplication. These are changes to how GPU work is launched and scheduled, not a new model.

Apple Silicon gets support for the float8_e4m3fn eight-bit format through PyTorch’s MPS backend, alongside scaled matrix multiplication. A new grouped-matrix operation has a decoding-specific fast path for mixture-of-experts models. The performance result below is PyTorch’s measurement on an M3 Ultra, not a promise of the same gain across Apple hardware.

1.32–1.34×MoE decoding throughput on M3 Ultra

PyTorch reports this throughput improvement from the new grouped_mm decoding fast path.

The installation baseline moves too

The release changes more than execution kernels. Its platform packages and supported runtimes also shift, including a different way of installing AMD’s GPU software libraries. The release blog details these installation changes:

• Python: Version 2.15 supports Python 3.11 through 3.15, including free-threaded 3.14t and 3.15t builds. Python 3.10 packages are no longer built or tested, and source builds on 3.10 are unsupported.
• NVIDIA: The default installation package moves to CUDA 13.2. CUDA 13.4 packages are also available for Linux x86_64, Linux aarch64 and Windows, adding experimental support for Rubin GPUs.
• AMD: ROCm 10.0 packages join the existing ROCm 7.14 option. All ROCm packages now use AMD’s TheRock builds and install the matching ROCm package as a pip dependency rather than bundling its libraries.