PyTorch 2.15 Release Blog

We are excited to announce the release of PyTorch® 2.15 ( release notes )! The PyTorch 2.15 release features the following changes: Rewritten NCCL backend becomes the default, with a flight recorder that dumps on hangs and TORCH_DIST_USE_NCCL2=0 or nccl-legacy as rollback paths One-sided transports for torch.distributed, with a NIXL backend and CUDA-graph-capturable transfers FSDP2 and pipelining gain finer memory and communication control, with fp32 for precision-sensitive parameters, CUDA-graph-captured gradient accumulation, and gradient reduction overlapped with compute Faster 4-bit (NVFP4) LLM decoding on NVIDIA Blackwell GPUs through fewer, better-overlapped kernel launches in NVGEMM Apple Silicon gains FP8 and faster decode, with float8_e4m3fn and _scaled_mm , a new grouped_mm with a decode fast path (1.32–1.34x tokens/s on MoE decode), and public torch.mps.Stream APIs Broader platform support with FlyDSL kernels on AMD gfx950, Tensor Data Mover support for gfx1250 (MI450), ROCm 10.0 wheels, simpler SYCL kernel authoring and multi-GPU profiling on Intel GPUs, CUDA 13.2 as the default wheel with CUDA 13.4 alongside it, and torch.compile on Python 3.15 This release is composed of 3,117 commits from 583 contributors since PyTorch 2.14. We want to sincerely thank our dedicated community for your contributions. As always, we encourage you to try these out and report any issues as we improve 2.15. More information about how to get started with the PyTorch 2-series can be found at our Getting Started page. Bring any questions you may have about PyTorch 2.15 to our Q&A webinar on Thursday, November 5, 2026, at 11 AM PT. Tristan Rice, Natalia Gimelshein, Huy Do, and Jane Xu (all Meta) will share an overview of the 2.15 release and answer community questions about PyTorch and the new capabilities in this release. Chris Gottbrath (Gottbrath Tech) will moderate. Register today. Throughout the 2.x series, PyTorch has been evolving from a research-first framework into a unified, hardware-agnostic platform for production training and inference at scale. PyTorch 2.13 landed FlexAttention on Apple Silicon, gave Inductor a CuTeDSL code path, and introduced torchcomms for large-cluster training. PyTorch 2.14 matured that path into NVGEMM, previewed the rewritten NCCL backend, and brought native linear algebra to Apple Silicon. PyTorch 2.15 builds on that work. The rewritten NCCL backend now backs every nccl process group, NVGEMM is tuned for low-precision decode, Apple Silicon adds FP8 and decode-specific kernels, AMD support extends to new compiler DSLs and (preview support) for next Instinct generation, and Intel GPUs gain a simpler kernel programming model and deeper profiling. Distributed Training The Rewritten NCCL Backend Becomes the Default When a model trains across many GPUs, those GPUs constantly exchange data through a communication backend, and on NVIDIA hardware that backend is NCCL. In PyTorch 2.15, every job that uses nccl runs on our rewritten backend by default. PyTorch 2.14 previewed our rewritten NCCL backend, ported from torchcomms, as the opt-in nccl2 backend. In 2.15 it is the default for every nccl process group, with TORCH_DIST_USE_NCCL2=0 and the nccl-legacy backend name as rollback paths. Communicators are now created eagerly with the process group, and reconfiguration can opt into a shrink/grow fast path (NCCL 2.29 or newer) so surviving ranks skip a full bootstrap. Reconfigure for fault tolerance is also enabled on ROCm with RCCL 2.30.7. To make the backend debuggable in production, a backend-agnostic flight recorder writes a post-mortem to disk when a collective hangs, and watchdog health is published to the debug server so a timeout on any rank triggers flight-recorder dumps from every rank. At cluster scale, a single stalled GPU can halt an entire training run, and finding the cause has often meant reproducing the failure by hand. The flight recorder captures what each rank was doing at the moment of the hang. API Unstable (PRs #192281 , #192895 , #199240 , #198865 and #198339 by Tristan Rice, Meta; #194774 by Chinmay Kuchinad, AMD; #192233 and #199092 by Tushar Jain, Meta; #192075 by Atharva Kshirsagar, Red Hat) One-Sided Transports with a NIXL Backend torch.distributed gains a Python transport API for one-sided data movement, with NIXL as the first backend. Transports connect to a peer through a store, without needing a process group or collectives, and Transport.read_stream/write_stream enqueue transfers on a CUDA stream so they can be captured with torch.cuda.graph . API Unstable (PRs #195643 , #197032 , #198889 and #198918 by Tristan Rice, Meta) FSDP2 and Pipeline Parallelism: Memory and Communication Control FSDP2 now gives finer control over precision and capture. Precision-sensitive parameters, such as MoE router gates and norm weights, can stay in fp32 while the rest of the model trains in bf16, still with one all-gather per fully_shard ( MixedPrecisionPolicy(param_dtype_override_fn=...) ). A full gradient-accumulation step can now be captured in a single CUDA graph. Gradient precision can also be set per parameter. Gradients accumulate directly in .grad and follow each parameter’s grad_dtype , which changes the default behavior when reduce_dtype=None (see Deprecations and Backwards-Incompatible Changes). Pipeline parallelism now allocates receive buffers just before each receive instead of for every microbatch up front, and releases forward outputs once their sends complete. Schedules accept a max_outstanding_sends budget, and FSDP unshards are staggered across each rank’s startup bubble instead of all being issued at the start of a step. Each unique edge between pipeline-parallel ranks, as determined by the schedule, now gets a dedicated P2P communicator. FSDP reduce grads can now be overlapped with subsequent compute. These improvements make PyTorch pipelining cudagraphable, deterministic, with an optimized memory profile, and expose performance tuning knobs for minimizing exposed communication. API Unstable (PRs #196640 , #196724 , #196727 , #196641 and #196963 by Chien-Chin Huang, Meta; #196626 by Anshul Sinha, Meta; #198417 by Andrew Or, Apache Spark PMC); #194114 by Animesh Jain, Meta; #196463 , #196464 and #196706 by Sanket Purandare, Meta) Performance Improvements Faster MPS Kernels for Decode and Linear Algebra Building on the 2.14 migration, more MPS operators move from MPSGraph to native Metal kernels. A single-row softmax over a 128,256-entry vocabulary drops by 8x, upsample backward is up to 21.7x faster, and reductions now share one set of kernels, making norm full reductions more than 100x faster. For decode, F.linear now uses the existing gemv kernels (1.23x geometric-mean speedup, up to 5x on small projections), and the new MPS grouped_mm has a gemv path that gives 1.32–1.34x higher MoE decode throughput on an M3 Ultra. Several linear algebra solvers were also rewritten: QR of a 512 x 512 matrix falls from about 7 seconds to 1.5 ms, and triangular solves are now blocked. (PRs #196071 , #199687 , #198611 , #198495 , #194933 , #198721 , #198725 , #198732 , #195944 and #196900 by Irakli Salia, Hugging Face; #196922 , #196241 and #195369 by Nikita Shulga, Thinking Machines Lab; #193153 by Sunghyun Choi, Tech University of Korea ; #188473 by Shrivas Shankar, University of Wisconsin-Madison) Core Features CUDA 13.2 by Default and CUDA 13.4 Wheels The default PyTorch wheel moves to CUDA 13.2 (13.2.2 toolkit), with CUDA 13.4 GA wheels shipping alongside it for Linux x86_64, Linux aarch64 and Windows. The CUDA 13.4 wheels add experimental support for NVIDIA Rubin GPUs. (PRs #198913 and #196014 by Andrey Talman, Meta; #194996 and #198843 by Ashish Karale, NVIDIA; #196560 , #199269 and #196351 by Ting Lu, NVIDIA; #192653 by Wei Wang, NVIDIA; #186245 by Lakshay Garg, NVIDIA) ROCm 10.0 and TheRock Wheels PyTorch 2.15 adds ROCm 10.0 wheels alongside the ROCm 7.14 wheels introduced in 2.14. All ROCm wheels are now bui