Meta Sends MetaRoCE to OCP for Loss-Tolerant AI Ethernet
The proposed RDMA transport treats out-of-order arrival and packet loss as normal network conditions, shifting more control to endpoint NICs. Meta’s reported results are promising, but hardware interoperability remains the consequential test.
Listen to this story
The audio brief
Story brief
3 key pointsMeta’s proposed answer to AI-cluster congestion is a transport that lets commodity Ethernet tolerate reordering and loss instead of relying on switch-level pauses. MetaRoCE’s endpoint NICs spray packets over multiple paths, place them directly in memory, and retransmit only missing packets on affected routes. In Meta’s tests, it retained roughly 86% of throughput at 1% loss and still delivered useful bandwidth at...
- 01
Standard RoCE depends on in-order delivery and PFC; MetaRoCE is designed to avoid both and reduce head-of-line blocking.
- 02
Per-path congestion tracking and selective acknowledgments localize retransmissions instead of stalling an entire connection.
- 03
Meta and AMD tested the transport on Pensando programmable NICs using RCCL collectives across 64 AMD GPU nodes.
Meta has designed MetaRoCE, a clean-sheet RDMA transport for AI workloads on commodity Ethernet, and is contributing its specification, reference software and compliance suite through the Open Compute Project. The protocol is built to accept packets arriving out of order and to recover from loss without asking Ethernet switches to pause traffic. That addresses a costly AI-cluster problem: during synchronized training work, the slowest transfer can set the job’s pace.
Packets take many routes, and the NIC puts them back to work
Meta says standard RoCE expects frames to arrive in order, uses priority flow control, or PFC, and discourages packet spraying across routes in large multiplane networks. MetaRoCE instead sprays packets across paths and treats out-of-order arrival as its normal operating condition.
Each packet carries destination information, allowing the receiving NIC to write data directly to its final memory location. Meta says this removes the need for a reorder buffer and avoids head-of-line blocking, where a delayed packet holds up later arrivals. The design also does not require PFC or pause frames.
Path-level recovery replaces a connection-wide stall
A MetaRoCE connection has distinct paths, each with its own round-trip measurement, congestion state, utilization data and congestion window. The NIC uses different UDP source ports to give equal-cost multipath routing distinct traffic to distribute, then can shift traffic away from a troubled route.
Selective acknowledgments identify a missing packet on the affected path and trigger its retransmission there. MetaRoCE pairs sender-side ECN congestion feedback with receiver-provided rate hints, so congestion can slow the marked path while traffic moves toward clearer ones.
Meta reports that MetaRoCE retained about 86% throughput at 1% packet loss and continued delivering useful bandwidth at 10% loss in its tests.
The open specification now needs more hardware
MetaRoCE separates ordered message streams from bandwidth control, allowing one connection to carry many streams and many paths under one congestion controller. Meta says existing RDMA Verbs APIs and software stacks work without modification, while extension APIs expose features such as multiplane support.
Meta and AMD implemented the transport on AMD Pensando programmable NICs and evaluated it on a 64-node AMD GPU cluster running RCCL collectives. Meta says it beat RoCEv2 on throughput and flow-completion time in those tests. It plans to release a DPDK-optimized reference implementation, specification and compliance framework at the OCP Global Summit in October 2026; additional vendor implementations are under way.
Editorial analysis
Our Read
MetaRoCE is a strategic bet that AI-network intelligence should live mainly at the endpoint, rather than in increasingly specialized switch fabrics. That makes it a useful counterpoint to NVIDIA’s recently described Spectrum-X Multiplane design, which assigns adaptive routing work to switches alongside network cards. The October OCP release is the next concrete milestone: a specification, reference stack and compliance framework can make independent implementations possible. But implementations beyond AMD Pensando hardware will determine whether MetaRoCE becomes a multi-vendor transport or remains a design validated in a narrow environment.
Sources
- engineering.fb.comMetaRoCE: A New RDMA Transport Built for AI-Scale Ethernet
