Sakana AI Releases PC-ALM, Training 1,000-Layer Networks Without Backpropagation

A neural network can now be trained across 1,000 layers without the usual network-wide backward pass—at least on a narrow image-classification test. Sakana AI has released PC-ALM, a training method that uses only local exchanges between neighboring layers; the team reports its 1,000-layer MNIST models stayed within roughly two percentage points of backpropagation.

The release targets a longstanding weakness in predictive coding, a training approach in which each layer adjusts using nearby information rather than receiving a calculated gradient from the model’s output. In deep, narrow networks, that local supervisory signal can fade before it reaches the earliest layers. PC-ALM adds a Lagrange multiplier—a value that tracks a layer’s accumulated constraint error—to each layer while keeping activation and weight updates local.

A local controller at every layer

The method reframes training as a constrained optimization problem: each layer should match the prediction from the layer before it. Standard predictive coding penalizes mismatches. PC-ALM also keeps a multiplier that accumulates them. The researchers describe the pairing as a proportional-integral, or PI, controller: the current mismatch is the proportional component, while the multiplier retains past mismatch as an integral component.

• Primal update: a layer adjusts its activation using local prediction errors and neighboring states.
• Dual update: the layer’s multiplier accumulates its remaining prediction error.
• Weight update: after settling, each layer uses the resulting local signal for a Hebbian-like learning step.

A proof with a boundary

The strongest theoretical result applies to linear networks. Under a stated stability condition, the researchers prove that PC-ALM’s multipliers converge to the same credit signals calculated by backpropagation. That is not an exact-equivalence guarantee for the nonlinear residual MLPs and ResNet-18 models used in the experiments; those results are empirical.

The experimental record is broader than the headline stress test, but still bounded. Across residual-MLP sweeps covering widths and depths from 8 to 128, the team reports PC-ALM matched backpropagation at a settling budget of 2L across identity, tanh and ReLU activations. It also outperformed standard predictive coding on the reported ResNet-18 evaluations for CIFAR-10 and Tiny ImageNet.

Depth is not free

PC-ALM removes the ordered reverse sweep of backpropagation, but it does not eliminate computation between weight changes. Its local updates must settle iteratively, and the reported budget rises with network depth. The research team also notes that the dual dynamics can produce damped oscillations, unlike standard predictive coding’s monotone gradient flow.

For now, this is a training method tested on small image benchmarks, not a deployable model. Sakana AI has released an MIT-licensed JAX reference implementation that runs on CPUs and reproduces the paper’s width-depth grid. The next question is whether its local dynamics retain their advantage when the networks, workloads and hardware demands become less forgiving.