Yifan Zhang Publishes RLT, a Transformer Design Still Waiting for Results

Yifan Zhang has published a technical report for the Recurrent Looped Transformer, or RLT, a proposed language-model architecture designed to keep a decoder’s state alive across every token in a prompt and response. The design gives each new token access to the prior decoder output and its retained attention cache, allowing the model’s computational path to grow with the sequence. But the report provides no measured results on reasoning quality, efficiency, or reinforcement-learning scaling—the outcomes that would determine whether the extra recurrence is useful.

RLT starts from a limitation of typical decoder-only language models: work completed at the final layer for one token does not directly feed into the first layer for the next. Tokens can refer back through cached attention keys and values, but the final hidden output is not itself carried forward as the next token’s input.

A loop that spans the whole exchange

RLT pairs a causal encoder with a recurrent decoder. The encoder processes the sequence into representations and memory. For each token, the decoder merges the new encoder representation with its previous final output, then applies sliding-window self-attention, cross-attention to encoder memory, and a feed-forward network. The resulting decoder output and layer-by-layer cache become state for the following token.

That recurrence continues through both the user’s prompt and the model’s reply, with no reset at the prompt-response boundary. The cache is bounded at each layer: its sliding window includes the current token and retains no more than W minus one earlier entries.

Depth is not a demonstrated reasoning gain

The report’s central idea is structural rather than a performance result. With 48 decoder layers, a token at position t ends a path containing 48t decoder-block applications. RLT therefore keeps the amount of specified work per new token fixed while lengthening the path information can travel through as the sequence continues.

That is not the same as establishing “infinite” practical reasoning. The report itself cautions that gates and contraction can suppress long paths. It offers no measurements showing that a longer recurrent path improves reasoning, and it explicitly leaves validation of reasoning, efficiency, and RL scaling for later work.

The speedup the design does not promise

The architecture separates parallel and sequential work. Known input tokens can be handled by token-parallel encoder kernels, and transitions from independent sequences can be batched. But within one sequence, decoder transitions remain sequential. RLT does not claim a parallel scan for its nonlinear decoder or a reduced-prefill speedup. It also says a conventional parallel sliding-window decoder pass is not equivalent to its recurrence.

The operational contract RLT specifies

• Training uses full-sequence next-token prediction and full backpropagation through time; supervised fine-tuning masks the loss on assistant targets but still updates state through user and tool tokens.
• For RL replay, the trainer rebuilds encoder memory, recurrent outputs, and every decoder cache from the sequence start under current parameters rather than reusing old rollout states.
• A reusable multi-turn prefix snapshot must preserve encoder cache and memory, complete decoder state, position metadata, window convention, and model version.

Those details make RLT more than a loose proposal for carrying hidden state: Zhang specifies one state transition for prefill, decoding, pretraining, fine-tuning, and current-policy RL replay. Still, the decisive question is empirical. The release defines how this loop should run; it has not shown that the loop produces better answers, lower cost, or more scalable training.