Ai2 finds leading AI models fall to 13% or less on its longest, hardest tracking tasks

Leading AI models can find information in long inputs yet lose track when the answer depends on a chain of changes. Ai2 released StateBench on November 3, 2026, to measure that distinction. In its evaluations, four frontier models scored 13% or lower on the longest problems at the hardest level.

The tested models were Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. Each read a sequence of variable assignments and returned every variable’s final value. All four performed well on simpler tasks, but accuracy dropped sharply as the harder sequences grew longer.

Finding a value is not following its history

State tracking means keeping up with information as it changes: a variable is reassigned in code, or an object passes between characters in a story. Some answers require only finding the latest update. Others require carrying earlier changes forward, so one missed step can spoil the result.

• R-trivial: Values never change. The model retrieves an original assignment, much as it would recall a birth date.
• Aperiodic: Values can be replaced. The latest setting wins, like a thermostat whose earlier settings no longer matter.
• Soluble: Updates depend on the current value. A running game score requires applying each change to what came before.
• General finite: Variables can take other variables’ current values. Tracking a ball through repeated swaps of five cups illustrates the challenge.

To keep comparisons controlled, Ai2 derives the three easier versions from a general-finite sequence. The number of variables, distinct values and positions of assignments stay the same. Only the permitted kind of update changes.

256 assignmentsMaximum training sequence

Ai2 trained its small experimental models on sequences containing up to 256 assignments.

64,000 assignmentsMaximum evaluation sequence

Evaluation extended to 250 times the maximum training length.

Small models reveal an architecture split

Ai2 then asked whether failures came from model design or from what pretraining had taught the models. It trained six 275-million-parameter models from scratch on StateBench: two transformer variants, two hybrids and two Gated DeltaNet variants. Recurrent designs such as Gated DeltaNet carry a running memory forward; hybrids combine them with attention layers.

On the first three levels, the Gated DeltaNet and hybrid models stayed almost perfect through the longest sequences tested—tens of thousands of assignments. Transformers learned the easier levels but lost accuracy on longer sequences. They never learned the general-finite tasks in this experiment.

The hardest level also separated the memory-based designs. A Gated DeltaNet variant with a more flexible memory update generalized farthest, remaining near perfect to roughly ten times its maximum training length. That result belongs to the small, directly trained setup, not the frontier-model evaluation.

Ai2 interprets the improvement from direct training as evidence that training, rather than architecture, explains some pretrained recurrent and hybrid models’ weaknesses. Those models could learn levels their pretrained counterparts could not handle. Transformers did not make the same recovery on the hardest level.

The same problem can fail in a different disguise

Presentation changed results, too. The pretrained 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on simple tasks presented in a retrieval-test format. Yet they were near chance when the same underlying problems appeared as spreadsheet edits. StateBench uses multiple formats to expose that sensitivity rather than treating one successful presentation as enough.

The benchmark also keeps the number of variables fixed. Real conversations can introduce new people and objects that a model must begin tracking as they appear. Ai2 says its current framework does not yet cover that expanding set of entities.

Ai2 has released data and code for researchers to run the benchmark. The accompanying paper, submitted November 2 by Jackson Petty, William Merrill and Ashish Sabharwal, sets out the four-tier framework and its theoretical foundations.