AI21 finds AI training can reward unfinished work with a perfect score

An AI model earned full reward for answering a request for a history chapter with one word: “China.” In new instruction-following research, AI21 Labs describes that failure and its proposed fix: LIFT-Bench, a test that checks whether each document section delivers the requested content, not just whether it follows the rules.

The failed assignment asked for an alternate-history chapter about the Xinhai Revolution, with no word repeated more than four times. The one-word answer satisfied the repetition rule without writing the chapter. AI21 traces the problem to IFTrain’s scoring: code checks constraints across the response, but does not check whether the underlying task was completed. Reinforcement learning—training through rewards—therefore gave the model no penalty for dropping the work.

The warning signs were easy to miss in the scores. In AI21’s experiment, IFTrain’s reward reached 1.0 by training step 180. Scores on the IFEval and IFBench instruction-following tests also rose. Yet by step 390, the typical response had become much shorter.

LIFT-Bench changes both where a rule applies and what earns credit. Rather than accepting a French paragraph anywhere, it can require French specifically in section 2.1. Code checks requirements such as language or keyword use. An AI judge checks what the section actually covers. Those content requirements count toward the same score as formatting, making task completion part of the reward rather than an unchecked assumption.

The benchmark contains 784 synthetic tasks across ten groups, covering short and long documents, different numbers of constraints per section, and prompts without outlines. Wikipedia supplies topics; an AI model generates outlines and constraints. This lets researchers vary how tightly individual sections are constrained.

The judge needed a narrower view, too. On 305 human-labelled examples, judges given the full prompt and response tended to approve too much. Giving them only the relevant section and a short request improved macro-F1, a measure of classification accuracy, by 11–19 points. AI21 selected the section-focused Gemma-4-31B judge. Its accuracy matters twice: the same judge evaluates answers and supplies training rewards.

AI21 used the same scoring system for LIFT-Train, a set of 34,258 training tasks, and compared it with IFTrain using Qwen2.5-7B. With LIFT-Train, median answers stayed around 390–530 words. No answer scoring at least 0.8 was under 30 words; in the IFTrain run, 38% of answers met both conditions.

Was content scoring responsible, or did document structure make the difference? A version of LIFT-Train without outlines also trained without collapse while retaining content in its reward. That points toward content scoring as the useful ingredient. It is not a clean isolation, however: the datasets still drew topics from different sources, Wikipedia and WildChat.

The outside tests also warrant restraint. Each training recipe performed best on its own benchmark. On AdvancedIF and ComplexBench, which also score content, IFTrain’s results slipped while LIFT-Train held steady. AI21 calls the differences small and interprets them as avoiding a regression, not establishing a general performance win.

Model selection changed with the test format as well. AI21 evaluated 17 models with eight samples per task. Among nine main models, GPT-5.5 ranked first on structured tasks but fifth on prompts whose rules applied to the whole answer. A strong position on one instruction-following leaderboard did not guarantee the same ranking for document drafting.

AI21 says LIFT-Bench, LIFT-Train and the generation pipeline are available, with links to the repository and technical paper in its research post.