AI21 says its coding benchmark keeps premium AI routing near a 25% target

A coding-agent budget can miss its target before the work is finished: AI21 Labs says a public benchmark led its router to send nearly half of live requests to an expensive model, against a 25% target. In findings published October 14, its new, traffic-informed coding benchmark brought that routing rate within 0.5 percentage points of the target.

The budget depends on the test

AI21's Intelligent Gateway sits between developers and the models answering their requests. Its router estimates each message's difficulty, then chooses between a premium model and a cheaper, faster alternative. Teams can tune the expected share sent to the premium option, trading spending against capability.

The experiment asked whether a routing threshold chosen on benchmark data would behave similarly on real work. AI21 applied the thresholds unchanged to 1,905 developer messages. The public-benchmark setting overshot the intended premium-routing share by 23 percentage points; the company's benchmark produced the much closer result.

That result concerns routing calibration: whether a planned allocation carries over to live requests. A matching premium-routing rate does not, by itself, establish that the cheaper answers are equally good. AI21 presents the close match as evidence that its evaluation better represents its developers' work.

Most messages did not ask for code

To build the evaluation, AI21 first classified seven weeks of internal developer traffic: 4,211 conversational turns across 15 broad categories and 84 subcategories. Only about 11% of turns asked an agent to write code. The most common message was a clarifying question, rather than a request for a finished patch.

Confirmations, redirections, git commands and pasted context accounted for about 40% of turns. AI21 compared its traffic with 23 public benchmarks, including SWE-bench, HumanEval and Spider. It found that public tests concentrated on code generation and bug fixes, with little coverage of those conversational interactions.

Context differed, too. Internal developer messages averaged around 80 characters and arrived amid conversation history, tool results and unfinished file changes. More than 60% of sessions contained multiple distinct tasks. AI21 argues that a single, neatly written assignment on a clean code checkout misses this switching between jobs.

Making messy work gradeable

Real conversations presented another problem: they were difficult to score automatically. A developer's intent might never become an explicit test or rubric. Public tasks offered the opposite bargain—repeatable grading, but less resemblance to internal work. AI21 built its 613-task benchmark to combine those properties rather than simply replay raw conversations.

• Match the task mix. Category quotas came from observed developer traffic, rather than a random sample of public coding problems.
• Include internal code. AI21 mined previously solved and merged tasks from its repositories, verified that internal tests could grade them, and mixed them with public tasks.
• Change the wording, not the checks. Prompts were rewritten to resemble short developer messages while preserving the underlying code environment and automated tests.
• Restore work in progress. Internal replay infrastructure can resume sessions with accumulated context and modified files, allowing evaluation from the middle of a workflow.

The company also recommends pricing whole sessions, not isolated tasks. Switching models during a conversation consumes money and context tokens, so a per-task figure can miss costs that emerge across the workflow. Its proposed test for an evaluation is operational: does a model choice or budget threshold selected there hold up on live traffic?