IBM Research Says Its Agent Tool Cut a Repeatability Gap in Half

An AI agent can look reliable in an average score and still fail the same request on a later run. IBM Research says its new diagnostic and generated guidance reduced that mismatch in AppWorld tests, where a GPT-4.1 agent’s repeated-run performance had lagged its average success rate.

IBM has added the Consistency Analyzer and consistency-guideline generation to its ALTK-Evolve open-source toolkit. In an evaluation of 168 AppWorld tasks, IBM says the generated guidelines lifted GPT-4.1’s Pass^5 score from 53.0% to 69.0%, while Mean@5 rose from 77.4% to 81.0%.

The score that averages leave out

Mean@k is an agent’s average pass rate over k runs. Pass^k is stricter: it measures the share of tasks passed on every run. It differs from Pass@k, which counts a task as successful if at least one attempt works. IBM calls the distance between Mean@k and Pass^k the consistency gap.

Before applying its guidelines, IBM’s GPT-4.1 ReAct agent recorded a 77.4% Mean@5 but a 53.0% Pass^5 on AppWorld test_normal—a 24.4-percentage-point gap. Afterward, IBM says that gap narrowed to 12.0 points.

Finding a decision before it changes

The analyzer starts with one recorded agent trajectory. It replays each decision against its existing context and requests five completions by default, looking for steps where the model’s response varies. IBM then turns flagged decisions into instructions the agent can retrieve later, such as checking which search result is correct before acting.

The method does not require access to model logits, correct answers, live tool calls, or a complete rerun of the task. That makes the diagnostic lighter than repeatedly executing an agent in its environment, though IBM’s reported results remain limited to its evaluation setup.

Some evidence of transfer, not a deployment verdict

IBM says guidelines derived from one task improved Pass^5 by 13.0 percentage points on related tasks in the same AppWorld scenario. On gpt-oss-120b, the company reported a 6.0-point same-task increase, from 10.1% to 16.1%, and an 8.7-point gain on related tasks.

Pass^k is intentionally unforgiving: one failed repeat means the task does not count as fully successful. That may be less useful when an answer can be cheaply checked and retried. But for agent workflows where a single inconsistent action matters, IBM’s work makes repeatability a distinct measure worth putting beside average accuracy.