IBM says learned rules make customer-service AI more reliable without retraining

A customer-service agent can improve by studying its mistakes without changing the model underneath it. In results published October 20, IBM Research says ALTK-Evolve turned past task attempts into written instructions, improving agents’ ability to complete simulated retail and airline requests correctly across repeated runs. The gains came without model retraining or human annotation.

A playbook extracted from past attempts

ALTK-Evolve examines an agent’s attempts on training tasks and distills successes and failures into short behavioral rules. The same model both extracts those rules and handles subsequent customer conversations. IBM adds the resulting playbook after the existing policy in the agent’s system prompt—the instructions it receives before a conversation. The benchmark’s tools, policy and basic think-then-act workflow stay unchanged.

• For identity checks, the retail playbook tells the agent to use a customer’s full name and ZIP code when email is unavailable.
• Before changing an order or issuing a return, it calls for a structured summary and explicit customer approval.

IBM extracts the guidelines from training tasks, then applies them unchanged to held-out tests. It compares no guidance, a compact set of rules found across several training tasks, and every rule retained after consolidation. For GPT-4.1, those last two options contain 31 versus 292 retail rules, and 14 versus 101 airline rules.

Getting it right four times

The test, Sierra’s τ²-bench, pairs an agent with a model-simulated customer, domain tools and a written policy. Success requires the final database state to match the expected outcome, not merely a convincing answer. Each task runs four times. Mean accuracy averages those attempts; Pass⁴ measures the share of tasks completed successfully on all four. IBM’s held-out evaluation covers 40 retail tasks and 20 airline tasks.

More instructions were not always better. With GPT-4.1 on airline tasks, the compact 14-rule set achieved 45% Pass⁴, versus 40% with all 101 rules and 35% without guidance. Yet the longer playbook had higher mean accuracy: 65%, against 60% for the compact set. The configuration that succeeded more often overall was not the one that completed more tasks reliably across every attempt.

IBM also compares its agents with an October 12 leaderboard snapshot. Its GPT-4.1 retail agent scored 67.5% Pass⁴, above every listed retail entry; its GPT-5.5 airline agent reached 82%, also above the listed competitors. Those leaderboard figures use the full task set, including training and test tasks, rather than only the held-out tests above. They are a separate evaluation, not interchangeable evidence of performance on unseen requests.

The bill grows with the playbook

The extra instructions carry a measurable cost. In IBM’s GPT-4.1 experiments across the full task set, the all-rules configuration increased retail conversation costs from $0.084 to $0.139, up 65%. Airline costs rose from $0.081 to $0.107, up 32%. These totals include both the agent and simulated customer. They are benchmark conversation costs, not a quoted production price or a calculation of savings from fewer failed requests.

Longer conversations did not explain the increase. For matched runs that succeeded both with and without guidance, no guideline configuration increased assistant turns. The all-rules version shortened conversations by roughly 0.3–0.4 turns in both domains. IBM attributes the added expense to longer input prompts. Its experiment therefore presents a specific tradeoff: better repeated-task performance through more instructions, with higher measured conversation costs rather than more back-and-forth.