Perplexity Finds 21% Fewer Tool Failures Between Trained Agent Versions

Perplexity has trained its Computer agent on mistakes from real user sessions, including sessions that failed. In a live test, a later trained version made 21.2% fewer failed tool calls than an earlier one. The improvement is measurable, but the test found no statistically significant drop in strong user dissatisfaction.

Why a failed session can still teach

The study tackles a blind spot in a common training approach: keep successful sessions and teach the model to repeat them. An agent can finish a task despite a bad intermediate step, so copying every action may preserve mistakes. Throwing out a failed session discards evidence of where the agent went wrong.

Perplexity separates actions worth imitating from errors worth correcting. It trains on non-error steps from successful sessions and on mistaken steps with validated corrective hints, even if the session failed. Other steps remain in the history for context but do not become training targets.

A correction the agent must remember

Hints must rely on information available before the mistake. A preference revealed for the first time in a later complaint cannot fairly become an instruction the agent should have followed earlier. Automated judges also trace complaints to the responsible step; Perplexity found that the last assistant step was the cause only about half the time.

The correction uses two passes through the same GLM 5.2 model, not a separate, stronger teacher. One sees the hint; the other sees only the original session. Training pushes the second pass toward the first pass’s predictions, so the agent can make the better choice later without a hint.

One search call used “year” for a recency setting that accepted only “day,” “week” or “month.” Its hint pointed to the validation error and an allowed choice. Before training, hints helped the unchanged base model avoid the original failure on 93.7% of 985 held-out tool-error turns, up from 75.1% without them. That test did not measure what the model retained.

The result stops short of happier users

The later checkpoint was not compared directly with the original GLM 5.2 model online. Tool-call failures fell from 2.24% to 1.77%, but strong dissatisfaction moved only from 2.58% to 2.54%, a difference that was not statistically significant. Fewer failed calls cannot be read as proof of better user satisfaction.

Offline tool-error rates were 2.79% for stock GLM 5.2, 1.35% after training on successful examples alone, and 0.87% with the added correction method. But those checkpoints used different training data, so the figures cannot isolate the method’s contribution. Task-level benchmark results were mixed.

What others can inspect

Training used eligible Computer sessions served by GLM 5.2, excluding sessions with personally identifiable information and those from users who opted out. Perplexity has not released the post-trained weights or training code. Although the base model is openly available, the trained version is available only inside Perplexity Computer.