Microsoft finds 15 tool and documentation problems despite working AI scripts

Five AI-generated scripts worked, yet the steps behind them exposed 15 product and documentation problems. In a November 5 post, Microsoft developer advocate Waldek Mastykarz described a Dev Proxy experiment that uncovered wasted requests, fragile workarounds and an overlooked system setting—problems that checking only whether the scripts worked would miss.

Microsoft ran the same assignment five times in GitHub Copilot Chat using Claude Sonnet 5.5, without extensions. The task was to write a continuous integration script—code for an automated build-and-test workflow—that used Dev Proxy to record which GitHub APIs a Node.js tool called. Dev Proxy intercepts an application’s web requests.

Success concealed the detours

Mastykarz examined each agent’s trajectory: the record of its steps and stated reasons for taking them. Those records showed agents repeatedly recovering from gaps rather than following a clear route. Across four runs, eight documentation fetches returned 404 errors. Two runs spent five to seven extra steps locating the correct API page; two abandoned the documentation.

The interface also invited trial and error. One request for a root certificate failed because it lacked a required format field. The error named the missing field but not its allowed values. The agent tried pem, crt, cer and der; only crt worked.

Other gaps produced homemade substitutes. With no command to list plugins, all five runs read a plugin library as raw bytes and searched for class names. Dev Proxy also gave no completion signal when reports finished writing. Four runs repeatedly checked the output folder; the fifth simply waited two seconds, a delay Mastykarz warned would eventually fail on a busy runner.

A system setting outside the score

The most consequential finding concerned Dev Proxy’s default behavior: it makes itself the system proxy, routing the machine’s traffic through it. Its help text sounded like an option to enable, rather than something already enabled. Disabling it required an explicit false value that the help did not explain.

Three runs never noticed that default. Mastykarz said their scripts would reroute all traffic on a Windows automation runner, while their fallback for terminating the process would skip cleanup. All three passed because system-proxy behavior was outside the evaluation criteria. Meanwhile, the automated judge wrongly failed a run that disabled it.

Even corrected documentation did not always reach the agent. One run found search snapshots omitting the API token required since Dev Proxy 3.3.1. Results included a translated reference and an agent-instruction file that had already been updated six weeks earlier. That run handled authentication correctly because it had read the installed command-line help.

Fix the shared path first

Mastykarz’s recommendation is to improve products and documentation before shipping an agent skill, a package of extra instructions and resources. Skills require developers to find, install and maintain them. A correction in the tool or its documentation reaches agents without that extra installation.

• For guessed documentation paths, add redirects and print the documentation URL in command-line output.
• For repeated workarounds, add discovery commands and a signal that delayed operations have finished.
• For ambiguous settings, state defaults and explain how to disable them, especially when they affect the wider machine.

The experiment covered five repetitions of one task with one model setup, not a broad comparison of coding agents. Mastykarz also cautions that trajectory findings remain hypotheses: teams should rerun the same tasks with and without a proposed fix, then compare quality and cost to establish whether it actually helped.