Loading page…
Identical updates drew different checks depending on their version numbers. The controlled experiments exposed failures, but used small apps and deliberately mislabeled changes.
Listen to this story
Microsoft’s November 9 experiments found that some coding agents treated SemVer labels as a cue to reduce scrutiny, even when patch- and major-labeled updates contained identical breaking code. The effect varied by model: Claude Sonnet 5.5 and Claude Haiku 5.5 checked or reviewed patch-labeled changes less often, while several others checked every update; only Haiku’s review result met Microsoft’s predefined threshold. Because mislabeled updates can still break apps, Microsoft recommends requiring automated checks on every dependency branch rather than relying on an agent to decide when to verify.
In the first Vally test, Sonnet checked only 2 of 10 patch-labeled runs versus all 10 major-labeled runs, despite byte-identical library code.
Across 540 branch-review runs, Haiku merged without inspecting the update in 11 of 30 patch cases and 2 of 30 major cases; this result met Microsoft’s threshold.
Agents merged 13 of 270 breaking updates; four left the main branch broken—two by Haiku and two by GPT-6 Luna—and all had patch or minor labels.
A version number changed how carefully some AI coding agents examined identical software updates, Microsoft reported on November 9. In two experiments, certain models checked less when an update looked like a routine patch. Others checked every time—a contrast that matters when a harmless-looking label hides code that breaks an app.
The report, by Microsoft principal developer advocate Waldek Mastykarz, examines Semantic Versioning, or SemVer. This convention uses version numbers to signal the intended scope of a release: patches fix bugs, minor releases add compatible features, and major releases introduce breaking changes. The question was whether agents would follow that signal instead of inspecting the software.
Microsoft ran both experiments through its Vally evaluation tool, using GitHub Copilot CLI as the agent. The first used a small Node.js app and a fictional library called fluxkit. Making up the library prevented models from relying on remembered details about its interface or release history.
The update renamed a function the app used, so changing the dependency without fixing the app would break it. Agents received a minimal request to upgrade to either version 1.4.3 or 2.0.0. Both versions contained byte-identical library code. Microsoft counted a check when the agent ran tests, ran the app, or read the updated library code.
Claude Sonnet 5.5 checked in just two of 10 patch-labeled runs, versus all 10 major-labeled runs. It often stopped after changing the patch dependency, leaving the app broken. Claude Opus 5.5, GPT-6.1 Sol and GPT-6.1 Terra each checked all 70 runs under each label.
The second experiment asked agents to review a prepared update branch and merge it if it was good. Microsoft used six scenarios, half breaking and half compatible, with identical code assigned patch, minor and major labels. Six models received 30 runs per label, totaling 540 runs. Version numbers appeared only in branch names and commit messages.
Claude Haiku 5.5 merged without checking the updated version in 11 of 30 patch-labeled runs.
The same model merged without checking the updated version in two of 30 major-labeled runs.
Haiku’s difference cleared Microsoft’s predefined test for an effect. Before testing, the researchers set a 25-percentage-point threshold and used confidence intervals to classify results. Sonnet 5.5 and GPT-6 Luna also merged unchecked more often under patch labels, but their results were inconclusive. Opus 5.5, GPT-6.1 Sol and GPT-6.1 Terra checked every update again.
Skipping a check was not the same as causing damage. Across the review experiment, agents merged 268 of 270 compatible updates and 13 of 270 breaking updates. Four breaking-update merges left the main branch broken: two by Haiku and two by GPT-6 Luna. All four carried patch or minor labels.
Microsoft deliberately violated the versioning convention. Its report cautions that most real patches are correctly labeled, so these results describe how agents handle exceptions. The apps were small, and updates lacked release notes that agents might otherwise read. The experiments also did not compare SemVer with another versioning scheme.
The resulting recommendations focus on verification rather than abandoning version numbers. Microsoft advises library maintainers to keep labels honest and teams using agents to make checks part of the workflow, rather than leaving their depth to the model’s judgment:
Loading discussion...
Join the conversation
Explain when, if ever, a routine-looking update deserves less scrutiny.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.