Open-source maintainers fixed 55 security issues after receiving checked, AI-generated findings with suggested repairs in a GitHub and Microsoft pilot. The companies published the results on November 17: thousands of potential issues became 99 private reports across six projects. As of November 15, 14 of those fixes had reached a release.
The experiment tackled a workload problem, not a lack of discoveries. GitHub says maintainers face a flood of AI-generated security reports, including duplicates and findings that are not real vulnerabilities. Each report still requires someone responsible for the software to decide whether it needs attention. The pilot aimed to take that first round of sorting off maintainers’ hands.
Six projects agreed to participate: AutoGPT, Caddy, Express, FastMCP, ImageMagick and Netty. All were alumni of the GitHub Secure Open Source Fund. Their maintainers consented to the scans and retained control over which issues to fix, how to repair them and when to act.
Before a finding reached a maintainer
Microsoft’s Codename MDASH security-scanning system used its MAI Cyber model to inspect the projects and propose fixes. But the scanner’s output was only the starting point. Two more review stages stood between an AI finding and a report in a maintainer’s inbox.
- AI filtering: A GitHub Security Lab agent, built with tailored Copilot skills, removed duplicates, checked validity and prioritized high-impact bugs.
- Human verification: Seven Microsoft FORGE Lab researchers manually checked every finding left after that filtering.
- Private delivery: Maintainers received reports through GitHub’s private vulnerability reporting, each with a proof of concept demonstrating the issue and a suggested fix.
That process reduced thousands of possible issues to the 99 reports delivered to maintainers. It also kept reports and repairs on GitHub, where the projects already worked. The suggested patches were starting points rather than instructions: maintainers used some unchanged and modified others.
Finding vulnerabilities is no longer our bottleneck. Fixing them is the hard part, and that depends on maintainers.
Netty maintainer, quoted by GitHub
From suggested patches to shipped repairs
GitHub reported that the first fixes shipped about a week after the initial scan. Caddy’s maintainers fixed all 22 reports within a week or less. At AutoGPT, a report arrived three days after maintainers shipped a broken check. It identified the relevant line. The team wrote its own repair within a day and put it into production less than a week later.
ImageMagick’s two-person maintenance team also received 22 reports. One initially resembled a familiar issue, but its summary explained the difference and identified the exact code involved. The maintainers pushed a fix in 15 minutes. An ImageMagick maintainer told GitHub that reading, understanding and fixing a security advisory normally took roughly 30 minutes to an hour.
The review work still needs to scale
Those examples come from a small, six-project pilot, which GitHub describes as early lessons. The results also depended on substantial human effort: seven researchers spent about two days per project checking findings by hand. GitHub acknowledges that this careful review process does not yet scale.
GitHub also identifies practical gaps beyond report quality. Maintainers need to run automated tests on private copies of their projects to check proposed repairs. It argues they also need access to security-capable models to verify demonstrations of vulnerabilities before and after a fix.
The next commitment reaches beyond the original six participants. GitHub says it will offer every Secure Open Source Fund alumni project AI security scans with findings checked before private delivery, and extend that support to future Fund sessions. It is working with Microsoft to scale researcher review without dropping that verification step.
Reader comments
Newest comments first. Replies stay oldest first.