GitHub previews AI checks to block passwords before code reaches repository history

GitHub says its smaller AI model cuts secret-scanning costs by more than 98%

GitHub says a smaller, purpose-built AI model is catching leaked credentials more accurately while costing far less to run. In technical results published October 13, the company reported more than 98% lower serving costs than its previous large-language-model pipeline. The new scanner combines fast pattern matching with a model that reads surrounding code.

The difficult targets are generic secrets: passwords and other credentials without a distinctive identifying pattern. A string that looks sensitive might instead be a checksum, a build identifier or a documentation example. Even the same password-shaped value can mean different things beside a production database address and inside a test file.

GitHub keeps regular expressions—rules that match text patterns—as the first filter. They generate a broad set of possible credentials. A fine-tuned ModernBERT classifier then scores each candidate using its value, file path, secret type, variable names and nearby code or text. High-confidence findings become alerts; likely false positives are filtered out before developers see them.

The model classifies rather than generates prose. GitHub chose that design for a narrow job requiring fast decisions over short windows of context. Training focused on confusing lookalikes, not just obvious credentials. Its examples came from three sources:

Product-derived repository examples, combining feedback from secret detection with additional human validation.

Repository candidates collected through broad pattern matching, with human annotations and less definitive labels used to expand coverage.

Synthetic examples covering sparse or difficult cases, including placeholders, test fixtures, public keys, checksums and near-matches. These passed rule-based validation and model-based quality review.

Before separating training and evaluation data, GitHub grouped rows sharing a repository, candidate value or example to keep near-duplicates out of both sets. It selected decision thresholds through offline tests, then checked behavior on production-like traffic before wider rollout. For deployment, it exported the model to ONNX, a model format, with 8-bit quantization for faster inference.

GitHub compared the scanner with its previous LLM pipeline in two stages. In an offline evaluation on the same public traffic, it reported 13% higher precision and 19% higher recall. Precision measures how many flagged findings are real; recall measures how many real secrets the scanner catches. Those results address both distracting alerts and missed credentials.

The customer experiment measured acceptance of findings, rather than repeating the offline accuracy test. GitHub weighted acceptance by repository owner to limit the influence of customers who resolve large batches of alerts at once. The cost comparison also has a specific scope: serving the compact classifier versus making remote LLM calls, not the total cost of GitHub's security products.

The speed improvement made generic-secret detection practical for push protection, which checks changes before GitHub accepts an upload. In its October 7 rollout announcement, GitHub said the classifier evaluates candidate batches in under two milliseconds. That is a batch-classification figure, distinct from the full post-push scan execution measurement published October 13.

That announcement placed push protection in private preview, with broader availability planned later in October for Secret Protection organizations on Enterprise Cloud and GitHub Teams. Those checks will consume AI credits. Organizations already using AI secret detection were set to receive the model automatically, with post-push alerts included in their secret-scanning purchase at no additional cost.

GitHub is also adding the classifier to security-review workflows in Copilot CLI and Copilot App. Its next research step is more ambitious: investigating models that detect and extract secrets in one pass. That remains an exploration; the deployed scanner still relies on separate pattern-matching and classification stages.