Anthropic Rebuilds Its Test System After AI Coding Drives 25-Fold CI Growth

Anthropic says it rebuilt a critical test-selection service after its continuous-integration, or CI, job volume rose 25-fold in six months. The pressure came as the company’s test suite grew tenfold and engineers shipped far more code with Claude, turning the system that decides which tests to run into a potential constraint on development.

CI is the automated checking stage that runs tests when a proposed code change is opened. Anthropic uses a test-impact analysis service rather than running every test on every change: a listener records results from CI runs, while a selector uses prior results and package relevance to choose tests for a new pull request.

That system was originally a single process. Because it maintained a running history for each test, it relied on one writer and could not be split across multiple machines. As traffic climbed, the listener fell behind; Anthropic says 20 minutes of lag could leave tens of thousands of test-result updates unapplied, leaving the selector to work from stale information.

Three short-lived fixes made the case for replacement

Anthropic first doubled the service’s processing capacity. It later split package state across workers, then tried daily restarts as memory pressure mounted. The company says those measures lasted about 70 days, 29 days, and less than a day, respectively. Restarts also gradually pushed the service further behind.

The operational consequence was not that CI stopped running or untested code went to production, Anthropic says. Instead, missed listener results meant the selector could use outdated data when choosing tests, which could cause it to include broadly failing or flaky tests and delay the use of repaired or newly added ones.

A journal separates fast intake from test history

The replacement removes the single writer from the busiest part of the path. Any listener worker can take a test result, append it to a journal in an in-memory data store, and move on. A separate consumer then rolls those entries into per-test history every few seconds, allowing the listener workers to scale horizontally while the selector can still retrieve a concise history.

What changed in the redesign

• Listener workers no longer retain the test history locally, so any worker can process a result.
• A journal buffers incoming results in the in-memory store.
• A separate consumer updates the per-test history that the selector reads.

Anthropic says one engineer completed the redesign in three weeks, compared with an estimate of roughly a quarter under its earlier development process. Claude generated code for an intermediate redesign and handled much of the later tuning, including adjustments to the journal and number of workers.

Stable backlog, higher operating cost

After cutover and tuning, Anthropic says the queued job-result backlog became flat rather than growing week over week, and the service has remained stable. The company also says the distributed architecture costs more to run. The reported gain is therefore not simply faster code production; it is a decision to spend more on the infrastructure needed to check that output.

Anthropic attributes the CI surge partly to Claude producing smaller pull requests, which create more test cycles, and to agents working outside normal hours while humans still direct and approve a significant share of changes. Its advice to other teams is to plan for infrastructure loads 10 to 20 times larger than their initial estimate and potentially 25-fold CI traffic within two quarters, budget permitting. That is guidance from one company’s internal experience, not an independently established forecast for every software team.