OpenAI reveals internal AI system for investigating ChatGPT slowdowns
Performance Factory coordinates investigations, proposed repairs and recovery checks. OpenAI’s early results show a wide gap between diagnosing problems and delivering fixes.
Loading page…
Performance Factory coordinates investigations, proposed repairs and recovery checks. OpenAI’s early results show a wide gap between diagnosing problems and delivering fixes.
Listen to this story
OpenAI’s Performance Factory turns performance alerts into persistent Codex-led investigations that can continue across agent sessions, pause for human review or deployment, and check whether a fix worked. In its first three months, it supported more than 1,700 investigations and helped move roughly 50 fixes that were influenced by or matched agent suggestions into production; about 15 were exact Codex-authored pull requests. Engineers still choose what ships, and the results show a wider gap between diagnosing a slowdown and producing a repair that is safe to deploy.
OpenAI says the system identified root causes in more than 600 investigations and advanced around 400 to proposed fixes.
In a manual review of 100 root-cause suggestions, 70% were graded accurate; a separate Codex review now returns about 25% of diagnoses for more investigation.
Tasks retain findings outside an individual agent’s context, run within OpenAI’s network, and use service identities to limit permissions and track actions.
OpenAI has disclosed an internal system that puts Codex agents to work investigating ChatGPT slowdowns, proposing repairs and checking recovery after deployment. Called Performance Factory, it tracked more than 1,700 investigations in just over three months, the company says. Engineers still decide which code changes ship—a boundary that matters in its early results.
In its November 19 engineering post, OpenAI says the system identified root causes in more than 600 cases. But finding an explanation is not the same as delivering a repair. The performance team told VentureBeat that around 15 shipped fixes were the exact pull requests—proposed code changes—authored by Codex.
The project grew out of an awkward handoff. Engineers had developed more than 30 performance tools, then captured investigation procedures in a reusable Codex skill called perf-forensics. Eventually, people were copying alerts into Codex and pasting findings into Slack. Moving investigations to the cloud removed that step, but left engineers to organize the follow-through.
Performance Factory turns each problem into a persistent task. Engineers define its stages and instructions in configuration files and Markdown prompts. A dispatch loop checks for ready work and starts a Codex session. The task retains findings outside any individual agent’s context, allowing another agent or an engineer to continue the investigation.
Those sessions run inside OpenAI’s network, with access to code, metrics, logs and other diagnostic tools. A service identity lets OpenAI limit permissions and track agent actions. Tasks can pause for a code review or deployment, then resume when a matching event arrives, rather than keeping an agent running while it waits.
The first running workflow begins with a configured performance alert. Codex establishes which metrics worsened, when the change began and which requests were affected. It also distinguishes genuinely slower requests from a shift toward serving more requests that were already slow. Those possibilities call for different explanations.
OpenAI found that agents sometimes accepted the first plausible cause too readily. It added adversarial review: a separate Codex session instructed to challenge the diagnosis. In a manual review of 100 root-cause suggestions, OpenAI graded 70% as accurate; the rest needed refinement or did not fully establish causality.
One investigation blamed failing ChatGPT conversations on image-download errors. The reviewer noticed that a file-location service had recovered while image requests kept failing, so it reopened the case. The performance team told VentureBeat that adversarial review currently sends back around 25% of diagnoses for further investigation.
OpenAI’s team says around 400 investigations progressed to a proposed fix.
Approximately 50 shipped fixes were influenced by or matched agent suggestions; around 15 were exact Codex-authored pull requests.
According to the performance team, Codex works best when the cause is clear and the change is contained—for example, removing repeated processing of the same conversation. Architectural changes are harder. An agent might propose a cache that speeds responses but leaves stale data, trading one problem for another. OpenAI has added adversarial review to fixes too.
Human review also requires usable evidence. OpenAI generates task summaries with the problem’s impact, suspected cause and proposed repair, alongside charts and traces. Engineers can edit the proposed code or write their own fix from the diagnosis. After deployment, another agent checks recovery; a persistent problem goes back for investigation.
OpenAI is exploring proactive workflows that compare performance across releases and identify inefficient autoscaling settings. For now, its performance team says automating regression investigations has freed engineers to spend more time on long-term improvements rather than continually responding to incidents.
Loading discussion...
Join the conversation
Explain how much credit useful diagnoses should earn.
Be the first to share a perspective or an experience.
Reader comments
Newest comments first. Replies stay oldest first.