Specific Labs Launches Private-Code Benchmark Where Top Coding Setup Solves 38.8%

Specific Labs has released Real-SWE, a benchmark built from licensed private enterprise codebases and the production tasks engineers actually face. Its first leaderboard offers a restrained answer to whether coding agents can reliably handle that work: the highest-scoring setup, Fable 5.1 running through Claude Code, resolved 38.8% of tasks.

That score measures a model-and-harness combination, not the underlying model alone. Specific Labs used each system’s native coding tool, aiming to reflect how enterprise engineers use agents in practice. Resolution rate is equivalent to pass@1: the average first-attempt success rate across eight independent runs per task.

Real-SWE’s central design choice is its source material: private production codebases licensed from real companies. The tasks cover billing, taxes, customer migrations, and infrastructure changes—work where a patch may need to preserve existing behavior while fitting company-specific rules and systems.

Specific Labs says keeping code and solutions off the public internet reduces the chance that a benchmark task or its answer was already exposed in training data. The tradeoff is intentional: agents must inspect the workspace available to them, rather than rely on familiarity with a public repository or issue history.

The next two reported results were GPT-6 Astra with Codex CLI at 33.8% and Gemini 3.8 Flash with Gemini CLI at 31.2%. GLM 5.3 with Claude Code followed at 28.8%; the remaining four published combinations scored 23.8% or lower.

The benchmark’s tasks are not necessarily long requests. Real-SWE reported a median instruction length of 1,742 characters across its repository-backed sample tasks. But its reference solutions edited a median of 11 files, compared with six in the cited FrontierCode and DeepSWE comparisons.

What the benchmark asks agents to do

• Find the relevant implementation details inside an unfamiliar, proprietary codebase.
• Reconcile instructions with business logic and coding patterns already present in the system.
• Make changes that can cross code, infrastructure, and connected business tools.

Specific Labs’ published task analysis illustrates the unevenness. Multi-region sweep reached a 67.2% aggregate resolution rate, while tax jurisdiction reached 3.1%. Analytics stream reducer recorded no successful rollouts across the eight tested model-and-harness combinations.

Longer runs did not produce a clear advantage in the reported rollout data. Specific Labs found that 71.4% of rollouts under 10 minutes failed, versus 73.4% of those lasting 10 minutes or more. Its analysis identified missed requirements as the most common failure mode and said models often struggle with company-specific logic and conventions.

That makes Real-SWE less a declaration of a best coding model than a test of complete coding setups under private, operational conditions. The leaderboard can show which tested combination did better in this environment. It cannot separate the effects of model capability from the native harnesses that supply tools, context handling, and workflow structure.