IBM and Kaggle launch Java migration leaderboard with different leaders for partial and full success

IBM and Kaggle launched a public leaderboard on October 27 for coding agents that move Java applications between software frameworks. The initial evaluation covers 204 migrations and 10 agent configurations. Its results expose a practical divide: the setup earning the most test credit is not the one completing the most migrations without a test failure.

The leaderboard builds on ScarfBench, an open benchmark IBM introduced earlier in 2026. It asks agents to move working applications between Spring, Jakarta EE and Quarkus without changing their behavior.

The original research paper describes 34 expert-written applications: 29 focused on a single software layer and five whole applications. Implementations in each framework yield 102 variants and 204 directed migration tasks. Each assignment starts with a working source application and a target framework.

Validation requires compilation, deployment in the target runtime and tests of observable behavior. For the new leaderboard, Kaggle consulted on evaluation design, independently reproduced and ran the benchmark on its platform, and sponsored the computing resources, according to IBM.

IBM’s launch results put Gemini 4 Argon paired with Antigravity first on the primary smoke-test score, at 75.2%. That measure awards partial credit and gives larger test suites more weight. Claude Opus 5.5 with Claude Code placed second at 70.5%.

Counting only migrations that passed every test changes the leader. Opus 5.5 with Claude Code reached 72.1%, narrowly ahead of Argon with Antigravity at 71.1%. These are results for model-and-harness combinations, not models alone. A harness is the software that controls how an agent interacts with the code repository and development environment.

535 of 745Runtime behavioral failures

About 72% of IBM’s classified failures involved applications that built and often started, but failed behavioral checks.

• Wrong results: 304 runtime failures involved content or assertion mismatches, where the output failed an expected check.
• Missing routes: 210 involved lost routes or HTTP 404 errors, leaving expected web addresses unavailable.
• Request errors: 21 produced explicit HTTP 500 or 405 errors.

One example makes the gap concrete. In an address-book migration from Jakarta EE to Quarkus, GPT-6 Astra with Codex passed just two of 10 tests after list and create pages returned HTTP 404.

The next step is ongoing evaluation rather than a one-time ranking. Kaggle will maintain the benchmark and add models as they are released, IBM says. Public model submissions are not available at launch; they remain on the roadmap.