OpenAI releases coding benchmark and reports sharp gains in COBOL

OpenAI released a coding benchmark on October 19 and reported large gains in programming languages with relatively little training material. Its results include a 72.1% COBOL pass rate for GPT-6 Astra. A separate migration test asks a more exacting question for older software: can AI rewrite a program without changing the business rules hidden inside it?

One database task, 20 languages

OpenAI calls languages such as COBOL, Fortran and Visual Basic “low-resource” because models have less code and documentation to learn from than for Python or JavaScript. Developers and enterprise customers told the company that coding models often struggle with them.

MultiLangBench gives each language the same assignment: build a database engine compatible with SQLite. Over an extended coding session, the model must organize its code, manage state, handle errors and add features without breaking functionality it has already built. The evaluation records both test pass rates and API costs.

The comparison works through database files rather than language-specific interfaces. Each candidate program receives an initial database and a SQL query, then produces a modified database. The reference SQLite implementation checks whether the resulting contents are correct. OpenAI adapted tests from SQLite’s existing suite, allowing the same checks to run across all 20 languages.

COBOL remains the hardest language

Astra improved over GPT-5.6 Sol in every language tested, according to OpenAI. Its COBOL result rose from 10.9% to 72.1%, but COBOL remained the toughest assignment. Astra exceeded 83% in each of the other 19 languages, including 85.8% in OCaml, 83.5% in Fortran and 89.4% in Visual Basic.

The smaller GPT-6 Luna also beat GPT-5.6 Sol in every tested language at maximum reasoning effort. OpenAI’s figures show a different balance between performance and spending:

• Performance: Luna averaged a 40.3% pass rate, compared with 26.8% for GPT-5.6 Sol.
• Estimated API spending: Luna cost $0.77 per sample versus $3.48 for GPT-5.6 Sol, approximately 78% less.

A rewrite must keep the old arithmetic

The second evaluation tests a different skill: translating existing COBOL programs into C++, Java, Kotlin, Python or Rust while preserving their behavior. That requires understanding both business logic and language-specific rules, not simply producing code that runs.

For example, COBOL’s fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested. A rewrite that rounds instead can run without errors yet produce different financial results. Missing-record handling can also encode business requirements that a migration needs to retain.

OpenAI checks observable outputs byte for byte using hidden tests. A migration counts as successful only if every test passes. On that measure, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort, versus 5.4% for GPT-5.6 Sol at the same setting, while using 39% less API spending. At higher reasoning effort, Sol reached 97.9%, OpenAI says.

OpenAI cautions that production migrations still require expert review and testing against a system’s business rules and dependencies. It describes a maintenance problem in which documentation may be incomplete, original developers may have left, and important rules may exist only in code.

The company says its MultiLangBench release includes the evaluation setup, test suite and per-language results on GitHub, so developers can inspect the tasks and identify gaps relevant to their work. It also invites developers to try its latest models through the API and provide feedback on languages and workflows that still need improvement.