OpenAI Reports Major Gains in COBOL Migration and Multilanguage Coding

OpenAI’s latest coding benchmarks point to a striking jump in COBOL migration performance: GPT-6.1 Sol scored 86.6% at low reasoning effort, compared with 5.4% for GPT-5.6 Sol, while costing 39% less. At higher reasoning effort, GPT-6.1 Sol reached 97.9%.
Those figures could interest teams facing the expense and risk of updating legacy systems. But the COBOL evaluation remains private, so developers can’t inspect its test cases or independently reproduce the results. That gap makes the scores promising evidence, not yet a clear measure of how the model will perform on a company’s production code.
The size of the jump is hard to dismiss, though its practical meaning depends on what the evaluation asks a model to do. Migration work can involve more than rewriting syntax: teams need to understand existing behavior, dependencies, and the assumptions that have accumulated around an application. A benchmark can test parts of that work, but its result can’t stand in for the full process.
What the COBOL score measures
OpenAI says its evaluation tests whether a model can translate existing COBOL programs into C++, Java, Kotlin, Python, and Rust while preserving their behavior. Submissions must pass hidden tests requiring “byte-for-byte parity on observable outputs.”

That’s a demanding target for a language with less public code and documentation available for model training than languages such as Python or JavaScript. Experienced COBOL developers can also be difficult to find, and system documentation may not reflect how older applications actually behave.
In this setting, matching observable output is a useful standard because it focuses attention on what a program does, not just whether a translation looks plausible. But passing a hidden test suite tells outsiders little about which behaviors were tested, how varied the programs were, or where the evaluation draws its boundaries. (And this is the part most people overlook.) A high score can signal progress without answering those questions.
The stakes are broad. IBM reports that COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions. Yet a high score on a private test can’t establish whether a migration will preserve the business rules and dependencies embedded in any particular system.
That distinction matters because production systems often have requirements that are specific to the organization running them. Teams need to account for the behavior they rely on, including behavior that may be poorly documented. The benchmark’s reported results don’t show how much work would remain to identify and verify those requirements in a real migration.
MultiLangBench opens a window
OpenAI has made its MultiLangBench setup and test suite available on GitHub. The benchmark asks models to build a SQLite-compatible database engine from scratch in 20 programming languages. Given a database and SQL query, the program must produce a modified database that matches the output of the reference SQLite implementation.

GPT-6 Astra averaged an 87.4% test pass rate, up from 26.8% for GPT-5.6 Sol. Astra exceeded 83% in 19 of the 20 languages and improved on every language tested. GPT-6 Luna, OpenAI’s lower-cost option, averaged 40.3% but also beat GPT-5.6 Sol in all 20 languages, with an estimated API cost about 78% lower.
The public materials make these results easier for other developers to examine. They can inspect the tasks and test suite, then compare model behavior against a shared target. That improves the quality of the discussion, though it doesn’t make the benchmark a direct forecast of how well a model will handle every codebase or engineering workflow.
The gap between Astra and Luna also points to a practical choice for teams: the highest-scoring model may not be the only one worth evaluating. Cost and performance can matter differently across tasks, and a team’s own results may not mirror a benchmark average. If you’re working in this space, the useful question is how a model performs on representative work your team can review.
These results are easier to assess than the COBOL scores because the MultiLangBench materials are public. Still, benchmark performance is only one signal. Real-SWE 1.1, which evaluates models against private codebases, listed Claude Opus 5.5 at 56.8% resolution as of Thursday. A previous Real-SWE run had a different leader and lower scores, a reminder that results depend on the test and its underlying tasks.
That’s why a score from one benchmark shouldn’t be treated as a universal ranking. Different tests emphasize different kinds of work, and private codebases introduce questions that a public task suite may not capture. The numbers here are striking, but the comparison has limits.
Production work still needs expert review
OpenAI says developer and enterprise customer feedback helped shape its work, but it hasn’t released the COBOL test suite or dataset. Without them, outsiders can’t judge how representative the migration tasks are or verify the reported accuracy.
The company also cautions that production use requires expert review and testing against each system’s business rules and dependencies. That qualification matters: a translation can pass output tests and still need careful validation against the assumptions built into a long-running application.
Expert review is part of how teams can connect a benchmark result to their own operating requirements. The model may help produce a translation, but the organization still has to determine whether that version behaves as expected in the contexts that matter to it. A test result can narrow uncertainty; it can’t remove the need to check the work.
OpenAI is joining an active effort to apply AI to mainframe modernization. Anthropic described ways Claude Code could assist with COBOL work on February 23, and AWS introduced AI agents for mainframe migrations in 2024. For engineering teams, the practical next step is to treat these scores as a reason to evaluate models on their own code and rules, while waiting for enough public detail to compare COBOL results directly.
What the results mean for migration teams
Here’s the thing: a dramatic benchmark gain can be a meaningful signal even when it doesn’t settle the production question. The COBOL result suggests that model capability may be improving quickly on the tasks OpenAI measures. The private evaluation, however, leaves an important part of that claim difficult to scrutinize.
For teams weighing modernization, the sensible response is to use benchmark results to guide evaluation, then test models against code and requirements they understand. That work can show where a model helps and where developers need to intervene. It can also reveal whether the cost difference matters for the team’s own workload.
This is more significant than it looks, but the evidence is uneven. MultiLangBench gives outsiders material they can inspect; the COBOL score currently doesn’t. Until more detail is available, the most useful reading is that OpenAI is reporting a large improvement on its evaluation, while the result’s reach beyond that test remains uncertain.
Potentially useful. Still unproven in your system.
The post OpenAI’s COBOL migration score has made a striking leap appeared first on The New Stack.