OpenAI Puts COBOL Migration to the Test, and Its Newest Models Mostly Pass

OpenAI wants to show that AI can handle the code most developers would rather not touch.

On Oct. 19, the company published results from two new evaluations built around what it calls low-resource programming languages. These include COBOL, Fortran, OCaml, and Visual Basic. They have far less public code and documentation for models to learn from than Python or JavaScript. But they still run a lot of critical infrastructure.

OpenAI cites IBM figures showing COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions. Fortran is still used in weather forecasting and nuclear reactor safety analysis.

The company says developers and enterprise customers told it that these languages often struggle to support modern coding models. That matches what DevOps teams see. Documentation is thin, original developers are often gone, and important business rules may exist only in the code.

Two Tests, Two Skills

The first evaluation, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in one of 20 languages. The model has to organize a codebase, manage state, handle errors, and add features without breaking what already works.

To compare results across languages, OpenAI uses database files as a shared interface. The program receives a database and a SQL query, then produces a modified database. The reference SQLite implementation checks the result. The tests were adapted from SQLite’s own test suite.

The jump is large. At maximum reasoning effort, GPT-6 Astra averaged an 87.4% test pass rate across the 20 languages, up from 26.8% for GPT-5.6 Sol. COBOL went from 10.9% to 72.1%. Fortran went from 21.4% to 83.5%. Astra scored above 83% in 19 of the 20 languages. COBOL was still the hardest.

GPT-6 Luna is the budget option. It reached a 40.3% average pass rate at an estimated $0.77 per sample, compared with $3.48 for GPT-5.6 Sol. OpenAI has also released the MultiLangBench setup and test suite on GitHub, with results for each language.

The second evaluation more closely reflects the daily pain of mainframe teams. It asks models to migrate COBOL programs to C++, Java, Kotlin, Python, or Rust while keeping each program’s behavior intact. Hidden tests require byte-for-byte parity on observable outputs. A migration counts only if every test passes.

Here, GPT-6.1 Sol hit 86.6% accuracy at low reasoning effort. GPT-5.6 Sol scored 5.4% at the same setting, and the new model ran at 61% of the cost. At higher reasoning effort, accuracy reached 97.9%.

Why Rounding Is the Real Test

Anyone who has worked a COBOL migration knows the hard part isn’t syntax. It’s arithmetic. OpenAI’s own example makes the point. COBOL’s fixed-point arithmetic drops extra decimal digits unless you explicitly request rounding. A rewrite that rounds those values instead will run cleanly and still produce different financial results.

That’s not a theoretical problem. One engineer who worked on a utility billing conversion noted that Java’s double type is the wrong choice for currency math, and even BigDecimal can apply a different rounding mode than the original COBOL specified. Multiply a one-cent error across millions of records, and you have an audit finding.

And that’s where the benchmark and the real world part ways.

“OpenAI’s migration benchmark checks code against hidden tests that someone already wrote,” said Mitch Ashley, vice president and practice lead for CIO & Technology Buyers and Software Lifecycle Engineering at The Futurum Group. “A mainframe estate whose business rules live only in the code has no such suite, so building that test harness lands in the critical path of the migration project.”

Read the Fine Print

Other reasons to keep expectations in check. These are OpenAI’s own benchmarks, run on OpenAI’s own models. They haven’t been independently verified. OpenAI also says production use still requires expert review and testing against the system’s business rules and dependencies.

A benchmark program is also not a mainframe estate. Real migrations involve copybooks, JCL, data stores, batch schedules and integrations nobody documented. Amdocs argues that AI is improving modernization platforms, not replacing them.

Researchers see the same gap. An IBM Research team found that COBOL’s domain-specific syntax and the shortage of quality training data limit how well LLMs translate it. Microsoft engineers, describing their own agent-based COBOL migration work, said deterministic controls, especially tests, were key to validating what the agents produced.

What DevOps Teams Should Do Now

The lesson is simple. The model is only as trustworthy as the test harness around it.

Before translating any code, teams should capture real production inputs and outputs. They should define what “equivalent” means for each program, down to rounding and missing-record handling. Then they should run old and new code side by side against the same data before cutover. That turns migration into a pipeline problem, which DevOps teams already know how to run.

Cost changes the math, too. Mainframe modernization has long meant multi-year projects led by large systems integrators. Cheaper translation makes smaller, safer migration slices more realistic. But the savings don’t disappear. They move.

“Cheaper per-program translation shifts the budget to verification,” Ashley said. “Watch whether OpenAI’s results hold on independent tests against real production data, because that evidence decides how fast teams can cut over.”

AI is getting better at reading and rewriting old code. That part is real. But proving the new code does exactly what the old code did is still the job. And that job belongs to the team, not the model.