COBOL Migration at 97.9% Accuracy? Why Banks Should Not Trust OpenAI's Number Alone

OpenAI says GPT-6.1 Sol reaches 97.9% accuracy on its COBOL migration evaluation at high reasoning effort. That is a major improvement on the same company’s earlier models, but it is not permission to convert a bank’s production system and trust the generated code.

The evaluation is deliberately demanding. A model must migrate COBOL programs into C++, Java, Kotlin, Python or Rust while preserving observable behavior. Hidden tests require byte-for-byte parity, and a migration counts as successful only when every test passes. Byte-for-byte parity is a stricter bar than it may sound: it means the translated program must not merely produce a similar answer, but the same answer, in the same form, under the same inputs. A single changed digit, a reordered record field or a differently formatted date is enough to fail.

That makes the result more useful than a subjective coding demo. It still measures a bounded benchmark designed and reported by OpenAI. A production mainframe includes undocumented business rules, batch schedules, external files, database behavior, operational procedures and decades of exceptions that no benchmark can automatically capture.

What OpenAI measured

OpenAI published two related evaluations for low-resource programming languages. MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in 20 languages. A separate COBOL migration evaluation tests behavior-preserving translation of existing programs. The two tasks probe different skills: building a system from a specification tests synthesis, while translating an existing program tests comprehension and fidelity. The second is closer to the work that dominates enterprise maintenance budgets.

On MultiLangBench, GPT-6 Astra averaged an 87.4% test pass rate across the 20 languages, up from 26.8% for GPT-5.6 Sol. Its COBOL result was lower than the overall average at 72.1%, which is a useful reminder that the headline capability varies by language. A model that performs well across a broad set of languages can still lag on the one that matters most to a particular organization, and the gap between an average and a single-language score is exactly where procurement decisions should slow down.

On the migration evaluation, GPT-6.1 Sol reached 86.6% accuracy at low reasoning effort and 97.9% at higher effort. OpenAI says the lower-effort configuration also cost less than GPT-5.6 Sol on the same task. That cost detail matters as much as the accuracy figure. Migration work is repetitive and voluminous, so token economics determine whether a model is used on a hundred programs or ten thousand. A cheaper configuration that still clears a high bar changes the arithmetic of a modernization program.

These results suggest that current models are better at reading unfamiliar code, preserving semantics and using tests to repair mistakes. The repair loop is the mechanism worth understanding: a model that can run tests, read failures and revise its output behaves differently from one that emits code in a single pass. It can iterate against evidence rather than guess. OpenAI also released the MultiLangBench setup and results, allowing developers to inspect and reproduce that portion of the work.

Why 97.9% is not a production safety rate

Accuracy in this evaluation means the migrated benchmark program passed its hidden tests. It does not mean 97.9% of every line is correct, nor that 97.9% of real migrations will be safe. The unit of measurement is the whole program under a fixed test suite, not the individual statement, and the two are easy to confuse when a number is quoted in a planning meeting.

The denominator matters. A compact benchmark can cover tricky language behavior while still omitting the scale and integration density of a bank, insurer or government system. Real applications interact with databases, message queues, job schedulers, proprietary file formats, transaction monitors and manual operating procedures. A benchmark program can be understood in isolation; a production system cannot. Its correctness is partly a property of everything it touches.

More importantly, tests only protect behavior they express. A legacy system may contain rules known only to experienced employees. Some apparently strange behaviors are deliberate responses to past incidents. Others are bugs that downstream systems now expect. An AI can preserve the wrong behavior perfectly or modernize away an undocumented requirement. Both failure modes look like success in a test report, because the test report encodes only what someone thought to check.

OpenAI acknowledges this boundary. Its announcement says production use still requires expert review and testing against the system’s business rules and dependencies. That caveat is not boilerplate. It locates the remaining risk precisely where benchmarks are weakest: in the requirements that were never written down.

COBOL semantics can fail quietly

OpenAI highlights fixed-point arithmetic as an example. COBOL may discard excess decimal digits unless rounding is explicitly requested. A translation that rounds instead can compile, run and produce plausible financial values while changing the result. Nothing throws an exception. No test written from the source alone necessarily notices. The number simply drifts, and the drift compounds across interest calculations, reconciliations and ledger entries.

That is the dangerous category of migration bug: not a crash, but a subtle semantic difference. Similar risks appear in date handling, record layouts, character encodings, file ordering, transaction boundaries and missing-value behavior. Each is a place where the source language carries an assumption the target language does not, and where a fluent translation can be confidently wrong.

A useful migration process therefore needs more than unit tests generated from the source. It needs golden outputs from production-like workloads, reconciliation against historical runs, domain-expert review and explicit tolerances for every financial field. Generated tests inherit the model’s understanding of the program; golden outputs inherit reality. The distinction is the difference between checking that code does what it says and checking that it does what the business has actually been doing.

A safer way to use the capability

The strongest near-term use is not autonomous replacement. It is accelerated modernization with a controlled verification pipeline.

First, inventory the system and its dependencies before generating target code. Second, preserve a runnable reference environment. Third, build characterization tests from real inputs and outputs. Fourth, let the model propose migrations in bounded modules. Fifth, compare results at byte, record and business-metric levels. Finally, run both systems in parallel before any cutover.

Each step exists to shrink the blast radius of a mistake. Bounded modules keep failures local. A runnable reference environment keeps the old behavior observable. Parallel running turns an argument about equivalence into a measurement. None of these steps is novel to software engineering, but the model changes their cost, and cheaper verification is what makes a broader migration economically rational.

Human experts should focus on the places where generated migration is least trustworthy: arithmetic, data conversion, transaction handling, error recovery and interfaces that are not represented in the code repository. These are the areas where the code underdetermines the behavior, and where a reviewer’s institutional memory outperforms any test suite the model can generate.

The model can reduce the cost of writing candidate code and tests. It cannot decide which behaviors the business is legally or operationally required to preserve.

The broader significance

Low-resource languages have less public code and documentation available for training than Python or JavaScript, yet they support critical infrastructure. Progress here expands the market for coding agents from greenfield development into maintenance and modernization budgets. That shift is significant because maintenance spending dwarfs new-build spending in most large organizations, and it has historically resisted automation precisely because the code is unfamiliar and the specifications are incomplete.

It also changes the economics of migration. Organizations may be able to test more modernization options, translate smaller modules earlier and document systems that were previously too expensive to analyze. Documentation may turn out to be an underrated output: a model that can explain a COBOL routine in plain language gives a team something it may never have had, even if the routine itself is never rewritten.

But the 97.9% number should start a proof of concept, not end a risk review. OpenAI has shown that benchmark COBOL migration is approaching saturation under its test conditions. Banks still need to prove that generated replacements preserve the rules their customers, regulators and accounting systems depend on. The benchmark answers whether the model can translate code. It does not answer whether the organization knows what its code is for.