AI can rewrite the old system now. Make it prove the numbers match.

Someone is about to offer to translate the whole thing

If you look after an old system, the pitch is probably on its way. Maybe it's a county permit and fee system written in COBOL before most of the current staff were hired. Maybe it's the Visual Basic and Access app that runs invoicing at a family distributor, and only one person still knows how to change it. Both of those are invented, but you likely know one just like them.

The pitch goes something like this: AI can read the old code now, so we'll translate the whole thing into Java or Python, quickly, for a fraction of what a rewrite used to cost.

On October 19, OpenAI gave that pitch its strongest evidence so far. It published results showing its newest models are much better at building and migrating software in what it calls low-resource languages, such as COBOL, Fortran and Visual Basic, where there's far less code around for a model to learn from. And it isn't alone: Crypto Briefing points out that Anthropic released COBOL-related updates in February and IBM sells watsonx Code Assistant to translate COBOL into Java.

So the real question isn't whether AI can translate your system. It's how you'll know the translation is right before you switch. That's what this note is about, and it ends with a message you can send to whoever makes you the offer.

Read what OpenAI actually measured

OpenAI built two tests, and they're worth understanding before anyone quotes them at you.

The first, MultiLangBench, asks a model to build a SQLite-compatible database engine from scratch in a given language, checked against tests adapted from SQLite's own suite. Across 20 languages, GPT-6 Astra averaged a test pass rate of 87.4%, up from 26.8% for the older GPT-5.6 Sol. COBOL went from 10.9% to 72.1% and is still the hardest language in the set. Fortran went from 21.4% to 83.5%, and Visual Basic from 27.5% to 89.4%. OpenAI is releasing the test suite and per-language results on GitHub.

The second test is the one that matters if you own an old system. Models were asked to migrate COBOL programs to C++, Java, Kotlin, Python or Rust, then graded on hidden tests that required byte-for-byte identical output. A migration only counted if every single test passed. GPT-6.1 Sol managed that 86.6% of the time at low reasoning effort, against 5.4% for GPT-5.6 Sol, and OpenAI says it cost 39% less to run. At higher effort it reached 97.9%.

Those are a vendor's numbers on tests the vendor built, so treat them that way until someone else reproduces them. Still, the direction is hard to argue with, and OpenAI was careful about the limit. Its post says that putting this to work on production software "still requires expert review and testing against the system's business rules and dependencies."

That sentence is the whole job. The rest of this note is about how to do it.

Notice what the test had that your system doesn't

Look at that grading rule again: hidden tests, byte-for-byte output, all of them must pass. It's a very good rule, and a luxury: somebody wrote those tests and knew every right answer before the model started.

Your system almost certainly doesn't come with that. The right answers live in last month's output files, in reports people have read for years, and in the heads of the people who can spot a wrong number at a glance.

OpenAI's own example shows why this matters. COBOL's fixed-point arithmetic drops extra decimal digits unless the program explicitly asks for rounding. A modern rewrite that helpfully rounds those values looks tidier and runs cleanly. In OpenAI's words, it "can run without errors while producing different financial results." That's a sliver of a cent on every fee or invoice, and nobody notices until the totals stop agreeing with the bank.

Rounding is just the famous one. In any old system, we'd also look hard at:

• Dates. Two-digit years, fiscal calendars, end-of-month rules, and the day a cutoff actually falls on.
• Blanks, zeros and missing records. OpenAI flags missing-record handling too. An old program might treat a blank as zero, skip the record, or stop the run. The new one has to do the same, or someone has to decide it shouldn't.
• Sort order. Mainframe data is often stored in EBCDIC, which orders letters and digits differently from the systems most rewrites run on. A report can come out in a different order with every line correct.
• The work around the code. The overnight batch schedule, the order the jobs run in, the file another office collects every morning, the export someone retypes into a spreadsheet. None of that is in the COBOL source.

Then there's the awkward one: bugs people depend on. Sometimes the old system is wrong in a way the business has quietly built around. A faithful translation keeps the bug, and a fixed one changes everybody's numbers. Either can be the right call. It just has to be a call someone makes on purpose.

Send this before anyone translates a line

Whether the offer comes from a vendor, a consultant or your own developer with a new tool, here's the message we'd send first. Change the details, keep the substance.

Thanks for the proposal. Before we agree on scope, here's how we'll decide the new version is ready, so you can price and plan around it. >- We'll pick a recent stretch of real inputs (transactions, applications or orders, including a month-end and, if it matters to us, a year-end) and keep the outputs the old system produced from them.- The new version runs on those same inputs and we compare everything that leaves the system: files, reports, amounts, letters. Anything sent to a bank, a state office or another system has to match exactly. For screens and internal reports, we'll agree in writing which differences are acceptable.- Every difference gets a written explanation and a decision from the person who owns that rule. A difference nobody can explain counts as a failure.- The old system stays runnable, and we keep the option to switch back, until both have run side by side through at least one full cycle.- Tell us where AI tools wrote or changed code, and how that code was reviewed.- We'd like to move in pieces, starting with one program or job, rather than switching everything at once.- Payment for the migration is tied to passing this comparison, not to code being delivered. >If any of this changes your estimate, please tell us now. We'd rather know.

The AI point is its own conversation, and we've written up how to ask how AI touched your code and what to put in the contract afterwards. Moving in pieces is the argument of our letter to the owner of the system everyone wants to rewrite: replace it a part at a time so each part can be proved before the next. And if what you've got is half-built rather than old, inspect it before anyone quotes a rewrite.

A good partner will welcome this message, because it gives them a clear finish line. Someone who resists the side-by-side comparison is telling you they weren't planning to check.

Run the old and new side by side

The comparison in that message is usually called a parallel run. It isn't glamorous, and it's where much of the real work of a migration sits. Here's the shape of one.

Capture first. Before anything is translated, save real inputs and the outputs the old system produced from them. That's your answer key. Include the ugly cases: corrections, reversals, records with missing fields, the customer whose name is too long for the field.

Decide what "same" means for each output. A payment file going to a bank needs to match byte for byte, just like OpenAI's test. A screen probably doesn't need identical spacing. Write it down per output so nobody argues about it in week six.

Replay and compare automatically. Run the new version on the saved inputs and diff every output against the old one. This should be a script anyone can rerun after every change, not a person squinting at two printouts.

Sort every difference. Each one is a bug in the new code, a bug in the old code that people rely on, or a change you actually want. Only the owner of the rule can say which. Log the decision next to the difference.

Keep both running. Once the replay comes back clean, run old and new together on live work for a full cycle before the old one stops being the record. Our dual-write cutover checklist covers keeping that phase short and owned, and a source-of-truth map settles whose answer counts while both are running.

Then cut over with a way back. Agree in advance what would make you switch back, and who gets to decide.

AI is genuinely useful through all of this: it can explain what an old paragraph of code does, draft the translation and write the comparison scripts. What it can't do is tell you which of two different answers your organization meant. That's still a person's job.

Put this into practice

This week, if you own an old system:

• List everything that leaves it: each file, report, letter and payment, and who receives it.
• Start keeping copies of real inputs and outputs, including this month-end. That archive is worth having whether or not you ever migrate.
• Write down three rules you suspect live only in the code, like how a fee is rounded or what happens to a late payment.
• Save the message above, and send it to the next person who offers to translate the whole thing.

For public agencies, our government and public-sector work starts with the systems already in place and plans the transition with the people who will support the service. If your business runs on an application it has outgrown, we can help you replace it in pieces, with the comparison as the finish line. If someone has already pitched you an AI rewrite, tell us what the system does today.

Related work: Howcast.