Before you merge the agent's work, ask what it took out

Two changes nobody asked for

On October 20, 2026, OpenAI published a report about two of its own coding agents. Neither did anything dramatic. Both were doing ordinary software work, and both took a safeguard out along the way.

In the first, on October 3, an agent was repairing an evaluation pipeline. Much earlier in the session, a researcher had told it not to send messages while it searched Slack. Eighty-odd messages later, the agent decided that instruction also covered the automatic alerts the monitoring system sends to OpenAI's Security team. Its patch emptied the alert recipient lists and switched notifications off. The monitor still ran and saved its reports. Nobody got paged.

In the second, on October 6, a user asked an agent to copy what a reference implementation did into their own code. The reference had no monitoring integration, so the agent removed theirs to match. Then it told the user it hadn't.

OpenAI says it's strengthening its monitoring in response. For everyone else, there's a simpler lesson: when software changes, look at what left.

Why removals slip through

Code review grew up around additions. A new feature is the point of the pull request, so that's where eyes go. Deleted lines show up in red and look like tidying.

AI coding agents make this worse in a specific way. They're very good at reaching the stated goal, and "make the tests pass" or "match the reference" or "clean this up" are goals that removing something can satisfy. A failing test can be fixed, or it can be loosened. A noisy alert can be tuned, or it can be switched off. A permission check that blocks the happy path can be handled, or it can quietly go.

Then there's the summary. Agents write a tidy description of what they did, and it's tempting to read that instead of the change. One of OpenAI's agents described its own work incorrectly. An agent's summary of its change is a claim. The diff is the evidence.

The first OpenAI case shows a second pattern. An instruction given for one purpose got stretched to cover another: "don't send any messages" was about posting in Slack, not about the alerts a monitor sends. Agents carry instructions through long sessions, and an old one can quietly govern a new decision. That's why the message below asks for the instruction behind any removal.

None of this is new. People delete tests too. What's changed is volume: an agent can open more pull requests in a week than a small team used to review in a month.

The message to send

If an agency, a contractor or your own team uses AI coding agents on your software, send them something like this. Edit the names, keep the substance.

Subject: How we review changes made with AI coding agents >For any pull request where an AI agent wrote or edited code, please do these before it's merged: >1. List removals separately in the description: deleted or loosened tests, validation, permission or role checks, logging, alerts and monitoring, rate limits, retries, and feature flags. One line each, with why.2. Flag test changes next to code changes. If the same PR changes a feature and edits the tests for it, say so at the top, and explain each edited assertion.3. Call out config. Changes to environment settings, CI steps, alert rules, or anything that decides who gets notified need a named person's sign-off.4. Quote the instruction the agent was given when it led to a removal, like "match the reference" or "make it pass", so the reviewer can judge whether that's what was meant.5. Don't treat the agent's summary as the review. A person reads the diff, including the red lines. >If any of this slows a change down, tell me which one and we'll talk about it.

That's the whole ask. It takes a few minutes per pull request, and most of them will say "no removals."

What a removal list catches

Picture an ordinary change: the agent is asked to speed up checkout. It caches the shipping rates, which is fine. It also drops a check that refused orders to addresses it couldn't validate, because that check was slow, and it edits a test that expected the refusal. All three changes are in one tidy pull request titled "Checkout performance."

Without a removal list, a reviewer sees faster checkout and green tests. With one, the description says, in plain words: "Removed address validation on submit (slow). Updated test to match." That line is the conversation you want to have before customers find it.

That's an illustrative example, but every piece of it is ordinary. We've written before about asking how AI touched your code and putting the answer in the contract. This is the day-to-day half of the same idea.

When a tool reviews the tool

Automated reviewers help. Our Claude Code Review GA review covered one that reads every pull request and never merges. Use them, and point them at removals specifically: ask the reviewer to list every deleted check, test assertion and alert.

Keep a person on the decision, though, for three kinds of change: anything that touches who gets told when something goes wrong, anything that touches who is allowed to do what, and anything that edits a test to make it pass. OpenAI's report is about exactly the first kind.

Put this into practice this week

Send the message above to whoever ships changes to your software. Ask them to apply it to the next five pull requests and tell you what showed up in the removal lists.

Check where your alerts go. Make a short list of the alerts that matter, the people who receive each, and when someone last confirmed one arrived. If an agent could change that list, someone should notice.

If you're not sure what's been changed in your software recently, or by whom, we can help you get a clear picture and set up a review routine that fits your team. That's part of our web application work. Talk to us.

Related work: Sutton|Place.