Railway's breach turned stored AI keys into a $740,000 bill. Cap every key, and make it swappable without a deploy
Railway's postmortem says the attackers went looking for one thing in customers' environment variables: AI provider keys, which they could resell. Stolen keys ran up an estimated $740,000. If your app keeps a model key next to its code, work out what one bad night would cost you.

Railway's November 10, 2026 postmortem says an October attacker decrypted environment variables for up to about 100,000 workspaces, deliberately hunting AI provider keys, and estimated unauthorized spend at about $740,000. Treat every AI key like a company card: one key per app and environment, a spending limit or alert at the provider where offered, a way to rotate without a rebuild, and a written refund path for each provider you use.
What Railway's report says
Railway, the app-hosting platform, published its full incident report on November 10. It's long and unusually candid, and the part that matters for anyone running a web app is simple.
Between October 20 and 21, an attacker got into Railway's build system, moved through its internal systems, and eventually read and decrypted a subset of customers' environment variables for at most 1.85% of workspaces, about 100,000 of them. Railway's forensic work says the attacker was "specifically targeting AI provider credentials," picking database rows where the variable name matched patterns the attacker supplied, and favoring customers with lots of variables. The decrypted data also included linked-account tokens for 1,852 users, which Railway reset.
Railway estimates unauthorized spend on the stolen keys at about $740,000. It's working with providers on refunds and says it will cover the cost where a provider won't.
Sequence of work
Railway's incident, as its report tells it
- Oct 20, 07:28
Attacker partly escapes build isolation and takes a build credential.
- Oct 21, 04:07 to 07:15
Attacker pulls encrypted variables, prioritizing AI provider keys, and decrypts a subset.
- Oct 21, 20:19
Every user session and token invalidated. Builds already paused, so rotated credentials couldn't be redeployed.
- Oct 22, 22:49
Builds restored for all plans.
- Nov 10
Full report: about $740,000 in estimated unauthorized spend; refunds coordinated with providers.
Why AI keys were the target
Railway believes the keys were wanted for resale, so that others could use them to distill models. Whatever the motive, the economics are plain. A stolen database password needs work to turn into money. A stolen AI provider key is money the moment it works, billed to you.
An AI key isn't a password anymore. It's a card number with no limit unless you set one.
Most teams still treat these keys like any other configuration value. One key gets created during a prototype, pasted into the hosting platform, copied to staging, then to a second app. Nobody sets a limit because nobody expects the bill to change. When it does, the first sign is often the invoice.
An AI key isn't a password anymore. It's a card number with no limit unless you set one.
The detail that caught people out
There's a second lesson in Railway's report, and it's about rotation. During the incident, Railway paused builds while it secured its infrastructure. Customers did the right thing and revoked their keys. But their running apps still held the old values, and with builds paused there was no way to deploy the new ones. Railway's report acknowledges that "some customers encountered downtime because of this fact," and suggested rotating manually over SSH where possible.
That's a failure mode worth planning for on any platform. If changing a key requires a rebuild, your security response depends on your build system being up, at exactly the moment it's most likely to be down or overloaded. Railway says the rush of customers redeploying with new credentials, on top of its own rebuild, pushed capacity problems to nearly two orders of magnitude above normal.
Decision guide
The same leak, set up two ways
| Consideration | One shared key | Keys set up for a bad night |
|---|---|---|
| Scope | One key for production, staging and the side project | One key per app and environment, named so you can tell them apart |
| Spend | Unlimited until the invoice arrives | Monthly limit or alert at the provider, where offered, set near normal use |
| Rotation | New key needs a rebuild and deploy | App reads the key at runtime; swap it without touching the build |
| Refunds | Discovered during the incident | Each provider's fraud and refund contact written down in advance |
Four changes worth making this month
None of this needs a new platform. It needs an afternoon per app.
One key per app and environment. Production, staging and the internal tool each get their own key, with a name that says which is which. When one leaks, you revoke one key and know exactly what stops working. Our note on the AI gateway that holds every key covers the case where a single service holds all of them.
A spending limit or alert at the provider. Many AI providers let you set a budget, a usage limit or an alert by project or key. Check what yours offers and set it near your normal monthly use, with an alert well below the limit. A limit that interrupts your product for an hour is cheaper than a stranger's month of usage.
Rotation that doesn't need a deploy. Have the app read its key at runtime from a secret store or the platform's variable service, and test that a new value is picked up without a rebuild. Then rehearse it. Our rotation-as-a-cutover note has a sequence that avoids the gap where the old key is dead and the new one isn't live yet.
A refund path per provider, written down. Railway's report shows how different providers' positions can be: some had fraud policies that covered the charges, some reviewed claims case by case, and one enforced a no-refund policy, so Railway said it would cover those costs itself. You don't want to learn your provider's position during an incident. Find the fraud contact and the terms now, and keep them with your incident notes.
How you'd know a key is being used by someone else
Most teams find out from the invoice. You can find out sooner with things your AI provider already shows you.
Look at usage by key or project once a week, and set an alert for anything well above normal. Watch for usage at hours your product is quiet, for models your app doesn't call, and for a sudden jump in output tokens on a key that normally does small jobs. None of these prove a theft, but each is worth a five-minute look.
If you see it, revoke first and investigate second. A revoked key that breaks a feature for an hour is an inconvenience. Waiting a day to be sure can be the expensive part. That's also why one key per app matters: you can revoke the suspicious one without taking everything else down.
Finally, keep the evidence. Railway asked affected customers to report charges they didn't make through a form, and providers reviewed refunds case by case. Screenshots of the usage spike, the time you revoked the key, and the key's name make that conversation much shorter.
What to ask your hosting platform
If you host on Railway, follow its report: rotate any keys it notified you about, and report charges you didn't make through its form. Railway says it has rebuilt its build isolation and now limits which services can decrypt which secrets, with every decrypt logged.
If you host anywhere else, three questions are fair to ask your provider. Who and what can decrypt our variables? Is every decrypt logged, and can we see those logs? If you pause builds during an incident, how do we change a secret? The answers tell you how much of your plan has to live in your own setup rather than the platform's.
We build and look after web applications where keys, deploys and incident response all have an owner. If you want help setting up your AI keys so one bad night stays small, see our web application development work or talk to us.
Related work: Sutton|Place
We build alongside the people doing the work.
Bring us the problem you’re thinking about.