twimco.Let’s talk ↗︎
Menu
Field notes / Product & engineering

Railway's breach turned stored AI keys into a $740,000 bill. Cap every key, and make it swappable without a deploy

Railway's postmortem says the attackers went looking for one thing in customers' environment variables: AI provider keys, which they could resell. Stolen keys ran up an estimated $740,000. If your app keeps a model key next to its code, work out what one bad night would cost you.

Editorial engraving of a bright office corner where an engineer swaps one brass key in an open wall key cabinet, a laptop and padlock on the desk beside her.

Railway's November 10, 2026 postmortem says an October attacker decrypted environment variables for up to about 100,000 workspaces, deliberately hunting AI provider keys, and estimated unauthorized spend at about $740,000. Treat every AI key like a company card: one key per app and environment, a spending limit or alert at the provider where offered, a way to rotate without a rebuild, and a written refund path for each provider you use.

What Railway's report says

Railway, the app-hosting platform, published its full incident report on November 10. It's long and unusually candid, and the part that matters for anyone running a web app is simple.

Between October 20 and 21, an attacker got into Railway's build system, moved through its internal systems, and eventually read and decrypted a subset of customers' environment variables for at most 1.85% of workspaces, about 100,000 of them. Railway's forensic work says the attacker was "specifically targeting AI provider credentials," picking database rows where the variable name matched patterns the attacker supplied, and favoring customers with lots of variables. The decrypted data also included linked-account tokens for 1,852 users, which Railway reset.

Railway estimates unauthorized spend on the stolen keys at about $740,000. It's working with providers on refunds and says it will cover the cost where a provider won't.

Sequence of work

Railway's incident, as its report tells it

  1. Oct 20, 07:28

    Attacker partly escapes build isolation and takes a build credential.

  2. Oct 21, 04:07 to 07:15

    Attacker pulls encrypted variables, prioritizing AI provider keys, and decrypts a subset.

  3. Oct 21, 20:19

    Every user session and token invalidated. Builds already paused, so rotated credentials couldn't be redeployed.

  4. Oct 22, 22:49

    Builds restored for all plans.

  5. Nov 10

    Full report: about $740,000 in estimated unauthorized spend; refunds coordinated with providers.

Dates from Railway's November 10, 2026 incident report (times in UTC).

Why AI keys were the target

Railway believes the keys were wanted for resale, so that others could use them to distill models. Whatever the motive, the economics are plain. A stolen database password needs work to turn into money. A stolen AI provider key is money the moment it works, billed to you.

An AI key isn't a password anymore. It's a card number with no limit unless you set one.

Most teams still treat these keys like any other configuration value. One key gets created during a prototype, pasted into the hosting platform, copied to staging, then to a second app. Nobody sets a limit because nobody expects the bill to change. When it does, the first sign is often the invoice.

An AI key isn't a password anymore. It's a card number with no limit unless you set one.

The detail that caught people out

There's a second lesson in Railway's report, and it's about rotation. During the incident, Railway paused builds while it secured its infrastructure. Customers did the right thing and revoked their keys. But their running apps still held the old values, and with builds paused there was no way to deploy the new ones. Railway's report acknowledges that "some customers encountered downtime because of this fact," and suggested rotating manually over SSH where possible.

That's a failure mode worth planning for on any platform. If changing a key requires a rebuild, your security response depends on your build system being up, at exactly the moment it's most likely to be down or overloaded. Railway says the rush of customers redeploying with new credentials, on top of its own rebuild, pushed capacity problems to nearly two orders of magnitude above normal.

Decision guide

The same leak, set up two ways

ConsiderationOne shared keyKeys set up for a bad night
ScopeOne key for production, staging and the side projectOne key per app and environment, named so you can tell them apart
SpendUnlimited until the invoice arrivesMonthly limit or alert at the provider, where offered, set near normal use
RotationNew key needs a rebuild and deployApp reads the key at runtime; swap it without touching the build
RefundsDiscovered during the incidentEach provider's fraud and refund contact written down in advance
Decision framework for AI provider keys in a hosted app; illustrative, not a measured result.

Four changes worth making this month

None of this needs a new platform. It needs an afternoon per app.

One key per app and environment. Production, staging and the internal tool each get their own key, with a name that says which is which. When one leaks, you revoke one key and know exactly what stops working. Our note on the AI gateway that holds every key covers the case where a single service holds all of them.

A spending limit or alert at the provider. Many AI providers let you set a budget, a usage limit or an alert by project or key. Check what yours offers and set it near your normal monthly use, with an alert well below the limit. A limit that interrupts your product for an hour is cheaper than a stranger's month of usage.

Rotation that doesn't need a deploy. Have the app read its key at runtime from a secret store or the platform's variable service, and test that a new value is picked up without a rebuild. Then rehearse it. Our rotation-as-a-cutover note has a sequence that avoids the gap where the old key is dead and the new one isn't live yet.

A refund path per provider, written down. Railway's report shows how different providers' positions can be: some had fraud policies that covered the charges, some reviewed claims case by case, and one enforced a no-refund policy, so Railway said it would cover those costs itself. You don't want to learn your provider's position during an incident. Find the fraud contact and the terms now, and keep them with your incident notes.

How you'd know a key is being used by someone else

Most teams find out from the invoice. You can find out sooner with things your AI provider already shows you.

Look at usage by key or project once a week, and set an alert for anything well above normal. Watch for usage at hours your product is quiet, for models your app doesn't call, and for a sudden jump in output tokens on a key that normally does small jobs. None of these prove a theft, but each is worth a five-minute look.

If you see it, revoke first and investigate second. A revoked key that breaks a feature for an hour is an inconvenience. Waiting a day to be sure can be the expensive part. That's also why one key per app matters: you can revoke the suspicious one without taking everything else down.

Finally, keep the evidence. Railway asked affected customers to report charges they didn't make through a form, and providers reviewed refunds case by case. Screenshots of the usage spike, the time you revoked the key, and the key's name make that conversation much shorter.

What to ask your hosting platform

If you host on Railway, follow its report: rotate any keys it notified you about, and report charges you didn't make through its form. Railway says it has rebuilt its build isolation and now limits which services can decrypt which secrets, with every decrypt logged.

If you host anywhere else, three questions are fair to ask your provider. Who and what can decrypt our variables? Is every decrypt logged, and can we see those logs? If you pause builds during an incident, how do we change a secret? The answers tell you how much of your plan has to live in your own setup rather than the platform's.

We build and look after web applications where keys, deploys and incident response all have an owner. If you want help setting up your AI keys so one bad night stays small, see our web application development work or talk to us.

Related work: Sutton|Place

twimco.

We build alongside the people doing the work.
Bring us the problem you’re thinking about.

Related work

Sutton Place ↗︎