Rebuild somewhere else once, before you have to.
A Japanese cloud provider just told hundreds of customers their data probably can't be recovered, and that the only way back is a backup they kept themselves. Do you have one? Has anyone ever rebuilt from it?
Yes, keep your own. A provider's snapshots usually sit behind the same console and logins as your servers, so one bad night can take both. Keep a copy somewhere that account can't reach, decide how old it's allowed to be, and every few months rebuild from it on a different provider with a clock running. The time you write down is your real recovery plan.
Two customers, one outage, two different Thursdays
At about 3:40 on Wednesday morning in Japan, someone got into IDCF Cloud, the cloud service run by SoftBank-owned IDC Frontier, and ransomware took down part of its East Japan Region 1. By the provider's count, 495 companies and local governments were affected. Railway email services, a travel booking site, an e-commerce platform and local government websites were among the services reported hit.
On Thursday evening, IDC Frontier's third notice said the hard part out loud. Customer data in four zones is expected to be difficult to extract or restore. The servers there are stopped and can't be restarted. In the company's words, restoring data is currently possible only from backups customers hold themselves, and those customers are being advised to set up a separate environment and rebuild.
Two of its customers show what that sentence means in practice.
Six Apart, which runs a hosted version of the Movable Type CMS, had 31 servers in the damaged region. It also kept seven generations of daily backups on Google Cloud, outside IDCF entirely. By 9 p.m. Thursday, Japan time, about a day and a half after the attack began, it had moved all 31 onto a different provider, Sakura Cloud, using the backup taken at 1 a.m. Wednesday.
Soliton Systems, whose remote-desktop service ran on the same platform, told its customers on Wednesday afternoon that its own backup servers, kept on the same infrastructure, had been hit too and couldn't be used. By Wednesday night it had decided to rebuild on a platform it had confirmed was safe, and as of its Thursday update it still couldn't say when.
Same provider, same night. The difference was where the copy lived.
A snapshot in the same account isn't a way out
Most people who rent servers believe they have backups because the provider takes snapshots. Often they do, and those snapshots are great for everyday problems: a bad deploy, a deleted table, a migration that went sideways. What they don't cover is the provider itself having a bad night, because the snapshots sit behind the same console, the same admin logins and often the same buildings as the servers they copy.
The message the attackers left on IDCF's console claims they deleted more than 554,000 snapshots and removed 41.5 petabytes of backup capacity, in seven minutes. Those are the attackers' claims, and IDC Frontier hasn't confirmed them. Whatever the real figure, the provider's own conclusion was that the way back runs through backups customers kept themselves.
There's a quieter detail in the same notice that matters even if you weren't in the damaged zones. IDC Frontier suspended the customer console in every region while it checked the rest of its platform. No break-in had been found in those other regions, and customers there were told to take their own backups. But they couldn't log in to start or stop their own servers; IDCF staff were doing it for them on request. If your plan for a bad day is "we'll take a fresh backup when we need one," that plan assumes a console you may not have.
None of this is special to one Japanese provider, or to ransomware. This week Yandex said one of its cloud zones was down after a drone attack and fire at its Sasovo data center, and that service wouldn't be restored in the near future. Last month AWS told customers it can't restore data held only in its Bahrain region after drone strikes there. Different causes, same lesson: the copy that saves you is one that doesn't share a fate with the thing it copies.
So the useful questions are short. Where does a copy live that your provider's console and credentials can't reach? How old is it allowed to be? Has anyone actually rebuilt from it somewhere else, and how long did that take? And if you could only bring one thing back first, what would it be?
Decision guide
Where the copy lives
| Consideration | What it covers | What can still take it down too |
|---|---|---|
| Snapshots in the same account | A bad deploy, a deleted table, a migration that went wrong. | Anything that reaches the account, its console or its admin logins, or the buildings the account runs in. |
| Backup in another region, same provider | Losing one building or zone. | A provider-wide problem: shared logins, or a console suspended in every region at once. |
| Copy at a different provider, separate logins | Losing the provider account entirely. | Its keys stored back in the main account, or a copy nobody checks is still being written. |
| Offline or write-once copy | Someone with your admin keys deleting everything they can reach. | Slower to restore, and only as fresh as the last time it was made. |
A backup nobody has restored is a hope with a storage bill.
Pick a Saturday and pretend the account is gone
You can answer the first two questions at a desk. The last two you only find out by doing it, so we suggest a drill. Call it the Saturday rebuild.
Pick a quiet morning. Get the people who would really do the work on one call: whoever looks after the app, whoever holds the account logins, and whoever would have to tell customers what's going on. For a small business that might be two people and a contractor.
There's one rule. For the whole drill, nobody touches the production cloud account. Pretend it's gone. No logging in to grab a file you forgot, no copying a setting from the old dashboard. Everything has to come from the copy you keep elsewhere and whatever is written down.
Start a clock and keep a plain log: when each step started, when it worked, and every moment someone said "hang on, where's the...?" Those pauses are the point of the exercise. They're where a real recovery stalls, and it's much cheaper to find them on a Saturday you picked than at 4 a.m. on one you didn't.
You aren't rebuilding production. You're building a second copy, on a different provider or at least in a separate account with separate logins, and pointing a test address at it. Your real site keeps running the whole time.
How the morning usually goes
Here's roughly how it goes for a typical web app or online store. Your times will be different, and finding out yours is the reason you're there.
Find the copy. Where is it, and who can open it? If the only person with the login is on holiday, or the key that decrypts the backup is stored in the account you just pretended to lose, you've found your first problem in the first fifteen minutes. That alone was worth the morning.
Check how old it is. Look at the newest backup and ask what happened after it. Six Apart's copy was from 1 a.m.; anything written after that wasn't in it. For a blog, a few hours is livable. For a store taking orders overnight, it means orders you'd have to piece back together from payment and email records. Decide how much of a day you could stand to re-enter, then check the backups are actually that fresh.
Set up an empty place to put it. Servers, a database, storage for files and images. If your setup is written down as code, this goes quickly. If it lives in one person's memory and an old dashboard, this is where the morning goes.
Restore, then look before you trust it. Load the data. Check it's complete, check the latest records look right, check nothing looks tampered with. According to Threadlinqs' summary, IDCF's recovery guidance includes inspecting backups for malware or tampering and checking data consistency before relying on them. A restored copy of a compromised system is still compromised.
Make it answer. Can someone log in? Does a test order go through, with the payment provider talking to the new address? Do password resets and order confirmations actually send? Do the scheduled jobs run? This is the step people skip, and it's where the surprises live: API keys tied to the old servers, a partner who only accepts traffic from your old IP address, email settings that still point at the old host. Six Apart warned its own customers that the move changes IP addresses and that storage and specs might not match exactly. Each of those is something you'd rather meet on purpose.
Write down the times. When it all works, or when you stop for the day, note how long each step took and what you had to go looking for.
Sequence of work
A Saturday rebuild, on the clock
- 9:00 Start the clock
Rule: nobody touches the production account. Log every step and every pause.
- 9:15 Find the copy
Where it lives, who can open it, where the decryption key is, how old the newest one is.
- 9:45 Set up an empty place
A separate account or provider: servers, database, file storage.
- 10:30 Restore and inspect
Load the data; check it's complete, recent and untampered.
- 11:30 Make it answer
Logins, a test order, outgoing email, scheduled jobs, keys and allowed addresses.
- 12:30 Write down the times
How long each step took, what stalled, and what has to come back first.
Decide what comes back first
The drill nearly always ends with the same discovery: you can't bring everything back at once, and nobody had decided what goes first.
Make that call now, while it's calm. For most web apps and stores, the order looks something like this:
- What customers touch. The storefront, checkout, login, booking. If people can't buy from you or reach you, nothing else matters yet.
- What staff need today. The order admin, the packing and shipping tools, the support inbox, the internal tool someone opens at 8 a.m.
- What can wait a day. Reports, analytics, search indexes, the nightly sync that can catch up later.
It's a short list, but it changes how things should be built. If checkout has to come back first, it shouldn't depend on the reporting database. If the order admin is the thing your team can't work without, it needs its own place in the backup and in the drill, not a spot on whatever server it happened to land on. For stores, a lot of our ecommerce work is exactly this: making sure orders can keep coming in while the rest catches up.
One honest caveat. Some businesses can't simply copy data to another provider or country, because of contracts or data-residency rules. If that's you, the answer isn't to skip the copy. It's to find where you're allowed to keep one, and write the constraint down so nobody discovers it in the middle of an incident.
Keep the times, and do it again
When the drill is done, you'll have something most companies don't: a real number. Not the recovery time in a provider's brochure, but how long it took your people to get your app answering somewhere else, from a copy you control.
Keep the log. Put the four answers on one page: where the copy lives and who can open it, how old it's allowed to be, how long the rebuild took, and what comes back first. Fix the two or three things that stalled you, then run it again in three months and see whether the number drops. If you already keep short recovery runbooks for everyday incidents, this becomes the big one at the back. And if the drill shows you couldn't leave your provider even if you wanted to, it's worth scoring your exit readiness before the next renewal.
If you run a web app, store or internal tool on one provider and nobody has ever rebuilt it anywhere else, we can help you plan the first Saturday and fix what it turns up. Tell us what you run.
Related work: Spin:Market
We build alongside the people doing the work.
Bring us the problem you’re thinking about.