Phantasialand's pool alarm went out by email during the Microsoft outage. Give your critical alerts a second way in
On November 13 a hotel pool near Cologne got far too much chlorine, and the system watching it did raise the alarm. By email, in the middle of the Microsoft outage. Most small businesses have alerts that would go just as quiet.

List every alert your business relies on and ask what would happen if it sat unread for a day. For the few that can't wait, such as failed payments, a down checkout, security sign-ins and freezer or equipment sensors, add a second path that doesn't depend on your email provider, send it to a person and a backup, make it escalate when nobody acknowledges it, and test it with email out of the picture.
The alarm went off. Nobody heard it
On Friday, November 13, the pool at Hotel Ling Bao, part of the Phantasialand theme park in Brühl near Cologne, started getting far too much chlorine. The automatic dosing system had a fault. The system watching it noticed, and did exactly what it was set up to do: it sent an alarm by email.
That was the second day of Microsoft's big Outlook and Exchange outage. According to what Phantasialand told WDR, the warning emails couldn't be delivered, so nobody on site knew. Guests who had used the pool complained of irritated airways. Fifteen people were affected in the end. Four went to hospital, and one stayed in for observation. The pool was closed, the water was replaced, and it has since reopened. RTL reports that the park plans to stop sending these warnings by email alone.
Outlook had nothing to do with the chlorine. What the outage broke was the path between a machine that knew something was wrong and the people who could fix it.
Microsoft's own explanation came on Sunday. According to its preliminary post-incident report, as described by The Register and TechCentral, a change made in July to its network gateway servers had a latent defect. A routine Teams failover added load to the gateway Exchange Online uses, while almost 14% of that gateway's capacity was already out for an unrelated reason. Together they tipped the servers into exhausting their CPU and memory, and the failure cascaded until more than 80% of them were unresponsive. Teams, OneDrive and SharePoint could move away from the broken gateway. Exchange Online couldn't, "for scale reasons," so email was out for the better part of a day. A final report is due November 20.
You can't do much about Microsoft's gateways. You can do something about the alerts in your own business that are quietly making the same bet the pool's alarm made.
Ask every alert one question
Most small businesses have more alarms than they think. They just don't call them that. The usual ones:
- Payments. A failed card run, a declined subscription renewal, a payout that didn't land.
- Stock. A low-stock or out-of-stock warning from the store or the warehouse.
- The website. An uptime monitor saying the site or checkout is down, or a certificate about to expire.
- Forms. A quote request, a booking, a "call me back" from the contact page.
- Security. A new sign-in on the admin account, a password reset nobody asked for, a payout detail being changed.
- Physical things. The walk-in freezer, a server-room temperature sensor, a water leak sensor, a pool, a boiler.
- Orders. A new order, especially one that has to ship today.
Sit down with whoever sets these things up and write each one on its own line. Next to it, write where it starts (the store, the payment processor, the sensor), where it lands (which inbox, whose phone), and who is supposed to act.
Then ask one question of every line: if this sat unread for a whole day, what would happen?
It's a better question than "is this important?", because everything feels important. Last week a day without email wasn't a thought experiment for a lot of Microsoft 365 customers. For many businesses, every line on that list landed in one inbox, on one provider, read by one person.
A forwarding rule isn't a second path. If the first mailbox never gets the message, there's nothing to forward.
Sort them by what a quiet day costs
Most answers fall into three piles.
It can wait. A weekly report, a newsletter signup, a review notification. If you see it tomorrow, nothing breaks. Leave these in email. Email is fine for them.
It costs money. Failed payments that won't retry, a bestseller going out of stock during a sale, a checkout that's been down since lunch, leads that go cold because nobody called back. A silent day here shows up in revenue, and sometimes in a customer who doesn't come back.
It costs safety or trust. A freezer drifting warm overnight, a leak, a dosing system or boiler misbehaving, someone signing in to your admin who shouldn't be. A silent day here can hurt people, spoil stock or hand someone your accounts. Most businesses have only a handful of these. They need to be heard in minutes, not by tomorrow.
Be honest when you sort. If the person who'd act on an alert is on holiday this week, write that down too. An alert that reaches the right inbox and the wrong person is still a silent alert.
Give the critical few a second way in
For everything in the last two piles, add a second path that doesn't fail for the same reason as the first. The test of a second path is simple: if your email provider went down tomorrow, would this one still arrive?
Paths that usually pass:
- A text message or an automated phone call sent by the monitoring tool or alerting service itself, not relayed through your mailbox.
- A push notification from the app that does the watching, on a phone that isn't logged in through the account that's down.
- A chat message in Slack or another tool, if it doesn't depend on the same provider. If your team signs in to that tool with Microsoft accounts, check what happens when Microsoft sign-in is having a bad day.
- Something local and loud. A buzzer on the freezer, a light on the panel, a siren in the plant room. Not glamorous, and it works with no internet at all.
Paths that look like a second path and aren't:
- A forwarding rule from your Outlook inbox to a personal Gmail. If the first mailbox never gets the message, there's nothing to forward.
- A second address on the same provider. It's the same mail system.
- "Someone checks the portal." A dashboard nobody's looking at is the same as no alert.
Two more things matter more than people expect. Send the alert to a person and a backup, not just one name. And write down whose phone number the text goes to. Numbers on alerting tools are often set up once and then go stale when people change roles or leave.
Decision guide
Is it really a second path?
| Consideration | Looks like a backup | Arrives when email is down |
|---|---|---|
| Text message | Your mailbox forwards the alert to a phone as a text. | The monitoring tool or alerting service sends the text itself. |
| Second inbox | A forwarding rule or another address on the same email provider. | A channel on a different provider, such as SMS, a phone call or app push. |
| Chat | A chat tool your team signs in to with the same account that's down. | A chat or push channel that doesn't depend on that sign-in. |
| Who gets it | One name, set up years ago. | A person and a backup, with phone numbers checked this quarter. |
| On site | Someone is meant to check the dashboard. | A local buzzer, light or siren on the equipment itself. |
Make it escalate when nobody answers
A second path solves a missed delivery. It doesn't solve a delivered alert that everyone assumes someone else saw.
For the alerts in the safety-or-trust pile, add an acknowledgement. Someone has to press a button, reply to the text or take the call to say "I've got it." If nobody does within a set time, the alert goes to the next person, and then the next. Many monitoring and on-call tools can do this, and some sensor systems can too. It's worth asking your vendor before you buy anything new.
Pick the times with the person who'd actually respond. Fifteen minutes might be right for a freezer, and five for a security alert on the payments account. Decide who sits at the bottom of the ladder, usually an owner or manager, and make sure they know they're there.
Keep the ladder short. Three steps, real names, phone numbers someone checked this month. Anything longer tends to fall apart the first time it's used for real.
How the work moves
Five steps for the alerts that can't wait
- List
Every alert: where it starts, where it lands, who acts.
- Sort
Ask what a day unread would cost: it can wait, it costs money, it costs safety or trust.
- Second path
Give the critical ones a channel that doesn't share your email provider, to a person and a backup.
- Escalate
Require an acknowledgement; if nobody answers in time, call the next person.
- Test quietly
Trigger a harmless alert with email out of the picture and write down how long it took.
Test the quiet path on purpose
Here's the part almost nobody does. Once a second path is set up, test it as if email were down.
Most tools have a "send test alert" button. Use it, but watch the second path only: don't count it as working because the email arrived. Better still, trigger a real-but-harmless event. Let the uptime monitor see a test page go down, or warm a sensor by hand for a minute. Time how long until a phone buzzes, and whose. Then don't acknowledge it, and see whether the escalation actually reaches the next person.
Write down what you find: the alert, the path, the time it took, who got it. Do it again whenever someone joins or leaves, whenever you change phone numbers or providers, and a couple of times a year anyway. Things drift.
A sensible order for this week:
- Today: list every alert you depend on, with where it starts, where it lands and who acts.
- This week: sort them by what a silent day would cost, and pick the handful that can't wait.
- Next: give those a second path that doesn't share your email provider, with a backup person.
- Then: add an acknowledgement and a short escalation ladder for the safety-or-trust pile.
- Every few months: test the quiet path with email out of the picture, and write down the time.
None of this needs a big project. It does need someone to own the list, because alerts are usually set up one at a time by whoever needed them that week, and nobody ever looks at all of them together.
If your alerts are spread across a store, a payment processor, a few sensors and a shared inbox, and you'd like them routed, escalated and tested properly, that's the kind of thing our business process automation work covers. Tell us which alerts you'd hate to miss and we'll help you map where they go.
Related work: Spin:Market
We build alongside the people doing the work.
Bring us the problem you’re thinking about.