Recover with a pattern library, not folklore.
The queue is stuck, a write landed on only one side, and whoever is on call is working from memory. That gets you a different recovery every time, sometimes the one that makes things worse. Short, rehearsed steps beat improvising at 2 a.m.
Replace late-night improvisation with a Recovery Runbook Pattern Library: short cards for stuck queues, partial writes, permission errors, and external timeouts. Each card needs symptom, check-first inspection, safe actions, escalation, and audit fields—bound to named owners.
Recovery should not be improvised
The queue is stuck. A write landed on one side. A permission error is blocking the only person who can fix it. An external call timed out after your system already committed something.
If the response is whatever the person on call remembers, you will get a different recovery every time, including the one that makes it worse. Give them a short card they have used before.
The same skeleton on every card
Keep the cards short and the shape stable.
Symptom. What the operator actually sees.
Likely causes. A short list, not an encyclopedia.
Check first. What to inspect before any write or retry. This is the rule that prevents a second side effect.
Safe actions. Ordered steps that avoid duplicate harm.
Escalate when. The exit to a named owner, or to the serious-incident path.
Audit. What to record when the card is closed.
Start with four patterns you already meet: a stuck queue, a partial write, a permission error, and an external timeout. For the timeout and the partial write, the first action is to see what already happened. Then compensate, or confirm, or stop. For a duplicate signal, look for a key before you create a second effect.
Retire cards that no longer match the system. A stale card is a hazard. If two cards conflict, merge them in daylight, not during the incident.
How the work moves
Use the same card skeleton every time
- Symptom
Describe what operators observe in plain language.
- Check-first
Inspect the record before any write or retry.
- Safe actions
Ordered steps that avoid duplicate harm.
- Escalate / audit
Exit to an owner when needed; record what was done.
Improvisation is not a recovery strategy.
Check first
The failure mode this library exists to stop is the helpful retry. Someone re-runs a job that already captured a payment, or already told a customer a thing that is now false.
The card's second step is not the fix. The second step is the inspection. If the card does not say what to look at before anyone writes, it is not ready to hand to a tired person.
Bind each card to an owner
A pattern with no owner is documentation. For each card, name who may run the safe actions, who may approve an exception to the card, and which severity the symptom usually is.
Keep the library next to the incident matrix, so the serious-incident path and the cards point at the same roles. When a person changes, update the card the same week. Do not learn the name is wrong during the next outage.
Version them lightly. A date and an owner are enough.
Decision guide
Patterned recovery vs improvisation
| Consideration | Pattern library | Improvisation |
|---|---|---|
| Stuck queue | Check consumers and poison messages before redrive. | Redeploy or redrive until motion appears. |
| Partial write | Reconcile both sides; compensate or complete deliberately. | Retry both sides and hope. |
| Permission error | Verify scope before elevating access. | Grant broader rights to unblock quickly. |
| External timeout | Determine remote commit state before acting again. | Resend immediately and risk duplicates. |
Put this into practice
List the last five failures that needed judgment. Write cards for the ones that match the four patterns. Rehearse one card against a past incident, on a table, not in production.
Recovery is part of the product. Four living cards beat a wiki nobody opens. If you want help designing the automation around those paths, start from the failures you already know. That work sits under business process automation.
Related work: Sutton Place.
We build alongside the people doing the work.
Bring us the problem you’re thinking about.