You rotated the webhook secret while events were still in flight
Security asked for a new signing secret by Friday. You flipped it at noon. The retries from 11:50 still signed with the old one. By evening the dead letter looked like an outage.
Rotating a webhook signing secret is a cutover, not a checkbox. Overlap old and new verification, drain in-flight retries, then disable the old secret—or you will treat valid late deliveries as forgeries.
The temptation
A scanner finds a webhook signing secret in an old ticket. Compliance wants it gone. Someone updates the provider dashboard, pastes the new value into the receiver's env, restarts, and tickets the task done. A secret rotation without an overlap window is just a scheduled outage.
Providers retry. Workers queue. Partners batch. The events signed at 11:50 do not care that you felt finished at 12:05.
What breaks
Your verifier accepts one secret. Late deliveries and retries still present HMAC signatures from the previous secret. You return 401. The provider backs off, then retries again—still old signature if they signed once and stored the payload. Or they re-sign with the new secret while your second region still runs the old env. Either way, the dead-letter pile grows with legitimate business events: paid invoices, shipment updates, seat changes.
On-call sees signature failures and hunts an attacker. Finance sees missing syncs and hunts your team.
How the work moves
Overlap cutover for signing secrets
- Generate
New secret created; old remains valid at provider.
- Accept both
Receiver verifies against a short list of secrets.
- Prove
Synthetic or shadow traffic shows green on both keys.
- Drain
Wait out retry window and queue lag.
- Retire
Disable old secret; remove code after cooling.
A secret rotation without an overlap window is just a scheduled outage.
The argument for overlap
Treat rotation like a dual-key cutover:
- Generate the new secret in the provider; keep the old one valid.
- Deploy the receiver to accept either signature (constant-time compare against a small list).
- Confirm healthy accepts on both keys with a synthetic event if the provider offers one.
- Disable the old secret only after the provider's max retry window (and your own queue lag) has passed.
- Remove old-key code paths after a cooling period so the overlap does not become permanent dual-custody theater.
If the provider cannot hold two secrets, you need a maintenance window with them—not a silent noon flip. Document that constraint before security sets a Friday deadline in a vacuum.
Decision guide
Hard flip vs overlap
| Consideration | Hard flip at noon | Overlap then retire |
|---|---|---|
| In-flight events | Look like forgeries. | Verify under either key until drained. |
| On-call story | Possible intrusion hunt. | Planned cutover with a known owner. |
| Business effect | Missing syncs until replay. | Continuous accepts through the window. |
| Compliance | Checkbox with collateral damage. | Evidence of controlled retirement. |
What "done" means
Done is not "env var updated." Done is:
- Zero signature failures attributable to key mismatch for a full retry window
- Dead-letter replay completed for anything rejected during the flip
- Runbook updated with the overlap steps and the owner who may disable the old key
- Secret storage (vault, not chat) lists only the active key after cooling
If you cannot measure signature-failure reasons separately from other 401s, add that label before the next rotation. Otherwise you will not know when overlap is safe to end.
Multi-region receivers need the overlap deploy to land everywhere before you disable the old provider secret. A region still on the old image will reject new signatures while another accepts them—or the reverse during a hard flip.
Partner-facing webhooks you send have a mirror problem: rotate the secret you share with them using the same overlap etiquette, and tell them the window in writing. Surprise rotations make you the villain in their dead letter.
Store secrets in a vault with access audit. Chat-pasted secrets are how the scanner finding happened.
Practice on a low-risk endpoint first
Pick an internal or low-volume webhook, run the full overlap drill, and write down how long drain actually took—not the vendor's brochure retry number. Use that measured drain when security sets the next deadline for a payment or inventory webhook.
Alert on signature-mismatch rates as a first-class signal during rotation windows, with a silence mode afterward so normal attacker probes do not page you the same way. Context in the alert ("rotation window for provider X") saves an hour of panic.
Document the break-glass: if the provider forces a single-secret flip with one hour notice, who joins the bridge, which queues pause, and how synthetic replay runs afterward.
Include consumer services that verify outbound webhooks you send—and inbound ones from billing, shipping, auth, and marketing automation. One forgotten endpoint will be the incident after you celebrate a clean rotation on the two you remembered.
If signatures include timestamps, confirm skew tolerance during overlap. A new key plus strict timestamp window plus clock drift is a false-failure triad. Measure NTP health as part of the rotation checklist.
Write the rotation into your change calendar with a rollback: re-enable the old secret only if the provider still holds it. Once disabled at the provider, rollback may mean replaying dead letters under the new key instead. Know which case you are in before you start.
Security questionnaires that ask "do you rotate webhook secrets?" should be answered with the overlap runbook attached. A yes without a method is how noon flips keep happening under audit pressure.
Pair secret rotation with endpoint URL rotation only when you must; doing both at once multiplies failure modes. If you change URLs, run a parallel endpoint that accepts both secrets while DNS and partner configs catch up.
Keep a printable one-page runbook in the on-call book: provider console path, vault path, overlap flag in config, who can disable the old key, and the chat channel for the window. Incidents at noon on Friday are not when you want to discover the vault path is wrong.
After success, schedule the next rotation before the scanner does. Cadence beats heroics.
Treat provider status pages and signing-secret emails as pageable events during the window. A provider who rotates on their side without your overlap plan is the same failure mode with a different author.
Log which key id verified each request during overlap (without logging the secret). That metric tells you when old-key traffic has truly drained so you are not guessing from the calendar alone.
If finance owns a billing webhook you do not, invite them to the overlap window. Orphan endpoints are still your incident.
Announce the overlap window in engineering chat the morning you start—not after the first 401 spike.
Overlap is the whole point.
Put this into practice
List every inbound webhook your product verifies. For each, note: provider dual-secret support, max retry duration, who can edit the receiver secrets, and whether verification already accepts a list. The gaps are your next rotation risks. Schedule the first overlap drill on a low-traffic hour before compliance picks the date for you.
We build web applications where credential cutovers drain work instead of dropping it. If your last secret flip created a mysterious dead letter—talk to us.
Related work: Howcast.
We build alongside the people doing the work.
Bring us the problem you’re thinking about.