twimco.Let’s talk ↗︎
Menu
Field notes / Operations

Replay a dead letter, or leave it named.

A webhook fails halfway, and by morning there's a folder of payloads nobody is clearly allowed to resend. Run the handler again and you might charge, notify or advance a deal twice. A failed delivery should stay a record with an owner.

Editorial illustration for Field Note: Replay a dead letter, or leave it named.

Before you automate retries on a webhook edge, decide who owns dead letters, whether ordering or last-write wins, which key dedupes a replay, how long replay stays allowed, and who may force-skip a message. A payload with no owner is not a queue policy; it is an unowned failure.

A failed delivery is a record, or it is a shrug

A webhook does not arrive, or it arrives and the handler fails halfway. The producer retries on a schedule you did not choose. The consumer inserts a row twice, or inserts nothing and leaves a log line. By morning the failures are a folder of payloads. Nobody can say who is allowed to send one again.

A dead letter is useful when it is a record: the original attempt stays visible, a role owns it, and replay is a separate action with a rule. It is not useful when “retry” means “run the handler again and hope the first attempt did not already charge, notify, or advance a deal step.”

This note is the TWIMCO Webhook Dead-Letter Replay Spec. It covers one edge—one inbound callback or one outbound call you make. You name the dead-letter owner, whether replay must keep order or may last-write, the dedupe key, the maximum replay window your team will honor, and who may force-skip. You do not copy a window or a retry count from this page. Those values belong to the edge. There is no live tool that fills this spec for you. The instruments that are live are the AI workflow readiness check, for whether a workflow should be automated at all, and the custom product build brief, for scoping the product around it. Neither one replaces the replay page.

On Sutton|Place, the public partnership keeps seller assertions, verification, and disclosure distinct. An explicit action carries an actor and a time. Replay deserves the same shape: the failure remains a fact, and sending again is a new fact, not a silent overwrite of the first. Howcast project work sat beside global video delivery and publishing decisions across channels—the kind of operation where a missed handoff is visible past a log line. There is no public Howcast case-study page. Do not treat the project name as proof of a webhook design. Use it as a reminder that delivery systems fail in public. If the edge is a handoff between systems, scoped help for that shape of work lives under business process automation.

Bring one real edge to the page before you discuss tooling. If you cannot name the producer, the consumer, and the business object the payload is about, you are not ready to automate replay.

Fill five fields before the first poison message

Copy this onto one page. Leave a blank rather than invent a value in the meeting.

Edge. Inbound or outbound. Producer system. Consumer system. The business object (deal step, shipment notice, publish job—whatever the payload represents). One sentence on the side effect a successful handling is allowed to commit.

Dead-letter owner. A role, not a channel name. This role sees the parked payload, decides replay or skip, and is accountable if it sits. “The on-call” is acceptable only if the rotation is a named role with a backup. A shared inbox with no role is a blank.

Ordering rule. Pick one and write why:

  • Strict order. Later messages for the same object must not apply until this one succeeds or is skipped.
  • Last-write. A newer accepted message makes this payload obsolete; replay would be wrong.
  • Independent. This event does not depend on siblings; replay does not reorder other work.

If two people in the room pick different rules, you do not have a policy. You have a future duplicate.

Dedupe key. The field or tuple that means “this is the same attempt.” Message id, event id plus object id, or a key you generate and send. “The whole JSON body” is a weak key if the producer changes whitespace. “The timestamp” is not a key. Write where the consumer stores keys it has already applied, and for how long that memory must outlive the replay window.

Maximum replay window. A duration your operation will still accept. After it, the payload expires into a skipped state with a reason, not back into an infinite retry. Do not publish a studio default here. A payment notice and a low-stakes cache refresh should not share a window just because they share a broker.

Force-skip authority. The role that may mark a payload skipped without applying it, and the reason they must record. Force-skip is not delete. The payload and the reason stay readable.

Add two lines the page often forgets. Already committed: what you must check before replay so you do not repeat a side effect the first attempt finished. Poison mark: how many failed replays, or which error class, moves the item from “retry” to “human only.” The count is yours to set. Write it down so the platform does not imply infinity.

The spec is incomplete if owner, ordering rule, dedupe key, window, or force-skip role is blank. Incomplete means you do not turn on automatic replay. Manual investigation can continue. Automation waits.

How the work moves

Write the replay spec before automation

  1. Name the edge

    One inbound or outbound webhook, one business object, one allowed side effect.

  2. Name the owner

    A role that can see the parked payload and is accountable if it sits.

  3. Choose the order rule

    Strict order, last-write, or independent—one sentence, not a debate left open.

  4. Set key and window

    Dedupe key, where applied keys live, and the replay horizon your team will honor.

  5. Name force-skip

    Who may decline the payload without deleting it, and the reason they must record.

Illustrative workflow, not a client screenshot or measured result.
A retry without an owner is just a second chance to write the wrong side effect.

Run replay as a named action

Treat the life of one payload as a short sequence. You can walk it on a whiteboard before any worker code exists.

  1. Receive. The handler either commits the side effect and records the dedupe key, or parks the payload with the error class and the owner. Parking is success of the capture, not success of the business action.
  2. Park. The dead letter shows object id, dedupe key, first-seen time, last error, and whether any side effect might already have committed. If you cannot tell that last point, the spec’s “already committed” line is unfinished.
  3. Decide. The owner replays, skips, or waits. Waiting is allowed inside the window. Waiting past the window without a decision is drift.
  4. Replay. A new attempt, linked to the original, under the ordering rule. If strict order is in force, siblings stay blocked. If last-write applies, replay is refused when a newer message has already been accepted.
  5. Close. Applied, skipped, or expired. Each close has an actor (human role or the named job) and a time. The original error text remains.

Automatic replay is allowed only inside the window, only when “already committed” is checkable, and only for error classes you listed as safe. Everything else stays with the owner. A platform default of “retry five times with backoff” is not this spec. Backoff without the five fields just spaces out the damage.

When you test, use one poisoned payload in a non-production environment. Show that a second delivery does not double the side effect, that a force-skip leaves the payload readable, and that an expired item does not re-enter the handler because a deploy restarted the consumer. If you cannot show those three, the spec is still a draft and the job should not run unattended.

Keep the filled page next to the integration note for that edge. When the producer changes the payload, re-read the dedupe key before you congratulate the new version. Schema drift is how a careful key becomes a key that never matches.

What replay must not quietly do

These are delivery-shaped composites, not client telemetry and not Howcast or Sutton|Place incident logs.

Replay as duplicate. The first attempt created the shipment notice or advanced the record. The second attempt does it again because the dedupe key was never stored, only logged.

Replay as overwrite. Last-write was the real business rule, but the worker applied an older payload after a newer one had been accepted. The record moved backward and no one saw an ordering decision.

Skip as delete. Someone cleared the queue to go home. The payload is gone, so the audit of who declined the business action is gone with it.

Infinite poison. The same payload fails, retries, fails, and pages a human who has no authority to skip and no page that explains the side effect. The alert is not a policy.

Shared window. Every event on the bus inherits one retry horizon. A stale cache refresh and a state-changing callback become the same operational object.

When you see these, do not add more retries. Return to the five fields. If the ordering rule is still an argument, stop automatic replay until it is a sentence. If force-skip has no role, you have built a parking lot with no exit except a database console.

The unreproducible asset is the filled TWIMCO Webhook Dead-Letter Replay Spec for your edge: your object, your key, your window, your skip role, and the three test notes from a poisoned payload. A generic essay about idempotency does not carry those lines. Competitors can advise you to “make webhooks idempotent.” They cannot reproduce the page you attached to the integration, with blanks closed against your producer’s actual id.

Use the spec beside the contract with the other system, not instead of it. A vendor’s retry documentation is a claim. Your spec is what your consumer will do when that claim is late, duplicated, or wrong. On a partnership shaped like Sutton|Place, the useful habit is that a check and a decision stay inspectable. A replay is a decision about a failed check. Write it so a person who was not in the incident can see what was attempted, what was skipped, and who had the right to say so.

Decision guide

Silent retry versus a named replay

ConsiderationLooks like handlingIs a replay policy
Retry settingA platform backoff number with no owner.A role, a window, and an error class list.
Same event twiceThe handler runs again and hopes.A stored dedupe key and an already-committed check.
Old payload laterWorkers apply whatever is next.An ordering rule that can refuse a stale replay.
Clearing the queueSomeone deletes poison to stop the noise.Force-skip leaves payload, actor, and reason readable.
Decision framework: assess these conditions against your own edge.

Put this into practice

  1. Pick one webhook edge that already fails in a way your team talks about in chat.
  2. Fill edge, owner, ordering rule, dedupe key, replay window, and force-skip role. Leave unknowns blank.
  3. Write the “already committed” check and the poison mark in one sentence each.
  4. Refuse automatic replay until the five required fields are filled and the three non-production tests have been seen: no double side effect, skip remains readable, expired items stay expired across a restart.
  5. File the page with the integration note. If the blanks are really about ownership between systems, bring them to a scoping conversation for business process automation or the application work under web applications. Ground the standard in how Sutton|Place keeps an action attached to an actor.

A broker will offer you retries. The spec is how you decide which of those retries you are willing to mean.

twimco.

We build alongside the people doing the work.
Bring us the problem you’re thinking about.

Related work

Sutton Place ↗︎Howcast ↗︎