Spec the batch-failure alert before the nightly job.
Just after midnight the batch fails, and someone gets "job failed" with no counts, no sample records and no safe first step. Or it half-fails, reports success, and the gap turns up at 9 a.m. in customer care.
Before the nightly job ships, fill the TWIMCO Batch-Failure Alert Spec: who is notified, what evidence rides with the alert, and the first operator action for partial and full failure. An alert without evidence and a first action is only noise with a pager.
Nightly jobs fail loudly or silently—both need a Spec
The batch runs after midnight. It fails. Someone gets a generic “job failed” ping with no counts, no sample ids, and no safe first action. Or worse: it fails partially, marks success on the wrapper, and the gap appears at 9 a.m. in customer care.
Batch failure is an operating event. Treating it as an infrastructure afterthought produces noise or silence—neither is operable.
This note gives you the TWIMCO Batch-Failure Alert Spec: who gets notified, what evidence rides with the alert, and the first operator action for partial and full failure. Distinct from a recovery runbook library (how to recover) and a partial-failure decision tree (compensate / retry / park). The Spec is the alert contract before the job is trusted in production.
Large content and catalog pipelines—work in the shape of Howcast—make batch boundaries and failure visibility non-optional. Verification-minded partnerships like Sutton|Place reinforce the same habit: evidence and who may act stay explicit. Automation scoping often lives under business process automation.
A forthcoming studio tool may help draft these fields from a job description; until then, fill one Spec per critical batch.
Teams often invest in dashboards and skip the alert contract. A dashboard that nobody is paged to open at failure time is not an operating control. The Spec forces the page path and the first human move into the same artifact as the job definition.
Teams often invest in dashboards and skip the alert contract. A dashboard that nobody is paged to open at failure time is not an operating control. The Spec forces the page path and the first human move into the same artifact as the job definition.
Fields of the Batch-Failure Alert Spec
Job identity. Name, schedule (timezone), system of record it writes, and upstream dependencies that can fail it indirectly.
Failure classes. Full failure (job did not complete meaningful work); partial failure (some units succeeded, some did not); success-with-warnings (completed but signals need human eyes). Define each in observable terms—not “degraded.”
Notify who. Primary on-call role, backup, and business owner if customer impact is likely. Channels and quiet-hours rules. Escalation if unacked within a time you set from your own staffing—not from a borrowed SLO blog.
Evidence payload. What must arrive with the alert: run id, start/end time, counts attempted / succeeded / failed, sample failed ids (bounded), link to logs or dead-letter, last successful run id. If the alert cannot carry evidence, the Spec is incomplete—fix the pipeline before you rely on the alert.
First operator action. For each failure class: the first safe action (ack and open evidence; freeze downstream; retry named subset; park and page owner). Explicitly forbid blind full retry when duplicate risk exists.
Done-means for the alert. When may the alert be closed: evidence reviewed, next action owned, customer impact assessed. Closing because “job was restarted” without checking duplicates is not done.
The unreproducible asset is your filled Spec for a real job—the failure classes, evidence payload, and first actions competitors cannot invent from “add monitoring.”
Include whether the alert is paging or ticket-only. Not every partial failure needs a phone wake-up; some need a morning ticket with full evidence. Write the severity mapping on the Spec so on-call does not invent it. Severity without evidence still fails the Spec.
Name the log retention expectation. An alert that links to logs that expire in an hour is an alert that cannot be reviewed. Retention is part of evidence, not an infra footnote.
Sequence of work
Alert contract before the job is trusted
- Classify failure
Full, partial, or success-with-warnings—observable definitions.
- Notify
Primary, backup, business owner when impact is likely.
- Attach evidence
Counts, sample ids, links, last success.
- First action
Safe next step—never blind retry when unsafe.
A batch alert that says “failed” without evidence asks operators to start from folklore.
Design for partial failure first
Full failure is obvious. Partial failure is where teams invent folklore. Require the job to emit counts and sample failures even when the wrapper exits zero. A green exit with red units is a Spec violation.
Decide whether partial failure pages immediately or accumulates to a threshold you set from your own volume. Write the rule. Do not leave it to the on-call’s mood at 2 a.m.
Downstream consumers need freeze rules. If a failed product sync would poison the storefront, the first action may be freeze publish—not retry. Coordinate with inventory truth and catalog owners.
Idempotency notes belong on the Spec. If retry is unsafe, say so in the first-action field. If retry is safe only with a token, name the token check.
Test the alert in staging by forcing a partial failure once before production trust. An untested alert path is a decorative pager.
Clock skew and timezone mistakes create false full failures and missed partials. Write the schedule timezone on the Spec and on the alert. If the job spans regions, say which clock is authoritative for “nightly.”
When a batch feeds another batch, chain the Specs: downstream first action may be “wait for upstream ack” rather than independent retry. Uncoordinated retries amplify damage.
Operate the Spec after the first real page
Within two business days of a real page, update evidence fields that were missing and first actions that were wrong. The Spec is living paper.
Correlate with incident ownership matrices for P1/P2. The alert Spec names the first move; the matrix names clock ownership and change authority. They should not contradict.
Do not invent MTTR targets to decorate the Spec. Track whether evidence arrived and whether the first action was followed—qualitative honesty beats fake precision.
When multiple batches share a pipeline, avoid alert storms: aggregate with clear unit counts rather than one page per unit. Storms train people to ignore alerts.
Share the Spec with the people who get paged—not only with the eng who wrote the job. A Spec that lives only in a repo README will not be read at 2 a.m. Put a short link in the alert body itself.
Decision guide
Noisy page vs operable alert
| Consideration | Common alert | Spec-compliant alert |
|---|---|---|
| Payload | “Job failed.” | Counts, samples, and log links included. |
| Partial failure | Wrapper exits zero; gap found at 9 a.m. | Partial class pages with unit evidence. |
| Retry | Blind full re-run. | First action respects duplicate risk. |
| Ownership | Eng-only ack. | Business owner notified when impact likely. |
Failure modes the Spec refuses
Empty page. “Failed” with no counts or links. Evidence payload blocks this.
Blind retry. First action is “run it again” despite duplicate risk. Forbidden unless the Spec proves safety.
Success theater. Wrapper green, units red, no alert. Failure class definitions block this.
Unowned business impact. Eng acks while customers wait. Business owner notify field blocks this when impact is likely.
These are delivery-shaped composites, not client telemetry.
Put this into practice
- Pick the next critical nightly job and draft failure classes with observable definitions.
- Specify notify who, evidence payload, and first actions for partial and full failure.
- Force a partial failure in staging and confirm the alert carries evidence.
- Attach the Spec to the job’s runbook; update after the first production page.
- Use Howcast and Sutton|Place as delivery references for pipeline and verification habits; scope under business process automation.
Trust the batch only when the alert Spec is filled—not when the happy path ran once.
Schedule a quarterly read of Specs for the top jobs. Jobs change; Specs rot. A thirty-minute review beats discovering missing evidence during an incident retrospective.
We build alongside the people doing the work.
Bring us the problem you’re thinking about.