Composition
A minimal stack for a batch or data pipeline
The shape
Scheduled jobs that read, transform, and write data. Wrong output is silent, and someone downstream discovers it weeks later.
The stack
One row per doubt. No two rows do the same job.
| Doubt | Sensor | Why this one |
|---|---|---|
| not internally consistent | Schema Validator Structural | Validate on read and on write; upstream shapes drift between runs. |
| unintended change | Snapshot Tests Behavioral | Golden output on a fixed input catches a transform that moved. |
| wrong logic | Metamorphic Testing Adversarial | The right output is unknown; a rerun must still not change it. |
| unasserted execution | Mutation Testing Test Effectiveness | Transform tests that assert nothing let wrong output through silently. |
| does not run where deployed | Smoke Tests Behavioral | One partition through the real scheduler and credentials first. |
| unjudged observation | Database Invariants Invariants | The engine checks every row written; nothing to schedule or scrub. |
| cannot carry the load | Load Testing Runtime | Run it at the biggest day's volume; falling behind is the failure. |
Left open, on purpose
-
A job that dies when its source is down and reruns from its checkpoint is the degradation you want. Spend on idempotent reruns before spending on fault injection.
-
Both closers split live traffic between two versions, and a scheduled job has no traffic to split.
-
Stages hand off through the store, and the schema row validates each handoff. Add integration tests when a stage starts talking to something other than the store.
What this stack is mistaken for proving
7 sensors close 7 of 19 doubts. It says nothing about unspecified property, wrong specification, missing behavior, correlated blind spot, late effects, unreached by tests, nobody knows where the coupling is, nobody knows where the risk sits, nobody can find out why. A green board from this stack is not evidence against any of those.
A service reports its own failure with a 500. A pipeline reports nothing; the wrong rows land and the next job reads them. So most of this stack runs before the job is scheduled, and the two rows that run in production, the schema check and the database constraints, both refuse the write, so the bad row never lands for anyone to discover.