Where you are

Compiler, linter, unit tests, a coverage number. Every merge is green. The first you hear of a bug is a user telling you about it.

You run Compiler, Linter, Example-Based Tests, Line Coverage. Already closed: wrong logic, not internally consistent, unreached by tests. 3 of 19 doubts.

Add in this order

Each step closes a doubt the previous ones left open.

  1. Smoke Tests Behavioral closes does not run where deployed

    Nothing you run today touches the deployed artifact. Four requests against the fresh deploy, one per vital sign, take an afternoon to wire up and close the doubt that the thing you tested is the thing that is running.

  2. Observability Events Runtime reveals unspecified property

    Closes nothing before shipping. It is the only sensor on this list that reveals unspecified property, which is where the bugs your users report currently live, and it is the data the next step and the last step both read from.

  3. Business Invariants Invariants closes unjudged observation

    The events from the previous step say what happened; nothing yet says whether it was right. One query per domain promise over those events is the judge the events lack, and it runs on real data rather than on the cases your unit tests thought to try.

  4. Mutation Testing Test Effectiveness closes unasserted execution

    Fourth because it tells you nothing new about the system; it improves the tests you already have. Your coverage number counts lines that ran, and this is the first step that asks whether any test would notice if those lines returned the wrong answer.

  5. Canary Analysis Change closes untested conditions, emergent failure, unintended change

    Three doubts from one sensor, and it depends on steps two and three: the events supply the metrics it compares between versions, and the invariants supply the violations that count as a divergence. Without them the canary has a tolerance band and nothing to apply it to.

After these steps: 9 of 19 doubts closed, 1 more revealed after the fact. Still untouched: wrong specification, missing behavior, correlated blind spot, late effects, falls over with a dependency, cannot carry the load, nobody knows where the coupling is, nobody knows where the risk sits, nobody can find out why.

What not to add yet

  • Branch Coverage

    Same doubt as line coverage. It tells you which side of a conditional never ran, and still says nothing about whether the side that ran was asserted on.

All plays