Sensor family 03
Test Effectiveness
“Do our tests actually detect failures?”
What belongs here
Every entry here points at the test suite rather than at the system. They answer a second-order question — would these tests have noticed? — which makes this the one family whose subject is other sensors.
Contested placements
Escaped Defect Rate measures the whole delivery pipeline after the fact and could equally sit in Evolution. The coverage entries are weak oracles by construction: they can prove a line was never executed, never that a behaviour was established.
Where its signals arrive
- Mutation testing — Test stage. Would our tests detect plausible wrong implementations?
The atlas places every family on the same lifecycle grid; the framework defines the six dimensions each entry below is characterized along.
Entries (5)
Branch Coverage
Did we exercise both sides of decisions? Better than line coverage. Path coverage is better still, but usually expensive and impractical at scale.
Diff Coverage
You don’t necessarily care whether some 15-year-old module has 43% coverage. You care: did the code I just changed have evidence attached to it?
Escaped Defect Rate
Of the bugs that reached users, which ones should the test suite have caught? Escaped defect rate is the slowest and most honest measure of test effectiveness: not “would the tests catch a…
Line Coverage
Did we execute this line? Useful but weak. A project can have 90% line coverage while mutation testing finds large numbers of mutations that tests don’t detect. Coverage measures execution.
Mutation Testing
Take if user.is_admin: allow() and mutate it to if not user.is_admin: allow(). If all your tests still pass, your tests did not actually establish the behavior you thought they established.
Adjacent families
- Previous: Behavioral — Does it do what we expect?
- Next: Invariants — What must always be true?
- All of them: the complete catalog (this family is also anchored there at /catalog/#test-effectiveness).