Sensor Properties

Oracle strength

How confidently does it know that something is wrong?

A compiler has maximum oracle strength because the implementation cannot argue with it. A complexity metric has minimum oracle strength because high complexity doesn't prove anything is wrong — it just suggests increased risk. The scale is ordinal (minimum → low → medium → high → maximum): a sensor two rungs up is stronger, not "twice as strong."

Independence

Can the thing being evaluated manipulate the sensor?

This is extremely important for agents. A model writing tests/ is allowed to write tests that make itself pass. The producer and evaluator should be separated wherever possible.

Scope

What level of the system does it tell us about?

Feedback latency

How long until the sensor tells you something?

Actionability

Does it merely say "bad" or does it tell you what to fix?

Three values, in order of how much the feedback directs the next action:

In Böckeler's framing, the interesting frontier is guiding sensors, where the feedback itself tells the agent what to do next.

Predictive vs retrospective

"This is wrong" or "this looks like things that became wrong before"?

Predictive sensors fire before the code ships — a compiler error, a failed test, a mutation that survives. Retrospective sensors fire after — they tell you that past changes look like changes that caused trouble before: revert rate, incident correlation, escaped defect rate.

This dimension is when the signal arrives, not what kind of feedback it gives. That is a separate axis — actionability: blocking, exploratory, guiding. The two are correlated but not the same: most predictive sensors gate (a compiler error blocks the build), and most retrospective sensors warn (revert rate is a signal, not a gate). But the correlation is not a rule. Build provenance & SBOM is retrospective — it fires after the build — and blocking: an unattested artifact does not ship. A retrospective sensor can gate; a predictive sensor can merely warn. Read the two dimensions independently.

You don't need to understand FooManagerFactoryImpl. You can observe: 27 changes in six months, 8 reverts, 4 incidents, touched by 11 teams. That's a retrospective signal — a black-box sensor of maintainability.

The catalog splits roughly evenly: predictive sensors catch bugs before they ship; retrospective sensors tell you where the bugs came from. Both matter — a sensor stack with only predictive sensors has no feedback loop; one with only retrospective sensors has no gate.

Combining sensors

Does a second sensor add anything?

The intuitive model of a sensor stack is additive: every green check is a little more evidence that the system is correct, so more sensors mean more confidence. The safety-case literature tested that model and it failed. Knight and Leveson had 27 versions of one program written independently from the same specification and ran a million tests against them. Each version was reliable on its own, but they failed together far more often than independence predicts, because their authors had misread the specification in the same ways. Littlewood and Wright showed the stranger case formally: a second line of evidence that entirely supports the first can leave you less confident than the first alone, because of what it reveals about an assumption the two share.

The catalog has its own instance. Line coverage and branch coverage sit in the same family, at the same rung of the stack, reading the same instrument. Counting them as two sensors is counting one sensor twice. The independence dimension does not catch this, and is not meant to: it asks whether the producer can manipulate the sensor, not whether two sensors share a blind spot.

So the composition rule this catalog uses is not accumulation. It is what the assurance-case field calls eliminative induction: confidence comes from ruling out specific ways the system could be wrong, and a check that rules out nothing new adds nothing, however green it is. The question to ask of every sensor in a stack is what doubt does this eliminate that the others leave open? Each entry opens by naming what it measures, and its "What it cannot detect" section names the doubts it leaves live. A stack is read from the second list, not the first. The what each sensor proves draws that answer for the whole catalog: 19 doubts, and for each one the sensors that close it, the sensors that only reveal it afterwards, and the sensors whose green reading is mistaken for closing it.

The same literature backs the meta-signal that most entries name at the end of "How it gets gamed": the second-order reading that shows a sensor going blind while its first-order reading stays green. Rushby's Bayesian model of a test-and-verify argument finds that a higher pass rate moves belief about the test oracle more than belief about the system: green tests are mostly evidence that the tests are in good shape. That is why a sensor that has stopped eliminating its doubt still reads green, and why the catalog tracks the reading that shows it going blind.

Knight & Leveson, An Experimental Evaluation of the Assumption of Independence in Multiversion Programming, IEEE TSE SE-12(1), 1986. Littlewood & Wright, The Use of Multi-legged Arguments to Increase Confidence in Safety Claims for Software-based Systems, IEEE TSE 33(5), 2007. Rushby, The Interpretation and Evaluation of Assurance Cases, SRI-CSL-15-01, 2015. Weinstock, Goodenough & Klein, Measuring Assurance Case Confidence using Baconian Probabilities, SEI, 2013.