An industry resource

What independent observations
would cause us to believe
this software is correct?

Software is increasingly an opaque artifact. We cannot — and often do not want to — fully understand every implementation. The Software Observatory is a catalog of epistemic sensors: the observable signals that reduce uncertainty about whether a system is correct, maintainable, and behaving as intended. Not "code quality metrics." Measurement instruments pointed at different failure modes.

Browse the catalog → Open the atlas

No single sensor measures correctness. Coverage measures execution. Mutation testing measures test sensitivity. Types measure a particular class of structural inconsistency. Contracts measure boundary assumptions. Observability measures what actually happened and preserves enough dimensionality to investigate unknown unknowns.

They are all measurement instruments pointed at different failure modes.

Eleven sensor families

The catalog is organized into eleven families, each asking a different question about the system. Together, they form a mesh of independent evidence — no single sensor is sufficient, but the combination constrains uncertainty from multiple directions.

01

Structural

"Is this artifact internally coherent?"

Compiler, type checker, linter, formatter, schema validator

 02

Behavioral

"Does it do what we expect?"

Unit, integration, E2E, contract, snapshot tests

 03

Test Effectiveness

"Do our tests actually detect failures?"

Coverage, diff coverage, mutation testing

 04

Invariants

"What must always be true?"

Balance >= 0, every FK valid, every request has one ID

 05

Adversarial

"Can we make our evidence of correctness fail?"

Fuzzing, mutation testing, fault injection, chaos

 06

Runtime

"What is it actually doing?"

Logs, traces, metrics, profiles, high-cardinality events

 07

Change

"What did this change actually affect?"

Diff coverage, API compatibility, canary, shadow traffic

 08

Architecture

"Is the system becoming harder to reason about?"

Dependency graphs, coupling, fitness functions, hotspots

 09

Evolution

"Does this look like changes that caused trouble before?"

Revert rate, regression rate, churn, incident correlation

 10

Human Comprehension

"Can another observer understand and challenge this?"

Review, explainability tests, documentation drift, onboarding

 11

AI-Generated Code

"What evidence do we have that this change is safe?"

Agent sensor stacks, computational gates, independence

The confidence stack

Sensors are not peers. They form a hierarchy — from the cheapest, most certain signals at the bottom to the most expensive, most meaningful signals at the top. Each layer depends on the layers below it.

User outcome Does the system produce the intended result for users?
Production behavior What is it actually doing in the real world?
Canary / shadow Does the new version behave differently from the old?
Integration tests Does it work connected to its real dependencies?
Behavioral tests Does it do what we expect for given inputs?
Property / metamorphic Does it obey generalized properties across input spaces?
Mutation testing Would our tests detect plausible wrong implementations?
Static analysis / types Is it internally coherent and structurally valid?
Compilation Is it a valid inhabitant of the language?
Source text The opaque artifact itself.
Open the atlas →
Test Effectiveness High oracle strength

Mutation Testing

Take if user.is_admin: allow() and mutate it to if not user.is_admin: allow(). If all your tests still pass, your tests did not actually establish the behavior you thought they established. Mutation testing is a sensor of test sensitivity rather than test presence — and it may be one of the most interesting sensors in the entire catalog.

Read the full entry →

Catalog entries