Catalog › Adversarial › Metamorphic Testing

Metamorphic Testing

Adversarial High oracle Predictive

You don’t know the answer, but you know how the answer should change. This is a particularly beautiful sensor because you don’t need an oracle.

Metamorphic relations

A metamorphic relation is a statement about how the output of a function should change when the input changes in a specific way:

sort(sort(x))       == sort(x)         # idempotence
decrypt(encrypt(x)) == x               # round-trip
sort(x) == sort(shuffle(x))            # order-independence

# Numeric examples:
abs(x) == abs(-x)                      # sign-independence
abs(x + y) <= abs(x) + abs(y)          # triangle inequality
max(x, y) == max(y, x)                 # commutativity

You don’t need to know what sort(x) returns. You just need to know that sort(sort(x)) should equal sort(x). If it doesn’t, something is wrong — and you’ve found a bug without ever needing to compute the correct answer.

Metamorphic testing is the answer to “how do you test something when you can’t compute the expected output?” Fuzzing explores the input space. Mutation testing perturbs the implementation. Metamorphic testing perturbs the input and checks relations between outputs. All three are adversarial — all three try to make the system fail.

In practice

A reading is a relation violation, already reduced. The relation below is order-independence — sorting a list and sorting its reverse must agree, which is checkable without knowing what the sorted answer is — run against a my_sort that makes one bubble pass instead of a full sort:

xs = [0, 0, -1]

    @given(st.lists(st.integers()))
    def test_sort_is_order_independent(xs):
>       assert my_sort(xs) == my_sort(list(reversed(xs)))
E       assert [0, -1, 0] == [-1, 0, 0]
E         At index 0 diff: 0 != -1
E         Use -v to get more diff
E       Failing test case: test_sort_is_order_independent(
E           xs=[0, 0, -1],
E       )

Reading it well:

  1. The named relation is the oracle. The failure says which relation broke (idempotence, round-trip, commutativity), which tells you what kind of bug to look for before you look at the code.
  2. The printed case is already the shrunk one. Hypothesis reports the reduced input and nothing else; the larger inputs that also failed never reach the console. If the printed case is still too big to hand-check, the shrinker was blocked — usually by an assume filter or a test that is not deterministic.
  3. A relation that never fails deserves a glance. It may be a strong invariant, or it may be vacuous. Check that it would have fired on a known-bad version of the code.

How it gets gamed

  • Weaken the relation. Replacing equality with “same length,” or adding assumption filters, makes violations disappear by narrowing what the relation claims. The test still runs and now detects less.
  • Shrink the campaign. Cutting the example count until failures stop appearing keeps the sensor’s name and discards its reach.
  • Label violations as flaky. A relation that fails on one input in a thousand is failing; retrying until it passes converts a finding into noise.

The meta-signal is the ratio of discarded (assumed-away) examples to generated ones. As it climbs, the relation is being strangled.

Response playbook

When a relation is violated:

  1. Work from the shrunk example. The minimized input is the one a human can verify by hand in seconds. If you cannot hand-check it, shrink it further before debugging.
  2. Decide which side of the relation is wrong. Usually the implementation. Occasionally the relation overclaims, and then the fix is a corrected relation, written with the reason, not a deleted test.
  3. Fix the implementation and re-run the campaign. A relation violation is rarely one input wide; the same bug usually breaks a neighborhood.
  4. Pin the counterexample as a regression test. The shrunk input is a free example-based test that runs in milliseconds and guards the fix forever.

What it cannot detect

Metamorphic testing can only check relations you know. If a function has no obvious metamorphic relations, this sensor has nothing to test. It also cannot detect missing behavior — if a feature is absent, there’s no function to check relations on.

Adversarial Fuzzing Test Effectiveness Mutation Testing Adversarial Property-Based Testing Family Adversarial Family Invariants
Categories: Adversarial Oracle-Free

References

Publications

Tooling