Catalog › Adversarial › Property-Based Testing

Property-Based Testing

Adversarial High oracle Predictive

You state a property that should hold for every input, and the tool generates inputs trying to break it. Where example-based testing checks one case you thought of, property-based testing searches the space of cases you didn’t.

Properties vs examples

A property is a universal statement. Here it is stated against a my_sort that orders numbers lexicographically — sorted(xs, key=str), a bug that hides behind any example whose inputs are non-negative single digits:

@given(st.lists(st.integers()))
def test_sort_is_ordered(xs):
    ys = my_sort(xs)
    assert ys == sorted(ys)

The reading is one falsifying input, already shrunk. Hypothesis runs the property until it fails, reduces the failure, and reports only the reduced case — never the input that first broke it:

xs = [-1, -2]

    @given(st.lists(st.integers()))
    def test_sort_is_ordered(xs):
        ys = my_sort(xs)
>       assert ys == sorted(ys)
E       assert [-1, -2] == [-2, -1]
E         At index 0 diff: -1 != -2
E         Use -v to get more diff
E       Failing test case: test_sort_is_ordered(
E           xs=[-1, -2],
E       )

Two elements and a minus sign, verifiable by hand in seconds: that reduction is the actionable part of the sensor, and it is invisible in the output: the larger inputs that also failed are never printed. (The banner is version-dependent — Hypothesis 6 prints Failing test case: where older releases printed Falsifying example: — so grep for the parameter line, not the banner.)

How it differs from metamorphic testing

Property-based testing and metamorphic testing are siblings in the adversarial family. Property-based testing states a property directly (f(x) == f(-x), sort(sort(xs)) == sort(xs)) and relies on the generator to find an x that breaks it. Metamorphic testing states a relation between outputs — you don’t know the answer, only how the answer should change when the input changes. In practice most property-based test suites contain metamorphic relations (idempotence, commutativity, round-trip) and the distinction is mostly about whether you can name the property outright or only the relation between two calls.

How it gets gamed

  • Weaken the property. Replacing equality with “same length,” or adding assume filters that discard failing inputs, makes the property hold while detecting less. The test still runs.
  • Shrink the campaign. Cutting the example count until failures stop appearing keeps the sensor’s name and discards its reach.
  • Label violations as flaky. A property that fails on one input in a thousand is failing; retrying until it passes converts a finding into noise.

The meta-signal is the ratio of discarded (assumed-away) examples to generated ones. As it climbs, the property is being strangled.

Response playbook

When a property fails:

  1. Work from the shrunk example. The minimized input is the one a human can verify by hand in seconds. If you can’t hand-check it, shrink further before debugging.
  2. Decide which side is wrong. Usually the implementation. Occasionally the property overclaims, and the fix is a corrected property, written with the reason, not a deleted test.
  3. Fix the implementation and re-run the campaign. A property violation is rarely one input wide; the same bug usually breaks a neighborhood.
  4. Pin the counterexample as a regression test. The shrunk input is a free example-based test that runs in milliseconds and guards the fix forever.

What it cannot detect

Property-based testing can only check properties you state. If a function has no obvious property, the generator has nothing to falsify. It also cannot detect missing behavior — if a feature is absent, there’s no property to check against it. And unlike fuzzing, it assumes you can characterize correctness; fuzzing finds crashes even when you can’t write down a property.

Adversarial Fuzzing Adversarial Metamorphic Testing Test Effectiveness Mutation Testing Behavioral Example-Based Tests Family Adversarial
Categories: Adversarial

References

Publications

Tooling