Catalog › Change › Feature Flag Exposure Telemetry

Feature Flag Exposure Telemetry

Change High oracle Retrospective

A change behind a flag is not a change anyone has experienced. Flag evaluation streams answer “who is actually seeing the new behavior, right now?” — the difference between deployed and live, and the ground truth underneath every canary analysis claim about a rollout’s state.

In practice

The core reading is an exposure matrix: which flags are evaluating true, for whom, and how stale the flag itself is.

Flag State Exposure Target Age Coupling
new-checkout-flow ramping 5% 100% 4 days 0 flags
dark-mode-v2 launched 100% n/a 14 months 0 flags
legacy-export-path decaying 100% off 26 months 3 flags
exp-pricing-b experimenting 50% n/a 3 weeks 0 flags

Reading it well:

  1. Deployed is not live. A flag at 0% exposure means nobody has seen the code, whatever the deploy dashboard says.
  2. Flags launched to 100% with no removal date are dead code with a gate. They accumulate coupling and mask which path actually runs.
  3. Coupled flags are a compound condition. When flag A only matters because flag B is on, you are testing one corner of a state space, not two features.
  4. Tie exposure to the verdict. Exposure tells you the denominator; the canary analysis tells you what happened to that population.

How it gets gamed

  • Launch and walk away. Flipping a flag to 100% counts as “done” while nobody removes the gate, so the codebase accrues dead branches that still count as live features in reporting.
  • Never delete the flag. Old flags at 100% exposure pad feature counts and hide which paths real traffic uses; removal is the work that nobody schedules.
  • Ramp only in quiet hours. Exposure telemetry read during a maintenance window shows a calm rollout that nothing real ever exercised.
  • Split flags to dodge review. A feature gated by three coupled flags is harder to reason about and easier to slip past a reviewer than one flag at full exposure.

The meta-signal is the ratio of launched-but-unremoved flags to active ramps.

Response playbook

When the exposure matrix reads badly:

  1. Kill the stale flags. Every flag launched to 100% more than a release cycle ago gets a removal task; delete the gate and the dead branch it guards.
  2. Map the coupled flags. For any flag whose behavior depends on another flag, write down the actual state space and decide which combinations are tested. Untested corners are the incident waiting to happen.
  3. Put an expiry date on every new flag. At creation time, record the target exposure and the removal date; telemetry makes both easy to audit.
  4. Reconcile exposure against reality. Compare the matrix with canary analysis verdicts; exposure without a behavioral verdict is a rollout nobody verified.

What it cannot detect

Whether the new behavior is correct — only where it is active. Exposure is the denominator; behavioral evidence still has to come from synthetic monitoring or real traffic.

Change Canary Analysis Runtime Continuous Profiling Behavioral Synthetic Monitoring Change A/B Testing
Categories: Change Production Sensors

References

Publications

Tooling