Canary Analysis
Does the new version behave differently from the old version? A sensor of behavioral drift between deployments, measured on real traffic.
Canary analysis routes a small percentage of production traffic to the new version and compares its behavior to the old. If error rates, latency, or invariant violations diverge, the canary fails.
In practice
A canary reading is a side-by-side comparison of the same metric across two populations, usually scored into a single pass/fail verdict rather than eyeballed per metric:
| Metric | Baseline v2.40.7 | Canary v2.41.0 (5%) | Deviance | Tolerance | Verdict |
|---|---|---|---|---|---|
| Error rate | 0.12% | 0.84% | 7.0x | 2x | FAIL |
| p99 latency | 212 ms | 231 ms | +9% | +10% | pass |
| Success rate | 99.88% | 99.16% | -0.72 pp | -0.10 pp | FAIL |
| CPU | 0.41 cores | 0.43 cores | +5% | +20% | pass |
| Overall score | 42 / 100 — ROLLBACK |
The tolerance column is the part that was decided before the canary ran. Read the table without it and the verdicts look arbitrary: a 9% latency bump passes and a 0.72-point drop in success rate fails, with nothing on the page saying why. Read it with the column and every verdict is arithmetic.
Reading it well requires four habits:
- Compare, don’t threshold. “Error rate 0.84%” means nothing on its own; the baseline at 0.12% is what makes it a signal. A canary judged against an absolute bar silently passes when everything degrades together.
- Set the tolerance before the canary runs. Every metric needs a band written into the analysis config in advance. Note where that bites in the table above: latency came in at +9% against a +10% band and passed. Decided afterwards, +9% is whatever the person reading it needs it to be — a regression if they were suspicious, a Tuesday if they want to ship. Pre-set bands are also what stop the argument that a failing canary was “close enough.”
- Know which metrics are allowed to move. Latency within the noise band is normal churn. Error rate and invariant violations are not. If the score weights treat them alike, noise drowns the real divergences.
- Match the populations. The canary and baseline must see comparable traffic. Comparing a canary that only handles new signups against a baseline serving the whole fleet reads drift into routing, and the “drift” is the routing.
The verdict is only as good as the metrics underneath it, which is why the canary is downstream of the observability work rather than a replacement for it.
What it cannot detect
Canary analysis can only detect differences in metrics you’re measuring. Unknown unknowns require high-cardinality events to investigate after the fact.
Related sensors
References
Publications
- Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla —
- Holistic Configuration Management at Facebook —
Tooling
- KayentaNetflix's automated canary analysis
- Argo RolloutsKubernetes progressive delivery
- FlaggerKubernetes progressive delivery and canary