Progressive rollout is often sold as a safety mechanism, and then configured in a way that provides very little safety.
The common mistake
Gate on error rate alone. Errors are the loudest signal, so they are the first one people wire up — and a release that is twice as slow but returns 200s will sail straight through.
A minimum useful set
Three signals cover most of what goes wrong:
- Error rate, because it is unambiguous
- p95 latency, because degradation rarely shows up in the mean
- Saturation of whatever your bottleneck is — connections, queue depth, CPU
Dwell time matters more than steps
Teams tune the percentages and leave the dwell time at thirty seconds. But a five-percent canary needs long enough to accumulate a statistically meaningful number of requests. On a low-traffic service, thirty seconds may be four requests.
Work backwards from your traffic. If five percent of traffic takes four minutes to produce a thousand requests, dwell for four minutes.
Roll back as a traffic change
If rolling back means running the pipeline again, you will hesitate. Keep the previous release warm for the duration of the rollout so reverting is a routing decision measured in seconds.