Product · Background agents · release watch
Every deploy, watched for days
So a drifting release is caught as a release, not as an outage three days later.
ewakeapp06:41Regression from a release
checkout-api error rate drifting since deploy #4821
0.41% → 1.94% · 2 days · no monitor firing
Tags: env:production service:checkout-api
Release watch
deploy #4821 · watched since Monday
From 03:10 today the error rate on checkout-api is running four times its baseline, and every failure is on the code path #4821 changed. Nothing has crossed a threshold yet, so no monitor has fired and nobody has been paged.
I compared it against the two releases either side of it: same traffic, same hosts, same hour of day. Only this one drifts. At the current rate it reaches the 2% alert threshold some time tomorrow afternoon, during peak, on the checkout path.
I have opened an investigation and I am watching it every five minutes.
What a background agent actually does
Watched past the first hour
Most monitoring stops paying attention once a deploy looks clean. Background agents keep watching the services it touched for days, not minutes.
Drift, read against its cause
A slow rise in error rate three days after a release is read as that release drifting, not as an unexplained anomaly with no history attached.
Nothing pages until the signal is real
A watched deploy doesn't mean a noisier on-call. It means the eventual page, if it comes, already has three days of context behind it.
The map updates on its own
What a release actually touched in production gets folded back into the live map automatically, so the next incident starts from better information.
The best incident is the one that never happens.
Thirty minutes, your own production, one perimeter. You will know inside a week whether it holds.