Product · Background agents · release watch
Every deploy, watched for days
So a drifting release is caught as a release, not as an outage three days later.
ewakeagent06:41Error rate drifting on checkout-api since deploy #4821
Snapshot
Error rate on checkout-api has run at 0.41% to 1.94% since 03:10, about four times the trailing baseline, with every failure on the code path #4821 changed. Nothing has crossed a threshold, so no monitor has fired and nobody has been paged. At the current slope it reaches the 2% alert threshold some time tomorrow afternoon, during peak.
deploy #4821 · watched since Monday
Hypotheses
Hypothesis 1 (Medium confidence)
The drift is a regression in #4821 rather than a change in traffic
Evidence:
- Compared against the two releases either side: same request volume, same hosts, same hour of day; only this one drifts [ref]
- Every failing request enters the handler #4821 modified [ref]
- Confidence is medium because 18 hours is short for a seasonality claim; the watch continues to day 5 [ref]
Nothing has been changed. Re-checking every five minutes.
A regression rarely announces itself. It ships on a Tuesday, moves error rates by a fraction of a percent, and stays under every threshold you have set until the traffic pattern changes on Friday and it becomes an outage. By then the deploy that caused it is three days back and nobody is looking there.
Background agents close that window. Every release stays watched for days after it ships, compared against how the service behaved before it, so a drift that no threshold would catch is still caught while it is small enough to be a rollback rather than an incident.
This is the cheapest part of the loop, because the work it saves is the work that never has to happen.
What a background agent actually does
Watched past the first hour
Most monitoring stops paying attention once a deploy looks clean. Background agents keep watching the services it touched for days, not minutes.
Drift, read against its cause
A slow rise in error rate three days after a release is read as that release drifting, not as an unexplained anomaly with no history attached.
Nothing pages until the signal is real
A watched deploy doesn't mean a noisier on-call. It means the eventual page, if it comes, already has three days of context behind it.
The map updates on its own
What a release actually touched in production gets folded back into the live map automatically, so the next incident starts from better information.
How it watches a release
- 01
It learns what normal looks like for that service
Not one global threshold across everything, but how this service behaves at this hour on this day, which is the only baseline a small drift shows up against.
- 02
It keeps watching after the deploy goes green
The window is days, not minutes. Most regressions worth catching do not appear during the deploy, they appear the first time real traffic hits the changed path.
- 03
It ties the change back to the release
When something moves, ewake already knows which deploy landed in that window and what it touched, so the finding arrives attached to a commit rather than to a graph.
- 04
It tells you while rolling back is still cheap
A finding lands in your channel with the release, the affected service and what changed, at the point where undoing it is a decision rather than an incident.
Where it stops
It does not roll back on its own
Reverting a release is a judgment call about your customers. ewake brings you the evidence for making it.
It stays quiet when nothing moved
The point of watching every release is to say nothing about almost all of them. An agent that reports on every deploy is another thing to ignore.
It needs no new instrumentation
It reads the observability data you already collect. No SDK, no code changes, no second agent on your hosts.
The best incident is the one that never happens.
Thirty minutes, your own production, one perimeter. You will know inside a week whether it holds.