Product · On-call agents
The agent that takes the page first
Triage, context and a named cause, in the channel, before anyone opens a laptop.
ewakeagent11:06Spike of 5xx errors in payment-api
Snapshot
Error rate on payment-api rose from 0.3% to 6.8% for tenant 4471pmqx, with every failure raised from acquireConnection() in services/payment-api/db.py:41. Most likely cause: connection pool exhaustion after #4821 lowered max_size from 32 to 8. Deploy landed 4 minutes before the first error. p99 unchanged on every service that does not call this handler.
checkout 5xx rate · 2% monitor threshold
Hypotheses
Hypothesis 1 (High confidence)
The pool size change in #4821 is exhausting connections under normal afternoon load
Evidence:
- All 1,204 failures raise PoolTimeout at db.py:41, none elsewhere [ref]
- Errors begin 11:03:40, 90s after #4821 reached the last canary [ref]
- Only payment-api and checkout-api degrade; both call the changed handler [ref]
Hypothesis 2 (Low confidence)
Database saturation on postgres-main is the underlying cause
Evidence:
- Pool utilisation held at 34% and query p99 was flat through the window [ref]
- Six other services share the instance and none of them slowed [ref]
How sure are you it isn't the database? It bit us last month.
Hypothesis 2 above. postgres-main pool utilisation was 34% and query p99 flat for the whole window, and the six other services on that instance were unaffected. If it were the database they would have degraded too.
Most of the time an on-call engineer spends on an alert goes before any debugging starts. You find the runbook, or find that there isn't one. You open the dashboard, work out which service actually owns the failing endpoint, check whether anything shipped in the last hour, and read enough logs to know if this is real. Twenty minutes later you finally know what you are looking at.
An on-call agent does that part. It watches the same alerts your team does, and by the time a human opens the thread the work of orienting is already done and written up, with the evidence attached.
At Brevo, whose transactional email team handles around 200 alerts a month, this took time to root cause on a typical high severity alert from 10 to 15 minutes down to 2 to 5.
What an on-call agent actually does
Every page lands here first
On-call agents watch the same alerts your team does, and get to the channel before the first human does.
Context arrives with the page
The affected service, the most recent deploy and the blast radius are pulled together automatically, not requested after the fact.
A named cause, not a guess
Hypotheses are checked against live production before anything is posted, so what lands in the channel is a conclusion, not a suggestion.
You can push back
Ask why it thinks so and it defends its reasoning with the evidence it used, not with a restated confidence score.
How it handles a page
- 01
It takes the alert at the same moment you do
ewake sits on the same feed as your rotation, reading from PagerDuty or OpsGenie and from Datadog, Grafana or Prometheus. There is nothing new to route and nothing to instrument.
- 02
It assembles the context before anyone asks
The affected service, its recent deploys, its dependencies and the blast radius get pulled together automatically, because ewake already holds a map of your production and does not have to go looking.
- 03
It tests what it thinks against live production
A hypothesis is checked before it is posted. That is the difference between a conclusion and a guess, and it is why what lands in the channel is worth reading rather than worth verifying.
- 04
It posts what it found, and what it ruled out
You get a named cause, a confidence score, the evidence behind it and the paths it eliminated. Ask it why and it answers with the reasoning it used, in the thread, not with the same number again.
Where it stops
Read only until you say otherwise
It reads metrics, logs, traces and code. It changes nothing in your infrastructure unless you grant write access, and that is scoped per service and revocable.
It wakes you when judgment is needed
Anything that needs a human call, or that has reached customers, goes to the on-call with the investigation already attached rather than being handled quietly.
It never trains on your data
Your telemetry, your code and your incidents are never used to train or fine tune any model, on any tier, in any deployment model.
The best incident is the one that never happens.
Thirty minutes, your own production, one perimeter. You will know inside a week whether it holds.