Product · On-call agents
The agent that takes the page first
Triage, context and a named cause, in the channel, before anyone opens a laptop.
ewakeapp13:42Root cause found for the global 502 spike
Twenty-seven minutes from the CPU spike to a named cause, with the obvious explanation ruled out.
edge-gateway CPU at 100% across every region
4% → 100% CPU · all regions · 13:42 UTC
Tags: env:production service:edge-gateway
Datadog Alert
What happened: CPU on every edge machine jumped from a few percent to 100% within about a minute of a new rule going live, and stayed pinned there for the next 27 minutes.
Customer impact: Around 82% of requests across the network came back as 502s until the rule was pulled, effectively a global outage for anything sitting behind it.
Root cause: The new rule shipped with .*(?:.*=.*). Its nested quantifiers triggered catastrophic backtracking on ordinary requests, turning routine traffic into a CPU-exhausting workload on every machine that evaluated it.
Wasn't this just a DDoS? The traffic graphs looked like one.
Request volume barely moved before the spike; it's processing time per request that exploded, not the number of requests. Every crashing worker's trace names the same WAF rule ID, and disabling only that rule brings CPU back to baseline just as fast as the global kill switch did.
What an on-call agent actually does
Every page lands here first
On-call agents watch the same alerts your team does, and get to the channel before the first human does.
Context arrives with the page
The affected service, the most recent deploy and the blast radius are pulled together automatically, not requested after the fact.
A named cause, not a guess
Hypotheses are checked against live production before anything is posted, so what lands in the channel is a conclusion, not a suggestion.
You can push back
Ask why it thinks so and it defends its reasoning with the evidence it used, not with a restated confidence score.
The best incident is the one that never happens.
Thirty minutes, your own production, one perimeter. You will know inside a week whether it holds.