Product · Incident agents
From hypothesis to a scoped pull request
Hypotheses checked against live production, the blast radius drawn, and the fix written as a scoped pull request.
ewakeagent11:08Connection pool exhaustion on payment-api
Snapshot
p99 on payment-api rose to 2.31s after #4821 lowered max_size from 32 to 8 in services/payment-api/db.py:41. Reverting the two values clears the waits when replayed against the last hour of traffic. Change is one file, one service; nothing else reads this pool and no migration is involved.
Proposed change
services/payment-api/db.py
Validation
Hypothesis 1 (High confidence)
Restoring max_size=32 removes the p99 regression without other effects
Evidence:
- Replaying 11:00-12:00 traffic at the old pool size clears every wait; at the current size 11% still queue [ref]
- Callers of get_pool() are limited to payment-api; no other service imports the module [ref]
- Value matches the one in production for 14 months before #4821 [ref]
Knowing the cause is not the same as knowing what to do about it. The gap between the two is where most of an incident actually goes: working out how far the damage reaches, deciding whether the obvious fix is safe, and writing something you are willing to merge at two in the morning.
An incident agent works that gap. It starts from the named cause and does the checking a careful engineer would do before touching production, then writes the change as a scoped pull request against the boundaries you set.
The pull request is a proposal, not a decision. Auto merge is off by default, and stays off until you turn it on for a service.
What an incident agent actually does
Hypotheses, ruled out
Every candidate cause is checked against live production data (logs, traces, metrics) until only one survives, not asserted from a plausible pattern.
The blast radius, drawn
Which services, which customers, which downstream calls are affected: mapped out before any action is proposed.
The fix arrives as code
A scoped pull request, opened by ewake, with the diff, the reasoning and the tests it ran attached, not a paragraph telling a human what to type.
Your rules decide what merges itself
Auto-merge is scoped per service. Where it's off, the PR waits on the approval you already require: bounded autonomy, not unattended production changes.
How it works toward a fix
- 01
It validates the hypothesis against production
Before anything is written, the explanation is checked against what your systems are actually doing right now. A cause that only holds in theory does not get a patch.
- 02
It maps the blast radius
Which services depend on the broken one, which of them are already degraded, and who owns them. This is the part that decides whether a fix is small or whether it needs a human in the room.
- 03
It writes the change inside your boundaries
Scoped to the services you have allowed, in the style of the surrounding code, as a pull request against your repository in GitHub or GitLab.
- 04
It shows the reasoning with the diff
The evidence, the alternatives it discarded and its confidence arrive with the change, so reviewing it does not mean reconstructing the investigation first.
Where it stops
Nothing merges on its own
Auto merge is off by default and enabled per service. Until you enable it, every change waits for a human to approve it.
You set what it may touch
Write access is granted per service and revoked the same way. What ewake can act on alone, what needs approval and what it may only suggest are your rules.
Repeats need to be earned
A pattern your team has already resolved once, under rules that allow it, can be handled again automatically. ewake tells you what it did.
The best incident is the one that never happens.
Thirty minutes, your own production, one perimeter. You will know inside a week whether it holds.