Get ahead of incidents
Reliability agents that catch and resolve production issues
before your customers notice
Trusted every day by
search-indexer disk usage above 85%
87.4% used · 12.6% free · 30 min
Datadog Alert · env:production service:search-indexer
What ewake did
- ✓Matched 11 prior firings
- ✓Checked the 4:00 AM merge job
- ✓Compared the threshold to staging
- ✓Closed it without paging
ewakeapp#prod-reliabilityMarked as noise
Nobody paged. This monitor has never once caught a real problem.
Ewake in action
End-to-end reliability by agents that know your production
Investigate
Find out what's actually happening.
As soon as an alert fires or ewake catches a faulty release, it starts investigating autonomously: pulling and enriching context, reasoning across signals, and determining the root cause, impact, and recommended next steps.
Learn more about on-call agentsewakeagent11:06Spike of 5xx errors in payment-api
Snapshot
Error rate on payment-api rose from 0.3% to 6.8% for tenant 4471pmqx, with every failure raised from acquireConnection() in services/payment-api/db.py:41. Most likely cause: connection pool exhaustion after #4821 lowered max_size from 32 to 8. Deploy landed 4 minutes before the first error. p99 unchanged on every service that does not call this handler.
checkout 5xx rate · 2% monitor threshold
Hypotheses
Hypothesis 1 (High confidence)
The pool size change in #4821 is exhausting connections under normal afternoon load
Evidence:
- All 1,204 failures raise PoolTimeout at db.py:41, none elsewhere [ref]
- Errors begin 11:03:40, 90s after #4821 reached the last canary [ref]
- Only payment-api and checkout-api degrade; both call the changed handler [ref]
Hypothesis 2 (Low confidence)
Database saturation on postgres-main is the underlying cause
Evidence:
- Pool utilisation held at 34% and query p99 was flat through the window [ref]
- Six other services share the instance and none of them slowed [ref]
How sure are you it isn't the database? It bit us last month.
Hypothesis 2 above. postgres-main pool utilisation was 34% and query p99 flat for the whole window, and the six other services on that instance were unaffected. If it were the database they would have degraded too.
Resolve
Work toward the safest fix.
ewake validates its hypotheses against live production, maps the blast radius, and prepares a scoped fix within the boundaries you have set.
Learn more about incident agentsewakeagent11:08Connection pool exhaustion on payment-api
Snapshot
p99 on payment-api rose to 2.31s after #4821 lowered max_size from 32 to 8 in services/payment-api/db.py:41. Reverting the two values clears the waits when replayed against the last hour of traffic. Change is one file, one service; nothing else reads this pool and no migration is involved.
Proposed change
services/payment-api/db.py
Validation
Hypothesis 1 (High confidence)
Restoring max_size=32 removes the p99 regression without other effects
Evidence:
- Replaying 11:00-12:00 traffic at the old pool size clears every wait; at the current size 11% still queue [ref]
- Callers of get_pool() are limited to payment-api; no other service imports the module [ref]
- Value matches the one in production for 14 months before #4821 [ref]
Detect
Catch a bad release before it becomes an incident.
ewake watches every release, catching regressions while they are still small, before they turn into customer-facing outages, so engineers don't have to watch monitors after every deploy.
Learn more about background agentsewakeagent06:41Error rate drifting on checkout-api since deploy #4821
Snapshot
Error rate on checkout-api has run at 0.41% to 1.94% since 03:10, about four times the trailing baseline, with every failure on the code path #4821 changed. Nothing has crossed a threshold, so no monitor has fired and nobody has been paged. At the current slope it reaches the 2% alert threshold some time tomorrow afternoon, during peak.
deploy #4821 · watched since Monday
Hypotheses
Hypothesis 1 (Medium confidence)
The drift is a regression in #4821 rather than a change in traffic
Evidence:
- Compared against the two releases either side: same request volume, same hosts, same hour of day; only this one drifts [ref]
- Every failing request enters the handler #4821 modified [ref]
- Confidence is medium because 18 hours is short for a seasonality claim; the watch continues to day 5 [ref]
Nothing has been changed. Re-checking every five minutes.
Escalate
Only ping a human when one is needed.
When human judgment is required or there is customer impact, ewake wakes the on-call, with the investigation, the evidence, and the recommended action already attached.
ewakeagent11:12Potential incident — review now and consider paging on-call
Checkout is returning 5xx to real traffic and the remaining fix needs a schema migration, which is outside what this agent is permitted to run.
5xx on checkout-api after schema change in #4818
Snapshot
6.8% of requests to checkout-api have failed for 14 minutes. Root cause is confirmed: #4818 added a non-null column to orders and the writer was deployed before the backfill completed. Reverting the config alone leaves the rows written since 10:58 in a state the new schema cannot read, so the revert is not sufficient on its own.
Why this stopped here
Hypothesis 1 (High confidence)
Repair requires an ALTER TABLE on orders, which is on the deny list for this environment
Evidence:
- 1,847 rows written between 10:58 and 11:12 fail the new constraint on read [ref]
- Migration policy for env:production lists DDL as human-only; no override is configured [ref]
- Rollback of #4818 alone reproduces the read failures in staging [ref]
Paged
Got it — running the backfill now. Nothing to re-investigate.
faster time to root cause
accuracy
from alert to fix
Point ewake at your own production
Thirty minutes to connect it, one perimeter, your real alerts. You will know inside a week whether it holds, on your stack, not in a sandbox.
Trusted by engineers
Built for real production environments
“ewake proved extremely useful during production investigations, consistently pointing us to the right root cause and saving precious time, especially during on-call.”
Piyush S.
Lead Software Engineer, Brevo
Live production map
Powered by a live production map, built and maintained by ewake
ewake continuously maps your services, deploys and dependencies from the signals already flowing through your stack, so every investigation starts from what actually changed, not a stale diagram.
Integrations
Able to work with your tools
Connect your stack in minutes. Use MCP, APIs, or webhooks to connect observability, alerting, code, chat, or your own custom MCP server.
Frequently asked questions
You've got questions, we've got answers
Still have questions?
Book a quick call with the team.
Talk to us