Diagnosis engine
A written brief on every incident, by the time you reach the laptop.
The diagnosis engine reads telemetry, system events, and recent changes the moment an alert fires. It produces a short engineering brief — what broke, the most likely cause, and the first three things worth trying — and attaches it to the incident before the page reaches your phone.
Memory on prod-api-01 grew 480MB in 2h. Correlates with deploy v3.1.7 at 09:15 UTC. Worker process is leaking upstream connections on client disconnect.
// add upstream keepalive + reload proxy_read_timeout 30s; proxy_send_timeout 30s; systemctl reload nginx
prod-db-01 growing 2.1 GB/day. 90% in 8 days.
● WAL filesstaging-api-02 jumped 120ms → 890ms after deploy.
● N+1 queryredis-cache OOM × 3 this week, peak 18–20 UTC.
● 256MB → 512MBPipeline
Four stages between the threshold and the page.
Most monitoring stops at “something crossed a line.” This pipeline begins there — joining signal, attribution, and prose into a single readable brief that lands with the alert.
What it produces
What lands in the channel when the page fires.
Sample briefs
Three real incidents, three real briefs.
Reproduced from production hosts, with identifiers redacted. This is the quality and tone of brief that arrives with every alert.
CRITICAL
OOM kill on db-prod-01 (postgres, PID 4128)
Postgres was reaped by the OOM killer at 03:42 UTC after RSS climbed past 95% of system memory and held there for twelve minutes. Two signals line up: a long-running analytical query opened against the reporting schema at 03:31, and the API connection pool jumped from a steady 80 to 240 over the next eight minutes — likely a retry storm following an upstream timeout.
Suggested first actions
- Restart
postgresql.serviceand confirmshared_bufferssettles - Pull the slow-query log for 03:30–03:42 UTC and identify the analytical query owner
- Cap pgbouncer
pool_sizeor tunework_membefore the next reporting window
WARNING
Disk pressure on web-03, /var/log at 87%
/var/log climbed past 87% and is trending toward the 90% page threshold within roughly six hours at the current ingestion rate. logrotate ran at 02:00 but appears to have skipped the nginx access logs — the rotation policy points at a path that no longer exists after last week’s nginx upgrade.
Suggested first actions
- Run
logrotate -f /etc/logrotate.d/nginxand confirm rotation completes - Update the rotation glob to match the current nginx access log path
- Consider mounting /var/log on its own volume before the next upgrade window
INFO
Periodic CPU saturation on worker-02 during nightly export
CPU sat at 92% between 02:00 and 02:15 UTC, coinciding with the export-reports cron job. The same shape has repeated nightly for seven days with no downstream impact and no growth in duration. Recording for context, not paging.
Suggested first actions
- No action required at this time
- If the window starts to expand, move the job to 04:00 UTC
- Track total runtime against the seven-day baseline
Operational impact