Monitoring and Alerting That Engineers Do Not Ignore
Most monitoring systems fail not because they miss problems but because they report too many. This guide is about signal quality and alerts people still read.
Ask an on call engineer what fraction of the pages they received last month required them to do anything. In a lot of teams the honest answer is under a quarter. That ratio is the real health metric of a monitoring setup, and no amount of dashboard polish improves it. An alert that fires when nothing is wrong teaches the recipient that alerts can be wrong, and that lesson gets applied to the next one too.
This guide is about the signal side of monitoring: what to measure, what deserves to wake someone, and how to keep the alert set honest as the system changes. It assumes you already have a collection pipeline of some kind and that the problem is not data but judgement.
Alert fatigue is a design failure, not a discipline problem
When people stop responding promptly to pages, the usual reaction is to talk about on call culture. That is almost always misdiagnosis. Humans calibrate to base rates. If most pages are noise, slower response to all pages is the rational adaptation, and no amount of encouragement will undo it.
The fix is on the sending side. Every alert that fires without requiring action should be either deleted, converted into a dashboard panel, or fixed so it only fires on the condition that actually matters. Treating a noisy alert as something to be endured rather than something to be repaired is how alert sets grow to several hundred rules that nobody can account for.
Start from what users feel
Infrastructure metrics are abundant and mostly uninteresting on their own. CPU at ninety percent is not a problem if requests are being served within their latency budget; it might be excellent utilisation. CPU at ten percent is not reassuring if the service is returning errors.
Work backwards from the experience you are protecting. For a request driven service that usually means four things: are requests succeeding, how long are they taking, how many are arriving, and how saturated is the resource that will run out first. Those four give you a compact picture that maps onto what someone would complain about.
For a queue based or batch system the equivalents are different: queue depth and its trend, oldest unprocessed item age, job success rate, and whether the last scheduled run actually ran. The age of the oldest item is the one people forget and the one that most directly reflects user impact.
Signals worth paging on
A page should mean a human needs to act now. In practice the list is short:
- The service is returning errors to real users at a rate above what you have agreed is tolerable.
- Latency has degraded past the point where the experience is broken rather than slow.
- A resource with a hard ceiling will be exhausted soon: disk, connections, a provider quota, a certificate expiry inside the renewal window.
- A pipeline that should have produced output has not.
- Something security relevant happened that cannot wait, as covered in detecting unexpected cloud activity.
Notice how much is absent. Single instance failures in an autoscaled group do not belong here, because the system is designed to absorb them. Deploys do not belong here. A spike in a metric that recovered on its own before anyone could log in definitely does not belong here.
Signals that belong on a dashboard instead
Plenty of information is worth having and not worth waking anyone for. Per instance resource graphs, cache hit rates, request distribution across regions, slow query logs, cost trends. These are investigation tools. Their value appears after an alert has fired and someone is trying to work out why.
Build dashboards for that moment. A good incident dashboard answers “what changed” in under a minute: deploy markers on the timeline, error rate broken down by endpoint and by dependency, and the saturation metric for whatever usually breaks. A dashboard with sixty panels answers nothing, because nobody can scan sixty panels under pressure.
Cost belongs on a dashboard too, with its own review rhythm rather than a pager. The exception is a hard anomaly threshold, which is worth an alert for the reasons set out in why cloud bills surprise people, since the gap between a runaway resource starting and the invoice arriving can be weeks.
Writing an alert that someone can act on
The text of an alert is part of its design. A page that says “HighCPU on web-07” forces the recipient to reconstruct the context from scratch at an hour when reconstruction is slow. A useful alert states what is broken in user terms, what the current value is against the threshold, which service and environment, and where the runbook lives.
Every paging alert should have a runbook, even if the runbook is three lines. Writing it is also a good test of whether the alert should exist. If you cannot describe a first action beyond “look at it”, the condition is probably not actionable and belongs on a dashboard.
Name the alert after the symptom
Alerts named after causes age badly, because the cause you had in mind when you wrote the rule is rarely the cause when it fires. “Checkout error rate elevated” stays accurate forever. “Database connection pool exhausted” becomes misleading the first time the same symptom comes from something else.
Thresholds, windows and flapping
Static thresholds are easy to reason about and wrong for anything seasonal. Traffic that is normal at midday is alarming at four in the morning, and a fixed floor will either miss the night time failure or scream through the daily peak.
Two adjustments fix most of this without introducing machine learning. Alert on rates and ratios rather than absolute counts, so the metric is already normalised against traffic. And require the condition to hold for a duration before firing, long enough to skip transient blips and short enough that the delay does not eat your response budget.
Flapping alerts, which fire and resolve repeatedly, are worse than consistently wrong ones. If a rule flaps, either the window is too short or the threshold sits right at the normal operating value. Both are fixable; neither is fixed by muting the alert, which is the usual response and which quietly removes the coverage you thought you had.
Routing, escalation and quiet hours
Not every alert needs the same delivery. Decide per rule whether it is a page, a message into a team channel, or a ticket. Most alert sets have exactly one delivery channel, which is why the page volume is high: there is nowhere else for a medium severity condition to go.
Escalation matters because people miss pages for ordinary reasons. An unacknowledged page should move to a second person after a defined interval, and that interval should be tested rather than assumed. The test is simple: have someone deliberately not acknowledge a non production alert and see whether the escalation actually reaches the next person.
Be careful with maintenance windows and silences. A silence set during an incident and never removed is one of the most common ways coverage disappears. Give silences expiry times by default, and review the list of active ones weekly.
Reviewing alerts on a schedule
Alert sets rot. Services get retired, thresholds that made sense at one traffic level stop making sense at another, and rules accumulate from incidents nobody remembers. Put a recurring review on the calendar, monthly or quarterly, and go through the alerts that fired since the last one.
For each, ask three questions. Did a human take an action because of it? Was that action something a machine could have done instead? If it never fired, do you believe it still would? Delete aggressively. An alert set small enough to read in one sitting is worth more than a comprehensive one nobody understands.
Also review the incidents that had no alert. Those are the gaps, and they are more valuable to find than another noisy rule is to tune. Each one should produce either a new alert or a deliberate decision that detection by customer report is acceptable for that failure.
Monitoring across more than one provider
If your infrastructure spans platforms, provider native monitoring gives you deep integration per platform and no unified view. The usual resolution is to keep provider tooling for infrastructure level detail and ship application metrics to one place where the service level alerts live, so that the page comes from a system that sees the whole request path.
Audit and activity logs are a separate stream and should be handled separately, preferably centralised from day one for the reasons in cloud audit logging from day one. Mixing operational metrics and audit events in one pipeline makes retention policy awkward, because the two have very different retention needs.
Platform depth varies here. The hyperscalers ship extensive native observability with a billing model attached to it, while smaller platforms tend to provide basic metrics and leave the rest to you. That gap shows up clearly in the AWS and Hetzner comparison. Teams running on a Hetzner account or a UpCloud account usually run their own collection stack, which costs setup time and gives complete control over retention and cardinality.
Whatever you run it on, judge the result by one number: the share of pages last month that led to an action. Drive that up and everything else about on call gets easier.