Monitoring what matters
Monitoring is only as useful as the decisions it supports. Dashboards help investigation and capacity planning, while alerts call attention to conditions that need action. Too many non-actionable notifications make that job harder. The goal is to notice meaningful change, route it to the right person, and make the next action obvious. Everything else is overhead.
Start with the question, not the metric
Before adding a metric, ask what decision it should support. If the answer is unclear, the metric is probably not worth alerting on. A CPU utilisation alert at eighty percent is a classic example. High CPU is often normal, often transient, and often not a problem. An alert that fires when a specific service stops responding is far more useful, because the action is clear.
Think in terms of outcomes. The outcome you care about is that a service is available, that a backup succeeded, that a directory service is replicating, or that a critical log source is still sending events. Metrics that do not map to an outcome are interesting to look at but poor candidates for alerts.
Alert on meaningful change
A meaningful change is one that requires action. A disk that goes from forty percent full to forty-five percent full is not meaningful. A disk that goes from eighty-five percent to ninety-two percent in a day probably is. Set thresholds based on what constitutes a real problem, not on round numbers.
Use rate-of-change alerts where appropriate. A service that suddenly stops sending logs is more suspicious than one that has never sent them. A host that reboots unexpectedly is worth noticing. A host that reboots on a planned maintenance window is not. In Prometheus terms this is a query over movement in a series rather than a level, and the built-in functions cover both cases. changes() counts how many times a value moved inside a window, while rate() gives the per-second average increase of a counter and is described as "best suited for alerting" (Prometheus, Query functions).
Deduplication and grouping
When something goes wrong, it often causes many symptoms. A database failure can show up as application errors, as user complaints, and as a monitoring blackout. Without deduplication, you receive an alert for each symptom. Deduplication removes repeated notifications for the same alert. Grouping combines related alerts into a notification; identifying the underlying cause still needs useful labels and, where appropriate, inhibition rules.
That behaviour has a standard implementation in Alertmanager. Alerts are grouped by a set of labels, and three timings decide how quickly a group is announced. group_wait is the wait before the first notification for a new group, group_interval is the wait before follow-up notifications for an existing group, and repeat_interval is the wait before the same notification is repeated (Prometheus, Alertmanager configuration). Inhibition is the other half. An inhibition rule "mutes an alert (target) matching a set of matchers" while "an alert (source) exists that matches another set of matchers", which is how a database failure silences the application errors it caused.
Define when a condition counts as recovered and whether the receiver should send a resolution notification. A useful recovery notice confirms that action worked; flapping between firing and resolved states is the noise to avoid. Prometheus has a clause for this. keep_firing_for tells Prometheus to "keep this alert firing for the specified duration after the firing condition was last met", while a rule without it deactivates on the first evaluation where the condition no longer holds (Prometheus, Alerting rules). The companion for clause is the delay before an alert fires at all, and an alert still inside that delay sits in a pending state.
Avoiding alert fatigue
Alert fatigue is what happens when too many alerts fire and most of them do not need action. The human response to repeated non-actionable alerts is to start ignoring them. That is when the one alert that did matter gets missed.
Review alerts regularly. If an alert fires often and is usually ignored, either fix the underlying issue, change the threshold, or remove the alert. An alert should earn its place by leading to action. If it does not, it is a distraction.
Keep the alerting system small and meaningful. A handful of well-chosen alerts is more useful than a hundred that fire daily. The goal is to notice what matters, not to monitor everything.
Review cadence
After every incident, review the alerts that fired. Were they useful? Did they point to the right problem? Were there alerts that should have fired but did not? Use the answers to adjust thresholds, add new alerts, and remove old ones. Monitoring is not a set-and-forget system. It needs to evolve with the environment it describes.
Checklist
- For each metric, state the decision it supports before alerting on it.
- Alert on outcomes rather than raw numbers, and prefer service health over resource utilisation.
- Set thresholds based on what constitutes a real problem, not round numbers.
- Use rate-of-change alerts where sudden deviation is the meaningful signal.
- Deduplicate and group related alerts, and add recovery conditions.
- Review alerts after incidents and remove those that do not lead to action.
- Keep notifications actionable without discarding the metrics and logs needed for investigation.