Nearly every monitoring setup begins the same way: install something, tick CPU, memory and disk, set all three to 80%, go to bed. Three weeks later the alerts are muted, because the ones that fired were noise and the outage that mattered arrived with no warning at all. The problem isn’t the tooling. It’s that the metrics easiest to collect are the ones least worth paging on.
Before an alert is worth existing, it has to survive one question: if this fires at 3am, is there something I would get up and do?
If the honest answer is “I’d look at it and go back to sleep”, it is not an alert. It might be a useful graph, a weekly review item, or a capacity-planning signal — but the moment it can wake you for nothing, it starts eating the credibility of every other alert you have. Alert fatigue isn’t a personal failing; it is the rational response to a system that has cried wolf.
Judged that way, the usual three metrics rank almost the opposite of how they’re usually configured.
High CPU means the machine is doing work. That is what it’s for. A build server at 100% is healthy. A video encoder at 100% is earning its keep. A database at 90% during the nightly batch is Tuesday.
Worse, the inverse is also true: plenty of severe outages happen at low CPU. A process that has deadlocked, a service that exited, a disk that went read-only — all of them look like a beautifully idle machine.
So CPU as an alert produces both false positives and false negatives, which is a remarkable achievement for a number everyone starts with. Chart it — it is genuinely useful when you’re already investigating — but page on it only in one narrow shape: sustained saturation over a long window on a machine you know should be mostly idle. “Above 95% for 30 minutes” can be meaningful. “Above 80%” is a noise generator.
Memory percentages mislead for a different reason: on Linux, unused RAM is wasted RAM, so the kernel fills it with cache. A healthy server frequently reads as 90%+ “used” and is completely fine, because that memory is instantly reclaimable.
What you actually care about is a machine that is running out and starting to hurt: heavy swapping, or the OOM killer taking processes. Those have consequences you'd act on. A number on a gauge doesn’t.
If your tooling can only give you a percentage, set the threshold high (95%+) and pair it with a long duration, so a brief cache spike can’t page you.
Disk space is the one classic metric that deserves a page, and it’s the one people most often leave at a useless default.
It qualifies because it has every property CPU lacks:
Two rules make it much better:
Alert per mount, not on an average. A machine with a 96%-full /var and three empty volumes averages out to something comfortable, and the average is the number that hides the problem about to take down your database.
Set it low enough to act on. 85% gives you room. 95% on a fast-filling disk can be twenty minutes of warning.
On owned hardware — a homelab, an office rack, an edge device in a cabinet — temperature earns its place. It’s a genuine leading indicator: a fan fails, a filter clogs, an air-conditioning unit dies over a long weekend, and temperature climbs steadily for hours before anything breaks. That is the rarest and most valuable thing a metric can offer, warning while there’s still time.
On someone else’s cloud VM it’s meaningless — you can neither see it usefully nor act on it. Skip it.
Here is the metric almost nobody configures, and it catches more real outages than the other four combined: the machine stopped reporting.
Think about what any threshold alert requires. It requires a healthy agent, on a running machine, with working networking, successfully sending a number. Every one of those is a dependency. Now consider the failures that actually take servers out: power cut, kernel panic, network gone, host terminated, disk so full the agent can’t function, someone unplugged the wrong cable. In none of those does a threshold alert fire — there is nobody left to send it.
A dead-man’s switch inverts the dependency. If a machine has checked in every twenty seconds for a month and then doesn’t, that absence is itself the page. It requires nothing of the failing machine, which is precisely why it works when everything else has stopped:
🔴 web-02 went quiet — no vitals for 60s last seen cpu 11% · mem 44% · disk 62% · 41°C # every number was healthy. that is the point.
Nothing crossed a threshold. The machine simply stopped existing.
| Signal | Page? | Setting that works |
|---|---|---|
| Stopped reporting | Always | 60–120s of silence |
| Disk per mount | Yes | 85% warn, 92% urgent |
| Service/endpoint failing | Yes | 2–3 consecutive failures, not 1 |
| Temperature (owned hardware) | Usually | ~10°C below the throttle point |
| Memory | Rarely | 95%+ sustained 15 min |
| CPU | Rarely | 95%+ sustained 30 min, idle machines only |
| Network throughput | No | Chart it. It’s an investigation tool. |
Require consecutive failures. A single failed HTTP check is often a dropped packet, not an outage. Requiring two or three in a row removes most false pages at the cost of one extra interval of delay. That trade is almost always worth taking, and it’s the single highest-value setting in most monitoring tools.
Alert on the leaf, not the root, and group the rest. When a database goes down, everything that depends on it fails too. Twelve pages for one incident trains you to ignore pages. If your tool understands dependencies, tell it — you want one alert naming the root cause, not a wall of symptoms.
A well-tuned server produces, on a normal month, zero alerts. Not few — zero. Every page it does send corresponds to something you genuinely needed to know, and you trust it enough that a notification at 3am gets you out of bed rather than reaching to silence it.
If that isn’t your experience, the fix usually isn’t better thresholds. It’s deleting alerts. Take the ones that have fired in the last month, ask which led to an action, and turn the rest into graphs.
PylonMon watches disk per mount, temperature and the silence signal by default, from a small open-source agent, and it checks from outside your network so an outage that takes the machine also can’t take the monitoring. Three machines free, no card.
Related: server monitoring without Prometheus · What a server monitoring system should cover