Self-hosted monitoring is genuinely good. Prometheus, Zabbix, Netdata, Uptime Kuma — they are capable, well-built, and free, and if you run one you probably have better visibility into your systems than most companies do. There is exactly one class of failure they cannot report on, and it happens to be the class that matters most: the one that takes them down too.
Picture the setup most people arrive at. A monitoring container on the same host, the same rack, or the same cluster as everything it watches. It scrapes, charts, and alerts. It works flawlessly for months.
Then the host dies. Power supply, kernel panic, a hypervisor that stopped, a datacentre with a bad afternoon, a home circuit breaker.
Everything on it stops — including the thing whose entire job was to tell you that everything stopped. There is no alert. There is no dashboard, because the dashboard was on the machine that died. From your side, the system is silent in exactly the way a perfectly healthy system is silent.
You find out from a customer. Or from opening a tab hours later. That is the failure mode, and no amount of configuration inside the box can fix it, because the problem isn’t configuration. It’s topology.
This isn’t one edge case — it’s a family, and it’s the family that produces real downtime:
| What went wrong | Can on-box monitoring report it? |
|---|---|
| A service crashed | Yes — this is what it’s good at |
| Disk filling up | Yes, with plenty of warning |
| The host lost power | No — it lost power too |
| Kernel panic / hard lockup | No |
| Network or ISP outage | No — the alert can’t leave the building |
| DNS broke for the outside world | No — internally everything resolves fine |
| TLS certificate expired | Rarely — internal checks often skip verification |
| Cloud provider terminated the instance | No |
| Someone unplugged the wrong cable | No |
Notice the pattern in the “no” rows. They’re not exotic. They’re the ordinary, boring ways that things actually go down — and every one of them is also a failure your users experience immediately.
This is the part that makes it dangerous rather than merely known. You cannot easily notice this problem in normal operation, because monitoring is only ever exercised by the failures it can see.
Kill a service to test your alerting: it works. Fill a disk: it works. Every drill you would think to run confirms the system is sound, because you're running the drills from a position where the monitoring is alive. The untested case — the whole host disappearing — is the one you find out about in production, once, at the worst time.
So the confidence is real, well-earned, and specific to a subset of failures that excludes the important ones.
A related version catches people who have thought about this: alerting that depends on the thing it's alerting about.
The rule generalises: the alerting path must not share failure domains with the thing being alerted on. Every shared dependency is a case where the alert cannot be delivered precisely when it is needed.
Something outside your failure domain has to be watching. That’s the whole requirement, and it can be small — this is not an argument for replacing the monitoring you already have.
Check from outside. Something on the public internet requests your service the way a user would. When it can't get an answer, that is the user experience, whatever the internal dashboard says. It also catches the failures that only exist from outside: expired certificates, DNS problems, a firewall rule that broke external access while everything internal kept working.
Use a dead-man’s switch for anything with no public address. Have the machine or job reach out on a schedule; the absence of that check-in becomes the alert. It requires nothing of the failing system at the moment of failure, which is exactly why it survives:
🔴 rack-nuc-01 went quiet — no check-in for 60s last seen cpu 9% · mem 37% · disk 55% · 38°C # every metric healthy right up to the moment the power went. # nothing on that machine could have sent this alert.
Deliver alerts through a path that doesn’t depend on you. Email from a provider you don’t host, SMS through a carrier, a push notification. If your alert has to traverse your own infrastructure to reach you, it will not arrive on the day you need it.
This is not “delete Prometheus.” The two layers answer different questions and the combination is strictly better than either:
| Layer | Answers | Fails when |
|---|---|---|
| Self-hosted, on-box | Why is it slow? What changed at 14:20? Which query is hot? | The host goes down |
| Off-site watchdog | Is it up at all? Can a user reach it? Is anything still alive in there? | — |
Keep your detailed internal monitoring — it’s where you'll spend your investigation time and it's better at that than anything external can be. Add a thin outside layer whose only job is to notice absence. The second one doesn’t need to be sophisticated. It needs to be elsewhere.
That is the entire argument for off-site monitoring, and it’s a structural one rather than a feature comparison. A watchdog inside the house cannot bark about the house burning down.
If you take one thing away: pick your single most important machine or job, and give it a heartbeat to somewhere that isn’t you. One line in a cron job is enough to cover the failure class your existing monitoring structurally cannot.
*/5 * * * * curl -fsS https://pylonmon.com/hb/YOUR-ID
PylonMon runs off-site by design — the production instance deliberately does not live on the infrastructure it watches, for exactly the reasons above. URL checks from the public internet, an open-source agent for machines with no public address, and heartbeats for jobs. Three machines free, no card.