Every monitoring tool ships the same five numbers: CPU, memory, disk, network, uptime. They are the easy ones, and they are almost never the thing that actually hurts you. The number that matters is usually specific to your business — the queue that is supposed to drain, the licence that expires, the export that runs nightly. If you can print it from a shell, you can monitor it.
A server at 30% CPU with plenty of disk is, by every standard dashboard, healthy. It can also be:
None of those move CPU. All of them ruin somebody’s week. The generic metrics cannot see them because they are not generic problems.
The agent has a [custom] section. Each entry is a name and a command. The command runs on the machine every check-in, and whatever number it prints becomes a metric that behaves exactly like CPU does — charted, alertable, on every plan.
Four numbers no generic agent would ever collect, on one machine.
Two practical notes from the parser. The value is kept as free text, so a # inside your command is part of the command and not treated as a comment. The name is cleaned into a metric key, so stick to letters, digits and underscores if you want it to read the way you wrote it.
This is the one rule that trips people up. The command must end up printing a bare number. A few patterns that keep it honest:
| Goal | Command shape |
|---|---|
| Count things | Pipe into wc -l. Empty output becomes 0, which is usually what you want. |
| Yes or no | Print 1 or 0. A service check is systemctl is-active --quiet nginx && echo 1 || echo 0. |
| Age of something | Subtract timestamps and divide. Hours or days read better than seconds when you are setting a threshold. |
| Pull one field | awk or cut down to the single value. Do the parsing on the box, not in your head at 3 AM. |
Keep each command fast and cheap. It runs on every check-in, so a query that takes ten seconds is a query you will regret at a thirty-second interval.
A custom metric is not a second-class citizen. It gets the same rule shape as CPU: a comparison, a value, and — the part worth using — a duration it must stay breached before anyone is paged.
The duration gate is why a queue that spikes for ninety seconds during a deploy does not wake anyone.
That sustained-breach setting matters more than the threshold itself. Most false pages are not wrong thresholds, they are correct thresholds crossed briefly by something that fixed itself. A queue over 500 right now is a Tuesday. A queue over 500 for ten minutes is a stuck worker.
Note the direction on the certificate rule: less than 14. Metrics that count down need lt, and getting that backwards is the classic way to build an alert that can never fire.
The useful test when choosing what to add: if this system broke at 3 AM and you had one command to run before deciding whether to get out of bed, what would you run? That command is your custom metric.
It is almost never CPU. It is the queue length, the replication lag, the age of the newest file in a directory, the number of rows in a table that should be empty.
The generic metrics tell you a machine is alive. Custom metrics tell you the thing the machine exists to do is still working — which is the question you actually wanted answered.
The agent is open source and vital alerting is on every plan including the free one, so the cheapest way to try this is to add the one command you already run by hand and put a threshold on it.
Agent documentation · The same idea on Windows · Which numbers should wake you up
