How to know your backups are actually running

Nobody loses data because a backup failed loudly. They lose it because the backup stopped quietly — months ago — and the first integrity check was the restore that didn't work. Here's how to make backup silence impossible, with real restic and borg examples you can paste tonight.

Every backup horror story has the same shape. The script worked when you wrote it. Then one day a key expired, a disk filled, a hostname changed, a cron entry got lost in a rebuild — and the job stopped. Not with an explosion: with silence. The folder of snapshots just quietly stopped growing, and everything looked fine because nobody looks at a thing that's supposed to be automatic.

So the goal isn't "alert me when the backup fails." That's half the job. The goal is: alert me unless the backup provably succeeded. Those are very different guarantees, and the second one is the only one that catches the quiet death.

The two ways backups betray you

• The job runs and fails — repository locked, destination full, credentials expired, network down. Detectable, if something checks the exit code.
• The job stops running at all — crontab clobbered, server rebuilt without it, systemd timer disabled, the box that ran it decommissioned. Nothing errors, because nothing runs. This is the one that eats data, and no amount of error-handling inside the script can catch it — the script never starts.

If you read our cron alerting guide, you know where this is going: the only mechanism that catches both is a dead-man's-switch — the backup checks in when it succeeds, and a watcher outside your infrastructure raises the alarm when the check-in doesn't come.

The pattern: succeed, then check in

One line change. Whatever your backup tool, chain a heartbeat ping to a successful exit:

# restic, nightly at 02:30
30 2 * * * restic backup /home /etc --exclude-caches && curl -fsS https://pylonmon.com/hb/your-monitor-id

# borg
0 3 * * * borg create ::'{hostname}-{now}' /home /etc && curl -fsS https://pylonmon.com/hb/your-monitor-id

# rclone to object storage
0 4 * * * rclone sync /srv/data remote:backups/data && curl -fsS https://pylonmon.com/hb/your-monitor-id

The && is the contract: the ping fires only if the backup exited zero. A failed run sends nothing. A run that never starts sends nothing. Both collapse into the same detectable event — silence — and silence is exactly what the watcher is listening for.

Setting the deadline

On the watcher side you create one heartbeat monitor per backup job and set a single number: alert if silent for… For a nightly backup, pick the built-in 26 hours preset — a day plus slack, so normal jitter (a big changeset, a slow destination) never pages you, but a dead job can’t hide past lunch:

One monitor per job. If you back up five things, make five — the free plan's 20 monitors leave room.

And when a backup dies quietly — the credentials expire on a Tuesday, say — this is what your Wednesday looks like instead of nothing:

The failure surfaced 25 hours after the last good run — not months later, during an emergency.

Level up: alert on the failure itself too

The heartbeat catches everything eventually, but its detection time is your backup interval. If you want to know the moment a run fails — not a day later — add a failure branch that tells your channel directly:

#!/usr/bin/env bash
# nightly-backup.sh — heartbeat on success, loud on failure
if restic backup /home /etc --exclude-caches; then
  curl -fsS https://pylonmon.com/hb/your-monitor-id
else
  curl -s -H "Content-Type: application/json" \
    -d "{\"content\": \"restic FAILED on $(hostname) — exit $?\"}" \
    "$DISCORD_WEBHOOK_URL"
fi

Now a failed run pages you within seconds, and a run that never happens still trips the deadline. Belt, suspenders, and the suspenders don't live on the machine that's failing.

What monitoring can't do: verify the restore

Honesty section. A heartbeat proves your backup ran and exited clean. It does not prove the archive is restorable — corruption, an empty include list, or backing up the wrong path all exit zero. Two habits close that gap:

• Verify integrity on a schedule. restic check weekly (or borg check) as its own cron job — with its own heartbeat monitor, because the verify job can die quietly too.
• Do a real restore quarterly. Pull one real file from the newest snapshot to a scratch directory and look at it. Ten minutes, four times a year, and you'll never discover a dead archive during an incident.

# weekly integrity check, with its own dead-man's-switch
0 5 * * 0 restic check --read-data-subset=5% && curl -fsS https://pylonmon.com/hb/your-check-monitor-id

The checklist

• Every backup job ends with && curl to its own heartbeat URL — success is the only thing that pings.
• Deadline = backup interval plus a little slack (nightly job → the 26-hour preset) so normal jitter never pages you.
• Failures also page directly (the bash branch above) if detection time matters to you.
• Integrity checks are scheduled and monitored like a backup — they can die quietly too.
• A real restore gets rehearsed quarterly. Monitoring buys you the confidence that there's something to restore from.
• The watcher lives off-site — an alert system on the same box that's failing goes down with it.

Frequently asked questions

How do I get notified if my backup fails?

Two layers: chain a heartbeat ping to successful exits (backup && curl your-ping-url) so an off-site watcher alerts on silence, and optionally add a failure branch that posts directly to Discord/Slack the moment a run errors. The heartbeat is the layer that also catches backups that stop running entirely.

How do I monitor restic or borg backups?

Append && curl -fsS <your heartbeat URL> to the cron line or systemd service that runs restic backup or borg create. Register that URL with a heartbeat monitor expecting a ping each cycle. Add a second monitor for your weekly restic check/borg check so integrity verification is watched too.

What happens if the whole backup server goes down?

That's exactly why the watcher must be off-site. If the machine dies, cron dies with it — no error can escape a dead box. But the heartbeat deadline still expires at the watcher, which pages you. The silence is the alert.

Does a successful backup mean I can restore?

Not by itself. A clean exit proves the run finished, not that the archive is sound. Schedule integrity checks (restic/borg check) with their own monitor, and rehearse a small real restore quarterly. Monitoring plus rehearsal is the full answer.

Put a dead-man's-switch on your backups free →   or read the cron alerting guide and the heartbeat docs.