A few Python tracebacks an hour is just the internet being the internet: a flaky mobile client, a DNS hiccup, someone’s phone going through a tunnel. Forty tracebacks in five minutes is an incident. The hard part isn’t catching a single error — it’s telling the difference between background noise and a real spike, and getting the actual error in front of you the moment it matters. Here’s a way to do that with one config line and no new infrastructure.
Say your app logs a traceback whenever an upstream call fails. Most of those are fine — transient network failures you can’t control and don’t want to be paged about. But when a dependency actually falls over, or you ship a bug, that trickle becomes a flood. You want to know about the flood, not the trickle. And when you’re paged, you want the stack trace, not a number that sends you SSH-ing into a box at 2 AM to run tail.
Stated plainly, the requirement is:
Every standard tool solves part of this and asks a lot in return:
grep -c + mail — the honest homelab move, and closer than you’d think. But it emails you on every run, has no rolling window, no escalation, no recovery notice, and no off-site delivery — if the box that runs the cron is the box that’s on fire, the email never leaves.What’s missing is the small version: count matches on the box, push just the number out, alert on a threshold, and carry the latest match along for context. No log shipping, no SDK, no query language.
Instead of shipping every log line somewhere central and counting there, count on the machine itself and push out only what matters: a running count, plus the most recent matched block. This is exactly the push-based, no-open-ports model we use for server vitals — the same agent, one more section in its config.
pylon-beacon is a single static binary that already pushes a machine’s vitals out to PylonMon over outbound HTTPS. Its [logwatch] section tails a file, counts regex matches inside a rolling window, and reports that count as a normal metric — while carrying the latest matched block (a whole traceback, not just its first line) along with the push.
# count Python tracebacks in the last 5 minutes, and keep the latest one [logwatch] tracebacks_5m = /var/log/app/app.log | Traceback \(most recent call last\): | 300
Three fields: the file, a regex matched per line, and the window in seconds. That’s the entire configuration.
Then, on the node’s monitor in PylonMon, you set a threshold rule — the same kind of vital rule you’d use for “disk over 90%”:
The sustained gate is what tolerates background noise: a one-off cluster of errors that recovers on its own is logged but never paged.
When the count crosses the line, the page carries the count and the most recent matched block — the actual traceback, straight from the log file, delivered off the box:
Traceback (most recent call last):
File "/srv/app/handlers.py", line 88, in fetch_profile
resp = session.get(url, timeout=2)
ConnectionError: HTTPSConnectionPool(host='api.example.com',
port=443): Max retries exceededYou now know it spiked, by how much, and exactly what’s throwing — before you’ve opened a terminal. The full block is also kept on the monitor’s detail card.
When the window decays back under the threshold — the old matches age out of the five-minute window — the next push clears the rule and you get a recovery notice. Same lifecycle as any other monitor: it opens an incident, escalates if you’ve set a ladder, and resolves itself when the numbers say so.
The pattern is a regex, so anything that shows up in a log line is fair game. A few that earn their keep:
[logwatch] # HTTP 500s in your access log, last 5 min http_500s = /var/log/nginx/access.log | " 500 | 300 # the kernel OOM-killer, last hour oom_kills = /var/log/kern.log | Out of memory | 3600 # failed logins, last 10 min (brute-force canary) auth_fails = /var/log/auth.log | Failed password | 600
Each becomes a number you can threshold, with the offending line delivered when it trips.
A log watcher that’s careless is worse than none. A few things worth knowing about how this one behaves:
Nothing, for a homelab or a small service. [logwatch] reports a normal beacon vital, and vital-threshold alerting is included on the free plan — three beacon nodes, no credit card. You add a beacon, add a [logwatch] line, set a rule, and you’re paging on error spikes with the traceback attached, for free. Paid tiers exist for more nodes and for the Prometheus/Alertmanager ingest, but this specific trick doesn’t require them.
Count the errors in your log with a rolling-window watcher on the machine, push the count to an off-site monitor, and set a threshold rule. pylon-beacon’s [logwatch] does this with one config line — no SDK in your app, no third-party error service, and the matched log line is delivered in the alert.
Yes. The watcher captures the full matched block — for a Python traceback that’s the Traceback… line, its indented frames, and the closing SomeError: message — and attaches it to the page, capped to a few KB. You see what’s throwing before you open a terminal.
Alert on a rate over a window, not on each occurrence, and use a sustained gate. “More than 20 in 5 minutes, for at least 60 seconds” ignores the steady background trickle and a brief blip that recovers on its own, and pages only on a real, lasting spike.
Yes — if you only need to count patterns and alert on a threshold, you don’t need to ship every log line anywhere. Counting on the box and pushing just the number (plus the latest match) skips the entire pipeline. Reach for ELK/Loki when you need full-text search and long retention, not for a rate alert.
Yes. The agent connects outbound over HTTPS, so there’s nothing to port-forward and no inbound rule. It works behind a home router, CGNAT, or a locked-down firewall — and because the watcher is off-site, it still alerts when the application itself is too broken to phone home.
Start alerting on error spikes — free → or see the pylon-beacon overview and lightweight server monitoring guide.