Alert on log errors without shipping every line

There are two completely different things people mean by “log monitoring”, and picking the wrong one is why log bills get frightening. One is counting: how many tracebacks in the last five minutes, as a number you can put a threshold on. The other is searching: keep the stream so you can read around the moment it broke. The agent does both, separately, and most machines only need the first.

The question decides the tool

A few tracebacks an hour is life on the internet. Forty in five minutes is an incident. That difference is a count, and a count is cheap: it is computed on your machine and leaves as a single metric. You do not need to ship a gigabyte of logs to learn that a number went up.

Counting: one line, one metric

Each entry names a file, a pattern, and the window to count over. The window is optional and defaults to five minutes:

Each becomes an ordinary metric. That means it gets the same alerting as CPU — a comparison, a value, and a duration it must stay breached:

The sustained gate is what separates a deploy blip from a real spike.

The part that makes the page useful

A number tells you something is wrong and nothing about what. So the most recent matching block rides along with the push as a text sample — capped at forty lines — and appears in the alert. You get the actual traceback in the page, not a count that sends you hunting for it.

This is the whole reason to prefer counting for alerting: you get the number and the evidence, without a log pipeline existing.

Searching: keep the stream

Sometimes you genuinely need the lines. Entries name a file or glob, optionally a service name, and optionally a filter:

Three details worth knowing:

• The filter is the cost control. The auth line above ships only the lines that match, leaving the other nine hundred and ninety-nine on disk. On a noisy file that is the difference between useful and unaffordable. Leave the service empty to filter while still naming the stream after the file.
• Globs are re-expanded every cycle, so /var/log/containers/*.log follows pods as they come and go. Container lines are unwrapped in place — both Docker json-file and Kubernetes CRI formats — so you see the line, not the JSON envelope around it.
• Shipped lines show up under incidents, so the lines from around the moment it broke are on the same screen that paged you.

Behaviour that will save you a bad first day

Both features share the same reading discipline, and the first one matters more than it sounds:

• A new file starts at the END. Installing the agent never replays last week’s errors at you. Nobody wants their first experience of a monitoring tool to be a page about a problem from March.
• Reading is incremental from a remembered byte offset, so each cycle only scans what is new.
• Rotation and truncation are handled — if the file is smaller than the remembered offset, reading restarts from the top of the new file.
• There is a scan cap per cycle, so a log flood costs bounded work instead of taking the machine with it.

The honest limits

• Shipping is lossy by contract. Lines go out once per interval in one bounded batch. If the server is shedding load, that cycle’s batch is dropped rather than retried forever — paging is protected above log intake, deliberately. If you need guaranteed-delivery audit logging, this is not that, and you should not pretend otherwise.
• Counting is regex on your machine. A pattern that matches everything costs CPU on the host. Anchor it.
• No parsing or structure. Lines are lines. There is no field extraction, no JSON indexing, no derived dimensions.
• Retention is a plan limit, not forever.

How to choose

Start with counting on every machine, because it is nearly free and it is what actually pages you. Add shipping to the two or three services where you know you will want to read around an incident — and put a filter on any file noisy enough that you hesitated before adding it.

Most fleets end up with a lot of [logwatch] and a little [logs]. That ratio is the point.

Agent documentation · Error spikes, end to end · Custom metrics