A Raspberry Pi running as a k3s node went quiet for 109 seconds one morning and came back on the wrong kernel. The journal was empty, last printed nothing, and wtmpdb boottime said the boot time could not be found. Here is how to find the cause anyway, and which two log files still had the answer.
An off-site monitor paged at 06:29. The node pushes its vitals every 20 seconds and the alert fires if nothing arrives for 60:
node-02 went quiet - silent for 67s - SLA 60s
By the time I looked, the node was up and the cluster was healthy. Easy to dismiss as a false alarm — except the check-in history was unambiguous:
06:28:15 ok
06:28:35 ok
<- 109 second gap
06:30:24 ok
06:30:44 ok220 check-ins over the previous three hours, all 20 seconds apart, mean 20.4 seconds. Exactly one gap over 60, and it was 109. Something real happened.
uptime said 13 minutes, which lines up with the gap. So it rebooted. That answers whether, not why.
$ journalctl -u <agent> --since 06:25 (nothing) $ journalctl --list-boots 0 <current boot only>
There is the first dead end, and it explains the first command. journald on this image is volatile — there is no /var/log/journal directory, so the journal lives in /run and everything from the boot that died went with it. If --list-boots shows only boot 0, stop looking in the journal.
Next instinct is the boot records:
$ last -x reboot shutdown wtmpdb begins Sun Jun 14 16:35:10 2026 (no entries) $ wtmpdb boottime Couldn't read boot entry: Boot time not found: no more rows available
Second dead end, and a subtle one. Newer Ubuntu replaced the binary /var/log/wtmp with a SQLite-backed wtmpdb, and the reboot / shutdown pseudo-entries do not come through the last compatibility shim. Note the giveaway: it claims records back to June while listing zero reboots, and that machine had definitely rebooted for a kernel update in early July. The tool was blind, not the history empty.
Worse, wtmpdb boottime reports no boot entry at all, so this image is not writing boot records either. Two standard tools, no boot history from either.
Neither of these is written by journald, which is exactly why they were still there.
$ sudo grep -iE 'reboot|linux-image|kernel' /var/log/unattended-upgrades/*.log 06:28:02 Packages that will be upgraded: libc-bin libc-gconv-modules-extra libc6 linux-image-raspi locales locales-all 06:31:56 Packages that will be upgraded: linux-image-raspi 06:34:26 Packages that were successfully auto-removed: ...
Unattended upgrades started at 06:28:02, pulling glibc and a new kernel. The last check-in was 06:28:35 — thirty-three seconds later.
Two things are odd. The run at 06:31:56 lists linux-image-raspi as “will be upgraded” again, three minutes after the first attempt. And the auto-remove at 06:34 happened after the reboot. That is the shape of a transaction that never finished and got picked up on the next pass.
Then dpkg, which timestamps every single package operation:
$ sudo grep '^2026-07-29 06:2[5-9]\|^2026-07-29 06:3' /var/log/dpkg.log 06:28:32 status half-configured systemd:arm64 259.5-0ubuntu3 06:28:32 status installed systemd:arm64 259.5-0ubuntu3 <- nothing for three and a half minutes 06:31:58 startup archives unpack 06:31:59 install linux-main-modules-zfs-...
There it is. dpkg reconfigured systemd at 06:28:32. The last check-in was 06:28:35 — three seconds later. Then dpkg goes silent mid-transaction, and a completely fresh run starts after the machine came back.
Configuring a new systemd package triggers a systemctl daemon-reexec: PID 1 re-executes itself. It did that in the same transaction that had just replaced libc6 — the glibc PID 1 is linked against — on a small ARM board with containerd and k3s live. That is about the most delicate moment in a Debian or Ubuntu upgrade, and this machine did not survive it.
$ grep -R Automatic-Reboot /etc/apt/apt.conf.d/ //Unattended-Upgrade::Automatic-Reboot "false"; //Unattended-Upgrade::Automatic-Reboot-WithUsers "true"; //Unattended-Upgrade::Automatic-Reboot-Time "02:00";
Every line commented out, so the setting sits at its default — which is false. Nothing asked this machine to reboot. Combined with dpkg stopping mid-transaction and no clean shutdown recorded anywhere, this was not scheduled maintenance. It fell over.
Worth doing before blaming the upgrade, because thermal, memory and power are the three things everyone guesses. The vitals either side of the gap:
| time | cpu | load | mem | temp |
|---|---|---|---|---|
| 06:27:55 | 4.5% | 0.96 | 24.3% | 31.6 C |
| 06:28:15 | 7.9% | 0.69 | 25.4% | 36.5 C |
| 06:28:35 | 27% | 0.93 | 34.6% | 35.1 C |
| 06:30:24 | 26.4% | 0.81 | 13.1% | 38.9 C |
| 06:31:04 | 34% | 4.16 | 19.6% | 38.0 C |
Not thermal — 36 C, and these throttle around 80. Not memory pressure — 25%, and the climb at the end is dpkg unpacking, not a leak. Not load — 0.93 when it died; the 4.16 afterwards is k3s restarting every container. Disk sat at 7%.
The detail I like most: memory drops from 34.6% to 13.1% across the gap. Page cache does not halve because a process died. It halves because the machine came up fresh. That is independent confirmation of a reboot without needing a single log line.
I also checked the other three nodes — same LAN, same internet connection, same image. Worst gap in any of them during that window was 20 seconds, which is simply their normal interval. So not the network, and not something environmental hitting the whole rack.
The kernel installed at 06:31, after the crash-reboot, so the node was running the old one with the new one sitting on disk:
$ uname -r 7.0.0-1014-raspi $ ls -la /var/run/reboot-required -rw-r--r-- 1 root root 32 Jul 29 06:33
Because this node carries MetalLB and Traefik, an undrained reboot takes ingress with it. So:
$ sudo dpkg --audit # anything left half-configured? $ kubectl drain <node> --ignore-daemonsets --delete-emptydir-data $ sudo reboot $ kubectl uncordon <node> $ uname -r 7.0.0-1015-raspi
Run dpkg --audit before the reboot, not after. A transaction that died mid-configure can leave packages half-done, and you would rather find that deliberately than during the next apt run.
1. Make journald persistent. This is the single highest-value change here — without it, the boot that fails takes its own evidence with it.
$ sudo mkdir -p /var/log/journal $ sudo systemd-tmpfiles --create --prefix /var/log/journal $ sudo systemctl restart systemd-journald
Set SystemMaxUse in /etc/systemd/journald.conf first — on an SD card you do not want an unbounded journal. Then journalctl -b -1 -e shows the tail of the boot that died, which is usually the whole answer in one screen.
2. Take the packages that need a re-exec or a reboot out of unattended-upgrades’ hands. In /etc/apt/apt.conf.d/50unattended-upgrades:
Unattended-Upgrade::Package-Blacklist { "libc6"; "libc-bin"; "systemd"; "linux-image-.*"; };
Security patching stays automatic. glibc, systemd and kernels get done deliberately, after draining. On a cluster you almost certainly want this — and check whether your nodes are all on the same apt-daily-upgrade.timer window, because every node rebooting itself at once is far worse than one doing it.
3. Look at kured if you want it hands-off. It watches for /var/run/reboot-required, drains the node, reboots one at a time, and uncordons. That is exactly the gap between “a reboot is needed” and “a reboot happened safely”.
Worth separating, because the internet is full of confident post-mortems that are half guesses.
Proven: the machine rebooted (uptime, the memory drop, and Kubernetes logging every pod on the node terminating at the same instant). Nothing configured asked it to. The upgrade started 33 seconds before the last check-in. dpkg reconfigured systemd 3 seconds before it. The dpkg transaction did not complete.
Inference: that the daemon-reexec against a just-replaced glibc is what actually killed it. No kernel log survived that boot, so I cannot show you the panic. The timing is three seconds and the mechanism is well known, but it is a strong reading rather than a proof — and now that the journal is persistent, the next occurrence will settle it.
The host looked completely healthy the entire time. 4% CPU, a quarter of its memory used, cool, disk nearly empty. If the only thing watching had been resource graphs, this would have been a two-minute hole that nobody ever noticed — and the node would still be quietly running a kernel I believed I had upgraded.
No log on the machine survived to tell me. The one thing that noticed was something outside the cluster observing that the node had stopped speaking. That is the whole argument for off-site monitoring in one incident: a monitor living on the rack cannot tell you the rack went away, and a healthy-looking host is not the same as a working one.
The silence was the signal. Everything else in this post is just the paper trail.
Watch a node from outside — free → or read about the agent and the monitor types.