Jasper Grahl

Running a Raspberry Pi 24/7: what a short power cut revealed about my system

· 6 min read · Also in Deutsch

My energy management system JG Energy runs around the clock on a Raspberry Pi 5 with an SSD. It controls the wallbox, the home battery and charge plans for two EVs. If the Pi fails, you notice in everyday life – at the latest when the car isn’t charged in the morning.

In early November there was a short power cut one morning. The Pi restarted on its own afterwards, and everything kept running. Still, the incident exposed four weak spots I hadn’t known about.

Step 1: was it really a power cut?

My first suspicion was a crash. A simple trick helps to tell the difference: compare the uptime of all devices. If the Pi, the Shellys and other devices all restarted in the same minute, it was the power. If only the Pi is back at zero, it was the Pi.

Shellys from generation 2 onwards report their uptime via the local API:

curl -s http://<shelly-ip>/rpc/Shelly.GetStatus | jq .sys.uptime
# → seconds since the last start

For the Pi itself, uptime -s is enough. In my case all start times matched within a few seconds: power cut.

Step 2: why were there no logs?

Next I wanted to check the system log for what happened before the restart. Result: nothing. The log started with the restart.

Yet I had already enabled the persistent journal earlier – with Storage=persistent in /etc/systemd/journald.conf. The problem: Raspberry Pi OS overrides this setting. It ships its own drop-in file:

/usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf
→ Storage=volatile

Files in journald.conf.d/ take precedence over the main file. So my setting had no effect, and the log only lived in RAM. It was gone after every restart – including exactly the part you most want to see after a crash: what happened just before.

This shows which setting actually applies:

systemd-analyze cat-config systemd/journald.conf

The fix is a drop-in of your own with a higher number, so it’s read last:

sudo mkdir -p /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/99-persistent.conf <<'EOF'
[Journal]
Storage=persistent
SystemMaxUse=500M
EOF
sudo systemctl restart systemd-journald
sudo journalctl --flush

journalctl --list-boots shows whether it worked: after the next restart there should be several entries. journalctl -b -1 then gives you the log of the previous boot.

On an SSD this is no problem. On an SD card you should weigh it more carefully because of the write load.

Step 3: the lost startup message

JG Energy sends a Telegram message after every start. After the power cut, none arrived. The log said:

⚠️ Telegram error: getaddrinfo EAI_AGAIN api.telegram.org

EAI_AGAIN means name resolution is temporarily unavailable. After a power cut the Pi boots faster than the router and the internet connection. When JG Energy tried to send the message, there simply was no internet yet. The message was lost.

After=network-online.target in the systemd service doesn’t solve this: the Pi then has a network connection, but the router doesn’t have a connection to the outside world yet.

The solution is a queue. If a message fails because of a network error, it’s put aside and retried every minute:

function retryable(err) {
  const status = err?.response?.status;
  return !status || status === 429 || status >= 500;   // no status = no response
}

Step 4: clean shutdown or crash?

The startup message now also answers the question I asked myself after the power cut:

🟢 JG Energy started
⚠️ The Pi restarted too (1 min ago)
⚠️ Last run did NOT shut down cleanly (last seen 09:22)

All it takes is two things:

Step 5: the hardware watchdog against hangs

A power cut is harmless because the Pi starts again on its own. A Pi that hangs is more dangerous: it’s switched on but no longer responds. Then nothing restarts.

The Raspberry Pi’s built-in hardware watchdog helps here. systemd has to “feed” it regularly. If that stops, the hardware restarts the Pi. The Pi’s watchdog allows at most about 15 seconds, so I use 14:

sudo mkdir -p /etc/systemd/system.conf.d
sudo tee /etc/systemd/system.conf.d/watchdog.conf <<'EOF'
[Manager]
RuntimeWatchdogSec=14s
EOF
sudo systemctl daemon-reexec

You can check it with:

systemctl show -p RuntimeWatchdogUSec     # → 14s
dmesg | grep -i watchdog

The watchdog monitors the system, not the program. If only JG Energy crashes, the systemd service with Restart=always takes care of it.

Step 6: the devices must be safe without the server

The most important question for 24/7 operation isn’t how quickly the server is back. It’s: what do the devices do while it’s gone?

For the wallbox I solved this by making every lock the server sets expire on its own after two hours at most. If the Pi fails, the wallbox then charges like any ordinary wallbox. The car is full in the morning – just not with solar power.

The home battery has no such option. If the server fails while discharging is locked during surplus charging (the “stays off” setting), it stays locked. I estimated how often exactly this combination occurs and decided to accept it. What matters is knowing the gap exists.

Conclusion

A short power cut didn’t do any damage in my case. But it showed that without logs and without a startup message, I wouldn’t have known what had happened in a real emergency.

My checklist for any Pi that controls something important: