If you have a product running in production, sooner or later you run into this question: Who is going to watch the system while you are busy with other things, or even asleep?
With a limited budget, we didn’t want to rent another server for this. We had an idle Raspberry Pi on hand. Production stayed in the cloud, and we gave the monitoring job to the Pi.
On paper, it was a simple idea. Of course, things are never as simple as they look at first glance.
First attempts: Constantly knocking on the door
In the first version, a workflow tool on the server checked the system every few minutes. If everything was fine, it stayed silent; if it found a problem, it dropped an alert file onto the server. An agent running on a small cloud machine checked for this file every 5 minutes.
Thinking about it now, it feels a bit funny. To see if anything was wrong, we first looked to see if a file had appeared.
The first issue was that the current setup of the workflow tool didn’t allow dropping an alert file directly onto the server. We tried another approach: we attempted to send the alert by making a request to the application. That didn’t work smoothly on the first try either.
We also had a chat bot. Its status command said everything was fine, but it wasn’t reading the actual state; we had written the answer directly into the code. Seeing “everything is fine” didn’t actually mean everything was fine. We would run into this again.
Then we stopped checking files and moved to a message queue. To cut costs, we moved the agent to the Raspberry Pi and added a watcher to my personal computer. But another issue popped up: someone had to check every night whether that watcher was still running. On top of that, waking the agent remotely without exposing my computer to the outside wasn’t easy. A classic “who watches the watchmen?” situation. I had to monitor the system that was monitoring the system.
At some point, we decided it made more sense to rethink the whole setup from scratch instead of patching it piece by piece.
Waiting to be told instead of asking
On 30 August, we set up a central hub on the Raspberry Pi.
The workflow tool on the server kept doing its job, but with a simplification: we shut down separate workflows and consolidated down to a single one. We wanted to start with a solid, working version first and build on it over time. This workflow checked the application’s health at regular intervals.
The real change was what happened when a problem was found.
Previously, my personal computer or the agent had to reach out to the server every 5 minutes to ask if there were any new alarms. That was no longer necessary. When the system running the checks found a problem, it notified the Raspberry Pi directly. That notification prompted the agent into action.
In addition, a small connectivity check ran on the Pi. It regularly checked whether it could reach the server. That way, we ensured not just the application was monitored, but also the access to the server.
That same day, we ran a small drill to test the system. We deliberately triggered a failure. The agent received the alarm, noticed the situation, and intervened.
It was a good test for learning, but it also showed that we needed to define the agent’s boundaries clearly. The logs from that day were full of failed attempts as well. The system certainly didn’t start working flawlessly on day one.
How much can we trust the agent?
After that experience, we made an important decision: An agent can make decisions, but it cannot set its own permissions.
The agent’s role was to inspect incoming data, figure out what the problem was, and decide what could be done to solve it. But we would define the boundary of where it was allowed to intervene.
Because of this, we only authorized the agent to perform a few recovery actions we had approved in advance. Everything else was forbidden. If it could solve the problem with one of those actions, it could intervene; if not, it stopped without touching anything and reported the situation to us.
We didn’t allow it to modify the application code either.
We kept the rules deliberately strict. As a result, the agent became a cautious assistant that rarely stepped in. We have since narrowed it further: in the new version it only makes suggestions and runs nothing itself.
A real test
On 3 September, the agent had to make a genuinely important call.
There was an issue in production, and alarms kept rolling in. The agent first tried one of the approved actions. However, that didn’t resolve the issue.
Instead of getting stuck in a loop by repeating the same action a second time, it reassessed the situation. It didn’t delete the logs, because those were our evidence for understanding what had gone wrong. In the end, it concluded that a solution wasn’t within its permissions and handed things over to us.
That was exactly the behavior I wanted to see. Not an agent that tries to fix everything and blindly retries when it fails, but an assistant that knows where to stop.
We moved, and the problems followed
After moving the observation system to the Pi, we ran into a series of classic mistakes. The kind even seasoned developers make at least once and kick themselves for afterwards. We managed to string several of them together in short order.
The first issue was about what we were basing the alarms on.
We had hooked the sensor up to the running total of failures accumulated since day one, rather than the current count. As a result, past failures kept firing alarms continuously. Every 5 minutes, a new alarm came in.
Soon, hundreds of pop-ups began stacking up on my screen. Whatever was happening in production, one thing had definitely crashed: my screen.
To fix this, instead of the total, we started looking at what was happening in the moment. That stopped it from constantly generating alarms because of old records.
On 6-7 September, we ran into another problem. I had asked for the alarms to be silenced for a while. But an assistant agent I was working with in another session stopped the observation system entirely instead of muting notifications.
The result was perfect: zero alarms, zero noise.
There was one small problem. For about 21 hours, nothing had been monitored and not a single record had been kept. No alarms appeared on the screen, but we had no idea what state the system was actually in.
The funny part was that we had added the mute feature that very day, specifically to silence notifications while keeping the logs going. But the observation system had already been stopped.
After that incident, we set a simple rule for ourselves:
You can mute notifications, but never stop the sensor. Because on screen, silence and blindness look like the exact same thing.
On 9 September, we ran into a network connectivity issue. The Raspberry Pi lost its own internet connection, and because it couldn’t reach the server, it repeatedly reported the server as unreachable.
We had set up a watchman that assumed the whole world had gone dark just because its own eyes were shut.
To solve this, we added one more check before raising an alarm. Before declaring the server unreachable, the Pi now checked its own internet connection. If it couldn’t reach the outside, instead of saying the server had crashed, it reported that it had lost its own connection. A real outage in November proved it: the Pi reported itself as blind, not the server as dead.
The final mistake was even simpler: we had assumed the system would automatically run again when the device restarted. But we had never actually tested this.
While investigating another issue, we noticed that when the Raspberry Pi restarted, the observation system didn’t come back up on its own. We fixed the issue. Rather than waiting for a real reboot, we deliberately killed the running process and confirmed that the system recovered on its own.
Turns out our continuous observation system was only continuous until the device restarted.
So why the Pi?
This setup comes with some drawbacks. The Raspberry Pi has limited memory, and the internet connection is far from reliable at times.
Yet interestingly enough, it was precisely those constraints that helped us catch certain issues sooner.
If the connection hadn’t dropped, we might never have seen the observation system confuse its own connection problems with a server crash. If notifications hadn’t flooded my screen, we wouldn’t have learned the difference between silencing notifications and stopping the observation system through 21 hours of blindness.
Strangely enough, I have no complaints about any of this. Over the years, these little machines have taught me the most when they break. I have to admit, I am still having a lot of fun.