Where it started
All my projects run on my Arch daily driver, and sysadmin_assistant watches it. A FastAPI daemon runs five scheduled agents (service health, log aggregation, file organisation, service discovery and, later, a watcher of watchers judging estate-manager’s published numbers). There is also a PyQt6 tray client which displays all the findings and pushes system notifications when alerts go past a threshold. One services.yaml declares every service on the box and joins each to its project id where it has one. I made it fail at load with every bad reference listed at once, so nothing gets monitored under a name nothing owns.
The first lesson I had with this project was restraint, and I had measured it before acting on it. One venture-assistant outage produces 186 alert rows, one internet outage produces 123, and steady state ran to 88 criticals a day. You can’t hear that kind of silent failure. Notifications now pass three gates (presence, dedup against already logged alerts, and severity) and arrive about four times a week. Every alerting path in the daemon ends at a database row, and the tray reads these rows, so nothing raises a toast directly.
There is a division of labour with the estate-manager at play. This repo watches system health, the estate-manager audits whether documents are honest, and neither judges itself. sysadmin_assistant monitors, while estate-manager audits. That rule sounded easy enough when I wrote it…
The decision
For thirty-nine and a half hours the daemon wrote no service-health rows at
all, hee-haw in the tables, missing outright as opposed to degraded. Three causes composed, each survivable alone. I had the migration written but never applied it, a restart picked up a config declaring services unmonitored, and one rejected row aborted the transaction carrying the other nineteen! The daemon logged its own failure 29 times in one morning but nothing reads those logs. The blackout was found by accident. I was looking at the schema drift guard, the one test that connects to the live database, and asking the question, why is it still green? This was because it skipped alembic_version, and because compare_metadata does not diff CHECK constraints. I checked that rather than taking it at face value. Restoring the pre-009 constraint inside a rolled-back transaction produced an empty diff, with the model declaring skipped and the database rejecting it. alembic current said 008 against the repo head of 009, and that’s what caused the whole blackout. That was the same failure again, twice in one incident! The alerting path could not report its own failure, and the test that pointed at the live database could not see the fault in plain sight.
From then on, I decided that the daemon should refuse to boot when there’s a schema mismatch, because a monitor that comes up degraded and writes alerts through the wrong schema looks healthy but evidently not in this case! The guard also runs at commit time, since the commit is the last scripted moment before a restart I type by hand. The second time round, I could afford the price. The same cause later produced a loud 23-hour outage but at least it was no longer silent. I had thought about applying the migrations at boot but that’s how a bad migration gets through to production with no eyes on it.
What it buys
In the end, I had four separate answers to the one problem, because no single one covered it. A systemd failure unit fires when the daemon itself dies, which is the case where the daemon can’t report its own failure. There’s the desktop understudy inside the daemon that covers the tray being absent. The boot guard covers the schema. The claim registry covers documentation. Every measured claim in the roadmap carries a marker naming the check that would refute it, never a copy of the figure, and every session opens by re-running the lot. I built the registry after having thirty open snags checked, five of which were already dead on the box, three of them P1s. The sweep cost a full session and was already stale by the next fix; the method was a waste but it highlighted the need for it. The registry may refute a claim but never closes the entry itself, because the checker that edits the document becomes a second author. The repo now carries more lines of testing than the daemon itself!
Where it’s going
The plan is to have the alerts shown on my phone but it’s still unbuilt. The broker credential arrives by systemd’s LoadCredential, because an alerting path with a live dependency on another service is not a true alerting path, and a missing credential fails the unit on purpose, fulfilling the self-fulfilling prophecy of silence being healthy ([ADR-0003]). I’ve wired the credential into the unit today while its publisher is a grand total of zero lines of code.
Parallel to this, project state, which this repo invented the manifest of, moved to the estate under [ADR-0005], and I verified the copy against the source before anything was dropped. What stayed was the swap that keeps both honest. The estate never judges its own scan and sysadmin_assistant never judges its own briefing pull, so neither watcher owns itself.