Watchtower: Testing an Operations Agent Before Any Customer Saw It

## The idea Small construction companies lose money in the gaps between their tools. A foreman agrees to extra work over a text message and it never becomes a change order, retainage gets released before the lien waiver is in, or a subcontractor keeps working after their insurance certificate has expired. Each of these leaves a trace somewhere in an email, a chat or a daily log, but nobody has the time to read across all of them. Watchtower is the engine behind Upright Ops, the small operations-AI product I'm building. It reads the tools a business already uses, links records across them and flags what's about to fall through the cracks, and every finding points back to the records it came from. ## Who it was for When I built this I hadn't talked to a contractor yet, so the target client came from desk research into small Ontario contractors, and I treat it as a hypothesis rather than a finding. That research pointed to a few clear requirements: nothing new for the team to learn, read-only access to start with and easy to switch off, findings tied to money (an unbilled change, a payment at risk, a compliance gap that could stop a job), and evidence attached to every finding, so a busy owner can check it in a minute instead of trusting a score. ## How do you test something like this? A detector for things that fall through the cracks is surprisingly hard to evaluate, because on real data you never know what it missed. So I flipped the order: write the answer key first, then build a world that contains it. ## What I built First, a simulated company. 16 LLM agents each played one role at a fictional 30-person general contractor, from the owner and project managers to the controller, superintendents, foremen and subcontractors. Over 90 simulated days across 4 projects they wrote emails, chat messages, invoices, change orders, RFIs and daily logs, and each new document could build on the ones before it. Before any of that was generated, I defined 15 problems and scheduled them into the calendar: work billed without a change order, a duplicate invoice, retainage released without a lien waiver, an expired insurance certificate, an unlogged safety incident, and a schedule that drifted away from the daily logs. The agents were never told these were anomalies, so they show up the way real ones do, as a few ordinary-looking records. Then came the investigator, a separate process with no access to the answer key, only structured queries and semantic search over the records. It's split into four roles (a financial auditor, a compliance officer, a schedule analyst and a synthesizer that challenges the other three), and they share a blackboard of findings. An evaluator sits outside all of it, matching each finding against the answer key and reporting what was found, what was missed and what was just noise. Finally, to see the engine on real data, I connected it to real Gmail, Google Calendar and Notion accounts, with read-only Gmail access. The first live use case was personal operations: unanswered emails, stalled tasks, upcoming commitments and obligations that show up across all three. Findings arrive as a daily Telegram digest where you can resolve, snooze or dismiss them, and the bot also takes voice notes. ## The stack Python for ingestion, agents and evaluation, Azure OpenAI for the simulated employees, SQLite for structured records, and vector search on Azure AI Search at first and Chroma locally later. I also tested Letta as an alternative memory backend. The dashboard is Next.js and TypeScript, and live mode runs on Google OAuth, the Notion API and a Telegram bot, with Whisper for voice notes. ## How it came together I started with a small 30-day world before moving to the full 90-day simulation, and I kept the investigator isolated by construction: it runs in its own process and never reads the seed file, so only the evaluator ever sees both sides. My first investigator was a single agent covering finance, compliance and scheduling at once, and it kept looping over ground it had already searched. Splitting it into four specialists with a shared blackboard found far more of the planted problems. After four runs I wrote a retrospective with a root cause for every miss and false positive, and a later benchmark reset all state before each run, capped each agent's steps and compared memory backends against each other and against a single-prompt baseline. ## What I learned The biggest lesson was about memory. The blackboard kept every finding across runs and nothing ever removed one, so while recall held at 12 of 15 for three runs in a row, the number of findings on the board grew from 18 to 60 and precision fell from 0.44 to 0.20. The synthesizer even had a tool to retract findings, and it never used it. A later benchmark that reset the board before every run (and also capped each agent's steps, so I can't credit the reset alone) got far higher precision, at the cost of some recall. A shared memory needs a way to forget that the agents will use, otherwise yesterday's guesses turn into today's evidence. A few other things stuck with me. The most expensive failures were the silent ones: for three runs, semantic search was pointing at the wrong vector field and returning nothing, without a single error, while the agents kept working and the scores looked plausible. I only caught it by asking why the synthesizer never found any counter-evidence. The test world also has to be dense enough. API rate limits and a batch deployment I never got around to setting up left the simulation at about 1,300 records instead of the 10,000 to 15,000 I'd planned, so most planted problems rested on one or two records, and coincidental matches started to look like patterns. And informal approvals were the hardest to catch: no run ever found a change that was approved only in a chat message, because there was nothing formal to cross-check it against. Adding more agents gave me more real findings, but also a lot more noise. The detector turned out to be the easy part to measure. Whether it's worth anyone's attention depends on precision, and in the end on customers.