Multi-Agent Specialization Took Recall From 13% to 80%. Then Precision Fell Apart.

Before Aquarium existed, I built a smaller multi-agent system for Upright: four agents investigating a synthetic construction company's records for planted anomalies, no hints, just the data. This is that experiment's real numbers, including the ones that got worse. ## The setup The dataset: a synthetic construction general contractor's operational records, generated across 90 days and 4 projects, with 15 anomalies planted in it (a duplicate invoice, a lien waiver gap, a safety incident never logged in the compliance system, that kind of thing). An investigator agent reads the records and reports findings with zero prior knowledge of what was planted. A separate evaluator scores each run against the 15-item answer key. ## Run 1: one agent, 13% recall The first version was a single agent trying to cover invoices, compliance, and scheduling at once. It looped, re-covering ground it had already searched, and found 2 of the 15 planted anomalies. 13% recall. ## Run 2: four agents, one blackboard, 80% recall The fix was specialization, not a bigger model or a longer prompt. Four agents, each with one domain: a financial auditor on invoices and change orders, a compliance officer on lien waivers and safety incidents, a schedule analyst on RFIs and timeline gaps, and a synthesizer whose job was to read what the other three found and challenge it. All four wrote to a shared blackboard so later agents could see earlier findings instead of duplicating them. Recall jumped to 80%, 12 of 15. Precision was 44%, 18 findings on the blackboard, F1 56%. That is the best run in the set, and it is not the one I want to talk about most, because it is the run where the second problem was still small enough to miss. ## Runs 3 and 4: recall held, precision did not | Run | Recall | Precision | F1 | Findings on blackboard | |---|---|---|---|---| | 1 (single agent) | 13% | (n/a) | (n/a) | (n/a) | | 2 (multi-agent) | 80% | 44% | 56% | 18 | | 3 | 80% | 21% | 34% | 56 | | 4 | 80% | 20% | 32% | 60 | Recall never moved after the architecture fix, 12 of 15 in every multi-agent run, including run 3 and run 4 where the agents were relying on memory of prior sessions rather than starting cold. That part worked exactly as intended. Precision collapsed. The blackboard was designed to accumulate across runs so agents could build on prior findings, but nothing on it ever got removed. The synthesizer's job was to prune false leads, and by run 4 it had 60 findings and roughly 25 tool-call steps to review them in, which is enough budget for 2 or 3 findings done properly, not 60. A dedicated tool called `retract_finding` shipped in run 4 for exactly this job. It was never called once, because the counter-evidence a synthesizer would need to justify a retraction barely existed in a dataset this sparse: most planted anomalies had only 1 or 2 supporting records, so a coincidental match to 2 records was often enough to clear the bar for a finding in the first place. Underneath both of those, a separate bug was quietly making things worse for three of the four runs. The semantic search call used the wrong vector field name (`embedding` instead of `contentVector`), so every semantic search silently failed for runs 1 through 3. Silently, meaning no error surfaced, the call just returned nothing useful. I only found it while diagnosing why the synthesizer wasn't finding counter-evidence. Fixed for run 4. It did not fix precision on its own, because the deeper cause, the sparse dataset and the unpruned blackboard, was still there. Three anomalies were never found in any of the four runs: a verbal change-order approval that only exists as a Slack message with no matching Procore record, a duplicate invoice from the same vendor under two different name spellings, and a submittal approval chain ambiguous enough that no single record proves it wrong. ## The framing I did not use Here is a real decision point from the retrospective I wrote at the time. I had two ways to describe run 4: > "AI found 80% of anomalies but also flagged 48 noise items." or > "AI found 12 of 15 planted anomalies plus 48 additional risk items worth review, in a real engagement these would all get validated." The second version is not false. A human reviewer would in fact triage all 60 findings, and some of the 48 "false positives" might turn out to be real issues the planted set didn't cover. But it is the version built to sound better rather than the version built to be checked, and it quietly moves the precision problem from "unsolved" to "somebody else's job." I am using the first framing here, the one with 48 as a cost, not a feature, because a system that surfaces 4 false leads for every real one is not done, whatever a later human reviewer would eventually catch. ## Where this actually stalled The retrospective's stated root cause for the low precision was data density: 1,316 generated records against a target of 5,000 to 20,000, because the batch-generation pipeline needed an Azure GlobalBatch deployment that was never created, so generation stayed capped at whatever the smaller real-time runs produced. With planted anomalies backed by only 1 or 2 records each, there wasn't enough surrounding evidence for the synthesizer to distinguish a real pattern from a coincidence, and the dataset was never expanded to test whether more records would have fixed it. The project is paused there. The plan on record before it stopped: move off the Azure AI Search and Batch API stack (cited at $75+/month for a demo that never got its batch generation working) toward OpenRouter plus a local vector store, actually reach the target record count, and rebuild the blackboard to snapshot per run instead of accumulating forever, before touching the agent logic again. Multi-agent specialization is a real, repeatable fix for recall. It is not a fix for precision, and treating a recall win as if it also solved precision is exactly the kind of framing this whole system exists to catch me making.