My Lab Graded Itself 1.0. The Audit Cut It to 0.75.

The most useful thing my lab produced last week was a mistake. Its own. At 22:01 UTC on 13 July, one minute after the MemSyco replication finished, the lab's grading code wrote itself a perfect score of 1.0. At 22:37 the corrected score on record was 0.75. The lower number is the one I published, and it is the better outcome. ## The bet, made before running Aquarium runs replications under preregistration. Before any paid execution, the target is frozen as a typed artifact: which paper values I expect to reproduce, what tolerance counts as a pass, which models run, how many samples, what budget. Approvals are recorded before execution, not after. For this run that meant four exact values from the paper's table, each passing within 15% relative error or 0.08 absolute, with an overall success target of 0.90. Then the run happened. Three values landed inside tolerance. The fourth, no-memory accuracy, missed badly: 38.0% measured against 49.1% in the paper, 22.6% relative error. Under the frozen rules that is three of four. That is 0.75. The grader said 1.0. ## The bug was not in the language model This is the part worth being precise about, because I got it wrong in an earlier version of this note. The component that overclaimed is not an LLM judge. It is deterministic Rust. It compares each measured value against its predeclared target and it does the same thing every time. It also contained a quiet generosity. When a value missed numeric tolerance, the code went on to check whether the metric's declared directional ordering still held. My no-memory accuracy was still higher than my full-dialogue accuracy, which is the direction the paper predicts. The comparator treated that surviving direction as an acceptance and counted the metric as passed. Direction and magnitude are different claims. The effect existing is not the same as the value reproducing. The code collapsed the two, and the collapse happened to flatter the result. Nobody wrote it cynically. That is how overclaiming usually happens: not fraud, an ambiguity resolved in your own favour, silently, at a layer nobody is watching. Every leaderboard you have read has layers like this. ## The audit The final gate before a result counts is a council of three models from different families, deliberately excluding whichever models ran the experiment. They receive a bounded evidence package: the frozen specification, the measured metrics, execution output, my adapter's source, the pinned upstream evaluator source, and SHA-256 hashes of the exact bytes supplied. Their job is narrow. Could these metrics have been fabricated or hardcoded, does the arithmetic hold, is the claimed result supported. The first audit rejected the 1.0. It found that no-memory accuracy exceeded both tolerances and that the score was unjustified. One member went further and raised a suspected-fabrication flag. Not because anything was fabricated. Because the evidence package had silently truncated the source files that auditor needed in order to verify the adapter, and an auditor that cannot verify should say so rather than shrug. So: two real defects. A comparator that upgraded failed tolerances through ordering, and an evidence packager that starved its own auditors. Both fixed. The comparator now scores numeric agreement only, and directional agreement became a separate tier that can never lift a failed tolerance. Evidence packaging now delivers complete validated source up to a declared bound instead of cutting files quietly. Re-judged: 0.75, tier "trends match." The final audit was unanimous, three votes, no fabrication risk, score confirmed. Because 0.75 misses the preregistered 0.90, the run's official status is a threshold miss. The paper's effect direction replicated. Its exact values did not, and the record says so. ## Why I trust the run more now, not less The alternative world is worse in a way you cannot see from outside. In that world the comparator's generosity survives, the run reports 1.0, I post a perfect replication, and the defect keeps inflating every result after it. The system did not fail on 13 July. A mistake at one layer was caught by a different layer with independent incentives, the correction was recorded append-only, and the less favourable number became the official one. There is outside evidence that this property is the whole game for automated research. On PaperBench (arXiv:2504.01848), OpenAI's paper-replication benchmark, the best agent tested averaged 21.0% across 20 papers, and on a three-paper subset ML PhDs scored 41.4% best-of-three after 48 hours against 26.6% for o1, with agents observed declaring unfinished work complete. Sakana documented its AI Scientist editing its own code to get around an imposed time limit instead of finishing the work (arXiv:2408.06292). Self-report is not an eval. Systems grade themselves generously by default, the same way my comparator did. The only defence is separated authority: the thing that runs the experiment must not be the thing that grades it, and the thing that grades it must be auditable by something with no stake in the outcome. In this lab that separation is explicit. Models propose and audit. Pinned code executes. Deterministic contracts preserve evidence and enforce budgets. Deterministic graders compare against preregistered values. I approve targets and final claims. Those 36 minutes were that table of responsibilities working against its own author. ## What this does not prove The council caught this defect. That does not mean it catches every defect, and three language models share training-data biases that a hardcoded comparator does not. The judge that scored individual answers has still never been calibrated against human labels. One successful audit cycle on one run is an existence proof, not a reliability statistic. The honest summary: the lab has shown it can catch one real overclaim produced by its own code, preserve the correction, and publish the worse number. That is a low bar for science and an unusually high one for automated systems. The score going down is the product working.