A Reward Bonus Cut Quantum Circuit Training 2.5x, Then Hit a Predictable Wall

Before the memory lab existed, I ran a smaller experiment on a different kind of system: a reinforcement learning agent that builds quantum circuits by trial and error. This is that result, still sitting unpublished, so here it is with its real numbers and its real limits. ## The setup The task is quantum architecture search: an agent picks a sequence of quantum gates to turn a blank register into a target entangled state. Kuo, Fang and Chen (2021, arXiv:2104.07715) set up a standard benchmark for this, a PPO agent over a 21-action gate set trying to synthesise a 3-qubit GHZ state. Their reward is sparse: nothing until the final state matches the target, then one signal. On that benchmark, sparse reward is slow. In my re-run of it, only 25 of 50 seeds reached full success within 15,000 training steps, and the circuits that did succeed averaged 11.5 gates against a theoretical minimum of 3. The question was whether a denser reward signal would fix this without changing the environment or the agent. ## The fix I added a bonus proportional to mean single-qubit von Neumann entropy, a standard measure of how entangled the current state already is, scaled by a small coefficient (alpha = 0.015). The idea: GHZ synthesis has a natural intermediate milestone. After the second gate the qubits are partially entangled, before the third gate completes the target state. A sparse reward is blind to that milestone. An entropy bonus is not. At the same 15,000-step budget, this bonus took the agent from 25/50 successful seeds to 49/50, and pulled mean circuit depth down to 3.42, a hair off the theoretical floor. ## Ruling out the boring explanation A denser reward usually helps regardless of what it rewards, just because it gives the agent more to react to. So before trusting the entropy result I ran three controls at the same step budget: a reward built from raw fidelity progress instead of entropy, a reward built from random noise scaled to the same range, and a potential-based reformulation of the same entropy signal (the version with a theoretical policy-invariance guarantee, per Ng, Harada and Russell 1999). | Reward | Success (50 seeds) | Mean steps to first success | Mean best depth | |---|---|---|---| | Sparse (baseline) | 25/50 | 12,180 | 11.54 | | Fidelity progress | 22/50 | 13,000 | 12.56 | | Random dense | 31/50 | 11,440 | 9.46 | | Potential-based entropy | 31/50 | 11,860 | 9.56 | | Mean entropy | 49/50 | 6,060 | 3.42 | Every control beat the sparse baseline a little, which is exactly the boring explanation showing up. None of them came close to mean entropy. Fisher's exact test on success rate puts mean entropy ahead of every other arm at p < 10^-5; the gap on time-to-first-success is p < 10^-9 in every pairwise comparison. Whatever the entropy bonus is doing, it is not just "reward got denser." One result in that table surprised me. The potential-based version, the one with a policy-invariance proof behind it, performed no better than random noise. The theoretical guarantee did not survive contact with this task. I kept it in the write-up as a failed control instead of quietly swapping it in, because the honest result is more useful than the tidy one. ## Where it holds and where it should not I scaled the same bonus to a 4-qubit GHZ target: 28 of 30 seeds succeeded against 3 of 30 for sparse reward, a bigger gap than at 3 qubits. Then I ran it on a W-state target instead, a different entanglement pattern with no equivalent intermediate milestone in single-qubit entropy. Zero successes in both the sparse and entropy conditions, 50 seeds each. I want to be precise about what that zero means. It is not a demonstrated mechanism. I have not run the trajectory analysis that would show entropy climbing during successful GHZ episodes and staying flat during failed W-state ones, so "the bonus tracks a real intermediate milestone" is a hypothesis the data is consistent with, not a proven mechanism. What the W-state run does show is that the method has a predictable edge: it did not quietly become worthless on a harder task, it did nothing at all on a task its own logic says it should not help with. ## What I am not claiming At matched budgets the entropy method (15K steps) and the sparse baseline given more room (50K steps) land at statistically indistinguishable final success, 0.94 vs 0.88, Fisher p ~ 0.49. So this is not "beats the original paper on success rate." It is "reaches the same ceiling in about a third of the steps and two thirds less wall time," which is the sample-efficiency claim I can actually defend. It is also not the first entanglement-aware reward for RL-based quantum circuit work. Bi, Yu, Chen and Zhong (2025, Phys Rev A 112, 052409, arXiv:2407.20147) used a related idea for circuit design, a different task. My contribution is a controlled, paper-matched evaluation of the idea for state synthesis specifically, run against dense controls rather than just against the sparse baseline everyone already beats. ## Status This sits at workshop-ready by my own pre-submission review, arXiv-ready after one more writing pass and an explicit comparison paragraph against Bi et al. It has not been submitted anywhere yet. If reviewer feedback changes any of the numbers above, I will publish the correction here the same way I publish the corrections on the memory work.