We overstated our own result: 44% was 37%, and the credit went to the wrong thing
Our first research post said that 44% of fresh second-look reviews caught a bug that would have shipped, and it credited the fresh session. Both parts were wrong. The corrected post now has the real numbers. This one explains how we got it wrong.
What we said, and what was true
Section titled “What we said, and what was true”| First version | Rebuilt as data | |
|---|---|---|
| Reviews that caught a would-ship bug | 12 of 27 (44%) | 10 of 27 (37%) |
| Main cause | the fresh session, then lighter orchestrated review | a human asking “any gaps?” (7 of 10) |
| Caught on the reviewer’s first pass | not reported | 3 of 10 |
| Caught by a step orchestrated review skips | ranked first | 0 |
The fresh session still earns its keep: it costs about 5% more tokens, and it found at most 3 of the 27 on its own. But most of the value came from a person pushing the reviewer for a second pass.
How it happened
Section titled “How it happened”- The number existed only as prose. The 44% was written up from a reading of transcripts. No per-review table existed, so nobody could re-count it.
- Some of the cases could not be found again. When we rebuilt the 27 from the surviving transcripts, the four cases the original cited by name were not among them. The original probably drew on a wider set of sessions.
- We credited the visible change. The fresh session was the thing we had just built, so it got the credit. The human’s follow-up was in almost every transcript, and that is exactly why it was easy to overlook.
What we changed
Section titled “What we changed”- The study is now a per-review table anyone can re-count, built in sequant-io/sequant#1067.
- The docs that repeated 44% were corrected in sequant-io/sequant#1309.
- Sequant has an automated version of the human’s follow-up: a full-weight second review with a required adversarial re-read. Whether it catches what the human’s question caught is not yet measured. That test is sequant-io/sequant#1344, and whichever way it comes out, the result belongs here.
Caveats
Section titled “Caveats”- n = 27, from one project’s workflows.
- The rebuilt labels were assigned by agents with a fixed rubric; a human spot-check is pending.
- “Would-ship bug” is a judgment made after the outcome was known.