A second look caught a would-ship bug in 10 of 27 reviews, and 7 of those came after a human asked again
An agent that has just written 400 lines of code is a weak reviewer of those 400 lines. It knows what it meant. A second look helps, but the data says the most useful part of the second look was a person asking for more.
The finding
Section titled “The finding”We rebuilt 27 recorded reviews in which a second review pass ran against work the implementing run had already declared finished.
10 of the 27 (37%) caught a real bug that would otherwise have shipped. 13 found only quality gaps, such as docs drift or missing tests for working code. 4 found nothing substantive.
A “would-ship bug” here means a functional defect, an unmet acceptance criterion, a missing implementation, a broken primary path, a test that guards nothing, or a merge that would delete something.
Where the catches came from
Section titled “Where the catches came from”Each catch was attributed to one mechanism:
| Mechanism | Catches |
|---|---|
| The human’s follow-up (“any gaps?“) | 7 |
| The reviewer’s first verdict, in a fresh session | 3 |
| A step that orchestrated review skips | 0 |
| Total | 10 |
- The human follow-up did most of the work. In 7 of the 10 catches, the reviewer’s first report missed the bug. It surfaced only after the person running the review asked it to look for gaps. Almost every session included that question, usually as the human’s first reply.
- The fresh session found 3 on the first pass. That is at most 3 of 27 reviews (11%), and it is an upper bound: the transcripts cannot separate a fresh session from a standalone one.
- The lighter orchestrated review accounted for none. The first version of this post ranked this mechanism first. In the recoverable data, no catch depended on a step the orchestrated review skips.
The headline in one line: 26% (7 of 27) of reviews caught a would-ship bug only because a human asked again, and 11% (3 of 27) caught one on the first pass.
What a fresh session costs
Section titled “What a fresh session costs”The natural implementation is to let the reviewer resume the implementer’s session, since the context is already warm. We measured the cost of not doing that across 82 orchestrator QA passes (44 resumed, 38 fresh):
| Metric (median) | Resumed | Fresh |
|---|---|---|
| Context at review start | 142k | 29k |
| Turns | 30 | 56 |
| Output tokens | 45k | 69k |
| Combined cost-weighted total | 1.58M | 1.65M |
A fresh reviewer takes nearly twice as many turns but starts from about a fifth of the context. The two effects nearly cancel: starting clean costs about 5% more, and that is an upper bound, because the fresh group also includes heavier full-weight passes and re-runs after failures.
The first version of this post said a reviewer that reads the author’s reasoning is “measurably more agreeable”. The rebuilt data does not show that. It shows the fresh session is cheap, and that its catches are at most 3 of 27.
What we changed
Section titled “What we changed”Sequant’s review phase no longer resumes the implementing session, so every review starts in a fresh session. Given the cost, we kept that.
The bigger lesson is the follow-up. (This paragraph was added in the 2026-10-07
revision.) Sequant now has an automated version of it:
a second, full-weight review with a required adversarial re-read (sequant ready, and sequant run --ready-gate). Whether it catches what the human’s
question caught has not been measured yet. That measurement is
sequant-io/sequant#1344,
and whichever way it comes out, the result belongs here.
Caveats
Section titled “Caveats”- n = 27. That is enough to show the follow-up matters, and not enough to pin any rate precisely.
- The labels were assigned by agents with a fixed rubric, from the first report and the turns after the gap question. A human spot-check of the labels is still pending.
- The four cases the original study cited could not be recovered as reviews from the surviving transcripts, which is part of why 12 became 10.
- “Would-ship bug” is a judgment made after the fact, against transcripts where the outcome was already known.
- All of it comes from one project’s workflows. The mechanism may generalize; the magnitude may not.
Source
Section titled “Source”The rebuilt per-review table, the attribution rule and the reconciliation with the original count are in qa-second-look-dataset.md (sequant-io/sequant#1067). The token-cost table and the original study are recorded in sequant-io/sequant#982.