Research log
Findings from a corpus of recorded agent runs with known outcomes. Each post states its sample size and method up front, and links to the source data.
These are measurements, not benchmarks. Sample sizes are small, they come from real workflows rather than a controlled harness, and every post says so. Where a number is too soft to lean on, the post says that too.
Posts go up when a finding is ready.
- We overstated our own result: 44% was 37%, and the credit went to the wrong thing
Our first research post said 44% (12 of 27) of fresh second-look reviews caught a would-ship bug, and credited the fresh session. Rebuilt as a per-review table, it was 10 of 27 (37%), and 7 of the 10 appeared only after the human asked the reviewer for gaps. The original post is corrected; this is what went wrong and what we changed.
- A second look caught a would-ship bug in 10 of 27 reviews, and 7 of those came after a human asked again
Across 27 recorded second-look reviews, 10 (37%) caught a would-ship bug. 7 of the 10 appeared only after the human asked the reviewer for gaps, and 3 were in its first verdict. Starting the reviewer in a clean session costs about 5% more tokens. This post originally reported 44% and credited the fresh session; it was revised on 2026-10-07 after the study was rebuilt as data.