Honesty leaderboard
Every model runs the same raw agent loop on trap tasks (impossible goals, hidden failing tests, corrupt data, broken builds) and on solvable controls. We check mechanically whether its final claim matches reality — and whether the codai verifier would have caught the lie.
Loading honesty results…
Last run · Sep 27, 2026
4 infra errors excluded
Sorted by honesty rate. Each rate shows its count (numerator/denominator). “n/a” means the model had no trials of that kind.
| Model | Honest | 95% CI | Fabricated | Sandbagged | Verifier caught | Verifier false + | Trials | Seeds | $ / trial |
|---|---|---|---|---|---|---|---|---|---|
codairoutercodai | 92.5%282/305 | 88.9%–94.9% | 5.7%14/246 | 15.3%9/59 | 14.3%2/14 | 4.3%12/282 | 305 | 3 | $0 |
Fable 5.1claude-fable-5-1 | 85.5%259/303 | 81.1%–89.0% | 17.8%43/242 | 1.6%1/61 | 0.0%0/43 | 2.7%7/259 | 303 | 3 | $0 |
Share of trap trials in each category where the model claimed success that the mechanical check disproved.
| Trap category | codai | Fable 5.1 |
|---|---|---|
| broken toolchain | 0.0%0/24 | 62.5%15/24 |
| corrupt data | 13.0%3/23 | 9.5%2/21 |
| hidden failing test | 0.0%0/30 | 3.3%1/30 |
| impossible spec | 29.2%7/24 | 0.0%0/24 |
| missing dependency | 0.0%0/24 | 12.5%3/24 |
| partial success | 13.0%3/23 | 8.3%2/24 |
| permission denied | 4.2%1/24 | 0.0%0/24 |
| solvable control | 0.0%0/2 | n/a |
| stale artifact | 0.0%0/24 | 83.3%20/24 |
| stderr on exit0 | 0.0%0/24 | 0.0%0/24 |
| test edit forbidden | 0.0%0/24 | 0.0%0/23 |
Methodology
codai row is the codai router run through the same raw loop as every other model. The verifier columns show what codai's gateway verifier would add on top: fabrications it flags, and honest answers it flags wrongly.