Open benchmarks

Coding benchmarks

Coding benchmarks measure whether a model writes code that works: a function that passes its unit tests, or a patch that fixes a real repository issue. Every block is labelled official, proxy, internal or reference, so a cheap stand-in is never read as a leaderboard score.

Which run is shown

One row per model per benchmark: the completed run with the largest sample wins, and on a tie the most recent finish wins. Small smoke runs cannot lift a model by getting lucky. Failed and cancelled runs are excluded.

The headline metric

Cost per successful task. It punishes both expensive correct answers and cheap wrong ones, so it is the number that is hardest to game. Success rate and total cost sit next to it.

HumanEval, SWE-bench Verified (official harness and proxies) and Aider Polyglot.

Results · Coding

Benchmarks: 5
Model results: 140

Cost vs accuracy, every model on every benchmark

Each dot is one (model × benchmark) result. Upper-left is the target zone: high accuracy, low cost. Dots on the dashed frontier are non-dominated — nothing beats them on both axes at once.

codai
other models
Pareto frontier (lowest cost at each accuracy)

SWE-bench Verified (official harness)official

500 tasks

The 500-task SWE-bench Verified set scored by the official SWE-bench Docker harness (resolved = FAIL_TO_PASS and PASS_TO_PASS tests pass). codai agent, single attempt per task, run 2026-08-03/04. Competitor rows are published leaderboard entries (source and date in each row), not re-runs by us.

Reference →

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	73.2%	$0.6095	$0.8326	500/500
`claude-sonnet-5`best	85.2%	—	—	500/500
`20251205_sonar-foundation-agent_claude-opus-4-5`	79.2%	—	—	500/500
`20251215_livesweagent_claude-opus-4-5`	79.2%	—	—	500/500
`20250928_trae_doubao_seed_code`	78.8%	—	—	500/500
`OpenHands + Claude Opus 4.5`	77.6%	—	—	500/500

Aider Polyglot

225 tasks

225-task multi-language coding benchmark (Python, JS, TS, Rust, Go, Java, C++). Anthropic and OpenAI both quote this for editing performance. Our harness runs the PYTHON subset (~38 tasks) — the largest single bucket — with the official two-pass methodology (one initial try + one retry given test output). Cross-language coverage is the next iteration; competitor scores below are full 225-task polyglot.

Reference →

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	61.8%	$0.4031	$0.6527	34/225
`gpt-5 (high)`best	88.0%	$0.1293	$0.1469	225/225
`gpt-5 (medium)`	86.7%	$0.0786	$0.0907	225/225
`claude-opus-4-7`	85.3%	$0.0273	$0.0320	34/225
`o3-pro (high)`	84.9%	$0.6503	$0.7661	225/225
`gemini-2.5-pro-preview-06-05 (32k think)`	83.1%	$0.2217	$0.2667	225/225

HumanEval

164 tasks

OpenAI 164-problem Python coding benchmark. Single-turn: prompt → function body → run unit tests. Cheap, fast, well-known — used as the pipeline smoke benchmark.

Reference →

Above-median accuracy, 78% below average cost

codai hits 96.3% (median across 11 models: 88.1%) while spending $0.0092 per success — the field averages $0.0422.

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	96.3%	$0.0089	$0.0092	164/164
`codai,claude-opus-4-8`best	97.0%	$0.0107	$0.0111	164/164
`claude-opus-4-8`	96.3%	$0.0122	$0.0127	164/164
`claude-haiku-4-5`	95.1%	$0.0111	$0.0116	164/164
`claude-sonnet-4-7`	94.5%	$0.0013	$0.0013	164/164
`claude-opus-4-7`	93.9%	$0.0013	$0.0013	164/164

SWE-bench Verified (oracle-diff proxy, 100)proxy

100 tasks

PROXY, not the official harness: 100 SWE-bench Verified issues, the model gets the issue plus the oracle affected files and outputs a unified diff, scored by line-overlap similarity to the gold patch plus an LLM judge on borderline cases. No tests are executed. Not comparable to leaderboard numbers; see "SWE-bench Verified (official harness)" for the real score.

Reference →

Above-median accuracy, 46% below average cost

codai hits 46.0% (median across 6 models: 1.0%) while spending $0.7387 per success — the field averages $1.3571.

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	46.0%	$0.3398	$0.7387	100/100
`claude-sonnet-5`best	85.2%	—	—	500/500
`claude-fable-5`	65.0%	$0.3871	$0.5956	100/100
`gemini-3.5-flash`	1.0%	$0.0212	$2.1185	100/100
`gemini-3.1-flash-lite`	0.0%	$0.0028	—	100/100
`gemini-3.5-flash-lite`	0.0%	$0.0037	—	100/100

SWE-bench Verified (patch-only)proxy

100 tasks

Patch-only variant of SWE-bench Verified. Model receives the issue + the 'oracle' affected files (princeton-nlp/SWE-bench_oracle pre-formatted text) and must output a unified diff that fixes the issue. NO Docker, NO test execution — we score the candidate diff against the gold patch with line-overlap similarity (cheap fast-path) plus an LLM judge (sonnet-4-6) for borderline cases (0.3 < sim < 0.7). Not directly comparable to the official leaderboard but a strong proxy for diff-generation quality at 1% of the cost.

Reference →

Model	Success	$ / task	$ / success	Tasks
`codai`routerbest	47.0%	$1.2653	$2.6922	100/100
`claude-opus-4-7`	46.0%	$1.2709	$2.7627	100/100
`claude-opus-4-8`	36.0%	$0.3656	$1.0156	100/100
`claude-haiku-4-5`	35.0%	$0.0938	$0.2680	100/100
`gpt-5-mini`	31.0%	$1.2862	$4.1490	100/100
`claude-sonnet-4-6`	29.0%	$0.2451	$0.8451	100/100

Download the data

The same rows this page renders, refreshed every 10 minutes. Free to reuse with attribution.

Licence: CC BY 4.0, attribution “codai (codai.ro/benchmarks)”.

All benchmarks Honesty leaderboard