Open benchmarks

Every benchmark. Every model. Live.

Name: Benchmarks
Creator: codai

How the codai router compares against each routable upstream on the benchmarks the market actually quotes. Numbers come straight from the gateway — the same path your requests hit — and every figure links to the underlying run, so you can reproduce it with your own key.

Public benchmarks: 17
Completed runs: 277
Spent on transparency: $2217.48

Last refresh · Aug 12, 2026

Pareto frontier

Cost vs accuracy, every model on every benchmark

Each dot is one (model × benchmark) result. Upper-left is the target zone: high accuracy, low cost. Dots on the dashed frontier are non-dominated — nothing beats them on both axes at once.

codai
other models
Pareto frontier (lowest cost at each accuracy)

Coding

Aider Polyglot

225 tasks

225-task multi-language coding benchmark (Python, JS, TS, Rust, Go, Java, C++). Anthropic and OpenAI both quote this for editing performance. Our harness runs the PYTHON subset (~38 tasks) — the largest single bucket — with the official two-pass methodology (one initial try + one retry given test output). Cross-language coverage is the next iteration; competitor scores below are full 225-task polyglot.

Model	Success	$ / task	$ / success	Tasks
`gpt-5 (high)`best	88.0%	$0.1293	$0.1469	225/225
`gpt-5 (medium)`	86.7%	$0.0786	$0.0907	225/225
`claude-opus-4-7`	85.3%	$0.0273	$0.0320	34/225
`o3-pro (high)`	84.9%	$0.6503	$0.7661	225/225
`gemini-2.5-pro-preview-06-05 (32k think)`	83.1%	$0.2217	$0.2667	225/225
`gpt-5 (low)`	81.3%	$0.0461	$0.0567	225/225
`codai`router	8.8%	$0.0257	$0.2910	34/225

Model	Success	$ / task	$ / success	Tasks
`codai,claude-opus-4-8`best	97.0%	$0.0107	$0.0111	164/164
`claude-opus-4-8`	96.3%	$0.0122	$0.0127	164/164
`claude-haiku-4-5`	95.1%	$0.0111	$0.0116	164/164
`claude-sonnet-4-7`	94.5%	$0.0013	$0.0013	164/164
`claude-opus-4-7`	93.9%	$0.0013	$0.0013	164/164
`gpt-5-mini`	82.3%	$0.0161	$0.0195	164/164
`codai`router	3.7%	$0.0014	$0.0370	164/164

Model	Success	$ / task	$ / success	Tasks
`claude-sonnet-5`best	85.2%	—	—	500/500
`20251205_sonar-foundation-agent_claude-opus-4-5`	79.2%	—	—	500/500
`20251215_livesweagent_claude-opus-4-5`	79.2%	—	—	500/500
`20250928_trae_doubao_seed_code`	78.8%	—	—	500/500
`OpenHands + Claude Opus 4.5`	77.6%	—	—	500/500
`20251120_livesweagent_gemini-3-pro-preview`	77.4%	—	—	500/500
`codai`router	46.0%	$0.3398	$0.7387	100/100

Model	Success	$ / task	$ / success	Tasks
`codai`routerbest	47.0%	$1.2653	$2.6922	100/100
`claude-opus-4-7`	46.0%	$1.2709	$2.7627	100/100
`claude-opus-4-8`	36.0%	$0.3656	$1.0156	100/100
`claude-haiku-4-5`	35.0%	$0.0938	$0.2680	100/100
`gpt-5-mini`	31.0%	$1.2862	$4.1490	100/100
`claude-sonnet-4-6`	29.0%	$0.2451	$0.8451	100/100

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	0.0%	$0.0167	—	2/2
`claude-opus-4-8`best	100.0%	$0.2595	$0.2595	2/2
`codai,claude-opus-4-8`	100.0%	$0.2681	$0.2681	2/2
`codai-alpha-1-0`	50.0%	$0.5973	$1.1946	2/2
`gemini-3.1-pro-preview`	0.0%	$0.0083	—	2/2
`gemini-3.5-flash-lite`	0.0%	$0.0109	—	2/2

Model	Success	$ / task	$ / success	Tasks
`codai,claude-opus-4-8`best	85.7%	$0.0734	$0.0857	7/7
`claude-fable-5`	71.4%	$0.4405	$0.6167	7/7
`claude-opus-4-8`	71.4%	$0.0552	$0.0773	7/7
`claude-haiku-4-5`	71.4%	$0.0244	$0.0341	7/7
`claude-sonnet-4-6`	57.1%	$0.0669	$0.1170	7/7
`gemini-3.5-flash-lite`	28.6%	$0.0008	$0.0029	7/7
`codai`router	0.0%	$0.0028	—	7/7

Model	Success	$ / task	$ / success	Tasks
`codai`routerbest	100.0%	$0.0038	$0.0038	10/10
`gemini-3.1-pro-preview`	100.0%	$0.0040	$0.0040	10/10
`gemini-3.1-flash-lite`	100.0%	$0.0003	$0.0003	10/10
`gemini-3.6-flash`	100.0%	$0.0026	$0.0026	10/10
`claude-opus-4-8`	100.0%	$0.0121	$0.0121	10/10
`codai,claude-opus-4-8`	100.0%	$0.0189	$0.0189	10/10

Model	Success	$ / task	$ / success	Tasks
`codai`routerbest	93.3%	$0.0413	$0.0442	60/60
`claude-opus-4-8`	91.7%	$0.0326	$0.0356	60/60
`codai,claude-opus-4-8`	90.0%	$0.0260	$0.0289	60/60
`gemini-3.5-flash-lite`	3.3%	$0.0130	$0.3888	60/60
`codai-alpha-1-0`	0.0%	$0	—	60/60

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	95.0%	$0.0360	$0.0378	20/20
`claude-opus-4-8`best	100.0%	$0.0312	$0.0312	20/20
`codai,claude-opus-4-8`	100.0%	$0.0330	$0.0330	20/20
`claude-sonnet-4-6`	100.0%	$0.1140	$0.1140	20/20
`claude-haiku-4-5`	95.0%	$0.0211	$0.0223	20/20
`codai-alpha-1-0`	0.0%	$0	—	20/20

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	70.0%	$0.0598	$0.0854	10/10
`gemini-3.6-flash`best	100.0%	$0.0449	$0.0449	10/10
`codai,claude-opus-4-8`	100.0%	$0.5520	$0.5520	10/10
`claude-opus-4-8`	100.0%	$0.4434	$0.4434	10/10
`gemini-3.5-flash`	70.0%	$0.0317	$0.0452	10/10
`gemini-3.1-pro-preview`	50.0%	$0.0135	$0.0270	10/10

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	81.5%	$0.0055	$0.0068	200/12032
`codai,claude-opus-4-8`best	82.0%	$0.0062	$0.0076	200/12032
`claude-opus-4-8`	79.0%	$0.0079	$0.0100	200/12032
`gpt-5-mini`	75.5%	$0.0102	$0.0135	200/200
`claude-haiku-4-5`	65.5%	$0.0101	$0.0155	200/12032
`claude-opus-4-7`	62.0%	$0.0004	$0.0007	200/200

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	98.0%	$0.0374	$0.0382	100/100
`claude-opus-4-7`best	99.0%	$0.0164	$0.0166	100/100
`claude-opus-4-8`	98.0%	$0.0232	$0.0237	100/100
`claude-haiku-4-5`	96.0%	$0.0099	$0.0103	100/100
`claude-sonnet-4-6`	96.0%	$0.0146	$0.0152	100/100
`gpt-5-mini`	80.0%	$0.0340	$0.0424	100/100

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	81.5%	$0.0055	$0.0068	200/12032
`codai,claude-opus-4-8`best	82.0%	$0.0062	$0.0076	200/12032
`claude-opus-4-8`	79.0%	$0.0079	$0.0100	200/12032
`gpt-5-mini`	75.5%	$0.0102	$0.0135	200/200
`claude-haiku-4-5`	65.5%	$0.0101	$0.0155	200/12032
`claude-opus-4-7`	62.0%	$0.0004	$0.0007	200/200

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	98.0%	$0.0374	$0.0382	100/100
`claude-opus-4-7`best	99.0%	$0.0164	$0.0166	100/100
`claude-opus-4-8`	98.0%	$0.0232	$0.0237	100/100
`claude-haiku-4-5`	96.0%	$0.0099	$0.0103	100/100
`claude-sonnet-4-6`	96.0%	$0.0146	$0.0152	100/100
`gpt-5-mini`	80.0%	$0.0340	$0.0424	100/100

Every benchmark. Every model. Live.

Public benchmarks

Coding

Agentic

Reasoning

Romanian

How we keep the numbers honest

Public benchmarks

Coding

Agentic

Reasoning

Romanian

How we keep the numbers honest