Open benchmarks

Reasoning benchmarks

Reasoning benchmarks measure whether a model reaches the right answer by a sound method: multiple-choice knowledge across domains, integer-answer competition maths, needle retrieval in long contexts and multi-step problem solving. Most are scored deterministically, without an LLM judge.

Which run is shown

One row per model per benchmark: the completed run with the largest sample wins, and on a tie the most recent finish wins. Small smoke runs cannot lift a model by getting lucky. Failed and cancelled runs are excluded.

The headline metric

Cost per successful task. It punishes both expensive correct answers and cheap wrong ones, so it is the number that is hardest to game. Success rate and total cost sit next to it.

MMLU-Pro, competition maths (AIME), long context, GraphWalks and problem solving.

Results · Reasoning

Benchmarks: 5
Model results: 29

Cost vs accuracy, every model on every benchmark

Each dot is one (model × benchmark) result. Upper-left is the target zone: high accuracy, low cost. Dots on the dashed frontier are non-dominated — nothing beats them on both axes at once.

codai
other models
Pareto frontier (lowest cost at each accuracy)

Competition Math (AIME)

90 tasks

Hard mathematical-reasoning benchmark built from AIME 2024, 2025, and 2026 (American Invitational Mathematics Examination). Every answer is an integer in 0..999, so scoring is fully DETERMINISTIC and un-gameable — we extract the model's final integer (`Final answer: N` sentinel, then \boxed{N}, then last integer) and compare for exact equality. No LLM judge, so no judge cost or judge variance poisoning the reasoning priors. Closes a frontier category (hard reasoning/math) the scorecard was previously blind to.

Reference →

Above-median accuracy, 71% below average cost

codai hits 93.3% (median across 5 models: 46.7%) while spending $0.0442 per success — the field averages $0.1511.

Model	Success	$ / task	$ / success	Tasks
`codai`routerbest	93.3%	$0.0413	$0.0442	60/60
`claude-opus-4-8`	91.7%	$0.0326	$0.0356	60/60
`codai,claude-opus-4-8`	90.0%	$0.0260	$0.0289	60/60
`gemini-3.5-flash-lite`	3.3%	$0.0130	$0.3888	60/60
`codai-alpha-1-0`	0.0%	$0	—	60/60

GraphWalks (Long-Context)

20 tasks

Frontier v8 long-context multi-hop reasoning gate. Each task embeds a large deterministically-generated directed graph in the prompt and asks a BFS-reachability or direct-parents query; scored by exact node-set match (no judge). Modelled on the GraphWalks eval in the Anthropic Mythos 5 / Fable 5 system card (§8, BFS 256K). Un-gameable: graphs are generated per-seed, never downloaded. Cost scales with nodeCount (context length).

Reference →

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	95.0%	$0.0360	$0.0378	20/20
`claude-opus-4-8`best	100.0%	$0.0312	$0.0312	20/20
`codai,claude-opus-4-8`	100.0%	$0.0330	$0.0330	20/20
`claude-sonnet-4-6`	100.0%	$0.1140	$0.1140	20/20
`claude-haiku-4-5`	95.0%	$0.0211	$0.0223	20/20
`codai-alpha-1-0`	0.0%	$0	—	20/20

Long Context (RULER-style)

10 tasks

Long-context retrieval benchmark (RULER-style needle-in-a-haystack). Fully SYNTHETIC + DETERMINISTIC: a filler haystack is generated at a target token budget (2k -> 128k), magic-number needles are inserted at spread depths, and the model must retrieve them. Scored by exact value match (all needles required). No external dataset, no LLM judge. Sweeps context lengths so we can see where each model’s effective window degrades. Closes the long-context frontier category.

Reference →

Above-median accuracy, 54% below average cost

codai hits 70.0% (median across 8 models: 70.0%) while spending $0.0854 per success — the field averages $0.1841.

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	70.0%	$0.0598	$0.0854	10/10
`gemini-3.6-flash`best	100.0%	$0.0449	$0.0449	10/10
`codai,claude-opus-4-8`	100.0%	$0.5520	$0.5520	10/10
`claude-opus-4-8`	100.0%	$0.4434	$0.4434	10/10
`gemini-3.5-flash`	70.0%	$0.0317	$0.0452	10/10
`gemini-3.1-pro-preview`	50.0%	$0.0135	$0.0270	10/10

MMLU-Pro

12,032 tasks

TIGER-Lab's 12,032-question reasoning benchmark across 14 domains (biology, business, chemistry, CS, economics, engineering, health, history, law, math, philosophy, physics, psychology, other). 10-choice multiple-choice, deterministic letter-match scoring. We run a 200-question stratified sample per category (28 per category, capped at the available count).

Reference →

Above-median accuracy, 97% below average cost

codai hits 81.5% (median across 9 models: 63.7%) while spending $0.0068 per success — the field averages $0.2178.

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	81.5%	$0.0055	$0.0068	200/12032
`codai,claude-opus-4-8`best	82.0%	$0.0062	$0.0076	200/12032
`claude-opus-4-8`	79.0%	$0.0079	$0.0100	200/12032
`gpt-5-mini`	75.5%	$0.0102	$0.0135	200/200
`claude-haiku-4-5`	65.5%	$0.0101	$0.0155	200/12032
`claude-opus-4-7`	62.0%	$0.0004	$0.0007	200/200

Problem Solving (Reasoning)

8 tasks

Frontier v9 #1-priority category. Single-shot HARD reasoning tasks that other harnesses miss: multi-step logic puzzles (balance-weighing, river/bridge crossings), quantitative word problems with intuitive-but-wrong traps, systematic root-cause/debugging strategy, constraint satisfaction, and Fermi estimation. NOT code-execution (humaneval/codai-swe own that) and NOT short-answer contest math (math-competition owns that) — rewards correct multi-step reasoning AND a sound, communicable method. Scored by a weighted-rubric LLM judge (correctness-critical criteria first); a task passes at >=0.7. Mirrors the training-side Rubrics-as-Rewards signal so the eval measures exactly what RLVR+RaR optimizes. Fixed, version-controlled tasks (problem-solving.ts) for reproducibility.

Reference →

No runs published yet for this benchmark.

Download the data

The same rows this page renders, refreshed every 10 minutes. Free to reuse with attribution.

Licence: CC BY 4.0, attribution “codai (codai.ro/benchmarks)”.

All benchmarks Honesty leaderboard