Benchmark-uri deschise

Benchmark-uri de raționament

Benchmark-urile de raționament măsoară dacă un model ajunge la răspunsul corect printr-o metodă solidă: grile de cunoștințe din mai multe domenii, matematică de concurs cu răspuns întreg, regăsire în contexte lungi și rezolvare de probleme în mai mulți pași. Majoritatea sunt punctate determinist, fără judecător LLM.

Ce rulare este afișată

Un rând per model per benchmark: câștigă rularea finalizată cu cel mai mare eșantion, iar la egalitate câștigă cea terminată cel mai recent. Rulările mici de test nu pot avantaja un model printr-un noroc. Rulările eșuate sau anulate sunt excluse.

Metrica principală

Costul per sarcină reușită. Penalizează atât răspunsurile corecte dar scumpe, cât și pe cele ieftine dar greșite, deci este cifra cel mai greu de manipulat. Rata de reușită și costul total stau alături.

MMLU-Pro, matematică de concurs (AIME), context lung, GraphWalks și rezolvare de probleme.

Rezultate · Raționament

Benchmark-uri: 5
Rezultate per model: 29

Cel mai recent rezultat afișat · 12 aug. 2026

Frontiera Pareto

Cost vs acuratețe, fiecare model pe fiecare benchmark

Fiecare punct este un rezultat (model × benchmark). Stânga-sus este zona țintă: acuratețe mare, cost mic. Punctele de pe frontiera punctată sunt nedominate — nimic nu le bate pe ambele axe simultan.

codai
alte modele
Frontiera Pareto (cel mai mic cost la fiecare nivel de acuratețe)

Competition Math (AIME)

90 de task-uri

Hard mathematical-reasoning benchmark built from AIME 2024, 2025, and 2026 (American Invitational Mathematics Examination). Every answer is an integer in 0..999, so scoring is fully DETERMINISTIC and un-gameable — we extract the model's final integer (`Final answer: N` sentinel, then \boxed{N}, then last integer) and compare for exact equality. No LLM judge, so no judge cost or judge variance poisoning the reasoning priors. Closes a frontier category (hard reasoning/math) the scorecard was previously blind to.

Referință →

Acuratețe peste mediană, cost cu 71% sub medie

codai atinge 93.3% (mediana celor 5 modele: 46.7%) cheltuind $0.0442 per succes — media câmpului este $0.1511.

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routercel mai bun	93.3%	$0.0413	$0.0442	60/60
`claude-opus-4-8`	91.7%	$0.0326	$0.0356	60/60
`codai,claude-opus-4-8`	90.0%	$0.0260	$0.0289	60/60
`gemini-3.5-flash-lite`	3.3%	$0.0130	$0.3888	60/60
`codai-alpha-1-0`	0.0%	$0	—	60/60

GraphWalks (Long-Context)

20 de task-uri

Frontier v8 long-context multi-hop reasoning gate. Each task embeds a large deterministically-generated directed graph in the prompt and asks a BFS-reachability or direct-parents query; scored by exact node-set match (no judge). Modelled on the GraphWalks eval in the Anthropic Mythos 5 / Fable 5 system card (§8, BFS 256K). Un-gameable: graphs are generated per-seed, never downloaded. Cost scales with nodeCount (context length).

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	95.0%	$0.0360	$0.0378	20/20
`claude-opus-4-8`cel mai bun	100.0%	$0.0312	$0.0312	20/20
`codai,claude-opus-4-8`	100.0%	$0.0330	$0.0330	20/20
`claude-sonnet-4-6`	100.0%	$0.1140	$0.1140	20/20
`claude-haiku-4-5`	95.0%	$0.0211	$0.0223	20/20
`codai-alpha-1-0`	0.0%	$0	—	20/20

Long Context (RULER-style)

10 task-uri

Long-context retrieval benchmark (RULER-style needle-in-a-haystack). Fully SYNTHETIC + DETERMINISTIC: a filler haystack is generated at a target token budget (2k -> 128k), magic-number needles are inserted at spread depths, and the model must retrieve them. Scored by exact value match (all needles required). No external dataset, no LLM judge. Sweeps context lengths so we can see where each model’s effective window degrades. Closes the long-context frontier category.

Referință →

Acuratețe peste mediană, cost cu 54% sub medie

codai atinge 70.0% (mediana celor 8 modele: 70.0%) cheltuind $0.0854 per succes — media câmpului este $0.1841.

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	70.0%	$0.0598	$0.0854	10/10
`gemini-3.6-flash`cel mai bun	100.0%	$0.0449	$0.0449	10/10
`codai,claude-opus-4-8`	100.0%	$0.5520	$0.5520	10/10
`claude-opus-4-8`	100.0%	$0.4434	$0.4434	10/10
`gemini-3.5-flash`	70.0%	$0.0317	$0.0452	10/10
`gemini-3.1-pro-preview`	50.0%	$0.0135	$0.0270	10/10

MMLU-Pro

12.032 de task-uri

TIGER-Lab's 12,032-question reasoning benchmark across 14 domains (biology, business, chemistry, CS, economics, engineering, health, history, law, math, philosophy, physics, psychology, other). 10-choice multiple-choice, deterministic letter-match scoring. We run a 200-question stratified sample per category (28 per category, capped at the available count).

Referință →

Acuratețe peste mediană, cost cu 97% sub medie

codai atinge 81.5% (mediana celor 9 modele: 63.7%) cheltuind $0.0068 per succes — media câmpului este $0.2178.

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	81.5%	$0.0055	$0.0068	200/12032
`codai,claude-opus-4-8`cel mai bun	82.0%	$0.0062	$0.0076	200/12032
`claude-opus-4-8`	79.0%	$0.0079	$0.0100	200/12032
`gpt-5-mini`	75.5%	$0.0102	$0.0135	200/200
`claude-haiku-4-5`	65.5%	$0.0101	$0.0155	200/12032
`claude-opus-4-7`	62.0%	$0.0004	$0.0007	200/200

Problem Solving (Reasoning)

8 task-uri

Frontier v9 #1-priority category. Single-shot HARD reasoning tasks that other harnesses miss: multi-step logic puzzles (balance-weighing, river/bridge crossings), quantitative word problems with intuitive-but-wrong traps, systematic root-cause/debugging strategy, constraint satisfaction, and Fermi estimation. NOT code-execution (humaneval/codai-swe own that) and NOT short-answer contest math (math-competition owns that) — rewards correct multi-step reasoning AND a sound, communicable method. Scored by a weighted-rubric LLM judge (correctness-critical criteria first); a task passes at >=0.7. Mirrors the training-side Rubrics-as-Rewards signal so the eval measures exactly what RLVR+RaR optimizes. Fixed, version-controlled tasks (problem-solving.ts) for reproducibility.

Referință →

Nicio rulare publicată încă pentru acest benchmark.

Descarcă datele

Aceleași rânduri pe care le afișează pagina, actualizate la fiecare 10 minute. Reutilizare liberă, cu atribuire.

Licență: CC BY 4.0, atribuire „codai (codai.ro/benchmarks)”.

Toate benchmark-urile Clasamentul onestității