Benchmark-uri deschise

Benchmark-uri de programare

Benchmark-urile de programare măsoară dacă un model scrie cod care funcționează: o funcție care trece testele unitare sau un patch care repară o problemă reală dintr-un depozit. Fiecare bloc este etichetat oficial, proxy, intern sau referință, ca un substitut ieftin să nu fie citit drept scor de clasament.

Ce rulare este afișată

Un rând per model per benchmark: câștigă rularea finalizată cu cel mai mare eșantion, iar la egalitate câștigă cea terminată cel mai recent. Rulările mici de test nu pot avantaja un model printr-un noroc. Rulările eșuate sau anulate sunt excluse.

Metrica principală

Costul per sarcină reușită. Penalizează atât răspunsurile corecte dar scumpe, cât și pe cele ieftine dar greșite, deci este cifra cel mai greu de manipulat. Rata de reușită și costul total stau alături.

HumanEval, SWE-bench Verified (harness oficial și variante proxy) și Aider Polyglot.

Rezultate · Programare

Benchmark-uri: 5
Rezultate per model: 140

Cel mai recent rezultat afișat · 27 sept. 2026

Frontiera Pareto

Cost vs acuratețe, fiecare model pe fiecare benchmark

Fiecare punct este un rezultat (model × benchmark). Stânga-sus este zona țintă: acuratețe mare, cost mic. Punctele de pe frontiera punctată sunt nedominate — nimic nu le bate pe ambele axe simultan.

codai
alte modele
Frontiera Pareto (cel mai mic cost la fiecare nivel de acuratețe)

SWE-bench Verified (official harness)oficial

500 de task-uri

The 500-task SWE-bench Verified set scored by the official SWE-bench Docker harness (resolved = FAIL_TO_PASS and PASS_TO_PASS tests pass). codai agent, single attempt per task, run 2026-08-03/04. Competitor rows are published leaderboard entries (source and date in each row), not re-runs by us.

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	73.2%	$0.6095	$0.8326	500/500
`claude-sonnet-5`cel mai bun	85.2%	—	—	500/500
`20251205_sonar-foundation-agent_claude-opus-4-5`	79.2%	—	—	500/500
`20251215_livesweagent_claude-opus-4-5`	79.2%	—	—	500/500
`20250928_trae_doubao_seed_code`	78.8%	—	—	500/500
`OpenHands + Claude Opus 4.5`	77.6%	—	—	500/500

Aider Polyglot

225 de task-uri

225-task multi-language coding benchmark (Python, JS, TS, Rust, Go, Java, C++). Anthropic and OpenAI both quote this for editing performance. Our harness runs the PYTHON subset (~38 tasks) — the largest single bucket — with the official two-pass methodology (one initial try + one retry given test output). Cross-language coverage is the next iteration; competitor scores below are full 225-task polyglot.

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	61.8%	$0.4031	$0.6527	34/225
`gpt-5 (high)`cel mai bun	88.0%	$0.1293	$0.1469	225/225
`gpt-5 (medium)`	86.7%	$0.0786	$0.0907	225/225
`claude-opus-4-7`	85.3%	$0.0273	$0.0320	34/225
`o3-pro (high)`	84.9%	$0.6503	$0.7661	225/225
`gemini-2.5-pro-preview-06-05 (32k think)`	83.1%	$0.2217	$0.2667	225/225

HumanEval

164 de task-uri

OpenAI 164-problem Python coding benchmark. Single-turn: prompt → function body → run unit tests. Cheap, fast, well-known — used as the pipeline smoke benchmark.

Referință →

Acuratețe peste mediană, cost cu 78% sub medie

codai atinge 96.3% (mediana celor 11 modele: 88.1%) cheltuind $0.0092 per succes — media câmpului este $0.0422.

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	96.3%	$0.0089	$0.0092	164/164
`codai,claude-opus-4-8`cel mai bun	97.0%	$0.0107	$0.0111	164/164
`claude-opus-4-8`	96.3%	$0.0122	$0.0127	164/164
`claude-haiku-4-5`	95.1%	$0.0111	$0.0116	164/164
`claude-sonnet-4-7`	94.5%	$0.0013	$0.0013	164/164
`claude-opus-4-7`	93.9%	$0.0013	$0.0013	164/164

SWE-bench Verified (oracle-diff proxy, 100)proxy

100 de task-uri

PROXY, not the official harness: 100 SWE-bench Verified issues, the model gets the issue plus the oracle affected files and outputs a unified diff, scored by line-overlap similarity to the gold patch plus an LLM judge on borderline cases. No tests are executed. Not comparable to leaderboard numbers; see "SWE-bench Verified (official harness)" for the real score.

Referință →

Acuratețe peste mediană, cost cu 46% sub medie

codai atinge 46.0% (mediana celor 6 modele: 1.0%) cheltuind $0.7387 per succes — media câmpului este $1.3571.

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	46.0%	$0.3398	$0.7387	100/100
`claude-sonnet-5`cel mai bun	85.2%	—	—	500/500
`claude-fable-5`	65.0%	$0.3871	$0.5956	100/100
`gemini-3.5-flash`	1.0%	$0.0212	$2.1185	100/100
`gemini-3.1-flash-lite`	0.0%	$0.0028	—	100/100
`gemini-3.5-flash-lite`	0.0%	$0.0037	—	100/100

SWE-bench Verified (patch-only)proxy

100 de task-uri

Patch-only variant of SWE-bench Verified. Model receives the issue + the 'oracle' affected files (princeton-nlp/SWE-bench_oracle pre-formatted text) and must output a unified diff that fixes the issue. NO Docker, NO test execution — we score the candidate diff against the gold patch with line-overlap similarity (cheap fast-path) plus an LLM judge (sonnet-4-6) for borderline cases (0.3 < sim < 0.7). Not directly comparable to the official leaderboard but a strong proxy for diff-generation quality at 1% of the cost.

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routercel mai bun	47.0%	$1.2653	$2.6922	100/100
`claude-opus-4-7`	46.0%	$1.2709	$2.7627	100/100
`claude-opus-4-8`	36.0%	$0.3656	$1.0156	100/100
`claude-haiku-4-5`	35.0%	$0.0938	$0.2680	100/100
`gpt-5-mini`	31.0%	$1.2862	$4.1490	100/100
`claude-sonnet-4-6`	29.0%	$0.2451	$0.8451	100/100

Descarcă datele

Aceleași rânduri pe care le afișează pagina, actualizate la fiecare 10 minute. Reutilizare liberă, cu atribuire.

Licență: CC BY 4.0, atribuire „codai (codai.ro/benchmarks)”.

Toate benchmark-urile Clasamentul onestității