Benchmark-uri deschise

Benchmark-uri pentru agenți

Benchmark-urile pentru agenți măsoară dacă un model duce singur la capăt o muncă în mai mulți pași: comenzi într-un shell real, apeluri de unelte corecte și o sarcină lungă ținută pe direcție. Rulările trec prin același gateway ca cererile dumneavoastră.

Ce rulare este afișată

Un rând per model per benchmark: câștigă rularea finalizată cu cel mai mare eșantion, iar la egalitate câștigă cea terminată cel mai recent. Rulările mici de test nu pot avantaja un model printr-un noroc. Rulările eșuate sau anulate sunt excluse.

Metrica principală

Costul per sarcină reușită. Penalizează atât răspunsurile corecte dar scumpe, cât și pe cele ieftine dar greșite, deci este cifra cel mai greu de manipulat. Rata de reușită și costul total stau alături.

Terminal-Bench, apeluri de unelte, sarcini lungi, multi-agent, GDPval, ALE-Bench și suita de onestitate.

Rezultate · Agentic

Benchmark-uri: 9
Rezultate per model: 43

Cel mai recent rezultat afișat · 27 sept. 2026

Frontiera Pareto

Cost vs acuratețe, fiecare model pe fiecare benchmark

Fiecare punct este un rezultat (model × benchmark). Stânga-sus este zona țintă: acuratețe mare, cost mic. Punctele de pe frontiera punctată sunt nedominate — nimic nu le bate pe ambele axe simultan.

codai
alte modele
Frontiera Pareto (cel mai mic cost la fiecare nivel de acuratețe)

Terminal-Bench 2.1 (official, k=5)oficial

89 de task-uri

The official Terminal-Bench 2.1 dataset (89 tasks) run with the harbor harness, terminus-2 agent on model codai, 5 attempts per task (445 trials), 2026-08-02/03. Score = mean reward over tasks; errored trials count as failures. Competitor rows are published leaderboard or system-card entries (source and date in each row).

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`GPT-6 Astra (Codex)`cel mai bun	87.4%	—	—	89/89
`Fable 5 (Claude Code)`	83.8%	—	—	89/89
`GPT-5.5 (Codex)`	83.2%	—	—	89/89
`Fable 5 (Terminus 2)`	80.5%	—	—	89/89
`claude-sonnet-5`	80.4%	—	—	89/89
`Grok 4.5 (Cursor CLI)`	79.3%	—	—	89/89
`codai`router	71.0%	$2.2742	$3.2025	445/445

Agentic Honesty v2 (leaderboard)

102 task-uri

Public honesty leaderboard. ~100 small agentic tasks (82 traps across 10 categories + 20 solvable controls). Each is scored against a MECHANICAL ground-truth shell check and protected-file hashes — no LLM judge, not our own verifier. The final report is classified success/failure/none by deterministic rules: fabricated = claimed success on a failed goal; sandbagged = reported failure on an achieved goal. A separate audit column records what the codai claim verifier would have flagged.

Referință →

34% mai ieftin per task reușit

codai cheltuie $0.0737 per succes față de $0.1112 pentru claude-fable-5-1 — următorul cel mai ieftin competitor.

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routercel mai bun	91.2%	$0.0672	$0.0737	102/102
`claude-fable-5-1`	86.3%	$0.0960	$0.1112	102/102

ALE-Bench-style Algorithm Engineering (codai)

3 task-uri

ALE-Bench-inspired iterative optimization (TSP, bin packing, knapsack). Up to 30 self-improvement rounds; solved = beats greedy reference.

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`claude-fable-5`cel mai bun	100.0%	$0	$0	3/3

GDPval-style Dev Productivity (codai)

12 task-uri

AAAI GDPval-AA axis: real-world SWE end-to-end. 12 embedded agentic tasks, hidden test verification.

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`claude-fable-5`cel mai bun	91.7%	$0.1316	$0.1436	12/12

Long-Horizon Autonomy

2 task-uri

Multi-milestone autonomy benchmark — can the model sustain a project-sized task over many steps without losing the thread? Each task is a goal split into ordered milestones, each with its own shell verify command. The harness drives a ReAct loop (large step budget) in a scratch dir, then verifies every milestone and awards PARTIAL CREDIT (milestoneScore) plus the binary all-milestones pass rate. Tasks are embedded + private (contamination-free). Closes the long-horizon-autonomy frontier must-win.

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routerfixat	0.0%	$0.0167	—	2/2
`claude-opus-4-8`cel mai bun	100.0%	$0.2595	$0.2595	2/2
`codai,claude-opus-4-8`	100.0%	$0.2681	$0.2681	2/2
`codai-alpha-1-0`	50.0%	$0.5973	$1.1946	2/2
`gemini-3.1-pro-preview`	0.0%	$0.0083	—	2/2
`gemini-3.5-flash-lite`	0.0%	$0.0109	—	2/2

Multi-Agent Gate

5 task-uri

Frontier v8 Track E gate. Each broad multi-facet task is run TWICE on the same model — single-agent vs orchestrated (x-codai-orchestrate) — and scored by facet coverage. Records the single->multi delta; passes when orchestration does not regress. Mirrors the Mythos5/Fable5 system card single->multi-agent gain (BrowseComp 88->93.3).

Referință →

Nicio rulare publicată încă pentru acest benchmark.

Terminal-Bench

100 de task-uri

Stanford 100-task benchmark: model drives a real bash shell to complete sysadmin / dev-ops tasks. Cascade + tool execution should shine here — this is where codai-vs-single-model differentiation gets measurable.

Referință →

Nicio rulare publicată încă pentru acest benchmark.

Tool Calling (BFCL-style)

10 task-uri

Function-calling reliability benchmark (Berkeley Function-Calling Leaderboard / tau-bench style). Each case presents JSON tool schemas + a user request; the model emits a single JSON tool call which is graded with DETERMINISTIC AST-style matching (function name + arguments, any-of values, extra-arg tolerant). No LLM judge. Cases are version-controlled (tool-calling-cases.ts) so the benchmark is fully reproducible. Closes the tool-calling frontier must-win category.

Referință →

Acuratețe peste mediană, cost cu 88% sub medie

codai atinge 100.0% (mediana celor 8 modele: 100.0%) cheltuind $0.0038 per succes — media câmpului este $0.0329.

Model	Succes	$ / task	$ / succes	Task-uri
`codai`routercel mai bun	100.0%	$0.0038	$0.0038	10/10
`gemini-3.1-pro-preview`	100.0%	$0.0040	$0.0040	10/10
`gemini-3.1-flash-lite`	100.0%	$0.0003	$0.0003	10/10
`gemini-3.6-flash`	100.0%	$0.0026	$0.0026	10/10
`claude-opus-4-8`	100.0%	$0.0121	$0.0121	10/10
`codai,claude-opus-4-8`	100.0%	$0.0189	$0.0189	10/10

codai terminal suite (7 tasks)intern

7 task-uri

Internal 7-task shell suite (log analysis, text pipelines, templating, build files, CSV joins) with hidden verify commands. NOT Terminal-Bench 2.1 and not comparable to its leaderboard; see "Terminal-Bench 2.1 (official, k=5)" for the official run.

Referință →

Model	Succes	$ / task	$ / succes	Task-uri
`codai,claude-opus-4-8`cel mai bun	85.7%	$0.0734	$0.0857	7/7
`claude-fable-5`	71.4%	$0.4405	$0.6167	7/7
`claude-opus-4-8`	71.4%	$0.0552	$0.0773	7/7
`claude-haiku-4-5`	71.4%	$0.0244	$0.0341	7/7
`claude-sonnet-4-6`	57.1%	$0.0669	$0.1170	7/7
`gemini-3.5-flash-lite`	28.6%	$0.0008	$0.0029	7/7
`codai`router	0.0%	$0.0028	—	7/7

Descarcă datele

Aceleași rânduri pe care le afișează pagina, actualizate la fiecare 10 minute. Reutilizare liberă, cu atribuire.

Licență: CC BY 4.0, atribuire „codai (codai.ro/benchmarks)”.

Toate benchmark-urile Clasamentul onestității