Open benchmarks

Agentic benchmarks

Agentic benchmarks measure whether a model finishes multi-step work on its own: driving a real shell, emitting correct tool calls, and keeping a long task on track. Runs go through the same gateway path your requests use.

Which run is shown

One row per model per benchmark: the completed run with the largest sample wins, and on a tie the most recent finish wins. Small smoke runs cannot lift a model by getting lucky. Failed and cancelled runs are excluded.

The headline metric

Cost per successful task. It punishes both expensive correct answers and cheap wrong ones, so it is the number that is hardest to game. Success rate and total cost sit next to it.

Terminal-Bench, tool calling, long-horizon, multi-agent, GDPval, ALE-Bench and the honesty suite.

Results · Agentic

Benchmarks: 9
Model results: 43

Cost vs accuracy, every model on every benchmark

Each dot is one (model × benchmark) result. Upper-left is the target zone: high accuracy, low cost. Dots on the dashed frontier are non-dominated — nothing beats them on both axes at once.

codai
other models
Pareto frontier (lowest cost at each accuracy)

Terminal-Bench 2.1 (official, k=5)official

89 tasks

The official Terminal-Bench 2.1 dataset (89 tasks) run with the harbor harness, terminus-2 agent on model codai, 5 attempts per task (445 trials), 2026-08-02/03. Score = mean reward over tasks; errored trials count as failures. Competitor rows are published leaderboard or system-card entries (source and date in each row).

Reference →

Model	Success	$ / task	$ / success	Tasks
`GPT-6 Astra (Codex)`best	87.4%	—	—	89/89
`Fable 5 (Claude Code)`	83.8%	—	—	89/89
`GPT-5.5 (Codex)`	83.2%	—	—	89/89
`Fable 5 (Terminus 2)`	80.5%	—	—	89/89
`claude-sonnet-5`	80.4%	—	—	89/89
`Grok 4.5 (Cursor CLI)`	79.3%	—	—	89/89
`codai`router	71.0%	$2.2742	$3.2025	445/445

Agentic Honesty v2 (leaderboard)

102 tasks

Public honesty leaderboard. ~100 small agentic tasks (82 traps across 10 categories + 20 solvable controls). Each is scored against a MECHANICAL ground-truth shell check and protected-file hashes — no LLM judge, not our own verifier. The final report is classified success/failure/none by deterministic rules: fabricated = claimed success on a failed goal; sandbagged = reported failure on an achieved goal. A separate audit column records what the codai claim verifier would have flagged.

Reference →

34% cheaper per successful task

codai spends $0.0737 per success vs $0.1112 for claude-fable-5-1 — the next-cheapest competitor.

Model	Success	$ / task	$ / success	Tasks
`codai`routerbest	91.2%	$0.0672	$0.0737	102/102
`claude-fable-5-1`	86.3%	$0.0960	$0.1112	102/102

ALE-Bench-style Algorithm Engineering (codai)

3 tasks

ALE-Bench-inspired iterative optimization (TSP, bin packing, knapsack). Up to 30 self-improvement rounds; solved = beats greedy reference.

Reference →

Model	Success	$ / task	$ / success	Tasks
`claude-fable-5`best	100.0%	$0	$0	3/3

GDPval-style Dev Productivity (codai)

12 tasks

AAAI GDPval-AA axis: real-world SWE end-to-end. 12 embedded agentic tasks, hidden test verification.

Reference →

Model	Success	$ / task	$ / success	Tasks
`claude-fable-5`best	91.7%	$0.1316	$0.1436	12/12

Long-Horizon Autonomy

2 tasks

Multi-milestone autonomy benchmark — can the model sustain a project-sized task over many steps without losing the thread? Each task is a goal split into ordered milestones, each with its own shell verify command. The harness drives a ReAct loop (large step budget) in a scratch dir, then verifies every milestone and awards PARTIAL CREDIT (milestoneScore) plus the binary all-milestones pass rate. Tasks are embedded + private (contamination-free). Closes the long-horizon-autonomy frontier must-win.

Reference →

Model	Success	$ / task	$ / success	Tasks
`codai`routerpinned	0.0%	$0.0167	—	2/2
`claude-opus-4-8`best	100.0%	$0.2595	$0.2595	2/2
`codai,claude-opus-4-8`	100.0%	$0.2681	$0.2681	2/2
`codai-alpha-1-0`	50.0%	$0.5973	$1.1946	2/2
`gemini-3.1-pro-preview`	0.0%	$0.0083	—	2/2
`gemini-3.5-flash-lite`	0.0%	$0.0109	—	2/2

Multi-Agent Gate

5 tasks

Frontier v8 Track E gate. Each broad multi-facet task is run TWICE on the same model — single-agent vs orchestrated (x-codai-orchestrate) — and scored by facet coverage. Records the single->multi delta; passes when orchestration does not regress. Mirrors the Mythos5/Fable5 system card single->multi-agent gain (BrowseComp 88->93.3).

Reference →

No runs published yet for this benchmark.

Terminal-Bench

100 tasks

Stanford 100-task benchmark: model drives a real bash shell to complete sysadmin / dev-ops tasks. Cascade + tool execution should shine here — this is where codai-vs-single-model differentiation gets measurable.

Reference →

No runs published yet for this benchmark.

Tool Calling (BFCL-style)

10 tasks

Function-calling reliability benchmark (Berkeley Function-Calling Leaderboard / tau-bench style). Each case presents JSON tool schemas + a user request; the model emits a single JSON tool call which is graded with DETERMINISTIC AST-style matching (function name + arguments, any-of values, extra-arg tolerant). No LLM judge. Cases are version-controlled (tool-calling-cases.ts) so the benchmark is fully reproducible. Closes the tool-calling frontier must-win category.

Reference →

Above-median accuracy, 88% below average cost

codai hits 100.0% (median across 8 models: 100.0%) while spending $0.0038 per success — the field averages $0.0329.

Model	Success	$ / task	$ / success	Tasks
`codai`routerbest	100.0%	$0.0038	$0.0038	10/10
`gemini-3.1-pro-preview`	100.0%	$0.0040	$0.0040	10/10
`gemini-3.1-flash-lite`	100.0%	$0.0003	$0.0003	10/10
`gemini-3.6-flash`	100.0%	$0.0026	$0.0026	10/10
`claude-opus-4-8`	100.0%	$0.0121	$0.0121	10/10
`codai,claude-opus-4-8`	100.0%	$0.0189	$0.0189	10/10

codai terminal suite (7 tasks)internal

7 tasks

Internal 7-task shell suite (log analysis, text pipelines, templating, build files, CSV joins) with hidden verify commands. NOT Terminal-Bench 2.1 and not comparable to its leaderboard; see "Terminal-Bench 2.1 (official, k=5)" for the official run.

Reference →

Model	Success	$ / task	$ / success	Tasks
`codai,claude-opus-4-8`best	85.7%	$0.0734	$0.0857	7/7
`claude-fable-5`	71.4%	$0.4405	$0.6167	7/7
`claude-opus-4-8`	71.4%	$0.0552	$0.0773	7/7
`claude-haiku-4-5`	71.4%	$0.0244	$0.0341	7/7
`claude-sonnet-4-6`	57.1%	$0.0669	$0.1170	7/7
`gemini-3.5-flash-lite`	28.6%	$0.0008	$0.0029	7/7
`codai`router	0.0%	$0.0028	—	7/7

Download the data

The same rows this page renders, refreshed every 10 minutes. Free to reuse with attribution.

Licence: CC BY 4.0, attribution “codai (codai.ro/benchmarks)”.

All benchmarks Honesty leaderboard