Every benchmark. Every model. Live.

225-task multi-language coding benchmark (Python, JS, TS, Rust, Go, Java, C++). Anthropic and OpenAI both quote this for editing performance. Our harness runs the PYTHON subset (~38 tasks) — the largest single bucket — with the official two-pass methodology (one initial try + one retry given test output). Cross-language coverage is the next iteration; competitor scores below are full 225-task polyglot.

Reference →

OpenAI 164-problem Python coding benchmark. Single-turn: prompt → function body → run unit tests. Cheap, fast, well-known — used as the pipeline smoke benchmark.

Reference →

Princeton 500-problem real-world software-engineering benchmark — the most-cited benchmark in 2025/2026 vendor announcements (Mythos 5 = 95.5, §8). Executed here via the cost-effective ORACLE proxy: the model receives the issue + oracle affected files and must output a unified diff, scored by line-overlap similarity + an LLM judge for borderline cases (shared executor with swe-bench-patch). Not the official Docker harness; a strong, cheap diff-quality proxy for the headline number.

Reference →

Patch-only variant of SWE-bench Verified. Model receives the issue + the 'oracle' affected files (princeton-nlp/SWE-bench_oracle pre-formatted text) and must output a unified diff that fixes the issue. NO Docker, NO test execution — we score the candidate diff against the gold patch with line-overlap similarity (cheap fast-path) plus an LLM judge (sonnet-4-6) for borderline cases (0.3 < sim < 0.7). Not directly comparable to the official leaderboard but a strong proxy for diff-generation quality at 1% of the cost.

Reference →

ALE-Bench-inspired iterative optimization (TSP, bin packing, knapsack). Up to 30 self-improvement rounds; solved = beats greedy reference.

Reference →

AAAI GDPval-AA axis: real-world SWE end-to-end. 12 embedded agentic tasks, hidden test verification.

Reference →

Multi-milestone autonomy benchmark — can the model sustain a project-sized task over many steps without losing the thread? Each task is a goal split into ordered milestones, each with its own shell verify command. The harness drives a ReAct loop (large step budget) in a scratch dir, then verifies every milestone and awards PARTIAL CREDIT (milestoneScore) plus the binary all-milestones pass rate. Tasks are embedded + private (contamination-free). Closes the long-horizon-autonomy frontier must-win.

Reference →

Frontier v8 Track E gate. Each broad multi-facet task is run TWICE on the same model — single-agent vs orchestrated (x-codai-orchestrate) — and scored by facet coverage. Records the single->multi delta; passes when orchestration does not regress. Mirrors the Mythos5/Fable5 system card single->multi-agent gain (BrowseComp 88->93.3).

Reference →

Stanford 100-task benchmark: model drives a real bash shell to complete sysadmin / dev-ops tasks. Cascade + tool execution should shine here — this is where codai-vs-single-model differentiation gets measurable.

Reference →

Frontier v8 Track D harder terminal-agent suite. Longer-horizon sysadmin / dev-ops shell tasks (log analysis, text pipelines, templating, build files, CSV joins) driven via a real shell ReAct loop with hidden verify commands. Matches the difficulty step of Terminal-Bench 2.1 in the Mythos5/Fable5 system card (§8, Mythos 5 = 88.0).

Reference →

Function-calling reliability benchmark (Berkeley Function-Calling Leaderboard / tau-bench style). Each case presents JSON tool schemas + a user request; the model emits a single JSON tool call which is graded with DETERMINISTIC AST-style matching (function name + arguments, any-of values, extra-arg tolerant). No LLM judge. Cases are version-controlled (tool-calling-cases.ts) so the benchmark is fully reproducible. Closes the tool-calling frontier must-win category.

Reference →

Hard mathematical-reasoning benchmark built from AIME 2024 + AIME 2025 (American Invitational Mathematics Examination). Every answer is an integer in 0..999, so scoring is fully DETERMINISTIC and un-gameable — we extract the model's final integer (`Final answer: N` sentinel, then \boxed{N}, then last integer) and compare for exact equality. No LLM judge, so no judge cost or judge variance poisoning the reasoning priors. Closes a frontier category (hard reasoning/math) the scorecard was previously blind to.

Reference →

Frontier v8 long-context multi-hop reasoning gate. Each task embeds a large deterministically-generated directed graph in the prompt and asks a BFS-reachability or direct-parents query; scored by exact node-set match (no judge). Modelled on the GraphWalks eval in the Anthropic Mythos 5 / Fable 5 system card (§8, BFS 256K). Un-gameable: graphs are generated per-seed, never downloaded. Cost scales with nodeCount (context length).

Reference →

Long-context retrieval benchmark (RULER-style needle-in-a-haystack). Fully SYNTHETIC + DETERMINISTIC: a filler haystack is generated at a target token budget (2k -> 128k), magic-number needles are inserted at spread depths, and the model must retrieve them. Scored by exact value match (all needles required). No external dataset, no LLM judge. Sweeps context lengths so we can see where each model’s effective window degrades. Closes the long-context frontier category.

Reference →

TIGER-Lab's 12,032-question reasoning benchmark across 14 domains (biology, business, chemistry, CS, economics, engineering, health, history, law, math, philosophy, physics, psychology, other). 10-choice multiple-choice, deterministic letter-match scoring. We run a 200-question stratified sample per category (28 per category, capped at the available count).

Reference →

Frontier v9 #1-priority category. Single-shot HARD reasoning tasks that other harnesses miss: multi-step logic puzzles (balance-weighing, river/bridge crossings), quantitative word problems with intuitive-but-wrong traps, systematic root-cause/debugging strategy, constraint satisfaction, and Fermi estimation. NOT code-execution (humaneval/codai-swe own that) and NOT short-answer contest math (math-competition owns that) — rewards correct multi-step reasoning AND a sound, communicable method. Scored by a weighted-rubric LLM judge (correctness-critical criteria first); a task passes at >=0.7. Mirrors the training-side Rubrics-as-Rewards signal so the eval measures exactly what RLVR+RaR optimizes. Fixed, version-controlled tasks (problem-solving.ts) for reproducibility.

Reference →

Romanian high-school mathematics benchmark from RoMathExam (Cuclea et al. 2026, 10,592 problems from official Bacalaureat exams 1895-2025, MIT license). Stratified sample across M1/M2/M3/M4 tracks. Scored by an LLM judge (sonnet-4-6) comparing the candidate's final answer to the official `barem` — math can't be exact-matched. Codai-unique angle: no major LLM router publishes Romanian-specific scores.

Reference →