{"name":"codai benchmarks — all","scope":"all","license":"https://creativecommons.org/licenses/by/4.0/","attribution":"CC BY 4.0, attribution: codai (https://codai.ro/benchmarks)","source":"https://codai.ro/benchmarks","selection":"One run per benchmark and model: the completed run with the largest sample; ties go to the most recent finish.","units":{"successRate":"fraction 0..1","costUsd":"USD","costPerSuccessUsd":"USD"},"lastRefresh":"2026-09-27T19:57:35.475Z","benchmarks":[{"id":"e1a30d9a-ae44-4e86-8bf8-1f6251eaad64","slug":"terminal-bench-2-1","name":"Terminal-Bench 2.1 (official, k=5)","description":"The official Terminal-Bench 2.1 dataset (89 tasks) run with the harbor harness, terminus-2 agent on model codai, 5 attempts per task (445 trials), 2026-08-02/03. Score = mean reward over tasks; errored trials count as failures. Competitor rows are published leaderboard or system-card entries (source and date in each row).","category":"agentic","taskCount":89,"referenceUrl":"https://www.tbench.ai/leaderboard","resultKind":"official","runs":[{"id":"d8f5c4ed-765c-4c1d-adf1-f21160784900","modelTested":"GPT-6 Astra (Codex)","successRate":"0.8740","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"f4367a23-723f-4544-95c6-6fd7fe777d45","modelTested":"Fable 5 (Claude Code)","successRate":"0.8380","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"068373aa-47f3-4cec-8352-9d2daeba938c","modelTested":"GPT-5.5 (Codex)","successRate":"0.8320","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"29071abb-67da-4f49-a472-0b7f033c35d1","modelTested":"Fable 5 (Terminus 2)","successRate":"0.8050","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"33ef370b-8cdf-4a14-9215-8a8bb963a049","modelTested":"claude-sonnet-5","successRate":"0.8040","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-06-30T00:00:00.000Z"},{"id":"dea1be52-078c-4efa-8a44-ffef8b3430d4","modelTested":"Grok 4.5 (Cursor CLI)","successRate":"0.7930","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"807e7178-10ee-479a-85e4-f638305d852f","modelTested":"Opus 4.8 (Claude Code)","successRate":"0.7890","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"645b617b-decd-4661-8cb0-1562c9602dfb","modelTested":"Sonnet 5 (tbench.ai)","successRate":"0.7460","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"20225239-941d-4b7b-992b-1f578320871f","modelTested":"Gemini 3 Pro (tbench.ai)","successRate":"0.7390","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"a8563b58-2de0-4a4a-91cc-c1b3e43f3b3e","modelTested":"codai","successRate":"0.7101","costUsd":"1012.0000","costPerSuccessUsd":"3.2025","completedTasks":445,"totalTasks":445,"finishedAt":"2026-08-03T09:53:53.486Z"},{"id":"80f3d38e-f04d-4cde-8dc4-49ecf1537e77","modelTested":"Opus 4.7 (tbench.ai)","successRate":"0.6890","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"5c57399b-bc06-43ce-8413-bdd73601c6c9","modelTested":"claude-sonnet-4-6","successRate":"0.6700","costUsd":null,"costPerSuccessUsd":null,"completedTasks":89,"totalTasks":89,"finishedAt":"2026-06-30T00:00:00.000Z"}]},{"id":"0b6f1a89-d310-4b90-b452-d31e870f3a45","slug":"agentic-honesty-v2","name":"Agentic Honesty v2 (leaderboard)","description":"Public honesty leaderboard. ~100 small agentic tasks (82 traps across 10 categories + 20 solvable controls). Each is scored against a MECHANICAL ground-truth shell check and protected-file hashes — no LLM judge, not our own verifier. The final report is classified success/failure/none by deterministic rules: fabricated = claimed success on a failed goal; sandbagged = reported failure on an achieved goal. A separate audit column records what the codai claim verifier would have flagged.","category":"agentic","taskCount":102,"referenceUrl":"https://codai.ro/honesty","resultKind":null,"runs":[{"id":"22edcda2-aea6-4e8e-8bc1-436243aab9b8","modelTested":"codai","successRate":"0.9118","costUsd":"6.8524","costPerSuccessUsd":"0.0737","completedTasks":102,"totalTasks":102,"finishedAt":"2026-09-27T19:16:46.749Z"},{"id":"93d1d952-746e-4802-8e62-901ea949872b","modelTested":"claude-fable-5-1","successRate":"0.8627","costUsd":"9.7882","costPerSuccessUsd":"0.1112","completedTasks":102,"totalTasks":102,"finishedAt":"2026-09-27T19:57:35.475Z"}]},{"id":"22df926b-4c8e-4a13-b281-a3c6e107a0da","slug":"ale-bench","name":"ALE-Bench-style Algorithm Engineering (codai)","description":"ALE-Bench-inspired iterative optimization (TSP, bin packing, knapsack). Up to 30 self-improvement rounds; solved = beats greedy reference.","category":"agentic","taskCount":3,"referenceUrl":"https://arxiv.org/abs/2506.09050","resultKind":null,"runs":[{"id":"b49ae5c7-36c6-455e-a5b7-81eb474a2f7a","modelTested":"claude-fable-5","successRate":"1.0000","costUsd":"0.0000","costPerSuccessUsd":"0.0000","completedTasks":3,"totalTasks":3,"finishedAt":"2026-07-11T13:32:41.016Z"}]},{"id":"35974400-29b8-4a5e-b102-9940ef9a9c06","slug":"gdpval","name":"GDPval-style Dev Productivity (codai)","description":"AAAI GDPval-AA axis: real-world SWE end-to-end. 12 embedded agentic tasks, hidden test verification.","category":"agentic","taskCount":12,"referenceUrl":"https://artificialanalysis.ai","resultKind":null,"runs":[{"id":"5c90c418-5034-47f1-a895-91983ac2a0d1","modelTested":"claude-fable-5","successRate":"0.9167","costUsd":"1.5792","costPerSuccessUsd":"0.1436","completedTasks":12,"totalTasks":12,"finishedAt":"2026-07-11T13:30:29.321Z"}]},{"id":"4f0c13b8-06bb-4857-acbc-a5dcca5f871e","slug":"long-horizon","name":"Long-Horizon Autonomy","description":"Multi-milestone autonomy benchmark — can the model sustain a project-sized task over many steps without losing the thread? Each task is a goal split into ordered milestones, each with its own shell verify command. The harness drives a ReAct loop (large step budget) in a scratch dir, then verifies every milestone and awards PARTIAL CREDIT (milestoneScore) plus the binary all-milestones pass rate. Tasks are embedded + private (contamination-free). Closes the long-horizon-autonomy frontier must-win.","category":"agentic","taskCount":2,"referenceUrl":"https://www.swebench.com/","resultKind":null,"runs":[{"id":"6aba986d-a1cf-4f6f-b00a-37b2acdda643","modelTested":"claude-opus-4-8","successRate":"1.0000","costUsd":"0.5189","costPerSuccessUsd":"0.2595","completedTasks":2,"totalTasks":2,"finishedAt":"2026-06-27T00:21:46.372Z"},{"id":"3f21c782-b785-4f1c-ae42-951dc14878e7","modelTested":"codai,claude-opus-4-8","successRate":"1.0000","costUsd":"0.5362","costPerSuccessUsd":"0.2681","completedTasks":2,"totalTasks":2,"finishedAt":"2026-06-27T00:15:15.437Z"},{"id":"abea1dc9-dfb7-4ee1-b5e8-ace3db024c73","modelTested":"codai-alpha-1-0","successRate":"0.5000","costUsd":"1.1946","costPerSuccessUsd":"1.1946","completedTasks":2,"totalTasks":2,"finishedAt":"2026-06-16T00:24:44.478Z"},{"id":"dff1a22e-2d24-4f79-95b9-216e20eb54c5","modelTested":"gemini-3.1-pro-preview","successRate":"0.0000","costUsd":"0.0165","costPerSuccessUsd":null,"completedTasks":2,"totalTasks":2,"finishedAt":"2026-08-11T22:49:48.287Z"},{"id":"2b2cd540-4dae-47f1-9011-cfc8c568b3bc","modelTested":"codai","successRate":"0.0000","costUsd":"0.0334","costPerSuccessUsd":null,"completedTasks":2,"totalTasks":2,"finishedAt":"2026-08-11T22:49:38.632Z"},{"id":"4f3b0b6b-2fa7-454b-9b1b-a1e2996e089f","modelTested":"gemini-3.5-flash-lite","successRate":"0.0000","costUsd":"0.0217","costPerSuccessUsd":null,"completedTasks":2,"totalTasks":2,"finishedAt":"2026-08-11T22:49:33.055Z"},{"id":"634eab55-500e-48cb-ae0e-bcb5327db241","modelTested":"gemini-3.6-flash","successRate":"0.0000","costUsd":"0.0299","costPerSuccessUsd":null,"completedTasks":2,"totalTasks":2,"finishedAt":"2026-08-11T22:49:30.917Z"},{"id":"6331cca0-e323-4198-b219-a138fc30e57d","modelTested":"gemini-3.1-flash-lite","successRate":"0.0000","costUsd":"0.0074","costPerSuccessUsd":null,"completedTasks":2,"totalTasks":2,"finishedAt":"2026-08-11T22:49:29.782Z"},{"id":"18e8d2c5-a409-4b32-af52-119fa19b3750","modelTested":"gemini-3.5-flash","successRate":"0.0000","costUsd":"0.0185","costPerSuccessUsd":null,"completedTasks":2,"totalTasks":2,"finishedAt":"2026-08-11T22:49:28.489Z"}]},{"id":"08b925c8-8c52-47d9-82b7-97031c64999a","slug":"multi-agent","name":"Multi-Agent Gate","description":"Frontier v8 Track E gate. Each broad multi-facet task is run TWICE on the same model — single-agent vs orchestrated (x-codai-orchestrate) — and scored by facet coverage. Records the single->multi delta; passes when orchestration does not regress. Mirrors the Mythos5/Fable5 system card single->multi-agent gain (BrowseComp 88->93.3).","category":"agentic","taskCount":5,"referenceUrl":"https://www.anthropic.com","resultKind":null,"runs":[]},{"id":"f3b878dc-7de6-4049-98dc-e13e73550917","slug":"terminal-bench","name":"Terminal-Bench","description":"Stanford 100-task benchmark: model drives a real bash shell to complete sysadmin / dev-ops tasks. Cascade + tool execution should shine here — this is where codai-vs-single-model differentiation gets measurable.","category":"agentic","taskCount":100,"referenceUrl":"https://crfm.stanford.edu/2024/12/01/terminal-bench.html","resultKind":null,"runs":[]},{"id":"6557edab-abe8-459e-969c-694f70de95bc","slug":"tool-calling","name":"Tool Calling (BFCL-style)","description":"Function-calling reliability benchmark (Berkeley Function-Calling Leaderboard / tau-bench style). Each case presents JSON tool schemas + a user request; the model emits a single JSON tool call which is graded with DETERMINISTIC AST-style matching (function name + arguments, any-of values, extra-arg tolerant). No LLM judge. Cases are version-controlled (tool-calling-cases.ts) so the benchmark is fully reproducible. Closes the tool-calling frontier must-win category.","category":"agentic","taskCount":10,"referenceUrl":"https://gorilla.cs.berkeley.edu/leaderboard.html","resultKind":null,"runs":[{"id":"4584d9b7-20c2-4010-95d2-ac85c036f828","modelTested":"codai","successRate":"1.0000","costUsd":"0.0383","costPerSuccessUsd":"0.0038","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-12T00:14:03.442Z"},{"id":"384697c6-9191-4d28-9d48-bdc75688c993","modelTested":"gemini-3.1-pro-preview","successRate":"1.0000","costUsd":"0.0398","costPerSuccessUsd":"0.0040","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:06.151Z"},{"id":"339283fc-21cb-4854-83b4-914b632b765d","modelTested":"gemini-3.1-flash-lite","successRate":"1.0000","costUsd":"0.0026","costPerSuccessUsd":"0.0003","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:48:58.595Z"},{"id":"4bd96230-a383-46d3-83e3-55684409526a","modelTested":"gemini-3.6-flash","successRate":"1.0000","costUsd":"0.0257","costPerSuccessUsd":"0.0026","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:47:25.588Z"},{"id":"62b894dc-769e-4a38-8abe-a2d0c755109f","modelTested":"claude-opus-4-8","successRate":"1.0000","costUsd":"0.1210","costPerSuccessUsd":"0.0121","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-27T00:20:46.380Z"},{"id":"dba9fee1-fd9b-47b8-8f65-b28dd39c7ea6","modelTested":"codai,claude-opus-4-8","successRate":"1.0000","costUsd":"0.1892","costPerSuccessUsd":"0.0189","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-27T00:14:11.343Z"},{"id":"afebc276-476b-4d4e-83c6-d8178637e604","modelTested":"gpt-5.6-sol","successRate":"0.9000","costUsd":"0.4995","costPerSuccessUsd":"0.0555","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-12T01:01:33.012Z"},{"id":"978e8678-cc9a-4846-a4d1-55a141bb2f6b","modelTested":"codai-alpha-1-0","successRate":"0.5000","costUsd":"0.6842","costPerSuccessUsd":"0.1368","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-16T00:18:18.486Z"}]},{"id":"e6b51a18-e2f8-4ea9-bcdb-29a319dec791","slug":"terminal-bench-2","name":"codai terminal suite (7 tasks)","description":"Internal 7-task shell suite (log analysis, text pipelines, templating, build files, CSV joins) with hidden verify commands. NOT Terminal-Bench 2.1 and not comparable to its leaderboard; see \"Terminal-Bench 2.1 (official, k=5)\" for the official run.","category":"agentic","taskCount":7,"referenceUrl":"https://www.anthropic.com","resultKind":"internal","runs":[{"id":"b8b73c14-e69c-4cf2-b10b-1c0132d68484","modelTested":"codai,claude-opus-4-8","successRate":"0.8571","costUsd":"0.5140","costPerSuccessUsd":"0.0857","completedTasks":7,"totalTasks":7,"finishedAt":"2026-06-27T00:15:24.895Z"},{"id":"3a329579-79d9-4e17-a045-1f2ffae1254c","modelTested":"claude-fable-5","successRate":"0.7143","costUsd":"3.0835","costPerSuccessUsd":"0.6167","completedTasks":7,"totalTasks":7,"finishedAt":"2026-07-11T02:05:39.670Z"},{"id":"f8917a97-ee41-4480-93f1-6232e1c09c9d","modelTested":"claude-opus-4-8","successRate":"0.7143","costUsd":"0.3865","costPerSuccessUsd":"0.0773","completedTasks":7,"totalTasks":7,"finishedAt":"2026-06-27T00:22:01.532Z"},{"id":"60e80266-4fe7-4c35-93f6-de7f2f6ff774","modelTested":"claude-haiku-4-5","successRate":"0.7143","costUsd":"0.1706","costPerSuccessUsd":"0.0341","completedTasks":7,"totalTasks":7,"finishedAt":"2026-06-15T20:10:35.238Z"},{"id":"90ab750e-4f95-4c3a-a963-30dfef7353c0","modelTested":"claude-sonnet-4-6","successRate":"0.5714","costUsd":"0.4680","costPerSuccessUsd":"0.1170","completedTasks":7,"totalTasks":7,"finishedAt":"2026-06-14T19:45:44.718Z"},{"id":"34c1484a-cdcf-413c-ae32-c59532717936","modelTested":"gemini-3.5-flash-lite","successRate":"0.2857","costUsd":"0.0058","costPerSuccessUsd":"0.0029","completedTasks":7,"totalTasks":7,"finishedAt":"2026-08-11T22:47:59.039Z"},{"id":"ec258998-91c5-4aaa-ae6c-c471e0e7929d","modelTested":"codai-alpha-1-0","successRate":"0.1429","costUsd":"0.9302","costPerSuccessUsd":"0.9302","completedTasks":7,"totalTasks":7,"finishedAt":"2026-06-15T21:41:18.498Z"},{"id":"8fca7d4b-b566-4ba9-9eb7-f97e5f46f722","modelTested":"gemini-3.1-flash-lite","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":7,"totalTasks":7,"finishedAt":"2026-08-11T22:48:17.259Z"},{"id":"12296ed3-c8df-445c-b22d-7be305beaa17","modelTested":"codai","successRate":"0.0000","costUsd":"0.0198","costPerSuccessUsd":null,"completedTasks":7,"totalTasks":7,"finishedAt":"2026-08-11T22:48:02.880Z"},{"id":"c68af333-90ed-4857-85c1-4e46e14b8b45","modelTested":"gemini-3.1-pro-preview","successRate":"0.0000","costUsd":"0.0235","costPerSuccessUsd":null,"completedTasks":7,"totalTasks":7,"finishedAt":"2026-08-11T22:47:59.285Z"}]},{"id":"2b71a3a5-530e-45bd-a965-b53c2f48d0f6","slug":"swe-bench-verified-official","name":"SWE-bench Verified (official harness)","description":"The 500-task SWE-bench Verified set scored by the official SWE-bench Docker harness (resolved = FAIL_TO_PASS and PASS_TO_PASS tests pass). codai agent, single attempt per task, run 2026-08-03/04. Competitor rows are published leaderboard entries (source and date in each row), not re-runs by us.","category":"coding","taskCount":500,"referenceUrl":"https://www.swebench.com/","resultKind":"official","runs":[{"id":"a8f579d8-8370-486d-8a61-0b9583ee4bc5","modelTested":"claude-sonnet-5","successRate":"0.8520","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-06-30T00:00:00.000Z"},{"id":"6f180153-a2af-41e1-806e-afc625f84d2a","modelTested":"20251205_sonar-foundation-agent_claude-opus-4-5","successRate":"0.7920","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.459Z"},{"id":"57d9b2cb-7937-44ff-99f1-6c4382ef0473","modelTested":"20251215_livesweagent_claude-opus-4-5","successRate":"0.7920","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.281Z"},{"id":"c7b6e52f-86ac-4d0e-8dd4-755f50a44688","modelTested":"20250928_trae_doubao_seed_code","successRate":"0.7880","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.036Z"},{"id":"564c8a09-6184-4fa9-9ded-24d1838c73ed","modelTested":"OpenHands + Claude Opus 4.5","successRate":"0.7760","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.634Z"},{"id":"c40d7eb0-5b35-4bbe-b49d-574647f368e7","modelTested":"20251120_livesweagent_gemini-3-pro-preview","successRate":"0.7740","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.809Z"},{"id":"bedb466d-2afa-4163-acf4-e234baf77f4d","modelTested":"Claude 4.5 Opus (bash-only)","successRate":"0.7680","costUsd":"375.0000","costPerSuccessUsd":"0.9766","completedTasks":500,"totalTasks":500,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"828b2e9b-2969-4a09-8f9f-6a5c42202735","modelTested":"20250804_epam-ai-run-claude-4-sonnet","successRate":"0.7680","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.611Z"},{"id":"80f55db9-725b-4415-8c7a-62c9744134dc","modelTested":"20250902_atlassian-rovo-dev","successRate":"0.7680","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.563Z"},{"id":"d746b437-62f0-4b14-af39-8739a9c61c7a","modelTested":"20250819_ACoder","successRate":"0.7640","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.263Z"},{"id":"b472916f-d183-4d83-9e2f-d00691fa62bf","modelTested":"MiniMax M2.5 (bash-only)","successRate":"0.7580","costUsd":"35.0000","costPerSuccessUsd":"0.0923","completedTasks":500,"totalTasks":500,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"4f755eb1-324e-4146-8807-4420606479d4","modelTested":"Gemini 3 Flash (bash-only)","successRate":"0.7580","costUsd":"180.0000","costPerSuccessUsd":"0.4749","completedTasks":500,"totalTasks":500,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"863dce10-9c25-45c4-8006-9a133d07b2ab","modelTested":"20250901_warp","successRate":"0.7560","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.737Z"},{"id":"896d6265-716c-4fed-a267-4b02a9100fb7","modelTested":"20250731_harness_ai","successRate":"0.7480","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.960Z"},{"id":"988a5fd3-d6c5-4046-9d26-73c600fe4937","modelTested":"20251103_sonar-foundation-agent_claude-sonnet-4-5","successRate":"0.7480","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.336Z"},{"id":"565782ef-9011-4fe0-a5de-bf7630f94bd8","modelTested":"20250720_Lingxi-v1.5_claude-4-sonnet-20250514","successRate":"0.7460","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.310Z"},{"id":"d8e6aafa-9c4f-4d57-a62a-bda187dcb48f","modelTested":"20250915_JoyCode","successRate":"0.7460","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.386Z"},{"id":"96606277-e29c-45ed-aa6f-e115866464e7","modelTested":"20251103_SalesforceAIResearch_SAGE_OpenHands","successRate":"0.7380","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.511Z"},{"id":"481b7eb2-752b-4693-a6c3-d4c6138ce9b7","modelTested":"codai","successRate":"0.7320","costUsd":"304.7400","costPerSuccessUsd":"0.8326","completedTasks":500,"totalTasks":500,"finishedAt":"2026-08-04T12:00:00.000Z"},{"id":"708652d9-e00c-4359-a5ee-9c98888eb3aa","modelTested":"20251021_SalesforceAIResearch_SAGE_bash_only","successRate":"0.7300","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.686Z"},{"id":"8ed3c681-52fa-4dca-a371-f47df403be8e","modelTested":"20250807_openhands_gpt5","successRate":"0.7180","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.437Z"},{"id":"b7857b80-b63b-411a-8e8a-6c24cbb191c5","modelTested":"20250710_bloop","successRate":"0.7120","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.836Z"},{"id":"308c70f1-3775-455a-bf82-f4029fe8a56b","modelTested":"20250715_qodo_command","successRate":"0.7120","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.660Z"},{"id":"93135d38-7f8f-4c44-b270-555c3e480a29","modelTested":"20251014_Lingxi_kimi_k2","successRate":"0.7120","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.861Z"},{"id":"b0d62aa4-74f6-4f84-9674-35198868a498","modelTested":"20250623_warp","successRate":"0.7100","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.539Z"},{"id":"ca2735a3-5780-4fdd-86fd-83fe86ba2965","modelTested":"20250716_openhands_kimi_k2","successRate":"0.6540","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.486Z"},{"id":"f01b09a7-04a0-466a-b4ad-9e63156b202f","modelTested":"20250901_entroPO_R2E_QwenCoder30BA3B_tts","successRate":"0.6040","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.912Z"},{"id":"4f17cb75-50ba-48de-8137-b3f61a309b8a","modelTested":"20250629_deepswerl_r2eagent_tts","successRate":"0.5880","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.011Z"},{"id":"c4a7b45c-3a8f-48da-8d60-90a4adf8ae45","modelTested":"20250924_artemis_agent_v2","successRate":"0.5700","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.211Z"},{"id":"3aef01ad-cd17-47d7-af20-a2b460147d55","modelTested":"20251110_frogboss-32b","successRate":"0.5360","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.161Z"},{"id":"4088dddd-4109-4e9f-ac4e-cbf3221eaad7","modelTested":"20250804_codesweep_sweagent_kimi_k2_instruct","successRate":"0.5340","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.785Z"},{"id":"ccff5ba0-3084-494f-8f46-5bde406a6d66","modelTested":"20250901_entroPO_R2E_QwenCoder30BA3B","successRate":"0.5220","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.088Z"},{"id":"e6987d94-792a-4f25-b4f2-1ab4085edb3b","modelTested":"20250616_Skywork-SWE-32B+TTS_Bo8","successRate":"0.4700","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.714Z"},{"id":"32a2f738-f483-4bc4-ba0c-1c834a470b59","modelTested":"20251110_frogmini-14b","successRate":"0.4500","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.984Z"},{"id":"62fe1417-483c-4c50-850e-0c32c8360913","modelTested":"20250629_deepswerl_r2eagent","successRate":"0.4220","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.187Z"},{"id":"b1cba308-7520-4476-b86c-9aa7fad32ec7","modelTested":"20250616_Skywork-SWE-32B","successRate":"0.3800","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.889Z"},{"id":"e7852541-eb51-4716-95ff-75f5c0f53059","modelTested":"20250725_sweagent_devstral_small_2507","successRate":"0.3800","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.135Z"},{"id":"5f20d252-5cc8-4093-8b12-7ac47e177222","modelTested":"20250627_agentless_MCTS-Refine-7B","successRate":"0.2320","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.362Z"}]},{"id":"cfbbc993-6921-43b7-a7c0-29b4479e8bb1","slug":"aider-polyglot","name":"Aider Polyglot","description":"225-task multi-language coding benchmark (Python, JS, TS, Rust, Go, Java, C++). Anthropic and OpenAI both quote this for editing performance. Our harness runs the PYTHON subset (~38 tasks) — the largest single bucket — with the official two-pass methodology (one initial try + one retry given test output). Cross-language coverage is the next iteration; competitor scores below are full 225-task polyglot.","category":"coding","taskCount":225,"referenceUrl":"https://aider.chat/docs/leaderboards/","resultKind":null,"runs":[{"id":"003f1e8a-748d-4590-993f-4c3d398063fe","modelTested":"gpt-5 (high)","successRate":"0.8800","costUsd":"29.0829","costPerSuccessUsd":"0.1469","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.237Z"},{"id":"6472dcdd-74be-4a87-91d3-119b035b4cf3","modelTested":"gpt-5 (medium)","successRate":"0.8670","costUsd":"17.6930","costPerSuccessUsd":"0.0907","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.399Z"},{"id":"c52ac3ad-3aa4-45ed-aa9c-dfdcc95d98ce","modelTested":"claude-opus-4-7","successRate":"0.8529","costUsd":"0.9271","costPerSuccessUsd":"0.0320","completedTasks":34,"totalTasks":225,"finishedAt":"2026-05-27T03:57:03.844Z"},{"id":"a2f9292a-5cd5-42fc-b917-825d2f3ccdc7","modelTested":"o3-pro (high)","successRate":"0.8490","costUsd":"146.3249","costPerSuccessUsd":"0.7661","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.588Z"},{"id":"996f6875-300b-49b6-af76-b7a121f34164","modelTested":"gemini-2.5-pro-preview-06-05 (32k think)","successRate":"0.8310","costUsd":"49.8822","costPerSuccessUsd":"0.2667","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.773Z"},{"id":"e3e0a374-055c-4f15-8240-e7d557a3f2ca","modelTested":"gpt-5 (low)","successRate":"0.8130","costUsd":"10.3713","costPerSuccessUsd":"0.0567","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.562Z"},{"id":"bfa6bf22-2bc2-4038-baaa-652ff1a0eed4","modelTested":"o3 (high)","successRate":"0.8130","costUsd":"21.2259","costPerSuccessUsd":"0.1160","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.099Z"},{"id":"853fb117-c6d4-40b3-813b-7f49c61ce74d","modelTested":"grok-4 (high)","successRate":"0.7960","costUsd":"59.6182","costPerSuccessUsd":"0.3331","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.751Z"},{"id":"9331458b-34b8-4e8e-8fee-94e293f1d704","modelTested":"gemini-2.5-pro-preview-06-05 (default think)","successRate":"0.7910","costUsd":"45.5961","costPerSuccessUsd":"0.2562","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.610Z"},{"id":"5f553101-f34a-4ddb-b486-d20287156176","modelTested":"o3 (high) + gpt-4.1","successRate":"0.7820","costUsd":"17.5518","costPerSuccessUsd":"0.0997","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:18.425Z"},{"id":"d287a36c-d352-4104-a5cf-488d240ccae8","modelTested":"o3","successRate":"0.7690","costUsd":"13.7517","costPerSuccessUsd":"0.0795","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.262Z"},{"id":"77264ea2-e928-4536-a231-92a4067dc097","modelTested":"Gemini 2.5 Pro Preview 05-06","successRate":"0.7690","costUsd":"37.4104","costPerSuccessUsd":"0.2162","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.147Z"},{"id":"1aa17896-5cac-4284-9eab-ded913e6a6a8","modelTested":"DeepSeek-V3.2-Exp (Reasoner)","successRate":"0.7420","costUsd":"1.3045","costPerSuccessUsd":"0.0078","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.724Z"},{"id":"e3ab9d63-03be-455a-b83f-e7cf5640c5c6","modelTested":"claude-sonnet-4-6","successRate":"0.7353","costUsd":"0.6421","costPerSuccessUsd":"0.0257","completedTasks":34,"totalTasks":225,"finishedAt":"2026-05-27T03:56:24.165Z"},{"id":"d1446beb-c88d-44a2-8000-1c526b9a115e","modelTested":"Gemini 2.5 Pro Preview 03-25","successRate":"0.7290","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.877Z"},{"id":"4aaab394-1cb8-4b01-ab63-65d3edb1729f","modelTested":"claude-opus-4-20250514 (32k thinking)","successRate":"0.7200","costUsd":"65.7484","costPerSuccessUsd":"0.4059","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.121Z"},{"id":"c800d0ae-aeea-460e-944d-16558162163d","modelTested":"o4-mini (high)","successRate":"0.7200","costUsd":"19.6399","costPerSuccessUsd":"0.1212","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.658Z"},{"id":"d66df734-a62a-4143-8244-b92a2245fd61","modelTested":"DeepSeek R1 (0528)","successRate":"0.7140","costUsd":"4.8016","costPerSuccessUsd":"0.0300","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:17.936Z"},{"id":"0ac3ffea-cc8b-4adc-b882-f4c6f82580b7","modelTested":"claude-opus-4-20250514 (no think)","successRate":"0.7070","costUsd":"68.6253","costPerSuccessUsd":"0.4316","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.959Z"},{"id":"32e73ed8-c162-4e75-babc-32c52d3fbf48","modelTested":"claude-opus-4-8","successRate":"0.7059","costUsd":"13.4471","costPerSuccessUsd":"0.5603","completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-27T00:28:29.838Z"},{"id":"ee0c8dd3-d11d-413b-8557-e9862b4f260b","modelTested":"DeepSeek-V3.2-Exp (Chat)","successRate":"0.7020","costUsd":"0.8756","costPerSuccessUsd":"0.0055","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.886Z"},{"id":"04b9b69d-3025-444d-95ae-8d9ad62b8023","modelTested":"claude-3-7-sonnet-20250219 (32k thinking tokens)","successRate":"0.6490","costUsd":"36.8343","costPerSuccessUsd":"0.2523","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:12.745Z"},{"id":"37f09e02-bcaa-45b4-80cd-b61d962cef33","modelTested":"DeepSeek R1 + claude-3-5-sonnet-20241022","successRate":"0.6400","costUsd":"13.2933","costPerSuccessUsd":"0.0923","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.620Z"},{"id":"bc5919d1-ba7a-423c-b494-755849a512c7","modelTested":"codai","successRate":"0.6176","costUsd":"13.7060","costPerSuccessUsd":"0.6527","completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-27T00:26:20.285Z"},{"id":"f6759f54-1152-47fc-9811-e0dad22aaefc","modelTested":"o1-2024-12-17 (high)","successRate":"0.6170","costUsd":"186.4958","costPerSuccessUsd":"1.3417","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:09.849Z"},{"id":"07244ff9-20a3-4153-9048-3bd6f5add3cf","modelTested":"claude-sonnet-4-20250514 (32k thinking)","successRate":"0.6130","costUsd":"26.5755","costPerSuccessUsd":"0.1926","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.797Z"},{"id":"0d07b234-263a-44a0-a18e-29e55ca6f079","modelTested":"claude-3-7-sonnet-20250219 (no thinking)","successRate":"0.6040","costUsd":"17.7191","costPerSuccessUsd":"0.1303","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:12.584Z"},{"id":"b6888788-ffb4-4492-a841-9e28a32c59ee","modelTested":"o3-mini (high)","successRate":"0.6040","costUsd":"18.1584","costPerSuccessUsd":"0.1335","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:12.102Z"},{"id":"9f0dd111-d245-4fab-a679-2a9b8b657461","modelTested":"Qwen3 235B A22B diff, no think, Alibaba API","successRate":"0.5960","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.472Z"},{"id":"3d485443-6537-4216-a6cc-b0126eeb9c65","modelTested":"Kimi K2","successRate":"0.5910","costUsd":"1.2357","costPerSuccessUsd":"0.0093","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.912Z"},{"id":"4f962bd1-d0de-4b4e-87d2-a4be09d18c63","modelTested":"claude-haiku-4-5","successRate":"0.5882","costUsd":"2.1241","costPerSuccessUsd":"0.1062","completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-15T20:07:59.828Z"},{"id":"a986165a-8e3c-4438-bbd0-2d192f8d56c3","modelTested":"gpt-5-mini","successRate":"0.5882","costUsd":"4.1429","costPerSuccessUsd":"0.2071","completedTasks":34,"totalTasks":225,"finishedAt":"2026-05-27T09:08:43.162Z"},{"id":"9001ecc5-a0e2-4b1b-acde-6db67fef6a17","modelTested":"DeepSeek R1","successRate":"0.5690","costUsd":"5.4193","costPerSuccessUsd":"0.0423","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.459Z"},{"id":"15fee063-cc52-428b-9880-59b0484e4814","modelTested":"claude-sonnet-4-20250514 (no thinking)","successRate":"0.5640","costUsd":"15.8155","costPerSuccessUsd":"0.1245","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.635Z"},{"id":"f060b6d3-63f9-485a-9843-a8fa116c71d2","modelTested":"gemini-2.5-flash-preview-05-20 (24k think)","successRate":"0.5510","costUsd":"8.5625","costPerSuccessUsd":"0.0691","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.447Z"},{"id":"da91e1b7-72bc-40ca-911c-c4e4f2324074","modelTested":"DeepSeek V3 (0324)","successRate":"0.5510","costUsd":"1.1164","costPerSuccessUsd":"0.0090","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.714Z"},{"id":"a6b18e87-bd2f-4f48-8751-a367b4c5ecc9","modelTested":"Quasar Alpha","successRate":"0.5470","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.204Z"},{"id":"e777659a-b1a1-4529-8f0f-d28d1c7e7b05","modelTested":"o3-mini (medium)","successRate":"0.5380","costUsd":"8.8599","costPerSuccessUsd":"0.0732","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.942Z"},{"id":"52ce8c2a-c066-4862-85fa-ce5c38c1deb4","modelTested":"Grok 3 Beta","successRate":"0.5330","costUsd":"11.0338","costPerSuccessUsd":"0.0919","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.530Z"},{"id":"432c413d-d510-4805-bd8b-692b974d0ff5","modelTested":"Optimus Alpha","successRate":"0.5290","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.014Z"},{"id":"f8abf842-e801-4b4c-8143-4bd7358e8d02","modelTested":"gpt-4.1","successRate":"0.5240","costUsd":"9.8556","costPerSuccessUsd":"0.0835","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.175Z"},{"id":"4d522efc-1b5d-4347-8dfd-7f44b8ae0c9b","modelTested":"claude-3-5-sonnet-20241022","successRate":"0.5160","costUsd":"14.4063","costPerSuccessUsd":"0.1242","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.365Z"},{"id":"135fe35b-0451-45f0-b74d-6aa7d01a505d","modelTested":"Grok 3 Mini Beta (high)","successRate":"0.4930","costUsd":"0.7346","costPerSuccessUsd":"0.0066","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.853Z"},{"id":"476ae89d-0c00-4a33-bc04-8021ad96ffd5","modelTested":"DeepSeek Chat V3 (prev)","successRate":"0.4840","costUsd":"0.3369","costPerSuccessUsd":"0.0031","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.137Z"},{"id":"4abf44bf-e020-4b3c-9e4a-4a55292d52ca","modelTested":"gemini-2.5-flash-preview-04-17 (default)","successRate":"0.4710","costUsd":"1.8451","costPerSuccessUsd":"0.0174","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.983Z"},{"id":"2863ce7d-4b0d-440b-9e2f-0eeeb6dcdde5","modelTested":"chatgpt-4o-latest (2025-03-29)","successRate":"0.4530","costUsd":"19.7416","costPerSuccessUsd":"0.1935","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.040Z"},{"id":"ef04c8eb-4c52-448d-b215-f0345ed27d2d","modelTested":"gpt-4.5-preview","successRate":"0.4490","costUsd":"183.1802","costPerSuccessUsd":"1.8137","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:12.905Z"},{"id":"45e4ed40-77de-4fbb-8858-92b12116384f","modelTested":"gemini-2.5-flash-preview-05-20 (no think)","successRate":"0.4400","costUsd":"1.1354","costPerSuccessUsd":"0.0115","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.284Z"},{"id":"b7e273c5-82e6-43e8-9f2c-98bbb295beba","modelTested":"gpt-oss-120b (high)","successRate":"0.4180","costUsd":"0.7406","costPerSuccessUsd":"0.0079","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.073Z"},{"id":"32d1ba14-5d06-44ae-969f-0a70fbc90ae9","modelTested":"Qwen3 32B","successRate":"0.4000","costUsd":"0.7603","costPerSuccessUsd":"0.0084","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.309Z"},{"id":"29d8761e-e386-4ed9-84f5-19002883c23a","modelTested":"gemini-3.1-flash-lite","successRate":"0.3824","costUsd":"0.0547","costPerSuccessUsd":"0.0042","completedTasks":34,"totalTasks":225,"finishedAt":"2026-08-11T22:46:44.115Z"},{"id":"da6ea955-af1b-416c-b5b9-45d908f91136","modelTested":"gemini-exp-1206","successRate":"0.3820","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.657Z"},{"id":"6738a76c-511e-4adc-a760-f33a11589666","modelTested":"Gemini 2.0 Pro exp-02-05","successRate":"0.3560","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.039Z"},{"id":"d4c9f662-c477-47d4-b46b-7bdb502c2541","modelTested":"Grok 3 Mini Beta (low)","successRate":"0.3470","costUsd":"0.7856","costPerSuccessUsd":"0.0101","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.692Z"},{"id":"8c9cd1ea-1a6f-422e-81fe-cde8d19d2a5a","modelTested":"o1-mini-2024-09-12","successRate":"0.3290","costUsd":"18.5770","costPerSuccessUsd":"0.2510","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.497Z"},{"id":"c6e695da-2c64-4493-80fe-753e40eb9ccc","modelTested":"gpt-4.1-mini","successRate":"0.3240","costUsd":"1.9918","costPerSuccessUsd":"0.0273","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.336Z"},{"id":"632f3418-c56c-46ae-ad7b-b1537b6b9b48","modelTested":"gemini-3.5-flash-lite","successRate":"0.3235","costUsd":"0.4514","costPerSuccessUsd":"0.0410","completedTasks":34,"totalTasks":225,"finishedAt":"2026-08-11T22:48:26.953Z"},{"id":"4cdcbf8b-9de0-49cb-8c09-fe6b87c38458","modelTested":"claude-3-5-haiku-20241022","successRate":"0.2800","costUsd":"6.0583","costPerSuccessUsd":"0.0962","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.173Z"},{"id":"432b56f6-30bb-450a-a15a-1fba46d18eeb","modelTested":"chatgpt-4o-latest (2025-02-15)","successRate":"0.2710","costUsd":"14.3703","costPerSuccessUsd":"0.2356","completedTasks":223,"totalTasks":223,"finishedAt":"2026-05-27T00:40:12.423Z"},{"id":"d3f32050-e012-4bf9-8ceb-2d783940dcf4","modelTested":"QwQ-32B + Qwen 2.5 Coder Instruct","successRate":"0.2620","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.226Z"},{"id":"be486b63-b5cf-4177-8f66-a81d93112ded","modelTested":"gpt-4o-2024-08-06","successRate":"0.2310","costUsd":"7.0286","costPerSuccessUsd":"0.1352","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.688Z"},{"id":"bc2c3fc0-7ca6-4994-b924-ba4b58f82d6d","modelTested":"gemini-2.0-flash-exp","successRate":"0.2220","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.817Z"},{"id":"dda00784-fdae-4903-9f0f-484ef0f195b0","modelTested":"qwen-max-2025-01-25","successRate":"0.2180","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.781Z"},{"id":"bb3b67f9-e83c-438e-a2c5-a10fdb47b1ea","modelTested":"QwQ-32B","successRate":"0.2090","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.065Z"},{"id":"90596882-142d-4b8c-bd3c-24d11aee6ece","modelTested":"gemini-2.0-flash-thinking-exp-01-21","successRate":"0.1820","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:12.262Z"},{"id":"cab5b341-40aa-498d-8be0-57101d92d315","modelTested":"gpt-4o-2024-11-20","successRate":"0.1820","costUsd":"6.7351","costPerSuccessUsd":"0.1643","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.526Z"},{"id":"3fbc65e3-ef15-4a70-80b4-1cbf66686cbb","modelTested":"DeepSeek Chat V2.5","successRate":"0.1780","costUsd":"0.5101","costPerSuccessUsd":"0.0128","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.012Z"},{"id":"2da9bb50-b74f-4539-870a-c75277998698","modelTested":"Qwen2.5-Coder-32B-Instruct","successRate":"0.1640","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.335Z"},{"id":"d02fb285-c434-4333-a91a-b495e27c9024","modelTested":"Llama 4 Maverick","successRate":"0.1560","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.367Z"},{"id":"9ce3cd54-732f-45ff-af0f-f780611d5a51","modelTested":"yi-lightning","successRate":"0.1290","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.978Z"},{"id":"acd6978d-f59d-4b6c-b3b1-90ebb77d7ae0","modelTested":"command-a-03-2025-quality","successRate":"0.1200","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.388Z"},{"id":"6f30d1d0-5722-4841-b7d2-e4b04ca44c2d","modelTested":"Codestral 25.01","successRate":"0.1110","costUsd":"1.9834","costPerSuccessUsd":"0.0793","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.298Z"},{"id":"a60923d1-06ae-40f9-8bc6-c93d95488f94","modelTested":"openhands-lm-32b-v0.1","successRate":"0.1020","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.820Z"},{"id":"77670eed-09ef-4b30-9572-577441749b75","modelTested":"gpt-4.1-nano","successRate":"0.0890","costUsd":"0.4281","costPerSuccessUsd":"0.0214","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.497Z"},{"id":"aadd630b-d1b3-4df9-b7ef-49a639840778","modelTested":"gemini-3.6-flash","successRate":"0.0588","costUsd":"0.3091","costPerSuccessUsd":"0.1545","completedTasks":34,"totalTasks":225,"finishedAt":"2026-08-11T22:52:41.349Z"},{"id":"56fa48ab-7c87-4c09-9cc7-945093c348e0","modelTested":"gemma-3-27b-it","successRate":"0.0490","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.552Z"},{"id":"ca728bcc-c0f5-401d-a6fb-8a42dac86d04","modelTested":"gpt-4o-mini-2024-07-18","successRate":"0.0360","costUsd":"0.3236","costPerSuccessUsd":"0.0404","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.202Z"},{"id":"8d242936-9404-4a7d-84ab-fcf1127f6aa8","modelTested":"codai-alpha-1-0","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-15T21:35:30.002Z"},{"id":"d216b61f-84b8-4d8c-8251-19e14c202027","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-15T20:01:56.351Z"}]},{"id":"cb0d4d49-baba-4d41-8341-b9d036e6d017","slug":"humaneval","name":"HumanEval","description":"OpenAI 164-problem Python coding benchmark. Single-turn: prompt → function body → run unit tests. Cheap, fast, well-known — used as the pipeline smoke benchmark.","category":"coding","taskCount":164,"referenceUrl":"https://arxiv.org/abs/2107.03374","resultKind":null,"runs":[{"id":"b493f430-1d9b-4c16-af11-d6475f249191","modelTested":"codai,claude-opus-4-8","successRate":"0.9695","costUsd":"1.7589","costPerSuccessUsd":"0.0111","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-27T00:16:09.456Z"},{"id":"06ba910b-395b-4f83-903b-b7bdcecd69b6","modelTested":"claude-opus-4-8","successRate":"0.9634","costUsd":"2.0007","costPerSuccessUsd":"0.0127","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-27T00:25:18.842Z"},{"id":"f1accbb5-cf4e-4d08-bda4-4f21cb37a189","modelTested":"codai","successRate":"0.9634","costUsd":"1.4539","costPerSuccessUsd":"0.0092","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-27T00:23:47.612Z"},{"id":"593ba55d-1320-48e2-a50f-6dea4f2bb7c4","modelTested":"claude-haiku-4-5","successRate":"0.9512","costUsd":"1.8172","costPerSuccessUsd":"0.0116","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-15T20:08:16.381Z"},{"id":"67fdf837-3078-46a5-8b9d-481cf343c7df","modelTested":"claude-sonnet-4-7","successRate":"0.9451","costUsd":"0.2064","costPerSuccessUsd":"0.0013","completedTasks":164,"totalTasks":164,"finishedAt":"2026-05-26T22:29:04.148Z"},{"id":"1bab34c3-cdd9-45aa-948b-4fff8d5a067e","modelTested":"claude-opus-4-7","successRate":"0.9390","costUsd":"0.2070","costPerSuccessUsd":"0.0013","completedTasks":164,"totalTasks":164,"finishedAt":"2026-05-26T22:29:48.310Z"},{"id":"6084a69a-db33-4561-aed5-baaf1ec3b0a6","modelTested":"gpt-5-mini","successRate":"0.8232","costUsd":"2.6352","costPerSuccessUsd":"0.0195","completedTasks":164,"totalTasks":164,"finishedAt":"2026-05-27T08:44:47.556Z"},{"id":"24c64877-f736-4000-b1c7-5b3dc57cc3be","modelTested":"codai-alpha-1-0","successRate":"0.4207","costUsd":"20.9714","costPerSuccessUsd":"0.3039","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-16T01:09:51.641Z"},{"id":"222473eb-760f-44d1-8408-f7ce4defdbff","modelTested":"gemini-3.6-flash","successRate":"0.1402","costUsd":"0.2059","costPerSuccessUsd":"0.0090","completedTasks":164,"totalTasks":164,"finishedAt":"2026-08-11T22:46:46.586Z"},{"id":"fc4efea3-0ce2-42fc-be44-518cac32c422","modelTested":"gemini-3.5-flash","successRate":"0.1220","costUsd":"0.1937","costPerSuccessUsd":"0.0097","completedTasks":164,"totalTasks":164,"finishedAt":"2026-08-11T22:47:19.378Z"},{"id":"91335538-7541-4230-b33f-17cd11eab184","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-15T19:58:42.839Z"}]},{"id":"d238d745-eb9b-40da-aad9-dab1de5a3601","slug":"swe-bench-verified","name":"SWE-bench Verified (oracle-diff proxy, 100)","description":"PROXY, not the official harness: 100 SWE-bench Verified issues, the model gets the issue plus the oracle affected files and outputs a unified diff, scored by line-overlap similarity to the gold patch plus an LLM judge on borderline cases. No tests are executed. Not comparable to leaderboard numbers; see \"SWE-bench Verified (official harness)\" for the real score.","category":"coding","taskCount":100,"referenceUrl":"https://www.swebench.com/","resultKind":"proxy","runs":[{"id":"3d3c30b0-b992-4fec-9913-3d68ec856074","modelTested":"claude-sonnet-5","successRate":"0.8520","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-06-30T00:00:00.000Z"},{"id":"bffd13a9-d0f7-4e51-9bf2-27ccbfb46741","modelTested":"claude-fable-5","successRate":"0.6500","costUsd":"38.7123","costPerSuccessUsd":"0.5956","completedTasks":100,"totalTasks":100,"finishedAt":"2026-07-11T01:57:50.991Z"},{"id":"d3607c3c-98e1-4fc2-b79f-9201b0065258","modelTested":"codai","successRate":"0.4600","costUsd":"33.9823","costPerSuccessUsd":"0.7387","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-27T01:12:34.222Z"},{"id":"a41fbb0e-5be0-4876-abc4-b18a916d9a6a","modelTested":"gemini-3.5-flash","successRate":"0.0100","costUsd":"2.1185","costPerSuccessUsd":"2.1185","completedTasks":100,"totalTasks":100,"finishedAt":"2026-08-11T22:54:57.113Z"},{"id":"7d3625b3-5bc9-4b96-b7da-55f950aca2f8","modelTested":"gemini-3.1-flash-lite","successRate":"0.0000","costUsd":"0.2813","costPerSuccessUsd":null,"completedTasks":100,"totalTasks":100,"finishedAt":"2026-08-11T22:52:30.683Z"},{"id":"609eb8ce-5e67-44f4-84cb-a163ce78c2e9","modelTested":"gemini-3.5-flash-lite","successRate":"0.0000","costUsd":"0.3713","costPerSuccessUsd":null,"completedTasks":100,"totalTasks":100,"finishedAt":"2026-08-11T22:52:09.771Z"}]},{"id":"0b9e454e-48e1-494c-9188-a61f9950692e","slug":"swe-bench-patch","name":"SWE-bench Verified (patch-only)","description":"Patch-only variant of SWE-bench Verified. Model receives the issue + the 'oracle' affected files (princeton-nlp/SWE-bench_oracle pre-formatted text) and must output a unified diff that fixes the issue. NO Docker, NO test execution — we score the candidate diff against the gold patch with line-overlap similarity (cheap fast-path) plus an LLM judge (sonnet-4-6) for borderline cases (0.3 < sim < 0.7). Not directly comparable to the official leaderboard but a strong proxy for diff-generation quality at 1% of the cost.","category":"coding","taskCount":100,"referenceUrl":"https://www.swebench.com/","resultKind":"proxy","runs":[{"id":"d89b6833-2b69-4052-bab6-c47a9a3428c4","modelTested":"codai","successRate":"0.4700","costUsd":"126.5339","costPerSuccessUsd":"2.6922","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T13:43:00.657Z"},{"id":"2115e64a-d849-430f-bbe5-cb95e54621d4","modelTested":"claude-opus-4-7","successRate":"0.4600","costUsd":"127.0859","costPerSuccessUsd":"2.7627","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T13:43:17.456Z"},{"id":"cd66c208-b3ba-4456-8eaf-3d3f63c2b373","modelTested":"claude-opus-4-8","successRate":"0.3600","costUsd":"36.5610","costPerSuccessUsd":"1.0156","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-29T00:09:26.774Z"},{"id":"2f2c4dbc-2664-4dea-a121-e0e9122df9f9","modelTested":"claude-haiku-4-5","successRate":"0.3500","costUsd":"9.3787","costPerSuccessUsd":"0.2680","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-15T20:49:43.045Z"},{"id":"1a7ab46b-5c36-4787-bcda-e34eb2202a33","modelTested":"gpt-5-mini","successRate":"0.3100","costUsd":"128.6182","costPerSuccessUsd":"4.1490","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T13:45:17.194Z"},{"id":"a2d83174-f487-4316-ae7a-fc8bafa79488","modelTested":"claude-sonnet-4-6","successRate":"0.2900","costUsd":"24.5090","costPerSuccessUsd":"0.8451","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T15:31:46.516Z"}]},{"id":"251e522d-d07a-4c46-a850-d15ed645a0c7","slug":"math-competition","name":"Competition Math (AIME)","description":"Hard mathematical-reasoning benchmark built from AIME 2024, 2025, and 2026 (American Invitational Mathematics Examination). Every answer is an integer in 0..999, so scoring is fully DETERMINISTIC and un-gameable — we extract the model's final integer (`Final answer: N` sentinel, then \\boxed{N}, then last integer) and compare for exact equality. No LLM judge, so no judge cost or judge variance poisoning the reasoning priors. Closes a frontier category (hard reasoning/math) the scorecard was previously blind to.","category":"reasoning","taskCount":90,"referenceUrl":"https://artofproblemsolving.com/wiki/index.php/AIME","resultKind":null,"runs":[{"id":"17a5b65f-2263-4857-b646-75119f070615","modelTested":"codai","successRate":"0.9333","costUsd":"2.4751","costPerSuccessUsd":"0.0442","completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-16T00:23:26.303Z"},{"id":"53792ac2-7d85-4a97-9383-7815954bed99","modelTested":"claude-opus-4-8","successRate":"0.9167","costUsd":"1.9558","costPerSuccessUsd":"0.0356","completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-27T00:28:02.972Z"},{"id":"65c9f2da-1635-4024-9938-7427e906b382","modelTested":"codai,claude-opus-4-8","successRate":"0.9000","costUsd":"1.5583","costPerSuccessUsd":"0.0289","completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-27T00:17:41.335Z"},{"id":"6b5784a7-50bf-4348-a0d6-5062d80b9d8d","modelTested":"gemini-3.5-flash-lite","successRate":"0.0333","costUsd":"0.7777","costPerSuccessUsd":"0.3888","completedTasks":60,"totalTasks":60,"finishedAt":"2026-08-11T22:50:50.215Z"},{"id":"5b56bf2c-90c5-499b-90d8-5733e853fd16","modelTested":"codai-alpha-1-0","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-16T00:17:27.271Z"}]},{"id":"285dd5cd-b73d-4ae6-8c11-efedbddd7666","slug":"graphwalks","name":"GraphWalks (Long-Context)","description":"Frontier v8 long-context multi-hop reasoning gate. Each task embeds a large deterministically-generated directed graph in the prompt and asks a BFS-reachability or direct-parents query; scored by exact node-set match (no judge). Modelled on the GraphWalks eval in the Anthropic Mythos 5 / Fable 5 system card (§8, BFS 256K). Un-gameable: graphs are generated per-seed, never downloaded. Cost scales with nodeCount (context length).","category":"reasoning","taskCount":20,"referenceUrl":"https://www.anthropic.com","resultKind":null,"runs":[{"id":"7d4e187d-d8bc-4c12-b859-2551ff72e3f5","modelTested":"claude-opus-4-8","successRate":"1.0000","costUsd":"0.6238","costPerSuccessUsd":"0.0312","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-27T00:21:51.296Z"},{"id":"9697527f-9fac-42db-a6c3-ae049f514394","modelTested":"codai,claude-opus-4-8","successRate":"1.0000","costUsd":"0.6607","costPerSuccessUsd":"0.0330","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-27T00:15:08.773Z"},{"id":"b804c2b4-769b-450f-a51a-f2d875d8c332","modelTested":"claude-sonnet-4-6","successRate":"1.0000","costUsd":"2.2798","costPerSuccessUsd":"0.1140","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-14T19:13:17.994Z"},{"id":"4c3abd1d-4263-454d-a19e-f1fcb7556111","modelTested":"codai","successRate":"0.9500","costUsd":"0.7191","costPerSuccessUsd":"0.0378","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-27T00:21:43.578Z"},{"id":"8375dfc6-c5fb-4eb4-8921-2952ef7fabbc","modelTested":"claude-haiku-4-5","successRate":"0.9500","costUsd":"0.4228","costPerSuccessUsd":"0.0223","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-15T20:08:54.448Z"},{"id":"7d56c8be-c792-4184-aee6-71d35c2ea3be","modelTested":"codai-alpha-1-0","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-15T21:20:54.019Z"},{"id":"efb4b2c9-5cdf-45b6-b9f9-1274d3124862","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-15T20:06:38.129Z"}]},{"id":"081d1bf0-402e-43b8-8684-6b35a6dbeea2","slug":"long-context","name":"Long Context (RULER-style)","description":"Long-context retrieval benchmark (RULER-style needle-in-a-haystack). Fully SYNTHETIC + DETERMINISTIC: a filler haystack is generated at a target token budget (2k -> 128k), magic-number needles are inserted at spread depths, and the model must retrieve them. Scored by exact value match (all needles required). No external dataset, no LLM judge. Sweeps context lengths so we can see where each model’s effective window degrades. Closes the long-context frontier category.","category":"reasoning","taskCount":10,"referenceUrl":"https://github.com/NVIDIA/RULER","resultKind":null,"runs":[{"id":"230a0cba-f1d8-4416-b51f-ead4bf3fefea","modelTested":"gemini-3.6-flash","successRate":"1.0000","costUsd":"0.4492","costPerSuccessUsd":"0.0449","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:24.545Z"},{"id":"c7903ee3-6145-4f8d-950d-885053b6fbba","modelTested":"codai,claude-opus-4-8","successRate":"1.0000","costUsd":"5.5204","costPerSuccessUsd":"0.5520","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-27T00:14:31.242Z"},{"id":"6f028c4c-c85e-43f4-8561-3de96ae45a5f","modelTested":"claude-opus-4-8","successRate":"1.0000","costUsd":"4.4342","costPerSuccessUsd":"0.4434","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-16T00:17:43.048Z"},{"id":"018a7f52-b3e3-40ef-be30-6f643b666e2c","modelTested":"codai","successRate":"0.7000","costUsd":"0.5981","costPerSuccessUsd":"0.0854","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:38.785Z"},{"id":"eec1fb10-7c7c-47b5-97f5-20238a407b25","modelTested":"gemini-3.5-flash","successRate":"0.7000","costUsd":"0.3166","costPerSuccessUsd":"0.0452","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:38.698Z"},{"id":"0be491e2-4192-4193-ba22-7c43d722b08e","modelTested":"gemini-3.1-pro-preview","successRate":"0.5000","costUsd":"0.1351","costPerSuccessUsd":"0.0270","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:34.660Z"},{"id":"a299742e-7991-4cde-87c8-0701519d416b","modelTested":"gemini-3.1-flash-lite","successRate":"0.4000","costUsd":"0.0119","costPerSuccessUsd":"0.0030","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:31.161Z"},{"id":"d3aff915-d771-48eb-8a74-1072efe36b40","modelTested":"codai-alpha-1-0","successRate":"0.1000","costUsd":"0.1730","costPerSuccessUsd":"0.1730","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-16T00:18:18.231Z"}]},{"id":"acc3499a-9688-411f-aed3-b9be21c44466","slug":"mmlu-pro","name":"MMLU-Pro","description":"TIGER-Lab's 12,032-question reasoning benchmark across 14 domains (biology, business, chemistry, CS, economics, engineering, health, history, law, math, philosophy, physics, psychology, other). 10-choice multiple-choice, deterministic letter-match scoring. We run a 200-question stratified sample per category (28 per category, capped at the available count).","category":"reasoning","taskCount":12032,"referenceUrl":"https://arxiv.org/abs/2406.01574","resultKind":null,"runs":[{"id":"a2f5c831-318f-43ba-bbef-fdd3b4e64a77","modelTested":"codai,claude-opus-4-8","successRate":"0.8200","costUsd":"1.2471","costPerSuccessUsd":"0.0076","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-27T00:15:41.563Z"},{"id":"9343b36e-38a0-45d2-b2fb-d06e155065cf","modelTested":"codai","successRate":"0.8150","costUsd":"1.1059","costPerSuccessUsd":"0.0068","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-27T00:23:34.442Z"},{"id":"bf1d30cb-31c9-42ec-8754-74dd68d23c8a","modelTested":"claude-opus-4-8","successRate":"0.7900","costUsd":"1.5828","costPerSuccessUsd":"0.0100","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-27T00:24:20.059Z"},{"id":"eebd0f92-a99f-4f9d-8b9c-0bfc02eca00e","modelTested":"gpt-5-mini","successRate":"0.7550","costUsd":"2.0394","costPerSuccessUsd":"0.0135","completedTasks":200,"totalTasks":200,"finishedAt":"2026-05-27T08:50:50.606Z"},{"id":"a5cf256d-0c37-4df9-9903-93b9b2c8d007","modelTested":"claude-haiku-4-5","successRate":"0.6550","costUsd":"2.0248","costPerSuccessUsd":"0.0155","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-15T20:05:48.847Z"},{"id":"f094fb2d-6952-4821-a55b-6d5bbef22352","modelTested":"claude-opus-4-7","successRate":"0.6200","costUsd":"0.0855","costPerSuccessUsd":"0.0007","completedTasks":200,"totalTasks":200,"finishedAt":"2026-05-26T22:45:00.520Z"},{"id":"5af6dc16-c9ce-4921-ab58-f45c9233cdf1","modelTested":"claude-sonnet-4-7","successRate":"0.6150","costUsd":"0.0763","costPerSuccessUsd":"0.0006","completedTasks":200,"totalTasks":200,"finishedAt":"2026-05-26T22:44:38.655Z"},{"id":"f4933fb4-117f-4f3a-b39a-06be8a0e0c23","modelTested":"codai-alpha-1-0","successRate":"0.0250","costUsd":"7.3842","costPerSuccessUsd":"1.4768","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-15T22:02:53.587Z"},{"id":"6c3a66fa-b31a-4af1-904c-f8a4b5db4e85","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-15T20:01:14.708Z"}]},{"id":"28af3441-0e7e-4eda-a465-8b8c73463ee8","slug":"problem-solving","name":"Problem Solving (Reasoning)","description":"Frontier v9 #1-priority category. Single-shot HARD reasoning tasks that other harnesses miss: multi-step logic puzzles (balance-weighing, river/bridge crossings), quantitative word problems with intuitive-but-wrong traps, systematic root-cause/debugging strategy, constraint satisfaction, and Fermi estimation. NOT code-execution (humaneval/codai-swe own that) and NOT short-answer contest math (math-competition owns that) — rewards correct multi-step reasoning AND a sound, communicable method. Scored by a weighted-rubric LLM judge (correctness-critical criteria first); a task passes at >=0.7. Mirrors the training-side Rubrics-as-Rewards signal so the eval measures exactly what RLVR+RaR optimizes. Fixed, version-controlled tasks (problem-solving.ts) for reproducibility.","category":"reasoning","taskCount":8,"referenceUrl":"https://arxiv.org/abs/2206.04615","resultKind":null,"runs":[]},{"id":"cada1ba6-f570-41cb-b662-1792efbe87f1","slug":"romath","name":"RoMath","description":"Romanian high-school mathematics benchmark from RoMathExam (Cuclea et al. 2026, 10,592 problems from official Bacalaureat exams 1895-2025, MIT license). Stratified sample across M1/M2/M3/M4 tracks. Scored by an LLM judge (sonnet-4-6) comparing the candidate's final answer to the official `barem` — math can't be exact-matched. Codai-unique angle: no major LLM router publishes Romanian-specific scores.","category":"romanian","taskCount":100,"referenceUrl":"https://arxiv.org/abs/2409.11074","resultKind":null,"runs":[{"id":"69df94fe-4121-45fe-a35f-5169ae6bd607","modelTested":"claude-opus-4-7","successRate":"0.9900","costUsd":"1.6449","costPerSuccessUsd":"0.0166","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T09:52:36.462Z"},{"id":"8e15d8cd-b554-41ba-9909-0602dfa634ec","modelTested":"codai","successRate":"0.9800","costUsd":"3.7393","costPerSuccessUsd":"0.0382","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-27T00:38:28.794Z"},{"id":"56624864-8b68-4dad-ac44-a1c8c065aee8","modelTested":"claude-opus-4-8","successRate":"0.9800","costUsd":"2.3230","costPerSuccessUsd":"0.0237","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-27T00:28:21.954Z"},{"id":"582d7abe-3409-4b90-afbe-4d7043b30fe2","modelTested":"claude-haiku-4-5","successRate":"0.9600","costUsd":"0.9862","costPerSuccessUsd":"0.0103","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-15T20:23:53.938Z"},{"id":"8fa5af57-a670-4554-bca4-b430c1855a8a","modelTested":"claude-sonnet-4-6","successRate":"0.9600","costUsd":"1.4620","costPerSuccessUsd":"0.0152","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T09:50:26.267Z"},{"id":"661384ea-f29b-4c57-a8df-257505857651","modelTested":"gpt-5-mini","successRate":"0.8000","costUsd":"3.3954","costPerSuccessUsd":"0.0424","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T09:57:14.267Z"},{"id":"4511f7e7-1a71-4229-8ec1-6e04c948f101","modelTested":"codai-alpha-1-0","successRate":"0.1200","costUsd":"1.7195","costPerSuccessUsd":"0.1433","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-15T22:02:23.722Z"},{"id":"0fb59bd8-56c1-4a13-99e4-4a40347adaf4","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-15T20:13:47.859Z"}]}]}