{"name":"codai benchmarks — reasoning","scope":"reasoning","license":"https://creativecommons.org/licenses/by/4.0/","attribution":"CC BY 4.0, attribution: codai (https://codai.ro/benchmarks)","source":"https://codai.ro/benchmarks/reasoning","selection":"One run per benchmark and model: the completed run with the largest sample; ties go to the most recent finish.","units":{"successRate":"fraction 0..1","costUsd":"USD","costPerSuccessUsd":"USD"},"lastRefresh":"2026-08-11T22:50:50.215Z","benchmarks":[{"id":"251e522d-d07a-4c46-a850-d15ed645a0c7","slug":"math-competition","name":"Competition Math (AIME)","description":"Hard mathematical-reasoning benchmark built from AIME 2024, 2025, and 2026 (American Invitational Mathematics Examination). Every answer is an integer in 0..999, so scoring is fully DETERMINISTIC and un-gameable — we extract the model's final integer (`Final answer: N` sentinel, then \\boxed{N}, then last integer) and compare for exact equality. No LLM judge, so no judge cost or judge variance poisoning the reasoning priors. Closes a frontier category (hard reasoning/math) the scorecard was previously blind to.","category":"reasoning","taskCount":90,"referenceUrl":"https://artofproblemsolving.com/wiki/index.php/AIME","resultKind":null,"runs":[{"id":"17a5b65f-2263-4857-b646-75119f070615","modelTested":"codai","successRate":"0.9333","costUsd":"2.4751","costPerSuccessUsd":"0.0442","completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-16T00:23:26.303Z"},{"id":"53792ac2-7d85-4a97-9383-7815954bed99","modelTested":"claude-opus-4-8","successRate":"0.9167","costUsd":"1.9558","costPerSuccessUsd":"0.0356","completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-27T00:28:02.972Z"},{"id":"65c9f2da-1635-4024-9938-7427e906b382","modelTested":"codai,claude-opus-4-8","successRate":"0.9000","costUsd":"1.5583","costPerSuccessUsd":"0.0289","completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-27T00:17:41.335Z"},{"id":"6b5784a7-50bf-4348-a0d6-5062d80b9d8d","modelTested":"gemini-3.5-flash-lite","successRate":"0.0333","costUsd":"0.7777","costPerSuccessUsd":"0.3888","completedTasks":60,"totalTasks":60,"finishedAt":"2026-08-11T22:50:50.215Z"},{"id":"5b56bf2c-90c5-499b-90d8-5733e853fd16","modelTested":"codai-alpha-1-0","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":60,"totalTasks":60,"finishedAt":"2026-06-16T00:17:27.271Z"}]},{"id":"285dd5cd-b73d-4ae6-8c11-efedbddd7666","slug":"graphwalks","name":"GraphWalks (Long-Context)","description":"Frontier v8 long-context multi-hop reasoning gate. Each task embeds a large deterministically-generated directed graph in the prompt and asks a BFS-reachability or direct-parents query; scored by exact node-set match (no judge). Modelled on the GraphWalks eval in the Anthropic Mythos 5 / Fable 5 system card (§8, BFS 256K). Un-gameable: graphs are generated per-seed, never downloaded. Cost scales with nodeCount (context length).","category":"reasoning","taskCount":20,"referenceUrl":"https://www.anthropic.com","resultKind":null,"runs":[{"id":"7d4e187d-d8bc-4c12-b859-2551ff72e3f5","modelTested":"claude-opus-4-8","successRate":"1.0000","costUsd":"0.6238","costPerSuccessUsd":"0.0312","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-27T00:21:51.296Z"},{"id":"9697527f-9fac-42db-a6c3-ae049f514394","modelTested":"codai,claude-opus-4-8","successRate":"1.0000","costUsd":"0.6607","costPerSuccessUsd":"0.0330","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-27T00:15:08.773Z"},{"id":"b804c2b4-769b-450f-a51a-f2d875d8c332","modelTested":"claude-sonnet-4-6","successRate":"1.0000","costUsd":"2.2798","costPerSuccessUsd":"0.1140","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-14T19:13:17.994Z"},{"id":"4c3abd1d-4263-454d-a19e-f1fcb7556111","modelTested":"codai","successRate":"0.9500","costUsd":"0.7191","costPerSuccessUsd":"0.0378","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-27T00:21:43.578Z"},{"id":"8375dfc6-c5fb-4eb4-8921-2952ef7fabbc","modelTested":"claude-haiku-4-5","successRate":"0.9500","costUsd":"0.4228","costPerSuccessUsd":"0.0223","completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-15T20:08:54.448Z"},{"id":"7d56c8be-c792-4184-aee6-71d35c2ea3be","modelTested":"codai-alpha-1-0","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-15T21:20:54.019Z"},{"id":"efb4b2c9-5cdf-45b6-b9f9-1274d3124862","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":20,"totalTasks":20,"finishedAt":"2026-06-15T20:06:38.129Z"}]},{"id":"081d1bf0-402e-43b8-8684-6b35a6dbeea2","slug":"long-context","name":"Long Context (RULER-style)","description":"Long-context retrieval benchmark (RULER-style needle-in-a-haystack). Fully SYNTHETIC + DETERMINISTIC: a filler haystack is generated at a target token budget (2k -> 128k), magic-number needles are inserted at spread depths, and the model must retrieve them. Scored by exact value match (all needles required). No external dataset, no LLM judge. Sweeps context lengths so we can see where each model’s effective window degrades. Closes the long-context frontier category.","category":"reasoning","taskCount":10,"referenceUrl":"https://github.com/NVIDIA/RULER","resultKind":null,"runs":[{"id":"230a0cba-f1d8-4416-b51f-ead4bf3fefea","modelTested":"gemini-3.6-flash","successRate":"1.0000","costUsd":"0.4492","costPerSuccessUsd":"0.0449","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:24.545Z"},{"id":"c7903ee3-6145-4f8d-950d-885053b6fbba","modelTested":"codai,claude-opus-4-8","successRate":"1.0000","costUsd":"5.5204","costPerSuccessUsd":"0.5520","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-27T00:14:31.242Z"},{"id":"6f028c4c-c85e-43f4-8561-3de96ae45a5f","modelTested":"claude-opus-4-8","successRate":"1.0000","costUsd":"4.4342","costPerSuccessUsd":"0.4434","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-16T00:17:43.048Z"},{"id":"018a7f52-b3e3-40ef-be30-6f643b666e2c","modelTested":"codai","successRate":"0.7000","costUsd":"0.5981","costPerSuccessUsd":"0.0854","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:38.785Z"},{"id":"eec1fb10-7c7c-47b5-97f5-20238a407b25","modelTested":"gemini-3.5-flash","successRate":"0.7000","costUsd":"0.3166","costPerSuccessUsd":"0.0452","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:38.698Z"},{"id":"0be491e2-4192-4193-ba22-7c43d722b08e","modelTested":"gemini-3.1-pro-preview","successRate":"0.5000","costUsd":"0.1351","costPerSuccessUsd":"0.0270","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:34.660Z"},{"id":"a299742e-7991-4cde-87c8-0701519d416b","modelTested":"gemini-3.1-flash-lite","successRate":"0.4000","costUsd":"0.0119","costPerSuccessUsd":"0.0030","completedTasks":10,"totalTasks":10,"finishedAt":"2026-08-11T22:49:31.161Z"},{"id":"d3aff915-d771-48eb-8a74-1072efe36b40","modelTested":"codai-alpha-1-0","successRate":"0.1000","costUsd":"0.1730","costPerSuccessUsd":"0.1730","completedTasks":10,"totalTasks":10,"finishedAt":"2026-06-16T00:18:18.231Z"}]},{"id":"acc3499a-9688-411f-aed3-b9be21c44466","slug":"mmlu-pro","name":"MMLU-Pro","description":"TIGER-Lab's 12,032-question reasoning benchmark across 14 domains (biology, business, chemistry, CS, economics, engineering, health, history, law, math, philosophy, physics, psychology, other). 10-choice multiple-choice, deterministic letter-match scoring. We run a 200-question stratified sample per category (28 per category, capped at the available count).","category":"reasoning","taskCount":12032,"referenceUrl":"https://arxiv.org/abs/2406.01574","resultKind":null,"runs":[{"id":"a2f5c831-318f-43ba-bbef-fdd3b4e64a77","modelTested":"codai,claude-opus-4-8","successRate":"0.8200","costUsd":"1.2471","costPerSuccessUsd":"0.0076","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-27T00:15:41.563Z"},{"id":"9343b36e-38a0-45d2-b2fb-d06e155065cf","modelTested":"codai","successRate":"0.8150","costUsd":"1.1059","costPerSuccessUsd":"0.0068","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-27T00:23:34.442Z"},{"id":"bf1d30cb-31c9-42ec-8754-74dd68d23c8a","modelTested":"claude-opus-4-8","successRate":"0.7900","costUsd":"1.5828","costPerSuccessUsd":"0.0100","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-27T00:24:20.059Z"},{"id":"eebd0f92-a99f-4f9d-8b9c-0bfc02eca00e","modelTested":"gpt-5-mini","successRate":"0.7550","costUsd":"2.0394","costPerSuccessUsd":"0.0135","completedTasks":200,"totalTasks":200,"finishedAt":"2026-05-27T08:50:50.606Z"},{"id":"a5cf256d-0c37-4df9-9903-93b9b2c8d007","modelTested":"claude-haiku-4-5","successRate":"0.6550","costUsd":"2.0248","costPerSuccessUsd":"0.0155","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-15T20:05:48.847Z"},{"id":"f094fb2d-6952-4821-a55b-6d5bbef22352","modelTested":"claude-opus-4-7","successRate":"0.6200","costUsd":"0.0855","costPerSuccessUsd":"0.0007","completedTasks":200,"totalTasks":200,"finishedAt":"2026-05-26T22:45:00.520Z"},{"id":"5af6dc16-c9ce-4921-ab58-f45c9233cdf1","modelTested":"claude-sonnet-4-7","successRate":"0.6150","costUsd":"0.0763","costPerSuccessUsd":"0.0006","completedTasks":200,"totalTasks":200,"finishedAt":"2026-05-26T22:44:38.655Z"},{"id":"f4933fb4-117f-4f3a-b39a-06be8a0e0c23","modelTested":"codai-alpha-1-0","successRate":"0.0250","costUsd":"7.3842","costPerSuccessUsd":"1.4768","completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-15T22:02:53.587Z"},{"id":"6c3a66fa-b31a-4af1-904c-f8a4b5db4e85","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":200,"totalTasks":12032,"finishedAt":"2026-06-15T20:01:14.708Z"}]},{"id":"28af3441-0e7e-4eda-a465-8b8c73463ee8","slug":"problem-solving","name":"Problem Solving (Reasoning)","description":"Frontier v9 #1-priority category. Single-shot HARD reasoning tasks that other harnesses miss: multi-step logic puzzles (balance-weighing, river/bridge crossings), quantitative word problems with intuitive-but-wrong traps, systematic root-cause/debugging strategy, constraint satisfaction, and Fermi estimation. NOT code-execution (humaneval/codai-swe own that) and NOT short-answer contest math (math-competition owns that) — rewards correct multi-step reasoning AND a sound, communicable method. Scored by a weighted-rubric LLM judge (correctness-critical criteria first); a task passes at >=0.7. Mirrors the training-side Rubrics-as-Rewards signal so the eval measures exactly what RLVR+RaR optimizes. Fixed, version-controlled tasks (problem-solving.ts) for reproducibility.","category":"reasoning","taskCount":8,"referenceUrl":"https://arxiv.org/abs/2206.04615","resultKind":null,"runs":[]}]}