{"name":"codai benchmarks — coding","scope":"coding","license":"https://creativecommons.org/licenses/by/4.0/","attribution":"CC BY 4.0, attribution: codai (https://codai.ro/benchmarks)","source":"https://codai.ro/benchmarks/coding","selection":"One run per benchmark and model: the completed run with the largest sample; ties go to the most recent finish.","units":{"successRate":"fraction 0..1","costUsd":"USD","costPerSuccessUsd":"USD"},"lastRefresh":"2026-09-27T00:00:00.000Z","benchmarks":[{"id":"2b71a3a5-530e-45bd-a965-b53c2f48d0f6","slug":"swe-bench-verified-official","name":"SWE-bench Verified (official harness)","description":"The 500-task SWE-bench Verified set scored by the official SWE-bench Docker harness (resolved = FAIL_TO_PASS and PASS_TO_PASS tests pass). codai agent, single attempt per task, run 2026-08-03/04. Competitor rows are published leaderboard entries (source and date in each row), not re-runs by us.","category":"coding","taskCount":500,"referenceUrl":"https://www.swebench.com/","resultKind":"official","runs":[{"id":"a8f579d8-8370-486d-8a61-0b9583ee4bc5","modelTested":"claude-sonnet-5","successRate":"0.8520","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-06-30T00:00:00.000Z"},{"id":"6f180153-a2af-41e1-806e-afc625f84d2a","modelTested":"20251205_sonar-foundation-agent_claude-opus-4-5","successRate":"0.7920","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.459Z"},{"id":"57d9b2cb-7937-44ff-99f1-6c4382ef0473","modelTested":"20251215_livesweagent_claude-opus-4-5","successRate":"0.7920","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.281Z"},{"id":"c7b6e52f-86ac-4d0e-8dd4-755f50a44688","modelTested":"20250928_trae_doubao_seed_code","successRate":"0.7880","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.036Z"},{"id":"564c8a09-6184-4fa9-9ded-24d1838c73ed","modelTested":"OpenHands + Claude Opus 4.5","successRate":"0.7760","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.634Z"},{"id":"c40d7eb0-5b35-4bbe-b49d-574647f368e7","modelTested":"20251120_livesweagent_gemini-3-pro-preview","successRate":"0.7740","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.809Z"},{"id":"bedb466d-2afa-4163-acf4-e234baf77f4d","modelTested":"Claude 4.5 Opus (bash-only)","successRate":"0.7680","costUsd":"375.0000","costPerSuccessUsd":"0.9766","completedTasks":500,"totalTasks":500,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"828b2e9b-2969-4a09-8f9f-6a5c42202735","modelTested":"20250804_epam-ai-run-claude-4-sonnet","successRate":"0.7680","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.611Z"},{"id":"80f55db9-725b-4415-8c7a-62c9744134dc","modelTested":"20250902_atlassian-rovo-dev","successRate":"0.7680","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.563Z"},{"id":"d746b437-62f0-4b14-af39-8739a9c61c7a","modelTested":"20250819_ACoder","successRate":"0.7640","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.263Z"},{"id":"b472916f-d183-4d83-9e2f-d00691fa62bf","modelTested":"MiniMax M2.5 (bash-only)","successRate":"0.7580","costUsd":"35.0000","costPerSuccessUsd":"0.0923","completedTasks":500,"totalTasks":500,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"4f755eb1-324e-4146-8807-4420606479d4","modelTested":"Gemini 3 Flash (bash-only)","successRate":"0.7580","costUsd":"180.0000","costPerSuccessUsd":"0.4749","completedTasks":500,"totalTasks":500,"finishedAt":"2026-09-27T00:00:00.000Z"},{"id":"863dce10-9c25-45c4-8006-9a133d07b2ab","modelTested":"20250901_warp","successRate":"0.7560","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.737Z"},{"id":"896d6265-716c-4fed-a267-4b02a9100fb7","modelTested":"20250731_harness_ai","successRate":"0.7480","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.960Z"},{"id":"988a5fd3-d6c5-4046-9d26-73c600fe4937","modelTested":"20251103_sonar-foundation-agent_claude-sonnet-4-5","successRate":"0.7480","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.336Z"},{"id":"565782ef-9011-4fe0-a5de-bf7630f94bd8","modelTested":"20250720_Lingxi-v1.5_claude-4-sonnet-20250514","successRate":"0.7460","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.310Z"},{"id":"d8e6aafa-9c4f-4d57-a62a-bda187dcb48f","modelTested":"20250915_JoyCode","successRate":"0.7460","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.386Z"},{"id":"96606277-e29c-45ed-aa6f-e115866464e7","modelTested":"20251103_SalesforceAIResearch_SAGE_OpenHands","successRate":"0.7380","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.511Z"},{"id":"481b7eb2-752b-4693-a6c3-d4c6138ce9b7","modelTested":"codai","successRate":"0.7320","costUsd":"304.7400","costPerSuccessUsd":"0.8326","completedTasks":500,"totalTasks":500,"finishedAt":"2026-08-04T12:00:00.000Z"},{"id":"708652d9-e00c-4359-a5ee-9c98888eb3aa","modelTested":"20251021_SalesforceAIResearch_SAGE_bash_only","successRate":"0.7300","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.686Z"},{"id":"8ed3c681-52fa-4dca-a371-f47df403be8e","modelTested":"20250807_openhands_gpt5","successRate":"0.7180","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.437Z"},{"id":"b7857b80-b63b-411a-8e8a-6c24cbb191c5","modelTested":"20250710_bloop","successRate":"0.7120","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.836Z"},{"id":"308c70f1-3775-455a-bf82-f4029fe8a56b","modelTested":"20250715_qodo_command","successRate":"0.7120","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.660Z"},{"id":"93135d38-7f8f-4c44-b270-555c3e480a29","modelTested":"20251014_Lingxi_kimi_k2","successRate":"0.7120","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.861Z"},{"id":"b0d62aa4-74f6-4f84-9674-35198868a498","modelTested":"20250623_warp","successRate":"0.7100","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.539Z"},{"id":"ca2735a3-5780-4fdd-86fd-83fe86ba2965","modelTested":"20250716_openhands_kimi_k2","successRate":"0.6540","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.486Z"},{"id":"f01b09a7-04a0-466a-b4ad-9e63156b202f","modelTested":"20250901_entroPO_R2E_QwenCoder30BA3B_tts","successRate":"0.6040","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.912Z"},{"id":"4f17cb75-50ba-48de-8137-b3f61a309b8a","modelTested":"20250629_deepswerl_r2eagent_tts","successRate":"0.5880","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.011Z"},{"id":"c4a7b45c-3a8f-48da-8d60-90a4adf8ae45","modelTested":"20250924_artemis_agent_v2","successRate":"0.5700","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:42.211Z"},{"id":"3aef01ad-cd17-47d7-af20-a2b460147d55","modelTested":"20251110_frogboss-32b","successRate":"0.5360","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:41.161Z"},{"id":"4088dddd-4109-4e9f-ac4e-cbf3221eaad7","modelTested":"20250804_codesweep_sweagent_kimi_k2_instruct","successRate":"0.5340","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.785Z"},{"id":"ccff5ba0-3084-494f-8f46-5bde406a6d66","modelTested":"20250901_entroPO_R2E_QwenCoder30BA3B","successRate":"0.5220","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:43.088Z"},{"id":"e6987d94-792a-4f25-b4f2-1ab4085edb3b","modelTested":"20250616_Skywork-SWE-32B+TTS_Bo8","successRate":"0.4700","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.714Z"},{"id":"32a2f738-f483-4bc4-ba0c-1c834a470b59","modelTested":"20251110_frogmini-14b","successRate":"0.4500","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:40.984Z"},{"id":"62fe1417-483c-4c50-850e-0c32c8360913","modelTested":"20250629_deepswerl_r2eagent","successRate":"0.4220","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.187Z"},{"id":"b1cba308-7520-4476-b86c-9aa7fad32ec7","modelTested":"20250616_Skywork-SWE-32B","successRate":"0.3800","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.889Z"},{"id":"e7852541-eb51-4716-95ff-75f5c0f53059","modelTested":"20250725_sweagent_devstral_small_2507","successRate":"0.3800","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:44.135Z"},{"id":"5f20d252-5cc8-4093-8b12-7ac47e177222","modelTested":"20250627_agentless_MCTS-Refine-7B","successRate":"0.2320","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-05-27T00:40:45.362Z"}]},{"id":"cfbbc993-6921-43b7-a7c0-29b4479e8bb1","slug":"aider-polyglot","name":"Aider Polyglot","description":"225-task multi-language coding benchmark (Python, JS, TS, Rust, Go, Java, C++). Anthropic and OpenAI both quote this for editing performance. Our harness runs the PYTHON subset (~38 tasks) — the largest single bucket — with the official two-pass methodology (one initial try + one retry given test output). Cross-language coverage is the next iteration; competitor scores below are full 225-task polyglot.","category":"coding","taskCount":225,"referenceUrl":"https://aider.chat/docs/leaderboards/","resultKind":null,"runs":[{"id":"003f1e8a-748d-4590-993f-4c3d398063fe","modelTested":"gpt-5 (high)","successRate":"0.8800","costUsd":"29.0829","costPerSuccessUsd":"0.1469","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.237Z"},{"id":"6472dcdd-74be-4a87-91d3-119b035b4cf3","modelTested":"gpt-5 (medium)","successRate":"0.8670","costUsd":"17.6930","costPerSuccessUsd":"0.0907","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.399Z"},{"id":"c52ac3ad-3aa4-45ed-aa9c-dfdcc95d98ce","modelTested":"claude-opus-4-7","successRate":"0.8529","costUsd":"0.9271","costPerSuccessUsd":"0.0320","completedTasks":34,"totalTasks":225,"finishedAt":"2026-05-27T03:57:03.844Z"},{"id":"a2f9292a-5cd5-42fc-b917-825d2f3ccdc7","modelTested":"o3-pro (high)","successRate":"0.8490","costUsd":"146.3249","costPerSuccessUsd":"0.7661","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.588Z"},{"id":"996f6875-300b-49b6-af76-b7a121f34164","modelTested":"gemini-2.5-pro-preview-06-05 (32k think)","successRate":"0.8310","costUsd":"49.8822","costPerSuccessUsd":"0.2667","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.773Z"},{"id":"e3e0a374-055c-4f15-8240-e7d557a3f2ca","modelTested":"gpt-5 (low)","successRate":"0.8130","costUsd":"10.3713","costPerSuccessUsd":"0.0567","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.562Z"},{"id":"bfa6bf22-2bc2-4038-baaa-652ff1a0eed4","modelTested":"o3 (high)","successRate":"0.8130","costUsd":"21.2259","costPerSuccessUsd":"0.1160","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.099Z"},{"id":"853fb117-c6d4-40b3-813b-7f49c61ce74d","modelTested":"grok-4 (high)","successRate":"0.7960","costUsd":"59.6182","costPerSuccessUsd":"0.3331","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.751Z"},{"id":"9331458b-34b8-4e8e-8fee-94e293f1d704","modelTested":"gemini-2.5-pro-preview-06-05 (default think)","successRate":"0.7910","costUsd":"45.5961","costPerSuccessUsd":"0.2562","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.610Z"},{"id":"5f553101-f34a-4ddb-b486-d20287156176","modelTested":"o3 (high) + gpt-4.1","successRate":"0.7820","costUsd":"17.5518","costPerSuccessUsd":"0.0997","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:18.425Z"},{"id":"d287a36c-d352-4104-a5cf-488d240ccae8","modelTested":"o3","successRate":"0.7690","costUsd":"13.7517","costPerSuccessUsd":"0.0795","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.262Z"},{"id":"77264ea2-e928-4536-a231-92a4067dc097","modelTested":"Gemini 2.5 Pro Preview 05-06","successRate":"0.7690","costUsd":"37.4104","costPerSuccessUsd":"0.2162","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.147Z"},{"id":"1aa17896-5cac-4284-9eab-ded913e6a6a8","modelTested":"DeepSeek-V3.2-Exp (Reasoner)","successRate":"0.7420","costUsd":"1.3045","costPerSuccessUsd":"0.0078","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.724Z"},{"id":"e3ab9d63-03be-455a-b83f-e7cf5640c5c6","modelTested":"claude-sonnet-4-6","successRate":"0.7353","costUsd":"0.6421","costPerSuccessUsd":"0.0257","completedTasks":34,"totalTasks":225,"finishedAt":"2026-05-27T03:56:24.165Z"},{"id":"d1446beb-c88d-44a2-8000-1c526b9a115e","modelTested":"Gemini 2.5 Pro Preview 03-25","successRate":"0.7290","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.877Z"},{"id":"4aaab394-1cb8-4b01-ab63-65d3edb1729f","modelTested":"claude-opus-4-20250514 (32k thinking)","successRate":"0.7200","costUsd":"65.7484","costPerSuccessUsd":"0.4059","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.121Z"},{"id":"c800d0ae-aeea-460e-944d-16558162163d","modelTested":"o4-mini (high)","successRate":"0.7200","costUsd":"19.6399","costPerSuccessUsd":"0.1212","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.658Z"},{"id":"d66df734-a62a-4143-8244-b92a2245fd61","modelTested":"DeepSeek R1 (0528)","successRate":"0.7140","costUsd":"4.8016","costPerSuccessUsd":"0.0300","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:17.936Z"},{"id":"0ac3ffea-cc8b-4adc-b882-f4c6f82580b7","modelTested":"claude-opus-4-20250514 (no think)","successRate":"0.7070","costUsd":"68.6253","costPerSuccessUsd":"0.4316","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.959Z"},{"id":"32e73ed8-c162-4e75-babc-32c52d3fbf48","modelTested":"claude-opus-4-8","successRate":"0.7059","costUsd":"13.4471","costPerSuccessUsd":"0.5603","completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-27T00:28:29.838Z"},{"id":"ee0c8dd3-d11d-413b-8557-e9862b4f260b","modelTested":"DeepSeek-V3.2-Exp (Chat)","successRate":"0.7020","costUsd":"0.8756","costPerSuccessUsd":"0.0055","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.886Z"},{"id":"04b9b69d-3025-444d-95ae-8d9ad62b8023","modelTested":"claude-3-7-sonnet-20250219 (32k thinking tokens)","successRate":"0.6490","costUsd":"36.8343","costPerSuccessUsd":"0.2523","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:12.745Z"},{"id":"37f09e02-bcaa-45b4-80cd-b61d962cef33","modelTested":"DeepSeek R1 + claude-3-5-sonnet-20241022","successRate":"0.6400","costUsd":"13.2933","costPerSuccessUsd":"0.0923","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.620Z"},{"id":"bc5919d1-ba7a-423c-b494-755849a512c7","modelTested":"codai","successRate":"0.6176","costUsd":"13.7060","costPerSuccessUsd":"0.6527","completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-27T00:26:20.285Z"},{"id":"f6759f54-1152-47fc-9811-e0dad22aaefc","modelTested":"o1-2024-12-17 (high)","successRate":"0.6170","costUsd":"186.4958","costPerSuccessUsd":"1.3417","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:09.849Z"},{"id":"07244ff9-20a3-4153-9048-3bd6f5add3cf","modelTested":"claude-sonnet-4-20250514 (32k thinking)","successRate":"0.6130","costUsd":"26.5755","costPerSuccessUsd":"0.1926","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.797Z"},{"id":"0d07b234-263a-44a0-a18e-29e55ca6f079","modelTested":"claude-3-7-sonnet-20250219 (no thinking)","successRate":"0.6040","costUsd":"17.7191","costPerSuccessUsd":"0.1303","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:12.584Z"},{"id":"b6888788-ffb4-4492-a841-9e28a32c59ee","modelTested":"o3-mini (high)","successRate":"0.6040","costUsd":"18.1584","costPerSuccessUsd":"0.1335","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:12.102Z"},{"id":"9f0dd111-d245-4fab-a679-2a9b8b657461","modelTested":"Qwen3 235B A22B diff, no think, Alibaba API","successRate":"0.5960","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.472Z"},{"id":"3d485443-6537-4216-a6cc-b0126eeb9c65","modelTested":"Kimi K2","successRate":"0.5910","costUsd":"1.2357","costPerSuccessUsd":"0.0093","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:18.912Z"},{"id":"4f962bd1-d0de-4b4e-87d2-a4be09d18c63","modelTested":"claude-haiku-4-5","successRate":"0.5882","costUsd":"2.1241","costPerSuccessUsd":"0.1062","completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-15T20:07:59.828Z"},{"id":"a986165a-8e3c-4438-bbd0-2d192f8d56c3","modelTested":"gpt-5-mini","successRate":"0.5882","costUsd":"4.1429","costPerSuccessUsd":"0.2071","completedTasks":34,"totalTasks":225,"finishedAt":"2026-05-27T09:08:43.162Z"},{"id":"9001ecc5-a0e2-4b1b-acde-6db67fef6a17","modelTested":"DeepSeek R1","successRate":"0.5690","costUsd":"5.4193","costPerSuccessUsd":"0.0423","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.459Z"},{"id":"15fee063-cc52-428b-9880-59b0484e4814","modelTested":"claude-sonnet-4-20250514 (no thinking)","successRate":"0.5640","costUsd":"15.8155","costPerSuccessUsd":"0.1245","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.635Z"},{"id":"f060b6d3-63f9-485a-9843-a8fa116c71d2","modelTested":"gemini-2.5-flash-preview-05-20 (24k think)","successRate":"0.5510","costUsd":"8.5625","costPerSuccessUsd":"0.0691","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.447Z"},{"id":"da91e1b7-72bc-40ca-911c-c4e4f2324074","modelTested":"DeepSeek V3 (0324)","successRate":"0.5510","costUsd":"1.1164","costPerSuccessUsd":"0.0090","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.714Z"},{"id":"a6b18e87-bd2f-4f48-8751-a367b4c5ecc9","modelTested":"Quasar Alpha","successRate":"0.5470","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.204Z"},{"id":"e777659a-b1a1-4529-8f0f-d28d1c7e7b05","modelTested":"o3-mini (medium)","successRate":"0.5380","costUsd":"8.8599","costPerSuccessUsd":"0.0732","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.942Z"},{"id":"52ce8c2a-c066-4862-85fa-ce5c38c1deb4","modelTested":"Grok 3 Beta","successRate":"0.5330","costUsd":"11.0338","costPerSuccessUsd":"0.0919","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.530Z"},{"id":"432c413d-d510-4805-bd8b-692b974d0ff5","modelTested":"Optimus Alpha","successRate":"0.5290","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.014Z"},{"id":"f8abf842-e801-4b4c-8143-4bd7358e8d02","modelTested":"gpt-4.1","successRate":"0.5240","costUsd":"9.8556","costPerSuccessUsd":"0.0835","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.175Z"},{"id":"4d522efc-1b5d-4347-8dfd-7f44b8ae0c9b","modelTested":"claude-3-5-sonnet-20241022","successRate":"0.5160","costUsd":"14.4063","costPerSuccessUsd":"0.1242","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.365Z"},{"id":"135fe35b-0451-45f0-b74d-6aa7d01a505d","modelTested":"Grok 3 Mini Beta (high)","successRate":"0.4930","costUsd":"0.7346","costPerSuccessUsd":"0.0066","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.853Z"},{"id":"476ae89d-0c00-4a33-bc04-8021ad96ffd5","modelTested":"DeepSeek Chat V3 (prev)","successRate":"0.4840","costUsd":"0.3369","costPerSuccessUsd":"0.0031","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.137Z"},{"id":"4abf44bf-e020-4b3c-9e4a-4a55292d52ca","modelTested":"gemini-2.5-flash-preview-04-17 (default)","successRate":"0.4710","costUsd":"1.8451","costPerSuccessUsd":"0.0174","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.983Z"},{"id":"2863ce7d-4b0d-440b-9e2f-0eeeb6dcdde5","modelTested":"chatgpt-4o-latest (2025-03-29)","successRate":"0.4530","costUsd":"19.7416","costPerSuccessUsd":"0.1935","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.040Z"},{"id":"ef04c8eb-4c52-448d-b215-f0345ed27d2d","modelTested":"gpt-4.5-preview","successRate":"0.4490","costUsd":"183.1802","costPerSuccessUsd":"1.8137","completedTasks":224,"totalTasks":224,"finishedAt":"2026-05-27T00:40:12.905Z"},{"id":"45e4ed40-77de-4fbb-8858-92b12116384f","modelTested":"gemini-2.5-flash-preview-05-20 (no think)","successRate":"0.4400","costUsd":"1.1354","costPerSuccessUsd":"0.0115","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:17.284Z"},{"id":"b7e273c5-82e6-43e8-9f2c-98bbb295beba","modelTested":"gpt-oss-120b (high)","successRate":"0.4180","costUsd":"0.7406","costPerSuccessUsd":"0.0079","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:19.073Z"},{"id":"32d1ba14-5d06-44ae-969f-0a70fbc90ae9","modelTested":"Qwen3 32B","successRate":"0.4000","costUsd":"0.7603","costPerSuccessUsd":"0.0084","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:16.309Z"},{"id":"29d8761e-e386-4ed9-84f5-19002883c23a","modelTested":"gemini-3.1-flash-lite","successRate":"0.3824","costUsd":"0.0547","costPerSuccessUsd":"0.0042","completedTasks":34,"totalTasks":225,"finishedAt":"2026-08-11T22:46:44.115Z"},{"id":"da6ea955-af1b-416c-b5b9-45d908f91136","modelTested":"gemini-exp-1206","successRate":"0.3820","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.657Z"},{"id":"6738a76c-511e-4adc-a760-f33a11589666","modelTested":"Gemini 2.0 Pro exp-02-05","successRate":"0.3560","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.039Z"},{"id":"d4c9f662-c477-47d4-b46b-7bdb502c2541","modelTested":"Grok 3 Mini Beta (low)","successRate":"0.3470","costUsd":"0.7856","costPerSuccessUsd":"0.0101","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.692Z"},{"id":"8c9cd1ea-1a6f-422e-81fe-cde8d19d2a5a","modelTested":"o1-mini-2024-09-12","successRate":"0.3290","costUsd":"18.5770","costPerSuccessUsd":"0.2510","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.497Z"},{"id":"c6e695da-2c64-4493-80fe-753e40eb9ccc","modelTested":"gpt-4.1-mini","successRate":"0.3240","costUsd":"1.9918","costPerSuccessUsd":"0.0273","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.336Z"},{"id":"632f3418-c56c-46ae-ad7b-b1537b6b9b48","modelTested":"gemini-3.5-flash-lite","successRate":"0.3235","costUsd":"0.4514","costPerSuccessUsd":"0.0410","completedTasks":34,"totalTasks":225,"finishedAt":"2026-08-11T22:48:26.953Z"},{"id":"4cdcbf8b-9de0-49cb-8c09-fe6b87c38458","modelTested":"claude-3-5-haiku-20241022","successRate":"0.2800","costUsd":"6.0583","costPerSuccessUsd":"0.0962","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.173Z"},{"id":"432b56f6-30bb-450a-a15a-1fba46d18eeb","modelTested":"chatgpt-4o-latest (2025-02-15)","successRate":"0.2710","costUsd":"14.3703","costPerSuccessUsd":"0.2356","completedTasks":223,"totalTasks":223,"finishedAt":"2026-05-27T00:40:12.423Z"},{"id":"d3f32050-e012-4bf9-8ceb-2d783940dcf4","modelTested":"QwQ-32B + Qwen 2.5 Coder Instruct","successRate":"0.2620","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.226Z"},{"id":"be486b63-b5cf-4177-8f66-a81d93112ded","modelTested":"gpt-4o-2024-08-06","successRate":"0.2310","costUsd":"7.0286","costPerSuccessUsd":"0.1352","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.688Z"},{"id":"bc2c3fc0-7ca6-4994-b924-ba4b58f82d6d","modelTested":"gemini-2.0-flash-exp","successRate":"0.2220","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.817Z"},{"id":"dda00784-fdae-4903-9f0f-484ef0f195b0","modelTested":"qwen-max-2025-01-25","successRate":"0.2180","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.781Z"},{"id":"bb3b67f9-e83c-438e-a2c5-a10fdb47b1ea","modelTested":"QwQ-32B","successRate":"0.2090","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.065Z"},{"id":"90596882-142d-4b8c-bd3c-24d11aee6ece","modelTested":"gemini-2.0-flash-thinking-exp-01-21","successRate":"0.1820","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:12.262Z"},{"id":"cab5b341-40aa-498d-8be0-57101d92d315","modelTested":"gpt-4o-2024-11-20","successRate":"0.1820","costUsd":"6.7351","costPerSuccessUsd":"0.1643","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.526Z"},{"id":"3fbc65e3-ef15-4a70-80b4-1cbf66686cbb","modelTested":"DeepSeek Chat V2.5","successRate":"0.1780","costUsd":"0.5101","costPerSuccessUsd":"0.0128","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.012Z"},{"id":"2da9bb50-b74f-4539-870a-c75277998698","modelTested":"Qwen2.5-Coder-32B-Instruct","successRate":"0.1640","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.335Z"},{"id":"d02fb285-c434-4333-a91a-b495e27c9024","modelTested":"Llama 4 Maverick","successRate":"0.1560","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:14.367Z"},{"id":"9ce3cd54-732f-45ff-af0f-f780611d5a51","modelTested":"yi-lightning","successRate":"0.1290","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:10.978Z"},{"id":"acd6978d-f59d-4b6c-b3b1-90ebb77d7ae0","modelTested":"command-a-03-2025-quality","successRate":"0.1200","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.388Z"},{"id":"6f30d1d0-5722-4841-b7d2-e4b04ca44c2d","modelTested":"Codestral 25.01","successRate":"0.1110","costUsd":"1.9834","costPerSuccessUsd":"0.0793","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:11.298Z"},{"id":"a60923d1-06ae-40f9-8bc6-c93d95488f94","modelTested":"openhands-lm-32b-v0.1","successRate":"0.1020","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.820Z"},{"id":"77670eed-09ef-4b30-9572-577441749b75","modelTested":"gpt-4.1-nano","successRate":"0.0890","costUsd":"0.4281","costPerSuccessUsd":"0.0214","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:15.497Z"},{"id":"aadd630b-d1b3-4df9-b7ef-49a639840778","modelTested":"gemini-3.6-flash","successRate":"0.0588","costUsd":"0.3091","costPerSuccessUsd":"0.1545","completedTasks":34,"totalTasks":225,"finishedAt":"2026-08-11T22:52:41.349Z"},{"id":"56fa48ab-7c87-4c09-9cc7-945093c348e0","modelTested":"gemma-3-27b-it","successRate":"0.0490","costUsd":null,"costPerSuccessUsd":null,"completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:13.552Z"},{"id":"ca728bcc-c0f5-401d-a6fb-8a42dac86d04","modelTested":"gpt-4o-mini-2024-07-18","successRate":"0.0360","costUsd":"0.3236","costPerSuccessUsd":"0.0404","completedTasks":225,"totalTasks":225,"finishedAt":"2026-05-27T00:40:09.202Z"},{"id":"8d242936-9404-4a7d-84ab-fcf1127f6aa8","modelTested":"codai-alpha-1-0","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-15T21:35:30.002Z"},{"id":"d216b61f-84b8-4d8c-8251-19e14c202027","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":34,"totalTasks":225,"finishedAt":"2026-06-15T20:01:56.351Z"}]},{"id":"cb0d4d49-baba-4d41-8341-b9d036e6d017","slug":"humaneval","name":"HumanEval","description":"OpenAI 164-problem Python coding benchmark. Single-turn: prompt → function body → run unit tests. Cheap, fast, well-known — used as the pipeline smoke benchmark.","category":"coding","taskCount":164,"referenceUrl":"https://arxiv.org/abs/2107.03374","resultKind":null,"runs":[{"id":"b493f430-1d9b-4c16-af11-d6475f249191","modelTested":"codai,claude-opus-4-8","successRate":"0.9695","costUsd":"1.7589","costPerSuccessUsd":"0.0111","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-27T00:16:09.456Z"},{"id":"06ba910b-395b-4f83-903b-b7bdcecd69b6","modelTested":"claude-opus-4-8","successRate":"0.9634","costUsd":"2.0007","costPerSuccessUsd":"0.0127","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-27T00:25:18.842Z"},{"id":"f1accbb5-cf4e-4d08-bda4-4f21cb37a189","modelTested":"codai","successRate":"0.9634","costUsd":"1.4539","costPerSuccessUsd":"0.0092","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-27T00:23:47.612Z"},{"id":"593ba55d-1320-48e2-a50f-6dea4f2bb7c4","modelTested":"claude-haiku-4-5","successRate":"0.9512","costUsd":"1.8172","costPerSuccessUsd":"0.0116","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-15T20:08:16.381Z"},{"id":"67fdf837-3078-46a5-8b9d-481cf343c7df","modelTested":"claude-sonnet-4-7","successRate":"0.9451","costUsd":"0.2064","costPerSuccessUsd":"0.0013","completedTasks":164,"totalTasks":164,"finishedAt":"2026-05-26T22:29:04.148Z"},{"id":"1bab34c3-cdd9-45aa-948b-4fff8d5a067e","modelTested":"claude-opus-4-7","successRate":"0.9390","costUsd":"0.2070","costPerSuccessUsd":"0.0013","completedTasks":164,"totalTasks":164,"finishedAt":"2026-05-26T22:29:48.310Z"},{"id":"6084a69a-db33-4561-aed5-baaf1ec3b0a6","modelTested":"gpt-5-mini","successRate":"0.8232","costUsd":"2.6352","costPerSuccessUsd":"0.0195","completedTasks":164,"totalTasks":164,"finishedAt":"2026-05-27T08:44:47.556Z"},{"id":"24c64877-f736-4000-b1c7-5b3dc57cc3be","modelTested":"codai-alpha-1-0","successRate":"0.4207","costUsd":"20.9714","costPerSuccessUsd":"0.3039","completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-16T01:09:51.641Z"},{"id":"222473eb-760f-44d1-8408-f7ce4defdbff","modelTested":"gemini-3.6-flash","successRate":"0.1402","costUsd":"0.2059","costPerSuccessUsd":"0.0090","completedTasks":164,"totalTasks":164,"finishedAt":"2026-08-11T22:46:46.586Z"},{"id":"fc4efea3-0ce2-42fc-be44-518cac32c422","modelTested":"gemini-3.5-flash","successRate":"0.1220","costUsd":"0.1937","costPerSuccessUsd":"0.0097","completedTasks":164,"totalTasks":164,"finishedAt":"2026-08-11T22:47:19.378Z"},{"id":"91335538-7541-4230-b33f-17cd11eab184","modelTested":"claude-fable-5","successRate":"0.0000","costUsd":"0.0000","costPerSuccessUsd":null,"completedTasks":164,"totalTasks":164,"finishedAt":"2026-06-15T19:58:42.839Z"}]},{"id":"d238d745-eb9b-40da-aad9-dab1de5a3601","slug":"swe-bench-verified","name":"SWE-bench Verified (oracle-diff proxy, 100)","description":"PROXY, not the official harness: 100 SWE-bench Verified issues, the model gets the issue plus the oracle affected files and outputs a unified diff, scored by line-overlap similarity to the gold patch plus an LLM judge on borderline cases. No tests are executed. Not comparable to leaderboard numbers; see \"SWE-bench Verified (official harness)\" for the real score.","category":"coding","taskCount":100,"referenceUrl":"https://www.swebench.com/","resultKind":"proxy","runs":[{"id":"3d3c30b0-b992-4fec-9913-3d68ec856074","modelTested":"claude-sonnet-5","successRate":"0.8520","costUsd":null,"costPerSuccessUsd":null,"completedTasks":500,"totalTasks":500,"finishedAt":"2026-06-30T00:00:00.000Z"},{"id":"bffd13a9-d0f7-4e51-9bf2-27ccbfb46741","modelTested":"claude-fable-5","successRate":"0.6500","costUsd":"38.7123","costPerSuccessUsd":"0.5956","completedTasks":100,"totalTasks":100,"finishedAt":"2026-07-11T01:57:50.991Z"},{"id":"d3607c3c-98e1-4fc2-b79f-9201b0065258","modelTested":"codai","successRate":"0.4600","costUsd":"33.9823","costPerSuccessUsd":"0.7387","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-27T01:12:34.222Z"},{"id":"a41fbb0e-5be0-4876-abc4-b18a916d9a6a","modelTested":"gemini-3.5-flash","successRate":"0.0100","costUsd":"2.1185","costPerSuccessUsd":"2.1185","completedTasks":100,"totalTasks":100,"finishedAt":"2026-08-11T22:54:57.113Z"},{"id":"7d3625b3-5bc9-4b96-b7da-55f950aca2f8","modelTested":"gemini-3.1-flash-lite","successRate":"0.0000","costUsd":"0.2813","costPerSuccessUsd":null,"completedTasks":100,"totalTasks":100,"finishedAt":"2026-08-11T22:52:30.683Z"},{"id":"609eb8ce-5e67-44f4-84cb-a163ce78c2e9","modelTested":"gemini-3.5-flash-lite","successRate":"0.0000","costUsd":"0.3713","costPerSuccessUsd":null,"completedTasks":100,"totalTasks":100,"finishedAt":"2026-08-11T22:52:09.771Z"}]},{"id":"0b9e454e-48e1-494c-9188-a61f9950692e","slug":"swe-bench-patch","name":"SWE-bench Verified (patch-only)","description":"Patch-only variant of SWE-bench Verified. Model receives the issue + the 'oracle' affected files (princeton-nlp/SWE-bench_oracle pre-formatted text) and must output a unified diff that fixes the issue. NO Docker, NO test execution — we score the candidate diff against the gold patch with line-overlap similarity (cheap fast-path) plus an LLM judge (sonnet-4-6) for borderline cases (0.3 < sim < 0.7). Not directly comparable to the official leaderboard but a strong proxy for diff-generation quality at 1% of the cost.","category":"coding","taskCount":100,"referenceUrl":"https://www.swebench.com/","resultKind":"proxy","runs":[{"id":"d89b6833-2b69-4052-bab6-c47a9a3428c4","modelTested":"codai","successRate":"0.4700","costUsd":"126.5339","costPerSuccessUsd":"2.6922","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T13:43:00.657Z"},{"id":"2115e64a-d849-430f-bbe5-cb95e54621d4","modelTested":"claude-opus-4-7","successRate":"0.4600","costUsd":"127.0859","costPerSuccessUsd":"2.7627","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T13:43:17.456Z"},{"id":"cd66c208-b3ba-4456-8eaf-3d3f63c2b373","modelTested":"claude-opus-4-8","successRate":"0.3600","costUsd":"36.5610","costPerSuccessUsd":"1.0156","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-29T00:09:26.774Z"},{"id":"2f2c4dbc-2664-4dea-a121-e0e9122df9f9","modelTested":"claude-haiku-4-5","successRate":"0.3500","costUsd":"9.3787","costPerSuccessUsd":"0.2680","completedTasks":100,"totalTasks":100,"finishedAt":"2026-06-15T20:49:43.045Z"},{"id":"1a7ab46b-5c36-4787-bcda-e34eb2202a33","modelTested":"gpt-5-mini","successRate":"0.3100","costUsd":"128.6182","costPerSuccessUsd":"4.1490","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T13:45:17.194Z"},{"id":"a2d83174-f487-4316-ae7a-fc8bafa79488","modelTested":"claude-sonnet-4-6","successRate":"0.2900","costUsd":"24.5090","costPerSuccessUsd":"0.8451","completedTasks":100,"totalTasks":100,"finishedAt":"2026-05-27T15:31:46.516Z"}]}]}