{ "benchmark": "DeepSWE 1.1", "description": "Pass@1 on Datacurve DeepSWE 1.1 (113 long-horizon engineering tasks), from the official leaderboard, lab reports and independent runs.", "generated_from": [ "data/models.csv", "data/deepswe-1.1.csv", "data/meta.json" ], "meta": { "as_of": "2026-10-09", "official_url": "https://deepswe.datacurve.ai/", "official_last_added": "2026-09-03", "official_last_added_basis": "Datacurve changelog: GPT-6 Astra (all efforts) added 2026-09-03; its trials finished 2026-09-01. The 2026-09-22 generated_at is a site re-export with no new runs.", "notes": [ "The official board has added no model since 3 Sep 2026 (GPT-6 Astra; its runs finished on 1 Sep).{{datacurve-changelog}} Its data file was regenerated on 22 Sep with no new runs.{{datacurve-artifact}} A GitHub issue on the benchmark's repository quotes Datacurve's CEO saying the team is building the next version.{{gh-issue-103}}", "Epoch AI's review of 7 Sep 2026 rated DeepSWE 1.1 'flawed', finding grading defects in 23 of 113 tasks, mostly hidden tests colliding with tests the agent wrote.{{epoch-review}} Tokenless separately documented 70 ways a submission can rewrite test outcomes.{{tokenless-envcheck}} Treat differences of a few points with care.", "Lab claims are run by the lab, on its own harness, effort settings and trial count, so they are not directly comparable with the official runs (mini-swe-agent, four passes over 113 tasks).{{datacurve-run}} The harness is listed on every row. xAI says Datacurve ran the Grok 4.7 evaluation for it, though the result never appeared on the public board.{{url:https://media.x.ai/v1/website/card4p7-3a96f40b.pdf}}", "Independent runs come from Mercor (mini-swe-agent, 500 steps, 2-hour limit, 3 passes),{{url:https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/}} Artificial Analysis (Grok Build harness),{{url:https://artificialanalysis.ai/articles/benchmarking-grok-4-7}} Fireworks (Kimi K3 at three effort levels){{url:https://fireworks.ai/blog/ember-1}} and entrpi (MiniMax M3).{{url:https://entrpi.github.io/misc/deep-swe-minimax-m3/}}", "Costs are per task in USD. The official page and its data file disagree for 19 of 70 configurations; this tracker shows the page's figure and keeps the file's in the row notes.{{datacurve-artifact}} Lab costs are the lab's own, shown only where the lab published them.", "Mercor shows no date for each model, so a model whose only result is Mercor’s appears only when the release slider is at any time.{{url:https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/}}", "A number is included only when it was read at the source that published it. Figures that appear only on aggregator sites are left out; research/ingest-log.txt in the repository lists each one and why.", "Anthropic has published no DeepSWE 1.1 score for Claude Haiku 5.5;{{anthropic-haiku-55}} Mercor's independent run is the only one. There is no GPT-6.1 Luna: OpenAI released GPT-6.1 Sol only.{{url:https://openai.com/index/introducing-gpt-6-1-sol/}}", "Where a source published token counts but no cost, the cost is estimated from Vercel AI Gateway’s list prices and shown as a range marked est.{{ai-gateway-pricing}} With only a total token count, the range runs from every token priced as input to every token priced as output; cached input is cheaper still, so the true cost may sit below it. No reading qualifies today: Poolside’s token counts for Laguna S 2.1 are output tokens only, and an agent’s cost is mostly its input.", "DeepSeek re-released V4 Pro (as 0813) and V4 Flash (as 0731) under the same names; the official board’s live rows are the re-releases, and the earlier previews are listed as separate models.{{url:https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813}} Gemini 3.1 Pro was only ever released as a preview, so Mercor’s “Gemini 3.1 Pro” is the same model as the board’s." ], "official_generated_at": "2026-09-22T06:27:15.860279+00:00", "official_latest_job": { "name": "20260901-deep-swe-1-1-gpt-6-astra", "finished_at": "2026-09-01T07:35:13Z" }, "official_configs": 70, "official_checked": "2026-10-09", "official_rows_from": "page", "context_sources": [ { "id": "datacurve-changelog", "title": "DeepSWE changelog", "publisher": "Datacurve", "type": "Official changelog", "url": "https://deepswe.datacurve.ai/changelog", "published": "2026-09-03", "accessed": "2026-10-09", "establishes": "Lists each model addition by date; the most recent is GPT-6 Astra (all efforts) on 3 Sep 2026." }, { "id": "datacurve-run", "title": "Run DeepSWE", "publisher": "Datacurve", "type": "Official documentation", "url": "https://deepswe.datacurve.ai/run", "published": "2026-06-15", "accessed": "2026-10-09", "establishes": "Official scores are produced with Pier running mini-swe-agent on Modal, in isolated containers." }, { "id": "datacurve-artifact", "title": "leaderboard-live.json (v1.1 data file)", "publisher": "Datacurve", "type": "Official data file", "url": "https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json", "published": "2026-09-22", "accessed": "2026-10-09", "establishes": "The board's data file: generated 22 Sep 2026, latest job finished 1 Sep; its costs differ from the page's for 19 of 70 configurations." }, { "id": "epoch-review", "title": "Benchmark review: DeepSWE v1.1", "publisher": "Epoch AI", "type": "Independent audit", "url": "https://epoch.ai/benchmarks/deepswe/review", "published": "2026-09-07", "accessed": "2026-10-09", "establishes": "Rates DeepSWE v1.1 flawed: grading defects in at least 23 of 113 tasks, most from hidden tests colliding with tests the agent wrote." }, { "id": "tokenless-envcheck", "title": "envcheck: auditing DeepSWE", "publisher": "Tokenless", "type": "Independent audit", "url": "https://usetokenless.com/blog/envcheck", "published": "2026-09-30", "accessed": "2026-10-09", "establishes": "Documents 70 ways a submission can rewrite test outcomes, because the test harness sits inside the repository the agent edits." }, { "id": "gh-issue-103", "title": "datacurve-ai/deep-swe issue #103", "publisher": "GitHub (community discussion)", "type": "Discussion", "url": "https://github.com/datacurve-ai/deep-swe/issues/103", "published": "2026-09-30", "accessed": "2026-10-09", "establishes": "Notes the board has published no new model since 3 Sep and quotes Datacurve's CEO saying the team is building the next generation of DeepSWE." }, { "id": "anthropic-haiku-55", "title": "Claude Haiku 5.5 System Card", "publisher": "Anthropic", "type": "Lab system card", "url": "https://www-cdn.anthropic.com/e1080d6bf5ae2018ea3c2f414064be03232f5be5/Claude%20Haiku%205.5%20System%20Card.pdf", "published": "2026-09-29", "accessed": "2026-10-09", "establishes": "Reports SWE-bench Pro, FrontierSWE v2 and ProgramBench for Haiku 5.5, and no DeepSWE 1.1 result." }, { "id": "ai-gateway-pricing", "title": "AI Gateway model list and prices", "publisher": "Vercel", "type": "Price list", "url": "https://ai-gateway.vercel.sh/v1/models", "published": "", "accessed": "2026-10-09", "establishes": "Per-token list prices used to estimate cost where a source published tokens but no cost: Laguna S 2.1 at $0.09 per million input tokens and $0.18 per million output tokens." } ] }, "source_types": { "official_leaderboard": "Official Datacurve leaderboard", "lab_self_reported": "Reported by the model’s lab", "third_party_run": "Independent third-party run" }, "models": 80, "observations": 215, "sources": [ { "id": "r-16wwn5l", "url": "https://deepswe.datacurve.ai/", "title": "Datacurve DeepSWE 1.1 leaderboard", "publisher": "Datacurve", "type": "Official leaderboard", "published": "2026-06-15", "accessed": "2026-10-09", "establishes": "Pass@1 for GPT-6 Astra 67%–74.1% across 5 settings; Gemini 3.8 Flash 71%–73.8% across 2 settings; Claude Opus 5 58.1%–73.7% across 5 settings; GPT-5.6 Sol 45.4%–72.7% across 5 settings; Claude Fable 5 59.6%–69.9% across 5 settings; GPT-5.6 Terra 24.1%–69.6% across 5 settings; and 22 more models, with cost per task, tokens, confidence intervals.", "readings": 70, "n": 1 }, { "id": "r-1fnts5d", "url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "title": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "publisher": "Internet Archive (capture of the Datacurve board)", "type": "Official leaderboard", "published": "2026-06-15", "accessed": "2026-10-09", "establishes": "Pass@1 for Claude Opus 4.7 31.6%–54.2% across 4 settings; Gemini 3.5 Flash 28.3% [medium]; Claude Opus 4.6 27.6% [max]; GPT-5.4 mini 24.3% [xhigh]; Kimi K2.6 23.9%; MiniMax M3 20.4%; and 10 more models, with cost per task, tokens, confidence intervals.", "readings": 19, "n": 2 }, { "id": "r-glshid", "title": "DeepSWE changelog", "publisher": "Datacurve", "type": "Official changelog", "url": "https://deepswe.datacurve.ai/changelog", "published": "2026-09-03", "accessed": "2026-10-09", "establishes": "Lists each model addition by date; the most recent is GPT-6 Astra (all efforts) on 3 Sep 2026.", "key": "datacurve-changelog", "readings": 0, "n": 3 }, { "id": "r-1dj68fa", "title": "Run DeepSWE", "publisher": "Datacurve", "type": "Official documentation", "url": "https://deepswe.datacurve.ai/run", "published": "2026-06-15", "accessed": "2026-10-09", "establishes": "Official scores are produced with Pier running mini-swe-agent on Modal, in isolated containers.", "key": "datacurve-run", "readings": 0, "n": 4 }, { "id": "r-qec9fu", "title": "leaderboard-live.json (v1.1 data file)", "publisher": "Datacurve", "type": "Official data file", "url": "https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json", "published": "2026-09-22", "accessed": "2026-10-09", "establishes": "The board's data file: generated 22 Sep 2026, latest job finished 1 Sep; its costs differ from the page's for 19 of 70 configurations.", "key": "datacurve-artifact", "readings": 0, "n": 5 }, { "id": "r-11h0zp7", "url": "https://huggingface.co/zai-org/GLM-5.2", "title": "GLM-5.2 HuggingFace Model Card", "publisher": "Z.ai", "type": "Lab self-report", "published": "2026-06-16", "accessed": "2026-10-09", "establishes": "Pass@1 for GLM-5.2 46.2% [max].", "readings": 1, "n": 6 }, { "id": "r-12e8ru1", "url": "https://poolside.ai/blog/introducing-laguna-s-2-1", "title": "Poolside Laguna S 2.1 launch post", "publisher": "Poolside", "type": "Lab self-report", "published": "2026-07-21", "accessed": "2026-10-10", "establishes": "Pass@1 for Laguna S 2.1 16.5%–40.4% across 2 settings, with tokens.", "readings": 2, "n": 7 }, { "id": "r-kyj4bu", "url": "https://huggingface.co/moonshotai/Kimi-K3", "title": "Kimi-K3 HuggingFace Model Card", "publisher": "Moonshot AI", "type": "Lab self-report", "published": "2026-07-23", "accessed": "2026-10-09", "establishes": "Pass@1 for Kimi K3 67.5% [max].", "readings": 1, "n": 8 }, { "id": "r-l1zffg", "url": "https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf", "title": "Claude Opus 5 System Card", "publisher": "Anthropic", "type": "Lab self-report", "published": "2026-07-24", "accessed": "2026-10-09", "establishes": "Pass@1 for Claude Opus 5 68.8% [max].", "readings": 1, "n": 9 }, { "id": "r-190dm7p", "url": "https://huggingface.co/Qwen/Qwen3.8-27B", "title": "Qwen3.8-27B HuggingFace Model Card", "publisher": "Alibaba (Qwen)", "type": "Lab self-report", "published": "2026-08-05", "accessed": "2026-10-10", "establishes": "Pass@1 for Qwen3.8-27B 42.2%; Qwen3.7 Plus 14.2%; Qwen3.6-27B 13.3%.", "readings": 3, "n": 10 }, { "id": "r-ouk7wk", "url": "https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B", "title": "Qwen3.8-2.4T-A95B (Qwen3.8-Max) HuggingFace Model Card", "publisher": "Alibaba (Qwen)", "type": "Lab self-report", "published": "2026-08-08", "accessed": "2026-10-09", "establishes": "Pass@1 for Qwen3.8 Max 56.6%; Qwen3.7 Max 21.6%.", "readings": 2, "n": 11 }, { "id": "r-7akg1l", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813", "title": "DeepSeek-V4-Pro-0813 HuggingFace Model Card", "publisher": "DeepSeek", "type": "Lab self-report", "published": "2026-08-13", "accessed": "2026-10-10", "establishes": "Pass@1 for DeepSeek V4 Pro 62.7% [max]; DeepSeek V4 Flash 54.4% [max]; DeepSeek V4 Pro (preview) 12.8% [max]; DeepSeek V4 Flash (preview) 7.3% [max].", "readings": 4, "n": 12 }, { "id": "r-tt3io3", "url": "https://media.x.ai/v1/website/card-4p6-4cd2dc57.pdf", "title": "xAI Grok 4.6 Model Card", "publisher": "xAI", "type": "Lab self-report", "published": "2026-08-18", "accessed": "2026-10-09", "establishes": "Pass@1 for Grok 4.6 65.9% [high].", "readings": 1, "n": 13 }, { "id": "r-qxom7y", "url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next", "title": "Qwen3.8-Flash-Next HuggingFace Model Card", "publisher": "Alibaba (Qwen)", "type": "Lab self-report", "published": "2026-08-24", "accessed": "2026-10-09", "establishes": "Pass@1 for Qwen3.8-Flash-Next 58.7%; Qwen3.7 Plus 16.5%.", "readings": 2, "n": 14 }, { "id": "r-1171e08", "url": "https://huggingface.co/zai-org/GLM-5.3", "title": "GLM-5.3 HuggingFace Model Card", "publisher": "Z.ai", "type": "Lab self-report", "published": "2026-08-25", "accessed": "2026-10-09", "establishes": "Pass@1 for GLM-5.3 66.9% [max].", "readings": 1, "n": 15 }, { "id": "r-p3aj15", "url": "https://huggingface.co/tencent/Hy4-preview", "title": "Tencent Hy4-preview Technical Report & Model Card", "publisher": "Tencent", "type": "Lab self-report", "published": "2026-08-27", "accessed": "2026-10-09", "establishes": "Pass@1 for Hy4 Preview 64.3%; Hy3 28%.", "readings": 2, "n": 16 }, { "id": "r-1b2qf5e", "url": "https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf", "title": "Claude Fable 5.1 & Claude Mythos 5.1 System Card", "publisher": "Anthropic", "type": "Lab self-report", "published": "2026-09-01", "accessed": "2026-10-09", "establishes": "Pass@1 for Claude Fable 5.1 67.4% [max].", "readings": 1, "n": 17 }, { "id": "r-1wqp1zm", "url": "https://research.meta.ai/static/muse-spark-1-3-multimodal-evaluation-methodology", "title": "Meta AI Research Muse Spark 1.3 Evaluation Methodology", "publisher": "Meta", "type": "Lab self-report", "published": "2026-09-02", "accessed": "2026-10-09", "establishes": "Pass@1 for Muse Spark 1.3 75.4% [max].", "readings": 1, "n": 18 }, { "id": "r-pixn6e", "url": "https://github.com/nex-agi/Nex-N2.5", "title": "Nex-AGI Nex-N2.5 GitHub README / Model Card", "publisher": "Nex-AGI", "type": "Lab self-report", "published": "2026-09-08", "accessed": "2026-10-09", "establishes": "Pass@1 for Nex-N2.5-Max 65.6%; Nex-N2.5-Pro 55.8%; Nex-N2.5-Mini 36.1%.", "readings": 3, "n": 19 }, { "id": "r-prwi23", "url": "https://cognition.com/blog/swe-2", "title": "Cognition SWE-2 launch post", "publisher": "Cognition", "type": "Lab self-report", "published": "2026-09-10", "accessed": "2026-10-09", "establishes": "Pass@1 for SWE-2 73%; SWE-1.7 37.7%.", "readings": 2, "n": 20 }, { "id": "r-1ftq9xa", "url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "title": "DeepSeek-V4.1-Flash HuggingFace Model Card", "publisher": "DeepSeek", "type": "Lab self-report", "published": "2026-09-10", "accessed": "2026-10-09", "establishes": "Pass@1 for DeepSeek V4.1 Flash 65.5%–74.2% across 8 settings.", "readings": 8, "n": 21 }, { "id": "r-s5arsj", "url": "https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/", "title": "Google Gemini 3.8 Flash launch blog and Model Evaluation Report", "publisher": "Google DeepMind", "type": "Lab self-report", "published": "2026-09-17", "accessed": "2026-10-09", "establishes": "Pass@1 for Gemini 3.8 Flash 73.7% [high], with cost per task, tokens.", "readings": 1, "n": 22 }, { "id": "r-1o5c47s", "url": "https://www.stepfun.com/", "title": "StepFun Step 5 Preview Announcement", "publisher": "StepFun", "type": "Lab self-report", "published": "2026-09-20", "accessed": "2026-10-09", "establishes": "Pass@1 for Step 5 Preview 67.7% [high].", "readings": 1, "n": 23 }, { "id": "r-1g5kt5u", "url": "https://media.x.ai/v1/website/card4p7-3a96f40b.pdf", "title": "xAI Grok 4.7 Model Card", "publisher": "xAI", "type": "Lab self-report", "published": "2026-09-21", "accessed": "2026-10-09", "establishes": "Pass@1 for Grok 4.7 71% [high].", "readings": 1, "n": 24 }, { "id": "r-1mttwp9", "url": "https://mimo.xiaomi.com/mimo-v2-6", "title": "Xiaomi MiMo-V2.6 Launch Announcement and Appendix", "publisher": "Xiaomi", "type": "Lab self-report", "published": "2026-09-21", "accessed": "2026-10-09", "establishes": "Pass@1 for MiMo-V2.6-Pro 71.9% [max]; MiMo-V2.6-Flash 67.9% [max].", "readings": 2, "n": 25 }, { "id": "r-1jseipb", "url": "https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf", "title": "Claude Opus 5.5 System Card", "publisher": "Anthropic", "type": "Lab self-report", "published": "2026-09-22", "accessed": "2026-10-09", "establishes": "Pass@1 for Claude Opus 5.5 74.2% [max].", "readings": 1, "n": 26 }, { "id": "r-1m7ewdf", "url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "title": "OpenAI GPT-6 Sol and Luna launch post", "publisher": "OpenAI", "type": "Lab self-report", "published": "2026-09-22", "accessed": "2026-10-09", "establishes": "Pass@1 for GPT-6 Sol 37.2%–68.8% across 5 settings; GPT-6 Luna 2.4%–66.6% across 5 settings; GPT-5.6 Luna 1.2%–62.2% across 5 settings, with cost per task.", "readings": 15, "n": 27 }, { "id": "r-1vlr43v", "url": "https://fireworks.ai/blog/ember-1", "title": "Fireworks AI Ember-1 Announcement", "publisher": "Fireworks AI", "type": "Lab self-report", "published": "2026-09-23", "accessed": "2026-10-09", "establishes": "Pass@1 for Ember-1 75.2% [thinking]; Kimi K3 55.8%–66.4% across 3 settings, with cost per task.", "readings": 4, "n": 28 }, { "id": "r-1ctctri", "url": "https://storage.googleapis.com/deepmind-media/gemini/gemini_4_argon_model_evaluation.pdf", "title": "Gemini 4 Argon Model Evaluation Report", "publisher": "Google DeepMind", "type": "Lab self-report", "published": "2026-09-24", "accessed": "2026-10-09", "establishes": "Pass@1 for Gemini 4 Argon 77.9%.", "readings": 1, "n": 29 }, { "id": "r-1tiaoyx", "url": "https://www-cdn.anthropic.com/870c8f525702625d2c62fc6dd04c857e3250bec1/Claude%20Sonnet%205.5%20System%20Card.pdf", "title": "Claude Sonnet 5.5 System Card", "publisher": "Anthropic", "type": "Lab self-report", "published": "2026-09-28", "accessed": "2026-10-09", "establishes": "Pass@1 for Claude Sonnet 5.5 71% [max].", "readings": 1, "n": 30 }, { "id": "r-1ntqzs0", "url": "https://openai.com/index/introducing-gpt-6-1-sol/", "title": "OpenAI GPT-6.1 Sol launch post", "publisher": "OpenAI", "type": "Lab self-report", "published": "2026-09-29", "accessed": "2026-10-09", "establishes": "Pass@1 for GPT-6.1 Sol 64.4%–75.2% across 5 settings, with cost per task.", "readings": 5, "n": 31 }, { "id": "r-1x8ip3p", "url": "https://reflection.ai/blog/introducing-beam", "title": "Reflection AI Beam launch post", "publisher": "Reflection AI", "type": "Lab self-report", "published": "2026-10-05", "accessed": "2026-10-09", "establishes": "Pass@1 for Beam 44.4%.", "readings": 1, "n": 32 }, { "id": "r-d2tk0p", "url": "https://mistral.ai/news/mistral-large-4/", "title": "Mistral Large 4 Announcement", "publisher": "Mistral AI", "type": "Lab self-report", "published": "2026-10-06", "accessed": "2026-10-09", "establishes": "Pass@1 for Mistral Large 4 61.7% [thinking].", "readings": 1, "n": 33 }, { "id": "r-1kqtfhm", "title": "Claude Haiku 5.5 System Card", "publisher": "Anthropic", "type": "Lab system card", "url": "https://www-cdn.anthropic.com/e1080d6bf5ae2018ea3c2f414064be03232f5be5/Claude%20Haiku%205.5%20System%20Card.pdf", "published": "2026-09-29", "accessed": "2026-10-09", "establishes": "Reports SWE-bench Pro, FrontierSWE v2 and ProgramBench for Haiku 5.5, and no DeepSWE 1.1 result.", "key": "anthropic-haiku-55", "readings": 0, "n": 34 }, { "id": "r-hzsdm7", "url": "https://entrpi.github.io/misc/deep-swe-minimax-m3/", "title": "Independent DeepSWE Audit by entrpi", "publisher": "entrpi (independent)", "type": "Independent run", "published": "2026-06-02", "accessed": "2026-10-09", "establishes": "Pass@1 for MiniMax M3 13.3%–16.8% across 2 settings, with cost per task, tokens.", "readings": 2, "n": 35 }, { "id": "r-17v6sj", "url": "https://artificialanalysis.ai/articles/benchmarking-grok-4-7", "title": "Artificial Analysis (Benchmarking Grok 4.7)", "publisher": "Artificial Analysis", "type": "Independent run", "published": "2026-09-21", "accessed": "2026-10-09", "establishes": "Pass@1 for Grok 4.7 73% [xhigh]; Grok 4.6 65% [high].", "readings": 2, "n": 36 }, { "id": "r-14zikky", "url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "title": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "publisher": "Mercor", "type": "Independent run", "published": "", "accessed": "2026-10-09", "establishes": "Pass@1 for Claude Opus 5.5 72.3% [max]; GPT-6.1 Sol 72.3% [max]; GPT-6 Astra 72% [max]; DeepSeek V4.1 Flash 71.7% [max]; Claude Opus 5 70.2%–71.4% across 2 settings; Gemini 3.8 Flash 71.4% [high]; and 43 more models, with confidence intervals.", "readings": 52, "n": 37 }, { "id": "r-182356w", "url": "https://sleev.ai/performance", "title": "Sleev Performance & Benchmarks", "publisher": "OpenAI", "type": "Independent run", "published": "", "accessed": "2026-10-10", "establishes": "Pass@1 for GPT-5.5 57.5% [high], with cost per task, confidence intervals.", "readings": 1, "n": 38 }, { "id": "r-1tuhx3r", "title": "Benchmark review: DeepSWE v1.1", "publisher": "Epoch AI", "type": "Independent audit", "url": "https://epoch.ai/benchmarks/deepswe/review", "published": "2026-09-07", "accessed": "2026-10-09", "establishes": "Rates DeepSWE v1.1 flawed: grading defects in at least 23 of 113 tasks, most from hidden tests colliding with tests the agent wrote.", "key": "epoch-review", "readings": 0, "n": 39 }, { "id": "r-18kp64f", "title": "envcheck: auditing DeepSWE", "publisher": "Tokenless", "type": "Independent audit", "url": "https://usetokenless.com/blog/envcheck", "published": "2026-09-30", "accessed": "2026-10-09", "establishes": "Documents 70 ways a submission can rewrite test outcomes, because the test harness sits inside the repository the agent edits.", "key": "tokenless-envcheck", "readings": 0, "n": 40 }, { "id": "r-i0raoa", "title": "datacurve-ai/deep-swe issue #103", "publisher": "GitHub (community discussion)", "type": "Discussion", "url": "https://github.com/datacurve-ai/deep-swe/issues/103", "published": "2026-09-30", "accessed": "2026-10-09", "establishes": "Notes the board has published no new model since 3 Sep and quotes Datacurve's CEO saying the team is building the next generation of DeepSWE.", "key": "gh-issue-103", "readings": 0, "n": 41 }, { "id": "r-isoxsa", "title": "AI Gateway model list and prices", "publisher": "Vercel", "type": "Price list", "url": "https://ai-gateway.vercel.sh/v1/models", "published": "", "accessed": "2026-10-09", "establishes": "Per-token list prices used to estimate cost where a source published tokens but no cost: Laguna S 2.1 at $0.09 per million input tokens and $0.18 per million output tokens.", "key": "ai-gateway-pricing", "readings": 0, "n": 42 } ], "rows": [ { "model_key": "gemini-4-argon", "model_as_reported": "Gemini 4 Argon", "effort": "", "harness": "mini-swe-agent", "score_pct": 77.9, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Gemini 4 Argon Model Evaluation Report", "source_url": "https://storage.googleapis.com/deepmind-media/gemini/gemini_4_argon_model_evaluation.pdf", "published": "2026-09-24", "accessed": "2026-10-09", "quote": "Gemini 4 Argon achieved 77.9% on DeepSWE v1.1 with mini-swe agent harness under highest thinking setting.", "notes": "Google eval PDF: self-computed with mini-swe-agent at the highest thinking setting; no level name given. Only Gemini 4 tier released. Evaluation report specifies highest thinking setting, pass@1, mini-swe-agent. No task cost or token counts given.", "display_name": "Gemini 4 Argon", "lab": "Google", "open_weights": false, "row_id": "row-1xty4qe", "model_released": "2026-09-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ctctri", "source_n": 29 }, { "model_key": "muse-spark-1.3", "model_as_reported": "Muse Spark 1.3", "effort": "max", "harness": "mini-swe-agent", "score_pct": 75.4, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Meta AI Research Muse Spark 1.3 Evaluation Methodology", "source_url": "https://research.meta.ai/static/muse-spark-1-3-multimodal-evaluation-methodology", "published": "2026-09-02", "accessed": "2026-10-09", "quote": "We run Muse Spark 1.3 max with a mini-swe agent and obtained the benchmarks for all other models on the official leaderboard on Datacurve.", "notes": "Corrects t5-community.json and t4-aggregators.json: Meta's primary methodology document explicitly states mini-swe-agent (not Muse Code) and gives no task cost ($0.55 was an external AA estimate).", "display_name": "Muse Spark 1.3", "lab": "Meta", "open_weights": false, "row_id": "row-47v276", "model_released": "2026-09-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1wqp1zm", "source_n": 18 }, { "model_key": "gpt-6.1-sol", "model_as_reported": "GPT-6.1 Sol", "effort": "high", "harness": "lab-internal", "score_pct": 75.22, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.6461, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6.1 Sol launch post", "source_url": "https://openai.com/index/introducing-gpt-6-1-sol/", "published": "2026-09-29", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6.1 Sol\",\"modelLabel\":\"GPT-6.1 Sol\",\"cost\":0.6461,\"score\":0.7522,\"effortLabel\":\"High\"}", "notes": "Reported in embedded Vega-Lite spec on launch page.", "display_name": "GPT-6.1 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-1j0dft9", "model_released": "2026-09-29", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ntqzs0", "source_n": 31 }, { "model_key": "ember-1", "model_as_reported": "Ember-1", "effort": "thinking", "harness": "mini-swe-agent", "score_pct": 75.2, "ci_pct": null, "n_samples": 113, "cost_per_task_usd": 3.62, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Fireworks AI Ember-1 Announcement", "source_url": "https://fireworks.ai/blog/ember-1", "published": "2026-09-23", "accessed": "2026-10-09", "quote": "Ember-1: 75.2% | Comparison (Ember-1 vs. K3 Max): -23.7% / -126.9 USD", "notes": "Post-trained Kimi K3 variant reducing reasoning tokens by 35-50%. Evaluated across N=113 DeepSWE 1.1 tasks. Total cost was $408.54 ($3.62/task), a 23.7% savings vs K3 Max.", "display_name": "Ember-1", "lab": "Fireworks AI", "open_weights": false, "row_id": "row-p2qq0j", "model_released": "2026-09-23", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1vlr43v", "source_n": 28 }, { "model_key": "claude-opus-5.5", "model_as_reported": "Claude Opus 5.5", "effort": "max", "harness": "lab-internal", "score_pct": 74.2, "ci_pct": null, "n_samples": 5, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Claude Opus 5.5 System Card", "source_url": "https://www-cdn.anthropic.com/fc1b44717c85dc068bc6ba5024219938094694bd/Claude%20Opus%205.5%20System%20Card.pdf", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "Claude Opus 5.5 scored an average of 74.2% over five trials.", "notes": "Section 8.3 reports 74.2% over 5 trials with adaptive thinking/max reasoning budget in internal evaluation. No cost or token figures given.", "display_name": "Claude Opus 5.5", "lab": "Anthropic", "open_weights": false, "row_id": "row-k5hr20", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1jseipb", "source_n": 26 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash", "effort": "max", "harness": "mini-swe-agent", "score_pct": 74.2, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 74.0 | 73.0 | 67.5 | 66.9 | 62.7 | 54.4 | **74.2** |", "notes": "Table: Comparison with frontier models (Max reasoning effort). All instruct results use reasoning_effort=100, temp=1.0, top_p=0.95. N=8 samples per task on DeepSWE v1.1, 1M-token context limit, max_steps=500 per agent.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-o4taeh", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "gpt-6-astra", "model_as_reported": "gpt-6-astra", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 74.12, "ci_pct": 2.87, "n_samples": 4, "cost_per_task_usd": 4.4291, "tokens_per_task": 1486484, "input_tokens_per_task": 1456927, "output_tokens_per_task": 29557, "steps_per_task": 28.8, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-09-03", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_6_astra_xhigh: pass_at_1 0.7411504424778761, ci_half 0.02865396174075141, n_runs 4, mean_cost_usd 4.429117203539823", "notes": "pass@4 80.5%; mean 19 min per attempt; cost basis: Current pricing at all context lengths: $10/M uncached input, $12.50/M cache writes, $1/M cache reads, $50/M output; no separate compute-unit fee.; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-6 Astra", "lab": "OpenAI", "open_weights": false, "row_id": "row-owdu43", "model_released": "2026-09-03", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gemini-3.8-flash", "model_as_reported": "gemini-3-8-flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 73.83, "ci_pct": 1.42, "n_samples": 4, "cost_per_task_usd": 2.3623, "tokens_per_task": 21877692, "input_tokens_per_task": 21734449, "output_tokens_per_task": 143243, "steps_per_task": 166.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-09-01", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_8_flash_high: pass_at_1 0.738255033557047, ci_half 0.014173132066358783, n_runs 4, mean_cost_usd 2.362349413758389", "notes": "pass@4 85.8%; mean 11 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.8 Flash", "lab": "Google", "open_weights": false, "row_id": "row-1w5t75g", "model_released": "2026-09-01", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gemini-3.8-flash", "model_as_reported": "Gemini 3.8 Flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 73.7, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 2.36, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": 143243, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Google Gemini 3.8 Flash launch blog and Model Evaluation Report", "source_url": "https://blog.google/innovation-and-ai/models-and-research/gemini-models/3-8-flash-and-3-8-flash-cyber/", "published": "2026-09-17", "accessed": "2026-10-09", "quote": "On DeepSWE v1.1 (Long-Horizon Software Engineering) 3.8 Flash outperforms most larger frontier models in autonomously solving complex engineering problems end to end, only at a fraction of the cost.", "notes": "Score reported as 73.7% in evaluation table image and methodology PDF (https://storage.googleapis.com/deepmind-media/gemini/gemini_3-8_flash_model_evaluation.pdf): 'Results for Gemini 3.8 Flash are self computed, and use a mini-swe agent harness with high thinking.' Also reported as 73.8% in blog overview text and OpenAI Astra comparison chart ($2.36 / 143,243 output tokens at high effort; $1.97 / 124,684 tokens at medium effort).", "display_name": "Gemini 3.8 Flash", "lab": "Google", "open_weights": false, "row_id": "row-14zbox", "model_released": "2026-09-01", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-s5arsj", "source_n": 22 }, { "model_key": "claude-opus-5", "model_as_reported": "claude-opus-5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 73.65, "ci_pct": 3.87, "n_samples": 4, "cost_per_task_usd": 11.8376, "tokens_per_task": 15143400, "input_tokens_per_task": 15025834, "output_tokens_per_task": 117566, "steps_per_task": 99, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-25", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_5_max: pass_at_1 0.7364864864864865, ci_half 0.03872310426371729, n_runs 4, mean_cost_usd 11.837583271396396", "notes": "pass@4 88.5%; mean 32 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-346qj2", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-astra", "model_as_reported": "gpt-6-astra", "effort": "high", "harness": "mini-swe-agent", "score_pct": 73.23, "ci_pct": 3.42, "n_samples": 4, "cost_per_task_usd": 3.9237, "tokens_per_task": 1309777, "input_tokens_per_task": 1283271, "output_tokens_per_task": 26506, "steps_per_task": 27.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-09-03", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_6_astra_high: pass_at_1 0.7323008849557522, ci_half 0.03423496057873005, n_runs 4, mean_cost_usd 3.9237237909292038", "notes": "pass@4 82.3%; mean 17 min per attempt; cost basis: Current pricing at all context lengths: $10/M uncached input, $12.50/M cache writes, $1/M cache reads, $50/M output; no separate compute-unit fee.; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-6 Astra", "lab": "OpenAI", "open_weights": false, "row_id": "row-vqmucw", "model_released": "2026-09-03", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-astra", "model_as_reported": "gpt-6-astra", "effort": "max", "harness": "mini-swe-agent", "score_pct": 73.23, "ci_pct": 0.83, "n_samples": 4, "cost_per_task_usd": 7.4978, "tokens_per_task": 2244182, "input_tokens_per_task": 2183034, "output_tokens_per_task": 61149, "steps_per_task": 28.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-09-03", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_6_astra_max: pass_at_1 0.7323008849557522, ci_half 0.008303197562056526, n_runs 4, mean_cost_usd 7.497827996681416", "notes": "pass@4 79.6%; mean 33 min per attempt; cost basis: Current pricing at all context lengths: $10/M uncached input, $12.50/M cache writes, $1/M cache reads, $50/M output; no separate compute-unit fee.; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-6 Astra", "lab": "OpenAI", "open_weights": false, "row_id": "row-5dvg8e", "model_released": "2026-09-03", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-opus-5", "model_as_reported": "claude-opus-5", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 73.15, "ci_pct": 3.06, "n_samples": 4, "cost_per_task_usd": 9.0722, "tokens_per_task": 11384401, "input_tokens_per_task": 11292729, "output_tokens_per_task": 91672, "steps_per_task": 88.7, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-25", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_5_xhigh: pass_at_1 0.7315436241610739, ci_half 0.030609818216384643, n_runs 4, mean_cost_usd 9.07220232326622", "notes": "pass@4 85.8%; mean 26 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1oeh1kj", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6.1-sol", "model_as_reported": "GPT-6.1 Sol", "effort": "medium", "harness": "lab-internal", "score_pct": 73.01, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.4196, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6.1 Sol launch post", "source_url": "https://openai.com/index/introducing-gpt-6-1-sol/", "published": "2026-09-29", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6.1 Sol\",\"modelLabel\":\"GPT-6.1 Sol\",\"cost\":0.4196,\"score\":0.7301,\"effortLabel\":\"Medium\"}", "notes": "Reported in embedded Vega-Lite spec on launch page.", "display_name": "GPT-6.1 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-16dfael", "model_released": "2026-09-29", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ntqzs0", "source_n": 31 }, { "model_key": "grok-4.7", "model_as_reported": "Grok 4.7", "effort": "xhigh", "harness": "Grok Build", "score_pct": 73, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Artificial Analysis (Benchmarking Grok 4.7)", "source_url": "https://artificialanalysis.ai/articles/benchmarking-grok-4-7", "published": "2026-09-21", "accessed": "2026-10-09", "quote": "DeepSWE v1.1: 73% (improved from 65% on Grok 4.6 xhigh)", "notes": "Corrects t5-community.json (which attributed 73% with Grok Build to xAI news). 73.0% was evaluated and published by Artificial Analysis as part of Coding Agent Index.", "display_name": "Grok 4.7", "lab": "xAI", "open_weights": false, "row_id": "row-3bvbtr", "model_released": "2026-09-21", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-17v6sj", "source_n": 36 }, { "model_key": "swe-2", "model_as_reported": "SWE-2", "effort": "", "harness": "Devin CLI", "score_pct": 73, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Cognition SWE-2 launch post", "source_url": "https://cognition.com/blog/swe-2", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "SWE-2 | 73.0%", "notes": "Evaluated using Devin CLI ('Devin CLI for open-weight models'). Best score across reasoning-effort settings. Post states: 'On FrontierCode 1.1 Main and DeepSWE 1.1, SWE-2 beats SWE-1.7 and Grok 4.6 on both score and cost'.", "display_name": "SWE-2", "lab": "Cognition", "open_weights": false, "row_id": "row-1o9vzqd", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-prwi23", "source_n": 20 }, { "model_key": "claude-opus-5", "model_as_reported": "claude-opus-5", "effort": "high", "harness": "mini-swe-agent", "score_pct": 72.83, "ci_pct": 1.95, "n_samples": 4, "cost_per_task_usd": 6.0761, "tokens_per_task": 7298087, "input_tokens_per_task": 7233879, "output_tokens_per_task": 64207, "steps_per_task": 72.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-25", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_5_high: pass_at_1 0.7282850779510023, ci_half 0.019454321571601495, n_runs 4, mean_cost_usd 6.076084489977728", "notes": "pass@4 87.6%; mean 19 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-lbxd05", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-astra", "model_as_reported": "gpt-6-astra", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 72.79, "ci_pct": 2.59, "n_samples": 4, "cost_per_task_usd": 3.0755, "tokens_per_task": 1028600, "input_tokens_per_task": 1008238, "output_tokens_per_task": 20362, "steps_per_task": 26, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-09-03", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_6_astra_medium: pass_at_1 0.7278761061946902, ci_half 0.025896490818318695, n_runs 4, mean_cost_usd 3.0754742975663714", "notes": "pass@4 82.3%; mean 15 min per attempt; cost basis: Current pricing at all context lengths: $10/M uncached input, $12.50/M cache writes, $1/M cache reads, $50/M output; no separate compute-unit fee.; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-6 Astra", "lab": "OpenAI", "open_weights": false, "row_id": "row-1wxhh73", "model_released": "2026-09-03", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-sol", "model_as_reported": "gpt-5-6-sol", "effort": "max", "harness": "mini-swe-agent", "score_pct": 72.67, "ci_pct": 2.83, "n_samples": 4, "cost_per_task_usd": 6.456, "tokens_per_task": 7967666, "input_tokens_per_task": 7907652, "output_tokens_per_task": 60014, "steps_per_task": 61.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_sol_max: pass_at_1 0.7266666666666667, ci_half 0.02829822249837175, n_runs 4, mean_cost_usd 6.455954682467823", "notes": "pass@4 85.8%; mean 19 min per attempt; leaderboard-live.json gives $8.39 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-y7m969", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash (DSH Minimal)", "effort": "max", "harness": "lab-internal", "score_pct": 72.6, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |", "notes": "Evaluated with DeepSeek Harness Minimal (DSH Minimal), reasoning_effort=100, N=8, temp=1.0, top_p=0.95, 1M context, max_steps=500.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-16p8paj", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "claude-opus-5.5", "model_as_reported": "Opus 5.5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 72.3, "ci_pct": 7.3, "n_samples": 336, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Opus 5.5 | effort: max | score: 72.3% ± 7.3% | n_samples: 336 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Opus 5.5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1sj5dau", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-6.1-sol", "model_as_reported": "GPT 6.1 Sol", "effort": "max", "harness": "mini-swe-agent", "score_pct": 72.3, "ci_pct": 7.4, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 6.1 Sol | effort: max | score: 72.3% ± 7.4% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-6.1 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-1unx8j9", "model_released": "2026-09-29", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-6-astra", "model_as_reported": "GPT 6 Astra", "effort": "max", "harness": "mini-swe-agent", "score_pct": 72, "ci_pct": 7.5, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 6 Astra | effort: max | score: 72% ± 7.5% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-6 Astra", "lab": "OpenAI", "open_weights": false, "row_id": "row-1mtbg4a", "model_released": "2026-09-03", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-6.1-sol", "model_as_reported": "GPT-6.1 Sol", "effort": "max", "harness": "lab-internal", "score_pct": 71.9, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 1.5711, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6.1 Sol launch post", "source_url": "https://openai.com/index/introducing-gpt-6-1-sol/", "published": "2026-09-29", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6.1 Sol\",\"modelLabel\":\"GPT-6.1 Sol\",\"cost\":1.5711,\"score\":0.7190,\"effortLabel\":\"Max\"}", "notes": "Corrects t5-community.json (71.0% via GitHub issue #102). Exact score is 71.90% with cost $1.5711.", "display_name": "GPT-6.1 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-z4hdo2", "model_released": "2026-09-29", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ntqzs0", "source_n": 31 }, { "model_key": "gpt-6.1-sol", "model_as_reported": "GPT-6.1 Sol", "effort": "xhigh", "harness": "lab-internal", "score_pct": 71.9, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.7886, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6.1 Sol launch post", "source_url": "https://openai.com/index/introducing-gpt-6-1-sol/", "published": "2026-09-29", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6.1 Sol\",\"modelLabel\":\"GPT-6.1 Sol\",\"cost\":0.7886,\"score\":0.7190,\"effortLabel\":\"Xhigh\"}", "notes": "Corrects t5-community.json (71.0% via GitHub issue #102). Exact score is 71.90% with cost $0.7886.", "display_name": "GPT-6.1 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-1xg0j7k", "model_released": "2026-09-29", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ntqzs0", "source_n": 31 }, { "model_key": "mimo-v2.6-pro", "model_as_reported": "MiMo-V2.6-Pro", "effort": "max", "harness": "lab-internal", "score_pct": 71.9, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Xiaomi MiMo-V2.6 Launch Announcement and Appendix", "source_url": "https://mimo.xiaomi.com/mimo-v2-6", "published": "2026-09-21", "accessed": "2026-10-09", "quote": "In Xiaomi’s launch appendix, MiMo-V2.6-Pro scored 71.9 on DeepSWE v1.1.", "notes": "1.02T total / 42B active parameter MoE model. Launch appendix reports 71.9% on DeepSWE v1.1. Also documented on Kingy.ai and FoneArena.", "display_name": "MiMo-V2.6-Pro", "lab": "Xiaomi", "open_weights": true, "row_id": "row-b1kgds", "model_released": "2026-09-21", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1mttwp9", "source_n": 25 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DeepSeek V4.1 Flash", "effort": "max", "harness": "mini-swe-agent", "score_pct": 71.7, "ci_pct": 6.3, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "DeepSeek V4.1 Flash | effort: max | score: 71.7% ± 6.3% | n_samples: 339 | provider: DeepSeek", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-1n55rtx", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-5", "model_as_reported": "Opus 5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 71.4, "ci_pct": 6.8, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Opus 5 | effort: max | score: 71.4% ± 6.8% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-nhlnrl", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gemini-3.8-flash", "model_as_reported": "Gemini 3.8 Flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 71.4, "ci_pct": 6.8, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Gemini 3.8 Flash | effort: high | score: 71.4% ± 6.8% | n_samples: 339 | provider: Google", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Gemini 3.8 Flash", "lab": "Google", "open_weights": false, "row_id": "row-1kjx08d", "model_released": "2026-09-01", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gemini-3.8-flash", "model_as_reported": "gemini-3-8-flash", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 71.02, "ci_pct": 2.28, "n_samples": 4, "cost_per_task_usd": 1.9671, "tokens_per_task": 16988624, "input_tokens_per_task": 16863940, "output_tokens_per_task": 124684, "steps_per_task": 147.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-09-01", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_8_flash_medium: pass_at_1 0.7101769911504425, ci_half 0.02280804572877925, n_runs 4, mean_cost_usd 1.9671024789823006", "notes": "pass@4 83.2%; mean 14 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.8 Flash", "lab": "Google", "open_weights": false, "row_id": "row-itcfkk", "model_released": "2026-09-01", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-5.5", "model_as_reported": "Claude Sonnet 5.5", "effort": "max", "harness": "lab-internal", "score_pct": 71, "ci_pct": null, "n_samples": 5, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Claude Sonnet 5.5 System Card", "source_url": "https://www-cdn.anthropic.com/870c8f525702625d2c62fc6dd04c857e3250bec1/Claude%20Sonnet%205.5%20System%20Card.pdf", "published": "2026-09-28", "accessed": "2026-10-09", "quote": "Sonnet 5.5 scored an average of 71.0% over five trials.", "notes": "Section 8.3 reports 71.0% over 5 trials with adaptive thinking/max reasoning budget in internal evaluation. No cost or token figures given.", "display_name": "Claude Sonnet 5.5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1ns4aaz", "model_released": "2026-09-28", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1tiaoyx", "source_n": 30 }, { "model_key": "grok-4.7", "model_as_reported": "Grok 4.7", "effort": "high", "harness": "mini-swe-agent", "score_pct": 71, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "xAI Grok 4.7 Model Card", "source_url": "https://media.x.ai/v1/website/card4p7-3a96f40b.pdf", "published": "2026-09-21", "accessed": "2026-10-09", "quote": "Grok 4.7 (xHigh): 71.0% on DeepSWE v1.1", "notes": "xAI card p.7: \"Grok 4.7 scores 71.0% at high\"; footnote: results taken from evaluations conducted by Datacurve, which never posted them on its public board. Published by xAI in official model card under mini-swe-agent.", "display_name": "Grok 4.7", "lab": "xAI", "open_weights": false, "row_id": "row-1kwajgk", "model_released": "2026-09-21", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1g5kt5u", "source_n": 24 }, { "model_key": "gpt-5.6-sol", "model_as_reported": "gpt-5-6-sol", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 70.73, "ci_pct": 0.82, "n_samples": 4, "cost_per_task_usd": 3.5986, "tokens_per_task": 4300063, "input_tokens_per_task": 4259318, "output_tokens_per_task": 40745, "steps_per_task": 44, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_sol_xhigh: pass_at_1 0.7073170731707317, ci_half 0.008191117217343103, n_runs 4, mean_cost_usd 3.5986416572286237", "notes": "pass@4 85.8%; mean 13 min per attempt; leaderboard-live.json gives $4.70 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-tdaaio", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-5.5", "model_as_reported": "Sonnet 5.5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 70.5, "ci_pct": 7.7, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Sonnet 5.5 | effort: max | score: 70.5% ± 7.7% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Sonnet 5.5", "lab": "Anthropic", "open_weights": false, "row_id": "row-qoxc9u", "model_released": "2026-09-28", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash (DSH Standard)", "effort": "max", "harness": "lab-internal", "score_pct": 70.5, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |", "notes": "Evaluated with DeepSeek Harness Standard (DSH Standard), reasoning_effort=100, N=8, temp=1.0, top_p=0.95, 1M context, max_steps=500.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-106utim", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "glm-5.3", "model_as_reported": "GLM 5.3", "effort": "max", "harness": "mini-swe-agent", "score_pct": 70.5, "ci_pct": 6.3, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GLM 5.3 | effort: max | score: 70.5% ± 6.3% | n_samples: 339 | provider: Zhipu", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GLM-5.3", "lab": "Z.ai", "open_weights": true, "row_id": "row-18ttaw", "model_released": "2026-08-20", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.6-sol", "model_as_reported": "GPT 5.6 Sol", "effort": "max", "harness": "mini-swe-agent", "score_pct": 70.5, "ci_pct": 6.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 5.6 Sol | effort: max | score: 70.5% ± 6.9% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-5.6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-1c4imbl", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-6-sol", "model_as_reported": "GPT 6 Sol", "effort": "max", "harness": "mini-swe-agent", "score_pct": 70.5, "ci_pct": 7.2, "n_samples": 338, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 6 Sol | effort: max | score: 70.5% ± 7.2% | n_samples: 338 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-tdvxi6", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-5", "model_as_reported": "Opus 5", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 70.2, "ci_pct": 6.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Opus 5 | effort: xhigh | score: 70.2% ± 6.9% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1dtxecg", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.6-terra", "model_as_reported": "GPT 5.6 Terra", "effort": "max", "harness": "mini-swe-agent", "score_pct": 70.2, "ci_pct": 7.2, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 5.6 Terra | effort: max | score: 70.2% ± 7.2% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-5.6 Terra", "lab": "OpenAI", "open_weights": false, "row_id": "row-n5l7y", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-fable-5", "model_as_reported": "claude-fable-5", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 69.91, "ci_pct": 3.24, "n_samples": 4, "cost_per_task_usd": 13.4145, "tokens_per_task": 7409513, "input_tokens_per_task": 7329161, "output_tokens_per_task": 80352, "steps_per_task": 68.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_fable_5_xhigh: pass_at_1 0.6991150442477876, ci_half 0.03244917575471319, n_runs 4, mean_cost_usd 13.414521495535714", "notes": "pass@4 88.5%; mean 24 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Fable 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1v14szb", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash (Claude Code)", "effort": "max", "harness": "Claude Code", "score_pct": 69.8, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |", "notes": "Performance across agent scaffolds table. Evaluated with Claude Code scaffold, reasoning_effort=100, N=8, temp=1.0, top_p=0.95, 1M context limit, max_steps=500.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-vejsjt", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "claude-fable-5", "model_as_reported": "claude-fable-5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 69.72, "ci_pct": 4.03, "n_samples": 4, "cost_per_task_usd": 21.6347, "tokens_per_task": 12757425, "input_tokens_per_task": 12638832, "output_tokens_per_task": 118593, "steps_per_task": 88.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_fable_5_max: pass_at_1 0.6972477064220184, ci_half 0.04033474250807936, n_runs 4, mean_cost_usd 21.634702092592594", "notes": "pass@4 84.1%; mean 35 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Fable 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-vw58k6", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-terra", "model_as_reported": "gpt-5-6-terra", "effort": "max", "harness": "mini-swe-agent", "score_pct": 69.62, "ci_pct": 2.56, "n_samples": 4, "cost_per_task_usd": 3.9567, "tokens_per_task": 9302499, "input_tokens_per_task": 9230561, "output_tokens_per_task": 71939, "steps_per_task": 75.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_terra_max: pass_at_1 0.6962305986696231, ci_half 0.025569389504458404, n_runs 4, mean_cost_usd 3.956677811086475", "notes": "pass@4 88.5%; mean 17 min per attempt; leaderboard-live.json gives $4.95 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Terra", "lab": "OpenAI", "open_weights": false, "row_id": "row-1ly571m", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-sol", "model_as_reported": "gpt-5-6-sol", "effort": "high", "harness": "mini-swe-agent", "score_pct": 69.4, "ci_pct": 1.43, "n_samples": 4, "cost_per_task_usd": 2.6618, "tokens_per_task": 2742023, "input_tokens_per_task": 2713572, "output_tokens_per_task": 28450, "steps_per_task": 36.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_sol_high: pass_at_1 0.6940133037694013, ci_half 0.014321452203772355, n_runs 4, mean_cost_usd 2.661832088842267", "notes": "pass@4 86.7%; mean 10 min per attempt; leaderboard-live.json gives $3.47 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-ajdh5w", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "glm-5.3", "model_as_reported": "glm-5-3", "effort": "max", "harness": "mini-swe-agent", "score_pct": 68.96, "ci_pct": 3.02, "n_samples": 4, "cost_per_task_usd": 3.9934, "tokens_per_task": 13352780, "input_tokens_per_task": 13272344, "output_tokens_per_task": 80436, "steps_per_task": 124.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-20", "accessed": "2026-10-09", "quote": "config mini_swe_agent_glm_5_3_max: pass_at_1 0.6895787139689579, ci_half 0.030186512715587, n_runs 4, mean_cost_usd 3.9933584893126386", "notes": "pass@4 87.6%; mean 35 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GLM-5.3", "lab": "Z.ai", "open_weights": true, "row_id": "row-fm38un", "model_released": "2026-08-20", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-opus-5", "model_as_reported": "claude-opus-5", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 68.9, "ci_pct": 1.17, "n_samples": 4, "cost_per_task_usd": 3.2898, "tokens_per_task": 3616797, "input_tokens_per_task": 3579815, "output_tokens_per_task": 36982, "steps_per_task": 52.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-25", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_5_medium: pass_at_1 0.6890380313199105, ci_half 0.011725586658930944, n_runs 4, mean_cost_usd 3.28978764541387", "notes": "pass@4 89.4%; mean 13 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1g3833n", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-sol", "model_as_reported": "GPT-6 Sol", "effort": "max", "harness": "lab-internal", "score_pct": 68.81, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 2.7439, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Sol\",\"cost_label\":\"$$2.74\",\"score\":0.6881,\"cost\":2.7439,\"effortLabel\":\"max\"}", "notes": "Reported in embedded Vega-Lite chart and text: 'On DeepSWE v1.1... GPT-6 Sol at max effort scores 68.8%, within 1.1 percentage points of Claude Fable 5’s highest score in the evaluation—69.9% at xhigh effort—at approximately 80% lower cost per task.'", "display_name": "GPT-6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-15hrore", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "claude-opus-5", "model_as_reported": "Claude Opus 5", "effort": "max", "harness": "lab-internal", "score_pct": 68.8, "ci_pct": null, "n_samples": 5, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Claude Opus 5 System Card", "source_url": "https://www-cdn.anthropic.com/c5fbac3f0b1280a933ebd26d3cb8bb9f5bdeaf48/Claude%20Opus%205%20System%20Card.pdf", "published": "2026-07-24", "accessed": "2026-10-09", "quote": "Claude Opus 5 scored an average of 68.8% over five trials.", "notes": "Section 8.3 DeepSWE v1.1 (p. 149) and Table 8.1 (p. 147).", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-kte6r1", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-l1zffg", "source_n": 9 }, { "model_key": "claude-fable-5", "model_as_reported": "claude-fable-5", "effort": "high", "harness": "mini-swe-agent", "score_pct": 68.6, "ci_pct": 1.12, "n_samples": 4, "cost_per_task_usd": 9.1776, "tokens_per_task": 4890465, "input_tokens_per_task": 4833178, "output_tokens_per_task": 57287, "steps_per_task": 58.7, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_fable_5_high: pass_at_1 0.686046511627907, ci_half 0.011208427174303877, n_runs 4, mean_cost_usd 9.177638214788733", "notes": "pass@4 86.7%; mean 18 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Fable 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1kfzh6g", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "kimi-k3", "model_as_reported": "kimi-k3", "effort": "max", "harness": "mini-swe-agent", "score_pct": 68.51, "ci_pct": 4.54, "n_samples": 4, "cost_per_task_usd": 4.6547, "tokens_per_task": 9776936, "input_tokens_per_task": 9695436, "output_tokens_per_task": 81500, "steps_per_task": 97.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-18", "accessed": "2026-10-09", "quote": "config mini_swe_agent_kimi_k3_max: pass_at_1 0.6851441241685144, ci_half 0.045370140792823206, n_runs 4, mean_cost_usd 4.654682129933482", "notes": "pass@4 89.4%; mean 76 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Kimi K3", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-123w736", "model_released": "2026-07-18", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "mimo-v2.6-flash", "model_as_reported": "MiMo-V2.6-Flash", "effort": "max", "harness": "lab-internal", "score_pct": 67.9, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Xiaomi MiMo-V2.6 Launch Announcement and Appendix", "source_url": "https://mimo.xiaomi.com/mimo-v2-6", "published": "2026-09-21", "accessed": "2026-10-09", "quote": "In Xiaomi’s launch appendix, MiMo-V2.6-Flash registered 67.9 on DeepSWE v1.1.", "notes": "Efficiency-optimized MoE model. Launch appendix reports 67.9% on DeepSWE v1.1.", "display_name": "MiMo-V2.6-Flash", "lab": "Xiaomi", "open_weights": true, "row_id": "row-q48lwe", "model_released": "2026-09-21", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1mttwp9", "source_n": 25 }, { "model_key": "step-5-preview", "model_as_reported": "Step 5 Preview", "effort": "high", "harness": "mini-swe-agent", "score_pct": 67.7, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "StepFun Step 5 Preview Announcement", "source_url": "https://www.stepfun.com/", "published": "2026-09-20", "accessed": "2026-10-09", "quote": "Step 5 Preview scored 67.7% on DeepSWE v1.1 under its High reasoning effort configuration.", "notes": "Source is StepFun’s homepage rather than a dated release page; the figure has not been re-checked on a page that is still reachable. 600B total / 27B active MoE foundation model with 1M context window. Evaluated at high reasoning effort with SWE-agent / mini-swe-agent harness.", "display_name": "Step 5 Preview", "lab": "StepFun", "open_weights": true, "row_id": "row-nj1v5q", "model_released": "2026-09-20", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1o5c47s", "source_n": 23 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash (DSH PTC)", "effort": "max", "harness": "lab-internal", "score_pct": 67.6, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |", "notes": "Evaluated with DeepSeek Harness Program-Test-Cycle (DSH PTC), reasoning_effort=100, N=8, temp=1.0, top_p=0.95, 1M context, max_steps=500.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-1r66635", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "kimi-k3", "model_as_reported": "Kimi K3", "effort": "max", "harness": "lab-internal", "score_pct": 67.5, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Kimi-K3 HuggingFace Model Card", "source_url": "https://huggingface.co/moonshotai/Kimi-K3", "published": "2026-07-23", "accessed": "2026-10-09", "quote": "DeepSWE | 67.5 | 70.0 | 73.0 | 59.0 | 67.0 | 46.2", "notes": "Footnote 2: DeepSWE. Kimi K3 is evaluated with the Kimi Code harness. We report the DeepSWE v1.1 tasks. All Kimi K3 results are obtained with reasoning effort set to 'max', temperature = 1.0, top-p = 1.0 for agentic tasks.", "display_name": "Kimi K3", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-ub0egg", "model_released": "2026-07-18", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-kyj4bu", "source_n": 8 }, { "model_key": "grok-4.6", "model_as_reported": "grok-4-6", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 67.48, "ci_pct": 2.28, "n_samples": 4, "cost_per_task_usd": 3.449, "tokens_per_task": 5774959, "input_tokens_per_task": 5725195, "output_tokens_per_task": 49764, "steps_per_task": 70.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-12", "accessed": "2026-10-09", "quote": "config mini_swe_agent_grok_4_6_medium: pass_at_1 0.6747787610619469, ci_half 0.0228080457287792, n_runs 4, mean_cost_usd 3.4489837522123894", "notes": "pass@4 84.1%; mean 15 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-cax4uv", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-fable-5.1", "model_as_reported": "Claude Fable 5.1", "effort": "max", "harness": "lab-internal", "score_pct": 67.4, "ci_pct": null, "n_samples": 5, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Claude Fable 5.1 & Claude Mythos 5.1 System Card", "source_url": "https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32c215d14235/Claude%20Fable%205.1%20&%20Claude%20Mythos%205.1%20System%20Card.pdf", "published": "2026-09-01", "accessed": "2026-10-09", "quote": "Fable 5.1 scored an average of 67.4% over five trials. A note on scoring: DeepSWE grades each task with hidden tests that are often written for a single reference solution. When we reviewed failing transcripts, we found that Fable 5.1 implemented some ambiguous tasks more thoroughly than the task required, for example by validating inputs and raising exceptions, or by keeping new code consistent with codebase conventions.", "notes": "Section 8.3 DeepSWE v1.1 (p. 168). Also noted over-implementation penalty where more rigorous implementations failed single-solution reference tests.", "display_name": "Claude Fable 5.1", "lab": "Anthropic", "open_weights": false, "row_id": "row-2f9pfs", "model_released": "2026-09-01", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1b2qf5e", "source_n": 17 }, { "model_key": "claude-fable-5", "model_as_reported": "Fable 5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 67.3, "ci_pct": 7.2, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Fable 5 | effort: max | score: 67.3% ± 7.2% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Fable 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1e5knmg", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-fable-5.1", "model_as_reported": "Fable 5.1", "effort": "high", "harness": "mini-swe-agent", "score_pct": 67.3, "ci_pct": 6.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Fable 5.1 | effort: high | score: 67.3% ± 6.9% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Fable 5.1", "lab": "Anthropic", "open_weights": false, "row_id": "row-t03qkb", "model_released": "2026-09-01", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "gpt-5-6-luna", "effort": "max", "harness": "mini-swe-agent", "score_pct": 67.19, "ci_pct": 3.99, "n_samples": 4, "cost_per_task_usd": 0.6056, "tokens_per_task": 15517118, "input_tokens_per_task": 15443718, "output_tokens_per_task": 73400, "steps_per_task": 101.7, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_luna_max: pass_at_1 0.671875, ci_half 0.03992001235671055, n_runs 4, mean_cost_usd 0.6056233620535714", "notes": "pass@4 90.3%; mean 19 min per attempt; leaderboard-live.json gives $3.03 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-1lhlm4s", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.5", "model_as_reported": "gpt-5-5", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 67.04, "ci_pct": 6.47, "n_samples": 4, "cost_per_task_usd": 7.2262, "tokens_per_task": 8419222, "input_tokens_per_task": 8372928, "output_tokens_per_task": 46295, "steps_per_task": 82, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_5_xhigh: pass_at_1 0.6703539823008849, ci_half 0.06465646340908864, n_runs 4, mean_cost_usd 7.226236674778761", "notes": "pass@4 88.5%; mean 30 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.5", "lab": "OpenAI", "open_weights": false, "row_id": "row-1st9iuc", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-astra", "model_as_reported": "gpt-6-astra", "effort": "low", "harness": "mini-swe-agent", "score_pct": 67.04, "ci_pct": 1.3, "n_samples": 4, "cost_per_task_usd": 1.5952, "tokens_per_task": 505086, "input_tokens_per_task": 494507, "output_tokens_per_task": 10580, "steps_per_task": 19.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-09-03", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_6_astra_low: pass_at_1 0.6703539823008849, ci_half 0.013008610516858731, n_runs 4, mean_cost_usd 1.595223439159292", "notes": "pass@4 79.6%; mean 10 min per attempt; cost basis: Current pricing at all context lengths: $10/M uncached input, $12.50/M cache writes, $1/M cache reads, $50/M output; no separate compute-unit fee.; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-6 Astra", "lab": "OpenAI", "open_weights": false, "row_id": "row-tb7k4q", "model_released": "2026-09-03", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-luna", "model_as_reported": "GPT 6 Luna", "effort": "max", "harness": "mini-swe-agent", "score_pct": 67, "ci_pct": 6.8, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 6 Luna | effort: max | score: 67% ± 6.8% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-12m5fyd", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "glm-5.3", "model_as_reported": "GLM-5.3", "effort": "max", "harness": "mini-swe-agent", "score_pct": 66.9, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "GLM-5.3 HuggingFace Model Card", "source_url": "https://huggingface.co/zai-org/GLM-5.3", "published": "2026-08-25", "accessed": "2026-10-09", "quote": "| DeepSWE (v1.1) | 66.9 | 46.2 | 67.5 | 62.7 | 56.6 | 58.0 | 69.7 | **72.7** |", "notes": "Footnote: We run DeepSWE using the mini-swe-agent harness with temperature=0.95, top_p=1.0, timeout=6h and 400K context. (Official board re-run reached 69.0% ±3% on mini-swe-agent as reported in CodingFleet).", "display_name": "GLM-5.3", "lab": "Z.ai", "open_weights": true, "row_id": "row-1so5ru7", "model_released": "2026-08-20", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1171e08", "source_n": 15 }, { "model_key": "grok-4.6", "model_as_reported": "grok-4-6", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 66.74, "ci_pct": 2.18, "n_samples": 4, "cost_per_task_usd": 5.4977, "tokens_per_task": 9415426, "input_tokens_per_task": 9344022, "output_tokens_per_task": 71404, "steps_per_task": 87.2, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-12", "accessed": "2026-10-09", "quote": "config mini_swe_agent_grok_4_6_xhigh: pass_at_1 0.6674057649667405, ci_half 0.02175684224405049, n_runs 4, mean_cost_usd 5.497668745011087", "notes": "pass@4 85%; mean 21 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-huu76o", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-luna", "model_as_reported": "GPT-6 Luna", "effort": "max", "harness": "lab-internal", "score_pct": 66.59, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.2169, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Luna\",\"cost_label\":\"$$0.2169\",\"score\":0.6659,\"cost\":0.2169,\"effortLabel\":\"max\"}", "notes": "Reported in embedded Vega-Lite chart in GPT-6 Sol and Luna launch post.", "display_name": "GPT-6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-m482jr", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "gpt-6-sol", "model_as_reported": "GPT-6 Sol", "effort": "xhigh", "harness": "lab-internal", "score_pct": 66.59, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 1.0033, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Sol\",\"cost_label\":\"$$1.00\",\"score\":0.6659,\"cost\":1.0033,\"effortLabel\":\"xhigh\"}", "notes": "Reported in embedded Vega-Lite chart and text: 'On DeepSWE v1.1... GPT-6 Sol at max effort scores 68.8%, within 1.1 percentage points of Claude Fable 5’s highest score in the evaluation—69.9% at xhigh effort—at approximately 80% lower cost per task.'", "display_name": "GPT-6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-1d2hsmx", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "kimi-k3", "model_as_reported": "K3 max", "effort": "max", "harness": "mini-swe-agent", "score_pct": 66.4, "ci_pct": null, "n_samples": 113, "cost_per_task_usd": 4.74, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Fireworks AI Ember-1 Announcement", "source_url": "https://fireworks.ai/blog/ember-1", "published": "2026-09-23", "accessed": "2026-10-09", "quote": "DeepSWE 1.1 | N=113 | K3 Low: 55.8% | K3 High: 62.8% | K3 max: 66.4% | Ember-1: 75.2% | -23.7% / -126.9 USD", "notes": "Evaluated by Fireworks AI across 113 tasks. Fireworks reports total run cost was $535.44 ($4.74/task), based on Kimi rates $3.00/M uncached, $0.30/M cached, $15.00/M output.", "display_name": "Kimi K3", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-11vuxf9", "model_released": "2026-07-18", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1vlr43v", "source_n": 28 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash (Pi)", "effort": "max", "harness": "Pi", "score_pct": 66.2, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |", "notes": "Evaluated with Pi scaffold, reasoning_effort=100, N=8, temp=1.0, top_p=0.95, 1M context, max_steps=500.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-1wpexpo", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "gpt-5.5", "model_as_reported": "GPT 5.5", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 66.1, "ci_pct": 6.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 5.5 | effort: xhigh | score: 66.1% ± 6.9% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-5.5", "lab": "OpenAI", "open_weights": false, "row_id": "row-e5sg10", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "grok-4.6", "model_as_reported": "Grok 4.6", "effort": "high", "harness": "mini-swe-agent", "score_pct": 65.9, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "xAI Grok 4.6 Model Card", "source_url": "https://media.x.ai/v1/website/card-4p6-4cd2dc57.pdf", "published": "2026-08-18", "accessed": "2026-10-09", "quote": "Grok 4.6 is evaluated with the mini-swe-agent harness. Grok 4.6 scores 65.9% at high thinking effort and 67.0% at xhigh thinking effort.", "notes": "Section 2.4 DeepSWE v1.1 (p. 11). Score at high thinking effort (reported as 65.2% in Grok 4.7 comparison table).", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-18oxn14", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-tt3io3", "source_n": 13 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash (Codex)", "effort": "max", "harness": "Codex", "score_pct": 65.6, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |", "notes": "Evaluated with Codex scaffold, reasoning_effort=100, N=8, temp=1.0, top_p=0.95, 1M context, max_steps=500.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-ils0lb", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "nex-n2.5-max", "model_as_reported": "Nex-N2.5-Max", "effort": "", "harness": "NexAU", "score_pct": 65.6, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Nex-AGI Nex-N2.5 GitHub README / Model Card", "source_url": "https://github.com/nex-agi/Nex-N2.5", "published": "2026-09-08", "accessed": "2026-10-09", "quote": "DeepSWE v1.1 | 36.1 | 55.8 | 65.6 | 73.7 | 72.7 | 67.5 | 66.9 | 62.8 | 69.3", "notes": "Coding tasks evaluated using the NexAU harness. Sampling parameters: temperature = 0.7, top_p = 0.95, top_k = 40. 1.6T parameter text-only MoE foundation model.", "display_name": "Nex-N2.5-Max", "lab": "Nex-AGI", "open_weights": true, "row_id": "row-1ogzpoy", "model_released": "2026-09-08", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-pixn6e", "source_n": 19 }, { "model_key": "deepseek-v4.1-flash", "model_as_reported": "DS-V4.1-Flash (OpenCode)", "effort": "max", "harness": "OpenCode", "score_pct": 65.5, "ci_pct": null, "n_samples": 8, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4.1-Flash HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "| DeepSWE v1.1 (Resolved) | 69.8 | 65.6 | 65.5 | 66.2 | 74.2 | 72.6 | 70.5 | 67.6 |", "notes": "Evaluated with OpenCode scaffold, reasoning_effort=100, N=8, temp=1.0, top_p=0.95, 1M context, max_steps=500.", "display_name": "DeepSeek V4.1 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-1n2qy16", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ftq9xa", "source_n": 21 }, { "model_key": "gemini-3.7-flash", "model_as_reported": "Gemini 3.7 Flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 65.5, "ci_pct": 7.1, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Gemini 3.7 Flash | effort: high | score: 65.5% ± 7.1% | n_samples: 339 | provider: Google", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Gemini 3.7 Flash", "lab": "Google", "open_weights": false, "row_id": "row-1rhhb5e", "model_released": "2026-08-13", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gemini-3.7-flash", "model_as_reported": "gemini-3-7-flash", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 65.49, "ci_pct": 3.09, "n_samples": 4, "cost_per_task_usd": 2.0251, "tokens_per_task": 15426057, "input_tokens_per_task": 15332066, "output_tokens_per_task": 93991, "steps_per_task": 117.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-13", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_7_flash_medium: pass_at_1 0.6548672566371682, ci_half 0.030865322764155222, n_runs 4, mean_cost_usd 2.0251351826880533", "notes": "pass@4 83.2%; mean 21 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.7 Flash", "lab": "Google", "open_weights": false, "row_id": "row-1q9if39", "model_released": "2026-08-13", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-fable-5", "model_as_reported": "claude-fable-5", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 65.37, "ci_pct": 4.42, "n_samples": 4, "cost_per_task_usd": 6.0882, "tokens_per_task": 2982453, "input_tokens_per_task": 2942252, "output_tokens_per_task": 40201, "steps_per_task": 48.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_fable_5_medium: pass_at_1 0.6536697247706422, ci_half 0.044219288792381656, n_runs 4, mean_cost_usd 6.088186528935186", "notes": "pass@4 83.2%; mean 14 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Fable 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-f4ic5k", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gemini-3.7-flash", "model_as_reported": "gemini-3-7-flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 65.27, "ci_pct": 1.79, "n_samples": 4, "cost_per_task_usd": 2.1763, "tokens_per_task": 17107101, "input_tokens_per_task": 16999853, "output_tokens_per_task": 107248, "steps_per_task": 124.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-13", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_7_flash_high: pass_at_1 0.6526548672566371, ci_half 0.017878625067843174, n_runs 4, mean_cost_usd 2.1762860351216813", "notes": "pass@4 82.3%; mean 21 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.7 Flash", "lab": "Google", "open_weights": false, "row_id": "row-z10ec", "model_released": "2026-08-13", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-sol", "model_as_reported": "GPT-6 Sol", "effort": "high", "harness": "lab-internal", "score_pct": 65.27, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.6404, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Sol\",\"cost_label\":\"$$0.64\",\"score\":0.6527,\"cost\":0.6404,\"effortLabel\":\"high\"}", "notes": "Reported in embedded Vega-Lite chart and text: 'On DeepSWE v1.1... GPT-6 Sol at max effort scores 68.8%, within 1.1 percentage points of Claude Fable 5’s highest score in the evaluation—69.9% at xhigh effort—at approximately 80% lower cost per task.'", "display_name": "GPT-6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-wxw9bn", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "grok-4.6", "model_as_reported": "grok-4-6", "effort": "high", "harness": "mini-swe-agent", "score_pct": 65.19, "ci_pct": 1.53, "n_samples": 4, "cost_per_task_usd": 4.3849, "tokens_per_task": 7409649, "input_tokens_per_task": 7348488, "output_tokens_per_task": 61161, "steps_per_task": 79, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-12", "accessed": "2026-10-09", "quote": "config mini_swe_agent_grok_4_6_high: pass_at_1 0.6518847006651884, ci_half 0.015336008186572951, n_runs 4, mean_cost_usd 4.3848732239467845", "notes": "pass@4 85%; mean 18 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-12wnw3m", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "grok-4.6", "model_as_reported": "Grok 4.6 (Grok Build)", "effort": "high", "harness": "Grok Build", "score_pct": 65, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Artificial Analysis (articles/benchmarking-grok-4-7)", "source_url": "https://artificialanalysis.ai/articles/benchmarking-grok-4-7", "published": "2026-09-21", "accessed": "2026-10-09", "quote": "Grok 4.6 | DeepSWE v1.1: 65.0% | Evaluated with Grok Build | Artificial Analysis Coding Agent Index", "notes": "Artificial Analysis benchmark baseline for Grok 4.6 on DeepSWE v1.1 component of Coding Agent Index.", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-69o7y9", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-17v6sj", "source_n": 36 }, { "model_key": "claude-fable-5.1", "model_as_reported": "Fable 5.1", "effort": "max", "harness": "mini-swe-agent", "score_pct": 64.9, "ci_pct": 7.2, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Fable 5.1 | effort: max | score: 64.9% ± 7.2% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Fable 5.1", "lab": "Anthropic", "open_weights": false, "row_id": "row-q5njty", "model_released": "2026-09-01", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.5", "model_as_reported": "gpt-5-5", "effort": "high", "harness": "mini-swe-agent", "score_pct": 64.38, "ci_pct": 3.12, "n_samples": 4, "cost_per_task_usd": 5.1004, "tokens_per_task": 4648941, "input_tokens_per_task": 4617782, "output_tokens_per_task": 31159, "steps_per_task": 61.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_5_high: pass_at_1 0.6438053097345132, ci_half 0.031168426495054677, n_runs 4, mean_cost_usd 5.100437004424779", "notes": "pass@4 90.3%; mean 34 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.5", "lab": "OpenAI", "open_weights": false, "row_id": "row-10u6f80", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6.1-sol", "model_as_reported": "GPT-6.1 Sol", "effort": "low", "harness": "lab-internal", "score_pct": 64.38, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.1714, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6.1 Sol launch post", "source_url": "https://openai.com/index/introducing-gpt-6-1-sol/", "published": "2026-09-29", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6.1 Sol\",\"modelLabel\":\"GPT-6.1 Sol\",\"cost\":0.1714,\"score\":0.6438,\"effortLabel\":\"Low\"}", "notes": "Reported in embedded Vega-Lite spec on launch page.", "display_name": "GPT-6.1 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-1u8z7hs", "model_released": "2026-09-29", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1ntqzs0", "source_n": 31 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "GPT 5.6 Luna", "effort": "max", "harness": "mini-swe-agent", "score_pct": 64.3, "ci_pct": 7.4, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 5.6 Luna | effort: max | score: 64.3% ± 7.4% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-y14yn6", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "hy4-preview", "model_as_reported": "Hy4 preview", "effort": "", "harness": "lab-internal", "score_pct": 64.3, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Tencent Hy4-preview Technical Report & Model Card", "source_url": "https://huggingface.co/tencent/Hy4-preview", "published": "2026-08-27", "accessed": "2026-10-09", "quote": "Hy4 preview achieved 64.3% on DeepSWE in the official Benchmark Appendix (up from 28.0% on Hy3).", "notes": "770B total / 49B active MoE flagship model. Evaluated on DeepSWE in official Benchmark Appendix.", "display_name": "Hy4 Preview", "lab": "Tencent", "open_weights": true, "row_id": "row-lcrv5b", "model_released": "2026-08-27", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-p3aj15", "source_n": 16 }, { "model_key": "grok-4.6", "model_as_reported": "Grok 4.6", "effort": "high", "harness": "mini-swe-agent", "score_pct": 63.7, "ci_pct": 7.4, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Grok 4.6 | effort: high | score: 63.7% ± 7.4% | n_samples: 339 | provider: xAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-2clyyf", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "glm-5.3-flash", "model_as_reported": "glm-5-3-flash", "effort": "max", "harness": "mini-swe-agent", "score_pct": 63.39, "ci_pct": 4.38, "n_samples": 4, "cost_per_task_usd": 0.241, "tokens_per_task": 12459855, "input_tokens_per_task": 12387025, "output_tokens_per_task": 72830, "steps_per_task": 122.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-26", "accessed": "2026-10-09", "quote": "config mini_swe_agent_glm_5_3_flash_max: pass_at_1 0.6339285714285714, ci_half 0.04377970751693644, n_runs 4, mean_cost_usd 0.24098185622767856", "notes": "pass@4 85%; mean 26 min per attempt; leaderboard-live.json gives $0.48 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GLM-5.3 Flash", "lab": "Z.ai", "open_weights": true, "row_id": "row-pa1eq4", "model_released": "2026-08-26", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "deepseek-v4-pro", "model_as_reported": "deepseek-v4-pro", "effort": "max", "harness": "mini-swe-agent", "score_pct": 62.83, "ci_pct": 6.33, "n_samples": 4, "cost_per_task_usd": 1.666, "tokens_per_task": 24297605, "input_tokens_per_task": 24191606, "output_tokens_per_task": 105999, "steps_per_task": 154.7, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-12", "accessed": "2026-10-09", "quote": "config mini_swe_agent_deepseek_v4_pro_max: pass_at_1 0.6283185840707964, ci_half 0.06333430597228876, n_runs 4, mean_cost_usd 1.6660232187256647", "notes": "pass@4 88.5%; mean 37 min per attempt; leaderboard-live.json gives $0.24 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "DeepSeek V4 Pro", "lab": "DeepSeek", "open_weights": true, "row_id": "row-eakg9h", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "kimi-k3", "model_as_reported": "K3 High", "effort": "high", "harness": "mini-swe-agent", "score_pct": 62.8, "ci_pct": null, "n_samples": 113, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Fireworks AI Ember-1 Announcement", "source_url": "https://fireworks.ai/blog/ember-1", "published": "2026-09-23", "accessed": "2026-10-09", "quote": "DeepSWE 1.1 | N=113 | K3 Low: 55.8% | K3 High: 62.8% | K3 max: 66.4% | Ember-1: 75.2%", "notes": "Evaluated by Fireworks AI across all 113 DeepSWE 1.1 tasks under high reasoning effort configuration.", "display_name": "Kimi K3", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-2nx1rj", "model_released": "2026-07-18", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1vlr43v", "source_n": 28 }, { "model_key": "deepseek-v4-pro", "model_as_reported": "DeepSeek-V4-Pro-0813", "effort": "max", "harness": "DeepSeek Harness (minimal mode)", "score_pct": 62.7, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4-Pro-0813 HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813", "published": "2026-08-13", "accessed": "2026-10-10", "quote": "| \"DeepSWE\" | \"62.7\" | \"54.4\" | \"12.8\" | \"7.3\" | \"46.2\" | \"67.5\" | \"58.0\" | \"70.0\" |; Note 1: \"For the code-agent tasks among the public benchmarks above, DeepSeek-V4-Pro-0813 is evaluated with the minimal mode of DeepSeek Harness as the agent framework, using the max reasoning effort level with temperature = 1.0, top_p = 0.95.\"", "notes": "Lab self-reported on official HF model card. Evaluated using DeepSeek Harness minimal mode at max reasoning effort (temp=1.0, top_p=0.95, 384K output token ceiling). Previous tracker only had Mercor 56.3.", "display_name": "DeepSeek V4 Pro", "lab": "DeepSeek", "open_weights": true, "row_id": "row-8oa950", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-7akg1l", "source_n": 12 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "GPT-5.6 Luna", "effort": "max", "harness": "lab-internal", "score_pct": 62.17, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.532, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-5.6 Luna\",\"cost_label\":\"$$0.532\",\"score\":0.6217,\"cost\":0.532,\"effortLabel\":\"max\"}", "notes": "OpenAI re-ran GPT-5.6 Luna internally for its GPT-6 Sol/Luna post; the official board measured 67.2% at max. Explains discrepancy: 62.17% was OpenAI internal re-evaluation chart data alongside GPT-6 Sol/Luna, whereas 67.2% (67.19%) was Datacurve official board 4-pass verified rollout.", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-1csb5tq", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "mistral-large-4", "model_as_reported": "Mistral Large 4", "effort": "thinking", "harness": "partner eval (Artificial Analysis, Surge AI)", "score_pct": 61.7, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Mistral Large 4 Announcement", "source_url": "https://mistral.ai/news/mistral-large-4/", "published": "2026-10-06", "accessed": "2026-10-09", "quote": "scoring 61.7% on DeepSWE v1.1, 59.4% on SWE-Atlas-QnA, and 28.3% on Terminal-Bench 4.", "notes": "Mistral reports the run was done with Artificial Analysis and Surge AI. Evaluated in partnership with Artificial Analysis and Surge AI using Artificial Analysis coding agent harness. Sole Mistral model with DeepSWE 1.1 score.; harness as reported: other name", "display_name": "Mistral Large 4", "lab": "Mistral", "open_weights": true, "row_id": "row-2arw03", "model_released": "2026-10-06", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-d2tk0p", "source_n": 33 }, { "model_key": "gpt-6-luna", "model_as_reported": "GPT-6 Luna", "effort": "xhigh", "harness": "lab-internal", "score_pct": 61.28, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.1096, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Luna\",\"cost_label\":\"$$0.1096\",\"score\":0.6128,\"cost\":0.1096,\"effortLabel\":\"xhigh\"}", "notes": "Reported in embedded Vega-Lite chart in GPT-6 Sol and Luna launch post.", "display_name": "GPT-6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-ddelns", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "gpt-5.6-sol", "model_as_reported": "gpt-5-6-sol", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 61.06, "ci_pct": 1.58, "n_samples": 4, "cost_per_task_usd": 1.4158, "tokens_per_task": 1524219, "input_tokens_per_task": 1505794, "output_tokens_per_task": 18425, "steps_per_task": 30.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_sol_medium: pass_at_1 0.6106194690265486, ci_half 0.01583357649307215, n_runs 4, mean_cost_usd 1.4157802656590275", "notes": "pass@4 80.5%; mean 7 min per attempt; leaderboard-live.json gives $1.86 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-5a37tj", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-terra", "model_as_reported": "gpt-5-6-terra", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 60.18, "ci_pct": 2.12, "n_samples": 4, "cost_per_task_usd": 1.7018, "tokens_per_task": 3285972, "input_tokens_per_task": 3246355, "output_tokens_per_task": 39617, "steps_per_task": 43.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_terra_xhigh: pass_at_1 0.6017699115044248, ci_half 0.021242972019271257, n_runs 4, mean_cost_usd 1.7017533805309735", "notes": "pass@4 80.5%; mean 10 min per attempt; leaderboard-live.json gives $2.13 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Terra", "lab": "OpenAI", "open_weights": false, "row_id": "row-ouwfw", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-haiku-5.5", "model_as_reported": "Haiku 5.5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 59.9, "ci_pct": 7.7, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX OSS DeepSWE Leaderboard", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Haiku 5.5 | effort: max | score: 59.9% ± 7.7% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Independent run on Mercor APEX. Confirms no Anthropic official DeepSWE figure exists.", "display_name": "Claude Haiku 5.5", "lab": "Anthropic", "open_weights": false, "row_id": "row-bm6mdq", "model_released": "2026-09-29", "model_released_basis": "stated", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-4.8", "model_as_reported": "Opus 4.8", "effort": "max", "harness": "mini-swe-agent", "score_pct": 59.6, "ci_pct": 7.1, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Opus 4.8 | effort: max | score: 59.6% ± 7.1% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Opus 4.8", "lab": "Anthropic", "open_weights": false, "row_id": "row-1gtq4fm", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-fable-5", "model_as_reported": "claude-fable-5", "effort": "low", "harness": "mini-swe-agent", "score_pct": 59.58, "ci_pct": 2.79, "n_samples": 4, "cost_per_task_usd": 3.7579, "tokens_per_task": 1716493, "input_tokens_per_task": 1691250, "output_tokens_per_task": 25243, "steps_per_task": 37.8, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_fable_5_low: pass_at_1 0.5958429561200924, ci_half 0.027940054451297294, n_runs 4, mean_cost_usd 3.757873125874126", "notes": "pass@4 81.4%; mean 11 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Fable 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-18j1o47", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-luna", "model_as_reported": "GPT-6 Luna", "effort": "high", "harness": "lab-internal", "score_pct": 59.29, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.0838, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Luna\",\"cost_label\":\"$$0.0838\",\"score\":0.5929,\"cost\":0.0838,\"effortLabel\":\"high\"}", "notes": "Reported in embedded Vega-Lite chart in GPT-6 Sol and Luna launch post.", "display_name": "GPT-6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-3vfxsy", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "claude-opus-4.8", "model_as_reported": "claude-opus-4-8", "effort": "max", "harness": "mini-swe-agent", "score_pct": 58.97, "ci_pct": 1.76, "n_samples": 4, "cost_per_task_usd": 13.2226, "tokens_per_task": 17182970, "input_tokens_per_task": 17047939, "output_tokens_per_task": 135032, "steps_per_task": 120, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_4_8_max: pass_at_1 0.5897435897435898, ci_half 0.01764815347429254, n_runs 4, mean_cost_usd 13.222583593240094", "notes": "pass@4 79.3%; mean 58 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 4.8", "lab": "Anthropic", "open_weights": false, "row_id": "row-1ojpt5b", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "qwen3.8-flash-next", "model_as_reported": "Qwen3.8-Flash-Next", "effort": "", "harness": "mini-swe-agent", "score_pct": 58.7, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Qwen3.8-Flash-Next HuggingFace Model Card", "source_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next", "published": "2026-08-24", "accessed": "2026-10-09", "quote": "
Agentic coding
DeepSWE 1.1
\n58.7", "notes": "Footnote 1: DeepSWE 1.1: evaluated with the Claude Code and mini-SWE-agent harnesses, temp=1.0, top_p=0.95, 256K context window. We report the highest score across the two harnesses; notably, Qwen3.8-Flash-Next performs best on mini-SWE-agent.", "display_name": "Qwen3.8-Flash-Next", "lab": "Alibaba", "open_weights": true, "row_id": "row-kipywi", "model_released": "2026-08-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-qxom7y", "source_n": 14 }, { "model_key": "claude-opus-5", "model_as_reported": "claude-opus-5", "effort": "low", "harness": "mini-swe-agent", "score_pct": 58.13, "ci_pct": 2.33, "n_samples": 4, "cost_per_task_usd": 1.6626, "tokens_per_task": 1584172, "input_tokens_per_task": 1564288, "output_tokens_per_task": 19884, "steps_per_task": 35.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-25", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_5_low: pass_at_1 0.5812917594654788, ci_half 0.023306879466842723, n_runs 4, mean_cost_usd 1.6626223919821828", "notes": "pass@4 85%; mean 8 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-blybci", "model_released": "2026-07-24", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.5", "model_as_reported": "GPT-5.5", "effort": "high", "harness": "Codex", "score_pct": 57.5, "ci_pct": 5, "n_samples": 4, "cost_per_task_usd": 3.71, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Sleev Performance & Benchmarks", "source_url": "https://sleev.ai/performance", "published": "", "accessed": "2026-10-10", "quote": "Standard runs: one task per context window running GPT-5.5 at high reasoning. Baseline 57.5% ± 5.0; mean cost per task $3.71", "notes": "Baseline (without Sleev) in Sleev's tool comparison; Codex CLI replaces DeepSWE's reference harness, so Sleev says the figures are not comparable to the official board; 4 runs, 95% interval; runs 21 Jun to 2 Jul 2026.; harness as reported: Codex CLI", "display_name": "GPT-5.5", "lab": "OpenAI", "open_weights": false, "row_id": "row-ve85ra", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-182356w", "source_n": 38 }, { "model_key": "qwen3.8-max", "model_as_reported": "qwen3-8-max", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 57.46, "ci_pct": 2.66, "n_samples": 4, "cost_per_task_usd": 3.7291, "tokens_per_task": 13091142, "input_tokens_per_task": 12996066, "output_tokens_per_task": 95075, "steps_per_task": 111.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-04", "accessed": "2026-10-09", "quote": "config mini_swe_agent_qwen3_8_max_xhigh: pass_at_1 0.5746102449888641, ci_half 0.026627926725075454, n_runs 4, mean_cost_usd 3.7290646500445437", "notes": "pass@4 83.2%; mean 43 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Qwen3.8 Max", "lab": "Alibaba", "open_weights": true, "row_id": "row-1nr3gb4", "model_released": "2026-08-04", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "gpt-5-6-luna", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 56.86, "ci_pct": 2.17, "n_samples": 4, "cost_per_task_usd": 0.3071, "tokens_per_task": 7655255, "input_tokens_per_task": 7610578, "output_tokens_per_task": 44678, "steps_per_task": 71.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_luna_xhigh: pass_at_1 0.5685840707964602, ci_half 0.021681017528097975, n_runs 4, mean_cost_usd 0.30712385371681417", "notes": "pass@4 78.8%; mean 12 min per attempt; leaderboard-live.json gives $1.54 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-19e0252", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-6-sol", "model_as_reported": "GPT-6 Sol", "effort": "medium", "harness": "lab-internal", "score_pct": 56.64, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.3798, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Sol\",\"cost_label\":\"$$0.38\",\"score\":0.5664,\"cost\":0.3798,\"effortLabel\":\"medium\"}", "notes": "Reported in embedded Vega-Lite chart and text: 'On DeepSWE v1.1... GPT-6 Sol at max effort scores 68.8%, within 1.1 percentage points of Claude Fable 5’s highest score in the evaluation—69.9% at xhigh effort—at approximately 80% lower cost per task.'", "display_name": "GPT-6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-vphi25", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "deepseek-v4-flash", "model_as_reported": "DeepSeek V4 Flash", "effort": "max", "harness": "mini-swe-agent", "score_pct": 56.6, "ci_pct": 8.8, "n_samples": 113, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "DeepSeek V4 Flash | effort: max | score: 56.6% ± 8.8% | n_samples: 113 | provider: DeepSeek", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "DeepSeek V4 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-1vnfpxa", "model_released": "2026-08-06", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "qwen3.8-max", "model_as_reported": "Qwen3.8-Max", "effort": "", "harness": "Claude Code", "score_pct": 56.6, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Qwen3.8-2.4T-A95B (Qwen3.8-Max) HuggingFace Model Card", "source_url": "https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B", "published": "2026-08-08", "accessed": "2026-10-09", "quote": "DeepSWE 1.1 | 59.0 | 70.0 | 73.0 | 21.6 | 56.6", "notes": "Footnote 4: DeepSWE 1.1: Evaluated with the Claude Code and mini-SWE-agent harnesses, temp=1.0, top_p=0.95, and a 256K context window. We report the highest score among both harnesses; notably, Qwen3.8-Max performs best on Claude Code.", "display_name": "Qwen3.8 Max", "lab": "Alibaba", "open_weights": true, "row_id": "row-17gjj3j", "model_released": "2026-08-04", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-ouk7wk", "source_n": 11 }, { "model_key": "deepseek-v4-pro", "model_as_reported": "DeepSeek V4 Pro 0813", "effort": "max", "harness": "mini-swe-agent", "score_pct": 56.3, "ci_pct": 7.2, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "DeepSeek V4 Pro 0813 | effort: max | score: 56.3% ± 7.2% | n_samples: 339 | provider: DeepSeek", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "DeepSeek V4 Pro", "lab": "DeepSeek", "open_weights": true, "row_id": "row-gjm39c", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "GPT-5.6 Luna", "effort": "xhigh", "harness": "lab-internal", "score_pct": 56.19, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.2737, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-5.6 Luna\",\"cost_label\":\"$$0.2737\",\"score\":0.5619,\"cost\":0.2737,\"effortLabel\":\"xhigh\"}", "notes": "OpenAI re-ran GPT-5.6 Luna internally for its GPT-6 Sol/Luna post; the official board measured 67.2% at max. Full effort breakdown from low to max reported in GPT-6 Sol and Luna launch post chart data.", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-15b4q5d", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "kimi-k3", "model_as_reported": "K3 Low", "effort": "low", "harness": "mini-swe-agent", "score_pct": 55.8, "ci_pct": null, "n_samples": 113, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Fireworks AI Ember-1 Announcement", "source_url": "https://fireworks.ai/blog/ember-1", "published": "2026-09-23", "accessed": "2026-10-09", "quote": "DeepSWE 1.1 | N=113 | K3 Low: 55.8% | K3 High: 62.8% | K3 max: 66.4% | Ember-1: 75.2%", "notes": "Evaluated by Fireworks AI across all 113 DeepSWE 1.1 tasks under low reasoning effort configuration.", "display_name": "Kimi K3", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-uvwqhb", "model_released": "2026-07-18", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1vlr43v", "source_n": 28 }, { "model_key": "nex-n2.5-pro", "model_as_reported": "Nex-N2.5-Pro", "effort": "", "harness": "NexAU", "score_pct": 55.8, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Nex-AGI Nex-N2.5 GitHub README / Model Card", "source_url": "https://github.com/nex-agi/Nex-N2.5", "published": "2026-09-08", "accessed": "2026-10-09", "quote": "DeepSWE v1.1 | 36.1 | 55.8 | 65.6 | 73.7 | 72.7 | 67.5 | 66.9 | 62.8 | 69.3", "notes": "Coding tasks evaluated using the NexAU harness. Sampling parameters: temperature = 0.7, top_p = 0.95, top_k = 40. Mid-sized multimodal agentic model.", "display_name": "Nex-N2.5-Pro", "lab": "Nex-AGI", "open_weights": true, "row_id": "row-1bzp4ea", "model_released": "2026-09-08", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-pixn6e", "source_n": 19 }, { "model_key": "muse-spark-1.2", "model_as_reported": "muse-spark-1-2", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 54.87, "ci_pct": 2.12, "n_samples": 4, "cost_per_task_usd": 3.6955, "tokens_per_task": 13793051, "input_tokens_per_task": 13693824, "output_tokens_per_task": 99226, "steps_per_task": 100.8, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-07", "accessed": "2026-10-09", "quote": "config mini_swe_agent_muse_spark_1_2_xhigh: pass_at_1 0.5486725663716814, ci_half 0.02124297201927127, n_runs 4, mean_cost_usd 3.6955048180309733", "notes": "pass@4 81.4%; mean 15 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Muse Spark 1.2", "lab": "Meta", "open_weights": false, "row_id": "row-qte2qf", "model_released": "2026-08-07", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "deepseek-v4-flash", "model_as_reported": "DeepSeek-V4-Flash-0731", "effort": "max", "harness": "DeepSeek Harness (minimal mode)", "score_pct": 54.4, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4-Pro-0813 HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813", "published": "2026-08-13", "accessed": "2026-10-10", "quote": "| \"DeepSWE\" | \"62.7\" | \"54.4\" | \"12.8\" | \"7.3\" | \"46.2\" | \"67.5\" | \"58.0\" | \"70.0\" |; Note 1: \"For the code-agent tasks among the public benchmarks above, DeepSeek-V4-Pro-0813 is evaluated with the minimal mode of DeepSeek Harness as the agent framework, using the max reasoning effort level with temperature = 1.0, top_p = 0.95.\"", "notes": "Second column of the DeepSeek-V4-Pro-0813 card table; same Note 1: DeepSeek Harness minimal mode, max reasoning effort, temperature 1.0, top_p 0.95.", "display_name": "DeepSeek V4 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-111pl4r", "model_released": "2026-08-06", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-7akg1l", "source_n": 12 }, { "model_key": "claude-opus-4.8", "model_as_reported": "claude-opus-4-8", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 54.36, "ci_pct": 3.71, "n_samples": 4, "cost_per_task_usd": 8.0064, "tokens_per_task": 9940741, "input_tokens_per_task": 9854652, "output_tokens_per_task": 86089, "steps_per_task": 94.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_4_8_xhigh: pass_at_1 0.5436241610738255, ci_half 0.03713430628094476, n_runs 4, mean_cost_usd 8.006355704138702", "notes": "pass@4 80.5%; mean 37 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 4.8", "lab": "Anthropic", "open_weights": false, "row_id": "row-f3dla0", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-opus-4.7", "model_as_reported": "claude-opus-4-7", "effort": "max", "harness": "mini-swe-agent", "score_pct": 54.2, "ci_pct": 4.72, "n_samples": null, "cost_per_task_usd": 18.19, "tokens_per_task": 28617242, "input_tokens_per_task": 28513814, "output_tokens_per_task": 103428, "steps_per_task": 202.8, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "claude-opus-4-7 [max]: 54.2% ±4.72%, cost $18.19, output tokens 103428, steps 202.8", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Claude Opus 4.7", "lab": "Anthropic", "open_weights": false, "row_id": "row-z88wg6", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "gpt-5.5", "model_as_reported": "gpt-5-5", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 53.98, "ci_pct": 2.55, "n_samples": 4, "cost_per_task_usd": 2.7493, "tokens_per_task": 2458164, "input_tokens_per_task": 2438539, "output_tokens_per_task": 19625, "steps_per_task": 46, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_5_medium: pass_at_1 0.5398230088495575, ci_half 0.02553087495290979, n_runs 4, mean_cost_usd 2.749342734513274", "notes": "pass@4 77.9%; mean 21 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.5", "lab": "OpenAI", "open_weights": false, "row_id": "row-1meuekt", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-5", "model_as_reported": "claude-sonnet-5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 53.85, "ci_pct": 4.24, "n_samples": 4, "cost_per_task_usd": 26.3999, "tokens_per_task": 72629874, "input_tokens_per_task": 72415756, "output_tokens_per_task": 214118, "steps_per_task": 268.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-02", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_sonnet_5_max: pass_at_1 0.5384615384615384, ci_half 0.04236916470174386, n_runs 4, mean_cost_usd 26.399858950791852", "notes": "pass@4 78.8%; mean 80 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Sonnet 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1q1xs9y", "model_released": "2026-07-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gemini-3.7-flash", "model_as_reported": "gemini-3-7-flash", "effort": "low", "harness": "mini-swe-agent", "score_pct": 53.76, "ci_pct": 2.59, "n_samples": 4, "cost_per_task_usd": 1.8323, "tokens_per_task": 15129930, "input_tokens_per_task": 15056565, "output_tokens_per_task": 73365, "steps_per_task": 130.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-13", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_7_flash_low: pass_at_1 0.5376106194690266, ci_half 0.02589649081831869, n_runs 4, mean_cost_usd 1.8322645629424779", "notes": "pass@4 77.9%; mean 21 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.7 Flash", "lab": "Google", "open_weights": false, "row_id": "row-7fe1ld", "model_released": "2026-08-13", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-terra", "model_as_reported": "gpt-5-6-terra", "effort": "high", "harness": "mini-swe-agent", "score_pct": 53.76, "ci_pct": 4.33, "n_samples": 4, "cost_per_task_usd": 0.9075, "tokens_per_task": 1578570, "input_tokens_per_task": 1557053, "output_tokens_per_task": 21517, "steps_per_task": 33.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_terra_high: pass_at_1 0.5376106194690266, ci_half 0.04328970467213552, n_runs 4, mean_cost_usd 0.9075010212389382", "notes": "pass@4 80.5%; mean 6 min per attempt; leaderboard-live.json gives $1.13 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Terra", "lab": "OpenAI", "open_weights": false, "row_id": "row-1imalva", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "grok-4.5", "model_as_reported": "grok-4-5", "effort": "high", "harness": "mini-swe-agent", "score_pct": 53.76, "ci_pct": 2.28, "n_samples": 4, "cost_per_task_usd": 2.4157, "tokens_per_task": 4094708, "input_tokens_per_task": 4059183, "output_tokens_per_task": 35525, "steps_per_task": 61.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-16", "accessed": "2026-10-09", "quote": "config mini_swe_agent_grok_4_5_high: pass_at_1 0.5376106194690266, ci_half 0.0228080457287792, n_runs 4, mean_cost_usd 2.4157481725663716", "notes": "pass@4 77.9%; mean 8 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Grok 4.5", "lab": "xAI", "open_weights": false, "row_id": "row-1jii3v9", "model_released": "2026-07-16", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "deepseek-v4-flash", "model_as_reported": "deepseek-v4-flash", "effort": "max", "harness": "mini-swe-agent", "score_pct": 53.32, "ci_pct": 3.57, "n_samples": 4, "cost_per_task_usd": 0.4641, "tokens_per_task": 19937618, "input_tokens_per_task": 19829931, "output_tokens_per_task": 107687, "steps_per_task": 152.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-06", "accessed": "2026-10-09", "quote": "config mini_swe_agent_deepseek_v4_flash_max: pass_at_1 0.5331858407079646, ci_half 0.035669502150324287, n_runs 4, mean_cost_usd 0.46407388012739975", "notes": "pass@4 80.5%; mean 24 min per attempt; leaderboard-live.json gives $0.10 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "DeepSeek V4 Flash", "lab": "DeepSeek", "open_weights": true, "row_id": "row-2xh2qw", "model_released": "2026-08-06", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "muse-spark-1.1", "model_as_reported": "muse-spark-1-1", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 53.32, "ci_pct": 3.04, "n_samples": 4, "cost_per_task_usd": 2.3611, "tokens_per_task": 12079408, "input_tokens_per_task": 12005399, "output_tokens_per_task": 74008, "steps_per_task": 95.8, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-14", "accessed": "2026-10-09", "quote": "config mini_swe_agent_muse_spark_1_1_xhigh: pass_at_1 0.5331858407079646, ci_half 0.030353424539337114, n_runs 4, mean_cost_usd 2.361142644469026", "notes": "pass@4 79.6%; mean 15 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Muse Spark 1.1", "lab": "Meta", "open_weights": false, "row_id": "row-ylw86l", "model_released": "2026-07-14", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "qwen3.8-max", "model_as_reported": "Qwen 3.8 Max", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 53.1, "ci_pct": 8.8, "n_samples": 113, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Qwen 3.8 Max | effort: xhigh | score: 53.1% ± 8.8% | n_samples: 113 | provider: Alibaba", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Qwen3.8 Max", "lab": "Alibaba", "open_weights": true, "row_id": "row-e8s2o9", "model_released": "2026-08-04", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-4.8", "model_as_reported": "claude-opus-4-8", "effort": "high", "harness": "mini-swe-agent", "score_pct": 51.77, "ci_pct": 4.56, "n_samples": 4, "cost_per_task_usd": 4.2819, "tokens_per_task": 4958751, "input_tokens_per_task": 4908687, "output_tokens_per_task": 50064, "steps_per_task": 72.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_4_8_high: pass_at_1 0.5176991150442478, ci_half 0.04561609145755843, n_runs 4, mean_cost_usd 4.281940017699115", "notes": "pass@4 77.9%; mean 18 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 4.8", "lab": "Anthropic", "open_weights": false, "row_id": "row-1o3rvd8", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.4", "model_as_reported": "gpt-5-4", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 51.77, "ci_pct": 1.5, "n_samples": 4, "cost_per_task_usd": 5.6525, "tokens_per_task": 9490330, "input_tokens_per_task": 9418921, "output_tokens_per_task": 71409, "steps_per_task": 70.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_4_xhigh: pass_at_1 0.5176991150442478, ci_half 0.015021049567382833, n_runs 4, mean_cost_usd 5.6524577245575225", "notes": "pass@4 77.9%; mean 23 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.4", "lab": "OpenAI", "open_weights": false, "row_id": "row-2ndzjq", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-5", "model_as_reported": "claude-sonnet-5", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 49.67, "ci_pct": 3.45, "n_samples": 4, "cost_per_task_usd": 11.8906, "tokens_per_task": 30816326, "input_tokens_per_task": 30695628, "output_tokens_per_task": 120699, "steps_per_task": 185.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-02", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_sonnet_5_xhigh: pass_at_1 0.49667405764966743, ci_half 0.034545489916778645, n_runs 4, mean_cost_usd 11.890634540022173", "notes": "pass@4 75.2%; mean 40 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Sonnet 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-k22gcp", "model_released": "2026-07-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-5", "model_as_reported": "Sonnet 5", "effort": "max", "harness": "mini-swe-agent", "score_pct": 49.6, "ci_pct": 7.2, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Sonnet 5 | effort: max | score: 49.6% ± 7.2% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Sonnet 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-495gbu", "model_released": "2026-07-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-4.7", "model_as_reported": "Opus 4.7", "effort": "max", "harness": "mini-swe-agent", "score_pct": 49.3, "ci_pct": 6.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Opus 4.7 | effort: max | score: 49.3% ± 6.9% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Opus 4.7", "lab": "Anthropic", "open_weights": false, "row_id": "row-1sa9dgv", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gemini-3.6-flash", "model_as_reported": "Gemini 3.6 Flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 49.3, "ci_pct": 7.1, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Gemini 3.6 Flash | effort: high | score: 49.3% ± 7.1% | n_samples: 339 | provider: Google", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Gemini 3.6 Flash", "lab": "Google", "open_weights": false, "row_id": "row-v4yd6l", "model_released": "2026-07-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "grok-4.5", "model_as_reported": "Grok 4.5", "effort": "high", "harness": "mini-swe-agent", "score_pct": 49.3, "ci_pct": 7.4, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Grok 4.5 | effort: high | score: 49.3% ± 7.4% | n_samples: 339 | provider: xAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Grok 4.5", "lab": "xAI", "open_weights": false, "row_id": "row-dvkuwq", "model_released": "2026-07-16", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.4", "model_as_reported": "GPT 5.4", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 49, "ci_pct": 6.8, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT 5.4 | effort: xhigh | score: 49% ± 6.8% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GPT-5.4", "lab": "OpenAI", "open_weights": false, "row_id": "row-oo7mzf", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-4.8", "model_as_reported": "claude-opus-4-8", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 48.67, "ci_pct": 2.24, "n_samples": 4, "cost_per_task_usd": 3.4439, "tokens_per_task": 3884916, "input_tokens_per_task": 3843603, "output_tokens_per_task": 41313, "steps_per_task": 65.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_4_8_medium: pass_at_1 0.48672566371681414, ci_half 0.02239205861737452, n_runs 4, mean_cost_usd 3.4438571028761062", "notes": "pass@4 76.1%; mean 15 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 4.8", "lab": "Anthropic", "open_weights": false, "row_id": "row-1gl1fwy", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-5", "model_as_reported": "claude-sonnet-5", "effort": "high", "harness": "mini-swe-agent", "score_pct": 48.23, "ci_pct": 4.51, "n_samples": 4, "cost_per_task_usd": 7.4256, "tokens_per_task": 18341786, "input_tokens_per_task": 18254491, "output_tokens_per_task": 87295, "steps_per_task": 146.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-02", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_sonnet_5_high: pass_at_1 0.4823008849557522, ci_half 0.045063148702148406, n_runs 4, mean_cost_usd 7.425560161197339", "notes": "pass@4 79.6%; mean 29 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Sonnet 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1ca4w9c", "model_released": "2026-07-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gemini-3.6-flash", "model_as_reported": "gemini-3-6-flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 46.68, "ci_pct": 3.7, "n_samples": 4, "cost_per_task_usd": 2.2095, "tokens_per_task": 12691802, "input_tokens_per_task": 12595957, "output_tokens_per_task": 95845, "steps_per_task": 116.7, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-22", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_6_flash_high: pass_at_1 0.4668141592920354, ci_half 0.037048538992472776, n_runs 4, mean_cost_usd 2.209506638666667", "notes": "pass@4 75.2%; mean 25 min per attempt; leaderboard-live.json gives $4.42 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.6 Flash", "lab": "Google", "open_weights": false, "row_id": "row-qiuq8l", "model_released": "2026-07-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "glm-5.2", "model_as_reported": "GLM-5.2", "effort": "max", "harness": "mini-swe-agent", "score_pct": 46.2, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "GLM-5.2 HuggingFace Model Card", "source_url": "https://huggingface.co/zai-org/GLM-5.2", "published": "2026-06-16", "accessed": "2026-10-09", "quote": "| DeepSWE | 46.2 | 18 | 18 | 20 | 8 | 58 | 70 | 10 |", "notes": "Footnote: We run DeepSWE with the official pier evaluation framework and the mini-swe-agent harness (temperature=1.0, top_p=1.0, timeout=2h, 400K context). Each task is solved in an isolated container with 2 CPUs, 8 GB RAM, and no internet access.", "display_name": "GLM-5.2", "lab": "Z.ai", "open_weights": true, "row_id": "row-1138rgg", "model_released": "2026-06-16", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-11h0zp7", "source_n": 6 }, { "model_key": "gpt-5.6-sol", "model_as_reported": "gpt-5-6-sol", "effort": "low", "harness": "mini-swe-agent", "score_pct": 45.35, "ci_pct": 2.39, "n_samples": 4, "cost_per_task_usd": 0.817, "tokens_per_task": 701413, "input_tokens_per_task": 690834, "output_tokens_per_task": 10579, "steps_per_task": 23.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_sol_low: pass_at_1 0.45353982300884954, ci_half 0.0238819467145892, n_runs 4, mean_cost_usd 0.8170052734939612", "notes": "pass@4 71.7%; mean 4 min per attempt; leaderboard-live.json gives $1.07 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-255enc", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-opus-4.7", "model_as_reported": "claude-opus-4-7", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 44.69, "ci_pct": 2.88, "n_samples": null, "cost_per_task_usd": 8.58, "tokens_per_task": 12224722, "input_tokens_per_task": 12159101, "output_tokens_per_task": 65621, "steps_per_task": 125.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "claude-opus-4-7 [xhigh]: 44.69% ±2.88%, cost $8.58, output tokens 65621, steps 125.9", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Claude Opus 4.7", "lab": "Anthropic", "open_weights": false, "row_id": "row-13997ak", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "gpt-6-luna", "model_as_reported": "GPT-6 Luna", "effort": "medium", "harness": "lab-internal", "score_pct": 44.47, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.0518, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Luna\",\"cost_label\":\"$$0.0518\",\"score\":0.4447,\"cost\":0.0518,\"effortLabel\":\"medium\"}", "notes": "Reported in embedded Vega-Lite chart in GPT-6 Sol and Luna launch post.", "display_name": "GPT-6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-1w8ic1j", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "beam", "model_as_reported": "Beam", "effort": "", "harness": "", "score_pct": 44.4, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Reflection AI Beam launch post", "source_url": "https://reflection.ai/blog/introducing-beam", "published": "2026-10-05", "accessed": "2026-10-09", "quote": "DeepSWE v1.1 | 44.4", "notes": "Open-weight 501B/23B MoE under Apache 2.0 license. Reported in the agentic coding evaluation table under DeepSWE v1.1. Evaluated across reasoning effort levels during RL training.", "display_name": "Beam", "lab": "Reflection AI", "open_weights": true, "row_id": "row-16j4ryu", "model_released": "2026-10-05", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1x8ip3p", "source_n": 32 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "gpt-5-6-luna", "effort": "high", "harness": "mini-swe-agent", "score_pct": 44.25, "ci_pct": 2.92, "n_samples": 4, "cost_per_task_usd": 0.1556, "tokens_per_task": 3398256, "input_tokens_per_task": 3372478, "output_tokens_per_task": 25778, "steps_per_task": 49, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_luna_high: pass_at_1 0.4424778761061947, ci_half 0.029195672479165317, n_runs 4, mean_cost_usd 0.15558014999999997", "notes": "pass@4 75.2%; mean 8 min per attempt; leaderboard-live.json gives $0.78 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-1kerb9g", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "glm-5.2", "model_as_reported": "glm-5-2", "effort": "max", "harness": "mini-swe-agent", "score_pct": 43.78, "ci_pct": 1.73, "n_samples": 4, "cost_per_task_usd": 3.9199, "tokens_per_task": 12684736, "input_tokens_per_task": 12606560, "output_tokens_per_task": 78175, "steps_per_task": 129.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-21", "accessed": "2026-10-09", "quote": "config mini_swe_agent_glm_5_2_max: pass_at_1 0.43777777777777777, ci_half 0.01725629620580811, n_runs 4, mean_cost_usd 3.9199075743111114", "notes": "pass@4 77%; mean 44 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GLM-5.2", "lab": "Z.ai", "open_weights": true, "row_id": "row-grzg6t", "model_released": "2026-06-16", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "glm-5.3-flash", "model_as_reported": "GLM 5.3 Flash", "effort": "max", "harness": "mini-swe-agent", "score_pct": 42.5, "ci_pct": 6.5, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GLM 5.3 Flash | effort: max | score: 42.5% ± 6.5% | n_samples: 339 | provider: Zhipu", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GLM-5.3 Flash", "lab": "Z.ai", "open_weights": true, "row_id": "row-j8z20a", "model_released": "2026-08-26", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "GPT-5.6 Luna", "effort": "high", "harness": "lab-internal", "score_pct": 42.37, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.1272, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-5.6 Luna\",\"cost_label\":\"$$0.1272\",\"score\":0.4237,\"cost\":0.1272,\"effortLabel\":\"high\"}", "notes": "OpenAI re-ran GPT-5.6 Luna internally for its GPT-6 Sol/Luna post; the official board measured 67.2% at max. Full effort breakdown from low to max reported in GPT-6 Sol and Luna launch post chart data.", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-1fvmcu6", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "qwen3.8-27b", "model_as_reported": "Qwen3.8-27B", "effort": "", "harness": "Claude Code", "score_pct": 42.2, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Qwen3.8-27B HuggingFace Model Card", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "published": "2026-08-05", "accessed": "2026-10-09", "quote": "
Agentic coding
DeepSWE 1.1
\n42.2", "notes": "Footnote 3: DeepSWE 1.1: Evaluated with the Claude Code harness at temp=1.0, top_p=0.95, and a 256K context window.", "display_name": "Qwen3.8-27B", "lab": "Alibaba", "open_weights": true, "row_id": "row-gz1a0a", "model_released": "2026-08-05", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-190dm7p", "source_n": 10 }, { "model_key": "grok-4.6", "model_as_reported": "grok-4-6", "effort": "low", "harness": "mini-swe-agent", "score_pct": 41.65, "ci_pct": 2.32, "n_samples": 4, "cost_per_task_usd": 1.0424, "tokens_per_task": 1663378, "input_tokens_per_task": 1646920, "output_tokens_per_task": 16458, "steps_per_task": 44.2, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-08-12", "accessed": "2026-10-09", "quote": "config mini_swe_agent_grok_4_6_low: pass_at_1 0.41648106904231624, ci_half 0.023186517644299014, n_runs 4, mean_cost_usd 1.0423604231625836", "notes": "pass@4 69%; mean 5 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-1ucz10t", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-opus-4.8", "model_as_reported": "claude-opus-4-8", "effort": "low", "harness": "mini-swe-agent", "score_pct": 40.8, "ci_pct": 1.46, "n_samples": 4, "cost_per_task_usd": 2.2934, "tokens_per_task": 2446911, "input_tokens_per_task": 2417988, "output_tokens_per_task": 28923, "steps_per_task": 54, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_opus_4_8_low: pass_at_1 0.4079822616407982, ci_half 0.014635847585691517, n_runs 4, mean_cost_usd 2.293369216740577", "notes": "pass@4 68.1%; mean 11 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Opus 4.8", "lab": "Anthropic", "open_weights": false, "row_id": "row-1pcx9r4", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "laguna-s-2.1", "model_as_reported": "Laguna S 2.1", "effort": "max", "harness": "pool", "score_pct": 40.4, "ci_pct": null, "n_samples": 3, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": 249000, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Poolside Laguna S 2.1 launch post", "source_url": "https://poolside.ai/blog/introducing-laguna-s-2-1", "published": "2026-07-21", "accessed": "2026-10-09", "quote": "On DeepSWE v1.1, Laguna S 2.1 scores 40.4 in thinking mode in pool harness.", "notes": "Token count is mean completion (output) tokens per trajectory, read from the launch post chart labels (\"DeepSWE: 40 at 249k, thinking\"; \"17 at 99k, no-thinking\"). Evaluated using Poolside's proprietary 'pool' agent harness (not mini-swe-agent as aggregators cited) with 500-step limit and 6-hour timeout. pass@1 averaged over 3 attempts per task. Thinking mode lifts score from 16.5% (99k tokens, thinking off) to 40.4% (249k tokens, thinking max).", "display_name": "Laguna S 2.1", "lab": "Poolside", "open_weights": true, "row_id": "row-befdi5", "model_released": "2026-07-21", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-12e8ru1", "source_n": 7 }, { "model_key": "muse-spark-1.1", "model_as_reported": "Muse Spark 1.1", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 40.4, "ci_pct": 6.5, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Muse Spark 1.1 | effort: xhigh | score: 40.4% ± 6.5% | n_samples: 339 | provider: Meta", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Muse Spark 1.1", "lab": "Meta", "open_weights": false, "row_id": "row-1bkgt8e", "model_released": "2026-07-14", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-4.7", "model_as_reported": "claude-opus-4-7", "effort": "high", "harness": "mini-swe-agent", "score_pct": 40.27, "ci_pct": 3.5, "n_samples": null, "cost_per_task_usd": 4.83, "tokens_per_task": 6214791, "input_tokens_per_task": 6169055, "output_tokens_per_task": 45736, "steps_per_task": 87.2, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "claude-opus-4-7 [high]: 40.27% ±3.5%, cost $4.83, output tokens 45736, steps 87.2", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Claude Opus 4.7", "lab": "Anthropic", "open_weights": false, "row_id": "row-r2zw6y", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "claude-sonnet-5", "model_as_reported": "claude-sonnet-5", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 39.78, "ci_pct": 3.13, "n_samples": 4, "cost_per_task_usd": 4.079, "tokens_per_task": 9343779, "input_tokens_per_task": 9286962, "output_tokens_per_task": 56817, "steps_per_task": 107.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-02", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_sonnet_5_medium: pass_at_1 0.3977777777777778, ci_half 0.03127576206329727, n_runs 4, mean_cost_usd 4.079036229", "notes": "pass@4 64.6%; mean 19 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Sonnet 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1iia63p", "model_released": "2026-07-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "glm-5.2", "model_as_reported": "GLM 5.2", "effort": "max", "harness": "mini-swe-agent", "score_pct": 38.9, "ci_pct": 6.5, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GLM 5.2 | effort: max | score: 38.9% ± 6.5% | n_samples: 339 | provider: Zhipu", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GLM-5.2", "lab": "Z.ai", "open_weights": true, "row_id": "row-1k7amnx", "model_released": "2026-06-16", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "swe-1.7", "model_as_reported": "SWE-1.7", "effort": "", "harness": "Devin CLI", "score_pct": 37.7, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Cognition SWE-2 launch post", "source_url": "https://cognition.com/blog/swe-2", "published": "2026-09-10", "accessed": "2026-10-09", "quote": "SWE-1.7 | 37.7%", "notes": "Reported retrospectively in the SWE-2 launch post DeepSWE 1.1 comparison table. The original SWE-1.7 launch post on 2026-07-08 (https://cognition.com/blog/swe-1-7) predated and did not report DeepSWE 1.1.", "display_name": "SWE-1.7", "lab": "Cognition", "open_weights": false, "row_id": "row-1bfjpb6", "model_released": "2026-09-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-prwi23", "source_n": 20 }, { "model_key": "gpt-6-sol", "model_as_reported": "GPT-6 Sol", "effort": "low", "harness": "lab-internal", "score_pct": 37.17, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.1623, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Sol\",\"cost_label\":\"$$0.16\",\"score\":0.3717,\"cost\":0.1623,\"effortLabel\":\"low\"}", "notes": "Reported in embedded Vega-Lite chart and text: 'On DeepSWE v1.1... GPT-6 Sol at max effort scores 68.8%, within 1.1 percentage points of Claude Fable 5’s highest score in the evaluation—69.9% at xhigh effort—at approximately 80% lower cost per task.'", "display_name": "GPT-6 Sol", "lab": "OpenAI", "open_weights": false, "row_id": "row-1bw434n", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "gemini-3.5-flash", "model_as_reported": "Gemini 3.5 Flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 36.6, "ci_pct": 6.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Gemini 3.5 Flash | effort: high | score: 36.6% ± 6.9% | n_samples: 339 | provider: Google", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Gemini 3.5 Flash", "lab": "Google", "open_weights": false, "row_id": "row-1twb52n", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "glm-5.2", "model_as_reported": "glm-5-2", "effort": "high", "harness": "mini-swe-agent", "score_pct": 36.28, "ci_pct": 4.75, "n_samples": 4, "cost_per_task_usd": 2.8355, "tokens_per_task": 9124851, "input_tokens_per_task": 9070606, "output_tokens_per_task": 54246, "steps_per_task": 121.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-21", "accessed": "2026-10-09", "quote": "config mini_swe_agent_glm_5_2_high: pass_at_1 0.36283185840707965, ci_half 0.047500729479216554, n_runs 4, mean_cost_usd 2.8355164725663715", "notes": "pass@4 68.1%; mean 30 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GLM-5.2", "lab": "Z.ai", "open_weights": true, "row_id": "row-a6zyiq", "model_released": "2026-06-16", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "nex-n2.5-mini", "model_as_reported": "Nex-N2.5-mini", "effort": "", "harness": "NexAU", "score_pct": 36.1, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Nex-AGI Nex-N2.5 GitHub README / Model Card", "source_url": "https://github.com/nex-agi/Nex-N2.5", "published": "2026-09-08", "accessed": "2026-10-09", "quote": "DeepSWE v1.1 | 36.1 | 55.8 | 65.6 | 73.7 | 72.7 | 67.5 | 66.9 | 62.8 | 69.3", "notes": "Coding tasks evaluated using the NexAU harness. Sampling parameters: temperature = 0.7, top_p = 0.95, top_k = 40. Lightweight multimodal agentic model.", "display_name": "Nex-N2.5-Mini", "lab": "Nex-AGI", "open_weights": true, "row_id": "row-92bvuc", "model_released": "2026-09-08", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-pixn6e", "source_n": 19 }, { "model_key": "gemini-3.5-flash", "model_as_reported": "gemini-3-5-flash", "effort": "high", "harness": "mini-swe-agent", "score_pct": 36.06, "ci_pct": 3.97, "n_samples": 4, "cost_per_task_usd": 3.4467, "tokens_per_task": 7704802, "input_tokens_per_task": 7629072, "output_tokens_per_task": 75730, "steps_per_task": 105.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_5_flash_high: pass_at_1 0.3606194690265487, ci_half 0.03966303010520439, n_runs 4, mean_cost_usd 3.4467114588691796", "notes": "pass@4 63.7%; mean 19 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.5 Flash", "lab": "Google", "open_weights": false, "row_id": "row-vnphv3", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-terra", "model_as_reported": "gpt-5-6-terra", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 35.11, "ci_pct": 3.38, "n_samples": 4, "cost_per_task_usd": 0.4666, "tokens_per_task": 736847, "input_tokens_per_task": 725101, "output_tokens_per_task": 11747, "steps_per_task": 25.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_terra_medium: pass_at_1 0.3511111111111111, ci_half 0.03381964303406632, n_runs 4, mean_cost_usd 0.46659949511111115", "notes": "pass@4 60.2%; mean 4 min per attempt; leaderboard-live.json gives $0.58 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Terra", "lab": "OpenAI", "open_weights": false, "row_id": "row-poihd0", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-4.6", "model_as_reported": "Sonnet 4.6", "effort": "high", "harness": "mini-swe-agent", "score_pct": 34.2, "ci_pct": 6.3, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Sonnet 4.6 | effort: high | score: 34.2% ± 6.3% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Sonnet 4.6", "lab": "Anthropic", "open_weights": false, "row_id": "row-kcyzlp", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "grok-4.7", "model_as_reported": "Grok 4.7", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 33.3, "ci_pct": 6, "n_samples": 337, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Grok 4.7 | effort: xhigh | score: 33.3% ± 6% | n_samples: 337 | provider: xAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Grok 4.7", "lab": "xAI", "open_weights": false, "row_id": "row-bfbz9v", "model_released": "2026-09-21", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-4.6", "model_as_reported": "Opus 4.6", "effort": "max", "harness": "mini-swe-agent", "score_pct": 31.9, "ci_pct": 6.3, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Opus 4.6 | effort: max | score: 31.9% ± 6.3% | n_samples: 339 | provider: Anthropic", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Claude Opus 4.6", "lab": "Anthropic", "open_weights": false, "row_id": "row-vnot5p", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-opus-4.7", "model_as_reported": "claude-opus-4-7", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 31.64, "ci_pct": 4.03, "n_samples": null, "cost_per_task_usd": 2.34, "tokens_per_task": 2572887, "input_tokens_per_task": 2545442, "output_tokens_per_task": 27445, "steps_per_task": 55.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "claude-opus-4-7 [medium]: 31.64% ±4.03%, cost $2.34, output tokens 27445, steps 55.4", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Claude Opus 4.7", "lab": "Anthropic", "open_weights": false, "row_id": "row-3ph296", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "kimi-k2.7-code", "model_as_reported": "kimi-k2-7-code", "effort": "", "harness": "mini-swe-agent", "score_pct": 30.53, "ci_pct": 0.5, "n_samples": 4, "cost_per_task_usd": 2.8155, "tokens_per_task": 13067339, "input_tokens_per_task": 13008041, "output_tokens_per_task": 59297, "steps_per_task": 149.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_kimi_k2_7_code_default: pass_at_1 0.3053097345132743, ci_half 0.005007016522460913, n_runs 4, mean_cost_usd 2.815536243274336", "notes": "pass@4 61.1%; mean 36 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Kimi K2.7 Code", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-1yoj145", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "claude-sonnet-5", "model_as_reported": "claude-sonnet-5", "effort": "low", "harness": "mini-swe-agent", "score_pct": 30.51, "ci_pct": 1.13, "n_samples": 4, "cost_per_task_usd": 2.1866, "tokens_per_task": 4570071, "input_tokens_per_task": 4534476, "output_tokens_per_task": 35595, "steps_per_task": 76.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-02", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_sonnet_5_low: pass_at_1 0.3051224944320713, ci_half 0.011260175414427497, n_runs 4, mean_cost_usd 2.1865550832962137", "notes": "pass@4 57.5%; mean 12 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Sonnet 5", "lab": "Anthropic", "open_weights": false, "row_id": "row-1em597q", "model_released": "2026-07-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "kimi-k2.7-code", "model_as_reported": "Kimi K2.7 Code", "effort": "high", "harness": "mini-swe-agent", "score_pct": 30.1, "ci_pct": 8.4, "n_samples": 113, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Kimi K2.7 Code | effort: high | score: 30.1% ± 8.4% | n_samples: 113 | provider: Kimi", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Kimi K2.7 Code", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-12ms1es", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-sonnet-4.6", "model_as_reported": "claude-sonnet-4-6", "effort": "high", "harness": "mini-swe-agent", "score_pct": 29.93, "ci_pct": 4.09, "n_samples": 4, "cost_per_task_usd": 5.5224, "tokens_per_task": 12946788, "input_tokens_per_task": 12870628, "output_tokens_per_task": 76160, "steps_per_task": 133.7, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_claude_sonnet_4_6_high: pass_at_1 0.29933481152993346, ci_half 0.04092141269189136, n_runs 4, mean_cost_usd 5.5223517987804875", "notes": "pass@4 56.6%; mean 47 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Claude Sonnet 4.6", "lab": "Anthropic", "open_weights": false, "row_id": "row-uei8w1", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gemini-3.5-flash", "model_as_reported": "gemini-3-5-flash", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 28.32, "ci_pct": 3.54, "n_samples": null, "cost_per_task_usd": 7.42, "tokens_per_task": 12736169, "input_tokens_per_task": 12547145, "output_tokens_per_task": 189024, "steps_per_task": 75.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "gemini-3-5-flash [medium]: 28.32% ±3.54%, cost $7.42, output tokens 189024, steps 75.1", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.; The board later revised this configuration: its leaderboard-live.json of 17 Jul 2026 (archived https://web.archive.org/web/20260721100146/https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json) gives 37.39% at $7.34 per task, and the configuration was then dropped from the board.", "display_name": "Gemini 3.5 Flash", "lab": "Google", "open_weights": false, "row_id": "row-1bibl2y", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "hy3", "model_as_reported": "Hy3", "effort": "", "harness": "lab-internal", "score_pct": 28, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Tencent Hy4-preview Technical Report & Model Card", "source_url": "https://huggingface.co/tencent/Hy4-preview", "published": "2026-08-27", "accessed": "2026-10-09", "quote": "Hy3 recorded 28.0% on DeepSWE, reported as previous generation baseline in the Hy4 Benchmark Appendix.", "notes": "295B MoE baseline model released mid-2026.", "display_name": "Hy3", "lab": "Tencent", "open_weights": true, "row_id": "row-1ilk7um", "model_released": "2026-08-27", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-p3aj15", "source_n": 16 }, { "model_key": "claude-opus-4.6", "model_as_reported": "claude-opus-4-6", "effort": "max", "harness": "mini-swe-agent", "score_pct": 27.6, "ci_pct": 3.68, "n_samples": null, "cost_per_task_usd": 5.39, "tokens_per_task": 7176852, "input_tokens_per_task": 7132371, "output_tokens_per_task": 44481, "steps_per_task": 103, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "claude-opus-4-6 [max]: 27.6% ±3.68%, cost $5.39, output tokens 44481, steps 103.0", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Claude Opus 4.6", "lab": "Anthropic", "open_weights": false, "row_id": "row-1thnwnf", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "gpt-5.5", "model_as_reported": "gpt-5-5", "effort": "low", "harness": "mini-swe-agent", "score_pct": 26.99, "ci_pct": 2.29, "n_samples": 4, "cost_per_task_usd": 1.2002, "tokens_per_task": 786194, "input_tokens_per_task": 776752, "output_tokens_per_task": 9443, "steps_per_task": 28.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_5_low: pass_at_1 0.26991150442477874, ci_half 0.02294503222007181, n_runs 4, mean_cost_usd 1.2002025442477877", "notes": "pass@4 47.8%; mean 9 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.5", "lab": "OpenAI", "open_weights": false, "row_id": "row-165hhz9", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "grok-4.6", "model_as_reported": "Grok 4.6", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 24.8, "ci_pct": 6.8, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Grok 4.6 | effort: xhigh | score: 24.8% ± 6.8% | n_samples: 339 | provider: xAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Grok 4.6", "lab": "xAI", "open_weights": false, "row_id": "row-f6bkeh", "model_released": "2026-08-12", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.4-mini", "model_as_reported": "gpt-5-4-mini", "effort": "xhigh", "harness": "mini-swe-agent", "score_pct": 24.34, "ci_pct": 2.96, "n_samples": null, "cost_per_task_usd": 2.08, "tokens_per_task": 12313472, "input_tokens_per_task": 12178956, "output_tokens_per_task": 134516, "steps_per_task": 85.9, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "gpt-5-4-mini [xhigh]: 24.34% ±2.96%, cost $2.08, output tokens 134516, steps 85.9", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "GPT-5.4 mini", "lab": "OpenAI", "open_weights": null, "row_id": "row-3oes9n", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "gpt-5.6-terra", "model_as_reported": "gpt-5-6-terra", "effort": "low", "harness": "mini-swe-agent", "score_pct": 24.05, "ci_pct": 0.78, "n_samples": 4, "cost_per_task_usd": 0.3422, "tokens_per_task": 489138, "input_tokens_per_task": 480565, "output_tokens_per_task": 8572, "steps_per_task": 21.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_terra_low: pass_at_1 0.24053452115812918, ci_half 0.007767613669447833, n_runs 4, mean_cost_usd 0.3421962084632517", "notes": "pass@4 44.2%; mean 3 min per attempt; leaderboard-live.json gives $0.43 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Terra", "lab": "OpenAI", "open_weights": false, "row_id": "row-1wrol78", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "kimi-k2.6", "model_as_reported": "kimi-k2-6", "effort": "", "harness": "mini-swe-agent", "score_pct": 23.89, "ci_pct": 2.45, "n_samples": null, "cost_per_task_usd": 3.16, "tokens_per_task": 12102875, "input_tokens_per_task": 12018459, "output_tokens_per_task": 84416, "steps_per_task": 146.8, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "kimi-k2-6 [None]: 23.89% ±2.45%, cost $3.16, output tokens 84416, steps 146.8", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Kimi K2.6", "lab": "Moonshot AI", "open_weights": null, "row_id": "row-1r0o7od", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "qwen3.7-max", "model_as_reported": "Qwen3.7-Max", "effort": "", "harness": "Claude Code", "score_pct": 21.6, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Qwen3.8-2.4T-A95B HuggingFace Model Card", "source_url": "https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B", "published": "2026-08-08", "accessed": "2026-10-09", "quote": "DeepSWE 1.1 | 59.0 | 70.0 | 73.0 | 21.6 | 56.6", "notes": "Reported as previous generation baseline in Qwen3.8-Max benchmark table. Evaluated with temp=1.0, top_p=0.95, 256K context window.", "display_name": "Qwen3.7 Max", "lab": "Alibaba", "open_weights": false, "row_id": "row-cpxpc0", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-ouk7wk", "source_n": 11 }, { "model_key": "minimax-m3", "model_as_reported": "minimax-m3", "effort": "", "harness": "mini-swe-agent", "score_pct": 20.44, "ci_pct": 3.73, "n_samples": null, "cost_per_task_usd": 5.57, "tokens_per_task": 41744490, "input_tokens_per_task": 41646948, "output_tokens_per_task": 97542, "steps_per_task": 314.2, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "minimax-m3 [None]: 20.44% ±3.73%, cost $5.57, output tokens 97542, steps 314.2", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "MiniMax M3", "lab": "MiniMax", "open_weights": null, "row_id": "row-9zllbx", "model_released": "2026-06-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "mimo-v2.5-pro", "model_as_reported": "mimo-v2-5-pro", "effort": "", "harness": "mini-swe-agent", "score_pct": 19.47, "ci_pct": 1.87, "n_samples": null, "cost_per_task_usd": 1.99, "tokens_per_task": 8685488, "input_tokens_per_task": 8636167, "output_tokens_per_task": 49321, "steps_per_task": 121.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "mimo-v2-5-pro [None]: 19.47% ±1.87%, cost $1.99, output tokens 49321, steps 121.5", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "MiMo-V2.5-Pro", "lab": "Xiaomi", "open_weights": true, "row_id": "row-1q4sncw", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "glm-5.1", "model_as_reported": "GLM 5.1", "effort": "", "harness": "mini-swe-agent", "score_pct": 18.9, "ci_pct": 5.5, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GLM 5.1 | effort: None | score: 18.9% ± 5.5% | n_samples: 339 | provider: Zhipu", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GLM-5.1", "lab": "Z.ai", "open_weights": true, "row_id": "row-65jvtm", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "qwen3.7-max", "model_as_reported": "qwen3-7-max", "effort": "", "harness": "mini-swe-agent", "score_pct": 17.7, "ci_pct": 1.42, "n_samples": null, "cost_per_task_usd": 2.12, "tokens_per_task": 6486282, "input_tokens_per_task": 6443835, "output_tokens_per_task": 42447, "steps_per_task": 110.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "qwen3-7-max [None]: 17.7% ±1.42%, cost $2.12, output tokens 42447, steps 110.3", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Qwen3.7 Max", "lab": "Alibaba", "open_weights": false, "row_id": "row-1rl4w7q", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "glm-5.1", "model_as_reported": "glm-5-1", "effort": "", "harness": "mini-swe-agent", "score_pct": 17.52, "ci_pct": 0.79, "n_samples": null, "cost_per_task_usd": 7.46, "tokens_per_task": 13589155, "input_tokens_per_task": 13539998, "output_tokens_per_task": 49157, "steps_per_task": 176.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "glm-5-1 [None]: 17.52% ±0.79%, cost $7.46, output tokens 49157, steps 176.6", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "GLM-5.1", "lab": "Z.ai", "open_weights": true, "row_id": "row-q9lc3f", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "minimax-m3", "model_as_reported": "MiniMax-M3 (Extended Window)", "effort": "thinking", "harness": "mini-swe-agent", "score_pct": 16.8, "ci_pct": null, "n_samples": 113, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Independent DeepSWE Audit by entrpi", "source_url": "https://entrpi.github.io/misc/deep-swe-minimax-m3/", "published": "2026-06-02", "accessed": "2026-10-09", "quote": "Extended pass rate: 16.8% (19/113, counting 4 solutions finished past the standard time window).", "notes": "19/113 tasks solved when counting 4 solutions completed past the 90-minute deadline but within the 135-minute ceiling.", "display_name": "MiniMax M3", "lab": "MiniMax", "open_weights": null, "row_id": "row-1hc37at", "model_released": "2026-06-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-hzsdm7", "source_n": 35 }, { "model_key": "laguna-s-2.1", "model_as_reported": "Laguna S 2.1", "effort": "off", "harness": "pool", "score_pct": 16.5, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": 99000, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Poolside Laguna S 2.1 launch post", "source_url": "https://poolside.ai/blog/introducing-laguna-s-2-1", "published": "2026-07-21", "accessed": "2026-10-10", "quote": "Max thinking lifts S 2.1’s score on Terminal-Bench 2.1 from 60.4% to 70.2% and on DeepSWE from 16.5% to 40.4%.", "notes": "Token count is mean completion (output) tokens per trajectory, read from the launch post chart labels (\"DeepSWE: 40 at 249k, thinking\"; \"17 at 99k, no-thinking\"). Poolside launch post states Laguna S 2.1 has two thinking modes: off and max (enabled by default); released without user-configurable low/medium/high control. In thinking-off mode, S 2.1 scored 16.5% (plotted as ~17% with 99k mean completion tokens/trajectory) in the pool harness, compared to 40.4% at max thinking (~249k completion tokens).", "display_name": "Laguna S 2.1", "lab": "Poolside", "open_weights": true, "row_id": "row-tg416", "model_released": "2026-07-21", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-12e8ru1", "source_n": 7 }, { "model_key": "qwen3.7-plus", "model_as_reported": "Qwen3.7-Plus", "effort": "", "harness": "mini-swe-agent", "score_pct": 16.5, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Qwen3.8-Flash-Next HuggingFace Model Card", "source_url": "https://huggingface.co/Qwen/Qwen3.8-Flash-Next", "published": "2026-08-24", "accessed": "2026-10-09", "quote": "DeepSWE 1.1\n58.7\n42.2\n16.5", "notes": "Reported as baseline in Qwen3.8-Flash-Next README (16.5%) and Qwen3.8-27B README (14.2% on Claude Code). Evaluated with temp=1.0, top_p=0.95, 256K context window.", "display_name": "Qwen3.7 Plus", "lab": "Alibaba", "open_weights": null, "row_id": "row-1exbwot", "model_released": "2026-08-13", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-qxom7y", "source_n": 14 }, { "model_key": "qwen3.7-plus", "model_as_reported": "Qwen3.7-Plus", "effort": "", "harness": "Claude Code", "score_pct": 14.2, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Qwen3.8-27B HuggingFace Model Card", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "published": "2026-08-13", "accessed": "2026-10-10", "quote": "| \"Coding\" | \"Agentic coding\" | \"DeepSWE 1.1\" | \"**42.2**\" | \"13.3\" | \"14.2\" | \"--\" | \"--\" |; Footnote 3: \"DeepSWE 1.1: Evaluated with the Claude Code harness at temp=1.0, top_p=0.95, and a 256K context window.\"", "notes": "New harness/setting measurement: 14.2% on Claude Code harness (temp=1.0, top_p=0.95, 256K context window, thinking mode/xhigh). Differs from earlier recorded 16.5% which was obtained via mini-SWE-agent. [lead: effort left unstated; the card documents the default setting but does not state the one used for DeepSWE]", "display_name": "Qwen3.7 Plus", "lab": "Alibaba", "open_weights": null, "row_id": "row-1uo50jh", "model_released": "2026-08-13", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-190dm7p", "source_n": 10 }, { "model_key": "minimax-m3", "model_as_reported": "MiniMax-M3", "effort": "thinking", "harness": "mini-swe-agent", "score_pct": 13.3, "ci_pct": null, "n_samples": 113, "cost_per_task_usd": 7.48, "tokens_per_task": 80000, "input_tokens_per_task": null, "output_tokens_per_task": 80000, "steps_per_task": 325, "source_type": "third_party_run", "verified_primary": true, "source_name": "Independent DeepSWE Audit by entrpi", "source_url": "https://entrpi.github.io/misc/deep-swe-minimax-m3/", "published": "2026-06-02", "accessed": "2026-10-09", "quote": "Strict pass rate: 13.3% (15/113 evaluated problems resolved within budget); Extended pass rate: 16.8% (19/113); Median steps: 325; Median output volume: 80k tokens; Median price: $7.48 per scenario.", "notes": "Third-party audit using unmodified mini-swe-agent in Pier with Harbor task format. MiniMax Anthropic-compatible API, pass@1, 90 min working time limit per task, air-gapped container.", "display_name": "MiniMax M3", "lab": "MiniMax", "open_weights": null, "row_id": "row-x0507n", "model_released": "2026-06-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-hzsdm7", "source_n": 35 }, { "model_key": "qwen3.6-27b", "model_as_reported": "Qwen3.6-27B", "effort": "", "harness": "Claude Code", "score_pct": 13.3, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "Qwen3.8-27B HuggingFace Model Card", "source_url": "https://huggingface.co/Qwen/Qwen3.8-27B", "published": "2026-08-05", "accessed": "2026-10-09", "quote": "| DeepSWE 1.1 | 42.2 | 13.3 | 14.2 | -- | -- |", "notes": "Reported as baseline in Qwen3.8-27B README benchmark table. Claude Code harness, temp=1.0, top_p=0.95, 256K context window.", "display_name": "Qwen3.6-27B", "lab": "Alibaba", "open_weights": true, "row_id": "row-1fk1m2l", "model_released": "2026-08-05", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-190dm7p", "source_n": 10 }, { "model_key": "grok-build-0.1", "model_as_reported": "grok-build-0-1", "effort": "", "harness": "mini-swe-agent", "score_pct": 13.08, "ci_pct": 2.44, "n_samples": null, "cost_per_task_usd": 6.6, "tokens_per_task": 10309658, "input_tokens_per_task": 10257762, "output_tokens_per_task": 51896, "steps_per_task": 176.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "grok-build-0-1 [None]: 13.08% ±2.44%, cost $6.6, output tokens 51896, steps 176.1", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Grok Build 0.1", "lab": "xAI", "open_weights": null, "row_id": "row-mbkfkb", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "deepseek-v4-pro-preview", "model_as_reported": "DeepSeek-V4-Pro (Preview)", "effort": "max", "harness": "DeepSeek Harness (minimal mode)", "score_pct": 12.8, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4-Pro-0813 HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813", "published": "2026-08-13", "accessed": "2026-10-10", "quote": "| \"DeepSWE\" | \"62.7\" | \"54.4\" | \"12.8\" | \"7.3\" | \"46.2\" | \"67.5\" | \"58.0\" | \"70.0\" |", "notes": "Sibling preview baseline reported in DeepSeek-V4-Pro-0813 and DeepSeek-V4-Flash-0731 benchmark tables under max reasoning effort.", "display_name": "DeepSeek V4 Pro (preview)", "lab": "DeepSeek", "open_weights": true, "row_id": "row-wg3gzd", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-7akg1l", "source_n": 12 }, { "model_key": "gemini-3.1-pro-preview", "model_as_reported": "gemini-3-1-pro-preview", "effort": "high", "harness": "mini-swe-agent", "score_pct": 11.73, "ci_pct": 1.48, "n_samples": 4, "cost_per_task_usd": 2.1434, "tokens_per_task": 2432046, "input_tokens_per_task": 2403677, "output_tokens_per_task": 28369, "steps_per_task": 75.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gemini_3_1_pro_preview_high: pass_at_1 0.1172566371681416, ci_half 0.01481095461108845, n_runs 4, mean_cost_usd 2.1434045568888886", "notes": "pass@4 28.3%; mean 14 min per attempt; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "Gemini 3.1 Pro (preview)", "lab": "Google", "open_weights": false, "row_id": "row-18zp7di", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "gpt-5-6-luna", "effort": "medium", "harness": "mini-swe-agent", "score_pct": 11.28, "ci_pct": 0.83, "n_samples": 4, "cost_per_task_usd": 0.0433, "tokens_per_task": 626007, "input_tokens_per_task": 617827, "output_tokens_per_task": 8180, "steps_per_task": 23.7, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_luna_medium: pass_at_1 0.11283185840707964, ci_half 0.008303197562056514, n_runs 4, mean_cost_usd 0.04326195796460178", "notes": "pass@4 27.4%; mean 3 min per attempt; leaderboard-live.json gives $0.22 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-7ygxig", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gemini-3.1-pro-preview", "model_as_reported": "gemini-3-1-pro-preview", "effort": "", "harness": "mini-swe-agent", "score_pct": 9.73, "ci_pct": 2.83, "n_samples": null, "cost_per_task_usd": 1.84, "tokens_per_task": 2100631, "input_tokens_per_task": 2047640, "output_tokens_per_task": 52991, "steps_per_task": 74.1, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "gemini-3-1-pro-preview [None]: 9.73% ±2.83%, cost $1.84, output tokens 52991, steps 74.1", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Gemini 3.1 Pro (preview)", "lab": "Google", "open_weights": false, "row_id": "row-xdw8fl", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "GPT-5.6 Luna", "effort": "medium", "harness": "lab-internal", "score_pct": 9.29, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.0313, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-5.6 Luna\",\"cost_label\":\"$$0.0313\",\"score\":0.0929,\"cost\":0.0313,\"effortLabel\":\"medium\"}", "notes": "OpenAI re-ran GPT-5.6 Luna internally for its GPT-6 Sol/Luna post; the official board measured 67.2% at max. Full effort breakdown from low to max reported in GPT-6 Sol and Luna launch post chart data.", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-yvja1r", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "deepseek-v4-pro-preview", "model_as_reported": "DeepSeek V4 Pro", "effort": "max", "harness": "mini-swe-agent", "score_pct": 9.1, "ci_pct": 3.8, "n_samples": 337, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "DeepSeek V4 Pro | effort: max | score: 9.1% ± 3.8% | n_samples: 337 | provider: DeepSeek", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor lists this unsuffixed DeepSeek V4 Pro beside a separate \"DeepSeek V4 Pro 0813\", so it is the April preview. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "DeepSeek V4 Pro (preview)", "lab": "DeepSeek", "open_weights": true, "row_id": "row-dy35ap", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "minimax-m3", "model_as_reported": "MiniMax M3", "effort": "high", "harness": "mini-swe-agent", "score_pct": 9.1, "ci_pct": 3.1, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "MiniMax M3 | effort: high | score: 9.1% ± 3.1% | n_samples: 339 | provider: MiniMax", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "MiniMax M3", "lab": "MiniMax", "open_weights": null, "row_id": "row-1ggd9h", "model_released": "2026-06-02", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gemini-3.1-pro-preview", "model_as_reported": "Gemini 3.1 Pro", "effort": "high", "harness": "mini-swe-agent", "score_pct": 7.7, "ci_pct": 3.1, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Gemini 3.1 Pro | effort: high | score: 7.7% ± 3.1% | n_samples: 339 | provider: Google", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor names it Gemini 3.1 Pro; Google released 3.1 Pro only as gemini-3.1-pro-preview (ai.google.dev model docs), the model the official board ran. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Gemini 3.1 Pro (preview)", "lab": "Google", "open_weights": false, "row_id": "row-166i7y8", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "deepseek-v4-pro-preview", "model_as_reported": "deepseek-v4-pro", "effort": "", "harness": "mini-swe-agent", "score_pct": 7.52, "ci_pct": 2.7, "n_samples": null, "cost_per_task_usd": 4.22, "tokens_per_task": 8403574, "input_tokens_per_task": 8353625, "output_tokens_per_task": 49949, "steps_per_task": 111.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "deepseek-v4-pro [None]: 7.52% ±2.7%, cost $4.22, output tokens 49949, steps 111.3", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.; This is the April preview of DeepSeek V4 Pro: DeepSeek re-released V4 Pro as 0813 under the same API name, and the live board row (62.83%) is the re-release.", "display_name": "DeepSeek V4 Pro (preview)", "lab": "DeepSeek", "open_weights": true, "row_id": "row-ppsxc9", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "deepseek-v4-flash-preview", "model_as_reported": "DeepSeek-V4-Flash (Preview)", "effort": "max", "harness": "DeepSeek Harness (minimal mode)", "score_pct": 7.3, "ci_pct": null, "n_samples": null, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "DeepSeek-V4-Pro-0813 HuggingFace Model Card", "source_url": "https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813", "published": "2026-08-13", "accessed": "2026-10-10", "quote": "| \"DeepSWE\" | \"62.7\" | \"54.4\" | \"12.8\" | \"7.3\" | \"46.2\" | \"67.5\" | \"58.0\" | \"70.0\" |", "notes": "Sibling preview baseline reported in DeepSeek-V4-Pro-0813 and DeepSeek-V4-Flash-0731 benchmark tables under max reasoning effort.", "display_name": "DeepSeek V4 Flash (preview)", "lab": "DeepSeek", "open_weights": true, "row_id": "row-l10ntr", "model_released": "2026-08-13", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-7akg1l", "source_n": 12 }, { "model_key": "gemini-3-flash-preview", "model_as_reported": "gemini-3-flash-preview", "effort": "", "harness": "mini-swe-agent", "score_pct": 5.14, "ci_pct": 2.48, "n_samples": null, "cost_per_task_usd": 1.53, "tokens_per_task": 5989122, "input_tokens_per_task": 5756060, "output_tokens_per_task": 233062, "steps_per_task": 70.8, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "gemini-3-flash-preview [None]: 5.14% ±2.48%, cost $1.53, output tokens 233062, steps 70.8", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Gemini 3 Flash (preview)", "lab": "Google", "open_weights": null, "row_id": "row-14jryq4", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "inkling", "model_as_reported": "Inkling", "effort": "high", "harness": "mini-swe-agent", "score_pct": 5, "ci_pct": 2.8, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Inkling | effort: high | score: 5% ± 2.8% | n_samples: 339 | provider: Thinking Machines", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Inkling", "lab": "Thinking Machines", "open_weights": false, "row_id": "row-1rprkfy", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "qwen3.6-plus", "model_as_reported": "qwen3-6-plus", "effort": "", "harness": "mini-swe-agent", "score_pct": 2.65, "ci_pct": 1.23, "n_samples": null, "cost_per_task_usd": 4.25, "tokens_per_task": 12729719, "input_tokens_per_task": 12662826, "output_tokens_per_task": 66893, "steps_per_task": 164.4, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "qwen3-6-plus [None]: 2.65% ±1.23%, cost $4.25, output tokens 66893, steps 164.4", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Qwen3.6 Plus", "lab": "Alibaba", "open_weights": null, "row_id": "row-1yu8727", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "gpt-6-luna", "model_as_reported": "GPT-6 Luna", "effort": "low", "harness": "lab-internal", "score_pct": 2.43, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.0057, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-6 Luna\",\"cost_label\":\"$$0.0057\",\"score\":0.0243,\"cost\":0.0057,\"effortLabel\":\"low\"}", "notes": "Reported in embedded Vega-Lite chart in GPT-6 Sol and Luna launch post.", "display_name": "GPT-6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-6tewm6", "model_released": "2026-09-22", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "glm-5", "model_as_reported": "GLM 5", "effort": "", "harness": "mini-swe-agent", "score_pct": 2.1, "ci_pct": 1.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GLM 5 | effort: None | score: 2.1% ± 1.9% | n_samples: 339 | provider: Zhipu", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "GLM-5", "lab": "Z.ai", "open_weights": true, "row_id": "row-w6r0fq", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "gpt-5-6-luna", "effort": "low", "harness": "mini-swe-agent", "score_pct": 1.55, "ci_pct": 0.83, "n_samples": 4, "cost_per_task_usd": 0.0145, "tokens_per_task": 153159, "input_tokens_per_task": 150032, "output_tokens_per_task": 3128, "steps_per_task": 12.5, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard", "source_url": "https://deepswe.datacurve.ai/", "published": "2026-07-10", "accessed": "2026-10-09", "quote": "config mini_swe_agent_gpt_5_6_luna_low: pass_at_1 0.015486725663716814, ci_half 0.008303197562056514, n_runs 4, mean_cost_usd 0.014481247876106192", "notes": "pass@4 4.4%; mean 1 min per attempt; leaderboard-live.json gives $0.07 per task under its own cost basis; source: rows embedded in https://deepswe.datacurve.ai/", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-7db7b6", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-16wwn5l", "source_n": 1 }, { "model_key": "gpt-5.6-luna", "model_as_reported": "GPT-5.6 Luna", "effort": "low", "harness": "lab-internal", "score_pct": 1.22, "ci_pct": null, "n_samples": null, "cost_per_task_usd": 0.0106, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "lab_self_reported", "verified_primary": true, "source_name": "OpenAI GPT-6 Sol and Luna launch post", "source_url": "https://openai.com/index/introducing-gpt-6-sol-and-luna/", "published": "2026-09-22", "accessed": "2026-10-09", "quote": "{\"model\":\"GPT-5.6 Luna\",\"cost_label\":\"$$0.0106\",\"score\":0.0122,\"cost\":0.0106,\"effortLabel\":\"low\"}", "notes": "OpenAI re-ran GPT-5.6 Luna internally for its GPT-6 Sol/Luna post; the official board measured 67.2% at max. Full effort breakdown from low to max reported in GPT-6 Sol and Luna launch post chart data.", "display_name": "GPT-5.6 Luna", "lab": "OpenAI", "open_weights": false, "row_id": "row-10y4p19", "model_released": "2026-07-10", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1m7ewdf", "source_n": 27 }, { "model_key": "deepseek-v3.2", "model_as_reported": "DeepSeek V3.2", "effort": "", "harness": "mini-swe-agent", "score_pct": 0.6, "ci_pct": 0.9, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "DeepSeek V3.2 | effort: None | score: 0.6% ± 0.9% | n_samples: 339 | provider: DeepSeek", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "DeepSeek V3.2", "lab": "DeepSeek", "open_weights": true, "row_id": "row-1irgvx5", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "nemotron-3-ultra", "model_as_reported": "Nemotron 3 Ultra", "effort": "high", "harness": "mini-swe-agent", "score_pct": 0.6, "ci_pct": 0.7, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Nemotron 3 Ultra | effort: high | score: 0.6% ± 0.7% | n_samples: 339 | provider: NVIDIA", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Nemotron 3 Ultra", "lab": "NVIDIA", "open_weights": true, "row_id": "row-p7xtiy", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "claude-haiku-4.5", "model_as_reported": "claude-haiku-4-5", "effort": "", "harness": "mini-swe-agent", "score_pct": 0.22, "ci_pct": 0.43, "n_samples": null, "cost_per_task_usd": 0.84, "tokens_per_task": 5332574, "input_tokens_per_task": 5293386, "output_tokens_per_task": 39188, "steps_per_task": 109.3, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "claude-haiku-4-5 [None]: 0.22% ±0.43%, cost $0.84, output tokens 39188, steps 109.3", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "Claude Haiku 4.5", "lab": "Anthropic", "open_weights": null, "row_id": "row-1lndd2c", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "minimax-m2.7", "model_as_reported": "minimax-m2-7", "effort": "", "harness": "mini-swe-agent", "score_pct": 0.22, "ci_pct": 0.43, "n_samples": null, "cost_per_task_usd": 0.7, "tokens_per_task": 9519377, "input_tokens_per_task": 9459354, "output_tokens_per_task": 60023, "steps_per_task": 135.6, "source_type": "official_leaderboard", "verified_primary": true, "source_name": "Datacurve DeepSWE 1.1 leaderboard, launch snapshot (Wayback)", "source_url": "https://web.archive.org/web/20260615111157/https://deepswe.datacurve.ai/", "published": "2026-06-15", "accessed": "2026-10-09", "quote": "minimax-m2-7 [None]: 0.22% ±0.43%, cost $0.7, output tokens 60023, steps 135.6", "notes": "On the v1.1 launch board on 2026-06-15; no longer on the live board.", "display_name": "MiniMax M2.7", "lab": "MiniMax", "open_weights": false, "row_id": "row-9s98xd", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-1fnts5d", "source_n": 2 }, { "model_key": "gemma-4-31b", "model_as_reported": "Gemma 4 31B", "effort": "", "harness": "mini-swe-agent", "score_pct": 0, "ci_pct": 0, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Gemma 4 31B | effort: None | score: 0% ± 0% | n_samples: 339 | provider: Google", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Gemma 4 31B", "lab": "Google", "open_weights": true, "row_id": "row-4yiqqu", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "gpt-oss-120b", "model_as_reported": "GPT OSS 120B", "effort": "high", "harness": "mini-swe-agent", "score_pct": 0, "ci_pct": 0, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "GPT OSS 120B | effort: high | score: 0% ± 0% | n_samples: 339 | provider: OpenAI", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "gpt-oss-120b", "lab": "OpenAI", "open_weights": true, "row_id": "row-1tvbm70", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "kimi-k2", "model_as_reported": "Kimi K2", "effort": "high", "harness": "mini-swe-agent", "score_pct": 0, "ci_pct": 0, "n_samples": 339, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Kimi K2 | effort: high | score: 0% ± 0% | n_samples: 339 | provider: Kimi", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Kimi K2", "lab": "Moonshot AI", "open_weights": false, "row_id": "row-1dj6krb", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "minimax-m2.7", "model_as_reported": "MiniMax M2.7", "effort": "high", "harness": "mini-swe-agent", "score_pct": 0, "ci_pct": 0, "n_samples": 337, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "MiniMax M2.7 | effort: high | score: 0% ± 0% | n_samples: 337 | provider: MiniMax", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "MiniMax M2.7", "lab": "MiniMax", "open_weights": false, "row_id": "row-1lhbuv", "model_released": "2026-06-15", "model_released_basis": "first DeepSWE 1.1 result", "source_id": "r-14zikky", "source_n": 37 }, { "model_key": "qwen3.5", "model_as_reported": "Qwen 3.5", "effort": "", "harness": "mini-swe-agent", "score_pct": 0, "ci_pct": 0, "n_samples": 113, "cost_per_task_usd": null, "tokens_per_task": null, "input_tokens_per_task": null, "output_tokens_per_task": null, "steps_per_task": null, "source_type": "third_party_run", "verified_primary": true, "source_name": "Mercor APEX (mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/)", "source_url": "https://www.mercor.com/apex/oss-benchmarks/oss-deep-swe-leaderboard/", "published": "", "accessed": "2026-10-09", "quote": "Qwen 3.5 | effort: None | score: 0% ± 0% | n_samples: 113 | provider: Alibaba", "notes": "Mercor shows no per-model dates; on its board as of the 2026-10-07 capture. Mercor independent evaluation using mini-swe-agent (500 max steps, 2 hour time limit), unit tests all must pass, judge of None, k=3 (113 tasks, 339 samples; 113 samples for k=1).; harness as reported: mini-swe-agent (500 max steps, 2 hour time limit)", "display_name": "Qwen3.5", "lab": "Alibaba", "open_weights": true, "row_id": "row-hdfwj0", "model_released": null, "model_released_basis": null, "source_id": "r-14zikky", "source_n": 37 } ] }