Current model coverage

No global ranks: common core 0 tasks; at least five are required. Every row uses its own latest nondefective task subset, with ≥3 valid runs per task. A macro needs ≥3 eligible tasks. Intervals describe sampling uncertainty, not selection bias. Dates: 2026-10-23 snapshot; source dates in footer.

Model / effortAgent harnessTasks / runsMacro pass rate 95% bootstrapMicro · pooled runs 95% Wilson
Claude Opus 5default (unverified)Claude Code · Harness not recorded98 / 53157.9% [48.8%, 66.7%]279/531 · 52.5% [48.3%, 56.8%]
GPT-5.6-soldefault (unverified)Codex · Harness not recorded24 / 15036.7% [19.6%, 55.0%]46/150 · 30.7% [23.8%, 38.5%]
GPT-6 Astradefault (unverified)Codex · offline terminal harness8 / 4120.5% [0.0%, 50.0%]6/41 · 14.6% [6.9%, 28.4%]
GPT-6 AstramediumCodex CLI · evaluation harness7 / 3128.6% [0.0%, 57.1%]Bootstrap cannot generate unseen boundary outcomes.8/31 · 25.8% [13.7%, 43.2%]
Claude Opus 5highClaude Code3 / 1526.7% [0.0%, 66.7%]4/15 · 26.7% [10.9%, 52.0%]
Tencent HY4 (preview)thinkClaude Code3 / 1720.0% [0.0%, 60.0%]3/17 · 17.6% [6.2%, 41.0%]
Tencent HY4 (preview)highClaude Code2 / 16Insufficient task coverageBootstrap cannot generate unseen boundary outcomes.0/16
Claude Opus 5xhighClaude Code1 / 4Insufficient task coverageBootstrap cannot generate unseen boundary outcomes.4/4 · preliminary

Where the evidence exists

Cells show macro pass rate with a bootstrap interval; counts are eligible tasks / valid runs. Shaded cells have ≥5 tasks each with ≥5 runs. Blank cells do not mean failure.

Model / effortCloud operationsData processing / data scienceDeepSWE (long-horizon SWE)Electrical engineering (RTL)GamesMachine learningMechanical engineeringMediaOperationsOther / uncategorisedSWE-MarathonScience / STEMScientific computingSecurity / cyberSnorkel-reviewed tasks (category unknown)Software engineering
Claude Opus 5 · default (unverified)Insufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage1 tasks / 5 runsLimited coverage37.8% [22.7%, 54.2%]27 tasks / 169 runs100.0% [100.0%, 100.0%]3 tasks / 11 runsLimited coverage96.0% [88.0%, 100.0%]10 tasks / 50 runs64.6% [47.6%, 80.0%]26 tasks / 139 runsInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage2 tasks / 10 runsLimited coverageInsufficient task coverage1 tasks / 6 runsLimited coverageInsufficient task coverage1 tasks / 3 runsLimited coverage47.8% [22.8%, 72.8%]12 tasks / 62 runsInsufficient task coverage0 tasks / 0 runsLimited coverage60.3% [37.9%, 81.7%]15 tasks / 76 runs
GPT-5.6-sol · default (unverified)Insufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverage36.7% [19.6%, 55.0%]24 tasks / 150 runsInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverage
GPT-6 Astra · default (unverified)Insufficient task coverage0 tasks / 0 runsLimited coverage20.5% [0.0%, 50.0%]8 tasks / 41 runsInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverage
GPT-6 Astra · mediumInsufficient task coverage0 tasks / 0 runsLimited coverage28.6% [0.0%, 57.1%]7 tasks / 31 runsInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverage
Claude Opus 5 · highInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage2 tasks / 10 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage1 tasks / 5 runsLimited coverage
Tencent HY4 (preview) · thinkInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage1 tasks / 4 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage1 tasks / 5 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage1 tasks / 8 runsLimited coverage
Tencent HY4 (preview) · highInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage2 tasks / 16 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverage
Claude Opus 5 · xhighInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage1 tasks / 4 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverageInsufficient task coverage0 tasks / 0 runsLimited coverage

Compare the shared tasks

Each comparison below keeps model and effort settings separate and uses identical benchmark task versions with ≥3 runs on each row. A paired difference needs at least five shared versions. This exploratory analysis uses a lower repetition threshold than a ranked core (≥5 runs).

GPT-5.6 Sol is lane-suspect in the supplied inventory. Harness differences and retrospective selection remain. Sign diagnostics are conditional on fair independent task signs, which unequal run counts can violate. They are not validated tests of equal model capability. The adjustment covers all testable snapshot pairs, including legacy rows. No superiority or equivalence inference is issued.

The task gallery

All 2,424 task versions remain inspectable, including exclusions. Calibration means repeated runs, not approval. Known curated delivery cohorts are badged; selection status for other records is not fully documented.

2,424 task versions

Legacy evidence

Historical model/effort rows use their own task sets and are never ranked alongside current models. Cross-era comparison: different task sets, harnesses, and dates.

Model / effortAgent harnessTasks / runsMacro pass rate 95% bootstrapMicro · pooled runs 95% Wilson
Claude Opus 4.8default (unverified)Claude Code · Harness not recorded · Terminus 2424 / 2,65549.8% [46.1%, 53.5%]1,294/2,655 · 48.7% [46.8%, 50.6%]
GPT-5.5default (unverified)Codex · Harness not recorded · Terminus 2355 / 1,95149.3% [45.3%, 53.3%]932/1,951 · 47.8% [45.6%, 50.0%]
Claude Opus 4.7default (unverified)Claude Code19 / 16233.5% [23.4%, 44.3%]53/162 · 32.7% [26.0%, 40.3%]
Claude Sonnet 4.6default (unverified)Claude Code · Harness not recorded5 / 3630.0% [10.0%, 51.3%]11/36 · 30.6% [18.0%, 46.9%]
Gemini Flash (latest alias)default (unverified)Gemini CLI3 / 1317.8% [0.0%, 48.9%]2/13 · 15.4% [4.3%, 42.2%]
Claude Haiku 4.5default (unverified)Claude Code2 / 19Insufficient task coverage18/19

Historical inventory audit

This separate view reproduces the original domain–model tables. It pools efforts and all versions, uses snapshot-valid records (including one out-of-range reward), and assigns domains from trial records. It is a provenance check, not the quality-gated benchmark or a model ranking. Oracle/nop are controls. TOTAL includes unrecorded models as the original table did.

Show the original descriptive tables

Table A · Full inventory

Cells: valid runs / descriptive failure %. Historical point estimates reproduce the archive, except n≤2 remains counts only. Full cell values are preserved in the local audit fixture; this inventory does not imply comparisons.

DomainTask keysclaude-opus-4-8gpt-5.5claude-opus-5gpt-5.6-solclaude-opus-4-7openai/gpt-6-astratencent/hy4-previewclaude-sonnet-4-6gemini-flash-latestclaude-haiku-4-5unrecordedTOTAL agent modelsoraclenopbase
Software engineering605713 / 47.3%623 / 53.8%94 / 45.7%0 / counts only0 / counts only0 / counts only12 / 66.7%0 / counts only0 / counts only9 / 11.1%0 / counts only1451 / 49.9%421 / 0.7%157 / 100%0 / counts only
Security / cyber229660 / 40.9%670 / 52.4%79 / 53.2%0 / counts only0 / counts only0 / counts only22 / 68.2%0 / counts only0 / counts only0 / counts only0 / counts only1431 / 47.4%443 / 0%155 / 100%0 / counts only
Scientific computing269687 / 55.7%602 / 49.8%11 / 0%0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only2 / counts only1302 / 52.5%382 / 0%136 / 100%0 / counts only
Data processing / data science287476 / 45.6%441 / 55.8%15 / 53.3%0 / counts only0 / counts only98 / 64.3%0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only1030 / 51.8%437 / 0.5%257 / 100%0 / counts only
Machine learning303428 / 54.7%325 / 51.1%168 / 42.3%0 / counts only0 / counts only0 / counts only9 / 100%5 / 100%0 / counts only0 / counts only0 / counts only935 / 51.9%266 / 0.4%124 / 100%0 / counts only
Games96290 / 45.5%243 / 46.9%52 / 3.8%0 / counts only0 / counts only0 / counts only0 / counts only31 / 64.5%0 / counts only10 / 0%0 / counts only626 / 42.8%169 / 0%67 / 100%0 / counts only
DeepSWE (long-horizon SWE)51100 / 81%0 / counts only242 / 64.5%183 / 67.8%0 / counts only0 / counts only29 / 75.9%0 / counts only27 / 81.5%2 / counts only5 / 80%588 / 69.9%26 / 0%17 / 100%1 / counts only
Electrical engineering (RTL)66130 / 52.3%26 / 26.9%13 / 0%0 / counts only0 / counts only0 / counts only2 / counts only0 / counts only0 / counts only0 / counts only0 / counts only171 / 43.9%14 / 0%9 / 100%0 / counts only
Cloud operations270 / counts only0 / counts only4 / 50%0 / counts only162 / 67.3%0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only166 / 66.9%2 / counts only2 / counts only0 / counts only
Science / STEM59109 / 28.4%20 / 55%18 / 33.3%0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only2 / counts only149 / 32.9%29 / 0%20 / 100%0 / counts only
Mechanical engineering26114 / 73.7%0 / counts only5 / 0%0 / counts only0 / counts only0 / counts only5 / 100%0 / counts only0 / counts only0 / counts only0 / counts only124 / 71.8%10 / 0%5 / 100%0 / counts only
Snorkel-reviewed tasks (category unknown)17740 / 40%40 / 47.5%0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only80 / 43.8%27 / 0%8 / 100%0 / counts only
Operations100 / counts only0 / counts only8 / 75%0 / counts only0 / counts only0 / counts only9 / 100%0 / counts only0 / counts only0 / counts only0 / counts only17 / 88.2%10 / 0%5 / 100%0 / counts only
SWE-Marathon70 / counts only0 / counts only16 / 62.5%0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only16 / 62.5%3 / 0%3 / 100%0 / counts only
Media60 / counts only0 / counts only3 / 66.7%0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only0 / counts only3 / 66.7%3 / 0%3 / 66.7%0 / counts only
TOTAL22263747 / 49.5%2990 / 51.8%728 / 47.8%183 / 67.8%162 / 67.3%98 / 64.3%88 / 77.3%36 / 69.4%27 / 81.5%21 / 14.3%9 / 66.7%8089 / 51.6%2242 / 0.3%968 / 99.9%1 / counts only

Table B · New since September 11, 2026

Cells: valid runs / descriptive failure %. Historical point estimates reproduce the archive, except n≤2 remains counts only. Full cell values are preserved in the local audit fixture; this inventory does not imply comparisons.

DomainTask keysclaude-opus-5openai/gpt-6-astratencent/hy4-previewTOTAL agent modelsoraclenopbase
Machine learning44163 / 41.7%0 / counts only1 / counts only164 / 42.1%50 / 0%51 / 100%0 / counts only
Data processing / data science2515 / 53.3%98 / 64.3%0 / counts only113 / 62.8%153 / 0%153 / 100%0 / counts only
Software engineering1894 / 45.7%0 / counts only12 / 66.7%106 / 48.1%19 / 10.5%19 / 100%0 / counts only
Security / cyber1979 / 53.2%0 / counts only22 / 68.2%101 / 56.4%13 / 0%9 / 100%0 / counts only
DeepSWE (long-horizon SWE)934 / 47.1%0 / counts only29 / 75.9%63 / 60.3%12 / 0%12 / 100%0 / counts only
Games1152 / 3.8%0 / counts only0 / counts only52 / 3.8%11 / 0%11 / 100%0 / counts only
Science / STEM418 / 33.3%0 / counts only0 / counts only18 / 33.3%10 / 0%9 / 100%0 / counts only
SWE-Marathon516 / 62.5%0 / counts only0 / counts only16 / 62.5%3 / 0%3 / 100%0 / counts only
Electrical engineering (RTL)713 / 0%0 / counts only2 / counts only15 / 0%14 / 0%9 / 100%0 / counts only
Scientific computing411 / 0%0 / counts only0 / counts only11 / 0%4 / 0%4 / 100%0 / counts only
Mechanical engineering25 / 0%0 / counts only5 / 100%10 / 50%10 / 0%5 / 100%0 / counts only
Cloud operations14 / 50%0 / counts only0 / counts only4 / 50%2 / counts only2 / counts only0 / counts only
Operations43 / 100%0 / counts only1 / counts only4 / 100%10 / 0%5 / 100%0 / counts only
Media23 / 66.7%0 / counts only0 / counts only3 / 66.7%3 / 0%3 / 66.7%0 / counts only
TOTAL155510 / 39.6%98 / 64.3%72 / 72.2%680 / 46.6%314 / 0.6%295 / 99.7%0 / counts only