Model evidence / current era
Recorded agent setup
GPT-5.6-sol
Effort: default (unverified) · current era
Harness: Codex · Harness not recorded
Effort mode was not recorded; no equivalence to a named effort is assumed.
Mixed harnesses; scores describe the recorded systems.
The snapshot flags this row as lane-suspect; recorded outcomes are retained and require caution.
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
36.7%macro pass estimate
24eligible benchmark task versions
150valid runs on those tasks
27HARNESS_ERROR · excluded in scope
Macro: 36.7% [19.6%, 55.0%] · Micro: 46/150 · 30.7% [23.8%, 38.5%]. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Data processing / data science | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| DeepSWE (long-horizon SWE) | 24 / 150 | 36.7% [19.6%, 55.0%] | 46/150 · 30.7% [23.8%, 38.5%] | ≥5 tasks × ≥5 runs |
| Electrical engineering (RTL) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Games | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Machine learning | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Mechanical engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Science / STEM | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Scientific computing | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Security / cyber | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | — | 0 / 152 |
| Cost (USD) | 1.69 | 15 / 152 |
| Agent turns | — | 0 / 152 |
| Duration (seconds) | — | 0 / 152 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| syncrope/crossterm-vt-interpreterVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/semver-pubgrub-resolverVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/marked-commonmark-strictVersion 1 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| oxigraph-write-policy-ec68e3dVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| path-to-regexp-route-fragments-bd12a33Version 1 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| syncrope/babel-messageformat2Version 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| syncrope/regexptree-exec-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | mediumBenchmark version |
| syncrope/ammonia-sanitizer-apiVersion 1 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| syncrope/fasthttp-websocket-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/2 | high-fail (unreviewed)Benchmark version |
| syncrope/cheerio-aria-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/networkx-graphml-fullVersion 1 | DeepSWE (long-horizon SWE) | 1/5 · 20.0% [3.6%, 62.4%] | hardExcluded / other version |
| gopher-lua-exact-integers-75f4976Version 1 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| syncrope/uniseg-bidi-uax9Version 1 | DeepSWE (long-horizon SWE) | 4/5 · 80.0% [37.6%, 96.4%] | mediumBenchmark version |
| syncrope/jsbarcode-qr-model2Version 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/paramiko-knownhosts-engineVersion 1 | DeepSWE (long-horizon SWE) | 1/5 · 20.0% [3.6%, 62.4%] | hardExcluded / other version |
| syncrope/paramiko-knownhosts-engineVersion 2 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| syncrope/nltk-subword-tokenizationVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/sqliteutils-btree-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/spdx-matching-guidelinesVersion 1 | DeepSWE (long-horizon SWE) | 0/1 | contested/defectiveExcluded / other version |
| syncrope/orb-projection-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | high-fail (unreviewed)Benchmark version |
| syncrope/oxigraph-shacl-coreVersion 1 | DeepSWE (long-horizon SWE) | 4/5 · 80.0% [37.6%, 96.4%] | mediumExcluded / other version |
| syncrope/oxigraph-shacl-coreVersion 2 | DeepSWE (long-horizon SWE) | 2/5 · 40.0% [11.8%, 76.9%] | hardBenchmark version |
| syncrope/iconv-lite-whatwg-modeVersion 1 | DeepSWE (long-horizon SWE) | 4/5 · 80.0% [37.6%, 96.4%] | mediumBenchmark version |
| paramiko-knownhosts-engineVersion 1 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| syncrope/gopher-lua-54-semanticsVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | hardExcluded / other version |
| syncrope/gopher-lua-54-semanticsVersion 2 | DeepSWE (long-horizon SWE) | 3/5 · 60.0% [23.1%, 88.2%] | mediumBenchmark version |
| syncrope/path-to-regexp-urlpatternVersion 1 | DeepSWE (long-horizon SWE) | 3/5 · 60.0% [23.1%, 88.2%] | mediumExcluded / other version |
| syncrope/path-to-regexp-urlpatternVersion 2 | DeepSWE (long-horizon SWE) | 4/5 · 80.0% [37.6%, 96.4%] | saturatedBenchmark version |
| syncrope/pydub-dsp-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | high-fail (unreviewed)Benchmark version |
| squeeze-qwen3.5-0.8b-300mbVersion 1 | Machine learning | 0/0 | uncalibratedBenchmark version |
| gopher-lua-exact-integers-75f4976Version 3 | DeepSWE (long-horizon SWE) | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| goja-temporal-durationVersion 1 | DeepSWE (long-horizon SWE) | 0/10 · 0.0% [0.0%, 27.8%] | saturatedBenchmark version |
| hono-range-requestsVersion 1 | DeepSWE (long-horizon SWE) | 0/10 · 0.0% [0.0%, 27.8%] | hardBenchmark version |
| click-constraint-groupsVersion 1 | DeepSWE (long-horizon SWE) | 4/10 · 40.0% [16.8%, 68.7%] | mediumBenchmark version |
| hono-http-cache-rfc9111Version 1 | DeepSWE (long-horizon SWE) | 0/10 · 0.0% [0.0%, 27.8%] | hardBenchmark version |
| etree-xpath10-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/10 · 0.0% [0.0%, 27.8%] | hardBenchmark version |
| python-docx-tracked-changesVersion 1 | DeepSWE (long-horizon SWE) | 0/10 · 0.0% [0.0%, 27.8%] | saturatedBenchmark version |