Model evidence / current era
Recorded agent setup
Claude Opus 5
Effort: default (unverified) · current era
Harness: Claude Code · Harness not recorded
Effort mode was not recorded; no equivalence to a named effort is assumed.
Mixed harnesses; scores describe the recorded systems.
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
57.9%macro pass estimate
98eligible benchmark task versions
531valid runs on those tasks
45HARNESS_ERROR · excluded in scope
Macro: 57.9% [48.8%, 66.7%] · Micro: 279/531 · 52.5% [48.3%, 56.8%]. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Data processing / data science | 1 / 5 | Insufficient task coverage | 2/5 | Limited coverage |
| DeepSWE (long-horizon SWE) | 27 / 169 | 37.8% [22.7%, 54.2%] | 65/169 · 38.5% [31.5%, 46.0%] | ≥5 tasks × ≥5 runs |
| Electrical engineering (RTL) | 3 / 11 | 100.0% [100.0%, 100.0%] | 11/11 · 100.0% [74.1%, 100.0%] | Limited coverage |
| Games | 10 / 50 | 96.0% [88.0%, 100.0%] | 48/50 · 96.0% [86.5%, 98.9%] | ≥5 tasks × ≥5 runs |
| Machine learning | 26 / 139 | 64.6% [47.6%, 80.0%] | 75/139 · 54.0% [45.7%, 62.0%] | ≥5 tasks × ≥5 runs |
| Mechanical engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 2 / 10 | Insufficient task coverage | 0/10 | Limited coverage |
| Science / STEM | 1 / 6 | Insufficient task coverage | 6/6 | Limited coverage |
| Scientific computing | 1 / 3 | Insufficient task coverage | 3/3 · preliminary | Limited coverage |
| Security / cyber | 12 / 62 | 47.8% [22.8%, 72.8%] | 26/62 · 41.9% [30.5%, 54.3%] | ≥5 tasks × ≥5 runs |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 15 / 76 | 60.3% [37.9%, 81.7%] | 43/76 · 56.6% [45.4%, 67.1%] | ≥5 tasks × ≥5 runs |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | 136,059 | 399 / 558 |
| Cost (USD) | 11.013 | 399 / 558 |
| Agent turns | 71 | 399 / 558 |
| Duration (seconds) | 2,089.915 | 399 / 558 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| tcl-rsync-snapshot-integrity-auditVersion 1 | Cloud operations | 2/4 · 50.0% [15.0%, 85.0%] · preliminary | contested/defectiveExcluded / other version |
| bikeshare-daily-leakage-regression-v4Version 1 | Data processing / data science | 5/10 · 50.0% [23.7%, 76.3%] | contested/defectiveExcluded / other version |
| bikeshare-daily-leakage-regressionVersion 1 | Data processing / data science | 2/5 · 40.0% [11.8%, 76.9%] | hardBenchmark version |
| syncrope/crossterm-vt-interpreterVersion 1 | DeepSWE (long-horizon SWE) | 0/3 · 0.0% [0.0%, 56.1%] · preliminary | high-fail (unreviewed)Benchmark version |
| syncrope/semver-pubgrub-resolverVersion 1 | DeepSWE (long-horizon SWE) | 0/2 | high-fail (unreviewed)Benchmark version |
| timescaledb-fix-invalidation-and-batchinVersion 1 | DeepSWE (long-horizon SWE) | 2/2 | uncalibratedBenchmark version |
| qdrant-fix-gridstore-has-pointer-lookupVersion 1 | DeepSWE (long-horizon SWE) | 0/3 · 0.0% [0.0%, 56.1%] · preliminary | contested/defectiveExcluded / other version |
| syncrope/marked-commonmark-strictVersion 1 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| syncrope/babel-messageformat2Version 1 | DeepSWE (long-horizon SWE) | 2/5 · 40.0% [11.8%, 76.9%] | hardBenchmark version |
| sqlite-distinct-agg-arityVersion 1 | DeepSWE (long-horizon SWE) | 0/3 · 0.0% [0.0%, 56.1%] · preliminary | contested/defectiveExcluded / other version |
| syncrope/regexptree-exec-engineVersion 1 | DeepSWE (long-horizon SWE) | 4/5 · 80.0% [37.6%, 96.4%] | mediumBenchmark version |
| syncrope/echo-http-signaturesVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | legacy-onlyBenchmark version |
| syncrope/ammonia-sanitizer-apiVersion 1 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| syncrope/fasthttp-websocket-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/cheerio-aria-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/7 · 0.0% [0.0%, 35.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/networkx-graphml-fullVersion 1 | DeepSWE (long-horizon SWE) | 0/9 · 0.0% [0.0%, 29.9%] | hardExcluded / other version |
| syncrope/uniseg-bidi-uax9Version 1 | DeepSWE (long-horizon SWE) | 1/5 · 20.0% [3.6%, 62.4%] | mediumBenchmark version |
| syncrope/jsbarcode-qr-model2Version 1 | DeepSWE (long-horizon SWE) | 0/4 · 0.0% [0.0%, 49.0%] · preliminary | high-fail (unreviewed)Benchmark version |
| syncrope/linkedom-range-selectionVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | contested/defectiveExcluded / other version |
| syncrope/paramiko-knownhosts-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/9 · 0.0% [0.0%, 29.9%] | hardExcluded / other version |
| syncrope/paramiko-knownhosts-engineVersion 2 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| syncrope/nltk-subword-tokenizationVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/sqliteutils-btree-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | high-fail (unreviewed)Benchmark version |
| syncrope/bibtexparser-biblatex-modelVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | legacy-onlyBenchmark version |
| syncrope/spdx-matching-guidelinesVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | contested/defectiveExcluded / other version |
| syncrope/orb-projection-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| syncrope/oxigraph-shacl-coreVersion 1 | DeepSWE (long-horizon SWE) | 1/5 · 20.0% [3.6%, 62.4%] | mediumExcluded / other version |
| syncrope/oxigraph-shacl-coreVersion 2 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| syncrope/iconv-lite-whatwg-modeVersion 1 | DeepSWE (long-horizon SWE) | 4/5 · 80.0% [37.6%, 96.4%] | mediumBenchmark version |
| faiss-scan-exactly-max-codes-elements-26Version 1 | DeepSWE (long-horizon SWE) | 0/1 | uncalibratedExcluded / other version |
| timescaledb-fix-segmentby-crash-in-caggVersion 1 | DeepSWE (long-horizon SWE) | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| syncrope/jimp-exif-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | legacy-onlyBenchmark version |
| qdrant-fix-wrong-multivectors-count-in-sVersion 1 | DeepSWE (long-horizon SWE) | 1/3 · 33.3% [6.1%, 79.2%] · preliminary | uncalibratedBenchmark version |
| syncrope/linkedom-range-selectionVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | legacy-onlyBenchmark version |
| syncrope/goldmark-cmark-sourceposVersion 1 | DeepSWE (long-horizon SWE) | 1/1 | uncalibratedBenchmark version |
| syncrope/gopher-lua-54-semanticsVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | hardExcluded / other version |
| syncrope/gopher-lua-54-semanticsVersion 2 | DeepSWE (long-horizon SWE) | 1/5 · 20.0% [3.6%, 62.4%] | mediumBenchmark version |
| syncrope/path-to-regexp-urlpatternVersion 1 | DeepSWE (long-horizon SWE) | 1/5 · 20.0% [3.6%, 62.4%] | mediumExcluded / other version |
| syncrope/path-to-regexp-urlpatternVersion 2 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| syncrope/pydub-dsp-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| noc-vc-wormhole-router-v3Version 1 | Electrical engineering (RTL) | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| rv32i-pipeline-precise-traps-v6Version 1 | Electrical engineering (RTL) | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| deflate-inflate-streaming-v3Version 1 | Electrical engineering (RTL) | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| modded-campaign-save-rebaseVersion 1 | Games | 0/0 | uncalibratedBenchmark version |
| mine-sweep-runVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| pathfinding-memory-optimize-gridVersion 1 | Games | 0/0 | uncalibratedBenchmark version |
| tower-siegeVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| negotiator-tableVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| alchemy-cauldronVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| kelp-ring-auctionVersion 1 | Games | 0/0 | uncalibratedBenchmark version |
| crusader-sprite-assemblyVersion 1 | Games | 0/0 | uncalibratedBenchmark version |
| orbit-ferryVersion 1 | Games | 4/5 · 80.0% [37.6%, 96.4%] | mediumBenchmark version |
| tbrain-slantway-move-legalityVersion 1 | Games | 0/0 | uncalibratedBenchmark version |
| outpost-arenaVersion 1 | Games | 4/5 · 80.0% [37.6%, 96.4%] | mediumBenchmark version |
| lattice-signalsVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| lines-of-action-move-legality-v3Version 1 | Games | 2/2 | contested/defectiveExcluded / other version |
| triage-wardVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| arb-deskVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| go-stone-arenaVersion 1 | Games | 0/0 | uncalibratedBenchmark version |
| reef-grazeVersion 1 | Games | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| bank-additional-macro-duration-leakageVersion 1 | Machine learning | 5/8 · 62.5% [30.6%, 86.3%] | mediumBenchmark version |
| squeeze-qwen3.5-0.8b-300mbVersion 1 | Machine learning | 0/0 | uncalibratedBenchmark version |
| cdsandvinyl-imbalance-sentmismatch-text-vocabdriftVersion 1 | Machine learning | 2/2 | uncalibratedBenchmark version |
| lalonde-doubly-robust-ateVersion 1 | Machine learning | 2/2 | uncalibratedBenchmark version |
| mbias-pretreatment-collider-ateVersion 1 | Machine learning | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| healthandhousehold-helpfulvotes-heteroscedastic-missing-monotoneVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| loan-default-cost-imbalanceVersion 1 | Machine learning | 2/5 · 40.0% [11.8%, 76.9%] | hardBenchmark version |
| movies-drama-imbalance-sentmismatch-textVersion 1 | Machine learning | 3/5 · 60.0% [23.1%, 88.2%] | mediumBenchmark version |
| critical-care-survival-censoring-calibrationVersion 1 | Machine learning | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| acsincome-shift-calibrationVersion 1 | Machine learning | 3/5 · 60.0% [23.1%, 88.2%] | mediumBenchmark version |
| maintenance-therapy-competing-risks-cifVersion 1 | Machine learning | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| hausman-taylor-time-invariant-endogenous-effectVersion 1 | Machine learning | 2/2 | uncalibratedBenchmark version |
| adult-calib-imbalanceVersion 1 | Machine learning | 2/2 | uncalibratedBenchmark version |
| clothing-imbalance-leakage-metafusion-textVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| nvss-conceptdrift-heteroscedastic-monotone-multioutVersion 1 | Machine learning | 2/2 | uncalibratedBenchmark version |
| delay-regression-group-holdoutVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| immortal-time-bias-tvc-survival-hr-v2Version 1 | Machine learning | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| forest-fires-skewed-regressionVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| meeting-multitrack-reconcileVersion 1 | Machine learning | 0/10 · 0.0% [0.0%, 27.8%] | hardBenchmark version |
| openfda-drug-route-calib-imbalance-temporalVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| hotel-bookings-modelingVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| sensor-placement-shift-rare-activityVersion 1 | Machine learning | 3/4 · 75.0% [30.1%, 95.4%] · preliminary | uncalibratedBenchmark version |
| airquality-benzene-sensor-leakageVersion 1 | Machine learning | 2/2 | uncalibratedBenchmark version |
| openfda-imbalance-multilabel-textVersion 1 | Machine learning | 0/2 | uncalibratedBenchmark version |
| terminal-bench/benchmark-gold-forensicsVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| petsupplies-coldstartuser-groupcv-metafusion-seqrec-v2Version 1 | Machine learning | 0/2 | uncalibratedBenchmark version |
| bike-sharing-regression-timeawareVersion 1 | Machine learning | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| aps-failure-cost-imbalanceVersion 1 | Machine learning | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| refresh/rlhf-decoding-primitivesVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| fars-leakage-multiclassVersion 1 | Machine learning | 4/10 · 40.0% [16.8%, 68.7%] | hardBenchmark version |
| nhis-adult-calib-pooling-selbias-svywtVersion 1 | Machine learning | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| ai4i-maintenance-leakage-imbalance-costVersion 1 | Machine learning | 0/7 · 0.0% [0.0%, 35.4%] | hardBenchmark version |
| immortal-time-bias-tvc-survival-hrVersion 1 | Machine learning | 1/10 · 10.0% [1.8%, 40.4%] | hardBenchmark version |
| petsupplies-coldstartuser-groupcv-metafusion-seqrecVersion 1 | Machine learning | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| nhanes-brfss-adult-conceptdrift-xcat-selbias-svywtVersion 1 | Machine learning | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| health-baseline-helpfulvotes-monotoneVersion 1 | Machine learning | 3/9 · 33.3% [12.1%, 64.6%] | hardBenchmark version |
| clinical-trial-survival-censoringVersion 1 | Machine learning | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| coil2000-imbalance-metricVersion 1 | Machine learning | 2/2 | uncalibratedBenchmark version |
| syncrope/ab-glyph-renderVersion 1 | Media | 1/1 | contested/defectiveExcluded / other version |
| syncrope/ttfparser-tt-hintingVersion 1 | Media | 0/2 | uncalibratedBenchmark version |
| claimsplit-v3Version 1 | Operations | 0/0 | contested/defectiveExcluded / other version |
| hello-worldVersion 1 | Other / uncategorised | 0/0 | uncalibratedBenchmark version |
| sqlite-file-engine-rs-v2Version 1 | SWE-Marathon | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| archives-mig-rs-v4Version 1 | SWE-Marathon | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| archives-mig-rsVersion 1 | SWE-Marathon | 0/0 | uncalibratedBenchmark version |
| jsonnet-interpreterVersion 1 | SWE-Marathon | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| implement-yaml2jsonVersion 1 | SWE-Marathon | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| terminal-bench/nmr-spin-inversionVersion 1 | Science / STEM | 4/5 · 80.0% [37.6%, 96.4%] | contested/defectiveExcluded / other version |
| syncrope/real-tandf-33so2-nu1nu3-coriolis-heldout-highka-linesVersion 1 | Science / STEM | 2/7 · 28.6% [8.2%, 64.1%] | contested/defectiveExcluded / other version |
| gamma-nuclide-assayVersion 1 | Science / STEM | 6/6 · 100.0% [61.0%, 100.0%] | saturatedBenchmark version |
| aquifer-test-diagnosisVersion 1 | Science / STEM | 0/0 | uncalibratedBenchmark version |
| syncrope/real-cxidb-pt-sapphire-450c-bcdi-heldout-002-fringe-streak-blocksVersion 1 | Scientific computing | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| syncrope/treewidth-pace2017-084-width-gapVersion 1 | Scientific computing | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| grid-inertia-audit-v3Version 1 | Scientific computing | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| contour-cutting-plasticityVersion 1 | Scientific computing | 0/0 | uncalibratedBenchmark version |
| syncrope/treewidth-pace2017-084-width-gapVersion 1 | Scientific computing | 2/2 | uncalibratedBenchmark version |
| zopfli-deflate-compressorVersion 1 | Security / cyber | 4/6 · 66.7% [30.0%, 90.3%] | mediumBenchmark version |
| implement-edVersion 1 | Security / cyber | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| reverse-engineer-listed-substance-timeline-registryVersion 1 | Security / cyber | 4/5 · 80.0% [37.6%, 96.4%] | mediumBenchmark version |
| reverse-engineer-bitmap-indexVersion 1 | Security / cyber | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| reverse-engineer-layout-policy-selectorVersion 1 | Security / cyber | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| archive-extract-repairVersion 1 | Security / cyber | 2/3 · 66.7% [20.8%, 93.9%] · preliminary | uncalibratedBenchmark version |
| deconstruct-header-compressorVersion 1 | Security / cyber | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| reverse-engineer-oscillator-discipline-governorVersion 1 | Security / cyber | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| http-smuggle-repairVersion 1 | Security / cyber | 0/10 · 0.0% [0.0%, 27.8%] | hardBenchmark version |
| seccompward-go-bpf-filter-reachability-v3Version 1 | Security / cyber | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| deconstruct-sat-solverVersion 1 | Security / cyber | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Benchmark version |
| reverse-engineer-divergent-weave-amendment-dossierVersion 1 | Security / cyber | 3/5 · 60.0% [23.1%, 88.2%] | mediumBenchmark version |
| reverse-vm-keygen-v2Version 1 | Security / cyber | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| image-loaderVersion 1 | Software engineering | 3/6 · 50.0% [18.8%, 81.2%] | hardBenchmark version |
| qdrant-fix-flusher-data-race-in-gridstorVersion 1 | Software engineering | 0/3 · 0.0% [0.0%, 56.1%] · preliminary | uncalibratedBenchmark version |
| etcd-fix-race-berween-read-index-and-leaVersion 1 | Software engineering | 0/5 · 0.0% [0.0%, 43.4%] | high-fail (unreviewed)Excluded / other version |
| qdrant-fix-flusher-data-race-in-gridstor-v2Version 1 | Software engineering | 4/7 · 57.1% [25.0%, 84.2%] | mediumBenchmark version |
| timescaledb-fix-set-returning-functionsVersion 1 | Software engineering | 3/10 · 30.0% [10.8%, 60.3%] | hardBenchmark version |
| rocksdb-fix-compaction-abort-reschedulin-v4Version 1 | Software engineering | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| audio-metadata-rust-portVersion 1 | Software engineering | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |
| etcd-fix-race-berween-read-index-and-lea-v3Version 1 | Software engineering | 0/7 · 0.0% [0.0%, 35.4%] | hardBenchmark version |
| timescaledb-fix-set-returning-functions-v4Version 1 | Software engineering | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | uncalibratedBenchmark version |
| rocksdb-fix-compaction-abort-reschedulinVersion 1 | Software engineering | 2/3 · 66.7% [20.8%, 93.9%] · preliminary | uncalibratedBenchmark version |
| tar-rust-rewriteVersion 1 | Software engineering | 5/5 · 100.0% [56.6%, 100.0%] | contested/defectiveExcluded / other version |
| uax14-linebreakVersion 1 | Software engineering | 7/7 · 100.0% [64.6%, 100.0%] | saturatedBenchmark version |
| float-to-stringVersion 1 | Software engineering | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | uncalibratedBenchmark version |
| minpack-solversVersion 1 | Software engineering | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | uncalibratedBenchmark version |
| bignum-arithmeticVersion 1 | Software engineering | 0/4 · 0.0% [0.0%, 49.0%] · preliminary | uncalibratedBenchmark version |
| xmllint-rust-rewriteVersion 1 | Software engineering | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | uncalibratedBenchmark version |
| expr-printf-rust-rewriteVersion 1 | Software engineering | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| wireformats-rsVersion 1 | Software engineering | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| gopher-lua-exact-integers-75f4976Version 3 | DeepSWE (long-horizon SWE) | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| goja-temporal-durationVersion 1 | DeepSWE (long-horizon SWE) | 10/10 · 100.0% [72.2%, 100.0%] | saturatedBenchmark version |
| hono-range-requestsVersion 1 | DeepSWE (long-horizon SWE) | 0/10 · 0.0% [0.0%, 27.8%] | hardBenchmark version |
| click-constraint-groupsVersion 1 | DeepSWE (long-horizon SWE) | 7/9 · 77.8% [45.3%, 93.7%] | mediumBenchmark version |
| hono-http-cache-rfc9111Version 1 | DeepSWE (long-horizon SWE) | 0/10 · 0.0% [0.0%, 27.8%] | hardBenchmark version |
| pypdf-revision-forensicsVersion 1 | DeepSWE (long-horizon SWE) | 2/10 · 20.0% [5.7%, 51.0%] | hardBenchmark version |
| etree-xpath10-engineVersion 1 | DeepSWE (long-horizon SWE) | 3/10 · 30.0% [10.8%, 60.3%] | hardBenchmark version |
| jiff-rrule-recurrenceVersion 1 | DeepSWE (long-horizon SWE) | 3/10 · 30.0% [10.8%, 60.3%] | hardBenchmark version |
| python-docx-tracked-changesVersion 1 | DeepSWE (long-horizon SWE) | 9/10 · 90.0% [59.6%, 98.2%] | saturatedBenchmark version |