Model evidence / current era
Recorded agent setup
Claude Opus 5
Effort: high · current era
Harness: Claude Code
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
26.7%macro pass estimate
3eligible benchmark task versions
15valid runs on those tasks
0HARNESS_ERROR · excluded in scope
Macro: 26.7% [0.0%, 66.7%] · Micro: 4/15 · 26.7% [10.9%, 52.0%]. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Data processing / data science | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| DeepSWE (long-horizon SWE) | 2 / 10 | Insufficient task coverage | 4/10 | Limited coverage |
| Electrical engineering (RTL) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Games | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Machine learning | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Mechanical engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Science / STEM | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Scientific computing | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Security / cyber | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 1 / 5 | Insufficient task coverage | 0/5 | Limited coverage |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | 118,709 | 7 / 17 |
| Cost (USD) | 15.926 | 7 / 17 |
| Agent turns | 105 | 17 / 17 |
| Duration (seconds) | 1,672.413 | 17 / 17 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| timescaledb-fix-invalidation-and-batchinVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| timescaledb-fix-invalidation-and-batchinVersion 2 | DeepSWE (long-horizon SWE) | 0/0 | contested/defectiveExcluded / other version |
| timescaledb-fix-invalidation-and-batchinVersion 3 | DeepSWE (long-horizon SWE) | 0/0 | contested/defectiveExcluded / other version |
| timescaledb-fix-invalidation-and-batchinVersion 4 | DeepSWE (long-horizon SWE) | 0/0 | contested/defectiveExcluded / other version |
| qdrant-fix-gridstore-has-pointer-lookupVersion 1 | DeepSWE (long-horizon SWE) | 0/1 | contested/defectiveExcluded / other version |
| sqlite-distinct-agg-arityVersion 2 | DeepSWE (long-horizon SWE) | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | contested/defectiveExcluded / other version |
| etcd-fix-race-hint-validationVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| faiss-scan-exactly-max-codes-elements-26Version 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedExcluded / other version |
| faiss-scan-exactly-max-codes-elements-26Version 2 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| faiss-scan-exactly-max-codes-elements-26Version 3 | DeepSWE (long-horizon SWE) | 0/0 | contested/defectiveExcluded / other version |
| timescaledb-fix-segmentby-crash-in-caggVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| qdrant-fix-wrong-multivectors-count-in-sVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| qdrant-fix-wrong-multivectors-count-in-sVersion 2 | DeepSWE (long-horizon SWE) | 0/0 | contested/defectiveExcluded / other version |
| qdrant-fix-wrong-multivectors-count-in-sVersion 3 | DeepSWE (long-horizon SWE) | 0/0 | contested/defectiveExcluded / other version |
| qdrant-fix-wrong-multivectors-count-in-sVersion 4 | DeepSWE (long-horizon SWE) | 5/5 · 100.0% [56.6%, 100.0%] | contested/defectiveExcluded / other version |
| archive-extract-repairVersion 1 | Security / cyber | 1/2 | uncalibratedBenchmark version |
| qdrant-fix-flusher-data-race-in-gridstorVersion 1 | Software engineering | 0/0 | uncalibratedBenchmark version |
| etcd-fix-race-berween-read-index-and-leaVersion 1 | Software engineering | 0/0 | high-fail (unreviewed)Excluded / other version |
| etcd-fix-race-berween-read-index-and-leaVersion 2 | Software engineering | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| rocksdb-fix-compaction-abort-reschedulinVersion 1 | Software engineering | 0/0 | uncalibratedBenchmark version |
| rocksdb-fix-compaction-abort-reschedulinVersion 2 | Software engineering | 0/0 | contested/defectiveExcluded / other version |
| gatt-database-serialize-and-diffVersion 1 | DeepSWE (long-horizon SWE) | 3/5 · 60.0% [23.1%, 88.2%] | mediumBenchmark version |
| slack-block-kit-validatorVersion 1 | DeepSWE (long-horizon SWE) | 1/5 · 20.0% [3.6%, 62.4%] | hardBenchmark version |
| landmark-competing-risk-scorecardVersion 4 | Machine learning | 2/5 · 40.0% [11.8%, 76.9%] | contested/defectiveExcluded / other version |
| occurrence-reinsurance-recovery-bordereauVersion 4 | Operations | 2/5 · 40.0% [11.8%, 76.9%] | contested/defectiveExcluded / other version |
| oci-layer-admission-auditVersion 1 | Security / cyber | 3/5 · 60.0% [23.1%, 88.2%] | contested/defectiveExcluded / other version |
| openssh-certificate-authorizationVersion 1 | Security / cyber | 3/5 · 60.0% [23.1%, 88.2%] | contested/defectiveExcluded / other version |