Claude Opus 5
Effort: xhigh · current era
Harness: Claude Code
Macro: Insufficient task coverage · Micro: 4/4 · preliminary. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows. Empirical bootstrap cannot generate unseen successes or failures in boundary cells; a narrow interval does not establish certainty.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Data processing / data science | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| DeepSWE (long-horizon SWE) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Electrical engineering (RTL) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Games | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Machine learning | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Mechanical engineering | 1 / 4 | Insufficient task coverage | 4/4 · preliminary | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Science / STEM | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Scientific computing | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Security / cyber | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | 102,565 | 9 / 9 |
| Cost (USD) | 4.971 | 9 / 9 |
| Agent turns | 43 | 9 / 9 |
| Duration (seconds) | — | 0 / 9 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| rtl-axi-event-snapshot-bankVersion 1 | Electrical engineering (RTL) | 2/2 | uncalibratedBenchmark version |
| landmark-competing-risk-scorecardVersion 1 | Machine learning | 0/0 | uncalibratedBenchmark version |
| cam-modal-toolpath-envelopeVersion 1 | Mechanical engineering | 1/1 | uncalibratedBenchmark version |
| dxf-nested-assembly-scheduleVersion 1 | Mechanical engineering | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | uncalibratedBenchmark version |
| family-benefit-accumulator-reprocessingVersion 1 | Operations | 0/2 | contested/defectiveExcluded / other version |
| occurrence-reinsurance-recovery-bordereauVersion 1 | Operations | 0/1 | uncalibratedBenchmark version |
| oci-layer-admission-auditVersion 1 | Security / cyber | 0/1 | mediumBenchmark version |