Model evidence / legacy era
Recorded agent setup
Claude Sonnet 4.6
Effort: default (unverified) · legacy era
Harness: Claude Code · Harness not recorded
Effort mode was not recorded; no equivalence to a named effort is assumed.
Mixed harnesses; scores describe the recorded systems.
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
30.0%macro pass estimate
5eligible benchmark task versions
36valid runs on those tasks
9HARNESS_ERROR · excluded in scope
Macro: 30.0% [10.0%, 51.3%] · Micro: 11/36 · 30.6% [18.0%, 46.9%]. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Data processing / data science | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| DeepSWE (long-horizon SWE) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Electrical engineering (RTL) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Games | 4 / 31 | 37.5% [19.2%, 59.2%] | 11/31 · 35.5% [21.1%, 53.1%] | Limited coverage |
| Machine learning | 1 / 5 | Insufficient task coverage | 0/5 | Limited coverage |
| Mechanical engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Science / STEM | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Scientific computing | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Security / cyber | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | — | 0 / 36 |
| Cost (USD) | — | 0 / 36 |
| Agent turns | — | 0 / 36 |
| Duration (seconds) | — | 0 / 36 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| task_0040_day_night_cycle_controllerVersion 1 | Games | 2/10 · 20.0% [5.7%, 51.0%] | legacy-onlyBenchmark version |
| camera-shake-rigVersion 1 | Games | 3/6 · 50.0% [18.8%, 81.2%] | legacy-onlyBenchmark version |
| crusader-sprite-assemblyVersion 1 | Games | 4/10 · 40.0% [16.8%, 68.7%] | legacy-onlyBenchmark version |
| tile-gradient-particlesVersion 1 | Games | 2/5 · 40.0% [11.8%, 76.9%] | legacy-onlyBenchmark version |
| qat-int2-cifarVersion 1 | Machine learning | 0/5 · 0.0% [0.0%, 43.4%] | legacy-onlyBenchmark version |