Model evidence / legacy era
Recorded agent setup
Claude Haiku 4.5
Effort: default (unverified) · legacy era
Harness: Claude Code
Effort mode was not recorded; no equivalence to a named effort is assumed.
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
—macro pass estimate
2eligible benchmark task versions
19valid runs on those tasks
52HARNESS_ERROR · excluded in scope
Macro: Insufficient task coverage · Micro: 18/19. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Data processing / data science | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| DeepSWE (long-horizon SWE) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Electrical engineering (RTL) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Games | 1 / 10 | Insufficient task coverage | 10/10 | Limited coverage |
| Machine learning | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Mechanical engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Science / STEM | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Scientific computing | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Security / cyber | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 1 / 9 | Insufficient task coverage | 8/9 | Limited coverage |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | 43,948.5 | 2 / 21 |
| Cost (USD) | 1.056 | 2 / 21 |
| Agent turns | 74.5 | 2 / 21 |
| Duration (seconds) | 538.887 | 2 / 21 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| pydub-dsp-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| jsbarcode-qr-model2Version 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| nltk-subword-tokenizationVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| syncrope/babel-messageformat2Version 1 | DeepSWE (long-horizon SWE) | 0/2 | legacy-onlyBenchmark version |
| crossterm-vt-interpreterVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| semver-pubgrub-resolverVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| orb-projection-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| regexptree-exec-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| cheerio-aria-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| sqliteutils-btree-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| fasthttp-websocket-engineVersion 1 | DeepSWE (long-horizon SWE) | 0/0 | uncalibratedBenchmark version |
| hello-worldVersion 1 | Other / uncategorised | 0/0 | uncalibratedBenchmark version |
| hello-worldVersion 1 | Other / uncategorised | 0/0 | uncalibratedBenchmark version |
| game-of-life-stepVersion 1 | Games | 10/10 · 100.0% [72.2%, 100.0%] | legacy-onlyBenchmark version |
| lru-cacheVersion 1 | Software engineering | 8/9 · 88.9% [56.5%, 98.0%] | legacy-onlyBenchmark version |