Model evidence / legacy era
Recorded agent setup
Claude Opus 4.7
Effort: default (unverified) · legacy era
Harness: Claude Code
Effort mode was not recorded; no equivalence to a named effort is assumed.
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
33.5%macro pass estimate
19eligible benchmark task versions
162valid runs on those tasks
18HARNESS_ERROR · excluded in scope
Macro: 33.5% [23.4%, 44.3%] · Micro: 53/162 · 32.7% [26.0%, 40.3%]. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 19 / 162 | 33.5% [23.4%, 44.3%] | 53/162 · 32.7% [26.0%, 40.3%] | ≥5 tasks × ≥5 runs |
| Data processing / data science | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| DeepSWE (long-horizon SWE) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Electrical engineering (RTL) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Games | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Machine learning | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Mechanical engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Science / STEM | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Scientific computing | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Security / cyber | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | — | 0 / 162 |
| Cost (USD) | — | 0 / 162 |
| Agent turns | — | 0 / 162 |
| Duration (seconds) | — | 0 / 162 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| appsync-graphql-cognito-resolver-cache-leakVersion 1 | Cloud operations | 1/10 · 10.0% [1.8%, 40.4%] | legacy-onlyBenchmark version |
| ecr-image-scan-lifecycle-immutable-tags-replicationVersion 1 | Cloud operations | 3/9 · 33.3% [12.1%, 64.6%] | legacy-onlyBenchmark version |
| sfn-secrets-rotation-chainVersion 1 | Cloud operations | 1/9 · 11.1% [2.0%, 43.5%] | legacy-onlyBenchmark version |
| secrets-rotation-kmsVersion 1 | Cloud operations | 5/9 · 55.6% [26.7%, 81.1%] | legacy-onlyBenchmark version |
| sfn-saga-compensation-orchestratorVersion 1 | Cloud operations | 4/8 · 50.0% [21.5%, 78.5%] | legacy-onlyBenchmark version |
| efs-access-point-posix-iam-mount-targetVersion 1 | Cloud operations | 3/9 · 33.3% [12.1%, 64.6%] | legacy-onlyBenchmark version |
| iam-revoke-older-sessionsVersion 1 | Cloud operations | 4/10 · 40.0% [16.8%, 68.7%] | legacy-onlyBenchmark version |
| ddb-outbox-eventbridge-fanoutVersion 1 | Cloud operations | 4/9 · 44.4% [18.9%, 73.3%] | legacy-onlyBenchmark version |
| athena-workgroup-result-encryption-cmk-enforcedVersion 1 | Cloud operations | 2/5 · 40.0% [11.8%, 76.9%] | legacy-onlyBenchmark version |
| glue-etl-catalog-security-configuration-kmsVersion 1 | Cloud operations | 3/8 · 37.5% [13.7%, 69.4%] | legacy-onlyBenchmark version |
| s3-lambda-ddb-pipelineVersion 1 | Cloud operations | 2/6 · 33.3% [9.7%, 70.0%] | legacy-onlyBenchmark version |
| cognito-m2m-httpapi-jwt-scope-gatedVersion 1 | Cloud operations | 1/8 · 12.5% [2.2%, 47.1%] | legacy-onlyBenchmark version |
| iam-cross-account-externalid-sourcearnVersion 1 | Cloud operations | 3/9 · 33.3% [12.1%, 64.6%] | legacy-onlyBenchmark version |
| apigw-sqs-fifo-direct-integrationVersion 1 | Cloud operations | 1/8 · 12.5% [2.2%, 47.1%] | legacy-onlyBenchmark version |
| apigw-http-api-jwt-authorizer-lambda-integrationVersion 1 | Cloud operations | 3/10 · 30.0% [10.8%, 60.3%] | legacy-onlyBenchmark version |
| iam-permissions-boundary-ceilingVersion 1 | Cloud operations | 2/9 · 22.2% [6.3%, 54.7%] | legacy-onlyBenchmark version |
| s3-sqs-image-pipeline-kmsVersion 1 | Cloud operations | 2/9 · 22.2% [6.3%, 54.7%] | legacy-onlyBenchmark version |
| iam-session-tag-tenant-scopeVersion 1 | Cloud operations | 3/10 · 30.0% [10.8%, 60.3%] | legacy-onlyBenchmark version |
| ecs-fargate-secrets-kms-exec-roleVersion 1 | Cloud operations | 6/7 · 85.7% [48.7%, 97.4%] | legacy-onlyBenchmark version |