Task gallery / Software engineering / Version 1
Task evidence / repo
oci-referrer-gc-import-reconcile
legacy-onlyChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- snorkel review: needs-revision x2
- Aggregate eligibility
- Eligible version
The task
An instruction excerpt is not available in the public snapshot.
Excerpt only; not a complete executable task specification. Task ID review-070ead12e193f6ca · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 9/10 · 90.0% [59.6%, 98.2%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.5 · default (unverified) | 4/10 · 40.0% [16.8%, 68.7%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
28 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
tbench-task__vZq57G6nop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__ATrD94eoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__mGXC2FGoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__8QtMeS8oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__d5kjY63Claude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 50,313 Turns: — Cost: $4.9996 | Not publicly available |
tbench-task__xsGWtW7Claude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 50,233 Turns: — Cost: $5.7867 | Not publicly available |
tbench-task__yGffDGuClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 44,395 Turns: — Cost: $5.0813 | Not publicly available |
tbench-task__AYDRxRgClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 46,849 Turns: — Cost: $3.9955 | Not publicly available |
tbench-task__uASE96DClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 46,098 Turns: — Cost: $5.5241 | Not publicly available |
tbench-task__aq9usi4GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 24,897 Turns: — Cost: $3.2711 | Not publicly available |
tbench-task__TUbUBcqGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 15,974 Turns: — Cost: $2.2973 | Not publicly available |
tbench-task__qVcuXcwGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,560 Turns: — Cost: $3.1197 | Not publicly available |
tbench-task__uLTatpmGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 15,649 Turns: — Cost: $2.5203 | Not publicly available |
tbench-task__rk8JXNYGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 25,078 Turns: — Cost: $4.1227 | Not publicly available |
tbench-task__F3WwqP5oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__iLhahUCoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__VHRUKUBoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__cbwHfcqnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__C3V9AsnClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 45,467 Turns: — Cost: $4.1595 | Not publicly available |
tbench-task__3zNTLKNClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 55,186 Turns: — Cost: $5.3272 | Not publicly available |
tbench-task__xUwUNBjClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 39,239 Turns: — Cost: $3.3442 | Not publicly available |
tbench-task__4G6Dx5XClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 52,676 Turns: — Cost: $4.5305 | Not publicly available |
tbench-task__pPA6NNqClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 42,848 Turns: — Cost: $3.7442 | Not publicly available |
tbench-task__LTsNTPRGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 16,686 Turns: — Cost: $2.2040 | Not publicly available |
tbench-task__k2owe2oGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 20,720 Turns: — Cost: $2.9845 | Not publicly available |
tbench-task__qHWJXmPGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 25,701 Turns: — Cost: $3.3704 | Not publicly available |
tbench-task__rXdFrP7GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,367 Turns: — Cost: $2.3880 | Not publicly available |
tbench-task__5PV6otiGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,416 Turns: — Cost: $3.3458 | Not publicly available |