Task gallery / Software engineering / Version 1
Task evidence / repo
relpack-npm10-payload-selector
contested/defectiveOutside aggregate scope- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- snorkel review: needs-revision x1, accept x1
- Aggregate eligibility
- Excluded / other version
Quality exclusion
- Known task defect recorded
These per-version observations remain visible, but this version does not contribute to aggregate pass rates.
The task
An instruction excerpt is not available in the public snapshot.
Excerpt only; not a complete executable task specification. Task ID review-b71ea4a732ff3d72 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| GPT-5.5 · default (unverified) | 0/10 · 0.0% [0.0%, 27.8%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| Claude Opus 4.8 · default (unverified) | 0/10 · 0.0% [0.0%, 27.8%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
28 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
tbench-task__8pKyVzForacle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__5voRvkCoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__7CurQZqoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__AiAkWyenop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__HgTMGoZGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 18,486 Turns: — Cost: $2.1817 | Not publicly available |
tbench-task__unCuENLClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,855 Turns: — Cost: $1.0344 | Not publicly available |
tbench-task__ZqGpAvZGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 33,460 Turns: — Cost: $3.6371 | Not publicly available |
tbench-task__v6wSx6bClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,405 Turns: — Cost: $1.0723 | Not publicly available |
tbench-task__33mBDAzClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 18,333 Turns: — Cost: $1.1218 | Not publicly available |
tbench-task__VAT8yWGGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 18,756 Turns: — Cost: $2.3801 | Not publicly available |
tbench-task__byRYLqJClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 22,381 Turns: — Cost: $1.4665 | Not publicly available |
tbench-task__aB2YzneGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,583 Turns: — Cost: $2.5775 | Not publicly available |
tbench-task__zdkwuMFGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 26,755 Turns: — Cost: $3.6456 | Not publicly available |
tbench-task__S8d8dmpClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 20,856 Turns: — Cost: $1.4104 | Not publicly available |
tbench-task__qfShejLoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__Sh5gEgGnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__czGyVakoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__ekC34NZoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__ore3dCRClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 20,955 Turns: — Cost: $1.4965 | Not publicly available |
tbench-task__6qQtDb4GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 25,140 Turns: — Cost: $3.2777 | Not publicly available |
tbench-task__4oeHxZzClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 22,219 Turns: — Cost: $1.4297 | Not publicly available |
tbench-task__y9NUGmcClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 28,413 Turns: — Cost: $2.0222 | Not publicly available |
tbench-task__R98SGtDGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,678 Turns: — Cost: $3.0361 | Not publicly available |
tbench-task__y2aadCMGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 33,719 Turns: — Cost: $4.1086 | Not publicly available |
tbench-task__8RrffXBClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 22,741 Turns: — Cost: $1.3710 | Not publicly available |
tbench-task__QTBCuUFGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 24,221 Turns: — Cost: $3.0616 | Not publicly available |
tbench-task__KhokpWPClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,559 Turns: — Cost: $1.2537 | Not publicly available |
tbench-task__9zQMhsPGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 29,098 Turns: — Cost: $3.4318 | Not publicly available |