Task gallery / Software engineering / Version 1
Task evidence / repo
fix-broken-logrotate
legacy-onlyChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- snorkel review: needs-revision x3
- Aggregate eligibility
- Eligible version
The task
An instruction excerpt is not available in the public snapshot.
Excerpt only; not a complete executable task specification. Task ID review-3fdc31cfc07ae700 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 13/15 · 86.7% [62.1%, 96.3%] | 0 / 15 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.5 · default (unverified) | 9/15 · 60.0% [35.7%, 80.2%] | 0 / 15 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
42 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
tbench-task__eUs8Bt2oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__LsbpZxfnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__LMw4L5Xoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__75pNDYdoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__8e9YYwRClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 25,667 Turns: — Cost: $1.3292 | Not publicly available |
tbench-task__sWnGFRXGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 8,407 Turns: — Cost: $0.7923 | Not publicly available |
tbench-task__s6qm9UeClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 26,611 Turns: — Cost: $1.4237 | Not publicly available |
tbench-task__H38MREpGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 10,857 Turns: — Cost: $0.9800 | Not publicly available |
tbench-task__EivojkLGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 13,532 Turns: — Cost: $1.2005 | Not publicly available |
tbench-task__dfuy8f9GPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 9,705 Turns: — Cost: $1.0500 | Not publicly available |
tbench-task__Gp8UQYjClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 22,382 Turns: — Cost: $1.1588 | Not publicly available |
tbench-task__CcwAvXwGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 9,007 Turns: — Cost: $0.6613 | Not publicly available |
tbench-task__acqvG9tClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 14,764 Turns: — Cost: $0.8036 | Not publicly available |
tbench-task__Yen732GClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 21,622 Turns: — Cost: $1.2207 | Not publicly available |
tbench-task__qaKeSmVnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__BhBKNwYoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__EUXd4Y4oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__jYbs8Kuoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__85bgqAKClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 16,750 Turns: — Cost: $0.7409 | Not publicly available |
tbench-task__bzVre6DClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 21,164 Turns: — Cost: $1.0603 | Not publicly available |
tbench-task__WjdsL3uClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 17,775 Turns: — Cost: $0.8569 | Not publicly available |
tbench-task__NpTMsk7Claude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 23,184 Turns: — Cost: $1.1379 | Not publicly available |
tbench-task__2wCYvdPClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 25,491 Turns: — Cost: $1.2190 | Not publicly available |
tbench-task__QabiKukGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 10,125 Turns: — Cost: $0.9264 | Not publicly available |
tbench-task__YafKmkAGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 14,286 Turns: — Cost: $1.1126 | Not publicly available |
tbench-task__bwzPNXaGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 7,915 Turns: — Cost: $0.6826 | Not publicly available |
tbench-task__b2t9HTQGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 10,511 Turns: — Cost: $1.0037 | Not publicly available |
tbench-task__BBv338cGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 8,264 Turns: — Cost: $0.8669 | Not publicly available |
tbench-task__m7hDMJsoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__FnAg5hmnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__Ek6NCj7oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__y67hXsgoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__uF8fTQiClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 16,748 Turns: — Cost: $0.8109 | Not publicly available |
tbench-task__aZC9E3rClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 27,312 Turns: — Cost: $1.3484 | Not publicly available |
tbench-task__dgdm4AUClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 24,545 Turns: — Cost: $1.2973 | Not publicly available |
tbench-task__QsdvzMBClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 14,596 Turns: — Cost: $0.7100 | Not publicly available |
tbench-task__g2omvyLClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,203 Turns: — Cost: $0.9699 | Not publicly available |
tbench-task__WDV6gpFGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 11,729 Turns: — Cost: $1.1321 | Not publicly available |
tbench-task__VFMJbN6GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 15,247 Turns: — Cost: $1.1842 | Not publicly available |
tbench-task__cbqG6iQGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 12,904 Turns: — Cost: $1.1789 | Not publicly available |
tbench-task__8B7bwL6GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 16,780 Turns: — Cost: $1.3412 | Not publicly available |
tbench-task__CbndEsoGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 9,740 Turns: — Cost: $0.8636 | Not publicly available |