Task gallery / Security / cyber / Version 1
Task evidence / repo
Snorkel-reviewed task review-a06271859ba7cab0
legacy-onlyChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- snorkel review: needs-revision x1, accept x1
- Aggregate eligibility
- Eligible version
The task
An instruction excerpt is not available in the public snapshot.
Excerpt only; not a complete executable task specification. Task ID review-a06271859ba7cab0 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| GPT-5.5 · default (unverified) | 9/10 · 90.0% [59.6%, 98.2%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| Claude Opus 4.8 · default (unverified) | 2/9 · 22.2% [6.3%, 54.7%] | 0 / 9 | 1 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
28 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
tbench-task__RX98KL2oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__fjNL72Aoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__3dJxyVFnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__gfH48kworacle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__uSvavWVGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 25,218 Turns: — Cost: $2.4969 | Not publicly available |
tbench-task__g6bESGuGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 31,819 Turns: — Cost: $3.0164 | Not publicly available |
tbench-task__7kshunXClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,290 Turns: — Cost: $0.7476 | Not publicly available |
tbench-task__bon8BQZClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 14,340 Turns: — Cost: $0.6539 | Not publicly available |
tbench-task__LcT25mPGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 21,290 Turns: — Cost: $1.7577 | Not publicly available |
tbench-task__m3WHpxUGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 23,573 Turns: — Cost: $1.9831 | Not publicly available |
tbench-task__PrQGe4EGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 23,384 Turns: — Cost: $1.8605 | Not publicly available |
tbench-task__2Dh4qWeClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 29,893 Turns: — Cost: $1.2275 | Not publicly available |
tbench-task__aMpQiHJClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,381 Turns: — Cost: $0.9864 | Not publicly available |
tbench-task__6sZZS3xClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 58,969 Turns: — Cost: $4.4373 | Not publicly available |
tbench-task__qLRmdKWoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__EQkoXY5oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__v5acPFUnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__Xjs2pfLoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__4jNuQeNGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 31,105 Turns: — Cost: $3.0977 | Not publicly available |
tbench-task__8XNojqNClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 13,429 Turns: — Cost: $0.5941 | Not publicly available |
tbench-task__bqRfeDRClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · excludedharness_error | HARNESS_ERROR derived | Tokens: 73,601 Turns: — Cost: $5.0241 | Not publicly available |
tbench-task__PqJ25VwClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 25,086 Turns: — Cost: $0.9746 | Not publicly available |
tbench-task__hrk939ZGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 31,994 Turns: — Cost: $3.3030 | Not publicly available |
tbench-task__WfAPjAxClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,489 Turns: — Cost: $0.8264 | Not publicly available |
tbench-task__eE5RJudClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 27,936 Turns: — Cost: $1.1847 | Not publicly available |
tbench-task__K3tEjSeGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 34,679 Turns: — Cost: $2.9334 | Not publicly available |
tbench-task__iMzF6uiGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 28,242 Turns: — Cost: $2.3549 | Not publicly available |
tbench-task__LukF6omGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 22,733 Turns: — Cost: $2.1823 | Not publicly available |