Task gallery / Machine learning / Version 1
Task evidence / repo
qualify-kotlin-h2-conformal-release-submit-reviewerfix
legacy-onlyChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- snorkel review: needs-revision x1, accept x1
- Aggregate eligibility
- Eligible version
The task
An instruction excerpt is not available in the public snapshot.
Excerpt only; not a complete executable task specification. Task ID review-1da75e480c20ded0 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 5/10 · 50.0% [23.7%, 76.3%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.5 · default (unverified) | 9/10 · 90.0% [59.6%, 98.2%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
28 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
tbench-task__vXRevkEnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__dpis9n8oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__Hd2zfoVoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__mktzR2Qoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__aDkVf8hClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 16,968 Turns: — Cost: $1.1675 | Not publicly available |
tbench-task__xGP3iPkClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,715 Turns: — Cost: $1.3361 | Not publicly available |
tbench-task__G3GUBViClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 20,131 Turns: — Cost: $1.4112 | Not publicly available |
tbench-task__TUVA8JSClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 23,668 Turns: — Cost: $1.6978 | Not publicly available |
tbench-task__MzkGdPsClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 21,490 Turns: — Cost: $1.4537 | Not publicly available |
tbench-task__BQbK9s8GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 33,986 Turns: — Cost: $4.0708 | Not publicly available |
tbench-task__5V9sNErGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 16,875 Turns: — Cost: $1.5271 | Not publicly available |
tbench-task__YxMNiYUGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,081 Turns: — Cost: $2.3649 | Not publicly available |
tbench-task__HJiCYrdGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 18,341 Turns: — Cost: $1.6447 | Not publicly available |
tbench-task__BpNQB4fGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,952 Turns: — Cost: $2.0359 | Not publicly available |
tbench-task__KXvZjMAnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__9KdSjvboracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__sYbqXThoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__GHJC6sBoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__rVzTHbjClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 18,655 Turns: — Cost: $1.4910 | Not publicly available |
tbench-task__kbmqqQLClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 23,172 Turns: — Cost: $1.6818 | Not publicly available |
tbench-task__GbqT2BKClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,171 Turns: — Cost: $1.6190 | Not publicly available |
tbench-task__ck32K2mClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 20,561 Turns: — Cost: $1.8974 | Not publicly available |
tbench-task__fVVA9G8Claude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 23,563 Turns: — Cost: $2.1734 | Not publicly available |
tbench-task__WXVcXyMGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 21,111 Turns: — Cost: $2.0248 | Not publicly available |
tbench-task__QU2szGQGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,182 Turns: — Cost: $1.9349 | Not publicly available |
tbench-task__rRWhF5UGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 18,967 Turns: — Cost: $1.9233 | Not publicly available |
tbench-task__w6DeVC3GPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,168 Turns: — Cost: $1.8915 | Not publicly available |
tbench-task__rHC33ysGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,253 Turns: — Cost: $1.9864 | Not publicly available |