Task gallery / Scientific computing / Version 1
Task evidence / repo
lease-quorum-log-recovery
legacy-onlyChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- snorkel review: needs-revision x4
- Aggregate eligibility
- Eligible version
The task
An instruction excerpt is not available in the public snapshot.
Excerpt only; not a complete executable task specification. Task ID review-dfc1160e8ab6f23b · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 0/15 · 0.0% [0.0%, 20.4%] | 0 / 15 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.5 · default (unverified) | 3/15 · 20.0% [7.0%, 45.2%] | 0 / 15 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
42 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
tbench-task__LK3GZYWoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__DbP9csBnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__ZYH32jporacle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__rg2Db4joracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__AgguAgXClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 11,580 Turns: — Cost: $0.6033 | Not publicly available |
tbench-task__M83toQjClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 10,597 Turns: — Cost: $0.5205 | Not publicly available |
tbench-task__srxAqvhClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,801 Turns: — Cost: $0.8168 | Not publicly available |
tbench-task__eTfNgBJClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 12,792 Turns: — Cost: $0.6771 | Not publicly available |
tbench-task__2ySekRaClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 12,682 Turns: — Cost: $0.6979 | Not publicly available |
tbench-task__AMEAq7tGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 14,247 Turns: — Cost: $1.4046 | Not publicly available |
tbench-task__kVJTtQ6GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,799 Turns: — Cost: $1.6677 | Not publicly available |
tbench-task__xM4uQHWGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,630 Turns: — Cost: $1.8480 | Not publicly available |
tbench-task__6TCj4JoGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 31,336 Turns: — Cost: $2.7788 | Not publicly available |
tbench-task__wd2MCJPGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 19,592 Turns: — Cost: $1.8360 | Not publicly available |
tbench-task__Xv9aqBAoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__BpuZkF5oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__gT9zh6Joracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__CtUvvh5nop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__ph6sTaTClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,612 Turns: — Cost: $0.9442 | Not publicly available |
tbench-task__czptZqUClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 13,994 Turns: — Cost: $0.6592 | Not publicly available |
tbench-task__UBPdJaYClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 15,287 Turns: — Cost: $0.7730 | Not publicly available |
tbench-task__MeyZKa9Claude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 11,099 Turns: — Cost: $0.5130 | Not publicly available |
tbench-task__cxdz7WdClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 16,869 Turns: — Cost: $0.8809 | Not publicly available |
tbench-task__mnaoqoKGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,583 Turns: — Cost: $1.7313 | Not publicly available |
tbench-task__KTk8y8FGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,863 Turns: — Cost: $1.8301 | Not publicly available |
tbench-task__ZTzqibyGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 13,946 Turns: — Cost: $1.3859 | Not publicly available |
tbench-task__tYpMXz6GPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 15,379 Turns: — Cost: $1.4213 | Not publicly available |
tbench-task__83TAiRDGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,435 Turns: — Cost: $1.7398 | Not publicly available |
tbench-task__B4XdGh2oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__URDZVuqnop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__zyMCqWuoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__hdV4yM3oracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__FzBUS3FClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 24,074 Turns: — Cost: $1.2450 | Not publicly available |
tbench-task__fCV3sFYClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 22,567 Turns: — Cost: $1.0044 | Not publicly available |
tbench-task__kattc5BClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 10,409 Turns: — Cost: $0.5446 | Not publicly available |
tbench-task__vMsjiH8Claude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,587 Turns: — Cost: $0.9036 | Not publicly available |
tbench-task__F9RGzzkClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 17,141 Turns: — Cost: $0.8339 | Not publicly available |
tbench-task__SogWSxSGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,062 Turns: — Cost: $1.6382 | Not publicly available |
tbench-task__xRS8rFRGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,924 Turns: — Cost: $1.8571 | Not publicly available |
tbench-task__S6PCQHbGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 14,707 Turns: — Cost: $1.1800 | Not publicly available |
tbench-task__2tm9aWWGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 12,570 Turns: — Cost: $1.3226 | Not publicly available |
tbench-task__WzJfVfRGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,407 Turns: — Cost: $2.2514 | Not publicly available |