Task gallery / Security / cyber / Version 1
Task evidence / repo
nocturne-ed25519-replay-registry
legacy-onlyChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- snorkel review: accept x1, needs-revision x1
- Aggregate eligibility
- Eligible version
The task
An instruction excerpt is not available in the public snapshot.
Excerpt only; not a complete executable task specification. Task ID review-58133a92c9deeb38 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 2/10 · 20.0% [5.7%, 51.0%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.5 · default (unverified) | 6/10 · 60.0% [31.3%, 83.2%] | 0 / 10 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
28 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
tbench-task__Jbvx79inop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__73YgGxroracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__b6JgHJaoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__eZWVrMWoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__kbPn3STClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 20,154 Turns: — Cost: $1.0167 | Not publicly available |
tbench-task__WKkLmaKClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 21,061 Turns: — Cost: $1.0528 | Not publicly available |
tbench-task__Mt8WgsiClaude Opus 4.8 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 23,628 Turns: — Cost: $1.1797 | Not publicly available |
tbench-task__qCAN7rsClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 23,964 Turns: — Cost: $1.1779 | Not publicly available |
tbench-task__H2cfMhpClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 19,727 Turns: — Cost: $0.9869 | Not publicly available |
tbench-task__gssKchPGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 24,020 Turns: — Cost: $2.2488 | Not publicly available |
tbench-task__qzJAVvaGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 23,358 Turns: — Cost: $2.2045 | Not publicly available |
tbench-task__DVnLa8qGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 18,337 Turns: — Cost: $1.6824 | Not publicly available |
tbench-task__G3Cx6KaGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 28,765 Turns: — Cost: $2.7143 | Not publicly available |
tbench-task__BR2qEU5GPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 20,159 Turns: — Cost: $1.7218 | Not publicly available |
tbench-task__wKR7mnLoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__tXvWQGRoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__JSz9NdRoracle@@ | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__A8kVvT7nop@@ | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
tbench-task__bksoq4NClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 20,962 Turns: — Cost: $1.0398 | Not publicly available |
tbench-task__EptDHR6Claude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 22,381 Turns: — Cost: $1.1217 | Not publicly available |
tbench-task__V5C38dbClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 22,347 Turns: — Cost: $1.1309 | Not publicly available |
tbench-task__8Vx3fWkClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,718 Turns: — Cost: $1.0511 | Not publicly available |
tbench-task__yAwj8LkClaude Opus 4.8 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 20,561 Turns: — Cost: $0.9857 | Not publicly available |
tbench-task__SwNP7RAGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 25,512 Turns: — Cost: $2.7372 | Not publicly available |
tbench-task__oXXCpRYGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,169 Turns: — Cost: $2.2321 | Not publicly available |
tbench-task__U5HRdHpGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 21,238 Turns: — Cost: $2.6066 | Not publicly available |
tbench-task__epM9gasGPT-5.5 · default (unverified) | Not recorded repo | 0 · valid | No diagnostic label derived | Tokens: 31,025 Turns: — Cost: $3.4585 | Not publicly available |
tbench-task__fbUXwgEGPT-5.5 · default (unverified) | Not recorded repo | 1 · valid | No diagnostic label derived | Tokens: 24,614 Turns: — Cost: $2.8753 | Not publicly available |