Task gallery / Software engineering / Version 1
Reference solution (oracle)
Pass
No-op resistance check
Pass
QA verdict
taxonomy: Rejected: too-easy (screen 3/3, SEE-confirmed) -> v5 TODO; QA: Local Q
Aggregate eligibility
Excluded / other version
Quality exclusion
  • Known task defect recorded

These per-version observations remain visible, but this version does not contribute to aggregate pass rates.

The task

Some operations in this RocksDB build abort instead of completing Teams using this build of RocksDB, a key-value store, have started filing reports. Some have the same shape: an operation that should complete instead brings the process down. No exception they can catch, no error message, just a dead process and whatever the caller was holding lost with it. Others are quieter: background compaction work that was abor…

Excerpt only; not a complete executable task specification. Task ID task_540eef91fb3045b3 · version 1. Selection status: not fully recorded.

Results by model and effort

This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.

Current evidence

Model / effortVerifier result [95% Wilson]Diagnostic labels / validHARNESS_ERROR · excludedOutcome classes
Claude Opus 5 · default (unverified)3/3 · 100.0% [43.9%, 100.0%] · preliminary3 / 33
GOOD_FAILURE: 0
BAD_FAILURE: 0
GOOD_SUCCESS: 3
BAD_SUCCESS: 0

Legacy evidence

Model / effortVerifier result [95% Wilson]Diagnostic labels / validHARNESS_ERROR · excludedOutcome classes

Every recorded run

19 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.

Run / modelDate / sourceReward / eligibilityDiagnostic class / sourceResourcesEvidence
trial_9e4e2861b7f74097
nop@@
Oct 13, 2026
console
0 · validGOOD_FAILURE
frozen-classifier
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_c841b435ade146d0
oracle@@
Oct 13, 2026
console
1 · validGOOD_SUCCESS
frozen-classifier
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_5e882883a0a74956
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
0 · excludedharness_errorHARNESS_ERROR
independent-review
Tokens: 50,493
Turns: 68
Cost: $5.0788
Not publicly available
trial_cdc3abf8d3664b47
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
0 · excludedharness_errorHARNESS_ERROR
independent-review
Tokens: 0
Turns: 1
Cost: $0.0000
Not publicly available
trial_ea7d37b35d3c41dc
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
0 · excludedharness_errorHARNESS_ERROR
independent-review
Tokens: 0
Turns: 1
Cost: $0.0000
Not publicly available
trial_15b017d38bf54b47
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_2327215492d94870
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_35611a52349343a4
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_511e73831a854a08
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_8be28d7f359643dc
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_c948206e440f4fad
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_e20a9d25be6a468c
Claude Opus 5 · default (unverified)
Oct 13, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_3e53968fe6af4d34
Claude Opus 5 · default (unverified)
Oct 14, 2026
console
1 · validGOOD_SUCCESS
frozen-classifier
Tokens: 128,048
Turns: 159
Cost: $18.4621
Not publicly available
trial_7a6db527c9824748
Claude Opus 5 · default (unverified)
Oct 14, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_9f636c7490824df7
Claude Opus 5 · default (unverified)
Oct 14, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_a13a60a5ca914cf1
Claude Opus 5 · default (unverified)
Oct 14, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_a967149f1add45d7
Claude Opus 5 · default (unverified)
Oct 14, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_e2b4ebf352694228
Claude Opus 5 · default (unverified)
Oct 14, 2026
console
1 · validGOOD_SUCCESS
frozen-classifier
Tokens: 279,983
Turns: 309
Cost: $50.9585
Not publicly available
trial_ee756743882b4a74
Claude Opus 5 · default (unverified)
Oct 14, 2026
console
1 · validGOOD_SUCCESS
frozen-classifier
Tokens: 128,868
Turns: 181
Cost: $20.6749
Not publicly available