Task gallery / DeepSWE (long-horizon SWE) / Version 1
Reference solution (oracle)
Unverified
No-op resistance check
Unverified
QA verdict
Not recorded
Aggregate eligibility
Eligible version

The task

Some operations in this etcd build return the wrong results Teams using this build of etcd, a distributed key-value store, have started filing reports: a read comes back with a value a later write already replaced, a write the cluster acknowledged is missing from the next read, or a key is returned at the wrong revision. There is no crash and no error message. It compiles cleanly and its own test suite is green, whi…

Excerpt only; not a complete executable task specification. Task ID task_32393fcd1ac640e3 · version 1. Selection status: not fully recorded.

Results by model and effort

This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.

Current evidence

Model / effortVerifier result [95% Wilson]Diagnostic labels / validHARNESS_ERROR · excludedOutcome classes
Claude Opus 5 · high0/00 / 00
GOOD_FAILURE: 0
BAD_FAILURE: 0
GOOD_SUCCESS: 0
BAD_SUCCESS: 0

Legacy evidence

Model / effortVerifier result [95% Wilson]Diagnostic labels / validHARNESS_ERROR · excludedOutcome classes

Every recorded run

4 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.

Run / modelDate / sourceReward / eligibilityDiagnostic class / sourceResourcesEvidence
trial_4e8938a7c8c742b2
Claude Opus 5 · high
Oct 22, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_5d731fce7e0b437e
Claude Opus 5 · high
Oct 22, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_ab98e78c44524ff0
tencent/hy4-preview@@
Oct 22, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available
trial_cc5e3d8f51094999
tencent/hy4-preview@@
Oct 22, 2026
console
No reward · excludedno_rewardNo diagnostic label
No diagnostic source
Tokens: —
Turns: —
Cost: —
Not publicly available