Task gallery / DeepSWE (long-horizon SWE) / Version 2
Task evidence / console
sqlite-distinct-agg-arity
contested/defectiveOutside aggregate scope- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- deepswe review: EXCLUDED from delivery (coordinator, 2026-10-22 ~00:25Z)
- Aggregate eligibility
- Excluded / other version
Quality exclusion
- Known task defect recorded
These per-version observations remain visible, but this version does not contribute to aggregate pass rates.
The task
This SQLite build crashes on some queries This build of SQLite has started crashing on the teams using it: certain statements kill the process outright where they should have come back with a plain, boring complaint that the statement is wrong -- and the same statement with one small change does produce exactly that complaint. The reports come from reporting queries that summarise stored data into groups. It reprodu…
Excerpt only; not a complete executable task specification. Task ID task_2ad9f7dfda1e4f8c · version 2. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Tencent HY4 (preview) · think | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | 4 / 4 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 4 BAD_SUCCESS: 0 |
| Claude Opus 5 · high | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | 3 / 3 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 3 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Every recorded run
26 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_f4a61845ffd24aafnop@@ | Oct 21, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_fed4e1404dce42beoracle@@ | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7ca5ad311e404bbcTencent HY4 (preview) · think | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 260,980 Turns: 314 Cost: $22.7839 | Not publicly available |
trial_19171be4e7384c4btencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5a0ee22a9b3a4676tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6ccc6009329442e6tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6cd7aff3f5ed4c57tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a4ca41061d034957tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c43f7afc8ce44af1tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c946acd37d024f07Tencent HY4 (preview) · think | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 415,577 Turns: 221 Cost: $30.8592 | Not publicly available |
trial_31641beef04646d3Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_39ec98debe304845Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_82ffccd673ae458cClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f52d724c98bb4c43Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f5a669d27e2f4dc4Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_0ed3c570b9984c8btencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_0edb1d9735db4caaTencent HY4 (preview) · think | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 68,704 Turns: 94 Cost: $5.3002 | Not publicly available |
trial_227f945878254337Tencent HY4 (preview) · think | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 234,609 Turns: 303 Cost: $21.7406 | Not publicly available |
trial_57cb763a4c1b4f2ctencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ad22c4b6944f4342tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c08e6cba74514bcftencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_52605585292d4c9aClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_742e68fd55f341e9Claude Opus 5 · high | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 251,213 Turns: 202 Cost: $34.2426 | Not publicly available |
trial_81e0a82bcc4449e7Claude Opus 5 · high | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 232,588 Turns: 197 Cost: $32.3040 | Not publicly available |
trial_827484eca4a14e72Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c8c74ce0087743d6Claude Opus 5 · high | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 279,930 Turns: 248 Cost: $42.4131 | Not publicly available |