Task gallery / DeepSWE (long-horizon SWE) / Version 4
Task evidence / console
qdrant-fix-wrong-multivectors-count-in-s
contested/defectiveCurrent calibratedOutside aggregate scope- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- deepswe review: OUT of delivery (coordinator, 2026-10-22 ~03:00Z): fails D2; rem
- Aggregate eligibility
- Excluded / other version
Quality exclusion
- Known task defect recorded
These per-version observations remain visible, but this version does not contribute to aggregate pass rates.
The task
Some searches in this Qdrant build return the wrong results Teams using this build of Qdrant have started filing reports: certain queries come back with results that are simply wrong. There is no panic and no error, the request succeeds and the payload it returns is incorrect. The workspace compiles cleanly and cargo test is green, which is exactly how this reached them unnoticed. The reports do have one thing in co…
Excerpt only; not a complete executable task specification. Task ID task_efac1df7688e4a98 · version 4. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · high | 5/5 · 100.0% [56.6%, 100.0%] | 5 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 5 BAD_SUCCESS: 0 |
| Tencent HY4 (preview) · think | 3/8 · 37.5% [13.7%, 69.4%] | 8 / 8 | 0 | GOOD_FAILURE: 5 BAD_FAILURE: 0 GOOD_SUCCESS: 3 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Every recorded run
19 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_9047b93ceabe4659oracle@@ | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_dc55b7129b5b464anop@@ | Oct 22, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_287e9d2b65404846Claude Opus 5 · high | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 112,568 Turns: 137 Cost: $16.7012 | Not publicly available |
trial_72f9ab2cb0d74d8cClaude Opus 5 · high | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 58,338 Turns: 102 Cost: $7.5345 | Not publicly available |
trial_7439e81bd7614e73Claude Opus 5 · high | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 130,743 Turns: 155 Cost: $19.8686 | Not publicly available |
trial_b650b87f62554494Claude Opus 5 · high | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 78,977 Turns: 121 Cost: $12.0364 | Not publicly available |
trial_b9bc13200e4a4b78Claude Opus 5 · high | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 69,384 Turns: 114 Cost: $9.1153 | Not publicly available |
trial_2831f448cd7644e5Tencent HY4 (preview) · think | Oct 22, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 124,380 Turns: 229 Cost: $18.8626 | Not publicly available |
trial_2d83bb7583fc4ffcTencent HY4 (preview) · think | Oct 22, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 188,887 Turns: 150 Cost: $16.6602 | Not publicly available |
trial_44a6534974da4565Tencent HY4 (preview) · think | Oct 22, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 308,934 Turns: 185 Cost: $25.1564 | Not publicly available |
trial_6e7f0be501904205tencent/hy4-preview@@ | Oct 22, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_72c130b67d1d4d92Tencent HY4 (preview) · think | Oct 22, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 312,605 Turns: 138 Cost: $34.4860 | Not publicly available |
trial_7da419046d2c4754tencent/hy4-preview@@ | Oct 22, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7dbe736393334ad2tencent/hy4-preview@@ | Oct 22, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f5d281eb586c40cdtencent/hy4-preview@@ | Oct 22, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2a8726307b3b4cf7Tencent HY4 (preview) · think | Oct 22, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 243,999 Turns: 190 Cost: $23.2637 | Not publicly available |
trial_3455aa6e2b504257Tencent HY4 (preview) · think | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 134,920 Turns: 190 Cost: $15.7186 | Not publicly available |
trial_74cde23298e1452aTencent HY4 (preview) · think | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 179,353 Turns: 119 Cost: $22.0880 | Not publicly available |
trial_cb8b7b36a2034466Tencent HY4 (preview) · think | Oct 22, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 190,875 Turns: 143 Cost: $23.9885 | Not publicly available |