Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
qdrant-fix-wrong-multivectors-count-in-s
uncalibratedChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
Some searches in this Qdrant build return the wrong results Teams using this build of Qdrant have started filing reports: certain queries come back with results that are simply wrong. There is no panic and no error, the request succeeds and the payload it returns is incorrect. The workspace compiles cleanly and cargo test is green, which is exactly how this reached them unnoticed. The reports do have one thing in co…
Excerpt only; not a complete executable task specification. Task ID task_efac1df7688e4a98 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 1/3 · 33.3% [6.1%, 79.2%] · preliminary | 2 / 3 | 0 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 1 BAD_SUCCESS: 0 |
| Claude Opus 5 · high | 0/0 | 0 / 0 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Every recorded run
19 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_6bff47ba1c3d47f7Claude Opus 5 · default (unverified) | Oct 8, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e9f319e7e58f4b6cClaude Opus 5 · default (unverified) | Oct 8, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_fad49761c3474cd4Claude Opus 5 · default (unverified) | Oct 8, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9bba0d34dfe04292oracle@@ | Oct 8, 2026 console | 1 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2b93b24edfad4e7cnop@@ | Oct 8, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d407cc1adf294ffcClaude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e1798f835c7b498dClaude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: 120,169 Turns: 144 Cost: $17.9432 | Not publicly available |
trial_b686f46892bd414fClaude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_63e2a89940344dbeClaude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_701ddcea36764365Claude Opus 5 · default (unverified) | Oct 9, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 110,145 Turns: 149 Cost: $17.9897 | Not publicly available |
trial_b5ba92699f684566Claude Opus 5 · default (unverified) | Oct 12, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 104,328 Turns: 118 Cost: $13.6299 | Not publicly available |
trial_eade8c7afa7d4281Claude Opus 5 · default (unverified) | Oct 12, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_04ac9d0db2e742edClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_25eb83b9e5854f0aClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3e37d0e2e639494cClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6e5ec6a4a1c5443cClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a2261d002b754fbdClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_759342e0c460441eClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_daf592d021814e0bClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |