Task gallery / DeepSWE (long-horizon SWE) / Version 2
Task evidence / console
faiss-scan-exactly-max-codes-elements-26
uncalibratedChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
Some searches in this FAISS build return the wrong results Teams using this build of FAISS have started filing reports: certain index operations come back with results that are simply wrong. There is no crash and no error message. The library compiles cleanly, which is exactly how this reached them unnoticed. The reports cluster around the scanned-vector budget limit applied during IVF index searches, but nobody has…
Excerpt only; not a complete executable task specification. Task ID task_c4f682a6d8d6463e · version 2. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · high | 0/0 | 0 / 0 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Every recorded run
17 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_94a27bda273e4a9enop@@ | Oct 21, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d6d4d9bf08f54e0coracle@@ | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2b886da9d1e742b7Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_33201b15228c4279Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6c422bd94b474030Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_819e2f4283e047f5Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_abb6f5922f8144b4Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_128c13f8731343eetencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_281d9a3bf0964493tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2dd4096d58894c19tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_343bafefd8c643a2tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5f890237a1de4034tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_99667bf1e6b94016tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_cac603557a5c437ftencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_da8e15243ff74ffatencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_37a07e1207854554tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_4bc1fc0f210a4fc6tencent/hy4-preview@@ | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |