Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
faiss-scan-exactly-max-codes-elements-26
uncalibratedOutside aggregate scope- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- Not recorded
- Aggregate eligibility
- Excluded / other version
The task
Some searches in this FAISS build return the wrong results Teams using this build of FAISS have started filing reports: certain index operations come back with results that are simply wrong. There is no crash and no error message. The library compiles cleanly, which is exactly how this reached them unnoticed. The reports cluster around the scanned-vector budget limit applied during IVF index searches, but nobody has…
Excerpt only; not a complete executable task specification. Task ID task_c4f682a6d8d6463e · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 0/1 | 1 / 1 | 3 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| Claude Opus 5 · high | 0/0 | 0 / 0 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Every recorded run
22 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_1dff8ccc11074073Claude Opus 5 · default (unverified) | Oct 8, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9662b5ff74db44e6Claude Opus 5 · default (unverified) | Oct 8, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b114a103fff64bbfClaude Opus 5 · default (unverified) | Oct 8, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e7e408d001fb4bceoracle@@ | Oct 8, 2026 console | 1 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_31ca205a43b84ca9nop@@ | Oct 8, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_cfdfef614c5440eaClaude Opus 5 · default (unverified) | Oct 8, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ed14353287a145e5Claude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d9f96d75e4064c35Claude Opus 5 · default (unverified) | Oct 9, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_719aecb13b4447c6Claude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1a951013e8e24893Claude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_94f88dbf06314628Claude Opus 5 · default (unverified) | Oct 9, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1da8204b1d084e2fClaude Opus 5 · default (unverified) | Oct 12, 2026 console | 0 · excludedharness_error | HARNESS_ERROR frozen-classifier | Tokens: 0 Turns: 1 Cost: $0.0000 | Not publicly available |
trial_f694224d3ee642e4Claude Opus 5 · default (unverified) | Oct 12, 2026 console | 0 · excludedharness_error | HARNESS_ERROR frozen-classifier | Tokens: 0 Turns: 1 Cost: $0.0000 | Not publicly available |
trial_296b210d70e7428fClaude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3995de1d4c1447c8Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_62c55d28233e48d8Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_884d53f1c68f4102Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b1000582f7d543d6Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_24698af3ea0a4028Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7eb0531c139e4a67Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f4d94c3c86eb40c2Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f68452ced7494983Claude Opus 5 · high | Oct 21, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |