Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/paramiko-knownhosts-engine
hardCurrent calibratedOutside aggregate scope- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Excluded / other version
The task
paramiko's HostKeys matches knownhosts names by literal string equality only: markers, wildcard patterns, bracketed ports, hashed-name writing, certificate trust, and authorizedkeys files are all beyond it. Add a trust-database module paramiko/trustdb.py implementing the OpenSSH formats as documented in sshd(8) (SSHKNOWNHOSTS FILE FORMAT and AUTHORIZEDKEYS FILE FORMAT), sshconfig(5) PATTERNS, and OpenSSH PROTOCOL.ce…
Excerpt only; not a complete executable task specification. Task ID task_752da4e1649540f2 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 0/9 · 0.0% [0.0%, 29.9%] | 9 / 9 | 0 | GOOD_FAILURE: 9 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 1/5 · 20.0% [3.6%, 62.4%] | 5 / 5 | 0 | GOOD_FAILURE: 3 BAD_FAILURE: 1 GOOD_SUCCESS: 1 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 0/5 · 0.0% [0.0%, 43.4%] | 5 / 5 | 0 | GOOD_FAILURE: 5 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| Gemini Flash (latest alias) · default (unverified) | 0/0 | 0 / 0 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Difficulty, with uncertainty
Selected-best difficulty: 80.0% [37.6%, 96.4%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
30 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_01ee568f12634d68Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 84,610 Turns: 37 Cost: $6.1230 | Not publicly available |
trial_0cfb4ab3471f4deaClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 85,629 Turns: 56 Cost: $6.5080 | Not publicly available |
trial_469262c109634e14Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 86,641 Turns: 45 Cost: $5.5248 | Not publicly available |
trial_62ef188e96d64ebbClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 77,112 Turns: 57 Cost: $6.0704 | Not publicly available |
trial_a7060820ff47405fClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 96,704 Turns: 61 Cost: $7.7895 | Not publicly available |
trial_04c022a61726497bClaude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: 140,152 Turns: 102 Cost: $13.0253 | Not publicly available |
trial_1bf76d7f7fd74316Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 96,092 Turns: 53 Cost: $6.5302 | Not publicly available |
trial_2e0899a1fdf847a2Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 120,517 Turns: 49 Cost: $7.3220 | Not publicly available |
trial_431b8c90f8f244cdClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 106,455 Turns: 55 Cost: $7.1920 | Not publicly available |
trial_4d65579cfd594bd7Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 93,187 Turns: 48 Cost: $5.8414 | Not publicly available |
trial_3741c7996df34a1dGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_97ea55210db34957Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9de1dc6101c8443dGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ce8ca68e52404246Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_dc49935dd5144bd6Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2bea4a7060a64580Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 101,279 Turns: 45 Cost: $6.3874 | Not publicly available |
trial_44a977b8fb3242edClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 87,685 Turns: 48 Cost: $5.8976 | Not publicly available |
trial_923c33e762ec4a7fClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 97,143 Turns: 49 Cost: $6.6761 | Not publicly available |
trial_b45ff459c86841ccClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 106,044 Turns: 53 Cost: $7.1469 | Not publicly available |
trial_dfd2cd28c0ce4c5fClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 113,648 Turns: 68 Cost: $8.9060 | Not publicly available |
trial_17c6be22675d4f6fGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a715d50f77764149Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d4f8244a52044d8cGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d6cd7cb704224eb2Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e7490cc265ab4373Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3420ebf0a97e4768GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_67f4a43e9e7d4398GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | BAD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_eabc5442fb9f41daGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f322bf8537ac46f4GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_fa1d7a8963594704GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |