Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/networkx-graphml-full
hardCurrent calibratedOutside aggregate scope- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Excluded / other version
The task
networkx's GraphML I/O drops nested graphs and ports and rejects hyperedges and per-edge directed overrides. Add a full-fidelity mode as new top-level API: readgraphmlfull(path) / parsegraphmlfull(string) return the list of top-level graphs in document order; writegraphmlfull(graphs, path, encoding="utf-8", prettyprint=True) / generategraphmlfull(graphs, encoding="utf-8", prettyprint=True) (yields lines) take a grap…
Excerpt only; not a complete executable task specification. Task ID task_48e33847ac1244bc · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 0/9 · 0.0% [0.0%, 29.9%] | 9 / 9 | 0 | GOOD_FAILURE: 9 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 1/5 · 20.0% [3.6%, 62.4%] | 5 / 5 | 0 | GOOD_FAILURE: 4 BAD_FAILURE: 0 GOOD_SUCCESS: 1 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 0/5 · 0.0% [0.0%, 43.4%] | 5 / 5 | 0 | GOOD_FAILURE: 5 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| Gemini Flash (latest alias) · default (unverified) | 0/0 | 0 / 0 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Difficulty, with uncertainty
Selected-best difficulty: 80.0% [37.6%, 96.4%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
30 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_5fddcb8311b34767Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 114,920 Turns: 53 Cost: $8.1013 | Not publicly available |
trial_78eef926744f44afClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 111,663 Turns: 57 Cost: $8.0608 | Not publicly available |
trial_7987554525844ab3Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 166,412 Turns: 65 Cost: $9.6193 | Not publicly available |
trial_a50be62ac048436fClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 136,543 Turns: 40 Cost: $7.5436 | Not publicly available |
trial_adcc4ab6f7a145d6Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 110,223 Turns: 49 Cost: $7.4054 | Not publicly available |
trial_2765f9c1c34e4e6eClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 139,583 Turns: 62 Cost: $9.9407 | Not publicly available |
trial_579b2372f0c345fbClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 156,044 Turns: 78 Cost: $12.2778 | Not publicly available |
trial_8451588ba7354739Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c12724f558084ccdClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 153,224 Turns: 77 Cost: $12.2051 | Not publicly available |
trial_d465f78a94804142Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 129,446 Turns: 58 Cost: $8.8475 | Not publicly available |
trial_73eb8edf21ed4a78Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9c09019720314ad7Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a23105e090a64f81Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_cb352c55fec64851Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d6760fa9057d4b90Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_14d7fc43c39f4866Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 151,165 Turns: 67 Cost: $11.2609 | Not publicly available |
trial_9355193040864ea1Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 154,452 Turns: 70 Cost: $11.4179 | Not publicly available |
trial_a92bebca8af84755Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 143,701 Turns: 65 Cost: $10.4185 | Not publicly available |
trial_e86d59386e4d4d49Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 141,146 Turns: 67 Cost: $10.4521 | Not publicly available |
trial_f8a5e0d5520e4ea7Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 150,584 Turns: 84 Cost: $12.4649 | Not publicly available |
trial_1e79390fc2ce4e88Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1f88546aed3443ffGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_4f8bac25631f468eGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_50678a04c9fb43baGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_63da93a73dfc480aGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_273b269510e74408GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5f5df6d4fd3640a6GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9977196059004087GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d3d6460ba1494910GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_dd7474e58b3440a2GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |