Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/marked-commonmark-strict
saturatedCurrent calibratedChosen benchmark version- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
marked deliberately trades full CommonMark conformance for speed and backward compatibility: even with gfm: false, its output departs from the spec's reference HTML in many places, from serialization details up to real structural corners. Add a strict-conformance mode. Public API: one new boolean option, commonmark (default false), declared alongside gfm/pedantic in the options types and defaults, and accepted every…
Excerpt only; not a complete executable task specification. Task ID task_14e103e2e9454d6b · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 5/5 · 100.0% [56.6%, 100.0%] | 5 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 5 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 5/5 · 100.0% [56.6%, 100.0%] | 5 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 5 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 4/5 · 80.0% [37.6%, 96.4%] | 5 / 5 | 0 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 4 BAD_SUCCESS: 0 |
| Gemini Flash (latest alias) · default (unverified) | 1/5 · 20.0% [3.6%, 62.4%] | 5 / 5 | 1 | GOOD_FAILURE: 3 BAD_FAILURE: 1 GOOD_SUCCESS: 0 BAD_SUCCESS: 1 |
Difficulty, with uncertainty
Selected-best difficulty: 0.0% [0.0%, 43.4%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
24 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_082519acdcbd4301Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 132,184 Turns: 112 Cost: $14.7127 | Not publicly available |
trial_3e92575deb164754Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 141,788 Turns: 133 Cost: $17.7354 | Not publicly available |
trial_53ec4a70bb7a4957Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 115,041 Turns: 71 Cost: $10.9451 | Not publicly available |
trial_6b0b5a4ad0994ed2Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 110,816 Turns: 88 Cost: $10.6629 | Not publicly available |
trial_e722752ce00a4a29Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 98,569 Turns: 88 Cost: $10.4293 | Not publicly available |
trial_1a10cad4ca174c29Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 126,873 Turns: 122 Cost: $16.8037 | Not publicly available |
trial_36563b3d1a614911Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 138,513 Turns: 98 Cost: $15.7982 | Not publicly available |
trial_39e62cfce2214162Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 123,507 Turns: 121 Cost: $16.2575 | Not publicly available |
trial_423da4a12f564b3aClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 119,753 Turns: 91 Cost: $13.0522 | Not publicly available |
trial_53ec029acc8544bdClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 139,127 Turns: 122 Cost: $18.5301 | Not publicly available |
trial_172507b1a5424600Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 1 · valid | BAD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_44915ec7015a4e00Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c3d91d9e795c48d4Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e44e26be83354ba7Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ee5904ba8ac2432fGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_482e47cea3784b66GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5c179d9b35034b35GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_610db8d83b9d42e7GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_80b3763a33e84254GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9c585597551d4554GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_21b02fc86c7215d0Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f1d12e3c69372a18Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · valid | BAD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_35f96d7abd020953Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedharness_error | HARNESS_ERROR frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d4cba4cb5bde42d4Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |