Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/iconv-lite-whatwg-mode
mediumCurrent calibratedChosen benchmark version- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
iconv-lite predates the WHATWG Encoding Standard, and its default tables and label handling deliberately keep their legacy lineage. Add a conformance mode that implements the Encoding Standard ([source link]) exactly, as a new whatwg namespace on the library's main export, leaving every existing behavior byte-identical. Pin the surface to: iconv.whatwg.getEncoding(label) returning the standard's …
Excerpt only; not a complete executable task specification. Task ID task_be13c5ae46354493 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 4/5 · 80.0% [37.6%, 96.4%] | 5 / 5 | 0 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 4 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 4/5 · 80.0% [37.6%, 96.4%] | 5 / 5 | 0 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 4 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 4/5 · 80.0% [37.6%, 96.4%] | 5 / 5 | 0 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 4 BAD_SUCCESS: 0 |
| Gemini Flash (latest alias) · default (unverified) | 1/3 · 33.3% [6.1%, 79.2%] · preliminary | 3 / 3 | 1 | GOOD_FAILURE: 2 BAD_FAILURE: 0 GOOD_SUCCESS: 1 BAD_SUCCESS: 0 |
Difficulty, with uncertainty
Selected-best difficulty: 20.0% [3.6%, 62.4%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
35 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_254992f3fd2849e6Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 132,647 Turns: 115 Cost: $14.6801 | Not publicly available |
trial_632f5f7db2654873Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 142,743 Turns: 115 Cost: $15.4455 | Not publicly available |
trial_8793e153636e44f6Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 135,102 Turns: 89 Cost: $13.4989 | Not publicly available |
trial_b107ebb2bda84b92Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 124,217 Turns: 100 Cost: $12.3350 | Not publicly available |
trial_f0c1c72f9dea49e4Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 145,955 Turns: 86 Cost: $12.8505 | Not publicly available |
trial_523d396e70e549b5Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7939e87f28614e69Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_91adb4a206144106Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b0e88e5986f14339Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_cc14b28934504c1eClaude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2df2d80d77f7487cGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6767b31039e04945Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_83e8c4e4b5c348a3Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a03e553ae0d14fcfGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a7e07d5fcd964ab1Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1828903399364077Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 150,740 Turns: 137 Cost: $19.3711 | Not publicly available |
trial_48be1a6530d54d40Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 165,527 Turns: 123 Cost: $15.8933 | Not publicly available |
trial_81fdf7f83ec84df3Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 162,027 Turns: 146 Cost: $19.9433 | Not publicly available |
trial_a8bcf414e2024fc8Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 155,670 Turns: 94 Cost: $13.9044 | Not publicly available |
trial_e490cba73cc04928Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 183,734 Turns: 139 Cost: $21.0641 | Not publicly available |
trial_04529e50ba9d4da9Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5e2e3cde183a402eGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c503095ac70441feGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ce5910a1285d46b6Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_fe3d7a078f664e19Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_023db348ab09480dGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_67fb41d1a76d4aecGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8b89cb52c8c342d1GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_95d8bf4902814812GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_bc4a97b098124490GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c2d052e394016ebaGemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e3765f8dab65b093Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a5b9a9c76f0251dfGemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a72841110169f275Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ad06e85a6da3d525Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedharness_error | HARNESS_ERROR frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |