Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/ammonia-sanitizer-api
saturatedCurrent calibratedChosen benchmark version- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
ammonia sanitizes HTML through its own Builder, but has no implementation of the WICG HTML Sanitizer API: no SanitizerConfig model, no config canonicalization, validity checking or reflection, and no way to run one config in safe and unsafe modes. Implement the configuration semantics of the WICG HTML Sanitizer API (Draft Community Group Report as of 24 November 2025) as a new public module ammonia::sanitizer (new s…
Excerpt only; not a complete executable task specification. Task ID task_40df7b0ce1654f8f · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 5/5 · 100.0% [56.6%, 100.0%] | 5 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 5 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 5/5 · 100.0% [56.6%, 100.0%] | 5 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 5 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 3/5 · 60.0% [23.1%, 88.2%] | 5 / 5 | 0 | GOOD_FAILURE: 2 BAD_FAILURE: 0 GOOD_SUCCESS: 3 BAD_SUCCESS: 0 |
| Gemini Flash (latest alias) · default (unverified) | 0/1 | 1 / 1 | 1 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Difficulty, with uncertainty
Selected-best difficulty: 0.0% [0.0%, 43.4%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
38 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_0ba3981e05384dd8Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 84,139 Turns: 60 Cost: $7.0017 | Not publicly available |
trial_557a3b8dfae54ceaClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 93,309 Turns: 55 Cost: $6.2024 | Not publicly available |
trial_5c43c07be63741f8Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 80,147 Turns: 49 Cost: $6.1511 | Not publicly available |
trial_782c4d7bf4a64e88Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 88,437 Turns: 55 Cost: $8.0174 | Not publicly available |
trial_b099ac9337f043a3Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 113,236 Turns: 64 Cost: $9.3897 | Not publicly available |
trial_17bfef6a48e84dbaClaude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_516ff574bf7d48f2Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5230231e865e43a0Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8da0cb7214cf41f3Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e304df0b94744b63Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1979702b3bc24028Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5a20f55124b4434dGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_851f2570b0c846edGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8b147b92ee9c46bfGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_caec970fd6434e61Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_07c68946fee54c09Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 90,146 Turns: 55 Cost: $7.0275 | Not publicly available |
trial_3f6638a7614d4c45Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 102,027 Turns: 64 Cost: $7.8656 | Not publicly available |
trial_6e36a7045aaf4d83Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 91,130 Turns: 58 Cost: $7.2145 | Not publicly available |
trial_980e29e30c8d46bcClaude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 89,274 Turns: 68 Cost: $7.1615 | Not publicly available |
trial_f012e173e45c4517Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 119,653 Turns: 88 Cost: $11.0125 | Not publicly available |
trial_1f6cf212cd72457bGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_30d57439cba549a7Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6ea17673425c4e8dGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9c8bc79882ea405eGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · excludedsource_evidence_exclusion | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_bf966889386640fdGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_39e08317a074471aGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_4d52db0db09c4a0bGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7cc41413aacc4214GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c38d2cf840c54145GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f302895a833a42c2GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3a039b9162dd2fdaGemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d39477e7c73e3b18Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9f0ac6b7290671a0Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6d15f9e5846ccf6dGemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7646a6bca4cebf54Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f8945745cf2bd409Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_4e5d07477cc6ad1dGemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedharness_error | HARNESS_ERROR frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_dcaf6fe8528ef176Gemini Flash (latest alias) · default (unverified) | Sep 3, 2026 console | 0 · excludedsource_evidence_exclusion | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |