Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/cheerio-aria-engine
high-fail (unreviewed)Current calibratedChosen benchmark version- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
Cheerio knows nothing about accessibility. Build that subsystem per WAI-ARIA 1.2, AccName 1.2, HTML-AAM 1.0 and the HTML focus rules, with six entry points. - $(sel).role(): string | null — computed role of the first match, null when none. role is a case-insensitive token list: the first concrete ARIA role token whose required context (nearest non-generic ancestor role) is satisfied wins; unknown/abstract/out-of-con…
Excerpt only; not a complete executable task specification. Task ID task_42603212f8434552 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 0/7 · 0.0% [0.0%, 35.4%] | 0 / 7 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 0/5 · 0.0% [0.0%, 43.4%] | 0 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Difficulty, with uncertainty
Selected-best difficulty: 100.0% [64.6%, 100.0%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
45 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_49b331edb04847b3Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_74d0020607624ea7Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d369261488ee4f55Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d38a510c1e0a49c6Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_eb3ed38bc1e84268Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_449febc02c964955GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_56cabc90f4e54178GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_665113ccd3d646b5GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_860b2e2b2adf4a2bGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8e9b91b0cd55442fGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_0ed2af576a624b3dClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1df6c657d1054788Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2c852bc261864330Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b7d718c0ca004221Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d32d53aa18d54417Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1326ef92f13f4d25GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7f0ad8fb3f1a4818GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8dccf51361ac47d9GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c0a5f9b89d1f48a1GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ebe9a38266ec41c1GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3a6aadb7c535423cClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3b9fd673ca45422cClaude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 172,633 Turns: 170 Cost: $21.7995 | Not publicly available |
trial_473bf6a65f7d49e8Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 162,316 Turns: 133 Cost: $17.4072 | Not publicly available |
trial_859c30923dc0403fClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8ec9a32d763243acClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_0eb22366bcc0469bGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_17d551ab7acb44bfGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_35001b6131f14b75GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a4413a5ff6e64ed7GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ee3790cd762e4171GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_16e0e293e33e460bClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5f0f6747ea6e4d02Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7d917002eac14460Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8bad61ccbacb4c32Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b95145763e474d6bClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_02e017ad580a4f0bClaude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 197,338 Turns: 175 Cost: $25.3816 | Not publicly available |
trial_2430e9ac702e40d1Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 161,794 Turns: 142 Cost: $18.6594 | Not publicly available |
trial_e2d9253c06c14556Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 184,684 Turns: 155 Cost: $22.6335 | Not publicly available |
trial_f59cede9e4014da4Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 170,159 Turns: 152 Cost: $21.1542 | Not publicly available |
trial_fadc53201eeb478aClaude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 172,783 Turns: 148 Cost: $19.5143 | Not publicly available |
trial_2554a3784b0a45f7GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5268d20cf8764c72GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5bcaabb430d447bbGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_aa552d8b78ad4be3GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d373e1e326a14f7eGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |