Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/regexptree-exec-engine
mediumCurrent calibratedChosen benchmark version- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
regexp-tree parses, traverses, transforms, optimizes and generates regular expressions, and its fa module builds NFA/DFA tables for a toy sub-language (no flags, no captures). What it has never had is a matcher of its own: regexpTree.exec hands the string to the host RegExp. Build one. Add src/engine/ exposing compile(pattern, flags?), re-exported as regexpTree.engine and as regexpTree.compile(...). pattern may be a…
Excerpt only; not a complete executable task specification. Task ID task_2f8a69fe4cfd42a0 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 4/5 · 80.0% [37.6%, 96.4%] | 0 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 0/0 | 0 / 0 | 5 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Difficulty, with uncertainty
Selected-best difficulty: 20.0% [3.6%, 62.4%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
45 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_03e2e1af453f46b9Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_133d5775751e478dClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_70efb94a27254074Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d5a742cf7b5947f8Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ed8947fbb93b404cClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2c3d56df04554a19GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8823bfd02a854996GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_96bd6254d71d4b91GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_df6ddb34c1b04d9aGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_eb91b171c1e6408aGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_592452877186491aClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7797965c15694d07Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_78e2078170e74273Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_918e2bb671ff4a91Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_bd67e91527b14be5Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_250d3c2a7fa94e16GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_65bd28ee8bcb40cdGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8a060bdf24934485GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a376b301d13c4a92GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f6b25daa96484f0bGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2c9f6d5c97294da0Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b2b55557adab4679Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d694ab4c193f4e47Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d8967f656f234627Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_dbf888632d6142f3Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_094406ba8cf443d0GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5dde0007dd4742cfGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6a8d2e96977f4937GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9caceb36e17641abGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b4f08987aac343d3GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_1e35b1ad3d3b45a5Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_396ca968090740a2Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_cd8eb3fcb5ad43feClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d16583609b94419dClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f9d4a7300d55496dClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6ae5e99e0a4e4709Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 1 · valid | No diagnostic label derived | Tokens: 227,689 Turns: 145 Cost: $21.7410 | Not publicly available |
trial_7d8ca76b16484585Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 1 · valid | No diagnostic label derived | Tokens: 313,659 Turns: 219 Cost: $42.7103 | Not publicly available |
trial_8598ab33ae6042e9Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 1 · valid | No diagnostic label derived | Tokens: 281,092 Turns: 169 Cost: $29.5195 | Not publicly available |
trial_cca1f06ac9844aa3Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 1 · valid | No diagnostic label derived | Tokens: 365,519 Turns: 200 Cost: $44.2328 | Not publicly available |
trial_f3477bc2ecb44b50Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 284,205 Turns: 142 Cost: $26.7241 | Not publicly available |
trial_401c44e9f8b0467eGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_56143220a7634433GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b3c7f0b974744b5fGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b9b63f882c094009GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_cf63befff54f484dGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |