Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/echo-http-signatures
legacy-onlyChosen benchmark version- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
Echo v5 has no support for HTTP Message Signatures. Add a request-verification middleware for RFC 9421 to the middleware package, implemented with the Go standard library only (the build is offline), following house style: HTTPSignatureConfig implementing ToMiddleware() (echo.MiddlewareFunc, error), plus HTTPSignatureWithConfig(config HTTPSignatureConfig) echo.MiddlewareFunc (panicking on invalid configuration) and …
Excerpt only; not a complete executable task specification. Task ID task_3b8bbd5b547043af · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 0/0 | 0 / 0 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 0/5 · 0.0% [0.0%, 43.4%] | 5 / 5 | 0 | GOOD_FAILURE: 5 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| Gemini Flash (latest alias) · default (unverified) | 0/0 | 0 / 0 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Every recorded run
15 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_4d2d33503cd44908Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 84,985 Turns: 43 Cost: $5.4788 | Not publicly available |
trial_543f8aa42ac74d67Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 78,041 Turns: 37 Cost: $4.5854 | Not publicly available |
trial_be1ce95e20d14d40Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 93,957 Turns: 38 Cost: $5.1202 | Not publicly available |
trial_dcbaa5b6a8894723Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 68,377 Turns: 29 Cost: $3.6881 | Not publicly available |
trial_de62c0471d7e4adeClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 84,484 Turns: 29 Cost: $4.4287 | Not publicly available |
trial_39e4b5280d1c4a0fClaude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_669835e933e94d29Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_7cd398cfeb94482eClaude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_bc1721b822ff46b3Claude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_bf786353ce0c479dClaude Opus 5 · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_001126e47a8e476bGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_310af3d7c25941daGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5000795a7e3f4fffGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6cff8e22f76c4a41Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9532dd217ddc4fe7Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |