Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/pydub-dsp-engine
high-fail (unreviewed)Current calibratedChosen benchmark version- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
Build pydub's missing DSP engine from scratch as a new self-contained package pydub/dsp/, re-exported (same objects) from a new pydub/loudness.py: ITU-R BS.1770-4 loudness, EBU R 128 normalisation, Tech 3341/3342 metering, true peak, resampling, dither. Pure stdlib — no numpy/scipy/audioop/ffmpeg in new code; existing behaviour unchanged. Samples: decodepcm(data,width[,"float32"]) (float32 passthrough), encodepcm, d…
Excerpt only; not a complete executable task specification. Task ID task_fcb17ab842554cdb · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 0/5 · 0.0% [0.0%, 43.4%] | 0 / 5 | 0 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 0/0 | 0 / 0 | 5 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Difficulty, with uncertainty
Selected-best difficulty: 100.0% [56.6%, 100.0%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
45 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_94a56c15567a4fc6Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a1ad35adbd4e43c2Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d75ea9f2b5a64f07Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d7c25bd3bd774f26Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f0ef44cea5774a88Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3738475cebf747cdGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3b09f54a7dac466fGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_508fd822f4d04139GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b26b8c015e064cf0GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_d11aeb14972e4c0eGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_31692c17bad84dc0Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3d5b22f4f8234f0bClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_54ac76c13c5c4e19Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_77ef6681f434485aClaude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_ab08a61c08bd4d65Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3f8525fee1a449b2GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5d6ad7ce63bf4069GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_89dcef20cb8345dbGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9a6121f7643649f2GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_e2c2443185194b4dGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_05e60a4967504237Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_3354a2346fcc4931Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_769e32c77e1344f3Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a8980f5a61434389Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_bcfcd3d351f44088Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_042806cd123b44d6GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_4b580930b4c24bddGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_988dec975e954beeGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_9f79222dcb964723GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_dea75c29c02844e5GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_387d39c9341949e7Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5bc64c68b4f44e64Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_911a8939af8b4386Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_955e3c4736dd44a2Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_a5bb186a975c4d90Claude Opus 5 · default (unverified) | Sep 6, 2026 console | No reward · excludedno_reward | No diagnostic label No diagnostic source | Tokens: — Turns: — Cost: — | Not publicly available |
trial_01d76f3109904960Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 112,570 Turns: 70 Cost: $7.0541 | Not publicly available |
trial_2f4d9cf612a14a0eClaude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 117,514 Turns: 58 Cost: $6.7794 | Not publicly available |
trial_48470b78887147ccClaude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 97,484 Turns: 61 Cost: $5.9947 | Not publicly available |
trial_a5f082f51f27415aClaude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 159,092 Turns: 105 Cost: $14.0858 | Not publicly available |
trial_ebe1ec53fe144fc2Claude Opus 5 · default (unverified) | Sep 6, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 106,517 Turns: 50 Cost: $6.2432 | Not publicly available |
trial_33d4c3d7c0d6436cGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_6b877699e61b405cGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8450b47d5c3a4dcaGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c88a0a0b71354fa4GPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |
trial_fae1e708636144cbGPT-5.6-sol · default (unverified) | Sep 6, 2026 console | 0 · excludedharness_error | HARNESS_ERROR derived | Tokens: — Turns: — Cost: — | Not publicly available |