Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console
syncrope/oxigraph-shacl-core
mediumCurrent calibratedOutside aggregate scope- Reference solution (oracle)
- Unverified
- No-op resistance check
- Unverified
- QA verdict
- Not recorded
- Aggregate eligibility
- Excluded / other version
The task
Oxigraph can store and query RDF but cannot validate data against a shapes graph. Implement a W3C SHACL Core validation engine as a new workspace crate oxshacl at lib/oxshacl, registered in the root Cargo.toml members list. Depend only on oxigraph (path dependency, default-features = false — RocksDB and test-suite git submodules are not vendored) plus crates already in Cargo.lock (the offline registry carries nothin…
Excerpt only; not a complete executable task specification. Task ID task_bb2e200d1de14882 · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · default (unverified) | 1/5 · 20.0% [3.6%, 62.4%] | 5 / 5 | 0 | GOOD_FAILURE: 4 BAD_FAILURE: 0 GOOD_SUCCESS: 1 BAD_SUCCESS: 0 |
| GPT-5.6-sol · default (unverified) | 4/5 · 80.0% [37.6%, 96.4%] | 5 / 5 | 0 | GOOD_FAILURE: 1 BAD_FAILURE: 0 GOOD_SUCCESS: 4 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 4.8 · default (unverified) | 1/5 · 20.0% [3.6%, 62.4%] | 5 / 5 | 0 | GOOD_FAILURE: 4 BAD_FAILURE: 0 GOOD_SUCCESS: 1 BAD_SUCCESS: 0 |
| Gemini Flash (latest alias) · default (unverified) | 0/5 · 0.0% [0.0%, 43.4%] | 5 / 5 | 0 | GOOD_FAILURE: 4 BAD_FAILURE: 1 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Difficulty, with uncertainty
Selected-best difficulty: 20.0% [3.6%, 62.4%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.
The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.
Every recorded run
20 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_21cc6e25ed9b4244Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 117,874 Turns: 119 Cost: $12.6351 | Not publicly available |
trial_28f2e59281cb498fClaude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 128,010 Turns: 99 Cost: $13.8611 | Not publicly available |
trial_506e906106514a53Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 127,879 Turns: 110 Cost: $13.9972 | Not publicly available |
trial_929bea89e5ec4d19Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 141,300 Turns: 104 Cost: $13.1787 | Not publicly available |
trial_c7a645d8c7f94510Claude Opus 4.8 · default (unverified) | Aug 29, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 116,625 Turns: 104 Cost: $11.2977 | Not publicly available |
trial_337fb21c4b624e63Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 154,160 Turns: 83 Cost: $11.9681 | Not publicly available |
trial_5581937d2d884021Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 123,713 Turns: 64 Cost: $8.8388 | Not publicly available |
trial_7fbad76a338f4723Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 150,865 Turns: 73 Cost: $10.6047 | Not publicly available |
trial_b2431fb60cee4a58Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 162,179 Turns: 104 Cost: $16.0875 | Not publicly available |
trial_f6da9b2486c24040Claude Opus 5 · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: 159,856 Turns: 113 Cost: $17.0814 | Not publicly available |
trial_85c5ae5aa7af45cdGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_bab4f0d903b14e56Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_c8c4ef4342a64d18Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_cde3ab3fe97c46e6Gemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | BAD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_f88a0e48345f448cGemini Flash (latest alias) · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_2fefde5a815e4826GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_5da30c36111b441dGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_8cd5b78ecc04408bGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_b1e3cf1d3c5f4466GPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |
trial_fb5dba0c8ccb434eGPT-5.6-sol · default (unverified) | Sep 1, 2026 console | 0 · valid | GOOD_FAILURE frozen-classifier | Tokens: — Turns: — Cost: — | Not publicly available |