Task gallery / Mechanical engineering / Version 1
Task evidence / console
dxf-nested-assembly-schedule
uncalibratedChosen benchmark version- Reference solution (oracle)
- Pass
- No-op resistance check
- Pass
- QA verdict
- Not recorded
- Aggregate eligibility
- Eligible version
The task
Assembly purchasing export Purchasing needs a schedule from a nested DXF assembly. Resolve the effective layer and attributes of each leaf, compose its placements and report quantities and locations by the part identity. The supported drawing profile is in /app/repo/CONTRACT.md. It excludes clipping because a clipped reference cannot be represented by the full-part schedule. Keep recursive blocks and attributes in t…
Excerpt only; not a complete executable task specification. Task ID task_f48ea8010149425d · version 1. Selection status: not fully recorded.
Results by model and effort
This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.
Current evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|---|---|---|---|
| Claude Opus 5 · xhigh | 4/4 · 100.0% [51.0%, 100.0%] · preliminary | 4 / 4 | 1 | GOOD_FAILURE: 0 BAD_FAILURE: 0 GOOD_SUCCESS: 4 BAD_SUCCESS: 0 |
| Tencent HY4 (preview) · think | 0/4 · 0.0% [0.0%, 49.0%] · preliminary | 3 / 4 | 5 | GOOD_FAILURE: 3 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0 |
Legacy evidence
| Model / effort | Verifier result [95% Wilson] | Diagnostic labels / valid | HARNESS_ERROR · excluded | Outcome classes |
|---|
Every recorded run
23 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.
| Run / model | Date / source | Reward / eligibility | Diagnostic class / source | Resources | Evidence |
|---|---|---|---|---|---|
trial_1c99630f26fb412cClaude Opus 5 · xhigh | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 136,513 Turns: 52 Cost: $7.3896 | Not publicly available |
trial_31fd1447c6b14032Tencent HY4 (preview) · think | Oct 21, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 106,741 Turns: 80 Cost: $7.7940 | Not publicly available |
trial_6be28e65983b4834Claude Opus 5 · xhigh | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 131,688 Turns: 70 Cost: $8.4868 | Not publicly available |
trial_807c97b775b34c66Tencent HY4 (preview) · think | Oct 21, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 119,525 Turns: 75 Cost: $7.4002 | Not publicly available |
trial_59d6106f2ff14542Claude Opus 5 · xhigh | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS independent-review | Tokens: 122,815 Turns: 68 Cost: $8.2135 | Not publicly available |
trial_3f1fd9285e0f4ba0Tencent HY4 (preview) · think | Oct 21, 2026 console | 0 · excludedharness_error | HARNESS_ERROR independent-review | Tokens: 28,921 Turns: 5 Cost: $0.9517 | Not publicly available |
trial_9d60aa7c241c454cTencent HY4 (preview) · think | Oct 21, 2026 console | 0 · valid | GOOD_FAILURE independent-review | Tokens: 102,237 Turns: 96 Cost: $8.9110 | Not publicly available |
trial_153e078dbdaf4c48Claude Opus 5 · xhigh | Oct 21, 2026 console | 1 · valid | GOOD_SUCCESS frozen-classifier | Tokens: 116,735 Turns: 52 Cost: $6.9811 | Not publicly available |
trial_00b37b48d9424b2fTencent HY4 (preview) · think | Oct 21, 2026 console | 0 · valid | No diagnostic label derived | Tokens: 124,666 Turns: 98 Cost: $8.9349 | Not publicly available |
trial_06dc7ae6031f4e93Claude Opus 5 · xhigh | Oct 21, 2026 console | No reward · excludedno_reward | HARNESS_ERROR independent-review | Tokens: — Turns: 0 Cost: — | Not publicly available |
trial_369669ab888b4d26Tencent HY4 (preview) · think | Oct 21, 2026 console | No reward · excludedno_reward | HARNESS_ERROR independent-review | Tokens: — Turns: 0 Cost: — | Not publicly available |
trial_182505e4244040dfTencent HY4 (preview) · think | Oct 21, 2026 console | No reward · excludedno_reward | HARNESS_ERROR independent-review | Tokens: — Turns: 0 Cost: — | Not publicly available |
trial_6173da4aa3e448f6Tencent HY4 (preview) · think | Oct 21, 2026 console | No reward · excludedno_reward | HARNESS_ERROR independent-review | Tokens: — Turns: 0 Cost: — | Not publicly available |
trial_8007fa1d65514b92Tencent HY4 (preview) · think | Oct 21, 2026 console | No reward · excludedno_reward | HARNESS_ERROR independent-review | Tokens: — Turns: 0 Cost: — | Not publicly available |
dxf-nested-assembly-schedule-oracle-1-219b1d75oracle@@ | Oct 21, 2026 delivery | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-nested-assembly-schedule-nop-2279c36enop@@ | Oct 21, 2026 delivery | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-nested-assembly-schedule-oracle-2-20abb6eforacle@@ | Oct 21, 2026 delivery | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-v3-opus5-oracle-1oracle@@ | Oct 21, 2026 delivery | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-v3-opus5-nop-1nop@@ | Oct 21, 2026 delivery | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-v3-opus5-oracle-2oracle@@ | Oct 21, 2026 delivery | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-nested-assembly-schedule-oracle-1-7ab95483oracle@@ | Oct 21, 2026 delivery | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-nested-assembly-schedule-nop-80159195nop@@ | Oct 21, 2026 delivery | 0 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |
dxf-nested-assembly-schedule-oracle-2-329f68baoracle@@ | Oct 21, 2026 delivery | 1 · valid | No diagnostic label derived | Tokens: — Turns: — Cost: — | Not publicly available |