tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it./workspace/data/public_pairs.json
1 {
2 "schema_version": "1.0",
3 "description": "Unlabeled public format examples for the DeepCAD canonical-equivalence task.",
4 "pairs": [
5 {
6 "pair_id": "P001",
7 "program_a": [
8 {"cmd": "param", "name": "w", "expr": "40"},
9 {"cmd": "param", "name": "h", "expr": "20"},
10 {"cmd": "param", "name": "d", "expr": "8"},
11 {"cmd": "sketch", "id": "s0", "plane": "XY"},
12 {"cmd": "rect", "id": "base", "sketch": "s0", "center": [0, 0], "size": ["w", "h"]},
13 {"cmd": "line", "id": "guide", "sketch": "s0", "p1": [0, -10], "p2": [0, 10], "construction": true},
14 {"cmd": "extrude", "id": "pad", "profile": "base", "operation": "new", "depth": "d", "extent": "one_side"}
15 ],
16 "program_b": [
17 {"cmd": "param", "name": "depth", "expr": "4 + 4"},
18 {"cmd": "param", "name": "height", "expr": "5 * 4"},
19 {"cmd": "param", "name": "width", "expr": "20 * 2"},
20 {"cmd": "sketch", "id": "profile_sketch", "plane": "XY"},
21 {"cmd": "rect", "id": "outer", "sketch": "profile_sketch", "center": [0, 0], "size": ["width", "height"]},
22 {"cmd": "extrude", "id": "solid", "profile": "outer", "operation": "new", "depth": "depth", "extent": "one_side"}
23 ]
24 },
25 {
26 "pair_id": "P002",
27 "program_a": [
28 {"cmd": "param", "name": "r", "expr": "4"},
29 {"cmd": "sketch", "id": "s", "plane": "XY"},
30 {"cmd": "circle", "id": "hole", "sketch": "s", "center": [8, 0], "radius": "r"},
31 {"cmd": "extrude", "id": "cut", "profile": "hole", "operation": "cut", "depth": 10, "extent": "one_side"}
32 ],
33 "program_b": [
34 {"cmd": "param", "name": "r2", "expr": "4.25"},
35 {"cmd": "sketch", "id": "renamed", "plane": "XY"},
36 {"cmd": "circle", "id": "hole2", "sketch": "renamed", "center": [8, 0], "radius": "r2"},
37 {"cmd": "extrude", "id": "cut2", "profile": "hole2", "operation": "cut", "depth": 10, "extent": "one_side"}
38 ]
39 },
40 {
41 "pair_id": "P003",
42 "program_a": [
43 {"cmd": "sketch", "id": "s", "plane": "XY"},
44 {"cmd": "rect", "id": "rib_a", "sketch": "s", "center": [-6, 0], "size": [2, 12]},
45 {"cmd": "rect", "id": "rib_b", "sketch": "s", "center": [6, 0], "size": [2, 12]},
46 {"cmd": "extrude", "id": "ribs", "profiles": ["rib_a", "rib_b"], "operation": "add", "depth": 5, "extent": "one_side"}
47 ],
48 "program_b": [
49 {"cmd": "sketch", "id": "renamed", "plane": "XY"},
50 {"cmd": "rect", "id": "right", "sketch": "renamed", "center": [6, 0], "size": [2, 12]},
51 {"cmd": "rect", "id": "left", "sketch": "renamed", "center": [-6, 0], "size": [2, 12]},
52 {"cmd": "extrude", "id": "ribs2", "profiles": ["right", "left"], "operation": "add", "depth": 5, "extent": "one_side"}
53 ]
54 }
55 ]
56 }
57
/workspace
EISDIR: illegal operation on a directory, read '/workspace' [error] tool reported failure
List workspace contents
ls -la /workspace && ls -la /workspace/data
total 24 drwxr-xr-x 1 root root 4096 Jul 6 02:01 . drwxr-xr-x 1 root root 4096 Jul 6 02:12 .. -rw-r--r-- 1 root root 299 Jul 4 05:02 README.md drwxr-xr-x 2 root root 4096 Jul 4 05:02 data drwxr-xr-x 2 root root 4096 Jul 6 02:01 output -rwxr-xr-x 1 root root 1011 Jul 4 05:02 solve.py total 12 drwxr-xr-x 2 root root 4096 Jul 4 05:02 . drwxr-xr-x 1 root root 4096 Jul 6 02:01 .. -rw-r--r-- 1 root root 2824 Jul 4 05:02 public_pairs.json [stdout] total 24 drwxr-xr-x 1 root root 4096 Jul 6 02:01 . drwxr-xr-x 1 root root 4096 Jul 6 02:12 .. -rw-r--r-- 1 root root 299 Jul 4 05:02 README.md drwxr-xr-x 2 root root 4096 Jul 4 05:02 data drwxr-xr-x 2 root root 4096 Jul 6 02:01 output -rwxr-xr-x 1 root root 1011 Jul 4 05:02 solve.py total 12 drwxr-xr-x 2 root root 4096 Jul 4 05:02 . drwxr-xr-x 1 root root 4096 Jul 6 02:01 .. -rw-r--r-- 1 root root 2824 Jul 4 05:02 public_pairs.json
/workspace/solve.py
1 #!/usr/bin/env python3
2 """Weak starter baseline for DeepCAD canonical-equivalence.
3
4 This intentionally predicts every pair as non-equivalent. It is useful only as
5 a schema example and should score reward 0.0 on the hidden verifier.
6 """
7
8 from __future__ import annotations
9
10 import json
11 import sys
12 from pathlib import Path
13
14
15 def main() -> int:
16 input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/workspace/data/public_pairs.json")
17 output_path = Path(sys.argv[2]) if len(sys.argv) > 2 else Path("/workspace/predictions.json")
18
19 payload = json.loads(input_path.read_text(encoding="utf-8"))
20 predictions = [
21 {"pair_id": pair["pair_id"], "equivalent": False}
22 for pair in payload.get("pairs", [])
23 ]
24
25 output_path.parent.mkdir(parents=True, exist_ok=True)
26 output_path.write_text(
27 json.dumps({"predictions": predictions}, indent=2, sort_keys=True) + "\n",
28 encoding="utf-8",
29 )
30 return 0
31
32
33 if __name__ == "__main__":
34 raise SystemExit(main())
35
#!/usr/bin/env python3
"""DeepCAD canonical-equivalence solver.
Builds a canonical signature for each CAD command program and compares the two
programs in a pair. Two programs are equivalent iff their canonical signatures
match.
Semantics implemented (per task spec):
* param expressions are evaluated to numeric values (tolerance 1e-6 via
rounding); parameter names are not compared, only values.
* sketch planes are case-insensitive literal tokens; axes are not reordered.
* rect/circle/slot profiles preserve their geometric fields + plane.
Rectangle width/height are ordered. Slot angle is taken modulo 180.
Unused profiles do not affect the solid.
* extrude features preserve operation/extent/direction/depth (defaults
new/one_side/normal), profile geometry, and body/channel partition topology.
Depth sign and direction are kept separate. Profile list order is not
semantic. Extrude command order is not semantic (features form a multiset).
* body/channel labels only induce partition topology; label spelling is not
preserved and body vs channel is not a semantic namespace. Features with no
body/channel share one unlabeled default partition; an explicit label does
not alias it.
* construction:true commands and annotation kinds (constraint, dimension,
note, metadata, view, comment) do not affect the solid.
* any other non-construction command is an opaque solid-affecting payload,
compared as an unordered multiset after stripping only its top-level id.
Opaque fields are compared literally (no rename/eval/tolerance/sort).
"""
from __future__ import annotations
import json
import sys
from pathlib import Path
TOL_DECIMALS = 6
PROFILE_KINDS = {"rect", "circle", "slot"}
ANNOTATION_KINDS = {"constraint", "dimension", "note", "metadata", "view", "comment"}
SUPPORTED_KINDS = {"param", "sketch", "extrude"} | PROFILE_KINDS
PROFILE_META_FIELDS = {"cmd", "id", "sketch", "construction"}
DEFAULT_PARTITION = "\x00default"
# --------------------------------------------------------------------------- #
# Expression evaluation
# --------------------------------------------------------------------------- #
def _tokenize(expr):
tokens = []
i = 0
n = len(expr)
while i < n:
c = expr[i]
if c.isspace():
i += 1
continue
if c in "+-*/()":
tokens.append(c)
i += 1
continue
if c.isdigit() or c == ".":
j = i
while j < n and (expr[j].isdigit() or expr[j] == "."):
j += 1
tokens.append(("num", float(expr[i:j])))
i = j
continue
if c.isalpha() or c == "_":
j = i
while j < n and (expr[j].isalnum() or expr[j] == "_"):
j += 1
tokens.append(("id", expr[i:j]))
i = j
continue
raise ValueError(f"bad char {c!r} in expr {expr!r}")
return tokens
def _eval_expr(expr, get):
"""Evaluate an arithmetic expression string. `get(name)` resolves params."""
tokens = _tokenize(expr)
pos = 0
def peek():
return tokens[pos] if pos < len(tokens) else None
def advance():
nonlocal pos
t = tokens[pos]
pos += 1
return t
def parse_expr():
v = parse_term()
while peek() in ("+", "-"):
op = advance()
rhs = parse_term()
v = v + rhs if op == "+" else v - rhs
return v
def parse_term():
v = parse_factor()
while peek() in ("*", "/"):
op = advance()
rhs = parse_factor()
v = v * rhs if op == "*" else v / rhs
return v
def parse_factor():
t = peek()
if t == "+":
advance()
return parse_factor()
if t == "-":
advance()
return -parse_factor()
if t == "(":
advance()
v = parse_expr()
if peek() != ")":
raise ValueError("missing )")
advance()
return v
if isinstance(t, tuple):
advance()
kind, val = t
if kind == "num":
return val
return get(val)
raise ValueError(f"unexpected token {t!r}")
v = parse_expr()
if pos != len(tokens):
raise ValueError("trailing tokens")
return v
def _make_env(param_exprs):
resolved = {}
resolving = set()
def get(name):
if name in resolved:
return resolved[name]
if name in resolving:
raise ValueError(f"cyclic param {name}")
if name not in param_exprs:
raise ValueError(f"unknown param {name}")
resolving.add(name)
val = _eval_value(param_exprs[name], get)
resolving.discard(name)
resolved[name] = val
return val
return get
def _eval_value(value, get):
"""Evaluate a scalar/list value to a numeric (rounded) or literal form."""
if isinstance(value, bool):
return value
if isinstance(value, (int, float)):
return round(float(value), TOL_DECIMALS)
if isinstance(value, str):
try:
return round(_eval_expr(value, get), TOL_DECIMALS)
except Exception:
return value
if isinstance(value, list):
return [_eval_value(v, get) for v in value]
if isinstance(value, dict):
return {k: _eval_value(v, get) for k, v in value.items()}
return value
# --------------------------------------------------------------------------- #
# Canonicalization
# --------------------------------------------------------------------------- #
def _jkey(obj):
return json.dumps(obj, sort_keys=True, separators=(",", ":"))
def _is_construction(cmd):
return bool(cmd.get("construction", False))
def _profile_canon(cmd, sketches, get):
kind = cmd.get("cmd")
sketch_id = cmd.get("sketch")
plane = ""
if sketch_id is not None and sketch_id in sketches:
plane = str(sketches[sketch_id]).upper()
fields = {}
for k, v in cmd.items():
if k in PROFILE_META_FIELDS:
continue
val = _eval_value(v, get)
if kind == "slot" and k == "angle" and isinstance(val, (int, float)) and not isinstance(val, bool):
val = round(((float(val) % 180.0) + 180.0) % 180.0, TOL_DECIMALS)
fields[k] = val
return {"kind": kind, "plane": plane, "fields": fields}
def _extrude_feature(cmd, profile_map, get):
op = str(cmd.get("operation", "new")).lower()
extent = str(cmd.get("extent", "one_side")).lower()
direction = str(cmd.get("direction", "normal")).lower()
depth = _eval_value(cmd.get("depth", 0), get)
prof_ids = []
if "profiles" in cmd and isinstance(cmd["profiles"], list):
prof_ids = list(cmd["profiles"])
elif "profile" in cmd:
p = cmd["profile"]
prof_ids = list(p) if isinstance(p, list) else [p]
prof_canons = [profile_map[pid] for pid in prof_ids if pid in profile_map]
prof_canons.sort(key=_jkey)
feature = {
"op": op,
"extent": extent,
"dir": direction,
"depth": depth,
"profiles": prof_canons,
}
if "body" in cmd:
label = ("label", _jkey(cmd["body"]))
elif "channel" in cmd:
label = ("label", _jkey(cmd["channel"]))
else:
label = DEFAULT_PARTITION
return feature, label
def _program_signature(program):
# Pass 1: collect params, sketches (ignore construction commands).
param_exprs = {}
sketches = {}
active = [] # non-construction commands
for cmd in program:
if not isinstance(cmd, dict):
continue
if _is_construction(cmd):
continue
active.append(cmd)
kind = cmd.get("cmd")
if kind == "param":
name = cmd.get("name")
if name is not None:
param_exprs[name] = cmd.get("expr", 0)
elif kind == "sketch":
sid = cmd.get("id")
if sid is not None:
sketches[sid] = cmd.get("plane", "")
get = _make_env(param_exprs)
# Pass 2: profile canons.
profile_map = {}
for cmd in active:
if cmd.get("cmd") in PROFILE_KINDS:
pid = cmd.get("id")
if pid is not None:
profile_map[pid] = _profile_canon(cmd, sketches, get)
# Pass 3: extrudes and opaque commands.
groups = {}
opaque = []
for cmd in active:
kind = cmd.get("cmd")
if kind == "extrude":
feature, label = _extrude_feature(cmd, profile_map, get)
groups.setdefault(label, []).append(feature)
elif kind in SUPPORTED_KINDS or kind in ANNOTATION_KINDS:
# param / sketch / profiles already handled or ignored; annotations
# do not affect the solid.
continue
else:
payload = {k: v for k, v in cmd.items() if k != "id"}
opaque.append(payload)
partition = []
for feats in groups.values():
feats_sorted = sorted(feats, key=_jkey)
partition.append(feats_sorted)
partition.sort(key=_jkey)
opaque.sort(key=_jkey)
return {"partition": partition, "opaque": opaque}
# --------------------------------------------------------------------------- #
# Driver
# --------------------------------------------------------------------------- #
def main() -> int:
input_path = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/workspace/data/public_pairs.json")
output_path = Path(sys.argv[2]) if len(sys.argv) > 2 else Path("/workspace/predictions.json")
payload = json.loads(input_path.read_text(encoding="utf-8"))
predictions = []
for pair in payload.get("pairs", []):
pid = pair.get("pair_id")
try:
sig_a = _program_signature(pair.get("program_a", []))
sig_b = _program_signature(pair.get("program_b", []))
key_a = _jkey(sig_a)
key_b = _jkey(sig_b)
equivalent = key_a == key_b
except Exception:
key_a = key_b = ""
equivalent = False
predictions.append({
"pair_id": pid,
"equivalent": bool(equivalent),
"signature_a": key_a,
"signature_b": key_b,
})
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(
json.dumps({"predictions": predictions}, indent=2) + "\n",
encoding="utf-8",
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
Run solver on public pairs
python3 solve.py data/public_pairs.json /workspace/predictions.json && python3 -c "
import json
d=json.load(open('/workspace/predictions.json'))
for p in d['predictions']:
print(p['pair_id'], p['equivalent'])
"P001 True P002 False P003 True [stdout] P001 True P002 False P003 True
Test edge cases
python3 -c "
import json, solve
def eq(a,b):
return solve._jkey(solve._program_signature(a))==solve._jkey(solve._program_signature(b))
S=lambda pl='XY':{'cmd':'sketch','id':'s','plane':pl}
R=lambda:{'cmd':'rect','id':'r','sketch':'s','center':[0,0],'size':[4,2]}
def ext(**kw):
d={'cmd':'extrude','profile':'r'}; d.update(kw); return d
# body vs channel same grouping -> equivalent
a=[S(),R(),ext(depth=5,body='X'),{'cmd':'rect','id':'r2','sketch':'s','center':[1,1],'size':[3,3]},ext(profile='r2',depth=5,body='X')]
b=[S(),R(),ext(depth=5,channel='Y'),{'cmd':'rect','id':'r2','sketch':'s','center':[1,1],'size':[3,3]},ext(profile='r2',depth=5,channel='Y')]
print('body==channel grouping:', eq(a,b)) # expect True
# single label vs default single group
a=[S(),R(),ext(depth=5,body='X')]
b=[S(),R(),ext(depth=5)]
print('label vs default single:', eq(a,b)) # expect True (one group each)
# mixed: two features one labeled one default vs both default
a=[S(),R(),ext(depth=5,body='X'),ext(depth=7)]
b=[S(),R(),ext(depth=5),ext(depth=7)]
print('mixed labeled distinct:', eq(a,b)) # expect False
# two separate labels vs one shared label (2 groups vs 1 group)
a=[S(),R(),ext(depth=5,body='X'),ext(depth=7,body='Y')]
b=[S(),R(),ext(depth=5,body='X'),ext(depth=7,body='X')]
print('2groups vs 1group:', eq(a,b)) # expect False
# depth sign preserved
print('depth sign:', eq([S(),R(),ext(depth=5)],[S(),R(),ext(depth=-5)])) # False
# direction preserved
print('direction:', eq([S(),R(),ext(depth=5)],[S(),R(),ext(depth=5,direction='reverse')])) # False
# extent literal
print('extent alias not folded:', eq([S(),R(),ext(depth=5,extent='blind')],[S(),R(),ext(depth=5,extent='one_side')])) # False
# slot angle mod 180
sl=lambda ang:[{'cmd':'sketch','id':'s','plane':'XY'},{'cmd':'slot','id':'r','sketch':'s','center':[0,0],'length':10,'width':2,'angle':ang},ext(depth=5)]
print('slot angle mod180:', eq(sl(30),sl(210))) # True
print('slot angle diff:', eq(sl(30),sl(40))) # False
# XZ vs ZX different
print('XZ vs ZX:', eq([S('XZ'),R(),ext(depth=5)],[S('ZX'),R(),ext(depth=5)])) # False
print('xy vs XY case:', eq([S('xy'),R(),ext(depth=5)],[S('XY'),R(),ext(depth=5)])) # True
# opaque multiset compare, strip id
op=lambda i,t:{'cmd':'fillet','id':i,'target':t,'radius':1}
print('opaque id-stripped:', eq([S(),R(),ext(depth=5),op('f1','e1')],[S(),R(),ext(depth=5),op('f2','e1')])) # True
print('opaque diff payload:', eq([S(),R(),ext(depth=5),op('f1','e1')],[S(),R(),ext(depth=5),op('f2','e2')])) # False
print('opaque no tolerance:', eq([S(),R(),ext(depth=5),{'cmd':'x','v':1.0}],[S(),R(),ext(depth=5),{'cmd':'x','v':1.0000001}])) # False literal
# annotation ignored
print('annotation ignored:', eq([S(),R(),ext(depth=5),{'cmd':'note','text':'hi'}],[S(),R(),ext(depth=5)])) # True
# construction ignored
print('construction ignored:', eq([S(),R(),ext(depth=5),{'cmd':'fillet','construction':True,'r':1}],[S(),R(),ext(depth=5)])) # True
# unused profile no effect
print('unused profile:', eq([S(),R(),{'cmd':'circle','id':'c','sketch':'s','center':[9,9],'radius':3},ext(depth=5)],[S(),R(),ext(depth=5)])) # True
# param value equivalence across expr
print('param expr eq:', eq([{'cmd':'param','name':'a','expr':'2*3'},S(),{'cmd':'rect','id':'r','sketch':'s','center':[0,0],'size':['a',2]},ext(depth=5)],[S(),{'cmd':'rect','id':'r','sketch':'s','center':[0,0],'size':[6,2]},ext(depth=5)])) # True
# rect ordered dims
rr=lambda w,h:[S(),{'cmd':'rect','id':'r','sketch':'s','center':[0,0],'size':[w,h]},ext(depth=5)]
print('rect ordered:', eq(rr(4,2),rr(2,4))) # False
"
body==channel grouping: True label vs default single: True mixed labeled distinct: False 2groups vs 1group: False depth sign: False direction: False extent alias not folded: False slot angle mod180: True slot angle diff: False XZ vs ZX: False xy vs XY case: True opaque id-stripped: True opaque diff payload: False opaque no tolerance: False annotation ignored: True construction ignored: True unused profile: True param expr eq: True rect ordered: False [stdout] body==channel grouping: True label vs default single: True mixed labeled distinct: False 2groups vs 1group: False depth sign: False direction: False extent alias not folded: False slot angle mod180: True slot angle diff: False XZ vs ZX: False xy vs XY case: True opaque id-stripped: True opaque diff payload: False opaque no tolerance: False annotation ignored: True construction ignored: True unused profile: True param expr eq: True rect ordered: False
Validate predictions schema
python3 -c "
import json
d=json.load(open('/workspace/predictions.json'))
assert set(d)=={'predictions'}
for p in d['predictions']:
assert isinstance(p['equivalent'], bool)
assert 'pair_id' in p
print('valid; rows:', len(d['predictions']))
print(json.dumps([{k:p[k] for k in ('pair_id','equivalent')} for p in d['predictions']]))
"valid; rows: 3
[{"pair_id": "P001", "equivalent": true}, {"pair_id": "P002", "equivalent": false}, {"pair_id": "P003", "equivalent": true}]
[stdout]
valid; rows: 3
[{"pair_id": "P001", "equivalent": true}, {"pair_id": "P002", "equivalent": false}, {"pair_id": "P003", "equivalent": true}]No file changes, the agent left the workspace untouched.
{
"metrics": {
"canonical_regime_min_accuracy": 0.934426,
"diagnostic_accuracy": {
"alias": 1.0,
"commutation": 1.0,
"enum_literal": 1.0,
"explicit_defaults": 1.0,
"feature_multiset_multiplicity": 1.0,
"hard_decoy": 0.934426,
"numeric_tolerance": 1.0,
"ood": 0.952941,
"opaque_id_alias": 1.0,
"opaque_list_literal": 1.0,
"opaque_nested_literal": 1.0,
"opaque_numeric_literal": 1.0,
"opaque_reference_literal": 1.0,
"ordered_rect_dimensions": 1.0,
"partition_alias_true": 1.0,
"partition_default": 1.0,
"partition_topology": 0.0,
"plane_literal": 1.0,
"profile_order": 1.0,
"redundant": 1.0,
"signed_depth_direction": 1.0,
"slot_angle_modulo": 1.0,
"unconsumed_profile": 1.0
},
"diagnostic_macro_accuracy": 0.951625,
"equivalent_recall": 1.0,
"overall_accuracy": 0.958763,
"regime_accuracy": {
"alias": 1.0,
"commutation": 1.0,
"hard_decoy": 0.934426,
"ood": 0.952941,
"redundant": 1.0
}
},
"passed": false,
"reward": 0.441163,
"reward_policy": {
"anchors": {
"diagnostic_macro_accuracy": {
"baseline": 0.45,
"strong_target": 1.0,
"target": 0.985,
"threshold": 0.8
},
"equivalent_recall": {
"baseline": 0.3,
"strong_target": 1.0,
"target": 0.99,
"threshold": 0.8
},
"overall_accuracy": {
"baseline": 0.65,
"strong_target": 1.0,
"target": 0.99,
"threshold": 0.88
}
},
"meets_targets": false,
"metric_scores": {
"diagnostic_macro_accuracy": 0.474231,
"equivalent_recall": 1.0,
"overall_accuracy": 0.441163
},
"required_metrics": [
"overall_accuracy",
"equivalent_recall",
"diagnostic_macro_accuracy"
],
"target_reward": 0.5,
"threshold_reward": 0.2,
"type": "v3_50pass"
}
}Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_eb2d83f35ae24617. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.
Trial trial_eb2d83f35ae24617 · verifier authoritative; classifier explanatory.