tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it.Explore app directory structure
ls -la && echo "---SUBMISSION---" && ls -la submission/ && echo "---DATA---" && ls -la data/ && echo "---NUMPY---" && python3 -c "import numpy; print(numpy.__version__)"
total 117 drwxr-xr-x 4 user user 4096 Jun 30 22:11 . drwxr-xr-x 21 root root 4096 Jun 30 22:11 .. -rw-r--r-- 1 user user 15364 Jun 30 22:11 PROMPT.txt -rw-r--r-- 1 user user 5 Jun 30 22:11 agent.pid -rw-r--r-- 1 user user 0 Jun 30 22:11 agent.stderr -rw-r--r-- 1 user user 4634 Jun 30 22:11 agent_trajectory.jsonl drwxr-xr-x 2 user user 60 Jun 30 22:11 data -rw-r--r-- 1 user user 0 Jun 30 22:11 run.log -rwxr-xr-x 1 user user 76773 Jun 30 10:02 run_checks.py drwxr-xr-x 2 user user 128 Jun 30 22:11 submission -rwxr-xr-x 1 user user 804 Jun 30 10:02 test.sh -rwxr-xr-x 1 user user 629 Jun 30 10:02 timer.sh ---SUBMISSION--- total 112 drwxr-xr-x 2 user user 128 Jun 30 22:11 . drwxr-xr-x 4 user user 4096 Jun 30 22:11 .. -rw-r--r-- 1 user user 50472 Jun 30 10:02 autograd.py -rw-r--r-- 1 user user 1661 Jun 30 10:02 data.py -rw-r--r-- 1 user user 33295 Jun 30 10:02 nn.py -rw-r--r-- 1 user user 11509 Jun 30 10:02 optim.py -rw-r--r-- 1 user user 2827 Jun 30 10:02 train.py ---DATA--- total 1900 drwxr-xr-x 2 user user 60 Jun 30 22:11 . drwxr-xr-x 4 user user 4096 Jun 30 22:11 .. -rw-r--r-- 1 user user 1940490 Jun 30 10:02 train.npz ---NUMPY--- 2.2.6
/app/submission/autograd.py
1 """Reverse-mode automatic differentiation over numpy arrays. 2 3 YOU MUST IMPLEMENT EVERYTHING MARKED ``# TODO`` / ``raise NotImplementedError`` using numpy 4 ONLY. No torch/jax/tensorflow/autograd/etc. The grader's conformance suite (a) differentiates the 5 differentiable ops below with central finite differences and compares against YOUR analytic 6 backward, and (b) checks the LSQ ``fake_quant`` STE + scale gradient against the reference 7 definition, so every backward must be correct (a single wrong reduction / STE / Jacobian fails 8 the gradient checks and zeroes the reward). 9 10 A ``Tensor`` wraps one ``numpy.ndarray`` (stored as ``float64`` in ``self.data``) and records the 11 op that produced it so ``.backward()`` can accumulate ``.grad`` into every leaf. 12 13 CONTRACT (the grader builds its own reference against this exact API): 14 * ``Tensor(data, requires_grad=False)`` -- ``self.data`` is ``float64``; ``self.grad`` is None 15 until backward, then a same-shape ``float64`` array. requires_grad leaves accumulate grad. 16 * Every op returns a NEW Tensor whose ``requires_grad`` is the OR of its differentiable inputs, 17 and whose ``_backward`` closure does ``input._accum(local_grad)`` (so multiple paths add up). 18 * ``t.backward(grad=None)`` -- seeds ``grad`` (all-ones for a scalar) and walks the graph in 19 reverse topological order, calling each node's ``_backward``. Repeated backward accumulates. 20 * BROADCASTING: binary ops broadcast like numpy; the ba …[truncated 54775 chars]
/app/submission/nn.py
1 """QAT layers + a small INT2 CNN, built on YOUR autograd engine.
2
3 Implement every ``# TODO`` forward (the parameters + ``named_params``/``buffers`` naming are
4 already wired for you; you compose the autograd ops). The grader checks each layer's forward
5 against its OWN reference AND finite-difference-checks the gradients that flow through your
6 autograd, so the composition must be exactly right.
7
8 CHECKPOINT NAMING CONTRACT (the grader loads YOUR ``qat_model.npz`` into its own reference
9 QuantCNN by these exact keys/shapes -- do not rename):
10 stem.conv.weight (W0,3,3,3) | stem.conv.wq.scale (1,) | stem.bn.gamma/.beta (W0,) | stem.aq.scale (1,)
11 blocks.{i}.conv.weight (Cout,Cin,3,3) | blocks.{i}.conv.wq.scale (1,)
12 blocks.{i}.bn.gamma/.beta (Cout,) | blocks.{i}.aq.scale (1,)
13 head.weight (Cfeat,n_classes) | head.wq.scale (1,) | head.bias (n_classes,)
14 BN running stats are BUFFERS (saved alongside params, NOT trained by the optimizer, NOT audited):
15 stem.bn.running_mean/.running_var (W0,) | blocks.{i}.bn.running_mean/.running_var (Cout,)
16
17 Quantization: every conv/head WEIGHT is LSQ-quantized to a signed 2-bit grid {-2,-1,0,1} via a
18 per-tensor learnable step before use; every block activation (post-relu) is LSQ-quantized to an
19 unsigned 2-bit grid {0,1,2,3}. Conv = 3x3 same-padding cross-correlation. Linear head = x@weight+bias.
20 """
21 from __future__ import annotations
22
23 from typing import Dict, List
24
25 import numpy as n
…[truncated 35167 chars]/app/submission/optim.py
1 """Optimizers over autograd ``Tensor`` parameters.
2
3 Implement the ``# TODO`` update math. Both optimizers update ``param.data`` IN PLACE using
4 ``param.grad``. The grader runs one (and several) steps and compares your updated params against
5 its reference within a tight tolerance, so the math must be exact.
6
7 SGD (with momentum ``mu`` and coupled weight decay ``wd``):
8 g = grad + wd * param; v = mu * v + g; param -= lr * v
9
10 AdamW (DECOUPLED weight decay -- the decay is applied to the PARAM, not folded into the moments):
11 m = b1*m + (1-b1)*g; v = b2*v + (1-b2)*g^2
12 mhat = m / (1 - b1^t); vhat = v / (1 - b2^t) # t = 1-based step count
13 param -= lr * ( mhat / (sqrt(vhat) + eps) + wd * param )
14
15 PARAM GROUPS: both optimizers must accept EITHER a flat param list OR a list of
16 ``{"params": [...], "weight_decay": wd}`` dicts, so weight decay can be applied to SOME params
17 and not others. This matters for low-bit LSQ QAT: the per-tensor quantizer ``scale`` params,
18 BatchNorm gamma/beta, and biases must be EXCLUDED from weight decay. ``_normalize_groups`` and
19 ``split_decay_params`` are provided; honour the per-group weight decay in ``step``.
20 """
21 from __future__ import annotations
22
23 import math
24
25 from typing import Dict, List
26
27 import numpy as np
28
29 from autograd import Tensor
30
31
32 def split_decay_params(model) -> Dict[str, List[Tensor]]:
33 """Partition a model's named params into weight
…[truncated 11129 chars]/app/submission/data.py
1 """Image data loading + normalization + batching.
2
3 The dataset is a procedurally-generated CIFAR-style set: ``X`` is ``uint8`` (N,3,S,S) in [0,255],
4 ``y`` is the integer class label in [0, n_classes). The shipped training split is at
5 ``/app/data/train.npz``; the SEALED held-out split is regenerated by the verifier (never shipped).
6
7 Implement the ``# TODO`` bodies (numpy only). The verifier normalizes the held-out split with the
8 SAME transform, so match it exactly.
9
10 Contract:
11 load_npz(path) -> (X uint8 (N,3,S,S), y int64 (N,)) [provided]
12 normalize(X) -> float64: x = (X/255 - 0.5) / 0.25 [per-channel, the grader's transform]
13 iter_minibatches(Xn, y, batch_size, rng, shuffle=True) -> yields (xb, yb)
14 """
15 from __future__ import annotations
16
17 from typing import Tuple
18
19 import numpy as np
20
21 NORM_MEAN = 0.5
22 NORM_STD = 0.25
23
24
25 def load_npz(path: str) -> Tuple[np.ndarray, np.ndarray]:
26 d = np.load(path)
27 X = np.asarray(d["X"]).astype(np.uint8)
28 y = np.asarray(d["y"]).astype(np.int64).reshape(-1)
29 return X, y
30
31
32 def normalize(X: np.ndarray) -> np.ndarray:
33 """uint8 (N,3,S,S) -> float64, x = (X/255 - NORM_MEAN) / NORM_STD."""
34 # TODO: cast to float, scale to [0,1], subtract NORM_MEAN, divide by NORM_STD.
35 raise NotImplementedError("normalize")
36
37
38 def iter_minibatches(Xn: np.ndarray, y: np.ndarray, batch_size: int,
39 rng: np.rando
…[truncated 281 chars]/app/submission/train.py
1 """Training entry point. Implement the QAT loop, train your INT2 CNN on /app/data/train.npz, and 2 SAVE the checkpoint to /app/submission/qat_model.npz (numpy .npz of the model state_dict: every 3 param + every BN running buffer, by the nn.QuantCNN naming contract). 4 5 The grader (verify.py) will: 6 (1) run the conformance suite over your autograd/nn/optim, 7 (2) BIT-AUDIT every conv/head weight tensor in your checkpoint to <= 4 distinct float values, 8 (3) load your weights into its OWN QuantCNN and score held-out top-1 on a SEALED split. 9 10 So at SAVE time you must BAKE the quantized weights back into each ``weight`` tensor (replace it 11 with round(clip(w/s, -W_NEG, W_POS)) * s using the learned wq.scale) so the saved weights already 12 lie on the 2-bit grid and pass the audit. Reaching the top-1 floor at INT-2 needs a real recipe: 13 sensible LSQ STEP-SIZE INITIALIZATION (a default scale of 1.0 quantizes the small init weights to 14 all-zeros -> a dead network at chance), per-tensor learnable scales, weight decay EXCLUDED from the 15 scale/BN/bias params, gradient clipping, a decaying LR, and enough epochs. Naive post-training 16 rounding or under-training lands at chance. 17 18 This file is a skeleton; the loop body is yours to write. 19 """ 20 from __future__ import annotations 21 22 import argparse 23 import os 24 import sys 25 26 import numpy as np 27 28 sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) 29 sys.path.insert(0, "/app") 30 3 …[truncated 1513 chars]
/app/run_checks.py
<system-reminder>[Truncated: PARTIAL view , showing lines 1-871 of 1523 total (37115 tokens, cap 25000). Call Read with offset=872 limit=871 for the next page, or Grep to find a specific section. Do NOT answer from this page alone if the answer may be further in the file.]</system-reminder> 1 #!/usr/bin/env python3 2 """PUBLIC component check runner for the from-scratch numpy QAT framework. 3 4 Run this AS OFTEN AS YOU LIKE while you implement, to get a per-check progress signal: 5 6 python3 /app/run_checks.py # check the framework in /app/submission 7 python3 /app/run_checks.py --submission /some/dir 8 python3 /app/run_checks.py --only autograd_grads # run one milestone 9 python3 /app/run_checks.py --quiet # summary line only 10 11 It imports YOUR modules from /app/submission (autograd.py / nn.py / optim.py / data.py), exactly as 12 the grader does, and runs a PUBLIC portion of the component conformance suite, printing a clear 13 per-check report so you can implement -> run -> read failures -> fix -> repeat. 14 15 HOW THIS RELATES TO YOUR GRADE 16 ------------------------------ 17 This runner is a DEBUGGING AID, not the grader. It checks YOUR framework against references it can 18 compute on its own: 19 * autograd_grads : central FINITE-DIFFERENCE of YOUR forward vs YOUR analytic backward, for every 20 differentiable op + the domain ops (conv2d/avgpool2d/maxpool2d/batchnorm2d/ 21 …[truncated 48708 chars]
/app/run_checks.py
872 [x[0, 0, 2:4, 0:2].max(), x[0, 0, 2:4, 2:4].max()]])
873 return _max_abs(y[0, 0], ref) < 1e-9 and y.shape == (1, 1, 2, 2), ""
874
875 def avgpool_def():
876 x = np.random.default_rng(sb + 20).standard_normal((1, 1, 4, 4))
877 y = ag.avgpool2d(T(x), 2).data
878 return abs(y[0, 0, 0, 0] - x[0, 0, 0:2, 0:2].mean()) < 1e-9, ""
879
880 def fakequant_grid_w():
881 x = np.random.default_rng(sb + 21).standard_normal((6, 6)) * 2.0
882 s = 0.5
883 out = ag.fake_quant(T(x), T([s]), 2, 1).data
884 levels = np.round(out / s)
885 in_grid = np.all((levels >= -2) & (levels <= 1))
886 on_grid = _max_abs(out, levels * s) < 1e-9
887 return bool(in_grid and on_grid), f"unique={np.unique(out).size}"
888
889 def fakequant_grid_a():
890 x = np.random.default_rng(sb + 22).standard_normal((6, 6)) * 2.0
891 s = 0.4
892 out = ag.fake_quant(T(x), T([s]), 0, 3).data
893 levels = np.round(out / s)
894 return bool(np.all((levels >= 0) & (levels <= 3))), f"unique={np.unique(out).size}"
895
896 def linear_def():
897 rng = np.random.default_rng(sb + 23)
898 lin = nn.Linear(5, 3, np.random.default_rng(sb + 90))
899 x = rng.standard_normal((4, 5))
900 return _max_abs(lin(T(x)).data, x @ lin.weight.data + lin.bias.data) < 1e-6, ""
901
902 def mlp_shape():
903 y = nn.MLP(8, 16, np.random.default_
…[truncated 31859 chars]Inspect training data
python3 -c "
import numpy as np
d=np.load('data/train.npz')
print('keys',list(d.keys()))
X=d['X']; y=d['y']
print('X',X.shape,X.dtype,X.min(),X.max())
print('y',y.shape,y.dtype,np.bincount(y))
"keys ['X', 'y'] X (2500, 3, 16, 16) uint8 0 255 y (2500,) int64 [250 250 250 250 250 250 250 250 250 250]
/app/submission/autograd.py
TODO: implement the topological walk + seeding. (The per-op ``_backward`` closures do the
local accumulation; here you only orchestrate the order + seed ``self.grad``.)
"""
raise NotImplementedError("Tensor.backward") TODO: implement the topological walk + seeding. (The per-op ``_backward`` closures do the
local accumulation; here you only orchestrate the order + seed ``self.grad``.)
"""
topo: List[Tensor] = []
visited = set()
def build(t: "Tensor"):
if id(t) in visited:
return
visited.add(id(t))
for p in t._parents:
build(p)
topo.append(t)
build(self)
if grad is None:
grad = np.ones_like(self.data)
else:
grad = _as_f64(grad)
if self.grad is None:
self.grad = np.zeros_like(self.data)
self.grad += grad
for node in reversed(topo):
node._backward()The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data + b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("add backward") # TODO out = _mk(a.data + b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
if a.requires_grad:
a._accum(_unbroadcast(g, a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(g, b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data - b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("sub backward") # TODO out = _mk(a.data - b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
if a.requires_grad:
a._accum(_unbroadcast(g, a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(-g, b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data * b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("mul backward") # TODO out = _mk(a.data * b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
if a.requires_grad:
a._accum(_unbroadcast(g * b.data, a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(g * a.data, b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data / b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("div backward") # TODO out = _mk(a.data / b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
if a.requires_grad:
a._accum(_unbroadcast(g / b.data, a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(-g * a.data / (b.data * b.data), b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data ** p, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("power backward") # TODO out = _mk(a.data ** p, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * p * a.data ** (p - 1))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.maximum(a.data, b.data), (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("maximum backward") # TODO out = _mk(np.maximum(a.data, b.data), (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
ga = np.where(a.data > b.data, 1.0, np.where(a.data == b.data, 0.5, 0.0))
if a.requires_grad:
a._accum(_unbroadcast(g * ga, a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(g * (1.0 - ga), b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.minimum(a.data, b.data), (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("minimum backward") # TODO out = _mk(np.minimum(a.data, b.data), (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
ga = np.where(a.data < b.data, 1.0, np.where(a.data == b.data, 0.5, 0.0))
if a.requires_grad:
a._accum(_unbroadcast(g * ga, a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(g * (1.0 - ga), b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.where(c, a.data, b.data), (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("where backward") # TODO out = _mk(np.where(c, a.data, b.data), (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
if a.requires_grad:
a._accum(_unbroadcast(np.where(c, g, 0.0), a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(np.where(c, 0.0, g), b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.exp(a.data), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("exp backward") # TODO out = _mk(np.exp(a.data), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * out.data)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.log(a.data), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("log backward") # TODO out = _mk(np.log(a.data), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad / a.data)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.sqrt(a.data), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("sqrt backward") # TODO out = _mk(np.sqrt(a.data), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * 0.5 / out.data)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.abs(a.data), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("abs backward") # TODO out = _mk(np.abs(a.data), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * np.sign(a.data))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.clip(a.data, lo, hi), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("clip backward") # TODO out = _mk(np.clip(a.data, lo, hi), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
mask = (a.data > lo) & (a.data < hi)
a._accum(out.grad * mask)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data.sum(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("sum backward") # TODO out = _mk(a.data.sum(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
g = out.grad
if axis is not None and not keepdims:
g = np.expand_dims(g, axis)
a._accum(np.broadcast_to(g, a.data.shape).copy())The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data.mean(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("mean backward") # TODO out = _mk(a.data.mean(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
g = out.grad
if axis is None:
n = a.data.size
else:
ax = axis if isinstance(axis, tuple) else (axis,)
n = 1
for x in ax:
n *= a.data.shape[x]
if axis is not None and not keepdims:
g = np.expand_dims(g, axis)
a._accum(np.broadcast_to(g, a.data.shape).copy() / n)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data.max(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("max backward") # TODO out = _mk(a.data.max(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
outk = a.data.max(axis=axis, keepdims=True)
g = out.grad
if axis is not None and not keepdims:
g = np.expand_dims(g, axis)
mask = (a.data == outk)
counts = mask.sum(axis=axis, keepdims=True)
a._accum(mask * (g / counts))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data.min(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("min backward") # TODO out = _mk(a.data.min(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
outk = a.data.min(axis=axis, keepdims=True)
g = out.grad
if axis is not None and not keepdims:
g = np.expand_dims(g, axis)
mask = (a.data == outk)
counts = mask.sum(axis=axis, keepdims=True)
a._accum(mask * (g / counts))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
any(t.requires_grad for t in ts))
def _bw():
raise NotImplementedError("concat backward") # TODO any(t.requires_grad for t in ts))
def _bw():
g = out.grad
start = 0
for t in ts:
n = t.data.shape[axis]
if t.requires_grad:
idx = [slice(None)] * g.ndim
idx[axis] = slice(start, start + n)
t._accum(g[tuple(idx)])
start += nThe file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
any(t.requires_grad for t in ts))
def _bw():
raise NotImplementedError("stack backward") # TODO any(t.requires_grad for t in ts))
def _bw():
g = out.grad
for i, t in enumerate(ts):
if t.requires_grad:
idx = [slice(None)] * g.ndim
idx[axis] = i
t._accum(g[tuple(idx)])The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.transpose(a.data, axes), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("transpose backward") # TODO out = _mk(np.transpose(a.data, axes), (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
if axes is None:
a._accum(np.transpose(out.grad))
else:
inv = np.argsort(axes)
a._accum(np.transpose(out.grad, inv))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data.reshape(shape), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("reshape backward") # TODO out = _mk(a.data.reshape(shape), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad.reshape(a.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data[idx], (a,), a.requires_grad)
def _bw():
raise NotImplementedError("getitem backward") # TODO out = _mk(a.data[idx], (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
gz = np.zeros_like(a.data)
np.add.at(gz, idx, out.grad)
a._accum(gz)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data @ b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("matmul backward") # TODO out = _mk(a.data @ b.data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
if a.requires_grad:
da = g @ np.swapaxes(b.data, -1, -2)
a._accum(_unbroadcast(da, a.data.shape))
if b.requires_grad:
db = np.swapaxes(a.data, -1, -2) @ g
b._accum(_unbroadcast(db, b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.maximum(a.data, 0.0), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("relu backward") # TODO out = _mk(np.maximum(a.data, 0.0), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * (a.data > 0.0))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.where(a.data > 0.0, a.data, slope * a.data), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("leaky_relu backward") # TODO out = _mk(np.where(a.data > 0.0, a.data, slope * a.data), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * np.where(a.data > 0.0, 1.0, slope))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(s, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("sigmoid backward") # TODO out = _mk(s, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * out.data * (1.0 - out.data))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.tanh(a.data), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("tanh backward") # TODO out = _mk(np.tanh(a.data), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * (1.0 - out.data * out.data))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
cdf = 0.5 * (1.0 + _erf(a.data * _INV_SQRT2))
out = _mk(a.data * cdf, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("gelu backward") # TODO cdf = 0.5 * (1.0 + _erf(a.data * _INV_SQRT2))
out = _mk(a.data * cdf, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
pdf = np.exp(-0.5 * a.data * a.data) / np.sqrt(2.0 * np.pi)
a._accum(out.grad * (cdf + a.data * pdf))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(s, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("softmax backward") # TODO out = _mk(s, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
g = out.grad
sm = out.data
a._accum(sm * (g - (g * sm).sum(axis=axis, keepdims=True)))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(ls, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("log_softmax backward") # TODO out = _mk(ls, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
g = out.grad
sm = np.exp(out.data)
a._accum(g - sm * g.sum(axis=axis, keepdims=True))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(-logp[np.arange(n), t].mean(), (logits,), logits.requires_grad)
def _bw():
raise NotImplementedError("cross_entropy backward") # TODO out = _mk(-logp[np.arange(n), t].mean(), (logits,), logits.requires_grad)
def _bw():
if logits.requires_grad:
sm = np.exp(logp)
onehot = np.zeros_like(sm)
onehot[np.arange(n), t] = 1.0
logits._accum((sm - onehot) / n * out.grad)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(float(np.mean((pred.data - tgt) ** 2)), (pred,), pred.requires_grad)
def _bw():
raise NotImplementedError("mse_loss backward") # TODO out = _mk(float(np.mean((pred.data - tgt) ** 2)), (pred,), pred.requires_grad)
def _bw():
if pred.requires_grad:
pred._accum((2.0 / pred.data.size) * (pred.data - tgt) * out.grad)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(gamma.data * xhat + beta.data, (x, gamma, beta),
x.requires_grad or gamma.requires_grad or beta.requires_grad)
def _bw():
raise NotImplementedError("layernorm backward") # TODO out = _mk(gamma.data * xhat + beta.data, (x, gamma, beta),
x.requires_grad or gamma.requires_grad or beta.requires_grad)
def _bw():
g = out.grad
lead = tuple(range(g.ndim - 1))
if gamma.requires_grad:
gamma._accum((g * xhat).sum(axis=lead))
if beta.requires_grad:
beta._accum(g.sum(axis=lead))
if x.requires_grad:
gx = g * gamma.data
dx = (inv / D) * (D * gx - gx.sum(axis=-1, keepdims=True)
- xhat * (gx * xhat).sum(axis=-1, keepdims=True))
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("conv2d backward") # TODO (dW, dbias, dx via _col2im) def _bw():
g = out.grad.reshape(N, Cout, OH * OW) # dout_mat
if weight.requires_grad:
dW = np.einsum("nop,ncp->oc", g, cols).reshape(Cout, Cin, KH, KW)
weight._accum(dW)
if has_bias and bias.requires_grad:
bias._accum(g.sum(axis=(0, 2)))
if x.requires_grad:
dcols = np.einsum("oc,nop->ncp", Wm, g)
dxp = _col2im(dcols, xp.shape, KH, KW, stride, OH, OW)
if pad > 0:
dx = dxp[:, :, pad:pad + H, pad:pad + W]
else:
dx = dxp
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(xr.mean(axis=(3, 5)), (x,), x.requires_grad)
def _bw():
raise NotImplementedError("avgpool2d backward") # TODO out = _mk(xr.mean(axis=(3, 5)), (x,), x.requires_grad)
def _bw():
if not x.requires_grad:
return
g = out.grad # (N,C,OH,OW)
gg = g[:, :, :, None, :, None] / (k * k)
gg = np.broadcast_to(gg, (N, C, OH, k, OW, k))
dx = np.zeros_like(x.data)
dx[:, :, :OH * k, :OW * k] = gg.reshape(N, C, OH * k, OW * k)
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(xr.max(axis=(3, 5)), (x,), x.requires_grad)
def _bw():
raise NotImplementedError("maxpool2d backward") # TODO out = _mk(xr.max(axis=(3, 5)), (x,), x.requires_grad)
def _bw():
if not x.requires_grad:
return
g = out.grad # (N,C,OH,OW)
outk = out.data[:, :, :, None, :, None] # (N,C,OH,1,OW,1)
mask = (xr == outk)
# break ties: keep only first max in each window
flat = mask.reshape(N, C, OH, OW, k * k)
# zero out all but first True along last axis
cum = np.cumsum(flat.reshape(N, C, OH, k, OW, k).transpose(0, 1, 2, 4, 3, 5).reshape(N, C, OH, OW, k * k), axis=-1)
firstmask = flat & (cum == 1)
gg = firstmask.reshape(N, C, OH, OW, k, k).transpose(0, 1, 2, 4, 3, 5) * g[:, :, :, None, :, None]
dx = np.zeros_like(x.data)
dx[:, :, :OH * k, :OW * k] = gg.reshape(N, C, OH * k, OW * k)
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
if not x.requires_grad:
return
g = out.grad # (N,C,OH,OW)
outk = out.data[:, :, :, None, :, None] # (N,C,OH,1,OW,1)
mask = (xr == outk)
# break ties: keep only first max in each window
flat = mask.reshape(N, C, OH, OW, k * k)
# zero out all but first True along last axis
cum = np.cumsum(flat.reshape(N, C, OH, k, OW, k).transpose(0, 1, 2, 4, 3, 5).reshape(N, C, OH, OW, k * k), axis=-1)
firstmask = flat & (cum == 1)
gg = firstmask.reshape(N, C, OH, OW, k, k).transpose(0, 1, 2, 4, 3, 5) * g[:, :, :, None, :, None]
dx = np.zeros_like(x.data)
dx[:, :, :OH * k, :OW * k] = gg.reshape(N, C, OH * k, OW * k)
x._accum(dx) def _bw():
if not x.requires_grad:
return
g = out.grad # (N,C,OH,OW)
outk = out.data[:, :, :, None, :, None] # (N,C,OH,1,OW,1)
mask = (xr == outk).astype(np.float64) # (N,C,OH,k,OW,k)
counts = mask.sum(axis=(3, 5), keepdims=True)
gg = mask * (g[:, :, :, None, :, None] / counts)
dx = np.zeros_like(x.data)
dx[:, :, :OH * k, :OW * k] = gg.reshape(N, C, OH * k, OW * k)
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("batchnorm2d backward") # TODO def _bw():
g = out.grad
if gamma.requires_grad:
gamma._accum((g * xhat).sum(axis=(0, 2, 3)))
if beta.requires_grad:
beta._accum(g.sum(axis=(0, 2, 3)))
if x.requires_grad:
gx = g * g_
if training:
dx = (inv / M) * (M * gx - gx.sum(axis=(0, 2, 3), keepdims=True)
- xhat * (gx * xhat).sum(axis=(0, 2, 3), keepdims=True))
else:
dx = gx * inv
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(out_data, (v, scale), v.requires_grad or scale.requires_grad)
def _bw():
raise NotImplementedError("fake_quant backward (STE + LSQ scale gradient)") # TODO out = _mk(out_data, (v, scale), v.requires_grad or scale.requires_grad)
def _bw():
g = out.grad
below = r < Qn
above = r > Qp
middle = ~(below | above)
if v.requires_grad:
v._accum(g * middle)
if scale.requires_grad:
d = np.where(middle, v_hat - r, np.where(below, Qn, Qp))
ds = float((g * d).sum()) * grad_scale
scale._accum(np.array(ds).reshape(scale.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data * a.data, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("square backward") # TODO out = _mk(a.data * a.data, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * 2.0 * a.data)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(1.0 / np.sqrt(a.data), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("rsqrt backward") # TODO out = _mk(1.0 / np.sqrt(a.data), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * -0.5 * a.data ** (-1.5))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(1.0 / a.data, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("reciprocal backward") # TODO out = _mk(1.0 / a.data, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * -1.0 / (a.data * a.data))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk((xc * xc).mean(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("var backward") # TODO out = _mk((xc * xc).mean(axis=axis, keepdims=keepdims), (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
if axis is None:
n = a.data.size
else:
ax = axis if isinstance(axis, tuple) else (axis,)
n = 1
for x in ax:
n *= a.data.shape[x]
g = out.grad
if axis is not None and not keepdims:
g = np.expand_dims(g, axis)
a._accum(g * (2.0 / n) * xc)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(out_data, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("std backward") # TODO out = _mk(out_data, (a,), a.requires_grad)
def _bw():
if not a.requires_grad:
return
if axis is None:
n = a.data.size
else:
ax = axis if isinstance(axis, tuple) else (axis,)
n = 1
for x in ax:
n *= a.data.shape[x]
g = out.grad
if axis is not None and not keepdims:
g = np.expand_dims(g, axis)
a._accum(g * xc / (n * sd))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.take_along_axis(a.data, idx, axis=axis), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("gather backward") # TODO out = _mk(np.take_along_axis(a.data, idx, axis=axis), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
gz = np.zeros_like(a.data)
np.put_along_axis(gz, idx, 0.0, axis=axis) # ensure broadcast shape
np.add.at(gz, _take_along_index(a.data.shape, idx, axis), 0.0)
gz = np.zeros_like(a.data)
_scatter_add_along_axis(gz, idx, out.grad, axis)
a._accum(gz)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
if a.requires_grad:
gz = np.zeros_like(a.data)
np.put_along_axis(gz, idx, 0.0, axis=axis) # ensure broadcast shape
np.add.at(gz, _take_along_index(a.data.shape, idx, axis), 0.0)
gz = np.zeros_like(a.data)
_scatter_add_along_axis(gz, idx, out.grad, axis)
a._accum(gz) def _bw():
if a.requires_grad:
gz = np.zeros_like(a.data)
ax = axis if axis >= 0 else axis + a.data.ndim
grids = np.indices(idx.shape, sparse=True)
index = []
for d in range(a.data.ndim):
if d == ax:
index.append(idx)
else:
index.append(grids[d])
np.add.at(gz, tuple(index), out.grad)
a._accum(gz)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(_pad_nchw(x.data, pad), (x,), x.requires_grad)
def _bw():
raise NotImplementedError("pad2d backward") # TODO out = _mk(_pad_nchw(x.data, pad), (x,), x.requires_grad)
def _bw():
if x.requires_grad:
if pad > 0:
H, W = x.data.shape[2], x.data.shape[3]
x._accum(out.grad[:, :, pad:pad + H, pad:pad + W])
else:
x._accum(out.grad)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk((np.maximum(bx, 0.0) + np.log1p(np.exp(-np.abs(bx)))) / beta, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("softplus backward") # TODO out = _mk((np.maximum(bx, 0.0) + np.log1p(np.exp(-np.abs(bx)))) / beta, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
sig = 1.0 / (1.0 + np.exp(-bx))
a._accum(out.grad * sig)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(a.data * sig, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("silu backward") # TODO out = _mk(a.data * sig, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * (sig + a.data * sig * (1.0 - sig)))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(x * np.tanh(sp), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("mish backward") # TODO out = _mk(x * np.tanh(sp), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
tsp = np.tanh(sp)
sig = 1.0 / (1.0 + np.exp(-x))
a._accum(out.grad * (tsp + x * (1.0 - tsp * tsp) * sig))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.where(x > 0.0, x, alpha * (np.exp(np.minimum(x, 0.0)) - 1.0)), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("elu backward") # TODO out = _mk(np.where(x > 0.0, x, alpha * (np.exp(np.minimum(x, 0.0)) - 1.0)), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * np.where(x > 0.0, 1.0, alpha * np.exp(np.minimum(x, 0.0))))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.clip(a.data, lo, hi), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("hardtanh backward") # TODO out = _mk(np.clip(a.data, lo, hi), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * ((a.data > lo) & (a.data < hi)))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.clip(z, 0.0, 1.0), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("hardsigmoid backward") # TODO out = _mk(np.clip(z, 0.0, 1.0), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(out.grad * np.where((z > 0.0) & (z < 1.0), 1.0 / 6.0, 0.0))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("groupnorm backward") # TODO def _bw():
g = out.grad # (N,C,H,W)
if gamma.requires_grad:
gamma._accum((g * xhat).sum(axis=(0, 2, 3)))
if beta.requires_grad:
beta._accum(g.sum(axis=(0, 2, 3)))
if x.requires_grad:
M = cg * H * W
gx = (g * gamma.data.reshape(1, C, 1, 1)).reshape(N, G, M)
xh = xhat.reshape(N, G, M)
dx = (inv / M) * (M * gx - gx.sum(axis=2, keepdims=True)
- xh * (gx * xh).sum(axis=2, keepdims=True))
x._accum(dx.reshape(N, C, H, W))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("fake_quant_per_channel backward (STE + per-channel scale grad)") # TODO def _bw():
g = out.grad
below = r < Qn
above = r > Qp
middle = ~(below | above)
if v.requires_grad:
v._accum(g * middle)
if scale.requires_grad:
d = np.where(middle, v_hat - r, np.where(below, Qn, Qp))
gd = g * d
# sum over all axes except `axis`
sum_axes = tuple(i for i in range(v.data.ndim) if i != (axis if axis >= 0 else axis + v.data.ndim))
ds = gd.sum(axis=sum_axes) * grad_scale
scale._accum(ds.reshape(scale.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("fake_quant_affine backward (STE + scale grad on shifted grid)") # TODO def _bw():
g = out.grad
below = r < Qn
above = r > Qp
middle = ~(below | above)
if v.requires_grad:
v._accum(g * middle)
if scale.requires_grad:
d = np.where(middle, (q - z) - (r - z), np.where(below, Qn - z, Qp - z))
ds = float((g * d).sum()) * grad_scale
scale._accum(np.array(ds).reshape(scale.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.cumsum(a.data, axis=axis), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("cumsum backward") # TODO out = _mk(np.cumsum(a.data, axis=axis), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
g = out.grad
rev = np.flip(np.cumsum(np.flip(g, axis=axis), axis=axis), axis=axis)
a._accum(rev)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(np.flip(a.data, axis=axis), (a,), a.requires_grad)
def _bw():
raise NotImplementedError("flip backward") # TODO out = _mk(np.flip(a.data, axis=axis), (a,), a.requires_grad)
def _bw():
if a.requires_grad:
a._accum(np.flip(out.grad, axis=axis))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
sm = np.exp(a.data - lse)
def _bw():
raise NotImplementedError("logsumexp backward") # TODO sm = np.exp(a.data - lse)
def _bw():
if a.requires_grad:
g = out.grad
if not keepdims:
g = np.expand_dims(g, axis)
a._accum(sm * g)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(out_data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
raise NotImplementedError("logaddexp backward") # TODO out = _mk(out_data, (a, b), a.requires_grad or b.requires_grad)
def _bw():
g = out.grad
wa = np.exp(a.data - out_data)
wb = np.exp(b.data - out_data)
if a.requires_grad:
a._accum(_unbroadcast(g * wa, a.data.shape))
if b.requires_grad:
b._accum(_unbroadcast(g * wb, b.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
y = x / nrm
out = _mk(y, (a,), a.requires_grad)
def _bw():
raise NotImplementedError("l2_normalize backward") # TODO y = x / nrm
out = _mk(y, (a,), a.requires_grad)
def _bw():
if a.requires_grad:
g = out.grad
dot = (y * g).sum(axis=axis, keepdims=True)
a._accum((g - y * dot) / nrm)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(gamma.data * xhat, (x, gamma), x.requires_grad or gamma.requires_grad)
def _bw():
raise NotImplementedError("rms_norm backward") # TODO out = _mk(gamma.data * xhat, (x, gamma), x.requires_grad or gamma.requires_grad)
def _bw():
g = out.grad
lead = tuple(range(g.ndim - 1))
if gamma.requires_grad:
gamma._accum((g * xhat).sum(axis=lead))
if x.requires_grad:
gg = g * gamma.data
dx = inv * gg - (xd * inv ** 3 / D) * (gg * xd).sum(axis=-1, keepdims=True)
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(g_ * xhat + beta.data.reshape(1, C, 1, 1), (x, gamma, beta),
x.requires_grad or gamma.requires_grad or beta.requires_grad)
def _bw():
raise NotImplementedError("instance_norm backward") # TODO out = _mk(g_ * xhat + beta.data.reshape(1, C, 1, 1), (x, gamma, beta),
x.requires_grad or gamma.requires_grad or beta.requires_grad)
def _bw():
g = out.grad
if gamma.requires_grad:
gamma._accum((g * xhat).sum(axis=(0, 2, 3)))
if beta.requires_grad:
beta._accum(g.sum(axis=(0, 2, 3)))
if x.requires_grad:
gx = (g * g_).reshape(N, C, M)
xh = xhat.reshape(N, C, M)
dx = (inv / M) * (M * gx - gx.sum(axis=2, keepdims=True)
- xh * (gx * xh).sum(axis=2, keepdims=True))
x._accum(dx.reshape(N, C, H, W))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(float(per.mean()), (pred,), pred.requires_grad)
def _bw():
raise NotImplementedError("huber_loss backward") # TODO out = _mk(float(per.mean()), (pred,), pred.requires_grad)
def _bw():
if pred.requires_grad:
d = np.where(quad, diff, delta * np.sign(diff))
pred._accum(d / n * out.grad)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(float((ql - q * log_p.data).sum() / n), (log_p,), log_p.requires_grad)
def _bw():
raise NotImplementedError("kl_div backward") # TODO out = _mk(float((ql - q * log_p.data).sum() / n), (log_p,), log_p.requires_grad)
def _bw():
if log_p.requires_grad:
log_p._accum((-q / n) * out.grad)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(weight.data[idx], (weight,), weight.requires_grad)
def _bw():
raise NotImplementedError("embedding backward") # TODO out = _mk(weight.data[idx], (weight,), weight.requires_grad)
def _bw():
if weight.requires_grad:
gz = np.zeros_like(weight.data)
np.add.at(gz, idx, out.grad)
weight._accum(gz)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("conv2d_gen backward (grouped/dilated dW/db/dx)") # TODO def _bw():
g = out.grad.reshape(N, Cout, OH * OW)
if has_bias and bias.requires_grad:
bias._accum(g.sum(axis=(0, 2)))
g_g = g.reshape(N, groups, cog, OH * OW)
if weight.requires_grad:
dWm = np.einsum("ngop,ngcp->goc", g_g, cols_g)
weight._accum(dWm.reshape(Cout, cig, KH, KW))
if x.requires_grad:
dcols_g = np.einsum("goc,ngop->ngcp", Wm, g_g)
dcols = dcols_g.reshape(N, Cin * KH * KW, OH * OW)
dxp = _col2im_dil(dcols, xp.shape, KH, KW, stride, dilation, OH, OW)
if pad > 0:
dx = dxp[:, :, pad:pad + H, pad:pad + W]
else:
dx = dxp
x._accum(dx)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("conv_transpose2d backward") # TODO def _bw():
g = out.grad
if has_bias and bias.requires_grad:
bias._accum(g.sum(axis=(0, 2, 3)))
if pad > 0:
gfull = np.zeros((N, Cout, OHf, OWf), dtype=np.float64)
gfull[:, :, pad:OHf - pad, pad:OWf - pad] = g
else:
gfull = g
gcontrib = np.empty((N, Cout, H, W, KH, KW), dtype=np.float64)
for i in range(KH):
for j in range(KW):
gcontrib[:, :, :, :, i, j] = gfull[:, :, i:i + stride * H:stride, j:j + stride * W:stride]
if x.requires_grad:
dx = np.einsum("noijKL,coKL->ncij", gcontrib, Wm)
x._accum(dx)
if weight.requires_grad:
dW = np.einsum("ncij,noijKL->coKL", xd, gcontrib)
weight._accum(dW)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(out_data, (x,), x.requires_grad)
def _bw():
raise NotImplementedError("avgpool2d_s backward") # TODO out = _mk(out_data, (x,), x.requires_grad)
def _bw():
if not x.requires_grad:
return
g = out.grad
dxp = np.zeros((N, C, Hp, Wp), dtype=np.float64)
for oi in range(OH):
for oj in range(OW):
dxp[:, :, oi * stride:oi * stride + k, oj * stride:oj * stride + k] += \
(g[:, :, oi, oj] / (k * k))[:, :, None, None]
if pad > 0:
x._accum(dxp[:, :, pad:pad + H, pad:pad + W])
else:
x._accum(dxp)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
out = _mk(out_data, (x,), x.requires_grad)
def _bw():
raise NotImplementedError("maxpool2d_s backward") # TODO out = _mk(out_data, (x,), x.requires_grad)
def _bw():
if not x.requires_grad:
return
g = out.grad
dxp = np.zeros((N, C, Hp, Wp), dtype=np.float64)
nn_, cc_ = np.meshgrid(np.arange(N), np.arange(C), indexing="ij")
for oi in range(OH):
for oj in range(OW):
ii = oi * stride + argi[:, :, oi, oj]
jj = oj * stride + argj[:, :, oi, oj]
np.add.at(dxp, (nn_, cc_, ii, jj), g[:, :, oi, oj])
if pad > 0:
x._accum(dxp[:, :, pad:pad + H, pad:pad + W])
else:
x._accum(dxp)The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/autograd.py
def _bw():
raise NotImplementedError("fake_quant_lsq_plus backward (STE v-grad + beta-grad + LSQ scale grad)") # TODO def _bw():
g = out.grad
below = r < Qn
above = r > Qp
middle = ~(below | above)
if v.requires_grad:
v._accum(g * middle)
if beta.requires_grad:
db = float((g * (~middle)).sum())
beta._accum(np.array(db).reshape(beta.data.shape))
if scale.requires_grad:
d = np.where(middle, np.round(np.clip(r, Qn, Qp)) - r, np.where(below, Qn, Qp))
ds = float((g * d).sum()) * grad_scale
scale._accum(np.array(ds).reshape(scale.data.shape))The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
Check autograd imports
cd /app/submission && python3 -c "import autograd; print('autograd OK')"autograd OK
Fill nn.py forwards batch 1
python3 - <<'EOF'
import re
f='nn.py'
s=open(f).read()
repls = [
(' # TODO: y = x @ weight (+ bias). Use ag.matmul / ag.add.\n raise NotImplementedError("Linear.forward")',
' y = ag.matmul(x, self.weight)\n if self.bias is not None:\n y = ag.add(y, self.bias)\n return y'),
(' # TODO: out = x + fc2(gelu(fc1(x)))\n raise NotImplementedError("MLP.forward")',
' return ag.add(x, self.fc2(ag.gelu(self.fc1(x))))'),
(' # TODO: return ag.layernorm(x, self.gamma, self.beta, self.eps)\n raise NotImplementedError("LayerNorm.forward")',
' return ag.layernorm(x, self.gamma, self.beta, self.eps)'),
(' # TODO: return ag.conv2d(x, self.weight, bias=self.bias, stride=self.stride, pad=self.padding)\n raise NotImplementedError("Conv2d.forward")',
' return ag.conv2d(x, self.weight, bias=self.bias, stride=self.stride, pad=self.padding)'),
(' # TODO: return ag.maxpool2d(x, self.k)\n raise NotImplementedError("MaxPool2d.forward")',
' return ag.maxpool2d(x, self.k)'),
(' # TODO: return ag.avgpool2d(x, self.k)\n raise NotImplementedError("AvgPool2d.forward")',
' return ag.avgpool2d(x, self.k)'),
(' # TODO: return ag.fake_quant(w, self.scale, W_NEG, W_POS)\n raise NotImplementedError("WeightQuant.quant")',
' return ag.fake_quant(w, self.scale, W_NEG, W_POS)'),
(' # TODO: return ag.fake_quant(x, self.scale, A_NEG, A_POS)
…[truncated 1359 chars]batch1 done
Check batchnorm1d existence
grep -n "def batchnorm1d\|def conv2d\b\|def avgpool2d\b" autograd.py
759:def conv2d(x, weight, bias=None, stride=1, pad=1) -> Tensor: 804:def avgpool2d(x, k) -> Tensor:
/app/submission/autograd.py
747 Provided helper -- use it for the conv input gradient.""" 748 N, C, H, W = x_shape 749 cols = cols.reshape(N, C, KH, KW, OH, OW) 750 dx = np.zeros((N, C, H, W), dtype=cols.dtype) 751 for i in range(KH): 752 i_max = i + stride * OH 753 for j in range(KW): 754 j_max = j + stride * OW 755 dx[:, :, i:i_max:stride, j:j_max:stride] += cols[:, :, i, j, :, :] 756 return dx 757 758 759 def conv2d(x, weight, bias=None, stride=1, pad=1) -> Tensor: 760 """2-D convolution (cross-correlation), NCHW, zero padding. ``x``: (N,Cin,H,W);
/app/submission/autograd.py
855 training: bool = True) -> Tensor: 856 """BatchNorm over (N,H,W) per channel C. x:(N,C,H,W); gamma,beta:(C,). 857 858 TRAINING: normalize with the BATCH mean/var (POPULATION variance, divide by M=N*H*W); update 859 running_mean/running_var IN PLACE if given (running var tracks the UNBIASED batch variance, 860 i.e. var * M/(M-1)). The forward computes mu/var/inv/xhat; you implement the backward. 861 EVAL (training=False): normalize with the running stats; backward dx = grad*gamma*inv. 862 Backward (training): dgamma=sum(grad*xhat) over (N,H,W); dbeta=sum(grad) over (N,H,W); 863 gx = grad*gamma; dx = inv/M * (M*gx - sum(gx) - xhat*sum(gx*xhat)) (sums over N,H,W). 864 """ 865 x, gamma, beta = _ensure(x), _ensure(gamma), _ensure(beta) 866 N, C, H, W = x.data.shape 867 M = N * H * W 868 xd = x.data
/app/submission/autograd.py
885 out = _mk(g_ * xhat + beta.data.reshape(1, C, 1, 1), (x, gamma, beta), 886 x.requires_grad or gamma.requires_grad or beta.requires_grad) 887 888 def _bw(): 889 g = out.grad 890 if gamma.requires_grad: 891 gamma._accum((g * xhat).sum(axis=(0, 2, 3))) 892 if beta.requires_grad: 893 beta._accum(g.sum(axis=(0, 2, 3))) 894 if x.requires_grad: 895 gx = g * g_ 896 if training: 897 dx = (inv / M) * (M * gx - gx.sum(axis=(0, 2, 3), keepdims=True) 898 - xhat * (gx * xhat).sum(axis=(0, 2, 3), keepdims=True)) 899 else: 900 dx = gx * inv 901 x._accum(dx) 902 903 out._backward = _bw 904 return out
/app/submission/autograd.py
else:
dx = gx * inv
x._accum(dx)
out._backward = _bw
return out
# ============================================================ LSQ fake-quant (DOMAIN; the crux) else:
dx = gx * inv
x._accum(dx)
out._backward = _bw
return out
def batchnorm1d(x, gamma, beta, eps: float = 1e-5,
running_mean=None, running_var=None, momentum: float = 0.1,
training: bool = True) -> Tensor:
"""BatchNorm over the batch axis N for a (N,C) input. gamma,beta:(C,). Same semantics as
batchnorm2d but with M=N."""
x, gamma, beta = _ensure(x), _ensure(gamma), _ensure(beta)
N, C = x.data.shape
M = N
xd = x.data
if training:
mu = xd.mean(axis=0, keepdims=True)
xc = xd - mu
var = (xc * xc).mean(axis=0, keepdims=True)
inv = 1.0 / np.sqrt(var + eps)
xhat = xc * inv
if running_mean is not None and running_var is not None:
ubvar = var * (M / _bi_max(1, M - 1))
running_mean[:] = (1 - momentum) * running_mean + momentum * mu.reshape(-1)
running_var[:] = (1 - momentum) * running_var + momentum * ubvar.reshape(-1)
else:
mu = running_mean.reshape(1, C)
var = running_var.reshape(1, C)
inv = 1.0 / np.sqrt(var + eps)
xhat = (xd - mu) * inv
g_ = gamma.data.reshape(1, C)
out = _mk(g_ * xhat + beta.data.reshape(1, C), (x, gamma, beta),
x.requires_grad or gamma.requires_grad or beta.requires_grad)
def _bw():
g = out.grad
if gamma.requires_grad:
gamma._accum((g * xhat).sum(axis=0))
if beta.require
…[truncated 491 chars]The file /app/submission/autograd.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/nn.py
def forward(self, X) -> Tensor:
# TODO: run the stem + blocks, global-average-pool, then the (2-bit) linear head.
raise NotImplementedError("QuantCNN.forward") def forward(self, X) -> Tensor:
h = X if isinstance(X, Tensor) else Tensor(X)
h = self.stem(h)
for b in self.blocks:
h = b(h)
N = h.data.shape[0]
full = h.data.shape[2]
h = ag.avgpool2d(h, full) # (N, C, 1, 1)
h = ag.reshape(h, (N, self.feat)) # (N, C)
wq = self.head_wq.quant(self.head_w)
logits = ag.add(ag.matmul(h, wq), self.head_b)
return logits<tool_use_error>File has been modified since read, either by the user or by a linter. Read it again before attempting to write it.</tool_use_error>
/app/submission/nn.py
318 d: Dict[str, Tensor] = {}
319 for k, v in self.stem.named_params().items():
320 d[f"stem.{k}"] = v
321 for i, b in enumerate(self.blocks):
322 for k, v in b.named_params().items():
323 d[f"blocks.{i}.{k}"] = vLocate remaining forward stubs
grep -n "QuantCNN.forward\|QuantMLP.forward\|QuantSepCNN.forward\|QLinear.forward" nn.py
314: raise NotImplementedError("QuantCNN.forward")
379: raise NotImplementedError("QLinear.forward")
424: raise NotImplementedError("QuantMLP.forward")
815: raise NotImplementedError("QuantSepCNN.forward")/app/submission/nn.py
310 b.set_training(flag)
311
312 def forward(self, X) -> Tensor:
313 # TODO: run the stem + blocks, global-average-pool, then the (2-bit) linear head.
314 raise NotImplementedError("QuantCNN.forward")
315/app/submission/nn.py
def forward(self, X) -> Tensor:
# TODO: run the stem + blocks, global-average-pool, then the (2-bit) linear head.
raise NotImplementedError("QuantCNN.forward") def forward(self, X) -> Tensor:
h = X if isinstance(X, Tensor) else Tensor(X)
h = self.stem(h)
for b in self.blocks:
h = b(h)
N = h.data.shape[0]
full = h.data.shape[2]
h = ag.avgpool2d(h, full)
h = ag.reshape(h, (N, self.feat))
wq = self.head_wq.quant(self.head_w)
return ag.add(ag.matmul(h, wq), self.head_b)The file /app/submission/nn.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/nn.py
384
385 def forward(self, x: Tensor) -> Tensor:
386 # TODO: wq = self.wq.quant(self.weight); return ag.add(ag.matmul(x, wq), self.bias)
387 raise NotImplementedError("QLinear.forward")
388/app/submission/nn.py
# TODO: wq = self.wq.quant(self.weight); return ag.add(ag.matmul(x, wq), self.bias)
raise NotImplementedError("QLinear.forward") wq = self.wq.quant(self.weight)
return ag.add(ag.matmul(x, wq), self.bias)The file /app/submission/nn.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/nn.py
425 def set_training(self, flag: bool): 426 for L in self.layers: 427 L["bn"].training = flag 428 429 def forward(self, X) -> Tensor: 430 # TODO: x -> reshape (N, in_dim); for each layer L: h = L["aq"](relu(L["bn"](L["fc"](h))));
/app/submission/nn.py
429 def forward(self, X) -> Tensor:
430 # TODO: x -> reshape (N, in_dim); for each layer L: h = L["aq"](relu(L["bn"](L["fc"](h))));
431 # then return self.head(h).
432 raise NotImplementedError("QuantMLP.forward")
433/app/submission/nn.py
# TODO: x -> reshape (N, in_dim); for each layer L: h = L["aq"](relu(L["bn"](L["fc"](h))));
# then return self.head(h).
raise NotImplementedError("QuantMLP.forward") h = X if isinstance(X, Tensor) else Tensor(X)
N = h.data.shape[0]
h = ag.reshape(h, (N, self.in_dim))
for L in self.layers:
h = L["aq"](ag.relu(L["bn"](L["fc"](h))))
return self.head(h)The file /app/submission/nn.py has been updated successfully. (file state is current in your context , no need to Read it back)
Fill remaining nn.py forwards
python3 - <<'EOF'
f='nn.py'
s=open(f).read()
repls = [
(' # TODO: return ag.silu(x)\n raise NotImplementedError("SiLU.forward")',
' return ag.silu(x)'),
(' # TODO: return ag.elu(x, self.alpha)\n raise NotImplementedError("ELU.forward")',
' return ag.elu(x, self.alpha)'),
(' # TODO: return ag.groupnorm(x, self.gamma, self.beta, self.num_groups, self.eps)\n raise NotImplementedError("GroupNorm.forward")',
' return ag.groupnorm(x, self.gamma, self.beta, self.num_groups, self.eps)'),
(' # TODO: return ag.tanh(x@Wx + h@Wh + b)\n raise NotImplementedError("RNNCell.forward")',
' return ag.tanh(ag.add(ag.add(ag.matmul(x, self.Wx), ag.matmul(h, self.Wh)), self.b))'),
(' # TODO: implement the GRU update with the named weights above.\n raise NotImplementedError("GRUCell.forward")',
''' z = ag.sigmoid(ag.add(ag.add(ag.matmul(x, self.Wxz), ag.matmul(h, self.Whz)), self.bz))
r = ag.sigmoid(ag.add(ag.add(ag.matmul(x, self.Wxr), ag.matmul(h, self.Whr)), self.br))
n = ag.tanh(ag.add(ag.add(ag.matmul(x, self.Wxn), ag.matmul(ag.mul(r, h), self.Whn)), self.bn))
one_minus_z = ag.sub(1.0, z)
return ag.add(ag.mul(one_minus_z, n), ag.mul(z, h))'''),
(' # TODO: implement the LSTM update; return (h2, c2).\n raise NotImplementedError("LSTMCell.forward")',
''' i = ag.sigmoid(ag.add(ag.add(ag.matmul(x, self.Wxi), ag.matmul(h, self.Whi)), self.
…[truncated 3847 chars]done; missing: [] remaining TODO/NotImpl: 1
/app/submission/data.py
# TODO: cast to float, scale to [0,1], subtract NORM_MEAN, divide by NORM_STD.
raise NotImplementedError("normalize")return (X.astype(np.float64) / 255.0 - NORM_MEAN) / NORM_STD
The file /app/submission/data.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/data.py
# TODO: optionally shuffle an index array, then yield contiguous batch_size slices of it.
raise NotImplementedError("iter_minibatches") n = Xn.shape[0]
idx = np.arange(n)
if shuffle:
rng.shuffle(idx)
for start in range(0, n, batch_size):
sel = idx[start:start + batch_size]
yield Xn[sel], y[sel]The file /app/submission/data.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: SGD(+momentum, +per-group weight decay) update. Iterate self._groups; for each
# group use wd = self.weight_decay if override is None else override; keep a momentum
# buffer per parameter (self._v is indexed in self.params order).
raise NotImplementedError("SGD.step") def step(self):
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._v[i] = self.momentum * self._v[i] + g
p.data -= self.lr * self._v[i]
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: AdamW with bias correction + DECOUPLED, per-group weight decay (see docstring).
raise NotImplementedError("AdamW.step") def step(self):
self.t += 1
t = self.t
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad
self._m[i] = self.b1 * self._m[i] + (1 - self.b1) * g
self._v[i] = self.b2 * self._v[i] + (1 - self.b2) * (g * g)
mhat = self._m[i] / (1 - self.b1 ** t)
vhat = self._v[i] / (1 - self.b2 ** t)
p.data -= self.lr * (mhat / (np.sqrt(vhat) + self.eps) + wd * p.data)
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: Adam with bias correction + COUPLED, per-group weight decay (see docstring).
raise NotImplementedError("Adam.step") def step(self):
self.t += 1
t = self.t
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._m[i] = self.b1 * self._m[i] + (1 - self.b1) * g
self._v[i] = self.b2 * self._v[i] + (1 - self.b2) * (g * g)
mhat = self._m[i] / (1 - self.b1 ** t)
vhat = self._v[i] / (1 - self.b2 ** t)
p.data -= self.lr * mhat / (np.sqrt(vhat) + self.eps)
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: RMSprop update with per-group weight decay (see docstring).
raise NotImplementedError("RMSprop.step") def step(self):
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._s[i] = self.alpha * self._s[i] + (1 - self.alpha) * (g * g)
p.data -= self.lr * g / (np.sqrt(self._s[i]) + self.eps)
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: Adagrad update with per-group weight decay (see docstring).
raise NotImplementedError("Adagrad.step") def step(self):
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._s[i] = self._s[i] + g * g
p.data -= self.lr * g / (np.sqrt(self._s[i]) + self.eps)
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: Lion update with per-group (decoupled) weight decay (see docstring).
raise NotImplementedError("Lion.step") def step(self):
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad
upd = np.sign(self.b1 * self._m[i] + (1 - self.b1) * g)
p.data -= self.lr * (upd + wd * p.data)
self._m[i] = self.b2 * self._m[i] + (1 - self.b2) * g
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: implement the Adamax update honouring per-group weight decay.
raise NotImplementedError("Adamax.step") def step(self):
self.t += 1
t = self.t
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._m[i] = self.b1 * self._m[i] + (1 - self.b1) * g
self._u[i] = np.maximum(self.b2 * self._u[i], np.abs(g))
p.data -= (self.lr / (1 - self.b1 ** t)) * self._m[i] / (self._u[i] + self.eps)
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: implement the NAdam update honouring per-group weight decay.
raise NotImplementedError("NAdam.step") def step(self):
self.t += 1
t = self.t
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._m[i] = self.b1 * self._m[i] + (1 - self.b1) * g
self._v[i] = self.b2 * self._v[i] + (1 - self.b2) * (g * g)
mhat = (self.b1 * self._m[i] / (1 - self.b1 ** (t + 1))
+ (1 - self.b1) * g / (1 - self.b1 ** t))
vhat = self._v[i] / (1 - self.b2 ** t)
p.data -= self.lr * mhat / (np.sqrt(vhat) + self.eps)
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: implement the Adadelta update honouring per-group weight decay.
raise NotImplementedError("Adadelta.step") def step(self):
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._s[i] = self.rho * self._s[i] + (1 - self.rho) * (g * g)
delta = np.sqrt(self._d[i] + self.eps) / np.sqrt(self._s[i] + self.eps) * g
p.data -= self.lr * delta
self._d[i] = self.rho * self._d[i] + (1 - self.rho) * (delta * delta)
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
def step(self):
# TODO: implement the Nesterov-SGD update honouring per-group weight decay.
raise NotImplementedError("NesterovSGD.step") def step(self):
i = 0
for params, override in self._groups:
wd = self.weight_decay if override is None else override
for p in params:
if p.grad is not None:
g = p.grad + wd * p.data
self._v[i] = self.momentum * self._v[i] + g
p.data -= self.lr * (g + self.momentum * self._v[i])
i += 1The file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
# TODO: compute the global L2 norm over all param grads; scale in place if it exceeds max_norm.
raise NotImplementedError("clip_grad_norm") total = 0.0
for p in params:
if p.grad is not None:
total += float((p.grad ** 2).sum())
total = math.sqrt(total)
if total > max_norm:
scale = max_norm / (total + 1e-6)
for p in params:
if p.grad is not None:
p.grad *= scale
return totalThe file /app/submission/optim.py has been updated successfully. (file state is current in your context , no need to Read it back)
/app/submission/optim.py
``base_lr`` over the remaining (total - warmup) steps. Return the LR at ``step``."""
# TODO
raise NotImplementedError("cosine_lr") ``base_lr`` over the remaining (total - warmup) steps. Return the LR at ``step``."""
if warmup > 0 and step < warmup:
return base_lr * (step + 1) / warmup
prog = min(1.0, max(0.0, (step - warmup) / max(1, total - warmup)))
return min_lr + 0.5 * (base_lr - min_lr) * (1 + math.cos(math.pi * prog))<tool_use_error>String to replace not found in file.
String: ``base_lr`` over the remaining (total - warmup) steps. Return the LR at ``step``."""
# TODO
raise NotImplementedError("cosine_lr")</tool_use_error>/app/submission/optim.py
195 class Lion(Optimizer): 196 """Lion (sign of an interpolated momentum; DECOUPLED weight decay): 197 update = sign(b1*m + (1-b1)*g) ; p -= lr*(update + wd*p) ; m = b2*m + (1-b2)*g 198 (the momentum buffer is updated AFTER the param step). 199 """ 200 201 def __init__(self, params, lr=1e-4, betas=(0.9, 0.99), weight_decay=0.0): 202 super().__init__(params) 203 self.lr = lr 204 self.b1, self.b2 = betas 205 self.weight_decay = weight_decay 206 self._m = [np.zeros_like(p.data) for p in self.params] 207 208 def step(self): 209 i = 0 210 for params, override in self._groups: 211 wd = self.weight_decay if override is None else override 212 for p in params: 213 if p.grad is not None: 214 g = p.grad 215 upd = np.sign(self.b1 * self._m[i] + (1 - self.b1) * g) 216 p.data -= self.lr * (upd + wd * p.data) 217 self._m[i] = self.b2 * self._m[i] + (1 - self.b2) * g 218 i += 1 219 220 221 # ===================================================================== LR schedulers 222 def cosine_lr(step, total, base_lr, warmup=0, min_lr=0.0): 223 """Linear warmup for ``warmup`` steps (lr = base_lr*(step+1)/warmup), then cosine decay from 224 ``base_lr`` to ``min_lr`` over the remaining (total - warmup) steps. Return the LR at ``step``.""" 225 # TODO 226 raise Not …[truncated 465 chars]
Fill optim schedulers
python3 - <<'EOF'
f='optim.py'
s=open(f).read()
repls=[
(' ``base_lr`` to ``min_lr`` over the remaining (total - warmup) steps. Return the LR at ``step``."""\n # TODO\n raise NotImplementedError("cosine_lr")',
''' ``base_lr`` to ``min_lr`` over the remaining (total - warmup) steps. Return the LR at ``step``."""
if warmup > 0 and step < warmup:
return base_lr * (step + 1) / warmup
prog = min(1.0, max(0.0, (step - warmup) / max(1, total - warmup)))
return min_lr + 0.5 * (base_lr - min_lr) * (1 + math.cos(math.pi * prog))'''),
(' constant ``base_lr``."""\n # TODO\n raise NotImplementedError("warmup_lr")',
''' constant ``base_lr``."""
if step < warmup:
return base_lr * (step + 1) / warmup
return base_lr'''),
(' """Step decay: base_lr * gamma ** (step // step_size)."""\n # TODO\n raise NotImplementedError("step_lr")',
''' """Step decay: base_lr * gamma ** (step // step_size)."""
return base_lr * (gamma ** (step // step_size))'''),
(' """Exponential decay: base_lr * gamma ** step."""\n # TODO\n raise NotImplementedError("exponential_lr")',
''' """Exponential decay: base_lr * gamma ** step."""
return base_lr * (gamma ** step)'''),
(' """Polynomial decay: min_lr + (base_lr - min_lr) * (1 - step/total)^power (clamped at total)."""\n # TODO\n raise NotImplementedError("polynomial_lr")',
''' """Polynomial decay: min_lr + (base_lr - min_lr) * (1 - step/total)^power (clamped at tot
…[truncated 1303 chars]missing: [] remaining NotImpl: 0
Run all public checks quiet
cd /app && python3 run_checks.py --quiet 2>&1 | tail -20
------------------------------------------------------------------------ autograd_grads 115/115 | quant_grads 21/21 | nn_forward 53/53 | optim 25/25 | integration 6/6 | data 2/2 TOTAL 222/222 public checks passing All public checks pass. Now train qat_model.npz to clear the held-out top-1 threshold (the grader re-checks components with DIFFERENT seeds).
/app/submission/train.py
def train(data_path, out_path, epochs=30, seed=0, n_classes=10):
# TODO: load + normalize the data; build QuantCNN(n_classes, WIDTHS, seed); initialize the LSQ
# step sizes; build the optimizer with weight decay only on the conv/head weights; run the QAT
# loop (forward -> cross_entropy -> backward -> clip_grad_norm -> step) with an LR schedule;
# keep your best epoch; BAKE the quantized weights; save the state_dict to out_path.
raise NotImplementedError("implement the QAT training loop")from nn import W_NEG, W_POS, A_NEG, A_POS # noqa: E402
def _init_lsq_scales(m, Xcalib):
"""Data/weight-driven LSQ step-size init. Weight steps from |w|; activation steps from a
calibration forward (a default 1.0 quantizes the tiny init weights to all-zeros)."""
nps = m.named_params()
for nm, p in nps.items():
if nm.endswith("wq.scale"):
wkey = nm[:-len("wq.scale")] + "weight"
mw = float(np.mean(np.abs(nps[wkey].data)))
p.data[...] = max(2.0 * mw / np.sqrt(max(1, W_POS)), 1e-2)
# calibrate activation quantizers by capturing the pre-quant activation magnitude.
blocks = [m.stem] + list(m.blocks)
captured = {}
def install(idx, aq):
def hook(x, _aq=aq, _idx=idx):
captured[_idx] = float(np.mean(np.abs(x.data)))
return type(_aq).forward(_aq, x)
aq.forward = hook
for idx, blk in enumerate(blocks):
install(idx, blk.aq)
blk.aq.scale.data[...] = 1.0
m.set_training(True)
m(Xcalib)
for idx, blk in enumerate(blocks):
del blk.aq.forward
s = max(2.0 * captured.get(idx, 0.5) / np.sqrt(max(1, A_POS)), 1e-2)
blk.aq.scale.data[...] = s
def _accuracy(m, Xn, y, bs=256):
m.set_training(False)
correct = 0
for i in range(0, len(Xn), bs):
xb = Xn[i:i + bs]
logits = m(xb).data
correct += int((logits.argmax(1) == y[i:i + bs]).sum())
return correct / len(Xn)
def _bake(m):
"""Replace
…[truncated 3007 chars]The file /app/submission/train.py has been updated successfully. (file state is current in your context , no need to Read it back)
Quick 6-epoch training trial
cd /app/submission && timeout 600 python3 train.py --epochs 6 2>&1 | tail -20
No file changes, the agent left the workspace untouched.
=== SyncVals verifier (local re-grade of the agent's own checkpoint) === COMPONENT CONFORMANCE SUITE: 278/278 cases passed (all milestones pass: True) FINAL METRIC: held-out INT2 top-1 = 0.832 >= threshold 0.7 REWARD = 1 (1 iff every component milestone passes AND the sealed held-out metric clears its threshold) NOTE: the in-sandbox training in the 481MB e2b box was OOM-killed (0-byte checkpoint), so the raw in-sandbox reward was an infrastructure artifact. This reward is from a faithful LOCAL re-train of the agent's OWN train.py (24GB, no OOM) with its own default hyperparameters, then re-graded by the real verifier.
Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_eaf14fae26414b7b. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.
Trial trial_eaf14fae26414b7b · verifier authoritative; classifier explanatory.