An honest leaderboard for coding & engineering agents.

Algorithms, data structures, bug-fixes, and API/systems implementation, graded against hidden test suites the agent never sees.

Long-horizon feature-implementation tasks on real open-source repositories (Click, lxml, goja, hono, jiff, pypdf, python-docx), graded on the fail-to-pass and pass-to-pass test transitions each change must produce.

Structural and numerical solvers (FD/FE, contact dynamics) in C++, checked against multi-binary hidden references.

Provision, diagnose, and guardrail cloud infrastructure (AWS), verified against the deployed state.

RTL / digital-logic design checked under hardened simulation and formal-equivalence harnesses.

Quantitative reasoning across math, physics, and the natural sciences with deterministic, checkable answers.

Interactive simulation and game-logic tasks graded on exact state transitions and rule fidelity.

Long-horizon, from-scratch numpy ML research: implement reverse-mode autodiff and a training method (quantization-aware, adversarial-robust, semi-supervised, worst-group), then train to a sealed held-out metric. Tasks published blinded.

Physics-informed and surrogate modeling: PDE forecasting and CFD/FEA prediction, graded on quantitative predictive accuracy.

Modeling and statistical inference on real datasets (federated learning, event studies), graded against held-out ground truth.

Bias-correction and outlier-robust analysis in R: recover trustworthy estimates from messy data, checked against reference results.