Follow every number.

Snapshot identity

Source snapshot SHA-256:

29b46d5ea53031efee082a7b9ff6f1c4535a37e21efce8924f9b05e6cdf2948f

The source snapshot hash fingerprints the read-only input files. Each sanitized download has its own SHA-256 below; it is not a byte-for-byte copy of its raw input. Derived metrics use a deterministic seed and stable ordering.

• derived.json ↓ 9,740,484 bytesSHA-256 ab789d5d77aafabb5586df8badb1f8c05289eee27de7472e5e5d6e6b45305009
• model-scores.json ↓ 250,147 bytesSHA-256 e5b3511745c7d8c03a2c881ecf6c4bb918434df0e4c3a817dfb85d5b23a54812
• task-catalog.json ↓ 3,224,052 bytesSHA-256 d6a93084a5925d8284acff51a0ec4bded11fadd986d83a481c14c36feffd7763
• run-results.json ↓ 5,730,019 bytesSHA-256 39a4cab93b0624b1b74f0a9c6126de8c6f31c8be59d6fe7cc394b8ddcf19db3d
• figure-data.json ↓ 41,829 bytesSHA-256 8e562f62f58406eb8fce67f74a385cb34e522444a210be7d90f412434c3a2632
• provenance.json ↓ 6,433 bytesSHA-256 29326b78f0237872ac999719abe74d1e9c9e3b8f851fbd5a410771f6ad836945
• manifest.json ↓ 1,021 bytesSHA-256 9cc1dd6dfcbdef16593fc5d329f40873414ff90ba12552a31949314cbe2aef86

Download the public file manifest

Analysis scope

• This is an observational catalog of recorded agent systems, not a balanced experiment or a representative sample of coding work.
• Current and legacy status comes from the frozen snapshot. No external release date or model identity was inferred.
• Aggregate rates use one latest nondefective version per task ID and at least three valid runs per task. Rows with fewer than three eligible tasks show counts only.
• Task names and display keys do not establish identity. Distinct task IDs remain distinct even if their names match.
• Confidence intervals describe sampling uncertainty under the stated assumptions. They do not correct selection bias, missing results, correlated task families, harness changes, or optional stopping.
• Two-stage empirical bootstrap intervals are approximate. Boundary cells cannot generate unobserved outcomes and may produce zero-width intervals; no certainty claim follows.
• The archive has no complete task-selection field. Delivery source marks a known curated proxy, while other tasks have unknown selection status.
• Frozen-classifier provenance is declared by the snapshot; classifier model IDs and prompt hashes are absent, so this build cannot independently attest them.
• Most historical effort modes, execution dates and CLI versions are missing. Unknown values remain unknown.
• One snapshot-valid negative reward is excluded as invalid_reward_range in analysis. The historical inventory retains original snapshot validity and trial-domain assignments for reproducibility.
• Classifiers explain verifier grades. HARNESS_ERROR is excluded under the inherited policy, including classifier inconsistencies; this policy can introduce outcome-dependent missingness.
• Duplicate trial IDs use console > site > delivery > repo precedence. Shared timestamps do not identify duplicate executions; timestamp collisions are retained.
• Pooled-run Wilson intervals are descriptive and assume independent Bernoulli trials. Headline comparisons use task macro averages and paired task resampling.
• Sign-test values are conditional sign diagnostics, not validated tests of equal model capability; unequal repetition counts can violate the fair-sign null. No superiority inference is issued.
• Reviews are already represented in the normalized trial labels. The site hashes the review inventory but does not adjudicate or independently establish reviewer independence.
• Legacy showcase trial evidence URLs have been retired because those routes return 404. Missing public evidence is shown explicitly, not replaced with a proof claim.

Independent arithmetic check

Download the Python verifier. Run it against run-results.json, task-catalog.json, and model-scores.json to independently check disclosed counts, point estimates, Wilson intervals, and quality rules. It does not authenticate original runs, private evidence, or statistical coverage. Instructions and audit scope.

Download the audit record and verification-file hashes.

Validity exclusions

From record to figure

The generator reads the frozen snapshot, validates run eligibility, selects one nondefective benchmark version per task, then calculates per-task intervals and task-weighted aggregates. The static pages and figures consume that same derived data. No model calls, analytics, or external runtime requests are involved.

Private evidence and reviewer identities are excluded from this public projection. Downloadable data does not grant access to private source systems or imply permission to redistribute original task packages.