What a score means.

Analysis v1.2 · source snapshot October 23, 2026 · methods audited October 24, 2026

These are the methods used by the current site. They replace the earlier page, which mixed an intended specification with implemented behavior. Reproducing a calculation is different from proving that the original run, grader, or diagnostic label was correct.

Definitions · Validity · Uncertainty · Diagnostic labels · Task tiers · Aggregation · Provenance · Audit and verification

What is established—and what is not

Evidence level	What this release supports
Recorded data	13,084 source trial records across 2,424 task versions and 2,408 task IDs. These are records in the supplied snapshot, not newly rerun experiments.
Reproducible calculations	Run counts, pass counts, per-task Wilson intervals, task selection, macro/micro point estimates, and label summaries can be checked against the sanitized downloads. A separate checker is linked below.
Diagnostic labels	1,643 of 8,079 analytically valid named-agent runs have an eligible diagnostic-source declaration. That declaration alone does not prove the diagnosis, a reviewer’s independence, or human authorship.
Incomplete provenance	Classifier model IDs and prompt hashes are absent. Original execution dates are absent for 6,882 valid named-agent runs; effort settings are absent for 7,885. Model identities and era labels are taken from the snapshot.
Statistical interpretation	Intervals are conditional summaries under stated assumptions. No balanced experiment, representative sampling, calibrated population-wide coverage, or causal model-progress claim is established.
Ranking	The common core across all eight current model/effort rows is empty. There is no global model ranking.

The source snapshot contains 11,300 records marked valid, including controls and unrecorded models. The analysis excludes one reward of −1 as an invalid grade, leaving 11,299 valid records overall and 8,079 valid named-agent runs before version and quality restrictions. These counts describe different scopes and must not be interchanged.

0. Definitions in one screen

Term	Current implementation
Run	One recorded trial ID linked to one task ID and version.
Model row	One model ID and effort value. Missing effort is shown as “default (unverified)”; it is not assumed equivalent to a named effort.
Valid run	A record passing the checks in Section 1. This is analytical eligibility, not an independent attestation of execution.
Pass	A finite, eligible reward of at least 1 − 1e-9. Fractional rewards below that threshold count as failures for pass rate.
Task pass rate	Passes divided by valid runs for one exact task version and model row.
Macro	Equal-weight mean of eligible per-task pass rates.
Micro	Pooled passes divided by pooled valid runs on the same eligible tasks.
Calibrated task	At least five valid runs on at least one current model row. This name denotes repetition, not certification or statistical calibration.
Benchmark version	Latest version of a task ID that does not meet the recorded quality-exclusion rules.
Diagnostic coverage	Eligible diagnostic labels divided by valid runs in the stated scope.
Historical inventory	A separate archive reproduction that retains original validity, original trial domains, and multiple versions; it is not the quality-gated benchmark.

1. Units and validity

1.1 Identity and duplicates

Task identity is (task_id, version). Names and display keys do not establish that two tasks are equivalent. Runs on different versions are kept separate.

The generator deduplicates identical trial_id values, preferring console, then site, delivery, and repository sources. It does not deduplicate by timestamp tuples: the snapshot contains 185 valid timestamp-collision groups, and shared batch timestamps do not prove duplicate execution. This snapshot has no repeated trial IDs removed by that rule. Records already marked as source duplicates can still be excluded by the validity rules.

1.2 Eligibility checks, in order

1. Exclude an absent reward.
2. Exclude non-finite rewards, rewards below zero, and rewards above 1 + 1e-9. Values outside that range are not silently clamped.
3. Exclude HARNESS_ERROR under the inherited analysis policy.
4. Exclude known nonpositive output-token counts; where tokens_out is absent, the code uses the generic tokens field. A positive generic count does not itself establish positive output.
5. Exclude an explicitly recorded infrastructure exception in the configured list: agent_not_found, sandbox_failure, provider_truncation, api_error, setup_timeout, nonzero_agent_exit_no_output, identity_incident, or address_pool_exhaustion.
6. Exclude a recorded timeout when the task explicitly has budget_stated == false.
7. Honor valid == false when both a reason and a nonempty evidence reference are recorded. A reference being present is not proof that its contents were independently inspected.
8. Exclude a run that cannot be linked to a recorded task version.

The exception, timeout, and budget fields in steps 5–6 are absent from this snapshot. Missing values remain unknown; this build cannot reconstruct those checks from absent metadata. Existing source exclusions are retained as recorded. No supplied invalid flag was re-enabled in this snapshot.

Invalid runs leave both numerator and denominator. BAD_FAILURE and BAD_SUCCESS do not automatically invalidate a run; they can instead exclude the task version from aggregate analysis under Section 4.

HARNESS_ERROR is a mixed inherited bucket. It may include infrastructure failures or reward/classifier inconsistencies. The snapshot lacks a reliable subtype field, so this site does not claim a verified split or an implemented review queue. Excluding completed but inconsistently classified runs can produce outcome-dependent missingness.

1.3 Grades and partial credit

The site consumes the recorded verifier reward. It does not rerun the verifier or independently establish that a grade matches the actual solution. Diagnostics do not rewrite the reward.

There are 159 analytically valid fractional-reward records in this snapshot. They remain available in the per-run tables and downloads and count as failures for the binary pass rate. A separate mean-reward chart and mean-reward confidence interval are not implemented.

All recorded valid runs within the selected scope count, including later top-ups. The build does not establish that the snapshot contains every original attempt, and it does not correct for optional stopping or outcome-driven selection of which runs were retained.

2. Pass rate and uncertainty

2.1 Per-task Wilson intervals

For k passes in n valid runs, the point estimate is p̂ = k/n. The implementation uses the Wilson score interval with z = 1.959963984540054, without continuity correction:

centre = (p̂ + z²/(2n)) / (1 + z²/n)
half   = z × sqrt(p̂(1−p̂)/n + z²/(4n²)) / (1 + z²/n)
interval = [max(0, centre−half), min(1, centre+half)]

Independent numerical tests check the score-equation inversion and published reference values. For example, 0/5 gives approximately [0%, 43.4%], and 5/5 gives [56.6%, 100%]. Five observed failures therefore do not establish that a task is impossible.

Wilson intervals assume binomial trials. Correlated attempts, heterogeneous setups, task selection, and missing outcomes can invalidate a broader interpretation. Pooled micro intervals additionally ignore variation and dependence across tasks. The formula is described by NIST; use of a standard formula is not validation of this dataset’s assumptions.

2.2 Display thresholds

• No valid runs: no rate.
• One or two valid runs per task/model cell: counts only, without a percentage or interval.
• Three or four: rate and Wilson interval, marked preliminary.
• At least five: eligible for the repetition-based tier rules.
• Aggregate macro/micro rates require at least three task versions with at least three valid runs each.

A calibrated aggregate has at least five tasks with five valid runs each. Its displayed score still uses all eligible tasks with at least three runs; the badge does not restrict the score to a five-run-only subset. Historical archive tables separately reproduce descriptive point estimates and are labelled accordingly.

2.3 Macro and paired bootstrap summaries

Macro uncertainty uses 2,000 replicates for both pages and downloads, with seed 20261023. Task cells are ordered lexicographically by the canonical task_id@vversion key before resampling; the snapshot’s keys are ASCII. The generator uses unsigned 32-bit random state, resamples eligible task cells with replacement, then resamples binary outcomes within each selected cell. For a cell with k/n, the latter is implemented as a binomial draw with probability k/n. It reports sorted replicate values at zero-based indices 50 and 1,949 as the empirical 2.5th and 97.5th percentile endpoints.

A paired comparison uses the same selected task versions for both rows, but resamples each row’s runs independently within a task. It requires at least five shared versions with at least three valid runs per row. This is an exploratory threshold; the proposed ranked common core requires five runs per row.

These are approximate bootstrap summaries, not demonstrated 95% coverage for this catalog. When all observed outcomes in a cell are failures, or all are successes, resampling cannot generate the opposite outcome. A zero-width interval is possible and is not certainty. The method also does not correct for nonrandom task selection, related task families, differing harnesses, missing results, or optional stopping. Paired evaluation principles and the importance of the sampling design are discussed in Adding Error Bars to Evals.

2.4 Sign diagnostics and comparisons

The export retains a two-sided binomial sign diagnostic from the per-task win/loss directions, ignoring ties; fewer than six non-tied tasks suppress it. Holm adjustment covers all testable pairs in the exported snapshot, including legacy pairs, not just the rows currently displayed. There are two such tests in this snapshot.

The binomial arithmetic is checked, but its model-comparison interpretation is not established. It assumes independent fair signs under its null. Unequal run counts can violate that assumption even when two models have equal underlying success probabilities. A significant sign diagnostic is therefore not presented as proven model superiority, and no automatic “supported” conclusion is issued.

A difference or an overlapping interval is not proof of equivalence either. Shared-task observations remain descriptive unless the evaluation design and inference assumptions are separately justified.

3. Genuine-failure rate and the five classes

The following are stored diagnostic categories, not facts independently established by this site:

Class	Intended meaning in the source taxonomy	Treatment
HARNESS_ERROR	Execution failure or inherited classifier inconsistency	Excluded from valid-run rates; counted separately.
GOOD_FAILURE	Failure attributed to the agent rather than the task	Eligible diagnostic count when its source declaration qualifies.
BAD_FAILURE	Failure attributed to a task, environment, instruction, or grader problem	Run reward remains unchanged; can trigger a task-quality exclusion.
GOOD_SUCCESS	Success considered legitimate by the source diagnosis	Pass/fail still follows the recorded reward; not held-out generalization evidence.
BAD_SUCCESS	Success considered illegitimate or task-compromised	Does not rewrite reward; can trigger a task-quality exclusion.

Valid-run outcome charts show the four non-HARNESS_ERROR categories. Excluded HARNESS_ERROR counts are separate. Missing or derived labels are not silently assigned to a diagnostic class.

3.1 Eligible labels and coverage

Only a recognized class with label_source declared as frozen-classifier or independent-review on a valid run qualifies. derived labels never establish genuine failure or a hard tier.

GFR = GOOD_FAILURE / eligible diagnostic labels
label coverage = eligible diagnostic labels / valid runs

GFR is exported only with at least three eligible labels and at least 80% label coverage. It describes that labelled subset; it does not establish the composition of unlabelled runs. GFR-specific Wilson intervals and a task-fault-share UI are not implemented.

3.2 What the review records establish

The generator uses the normalized labels already present on trials. It hashes the separate review inventory for provenance but does not implement a new review/classifier precedence merge, adjudicate disagreements, or establish reviewer independence.

The audit found that all 249 trials declared independent-review have a matching linked outcome-bearing review record. Their stored categories agree after mapping the review verdict vocabulary. This establishes record consistency, not human authorship, independence, or correctness. Structured reviewer-independence fields are absent.

Classifier model IDs and prompt hashes are absent, so the frozen-classifier designation is a source declaration. There are no verified C/R provenance badges or public disagreement counts. A source label is not a certificate.

4. Task-level tiers, badges, and difficulty

4.1 Quality exclusions and tiers

A task version is excluded from aggregates if any of these recorded conditions holds:

• Nonempty known-defect list, failed oracle control, passed no-op control, or QA verdict REJECT, REVISE, BROKEN, or NOT READY (case and separator normalization applied).
• At least three eligible labelled current-model failures, with at least half classified BAD_FAILURE.
• At least one eligible current BAD_SUCCESS declared independent review, or two eligible current frozen-classifier BAD_SUCCESS records.

Unknown controls do not automatically exclude a version. The site reports their missing status. These rules identify recorded concerns; absence of a recorded concern is not proof that a task is defect-free.

Among current rows with at least five valid runs, the strongest observed row is selected by pass rate, then run count, then Wilson lower bound.

Tier	Implemented rule
Saturated	Selected row pass rate at least 90%.
Hard	Selected row pass rate at most 50%, at least one eligible current failure labelled GOOD_FAILURE, and no quality exclusion.
High-fail (unreviewed)	Selected row pass rate at most 50%, without the qualifying diagnostic declaration.
Medium	Selected row pass rate above 50% and below 90%.
Uncalibrated	No current row reaches five valid runs; includes thin current evidence.
Legacy-only	Valid named-agent evidence exists only on legacy rows.
Contested/defective	Any quality-exclusion rule holds; takes precedence over the other tiers.

The exported confirmed flag is a mechanical rule: saturated additionally needs Wilson lower bound ≥70%; hard additionally needs Wilson upper bound ≤50%, oracle pass, and no-op fail. It is not independent confirmation of the run or diagnosis. The task page labels it as a threshold flag.

4.2 Difficulty score

Difficulty is 1 − selected current pass rate, with interval [1−Wilson high, 1−Wilson low]. It is undefined for contested, uncalibrated, and legacy-only versions.

Selecting the best observed row is optimistic. Its ordinary interval is not adjusted for that selection. This is a conditional description of the evaluated rows, not a population difficulty estimate. The earlier page’s promised fixed-reference-model difficulty view is not implemented. No Elo, Bradley–Terry, Rasch, or latent-capability score is used.

5. Aggregation

Each model row uses the latest nonexcluded version per task ID, ordered numerically by version. Earlier versions remain inspectable and are not pooled into aggregates. A newer excluded version can therefore leave an older eligible version selected.

Let T_m be that row’s task versions with at least three valid runs:

macro = mean(k_t / n_t for t in T_m)
micro = sum(k_t for t in T_m) / sum(n_t for t in T_m)

Both use the same task set. At least three such tasks are needed to display an aggregate rate. Coverage is eligible task count divided by eligible benchmark versions in the chosen domain or catalog. This is not coverage of all possible work.

Current and legacy rows have separate displays and their own eligible task sets. Legacy rows have descriptive aggregates, but never contribute to current task tiers or the all-current common core. The current common core contains zero tasks, so no global ranks are assigned.

Harness names are disclosed when recognized. Unknown harnesses, absent CLI versions, absent release dates, and default effort remain unknown. Different setups are not assumed equivalent.

6. Provenance and honesty

6.1 Snapshot and public downloads

The data page lists the snapshot fingerprint, per-file SHA-256 values, source dates, and validity exclusions. The source fingerprint covers the supplied input files and supports checking that the same bytes were used; it does not establish the truth of their contents. A stable hash does not demonstrate execution, prevent all duplication, or certify a grader.

Public downloads contain sanitized run records, task metadata, scores, and figure data. Private review locations, reviewer identities, and raw operational text are not published. Public readers can reproduce the disclosed arithmetic while some underlying source judgments remain unauditable publicly.

The audit found 1,018 inherited links to retired showcase trial pages. Representative endpoints returned 404 and those routes are absent from this build. They have been removed from evidence links; the site now says the evidence is not publicly available. A sanitized result row is not a replacement for a trajectory or review document.

6.2 Selection and historical scope

Explicit selection metadata is absent. Delivery source is used as a proxy for a known curated cohort; other selection histories remain unknown. Gallery filters can exclude known curated tasks from the gallery, not recompute the aggregate tables. Removing known curated records does not establish an unbiased sample.

The historical inventory retains snapshot validity, multiple task versions, pooled effort modes, and original trial-domain assignments to reproduce earlier tables. It includes the one out-of-range reward retained by the source. It is labelled an archive audit and is never a model leaderboard.

6.3 Limits on conclusions

This release does not establish SOTA, population-wide model capability, causal progress between model generations, universal task hardness, grader correctness, human-only review, or complete failure attribution. No new model calls or trial reruns were performed for this website audit. Source records tagged “measured” are recorded observations, not independent attestations.

7. How to read this bench

Start with the exact task version, passes/valid runs, and the displayed scope. Read label coverage and missing provenance before interpreting a diagnostic class. Treat task and model comparisons as descriptions of the recorded subset.

Verify the arithmetic yourself

Download run-results.json, task-catalog.json, and model-scores.json from the data page, and the independent Python verification script. Put the three JSON files in one folder, then run:

python3 verify-scores.py /path/to/downloaded-data

The checker separately recomputes point/count/Wilson statistics from disclosed records and compares them with the published values. It accepts the published validity decisions, quality flags, model-era assignments, and label-source declarations as inputs; agreement does not independently validate those judgments. It does not authenticate original execution, validate private evidence, or establish bootstrap coverage. The generator also has numerical regression tests and deterministic rebuild checks; neither substitutes for reviewing the original experiments.

For a full numerical replay, also download figure-data.json and run:

python3 verify-scores.py /path/to/downloaded-data --bootstrap

This also reproduces macro and paired bootstrap endpoints, sign diagnostics, and Holm adjustment. The audit record and file hashes describe the verification scope and data limitations. The audit completed 83,006 comparisons across 1,572 task-model objects, 238 model/domain scopes, and 12 paired comparisons. Reproducing these values does not establish the statistical assumptions or the truth of the source judgments.

October 24 correction record

• Replaced intended-but-unimplemented feature and provenance promises with current behavior.
• Corrected the bootstrap generator’s integer wrap, added a long-sequence numerical regression, and made task-cell ordering deterministic. These changes affect some bootstrap endpoints; they do not add new trial observations.
• The canonical ordering correction changes 58 macro interval endpoints across 33 aggregate objects (maximum 2.68 percentage points). The current 19-task pair now has an approximate interval of [−7.89, +28.30] percentage points, around the same +9.36-point estimate. Counts and denominators are unchanged; tiny floating-point rounding differences do not change the displayed point scores.
• Fixed sign-diagnostic underflow for very large samples; current snapshot sign values are unchanged.
• Removed automatic superiority conclusions and documented the conditional sign assumptions.
• Retired broken legacy evidence URLs and preserved missing evidence explicitly.
• Documented the actual 2,000-replicate export convention, trial-ID deduplication, and snapshot-wide Holm family.

This is a reproducible snapshot analysis with incomplete evaluation provenance. Stronger claims require original execution artifacts, documented review independence, grader/control validation, and a comparison design suited to the claim.