Model evidence / current era
Recorded agent setup

GPT-6 Astra

Effort: default (unverified) · current era
Harness: Codex · offline terminal harness

Release date: not recorded
Effort mode was not recorded; no equivalence to a named effort is assumed.
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
20.5%macro pass estimate
8eligible benchmark task versions
41valid runs on those tasks
0HARNESS_ERROR · excluded in scope

Macro: 20.5% [0.0%, 50.0%] · Micro: 6/41 · 14.6% [6.9%, 28.4%]. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows.

Domain results

DomainTasks / runsMacro [95% bootstrap]Micro [95% Wilson]Calibration
Cloud operations0 / 0Insufficient task coverage0/0Limited coverage
Data processing / data science8 / 4120.5% [0.0%, 50.0%]6/41 · 14.6% [6.9%, 28.4%]≥5 tasks × ≥5 runs
DeepSWE (long-horizon SWE)0 / 0Insufficient task coverage0/0Limited coverage
Electrical engineering (RTL)0 / 0Insufficient task coverage0/0Limited coverage
Games0 / 0Insufficient task coverage0/0Limited coverage
Machine learning0 / 0Insufficient task coverage0/0Limited coverage
Mechanical engineering0 / 0Insufficient task coverage0/0Limited coverage
Media0 / 0Insufficient task coverage0/0Limited coverage
Operations0 / 0Insufficient task coverage0/0Limited coverage
Other / uncategorised0 / 0Insufficient task coverage0/0Limited coverage
SWE-Marathon0 / 0Insufficient task coverage0/0Limited coverage
Science / STEM0 / 0Insufficient task coverage0/0Limited coverage
Scientific computing0 / 0Insufficient task coverage0/0Limited coverage
Security / cyber0 / 0Insufficient task coverage0/0Limited coverage
Snorkel-reviewed tasks (category unknown)0 / 0Insufficient task coverage0/0Limited coverage
Software engineering0 / 0Insufficient task coverage0/0Limited coverage

Resource coverage

Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.

Recorded resourceMedianObserved / eligible runs
Recorded tokens—0 / 49
Cost (USD)—0 / 49
Agent turns—0 / 49
Duration (seconds)—0 / 49

Task-level evidence

Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.

Task / versionDomainVerifier outcome [95% Wilson]Tier / aggregate scope
skglm-perspective-certificatesVersion 1Data processing / data science2/4 · 50.0% [15.0%, 85.0%] · preliminaryuncalibratedBenchmark version
weak-moment-location-scaleVersion 1Data processing / data science0/5 · 0.0% [0.0%, 43.4%]high-fail (unreviewed)Benchmark version
compact-noise-reflected-divergenceVersion 1Data processing / data science2/2uncalibratedBenchmark version
pyclustering-selective-jointVersion 1Data processing / data science0/7 · 0.0% [0.0%, 35.4%]high-fail (unreviewed)Benchmark version
esda-mh-diffusion-distanceVersion 1Data processing / data science2/2uncalibratedBenchmark version
pyfixest-picchetti-riVersion 1Data processing / data science2/2uncalibratedBenchmark version
augmented-gmm-covarianceVersion 1Data processing / data science1/7 · 14.3% [2.6%, 51.3%]high-fail (unreviewed)Benchmark version
nonconformist-kernel-quantile-pathVersion 1Data processing / data science0/7 · 0.0% [0.0%, 35.4%]high-fail (unreviewed)Benchmark version
lifelines-dr-cosarcVersion 1Data processing / data science0/5 · 0.0% [0.0%, 43.4%]high-fail (unreviewed)Benchmark version
generative-process-two-sampleVersion 1Data processing / data science0/3 · 0.0% [0.0%, 56.1%] · preliminaryuncalibratedBenchmark version
ruptures_graceVersion 1Data processing / data science2/2uncalibratedBenchmark version
dcor-taustar-tiesVersion 1Data processing / data science3/3 · 100.0% [43.9%, 100.0%] · preliminaryuncalibratedBenchmark version