Model evidence / current era
Recorded agent setup
GPT-6 Astra
Effort: medium · current era
Harness: Codex CLI · evaluation harness
Task selection is incompletely recorded; catalog-wide generalization is unsupported.
28.6%macro pass estimate
7eligible benchmark task versions
31valid runs on those tasks
12HARNESS_ERROR · excluded in scope
Macro: 28.6% [0.0%, 57.1%] · Micro: 8/31 · 25.8% [13.7%, 43.2%]. Latest nondefective task versions, each with ≥3 valid runs; a macro needs ≥3 tasks. The model’s measured subset differs from other rows. Empirical bootstrap cannot generate unseen successes or failures in boundary cells; a narrow interval does not establish certainty.
Domain results
| Domain | Tasks / runs | Macro [95% bootstrap] | Micro [95% Wilson] | Calibration |
|---|---|---|---|---|
| Cloud operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Data processing / data science | 7 / 31 | 28.6% [0.0%, 57.1%] | 8/31 · 25.8% [13.7%, 43.2%] | ≥5 tasks × ≥5 runs |
| DeepSWE (long-horizon SWE) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Electrical engineering (RTL) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Games | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Machine learning | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Mechanical engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Media | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Operations | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Other / uncategorised | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| SWE-Marathon | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Science / STEM | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Scientific computing | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Security / cyber | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Snorkel-reviewed tasks (category unknown) | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
| Software engineering | 0 / 0 | Insufficient task coverage | 0/0 | Limited coverage |
Resource coverage
Valid runs on nondefective benchmark versions. Missing values are excluded, never counted as zero. Medians are descriptive and can reflect different task difficulty and harness behavior.
| Recorded resource | Median | Observed / eligible runs |
|---|---|---|
| Recorded tokens | 16,620 | 49 / 49 |
| Cost (USD) | — | 0 / 49 |
| Agent turns | 34 | 49 / 49 |
| Duration (seconds) | 471.2 | 49 / 49 |
Task-level evidence
Includes excluded and superseded versions for inspection; only the chosen nondefective versions enter aggregate results. Each task links to the exact runs and their diagnostic provenance.
| Task / version | Domain | Verifier outcome [95% Wilson] | Tier / aggregate scope |
|---|---|---|---|
| datasciresearch/augmented-gmm-covarianceVersion 1 | Data processing / data science | 1/2 | uncalibratedBenchmark version |
| datasciresearch/augmented-gmm-covarianceVersion 2 | Data processing / data science | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| datasciresearch/augmented-gmm-covarianceVersion 3 | Data processing / data science | 2/2 | uncalibratedBenchmark version |
| compact-noise-reflected-divergenceVersion 1 | Data processing / data science | 2/2 | uncalibratedBenchmark version |
| datasciresearch/dcor-taustar-tiesVersion 1 | Data processing / data science | 3/3 · 100.0% [43.9%, 100.0%] · preliminary | uncalibratedBenchmark version |
| datasciresearch/esda-mh-diffusion-distanceVersion 1 | Data processing / data science | 2/2 | uncalibratedBenchmark version |
| datasciresearch/generative-process-two-sampleVersion 1 | Data processing / data science | 0/3 · 0.0% [0.0%, 56.1%] · preliminary | uncalibratedBenchmark version |
| datasciresearch/gmm-misspecified-covarianceVersion 1 | Data processing / data science | 2/2 | uncalibratedBenchmark version |
| syncrope/lifelines-dr-cosarcVersion 1 | Data processing / data science | 0/0 | uncalibratedBenchmark version |
| syncrope/nonconformist-kernel-quantile-pathVersion 1 | Data processing / data science | 0/2 | uncalibratedBenchmark version |
| syncrope/nonconformist-kernel-quantile-pathVersion 2 | Data processing / data science | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| syncrope/pyclustering-selective-jointVersion 1 | Data processing / data science | 0/2 | uncalibratedBenchmark version |
| syncrope/pyclustering-selective-jointVersion 2 | Data processing / data science | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| datasciresearch/pyfixest-picchetti-riVersion 1 | Data processing / data science | 2/2 | uncalibratedBenchmark version |
| datasciresearch/rlm-sandwich-covarianceVersion 1 | Data processing / data science | 2/2 | uncalibratedBenchmark version |
| syncrope/skglm-perspective-certificatesVersion 1 | Data processing / data science | 0/0 | uncalibratedBenchmark version |
| datasciresearch/weak-moment-location-scaleVersion 1 | Data processing / data science | 0/5 · 0.0% [0.0%, 43.4%] | hardBenchmark version |
| datasciresearch/weak-moment-location-scaleVersion 2 | Data processing / data science | 5/5 · 100.0% [56.6%, 100.0%] | saturatedBenchmark version |