Syncvals

A catalog, with context.

Passing a task is one observation. Understanding that result means knowing which task version ran, the agent setup, the grader’s verdict, and whether a later review found a problem with the task itself.

Syncvals brings those records together. This snapshot contains 2,424 task versions and 8,079 valid named-agent runs. It preserves older models alongside current ones without treating different evaluation histories as a controlled comparison.

Coverage before rank.

Each row below is a recorded model and effort setting. Macro pass rate gives equal weight to eligible task versions; the shaded bar shows the point estimate and the whisker its bootstrap interval. Different rows cover different tasks.

Current rows only · latest nondefective version per task · ≥3 runs per task and ≥3 tasks to display a macro · shared core: 0 tasks · no ranking

Model / effort	Agent harness	Tasks / runs	Macro pass rate 95% bootstrap
Claude Opus 5default (unverified)	Claude Code · Harness not recorded	98 / 531	57.9% [48.8%, 66.7%]
GPT-5.6-soldefault (unverified)	Codex · Harness not recorded	24 / 150	36.7% [19.6%, 55.0%]
GPT-6 Astradefault (unverified)	Codex · offline terminal harness	8 / 41	20.5% [0.0%, 50.0%]
GPT-6 Astramedium	Codex CLI · evaluation harness	7 / 31	28.6% [0.0%, 57.1%] Bootstrap cannot generate unseen boundary outcomes.
Claude Opus 5high	Claude Code	3 / 15	26.7% [0.0%, 66.7%]
Tencent HY4 (preview)think	Claude Code	3 / 17	20.0% [0.0%, 60.0%]
Tencent HY4 (preview)high	Claude Code	2 / 16	Insufficient task coverageBootstrap cannot generate unseen boundary outcomes.
Claude Opus 5xhigh	Claude Code	1 / 4	Insufficient task coverageBootstrap cannot generate unseen boundary outcomes.

Default effort is unverified where not recorded. GPT-5.6 Sol has a documented lane-suspect flag. Model identities and era designations follow the supplied snapshot; no new model calls were made.

Read the brackets. Then read the task.

A result from five runs is still uncertain. We show denominators, preserve task versions, and keep infrastructure and classifier exclusions visible. A high failure count becomes evidence of difficulty only when qualifying review or classifier evidence attributes a failure to the agent.

The figures are descriptive views of a retrospective catalog, not estimates of every model’s capability on unseen work. Task selection, harness differences, and missing labels limit the conclusions.

Different models were evaluated on different tasks.

Domain results answer a narrow question: how did this recorded setup perform on the eligible tasks available for it? The matrix makes missing and thinly measured areas visible, rather than smoothing them into a score.

Pass rates on each row’s measured domainsFigure 01 / measured evidence

DeepSWE (long-horizon SWE)

Claude Opus 5default (unverified)

37.8%

27 tasks · 169 runs · 95% bootstrap [22.7%, 54.2%]

GPT-5.6-soldefault (unverified)

36.7%

24 tasks · 150 runs · 95% bootstrap [19.6%, 55.0%]

0%50%100%

Electrical engineering (RTL)

Claude Opus 5default (unverified)

100.0%

3 tasks · 11 runs · 95% bootstrap [100.0%, 100.0%]

0%50%100%

Games

Claude Opus 5default (unverified)

96.0%

10 tasks · 50 runs · 95% bootstrap [88.0%, 100.0%]

0%50%100%

Machine learning

Claude Opus 5default (unverified)

64.6%

26 tasks · 139 runs · 95% bootstrap [47.6%, 80.0%]

0%50%100%

Security / cyber

Claude Opus 5default (unverified)

47.8%

12 tasks · 62 runs · 95% bootstrap [22.8%, 72.8%]

0%50%100%

Software engineering

Claude Opus 5default (unverified)

60.3%

15 tasks · 76 runs · 95% bootstrap [37.9%, 81.7%]

0%50%100%

Data processing / data science

GPT-6 Astradefault (unverified)

20.5%

8 tasks · 41 runs · 95% bootstrap [0.0%, 50.0%]

0%50%100%

Claude Opus 5GPT-5.6-solGPT-6 Astra

Read the figure as a data table (8 observations)

label	Tasks	Runs	Macro [95% bootstrap]	Micro [95% Wilson]
DeepSWE (long-horizon SWE) / Opus 5	27	169	37.8% [22.7%, 54.2%]	65/169 · 38.5% [31.5%, 46.0%]
Electrical engineering (RTL) / Opus 5	3	11	100.0% [100.0%, 100.0%]	11/11 · 100.0% [74.1%, 100.0%]
Games / Opus 5	10	50	96.0% [88.0%, 100.0%]	48/50 · 96.0% [86.5%, 98.9%]
Machine learning / Opus 5	26	139	64.6% [47.6%, 80.0%]	75/139 · 54.0% [45.7%, 62.0%]
Security / cyber / Opus 5	12	62	47.8% [22.8%, 72.8%]	26/62 · 41.9% [30.5%, 54.3%]
Software engineering / Opus 5	15	76	60.3% [37.9%, 81.7%]	43/76 · 56.6% [45.4%, 67.1%]
DeepSWE (long-horizon SWE) / GPT-5.6-sol	24	150	36.7% [19.6%, 55.0%]	46/150 · 30.7% [23.8%, 38.5%]
Data processing / data science / GPT-6 Astra	8	41	20.5% [0.0%, 50.0%]	6/41 · 14.6% [6.9%, 28.4%]

Complex engineering deserves its own view.

Multi-step software tasks can expose failure modes that disappear in a catalog-wide average. Here we isolate the long-horizon software engineering category while keeping the same version and quality gates.

Long-horizon software engineeringFigure 03 / measured evidence

Claude Opus 5default (unverified)

37.8%

27 tasks · 169 runs · 95% bootstrap [22.7%, 54.2%]

GPT-5.6-soldefault (unverified)

36.7%

24 tasks · 150 runs · 95% bootstrap [19.6%, 55.0%]

0%50%100%

Read the figure as a data table (2 observations)

model	domain	Tasks	Runs	Macro [95% bootstrap]	Micro [95% Wilson]
Claude Opus 5 · default (unverified)	DeepSWE (long-horizon SWE)	27	169	37.8% [22.7%, 54.2%]	65/169 · 38.5% [31.5%, 46.0%]
GPT-5.6-sol · default (unverified)	DeepSWE (long-horizon SWE)	24	150	36.7% [19.6%, 55.0%]	46/150 · 30.7% [23.8%, 38.5%]

Unmeasured is not the same as hard.

Only a small part of the full catalog has repeated current-model evidence. A task is calibrated when at least one current row has five valid runs. A hard label additionally needs agent-attributed failure evidence and no known task defect.

Tiers: saturated ≥90%; medium >50% and <90%; hard ≤50% with qualifying evidence; otherwise high-fail (unreviewed). Read the precise rules.

A failed test does not explain the failure.

The verifier decides pass or fail. Diagnostic labels address a separate question: was the observed outcome attributable to the agent or affected by a task or grading problem? The site reads normalized source labels; it does not independently establish review correctness or independence. Reward-only labels never become diagnostic evidence.

Only 1,643 of 8,079 valid named-agent runs have eligible diagnostic labels in the full snapshot. That is a limitation of the evidence, not proof that the remaining runs were sound. “Frozen classifier” provenance is declared in the snapshot; this build does not independently verify its prompt hash.

Effort is part of the result.

Token, cost, and turn records help explain how runs were executed. Their availability differs by source and harness, so missing records never become zeros. These measurements do not establish causal efficiency.

Preserve the past. Compare the overlap.

Older results remain useful evidence. They do not become a fair measure of progress simply by sitting beside newer results. A comparison must use the same task versions and disclose the two model/effort rows, harnesses, dates, and selection rules.

A small set of explicit rules.

A valid run needs a recorded reward and must pass the documented execution and exclusion checks. HARNESS_ERROR and other invalid runs leave both numerator and denominator. A full pass is reward ≥1 within numerical tolerance; fractional rewards count as failures and remain visible on run records.

Per-task pass rates use Wilson 95% intervals. Pooled micro intervals assume independent trials and are descriptive. Macro means use an approximate, fixed-seed two-stage bootstrap over tasks and their runs. Resampling an all-zero or all-one cell cannot generate an unseen outcome, so a narrow boundary interval does not establish certainty. No interval removes selection bias or harness differences.

Rates with one or two runs are counts only. Three or four runs are marked preliminary. Genuine-failure rates use only qualifying diagnostic labels and require sufficient label coverage.

The same tasks. The same question.

A ranked comparison needs at least five shared task versions, each with at least five valid runs for every displayed row. No such all-current-model core exists in this snapshot. Pairwise overlap can still be inspected on the leaderboard page.

• Versioned: only the latest nondefective version of a task enters aggregates.
• Transparent: current and legacy rows remain separate; effort and harness are disclosed.
• Conditional: curated failure sets and sparse review coverage limit generalization.

This is a different measurement problem from Autoresearch Bench’s iterative optimization score. This site borrows its research-report clarity, while retaining verifier outcomes and task-quality evidence specific to this catalog.

Built on careful task work.

Syncrope evaluation team and task authors. We acknowledge the reference site’s editorial approach and the broader evaluation community’s work on standardized comparisons, transparent uncertainty, and reproducible evidence.

Cite the snapshot, not a moving target.

@misc{syncrope2026syncvals,
  title  = {Syncvals},
  author = {Syncrope},
  year   = {2026},
  url    = {https://syncvals.syncrope.com/},
  note   = {Snapshot 2026-10-23}
}

The citation points to the public research site. Include the snapshot date and content hash when reusing results.