syncrope/babel-messageformat2

Task gallery / DeepSWE (long-horizon SWE) / Version 1

Reference solution (oracle): Pass
No-op resistance check: Pass
QA verdict: Not recorded
Aggregate eligibility: Eligible version

The task

Implement Unicode MessageFormat 2.0 in Babel, per Unicode LDML TR35 Part 9, "MessageFormat", revision 48.2 (CLDR 48.2). That document governs everything not pinned below: the syntax ABNF, the Interchange Data Model, the error taxonomy, expression/markup/option resolution, fallback values, best-variant pattern selection, the Default Bidi Strategy, the functions :string, :number, :integer with all REQUIRED options, th…

Excerpt only; not a complete executable task specification. Task ID task_5716741dd58d4ee5 · version 1. Selection status: not fully recorded.

Results by model and effort

This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.

Current evidence

Model / effort	Verifier result [95% Wilson]	Diagnostic labels / valid	HARNESS_ERROR · excluded	Outcome classes

Legacy evidence

Model / effort	Verifier result [95% Wilson]	Diagnostic labels / valid	HARNESS_ERROR · excluded	Outcome classes
Claude Opus 4.8 · default (unverified)	0/5 · 0.0% [0.0%, 43.4%]	5 / 5	0	GOOD_FAILURE: 5 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0
Claude Haiku 4.5 · default (unverified)	0/2	2 / 2	1	GOOD_FAILURE: 2 BAD_FAILURE: 0 GOOD_SUCCESS: 0 BAD_SUCCESS: 0

Every recorded run

10 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.

Run / model	Date / source	Reward / eligibility	Diagnostic class / source	Resources	Evidence
`trial_0246d18ca3324081` Claude Opus 4.8 · default (unverified)	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: 129,964 Turns: 48 Cost: $7.2837	Not publicly available
`trial_6e04d0ac4cd548ca` Claude Opus 4.8 · default (unverified)	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: 143,889 Turns: 108 Cost: $12.9890	Not publicly available
`trial_78b0148f9aa64a21` Claude Opus 4.8 · default (unverified)	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: 193,456 Turns: 122 Cost: $20.2791	Not publicly available
`trial_874a0de4cff44a96` Claude Opus 4.8 · default (unverified)	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: 175,763 Turns: 125 Cost: $20.0903	Not publicly available
`trial_8f0b97bc43f843ec` Claude Opus 4.8 · default (unverified)	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: 139,934 Turns: 95 Cost: $12.8080	Not publicly available
`trial_0ad653ca38be43ad` oracle@@	Aug 29, 2026 console	1 · valid	GOOD_SUCCESS frozen-classifier	Tokens: — Turns: — Cost: —	Not publicly available
`trial_2b6230d553124d5e` nop@@	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: — Turns: — Cost: —	Not publicly available
`trial_c95370f1b7e547ec` Claude Haiku 4.5 · default (unverified)	Aug 29, 2026 console	0 · excludedharness_error	HARNESS_ERROR frozen-classifier	Tokens: 48,315 Turns: 69 Cost: $0.9193	Not publicly available
`trial_e64502e466094362` Claude Haiku 4.5 · default (unverified)	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: 56,838 Turns: 94 Cost: $1.3253	Not publicly available
`trial_da131d55643d453b` Claude Haiku 4.5 · default (unverified)	Aug 29, 2026 console	0 · valid	GOOD_FAILURE frozen-classifier	Tokens: 31,059 Turns: 55 Cost: $0.7862	Not publicly available