Task gallery / DeepSWE (long-horizon SWE) / Version 1
Task evidence / console

syncrope/babel-messageformat2

hardCurrent calibratedChosen benchmark version
Reference solution (oracle)
Pass
No-op resistance check
Unverified
QA verdict
Not recorded
Aggregate eligibility
Eligible version

The task

Implement Unicode MessageFormat 2.0 in Babel, per Unicode LDML TR35 Part 9, "MessageFormat", revision 48.2 (CLDR 48.2). That document governs everything not pinned below: the syntax ABNF, the Interchange Data Model, the error taxonomy, expression/markup/option resolution, fallback values, best-variant pattern selection, the Default Bidi Strategy, the functions :string, :number, :integer with all REQUIRED options, th…

Excerpt only; not a complete executable task specification. Task ID task_23152389f9124dcb · version 1. Selection status: not fully recorded.

Results by model and effort

This exact task version only. Every rate is passes / valid runs with a Wilson 95% interval; n≤2 remains counts only. Macro aggregation does not apply within a single task. Baselines are summarized in the quality gates, not ranked. A passed no-op resistance check means that doing nothing did not solve the task.

Current evidence

Model / effortVerifier result [95% Wilson]Diagnostic labels / validHARNESS_ERROR · excludedOutcome classes
Claude Opus 5 · default (unverified)2/5 · 40.0% [11.8%, 76.9%]5 / 50
GOOD_FAILURE: 3
BAD_FAILURE: 0
GOOD_SUCCESS: 2
BAD_SUCCESS: 0
GPT-5.6-sol · default (unverified)0/5 · 0.0% [0.0%, 43.4%]5 / 50
GOOD_FAILURE: 5
BAD_FAILURE: 0
GOOD_SUCCESS: 0
BAD_SUCCESS: 0

Legacy evidence

Model / effortVerifier result [95% Wilson]Diagnostic labels / validHARNESS_ERROR · excludedOutcome classes
Gemini Flash (latest alias) · default (unverified)0/11 / 10
GOOD_FAILURE: 1
BAD_FAILURE: 0
GOOD_SUCCESS: 0
BAD_SUCCESS: 0

Difficulty, with uncertainty

Selected-best difficulty: 60.0% [23.1%, 88.2%]. This is one minus the best measured current-row pass rate among rows with at least five valid runs.

The maximum is selected from the observed rows, so the estimate is optimistic. Its Wilson interval is not adjusted for model selection. “Hard” requires qualifying GOOD_FAILURE evidence; the threshold flag additionally requires the interval and recorded quality gates to meet the documented rule. This is not independent validation. Threshold flag: no.

Every recorded run

16 recorded trials, including controls. Diagnostic labels do not replace verifier reward. Labels and quality checks are source declarations, not independently attested here. Internal evidence remains private; a missing public URL is shown explicitly. Records with invalid or absent rewards are not scored.

Run / modelDate / sourceReward / eligibilityDiagnostic class / sourceResourcesEvidence
trial_4356abb9dada4802
oracle@@		Aug 29, 2026
console		1 · excludedsource_evidence_exclusionNo diagnostic label
derived		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_73535b60f5fc415f
Claude Opus 5 · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: 149,954
Turns: 71
Cost: $11.5874		Not publicly available
trial_7d136b0c50954f95
Claude Opus 5 · default (unverified)		Sep 1, 2026
console		1 · validGOOD_SUCCESS
frozen-classifier		Tokens: 174,793
Turns: 128
Cost: $19.8720		Not publicly available
trial_a0320acfbcf242f7
Claude Opus 5 · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: 153,649
Turns: 111
Cost: $17.0553		Not publicly available
trial_cff94d5730c6465b
Claude Opus 5 · default (unverified)		Sep 1, 2026
console		1 · validGOOD_SUCCESS
frozen-classifier		Tokens: 186,485
Turns: 122
Cost: $21.1239		Not publicly available
trial_f35df7c5ba844224
Claude Opus 5 · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: 172,712
Turns: 120
Cost: $21.5633		Not publicly available
trial_5fb1b100be8d4f01
Gemini Flash (latest alias) · default (unverified)		Sep 1, 2026
console		0 · excludedsource_evidence_exclusionGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_7d5edd02ba7b4198
Gemini Flash (latest alias) · default (unverified)		Sep 1, 2026
console		0 · excludedsource_evidence_exclusionGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_9e712b1fd6a942c3
Gemini Flash (latest alias) · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_a080757d78ae41b1
Gemini Flash (latest alias) · default (unverified)		Sep 1, 2026
console		0 · excludedsource_evidence_exclusionNo diagnostic label
No diagnostic source		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_d39c220191d849e4
Gemini Flash (latest alias) · default (unverified)		Sep 1, 2026
console		0 · excludedsource_evidence_exclusionNo diagnostic label
No diagnostic source		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_5bd77c1a3ad94db8
GPT-5.6-sol · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_87c5f00299bc4abb
GPT-5.6-sol · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_91502462e8094ac5
GPT-5.6-sol · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_b9362c372bbd4404
GPT-5.6-sol · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available
trial_bfed719520514f56
GPT-5.6-sol · default (unverified)		Sep 1, 2026
console		0 · validGOOD_FAILURE
frozen-classifier		Tokens: —
Turns: —
Cost: —		Not publicly available