tests/test.sh exit code (0 → resolved); the classification below is post-hoc and cannot change it./home/user/instruction.md
1 # Ticket: Make the resilient HTTP client's policies co-evolve in `ResilientClient.send` 2 3 ## Context 4 5 `resilient-http` is the library the **checkout service** uses to call its flaky 6 downstream dependencies (TypeScript, Node 20). It already ships the building 7 blocks , an injectable `Clock`, a backoff calculator, a three-state 8 `CircuitBreaker`, a `Bulkhead` (AIMD + hard cap), a `RetryBudget` (token bucket), 9 typed errors, `runAttempt` (a single attempt under a timeout), and `runHedged` 10 (request hedging with deadline-aware suppression). They are **provided complete**. 11 12 What is missing is the hard part: an orchestrator that makes the circuit breaker, 13 retry budget, deadline, and hedging **co-evolve** across the retry/hedge loop, so 14 that an attempt's outcome changes what later iterations are even allowed to do. 15 16 **Only `ResilientClient.send(req, ctx)` is unimplemented.** 17 18 ## Your task 19 20 Implement `src/resilientClient.ts → ResilientClient.send`. The authoritative spec 21 is the JSDoc directly above the `send` stub , read it; it pins every rule exactly. 22 The shape: 23 24 - **Once per call:** deposit to the retry budget, then acquire a bulkhead slot 25 (shed with `BulkheadRejectedError` and no transport call if refused). Hold the 26 slot for the whole call and release it **exactly once** , `recordSuccess()` on a 27 terminal success, `recordFailure()` on any terminal failure (including a 28 short-circuit). That release drives AIMD. 29 30 - **Per loop iteration, in a pinned precedence:** deadline first (a request past 31 its deadline is dead regardless of the breaker), then breaker admission. Run the 32 attempt with the timeout set to `min(timeoutMs, deadline - now)` (the configured 33 timeout, never extended past the remaining deadline budget) via 34 `this.runHedged(...)`. Report the outcome to the breaker. 35 36 - **On a failed attempt, in a pinned precedence:** stop on max-retries, else gate 37 the retry on the budget, else suppress the retry when `now + delay >= deadline` 38 (reaching the deadline counts , do not sleep up to it only to fail); otherwise 39 sleep the jittered backoff and loop. The exact error class and `reason` for each 40 stop are in the JSDoc. 41 42 - **Retry-budget co-evolution:** every retry and every hedge withdraws one whole 43 token; a call that **ultimately succeeds refunds** every token it withdrew (a 44 productive retry is not storm load), while a call that ultimately fails keeps 45 them debited. Integer tokens only. 46 47 Use the provided pieces; keep the public exports. With the defaults (no deadline, 48 hedging off, generous bulkhead/budget) behaviour must stay **backward-compatible** 49 with a plain retry + circuit-breaker client. Everything is deterministic: route 50 all time and randomness through the injected `Clock` (the tests drive a 51 `ManualClock` and an injected RNG; never read real time or `Math.random`). 52 53 ## Definition of done 54 55 - `npm run typecheck` is clean. 56 - `npm test` passes. The shipped image contains only a smoke test that pins the 57 public surface plus the provided primitive unit tests; the behavioural suite is 58 applied at grading time. Implement the feature entirely within 59 `ResilientClient.send` , do not change the tests, the primitives, the clock, the 60 errors, `runAttempt`, or `runHedged`. 61 62 ## Running locally 63 64 ```bash 65 npm install # already done in the provided environment 66 npm run typecheck 67 npm test 68 ``` 69
/home/user/app/src/resilientClient.ts
1 import { CircuitBreaker } from "./circuitBreaker.js";
2 import { backoffDelay } from "./backoff.js";
3 import { Bulkhead } from "./bulkhead.js";
4 import { RetryBudget } from "./retryBudget.js";
5 import { systemClock } from "./clock.js";
6 import type { Clock } from "./clock.js";
7 import {
8 AttemptTimeoutError,
9 BulkheadRejectedError,
10 CircuitOpenError,
11 DeadlineExceededError,
12 RetriesExhaustedError,
13 } from "./errors.js";
14 import type {
15 HttpRequest,
16 HttpResponse,
17 RequestContext,
18 ResilienceOptions,
19 ResolvedResilienceOptions,
20 Transport,
21 } from "./types.js";
22
23 const DEFAULTS: ResolvedResilienceOptions = {
24 maxRetries: 2,
25 baseDelayMs: 50,
26 maxDelayMs: 2000,
27 failureThreshold: 5,
28 openDurationMs: 30_000,
29 halfOpenMaxAttempts: 1,
30 timeoutMs: 1000,
31 isRetryableStatus: (status: number) => status >= 500 || status === 429,
32 maxConcurrency: 100,
33 initialConcurrencyLimit: 100,
34 minConcurrencyLimit: 1,
35 concurrencyAdditiveIncrease: 1,
36 concurrencyDecreaseFactor: 0.5,
37 retryBudgetRatio: 0.2,
38 retryBudgetMaxTokens: 10,
39 hedgeDelayMs: 0,
40 maxHedges: 1,
41 };
42
43 export function resolveOptions(options: ResilienceOptions = {}): ResolvedResilienceOptions {
44 const maxConcurrency = options.maxConcurrency ?? DEFAULTS.maxConcurrency;
45 return {
46 maxRetries: options.maxRetries ?? DEFAULTS.maxRetries,
47 baseDelayMs: options.baseDelayMs ?? DEFAULTS.baseDelayMs,
48 maxDelayMs: options.maxDelayMs ?? DEFAULTS.maxDelayMs,
49 failureThreshold: options.failureThreshold ?? DEFAULTS.failureThreshold,
50 openDurationMs: options.openDurationMs ?? DEFAULTS.openDurationMs,
51 halfOpenMaxAttempts: options.halfOpenMaxAttempts ?? DEFAULTS.halfOpenMaxAttempts,
52 timeoutMs: options.timeoutMs ?? DEFAULTS.timeoutMs,
53 isRetryableStatus: options.isRetryableStatus ?? DEFAULTS.isRetryableStatus,
54 maxConcurrency,
55 initialConcurrencyLimit: options.initialConcurrencyLimit ?? maxConcurrency,
56 minConcurrencyLimit: options.minConcurrencyLimit ?? DEFAULTS.minConcurrencyLimit,
57 concurrencyAdditiveIncrease: options.concurrencyAdditiveIncrease ?? DEFAULTS.concurrencyAdditiveIncrease,
58 concurrencyDecreaseFactor: options.concurrencyDecreaseFactor ?? DEFAULTS.concurrencyDecreaseFactor,
59 retryBudgetRatio: options.retryBudgetRatio ?? DEFAULTS.retryBudgetRatio,
60 retryBudgetMaxTokens: options.retryBudgetMaxTokens ?? DEFAULTS.retryBudgetMaxTokens,
61 hedgeDelayMs: options.hedgeDelayMs ?? DEFAULTS.hedgeDelayMs,
62 maxHedges: options.maxHedges ?? DEFAULTS.maxHedges,
63 };
64 }
65
66 /**
67 * Run a single transport attempt under a per-attempt timeout. Resolves with the
68 * transport's response; rejects with {@link AttemptTimeoutError} if the timeout
69 * fires first (aborting the transport), or with whatever the transport rejected
70 * with. An optional external `abort` signal lets a hedge winner cancel the
71 * losers. Provided complete.
72 */
73 export async function runAttempt(
74 transport: Transport,
75 req: HttpRequest,
76 clock: Clock,
77 timeoutMs: number,
78 abort?: AbortSignal,
79 ): Promise<HttpResponse> {
80 const transportCtl = new AbortController();
81 const timerCtl = new AbortController();
82
83 const onExternalAbort = (): void => transportCtl.abort();
84 if (abort) {
85 if (abort.aborted) transportCtl.abort();
86 else abort.addEventListener("abort", onExternalAbort, { once: true });
87 }
88
89 const result = await Promise.race([
90 transport(req, transportCtl.signal).then(
91 (res) => ({ kind: "res" as const, res }),
92 (err) => ({ kind: "err" as const, err }),
93 ),
94 clock.sleep(timeoutMs, timerCtl.signal).then(() => ({ kind: "timeout" as const })),
95 ]);
96
97 if (abort) abort.removeEventListener("abort", onExternalAbort);
98
99 if (result.kind === "timeout") {
100 transportCtl.abort();
101 throw new AttemptTimeoutError(timeoutMs);
102 }
103 timerCtl.abort();
104 if (result.kind === "err") throw result.err;
105 return result.res;
106 }
107
108 interface AttemptOutcome {
109 ok: boolean;
110 res?: HttpResponse;
111 err?: unknown;
112 /** EXTRA hedge attempts launched (and debited from the budget) for this attempt. */
113 hedgesSpent: number;
114 }
115
116 /**
117 * A resilient HTTP client. See the original stub for the full `send` contract.
118 */
119 export class ResilientClient {
120 readonly options: ResolvedResilienceOptions;
121 private readonly breaker: CircuitBreaker;
122 private readonly bulkhead: Bulkhead;
123 private readonly retryBudget: RetryBudget;
124 private readonly clock: Clock;
125 private readonly transport: Transport;
126
127 constructor(transport: Transport, options: ResilienceOptions = {}, clock: Clock = systemClock) {
128 this.transport = transport;
129 this.clock = clock;
130 this.options = resolveOptions(options);
131 this.breaker = new CircuitBreaker(
132 clock,
133 this.options.failureThreshold,
134 this.options.openDurationMs,
135 this.options.halfOpenMaxAttempts,
136 );
137 this.bulkhead = new Bulkhead({
138 maxConcurrency: this.options.maxConcurrency,
139 initialLimit: this.options.initialConcurrencyLimit,
140 minLimit: this.options.minConcurrencyLimit,
141 additiveIncrease: this.options.concurrencyAdditiveIncrease,
142 decreaseFactor: this.options.concurrencyDecreaseFactor,
143 });
144 this.retryBudget = new RetryBudget(this.options.retryBudgetRatio, this.options.retryBudgetMaxTokens);
145 }
146
147 circuitState() {
148 return this.breaker.currentState();
149 }
150
151 /** Bulkhead snapshot (tests/observability). */
152 concurrency() {
153 return { inFlight: this.bulkhead.inFlight, limit: this.bulkhead.effectiveLimit };
154 }
155
156 /** Retry-budget balance (tests/observability). */
157 retryTokens(): number {
158 return this.retryBudget.tokens;
159 }
160
161 /**
162 * Send `req` under the full resilience policy. THIS METHOD IS THE TASK.
163 *
164 * Orchestrate retries + 3-state circuit breaker together with deadline
165 * propagation, a retry budget, bulkhead/AIMD admission, and request hedging,
166 * so the four interacting policies CO-EVOLVE across the attempt loop. The
167 * provided pieces are `this.breaker`, `this.bulkhead`, `this.retryBudget`,
168 * `this.runHedged` (hedging + deadline-aware suppression , already complete),
169 * `backoffDelay`, the error classes, and `this.clock`.
170 *
171 * Once per call (in this order): `this.retryBudget.deposit()`; then
172 * `this.bulkhead.tryAcquire()` , if refused, throw
173 * `BulkheadRejectedError(inFlight, effectiveLimit)` WITHOUT calling the
174 * transport; otherwise hold ONE slot and release it EXACTLY once at the end via
175 * `recordSuccess()` (terminal success) or `recordFailure()` (any terminal
176 * failure, including a short-circuit). That release drives AIMD.
177 *
178 * Per loop iteration the conditions are evaluated in a PINNED PRECEDENCE:
179 * 1. DEADLINE first: with `now = this.clock.now()` and `deadline = ctx.deadline`,
180 * if `deadline !== undefined && now >= deadline` throw
181 * `DeadlineExceededError(deadline, now)` (a request past its deadline is
182 * dead regardless of breaker state). Never call the transport.
183 * 2. BREAKER: `this.breaker.tryAcquire()`. Refused on the FIRST iteration →
184 * `CircuitOpenError`; refused LATER → `RetriesExhaustedError(attempts,
185 * lastError, "circuit_open")`.
186 * Then run the attempt with timeout `deadline === undefined ? timeoutMs :
187 * min(timeoutMs, deadline - now)` via `this.runHedged(req, attemptTimeout,
188 * deadline)`. It returns `{ ok, res?, err?, hedgesSpent }`, already classified
189 * (a retryable status is `ok:false`), where `hedgesSpent` is how many hedge
190 * tokens it withdrew. Report the outcome to the breaker (`onSuccess`/`onFailure`).
191 *
192 * On a failed attempt the loop stops in this PINNED PRECEDENCE:
193 * 3. MAX-RETRIES: if `retryIndex >= this.options.maxRetries` →
194 * `RetriesExhaustedError(attempts, lastError, "max_retries")`.
195 * 4. RETRY-BUDGET: else `this.retryBudget.tryWithdraw()`; if it refuses →
196 * `RetriesExhaustedError(attempts, lastError, "retry_budget")`.
197 * Otherwise compute `backoffDelay(retryIndex, baseDelayMs, maxDelayMs, () =>
198 * this.clock.random())`; if `deadline !== undefined && now + delay >= deadline`
199 * throw `DeadlineExceededError` (don't sleep past the deadline only to fail);
200 * else `await this.clock.sleep(delay)` and loop.
201 *
202 * RETRY-BUDGET ACCOUNTING (co-evolution): every retry AND every hedge withdraws
203 * one whole token. A call that ULTIMATELY SUCCEEDS refunds (via
204 * `this.retryBudget.refund(n)`) every token it withdrew , those attempts were
205 * productive, not storm load , so an early productive retry restores budget that
206 * a later call can spend. A call that ultimately FAILS keeps its tokens debited.
207 * Tokens are integers throughout (no fractional costs/refunds).
208 *
209 * Backward-compatible: with the defaults (no deadline, hedging off, generous
210 * bulkhead/budget) behaviour matches the plain retry + breaker client.
211 */
212 async send(req: HttpRequest, ctx: RequestContext = {}): Promise<HttpResponse> {
213 // The symbols below are intentionally referenced so a partial implementation
214 // still type-checks; observe/remove as you implement.
215 void this.retryBudget;
216 void this.bulkhead;
217 void this.breaker;
218 void this.clock;
219 void this.options;
220 void this.runHedged;
221 void backoffDelay;
222 void CircuitOpenError;
223 void RetriesExhaustedError;
224 void DeadlineExceededError;
225 void BulkheadRejectedError;
226 void req;
227 void ctx;
228
229 // TODO(resilient-client): implement the contract documented above.
230 throw new Error("ResilientClient.send not implemented");
231 }
232
233 /**
234 * Run one logical attempt with request hedging. Fire the primary; if it is
235 * still pending after `hedgeDelayMs`, fire up to `maxHedges` extra parallel
236 * attempts (each consuming a retry-budget token) and race them; the first
237 * success/non-retryable response wins and the losers are aborted.
238 *
239 * Returns `{ ok, res?, err?, hedgesSpent }` where `hedgesSpent` counts the EXTRA
240 * hedge attempts that were launched (and debited from the budget).
241 *
242 * Deadline-aware hedge suppression: a new hedge is only launched while
243 * `clock.now() < deadline` (there is still budget left to receive its answer);
244 * once the deadline is reached, no further hedge starts and the call awaits the
245 * in-flight attempts (which are themselves deadline-bounded by their shortened
246 * timeout).
247 */
248 private async runHedged(
249 req: HttpRequest,
250 timeoutMs: number,
251 deadline: number | undefined,
252 ): Promise<AttemptOutcome> {
253 const hedgeDelay = this.options.hedgeDelayMs;
254 if (hedgeDelay <= 0) {
255 const outcome = await this.classify(req, timeoutMs, undefined);
256 return { ...outcome, hedgesSpent: 0 };
257 }
258
259 const controllers: AbortController[] = [];
260 const live = new Map<number, Promise<{ idx: number; outcome: AttemptOutcome }>>();
261 let launched = 0;
262 let hedgesSpent = 0;
263 let lastFailure: AttemptOutcome = { ok: false, err: new Error("no attempt made"), hedgesSpent: 0 };
264
265 const launch = (): void => {
266 const ctl = new AbortController();
267 controllers.push(ctl);
268 const idx = launched++;
269 live.set(
270 idx,
271 this.classify(req, timeoutMs, ctl.signal).then((outcome) => ({
272 idx,
273 outcome: { ...outcome, hedgesSpent: 0 },
274 })),
275 );
276 };
277
278 launch(); // primary
279
280 while (live.size > 0) {
281 const deadlineAllowsHedge = deadline === undefined || this.clock.now() < deadline;
282 const canHedge = launched < 1 + this.options.maxHedges && deadlineAllowsHedge;
283 const hedgeCtl = new AbortController();
284 const racers: Promise<{ idx: number; outcome: AttemptOutcome }>[] = [...live.values()];
285 if (canHedge) {
286 racers.push(
287 this.clock
288 .sleep(hedgeDelay, hedgeCtl.signal)
289 .then(() => ({ idx: -1, outcome: { ok: false, hedgesSpent: 0 } as AttemptOutcome })),
290 );
291 }
292
293 const winner = await Promise.race(racers);
294
295 if (winner.idx === -1) {
296 // Hedge timer fired first: launch another attempt iff the deadline still
297 // permits a reply AND the retry budget allows it.
298 const stillRoom = deadline === undefined || this.clock.now() < deadline;
299 if (stillRoom && this.retryBudget.tryWithdraw()) {
300 hedgesSpent += 1;
301 launch();
302 } else {
303 const settled = await Promise.race([...live.values()]);
304 hedgeCtl.abort();
305 live.delete(settled.idx);
306 if (settled.outcome.ok) {
307 for (const c of controllers) c.abort();
308 return { ...settled.outcome, hedgesSpent };
309 }
310 lastFailure = settled.outcome;
311 }
312 continue;
313 }
314
315 hedgeCtl.abort();
316 live.delete(winner.idx);
317 if (winner.outcome.ok) {
318 for (const c of controllers) c.abort();
319 return { ...winner.outcome, hedgesSpent };
320 }
321 lastFailure = winner.outcome;
322 }
323
324 for (const c of controllers) c.abort();
325 return { ...lastFailure, hedgesSpent };
326 }
327
328 /** Run a single transport attempt and classify it as success/failure. */
329 private async classify(
330 req: HttpRequest,
331 timeoutMs: number,
332 abort: AbortSignal | undefined,
333 ): Promise<{ ok: boolean; res?: HttpResponse; err?: unknown }> {
334 try {
335 const res = await runAttempt(this.transport, req, this.clock, timeoutMs, abort);
336 if (this.options.isRetryableStatus(res.status)) {
337 return { ok: false, err: new Error(`retryable status ${res.status}`) };
338 }
339 return { ok: true, res };
340 } catch (err) {
341 return { ok: false, err };
342 }
343 }
344 }
345cat src/circuitBreaker.ts src/bulkhead.ts src/retryBudget.ts src/backoff.ts src/clock.ts src/errors.ts src/types.ts
import type { Clock } from "./clock.js";
import type { CircuitState } from "./types.js";
/**
* A standard three-state circuit breaker.
*
* closed normal operation; requests flow. Consecutive failures are counted;
* reaching `failureThreshold` trips the breaker to `open`.
* open requests are short-circuited (rejected without calling the
* transport). After `openDurationMs` has elapsed since it opened,
* the breaker permits a limited number of trial requests by moving
* to `half_open`.
* half_open up to `halfOpenMaxAttempts` trial requests are admitted. The first
* SUCCESS closes the breaker and resets counters; any FAILURE
* re-opens it and restarts the open timer.
*
* Single-threaded usage model: Node executes one continuation at a time, so the
* breaker does not lock , but it deliberately tracks how many half-open trials
* are *in flight* so concurrent callers cannot all be admitted at once during
* recovery.
*
* This class is provided complete. The resilient client consults it via
* {@link tryAcquire} before each attempt and reports the result via
* {@link onSuccess} / {@link onFailure}.
*/
export class CircuitBreaker {
private state: CircuitState = "closed";
/** Consecutive failures while closed. */
private consecutiveFailures = 0;
/** Epoch ms at which the breaker last opened. */
private openedAt = 0;
/** Half-open trials currently admitted but not yet settled. */
private halfOpenInFlight = 0;
/** Successful half-open trials so far in the current half-open window. */
private halfOpenSuccesses = 0;
constructor(
private readonly clock: Clock,
private readonly failureThreshold: number,
private readonly openDurationMs: number,
private readonly halfOpenMaxAttempts: number,
) {}
/** Current externally-visible state (after applying any time-based transition). */
currentState(): CircuitState {
this.maybeHalfOpen();
return this.state;
}
/**
* Ask permission to make an attempt.
*
* - `closed` -> always admitted.
* - `open` -> if the open window has elapsed, transition to `half_open`
* and admit a trial; otherwise refuse.
* - `half_open` -> admit only while fewer than `halfOpenMaxAttempts` trials
* are in flight; otherwise refuse.
*
* Returns `true` when the caller may proceed to call the transport. When it
* returns `true` in a half-open state it has reserved a trial slot, which the
* caller MUST release by calling exactly one of {@link onSuccess} /
* {@link onFailure}.
*/
tryAcquire(): boolean {
this.maybeHalfOpen();
if (this.state === "open") {
return false;
}
if (this.state === "half_open") {
if (this.halfOpenInFlight >= this.halfOpenMaxAttempts) return false;
this.halfOpenInFlight += 1;
return true;
}
// closed
return true;
}
/** Report a successful attempt. */
onSuccess(): void {
if (this.state === "half_open") {
this.halfOpenInFlight = Math.max(0, this.halfOpenInFlight - 1);
this.halfOpenSuccesses += 1;
// A single success is enough to consider the dependency healthy again.
this.close();
return;
}
// closed: reset the failure streak.
this.consecutiveFailures = 0;
}
/** Report a failed attempt (transport reject, timeout, or retryable status). */
onFailure(): void {
if (this.state === "half_open") {
this.halfOpenInFlight = Math.max(0, this.halfOpenInFlight - 1);
// A failed trial re-opens the breaker and restarts the timer.
this.open();
return;
}
if (this.state === "open") {
// Shouldn't normally happen (open refuses), but stay defensive.
return;
}
// closed
this.consecutiveFailures += 1;
if (this.consecutiveFailures >= this.failureThreshold) {
this.open();
}
}
private open(): void {
this.state = "open";
this.openedAt = this.clock.now();
this.halfOpenInFlight = 0;
this.halfOpenSuccesses = 0;
this.consecutiveFailures = 0;
}
private close(): void {
this.state = "closed";
this.consecutiveFailures = 0;
this.halfOpenInFlight = 0;
this.halfOpenSuccesses = 0;
}
/** If open and the cool-down has elapsed, move to half_open. */
private maybeHalfOpen(): void {
if (this.state !== "open") return;
if (this.clock.now() - this.openedAt >= this.openDurationMs) {
this.state = "half_open";
this.halfOpenInFlight = 0;
this.halfOpenSuccesses = 0;
}
}
}
/**
* Bulkhead isolation + AIMD adaptive concurrency.
*
* Two limits gate concurrency:
* - `maxConcurrency` , a HARD ceiling on simultaneous in-flight attempts (the
* bulkhead). Never exceeded.
* - an ADAPTIVE limit that moves by AIMD (additive-increase /
* multiplicative-decrease): each success nudges it up by `additiveIncrease`
* (capped at `maxConcurrency`); each failure/timeout multiplies it down by
* `decreaseFactor` (floored at `minLimit`). Under load this converges toward
* the concurrency the dependency can actually sustain.
*
* The effective admission limit at any instant is `min(adaptiveLimit,
* maxConcurrency)`. `tryAcquire` admits only while `inFlight` is below it.
*
* Single-threaded model: Node runs one continuation at a time, so plain counters
* are race-free. Provided complete , the client calls
* `tryAcquire`/`recordSuccess`/`recordFailure`; it does not re-implement AIMD.
*/
export interface BulkheadOptions {
/** Hard ceiling on simultaneous in-flight attempts. Integer >= 1. */
maxConcurrency: number;
/** Starting adaptive limit. Integer in [minLimit, maxConcurrency]. Default = maxConcurrency. */
initialLimit?: number;
/** Floor for the adaptive limit. Integer >= 1. Default 1. */
minLimit?: number;
/** Additive step applied to the adaptive limit on success. Default 1. */
additiveIncrease?: number;
/** Multiplicative factor (in (0,1)) applied on failure. Default 0.5. */
decreaseFactor?: number;
}
export class Bulkhead {
private readonly maxConcurrency: number;
private readonly minLimit: number;
private readonly additiveIncrease: number;
private readonly decreaseFactor: number;
private limit: number;
private inFlightCount = 0;
constructor(options: BulkheadOptions) {
if (!Number.isInteger(options.maxConcurrency) || options.maxConcurrency < 1) {
throw new Error("maxConcurrency must be an integer >= 1");
}
this.maxConcurrency = options.maxConcurrency;
this.minLimit = Math.max(1, Math.floor(options.minLimit ?? 1));
this.additiveIncrease = options.additiveIncrease ?? 1;
this.decreaseFactor = options.decreaseFactor ?? 0.5;
const init = options.initialLimit ?? options.maxConcurrency;
this.limit = Math.min(this.maxConcurrency, Math.max(this.minLimit, Math.floor(init)));
}
/** Current effective admission limit. */
get effectiveLimit(): number {
return Math.min(this.maxConcurrency, this.limit);
}
/** Current adaptive limit (may exceed nothing; informational). */
get adaptiveLimit(): number {
return this.limit;
}
get inFlight(): number {
return this.inFlightCount;
}
/** Try to reserve a slot. Returns true (and increments in-flight) iff admitted. */
tryAcquire(): boolean {
if (this.inFlightCount >= this.effectiveLimit) return false;
this.inFlightCount += 1;
return true;
}
/** Release a slot after a SUCCESS and additively increase the adaptive limit. */
recordSuccess(): void {
this.inFlightCount = Math.max(0, this.inFlightCount - 1);
this.limit = Math.min(this.maxConcurrency, this.limit + this.additiveIncrease);
}
/** Release a slot after a FAILURE and multiplicatively decrease the adaptive limit. */
recordFailure(): void {
this.inFlightCount = Math.max(0, this.inFlightCount - 1);
this.limit = Math.max(this.minLimit, Math.floor(this.limit * this.decreaseFactor));
}
}
/**
* A retry budget (token bucket over RETRIES), the classic guard against retry
* storms (Envoy/Finagle style).
*
* The balance is capped at `maxTokens`. Every top-level request deposits `ratio`
* tokens (so a steady request stream refills the budget at `ratio` retries per
* request); every retry or hedge withdraws `1` token and is only permitted when
* the balance is `>= 1`. With `ratio = 0.2` the system tolerates roughly one
* retry per five requests in steady state , plus an initial cushion of
* `maxTokens` so a brief early burst can still retry.
*
* A withdrawn token can be REFUNDED whole (`refund`) when the retry/hedge it
* authorized turned out to be productive , i.e. the call ultimately succeeded ,
* so that productive retries are not counted as storm load. Refunds are integer
* and capped at `maxTokens` just like deposits.
*
* Provided complete , the client calls `deposit`/`tryWithdraw`/`refund`; it does
* not re-implement budget accounting. Deterministic (no time, no RNG).
*/
export class RetryBudget {
private balance: number;
private readonly ratio: number;
private readonly maxTokens: number;
constructor(ratio = 0.2, maxTokens = 10) {
if (!(ratio >= 0)) throw new Error("retry budget ratio must be >= 0");
if (!(maxTokens >= 0)) throw new Error("retry budget maxTokens must be >= 0");
this.ratio = ratio;
this.maxTokens = maxTokens;
this.balance = maxTokens; // start with the full cushion
}
/** Record a top-level request: deposit `ratio` tokens, capped at `maxTokens`. */
deposit(): void {
this.balance = Math.min(this.maxTokens, this.balance + this.ratio);
}
/** Try to spend one retry token. Returns true (and debits) iff a retry is permitted. */
tryWithdraw(): boolean {
if (this.balance >= 1) {
this.balance -= 1;
return true;
}
return false;
}
/**
* Refund `n` whole tokens previously withdrawn for retries/hedges that turned
* out to be productive (the call succeeded). Capped at `maxTokens`. `n` defaults
* to 1 and must be a non-negative integer.
*/
refund(n = 1): void {
if (!Number.isInteger(n) || n < 0) throw new Error("refund count must be a non-negative integer");
this.balance = Math.min(this.maxTokens, this.balance + n);
}
/** Current balance (observability/tests). */
get tokens(): number {
return this.balance;
}
}
/**
* Exponential backoff with "full jitter".
*
* For retry attempt `n` (0-based: n=0 is the delay before the FIRST retry, i.e.
* after the initial attempt failed), the uncapped exponential delay is
* `base * 2^n`, capped at `maxDelay`. Full jitter then picks a uniform random
* value in `[0, capped]` to avoid thundering-herd synchronization across many
* clients retrying at once.
*
* The randomness is injected (`rng` returns a value in `[0, 1)`) so the delay is
* deterministic in tests. This module is provided complete; the resilient client
* uses {@link backoffDelay} to decide how long to sleep between attempts.
*/
export function backoffDelay(
attempt: number,
baseDelayMs: number,
maxDelayMs: number,
rng: () => number,
): number {
if (attempt < 0) throw new Error("attempt must be >= 0");
const exponential = baseDelayMs * 2 ** attempt;
const capped = Math.min(exponential, maxDelayMs);
// Full jitter: uniform in [0, capped].
const jittered = rng() * capped;
// Whole milliseconds; never negative.
return Math.max(0, Math.floor(jittered));
}
/**
* Injectable clock + sleep + RNG.
*
* The resilient client never reads `Date.now()`, `setTimeout`, or `Math.random`
* directly. Routing all of that through a {@link Clock} keeps the retry/breaker
* timing logic deterministic and unit-testable: tests supply a
* {@link ManualClock} and advance it explicitly instead of waiting in real time.
*/
export interface Clock {
/** Current time in epoch milliseconds. */
now(): number;
/**
* Resolve after `ms` (logical) milliseconds. If a `signal` is supplied and it
* aborts first, the timer is cancelled and the promise resolves early , so a
* caller that loses a `Promise.race` does not leak a pending timer.
*/
sleep(ms: number, signal?: AbortSignal): Promise<void>;
/** Uniform random in [0, 1). Used for jitter. */
random(): number;
}
export const systemClock: Clock = {
now: () => Date.now(),
sleep: (ms, signal) =>
new Promise((resolve) => {
if (signal?.aborted) {
resolve();
return;
}
const timer = setTimeout(() => {
signal?.removeEventListener("abort", onAbort);
resolve();
}, ms);
const onAbort = (): void => {
clearTimeout(timer);
resolve();
};
signal?.addEventListener("abort", onAbort, { once: true });
}),
random: () => Math.random(),
};
/**
* A controllable clock for tests. `sleep` does not block real time: a sleeping
* caller is parked until {@link ManualClock.advance} moves the clock past its
* wake time, at which point it resolves on the microtask queue. Jitter is made
* deterministic via an injectable RNG (defaults to always returning 0, i.e. no
* jitter, which is the most test-friendly default).
*/
export class ManualClock implements Clock {
private current: number;
private rng: () => number;
private waiters: Array<{ at: number; resolve: () => void }> = [];
constructor(start = 0, rng: () => number = () => 0) {
this.current = start;
this.rng = rng;
}
now(): number {
return this.current;
}
random(): number {
return this.rng();
}
sleep(ms: number, signal?: AbortSignal): Promise<void> {
if (ms <= 0 || signal?.aborted) return Promise.resolve();
return new Promise<void>((resolve) => {
const waiter = { at: this.current + ms, resolve };
this.waiters.push(waiter);
if (signal) {
const onAbort = (): void => {
// Drop the parked waiter and resolve early on abort.
this.waiters = this.waiters.filter((w) => w !== waiter);
resolve();
};
signal.addEventListener("abort", onAbort, { once: true });
}
});
}
/** Advance logical time by `ms`, waking any sleepers whose deadline passed. */
async advance(ms: number): Promise<void> {
this.current += ms;
const due = this.waiters.filter((w) => w.at <= this.current);
this.waiters = this.waiters.filter((w) => w.at > this.current);
for (const w of due) w.resolve();
// Let woken continuations run before returning.
await Promise.resolve();
}
/** Number of callers currently parked in {@link sleep}. */
pending(): number {
return this.waiters.length;
}
/**
* The earliest wake time among parked sleepers, or `undefined` if none. Tests
* use this to advance EXACTLY to the next event (no fixed-step overshoot), so
* logical time is bit-for-bit reproducible.
*/
nextWakeAt(): number | undefined {
if (this.waiters.length === 0) return undefined;
let min = Infinity;
for (const w of this.waiters) if (w.at < min) min = w.at;
return min;
}
}
/**
* Typed failures surfaced by the resilient client. The `code` is stable so
* callers can switch on it (e.g. to map to a 503 vs a 504).
*/
export type ResilienceErrorCode =
| "circuit_open"
| "attempt_timeout"
| "retries_exhausted"
| "deadline_exceeded"
| "bulkhead_rejected";
export class CircuitOpenError extends Error {
readonly code = "circuit_open" as const;
constructor(message = "circuit breaker is open") {
super(message);
this.name = "CircuitOpenError";
}
}
export class AttemptTimeoutError extends Error {
readonly code = "attempt_timeout" as const;
constructor(public readonly timeoutMs: number) {
super(`attempt exceeded ${timeoutMs}ms timeout`);
this.name = "AttemptTimeoutError";
}
}
/**
* Thrown when every permitted attempt failed. Wraps the last underlying error
* (a transport rejection, an {@link AttemptTimeoutError}, or a synthetic error
* representing the last retryable HTTP status). `reason` distinguishes WHY
* retrying stopped: out of attempts, the retry budget ran dry, or the breaker
* opened mid-flight.
*/
export class RetriesExhaustedError extends Error {
readonly code = "retries_exhausted" as const;
constructor(
public readonly attempts: number,
public readonly lastError: unknown,
public readonly reason: "max_retries" | "retry_budget" | "circuit_open" = "max_retries",
) {
super(`exhausted ${attempts} attempt(s) (${reason})`);
this.name = "RetriesExhaustedError";
}
}
/**
* Thrown when the request's deadline (absolute time budget) elapsed before a
* successful response could be obtained , either before any attempt, or before a
* needed retry could complete.
*/
export class DeadlineExceededError extends Error {
readonly code = "deadline_exceeded" as const;
constructor(
public readonly deadline: number,
public readonly now: number,
) {
super(`deadline ${deadline} exceeded at ${now}`);
this.name = "DeadlineExceededError";
}
}
/**
* Thrown when the client's bulkhead is saturated: too many requests are already
* in flight (above the adaptive concurrency limit / the hard bulkhead cap), so
* this request is shed immediately without calling the transport.
*/
export class BulkheadRejectedError extends Error {
readonly code = "bulkhead_rejected" as const;
constructor(
public readonly inFlight: number,
public readonly limit: number,
) {
super(`bulkhead rejected: ${inFlight} in flight >= limit ${limit}`);
this.name = "BulkheadRejectedError";
}
}
/**
* Transport + policy types for the resilient HTTP client.
*
* The client is built around an injectable {@link Transport} (a thin fetch-like
* function) so it can wrap any HTTP library , and so tests can drive it with a
* deterministic fake instead of real sockets.
*/
export interface HttpRequest {
method: "GET" | "POST" | "PUT" | "PATCH" | "DELETE";
url: string;
headers?: Record<string, string>;
body?: string;
}
export interface HttpResponse {
status: number;
headers: Record<string, string>;
body: string;
}
/**
* A single HTTP round-trip. Receives an {@link AbortSignal} that the resilient
* client trips when a per-attempt timeout fires; a well-behaved transport should
* abort its in-flight work when the signal aborts. Resolving with any
* {@link HttpResponse} (including a 5xx) is "no exception thrown"; the client
* decides retriability from the status. Rejecting models a transport-level
* failure (DNS, connection reset, timeout abort).
*/
export type Transport = (req: HttpRequest, signal: AbortSignal) => Promise<HttpResponse>;
export type CircuitState = "closed" | "open" | "half_open";
/** Tunables for the retry + circuit-breaker policy. */
export interface ResilienceOptions {
/**
* Maximum number of RETRIES after the initial attempt. `maxRetries: 2` means
* up to 3 total attempts. Defaults to 2.
*/
maxRetries?: number;
/** Base backoff delay in ms for the first retry. Defaults to 50. */
baseDelayMs?: number;
/** Cap on a single backoff delay in ms. Defaults to 2000. */
maxDelayMs?: number;
/**
* Consecutive failures (as seen by the breaker) that trip the circuit from
* closed to open. Defaults to 5.
*/
failureThreshold?: number;
/**
* How long the circuit stays open before allowing a half-open trial, in ms.
* Defaults to 30000.
*/
openDurationMs?: number;
/**
* Number of trial requests permitted while half-open. Defaults to 1. The
* first success closes the circuit; a failure re-opens it.
*/
halfOpenMaxAttempts?: number;
/** Per-attempt timeout in ms. Defaults to 1000. */
timeoutMs?: number;
/**
* Decide whether a resolved {@link HttpResponse} counts as a failure for retry
* + breaker purposes. Defaults to "5xx and 429 are failures; everything else
* is a success".
*/
isRetryableStatus?: (status: number) => boolean;
// --- adaptive concurrency / bulkhead ---
/** Hard ceiling on simultaneous in-flight attempts. Integer >= 1. Defaults to 100. */
maxConcurrency?: number;
/** Starting adaptive concurrency limit. Defaults to maxConcurrency. */
initialConcurrencyLimit?: number;
/** Floor for the adaptive limit. Defaults to 1. */
minConcurrencyLimit?: number;
/** AIMD additive increase per success. Defaults to 1. */
concurrencyAdditiveIncrease?: number;
/** AIMD multiplicative decrease factor in (0,1). Defaults to 0.5. */
concurrencyDecreaseFactor?: number;
// --- retry budget ---
/** Retry-budget refill ratio (retries permitted per request in steady state). Defaults to 0.2. */
retryBudgetRatio?: number;
/** Retry-budget cushion (max tokens). Defaults to 10. */
retryBudgetMaxTokens?: number;
// --- hedging ---
/**
* If a non-settled attempt is still pending after `hedgeDelayMs`, fire an
* additional parallel ("hedged") attempt and race them; the first
* success/non-retryable response wins and the losers are aborted. `0` (default)
* disables hedging.
*/
hedgeDelayMs?: number;
/** Maximum number of EXTRA hedged attempts in flight per logical attempt. Defaults to 1. */
maxHedges?: number;
}
/**
* Per-call context. The DEADLINE is an absolute logical instant (ms, same scale
* as `clock.now()`) by which the call must succeed; attempts whose per-attempt
* timeout would run past it are shortened, and a retry that cannot start (after
* its backoff) before the deadline is not attempted , the call fails fast with a
* `DeadlineExceededError`. Propagate the SAME context to fan-out calls to share
* one budget.
*/
export interface RequestContext {
/** Absolute deadline in ms (clock scale). Omit for no deadline. */
deadline?: number;
}
export interface ResolvedResilienceOptions {
maxRetries: number;
baseDelayMs: number;
maxDelayMs: number;
failureThreshold: number;
openDurationMs: number;
halfOpenMaxAttempts: number;
timeoutMs: number;
isRetryableStatus: (status: number) => boolean;
maxConcurrency: number;
initialConcurrencyLimit: number;
minConcurrencyLimit: number;
concurrencyAdditiveIncrease: number;
concurrencyDecreaseFactor: number;
retryBudgetRatio: number;
retryBudgetMaxTokens: number;
hedgeDelayMs: number;
maxHedges: number;
}/home/user/app/src/resilientClient.ts
async send(req: HttpRequest, ctx: RequestContext = {}): Promise<HttpResponse> {
// The symbols below are intentionally referenced so a partial implementation
// still type-checks; observe/remove as you implement.
void this.retryBudget;
void this.bulkhead;
void this.breaker;
void this.clock;
void this.options;
void this.runHedged;
void backoffDelay;
void CircuitOpenError;
void RetriesExhaustedError;
void DeadlineExceededError;
void BulkheadRejectedError;
void req;
void ctx;
// TODO(resilient-client): implement the contract documented above.
throw new Error("ResilientClient.send not implemented");
} async send(req: HttpRequest, ctx: RequestContext = {}): Promise<HttpResponse> {
// Once per call: deposit to the budget, then claim ONE bulkhead slot.
this.retryBudget.deposit();
if (!this.bulkhead.tryAcquire()) {
throw new BulkheadRejectedError(this.bulkhead.inFlight, this.bulkhead.effectiveLimit);
}
const deadline = ctx.deadline;
let tokensWithdrawn = 0;
let attempts = 0;
let retryIndex = 0;
let lastError: unknown = undefined;
try {
for (;;) {
const now = this.clock.now();
// 1. DEADLINE first: a request past its deadline is dead regardless of
// the breaker. Never call the transport.
if (deadline !== undefined && now >= deadline) {
throw new DeadlineExceededError(deadline, now);
}
// 2. BREAKER admission.
if (!this.breaker.tryAcquire()) {
if (attempts === 0) {
throw new CircuitOpenError();
}
throw new RetriesExhaustedError(attempts, lastError, "circuit_open");
}
// Run the attempt, never extending the timeout past the remaining budget.
const attemptTimeout =
deadline === undefined ? this.options.timeoutMs : Math.min(this.options.timeoutMs, deadline - now);
const outcome = await this.runHedged(req, attemptTimeout, deadline);
attempts += 1;
tokensWithdrawn += outcome.hedgesSpent;
if (outcome.ok) {
this.breaker.onSuccess();
// Terminal success: release the slot via AIMD success and refund every
// token this (productive) call withdrew.
this.bulkhead.recordSuccess();
if (tokensWithdrawn > 0) this.retryBudget.refund(tokensWithdrawn);
return outcome.res as HttpResponse;
}
this.breaker.onFailure();
lastError = outcome.err;
// On a failed attempt, decide whether to retry in a pinned precedence.
// 3. MAX-RETRIES.
if (retryIndex >= this.options.maxRetries) {
throw new RetriesExhaustedError(attempts, lastError, "max_retries");
}
// 4. RETRY-BUDGET.
if (!this.retryBudget.tryWithdraw()) {
throw new RetriesExhaustedError(attempts, lastError, "retry_budget");
}
tokensWithdrawn += 1;
// Suppress the retry if its backoff would reach the deadline.
const delay = backoffDelay(retryIndex, this.options.baseDelayMs, this.options.maxDelayMs, () =>
this.clock.random(),
);
if (deadline !== undefined && now + delay >= deadline) {
throw new DeadlineExceededError(deadline, now);
}
await this.clock.sleep(delay);
retryIndex += 1;
}
} catch (err) {
// Any terminal failure (including a short-circuit) releases the slot via
// AIMD failure; withdrawn tokens stay debited.
this.bulkhead.recordFailure();
throw err;
}
}The file /home/user/app/src/resilientClient.ts has been updated successfully. (file state is current in your context , no need to Read it back)
npm run typecheck 2>&1 | tail -20 && npm test 2>&1 | tail -25
> resilient-http@2.1.0 typecheck
> tsc -p tsconfig.json --noEmit
> resilient-http@2.1.0 test
> vitest run
RUN v1.6.0 /home/user/app
✓ test/circuitBreaker.test.ts (7 tests) 5ms
✓ test/smoke.test.ts (2 tests) 3ms
✓ test/bulkhead.test.ts (3 tests) 3ms
✓ test/retryBudget.test.ts (3 tests) 3ms
Test Files 4 passed (4)
Tests 15 passed (15)
Start at 20:23:19
Duration 999ms (transform 165ms, setup 0ms, collect 227ms, tests 14ms, environment 1ms, prepare 295ms)No file changes, the agent left the workspace untouched.
VERIFIER: injecting hidden tests from /home/user/tests/hidden -> /home/user/app/test
VERIFIER: test files now present:
_oracle.ts
bulkhead.test.ts
circuitBreaker.test.ts
fuzz.test.ts
resilientClient.test.ts
retryBudget.test.ts
VERIFIER: typecheck
=== running hidden test suite in /home/user/app ===
=== vitest exit status: 1 ===
✓ test/circuitBreaker.test.ts > CircuitBreaker > opens after the configured number of consecutive failures
✓ test/circuitBreaker.test.ts > CircuitBreaker > a success resets the consecutive-failure streak while closed
✓ test/circuitBreaker.test.ts > CircuitBreaker > moves to half_open after the open window elapses and admits one trial
✓ test/circuitBreaker.test.ts > CircuitBreaker > a successful half-open trial closes the breaker
✓ test/circuitBreaker.test.ts > CircuitBreaker > a failed half-open trial re-opens the breaker and restarts the timer
✓ test/circuitBreaker.test.ts > CircuitBreaker > admits up to halfOpenMaxAttempts concurrent trials
✓ test/bulkhead.test.ts > Bulkhead (AIMD adaptive concurrency, provided sibling) > admits up to the effective limit and rejects beyond it
✓ test/bulkhead.test.ts > Bulkhead (AIMD adaptive concurrency, provided sibling) > additively increases the adaptive limit on success (capped at maxConcurrency)
✓ test/bulkhead.test.ts > Bulkhead (AIMD adaptive concurrency, provided sibling) > multiplicatively decreases on failure (floored at minLimit)
✓ test/retryBudget.test.ts > RetryBudget (token bucket over retries, provided sibling) > starts with the full cushion and spends one token per retry
✓ test/retryBudget.test.ts > RetryBudget (token bucket over retries, provided sibling) > deposits `ratio` tokens per request, capped at maxTokens
✓ test/retryBudget.test.ts > RetryBudget (token bucket over retries, provided sibling) > never exceeds the cap no matter how many deposits
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
FAIL test/fuzz.test.ts > engine == independent oracle (seeded fuzz, non-hedging regime) > matches the synchronous state-machine model across many random workloads
AssertionError: trial 8 call 0 (jit=0.5 mr=1 ft=3 to=50 dl=55) now: expected 55 to be 50 // Object.is equality
- Expected
+ Received
- 50
+ 55
❯ test/fuzz.test.ts:210:45
208| }
209| // logical clocks must stay in lockstep
210| expect(clock.now(), where + " now").toBe(oracle.getNow());
| ^
211| expect(engine.circuitState(), where + " circuit").toBe(expecte…
212| expect(engine.retryTokens(), where + " tokens").toBeCloseTo(ex…
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯
Test Files 1 failed | 4 passed (5)
Tests 1 failed | 36 passed (37)
Start at 20:23:40
Duration 1.22s (transform 180ms, setup 1ms, collect 286ms, tests 47ms, environment 1ms, prepare 350ms)
=== vitest exit status: 1 ===
RESULT: FAIL (reward=0)Reproduce this trial: git checkout 2f94510 && PYTHONPATH=src python3 scripts/build_site.py , then open trial/trial_e03b5cd045d04e3f. Re-running the agent live requires EVAL_PLATFORM_ENABLE_OAUTH_SMOKE=1 and is non-deterministic.
Trial trial_e03b5cd045d04e3f · verifier authoritative; classifier explanatory.