resilient-http-client

1	# Ticket: Make the resilient HTTP client's policies co-evolve in `ResilientClient.send`
2	
3	## Context
4	
5	`resilient-http` is the library the **checkout service** uses to call its flaky
6	downstream dependencies (TypeScript, Node 20). It already ships the building
7	blocks , an injectable `Clock`, a backoff calculator, a three-state
8	`CircuitBreaker`, a `Bulkhead` (AIMD + hard cap), a `RetryBudget` (token bucket),
9	typed errors, `runAttempt` (a single attempt under a timeout), and `runHedged`
10	(request hedging with deadline-aware suppression). They are **provided complete**.
11	
12	What is missing is the hard part: an orchestrator that makes the circuit breaker,
13	retry budget, deadline, and hedging **co-evolve** across the retry/hedge loop, so
14	that an attempt's outcome changes what later iterations are even allowed to do.
15	
16	**Only `ResilientClient.send(req, ctx)` is unimplemented.**
17	
18	## Your task
19	
20	Implement `src/resilientClient.ts → ResilientClient.send`. The authoritative spec
21	is the JSDoc directly above the `send` stub , read it; it pins every rule exactly.
22	The shape:
23	
24	- **Once per call:** deposit to the retry budget, then acquire a bulkhead slot
25	  (shed with `BulkheadRejectedError` and no transport call if refused). Hold the
26	  slot for the whole call and release it **exactly once** , `recordSuccess()` on a
27	  terminal success, `recordFailure()` on any terminal failure (including a
28	  short-circuit). That release drives AIMD.
29	
30	- **Per loop iteration, in a pinned precedence:** deadline first (a request past
31	  its deadline is dead regardless of the breaker), then breaker admission. Run the
32	  attempt with the timeout set to `min(timeoutMs, deadline - now)` (the configured
33	  timeout, never extended past the remaining deadline budget) via
34	  `this.runHedged(...)`. Report the outcome to the breaker.
35	
36	- **On a failed attempt, in a pinned precedence:** stop on max-retries, else gate
37	  the retry on the budget, else suppress the retry when `now + delay >= deadline`
38	  (reaching the deadline counts , do not sleep up to it only to fail); otherwise
39	  sleep the jittered backoff and loop. The exact error class and `reason` for each
40	  stop are in the JSDoc.
41	
42	- **Retry-budget co-evolution:** every retry and every hedge withdraws one whole
43	  token; a call that **ultimately succeeds refunds** every token it withdrew (a
44	  productive retry is not storm load), while a call that ultimately fails keeps
45	  them debited. Integer tokens only.
46	
47	Use the provided pieces; keep the public exports. With the defaults (no deadline,
48	hedging off, generous bulkhead/budget) behaviour must stay **backward-compatible**
49	with a plain retry + circuit-breaker client. Everything is deterministic: route
50	all time and randomness through the injected `Clock` (the tests drive a
51	`ManualClock` and an injected RNG; never read real time or `Math.random`).
52	
53	## Definition of done
54	
55	- `npm run typecheck` is clean.
56	- `npm test` passes. The shipped image contains only a smoke test that pins the
57	  public surface plus the provided primitive unit tests; the behavioural suite is
58	  applied at grading time. Implement the feature entirely within
59	  `ResilientClient.send` , do not change the tests, the primitives, the clock, the
60	  errors, `runAttempt`, or `runHedged`.
61	
62	## Running locally
63	
64	```bash
65	npm install      # already done in the provided environment
66	npm run typecheck
67	npm test
68	```
69

1	import { CircuitBreaker } from "./circuitBreaker.js";
2	import { backoffDelay } from "./backoff.js";
3	import { Bulkhead } from "./bulkhead.js";
4	import { RetryBudget } from "./retryBudget.js";
5	import { systemClock } from "./clock.js";
6	import type { Clock } from "./clock.js";
7	import {
8	  AttemptTimeoutError,
9	  BulkheadRejectedError,
10	  CircuitOpenError,
11	  DeadlineExceededError,
12	  RetriesExhaustedError,
13	} from "./errors.js";
14	import type {
15	  HttpRequest,
16	  HttpResponse,
17	  RequestContext,
18	  ResilienceOptions,
19	  ResolvedResilienceOptions,
20	  Transport,
21	} from "./types.js";
22	
23	const DEFAULTS: ResolvedResilienceOptions = {
24	  maxRetries: 2,
25	  baseDelayMs: 50,
26	  maxDelayMs: 2000,
27	  failureThreshold: 5,
28	  openDurationMs: 30_000,
29	  halfOpenMaxAttempts: 1,
30	  timeoutMs: 1000,
31	  isRetryableStatus: (status: number) => status >= 500 || status === 429,
32	  maxConcurrency: 100,
33	  initialConcurrencyLimit: 100,
34	  minConcurrencyLimit: 1,
35	  concurrencyAdditiveIncrease: 1,
36	  concurrencyDecreaseFactor: 0.5,
37	  retryBudgetRatio: 0.2,
38	  retryBudgetMaxTokens: 10,
39	  hedgeDelayMs: 0,
40	  maxHedges: 1,
41	};
42	
43	export function resolveOptions(options: ResilienceOptions = {}): ResolvedResilienceOptions {
44	  const maxConcurrency = options.maxConcurrency ?? DEFAULTS.maxConcurrency;
45	  return {
46	    maxRetries: options.maxRetries ?? DEFAULTS.maxRetries,
47	    baseDelayMs: options.baseDelayMs ?? DEFAULTS.baseDelayMs,
48	    maxDelayMs: options.maxDelayMs ?? DEFAULTS.maxDelayMs,
49	    failureThreshold: options.failureThreshold ?? DEFAULTS.failureThreshold,
50	    openDurationMs: options.openDurationMs ?? DEFAULTS.openDurationMs,
51	    halfOpenMaxAttempts: options.halfOpenMaxAttempts ?? DEFAULTS.halfOpenMaxAttempts,
52	    timeoutMs: options.timeoutMs ?? DEFAULTS.timeoutMs,
53	    isRetryableStatus: options.isRetryableStatus ?? DEFAULTS.isRetryableStatus,
54	    maxConcurrency,
55	    initialConcurrencyLimit: options.initialConcurrencyLimit ?? maxConcurrency,
56	    minConcurrencyLimit: options.minConcurrencyLimit ?? DEFAULTS.minConcurrencyLimit,
57	    concurrencyAdditiveIncrease: options.concurrencyAdditiveIncrease ?? DEFAULTS.concurrencyAdditiveIncrease,
58	    concurrencyDecreaseFactor: options.concurrencyDecreaseFactor ?? DEFAULTS.concurrencyDecreaseFactor,
59	    retryBudgetRatio: options.retryBudgetRatio ?? DEFAULTS.retryBudgetRatio,
60	    retryBudgetMaxTokens: options.retryBudgetMaxTokens ?? DEFAULTS.retryBudgetMaxTokens,
61	    hedgeDelayMs: options.hedgeDelayMs ?? DEFAULTS.hedgeDelayMs,
62	    maxHedges: options.maxHedges ?? DEFAULTS.maxHedges,
63	  };
64	}
65	
66	/**
67	 * Run a single transport attempt under a per-attempt timeout. Resolves with the
68	 * transport's response; rejects with {@link AttemptTimeoutError} if the timeout
69	 * fires first (aborting the transport), or with whatever the transport rejected
70	 * with. An optional external `abort` signal lets a hedge winner cancel the
71	 * losers. Provided complete.
72	 */
73	export async function runAttempt(
74	  transport: Transport,
75	  req: HttpRequest,
76	  clock: Clock,
77	  timeoutMs: number,
78	  abort?: AbortSignal,
79	): Promise<HttpResponse> {
80	  const transportCtl = new AbortController();
81	  const timerCtl = new AbortController();
82	
83	  const onExternalAbort = (): void => transportCtl.abort();
84	  if (abort) {
85	    if (abort.aborted) transportCtl.abort();
86	    else abort.addEventListener("abort", onExternalAbort, { once: true });
87	  }
88	
89	  const result = await Promise.race([
90	    transport(req, transportCtl.signal).then(
91	      (res) => ({ kind: "res" as const, res }),
92	      (err) => ({ kind: "err" as const, err }),
93	    ),
94	    clock.sleep(timeoutMs, timerCtl.signal).then(() => ({ kind: "timeout" as const })),
95	  ]);
96	
97	  if (abort) abort.removeEventListener("abort", onExternalAbort);
98	
99	  if (result.kind === "timeout") {
100	    transportCtl.abort();
101	    throw new AttemptTimeoutError(timeoutMs);
102	  }
103	  timerCtl.abort();
104	  if (result.kind === "err") throw result.err;
105	  return result.res;
106	}
107	
108	interface AttemptOutcome {
109	  ok: boolean;
110	  res?: HttpResponse;
111	  err?: unknown;
112	  /** EXTRA hedge attempts launched (and debited from the budget) for this attempt. */
113	  hedgesSpent: number;
114	}
115	
116	/**
117	 * A resilient HTTP client. See the original stub for the full `send` contract.
118	 */
119	export class ResilientClient {
120	  readonly options: ResolvedResilienceOptions;
121	  private readonly breaker: CircuitBreaker;
122	  private readonly bulkhead: Bulkhead;
123	  private readonly retryBudget: RetryBudget;
124	  private readonly clock: Clock;
125	  private readonly transport: Transport;
126	
127	  constructor(transport: Transport, options: ResilienceOptions = {}, clock: Clock = systemClock) {
128	    this.transport = transport;
129	    this.clock = clock;
130	    this.options = resolveOptions(options);
131	    this.breaker = new CircuitBreaker(
132	      clock,
133	      this.options.failureThreshold,
134	      this.options.openDurationMs,
135	      this.options.halfOpenMaxAttempts,
136	    );
137	    this.bulkhead = new Bulkhead({
138	      maxConcurrency: this.options.maxConcurrency,
139	      initialLimit: this.options.initialConcurrencyLimit,
140	      minLimit: this.options.minConcurrencyLimit,
141	      additiveIncrease: this.options.concurrencyAdditiveIncrease,
142	      decreaseFactor: this.options.concurrencyDecreaseFactor,
143	    });
144	    this.retryBudget = new RetryBudget(this.options.retryBudgetRatio, this.options.retryBudgetMaxTokens);
145	  }
146	
147	  circuitState() {
148	    return this.breaker.currentState();
149	  }
150	
151	  /** Bulkhead snapshot (tests/observability). */
152	  concurrency() {
153	    return { inFlight: this.bulkhead.inFlight, limit: this.bulkhead.effectiveLimit };
154	  }
155	
156	  /** Retry-budget balance (tests/observability). */
157	  retryTokens(): number {
158	    return this.retryBudget.tokens;
159	  }
160	
161	  /**
162	   * Send `req` under the full resilience policy. THIS METHOD IS THE TASK.
163	   *
164	   * Orchestrate retries + 3-state circuit breaker together with deadline
165	   * propagation, a retry budget, bulkhead/AIMD admission, and request hedging,
166	   * so the four interacting policies CO-EVOLVE across the attempt loop. The
167	   * provided pieces are `this.breaker`, `this.bulkhead`, `this.retryBudget`,
168	   * `this.runHedged` (hedging + deadline-aware suppression , already complete),
169	   * `backoffDelay`, the error classes, and `this.clock`.
170	   *
171	   * Once per call (in this order): `this.retryBudget.deposit()`; then
172	   * `this.bulkhead.tryAcquire()` , if refused, throw
173	   * `BulkheadRejectedError(inFlight, effectiveLimit)` WITHOUT calling the
174	   * transport; otherwise hold ONE slot and release it EXACTLY once at the end via
175	   * `recordSuccess()` (terminal success) or `recordFailure()` (any terminal
176	   * failure, including a short-circuit). That release drives AIMD.
177	   *
178	   * Per loop iteration the conditions are evaluated in a PINNED PRECEDENCE:
179	   *   1. DEADLINE first: with `now = this.clock.now()` and `deadline = ctx.deadline`,
180	   *      if `deadline !== undefined && now >= deadline` throw
181	   *      `DeadlineExceededError(deadline, now)` (a request past its deadline is
182	   *      dead regardless of breaker state). Never call the transport.
183	   *   2. BREAKER: `this.breaker.tryAcquire()`. Refused on the FIRST iteration →
184	   *      `CircuitOpenError`; refused LATER → `RetriesExhaustedError(attempts,
185	   *      lastError, "circuit_open")`.
186	   * Then run the attempt with timeout `deadline === undefined ? timeoutMs :
187	   * min(timeoutMs, deadline - now)` via `this.runHedged(req, attemptTimeout,
188	   * deadline)`. It returns `{ ok, res?, err?, hedgesSpent }`, already classified
189	   * (a retryable status is `ok:false`), where `hedgesSpent` is how many hedge
190	   * tokens it withdrew. Report the outcome to the breaker (`onSuccess`/`onFailure`).
191	   *
192	   * On a failed attempt the loop stops in this PINNED PRECEDENCE:
193	   *   3. MAX-RETRIES: if `retryIndex >= this.options.maxRetries` →
194	   *      `RetriesExhaustedError(attempts, lastError, "max_retries")`.
195	   *   4. RETRY-BUDGET: else `this.retryBudget.tryWithdraw()`; if it refuses →
196	   *      `RetriesExhaustedError(attempts, lastError, "retry_budget")`.
197	   * Otherwise compute `backoffDelay(retryIndex, baseDelayMs, maxDelayMs, () =>
198	   * this.clock.random())`; if `deadline !== undefined && now + delay >= deadline`
199	   * throw `DeadlineExceededError` (don't sleep past the deadline only to fail);
200	   * else `await this.clock.sleep(delay)` and loop.
201	   *
202	   * RETRY-BUDGET ACCOUNTING (co-evolution): every retry AND every hedge withdraws
203	   * one whole token. A call that ULTIMATELY SUCCEEDS refunds (via
204	   * `this.retryBudget.refund(n)`) every token it withdrew , those attempts were
205	   * productive, not storm load , so an early productive retry restores budget that
206	   * a later call can spend. A call that ultimately FAILS keeps its tokens debited.
207	   * Tokens are integers throughout (no fractional costs/refunds).
208	   *
209	   * Backward-compatible: with the defaults (no deadline, hedging off, generous
210	   * bulkhead/budget) behaviour matches the plain retry + breaker client.
211	   */
212	  async send(req: HttpRequest, ctx: RequestContext = {}): Promise<HttpResponse> {
213	    // The symbols below are intentionally referenced so a partial implementation
214	    // still type-checks; observe/remove as you implement.
215	    void this.retryBudget;
216	    void this.bulkhead;
217	    void this.breaker;
218	    void this.clock;
219	    void this.options;
220	    void this.runHedged;
221	    void backoffDelay;
222	    void CircuitOpenError;
223	    void RetriesExhaustedError;
224	    void DeadlineExceededError;
225	    void BulkheadRejectedError;
226	    void req;
227	    void ctx;
228	
229	    // TODO(resilient-client): implement the contract documented above.
230	    throw new Error("ResilientClient.send not implemented");
231	  }
232	
233	  /**
234	   * Run one logical attempt with request hedging. Fire the primary; if it is
235	   * still pending after `hedgeDelayMs`, fire up to `maxHedges` extra parallel
236	   * attempts (each consuming a retry-budget token) and race them; the first
237	   * success/non-retryable response wins and the losers are aborted.
238	   *
239	   * Returns `{ ok, res?, err?, hedgesSpent }` where `hedgesSpent` counts the EXTRA
240	   * hedge attempts that were launched (and debited from the budget).
241	   *
242	   * Deadline-aware hedge suppression: a new hedge is only launched while
243	   * `clock.now() < deadline` (there is still budget left to receive its answer);
244	   * once the deadline is reached, no further hedge starts and the call awaits the
245	   * in-flight attempts (which are themselves deadline-bounded by their shortened
246	   * timeout).
247	   */
248	  private async runHedged(
249	    req: HttpRequest,
250	    timeoutMs: number,
251	    deadline: number | undefined,
252	  ): Promise<AttemptOutcome> {
253	    const hedgeDelay = this.options.hedgeDelayMs;
254	    if (hedgeDelay <= 0) {
255	      const outcome = await this.classify(req, timeoutMs, undefined);
256	      return { ...outcome, hedgesSpent: 0 };
257	    }
258	
259	    const controllers: AbortController[] = [];
260	    const live = new Map<number, Promise<{ idx: number; outcome: AttemptOutcome }>>();
261	    let launched = 0;
262	    let hedgesSpent = 0;
263	    let lastFailure: AttemptOutcome = { ok: false, err: new Error("no attempt made"), hedgesSpent: 0 };
264	
265	    const launch = (): void => {
266	      const ctl = new AbortController();
267	      controllers.push(ctl);
268	      const idx = launched++;
269	      live.set(
270	        idx,
271	        this.classify(req, timeoutMs, ctl.signal).then((outcome) => ({
272	          idx,
273	          outcome: { ...outcome, hedgesSpent: 0 },
274	        })),
275	      );
276	    };
277	
278	    launch(); // primary
279	
280	    while (live.size > 0) {
281	      const deadlineAllowsHedge = deadline === undefined || this.clock.now() < deadline;
282	      const canHedge = launched < 1 + this.options.maxHedges && deadlineAllowsHedge;
283	      const hedgeCtl = new AbortController();
284	      const racers: Promise<{ idx: number; outcome: AttemptOutcome }>[] = [...live.values()];
285	      if (canHedge) {
286	        racers.push(
287	          this.clock
288	            .sleep(hedgeDelay, hedgeCtl.signal)
289	            .then(() => ({ idx: -1, outcome: { ok: false, hedgesSpent: 0 } as AttemptOutcome })),
290	        );
291	      }
292	
293	      const winner = await Promise.race(racers);
294	
295	      if (winner.idx === -1) {
296	        // Hedge timer fired first: launch another attempt iff the deadline still
297	        // permits a reply AND the retry budget allows it.
298	        const stillRoom = deadline === undefined || this.clock.now() < deadline;
299	        if (stillRoom && this.retryBudget.tryWithdraw()) {
300	          hedgesSpent += 1;
301	          launch();
302	        } else {
303	          const settled = await Promise.race([...live.values()]);
304	          hedgeCtl.abort();
305	          live.delete(settled.idx);
306	          if (settled.outcome.ok) {
307	            for (const c of controllers) c.abort();
308	            return { ...settled.outcome, hedgesSpent };
309	          }
310	          lastFailure = settled.outcome;
311	        }
312	        continue;
313	      }
314	
315	      hedgeCtl.abort();
316	      live.delete(winner.idx);
317	      if (winner.outcome.ok) {
318	        for (const c of controllers) c.abort();
319	        return { ...winner.outcome, hedgesSpent };
320	      }
321	      lastFailure = winner.outcome;
322	    }
323	
324	    for (const c of controllers) c.abort();
325	    return { ...lastFailure, hedgesSpent };
326	  }
327	
328	  /** Run a single transport attempt and classify it as success/failure. */
329	  private async classify(
330	    req: HttpRequest,
331	    timeoutMs: number,
332	    abort: AbortSignal | undefined,
333	  ): Promise<{ ok: boolean; res?: HttpResponse; err?: unknown }> {
334	    try {
335	      const res = await runAttempt(this.transport, req, this.clock, timeoutMs, abort);
336	      if (this.options.isRetryableStatus(res.status)) {
337	        return { ok: false, err: new Error(`retryable status ${res.status}`) };
338	      }
339	      return { ok: true, res };
340	    } catch (err) {
341	      return { ok: false, err };
342	    }
343	  }
344	}
345

1	/**
2	 * Transport + policy types for the resilient HTTP client.
3	 *
4	 * The client is built around an injectable {@link Transport} (a thin fetch-like
5	 * function) so it can wrap any HTTP library , and so tests can drive it with a
6	 * deterministic fake instead of real sockets.
7	 */
8	
9	export interface HttpRequest {
10	  method: "GET" | "POST" | "PUT" | "PATCH" | "DELETE";
11	  url: string;
12	  headers?: Record<string, string>;
13	  body?: string;
14	}
15	
16	export interface HttpResponse {
17	  status: number;
18	  headers: Record<string, string>;
19	  body: string;
20	}
21	
22	/**
23	 * A single HTTP round-trip. Receives an {@link AbortSignal} that the resilient
24	 * client trips when a per-attempt timeout fires; a well-behaved transport should
25	 * abort its in-flight work when the signal aborts. Resolving with any
26	 * {@link HttpResponse} (including a 5xx) is "no exception thrown"; the client
27	 * decides retriability from the status. Rejecting models a transport-level
28	 * failure (DNS, connection reset, timeout abort).
29	 */
30	export type Transport = (req: HttpRequest, signal: AbortSignal) => Promise<HttpResponse>;
31	
32	export type CircuitState = "closed" | "open" | "half_open";
33	
34	/** Tunables for the retry + circuit-breaker policy. */
35	export interface ResilienceOptions {
36	  /**
37	   * Maximum number of RETRIES after the initial attempt. `maxRetries: 2` means
38	   * up to 3 total attempts. Defaults to 2.
39	   */
40	  maxRetries?: number;
41	  /** Base backoff delay in ms for the first retry. Defaults to 50. */
42	  baseDelayMs?: number;
43	  /** Cap on a single backoff delay in ms. Defaults to 2000. */
44	  maxDelayMs?: number;
45	  /**
46	   * Consecutive failures (as seen by the breaker) that trip the circuit from
47	   * closed to open. Defaults to 5.
48	   */
49	  failureThreshold?: number;
50	  /**
51	   * How long the circuit stays open before allowing a half-open trial, in ms.
52	   * Defaults to 30000.
53	   */
54	  openDurationMs?: number;
55	  /**
56	   * Number of trial requests permitted while half-open. Defaults to 1. The
57	   * first success closes the circuit; a failure re-opens it.
58	   */
59	  halfOpenMaxAttempts?: number;
60	  /** Per-attempt timeout in ms. Defaults to 1000. */
61	  timeoutMs?: number;
62	  /**
63	   * Decide whether a resolved {@link HttpResponse} counts as a failure for retry
64	   * + breaker purposes. Defaults to "5xx and 429 are failures; everything else
65	   * is a success".
66	   */
67	  isRetryableStatus?: (status: number) => boolean;
68	
69	  // --- adaptive concurrency / bulkhead ---
70	  /** Hard ceiling on simultaneous in-flight attempts. Integer >= 1. Defaults to 100. */
71	  maxConcurrency?: number;
72	  /** Starting adaptive concurrency limit. Defaults to maxConcurrency. */
73	  initialConcurrencyLimit?: number;
74	  /** Floor for the adaptive limit. Defaults to 1. */
75	  minConcurrencyLimit?: number;
76	  /** AIMD additive increase per success. Defaults to 1. */
77	  concurrencyAdditiveIncrease?: number;
78	  /** AIMD multiplicative decrease factor in (0,1). Defaults to 0.5. */
79	  concurrencyDecreaseFactor?: number;
80	
81	  // --- retry budget ---
82	  /** Retry-budget refill ratio (retries permitted per request in steady state). Defaults to 0.2. */
83	  retryBudgetRatio?: number;
84	  /** Retry-budget cushion (max tokens). Defaults to 10. */
85	  retryBudgetMaxTokens?: number;
86	
87	  // --- hedging ---
88	  /**
89	   * If a non-settled attempt is still pending after `hedgeDelayMs`, fire an
90	   * additional parallel ("hedged") attempt and race them; the first
91	   * success/non-retryable response wins and the losers are aborted. `0` (default)
92	   * disables hedging.
93	   */
94	  hedgeDelayMs?: number;
95	  /** Maximum number of EXTRA hedged attempts in flight per logical attempt. Defaults to 1. */
96	  maxHedges?: number;
97	}
98	
99	/**
100	 * Per-call context. The DEADLINE is an absolute logical instant (ms, same scale
101	 * as `clock.now()`) by which the call must succeed; attempts whose per-attempt
102	 * timeout would run past it are shortened, and a retry that cannot start (after
103	 * its backoff) before the deadline is not attempted , the call fails fast with a
104	 * `DeadlineExceededError`. Propagate the SAME context to fan-out calls to share
105	 * one budget.
106	 */
107	export interface RequestContext {
108	  /** Absolute deadline in ms (clock scale). Omit for no deadline. */
109	  deadline?: number;
110	}
111	
112	export interface ResolvedResilienceOptions {
113	  maxRetries: number;
114	  baseDelayMs: number;
115	  maxDelayMs: number;
116	  failureThreshold: number;
117	  openDurationMs: number;
118	  halfOpenMaxAttempts: number;
119	  timeoutMs: number;
120	  isRetryableStatus: (status: number) => boolean;
121	  maxConcurrency: number;
122	  initialConcurrencyLimit: number;
123	  minConcurrencyLimit: number;
124	  concurrencyAdditiveIncrease: number;
125	  concurrencyDecreaseFactor: number;
126	  retryBudgetRatio: number;
127	  retryBudgetMaxTokens: number;
128	  hedgeDelayMs: number;
129	  maxHedges: number;
130	}
131

1	/**
2	 * Typed failures surfaced by the resilient client. The `code` is stable so
3	 * callers can switch on it (e.g. to map to a 503 vs a 504).
4	 */
5	export type ResilienceErrorCode =
6	  | "circuit_open"
7	  | "attempt_timeout"
8	  | "retries_exhausted"
9	  | "deadline_exceeded"
10	  | "bulkhead_rejected";
11	
12	export class CircuitOpenError extends Error {
13	  readonly code = "circuit_open" as const;
14	  constructor(message = "circuit breaker is open") {
15	    super(message);
16	    this.name = "CircuitOpenError";
17	  }
18	}
19	
20	export class AttemptTimeoutError extends Error {
21	  readonly code = "attempt_timeout" as const;
22	  constructor(public readonly timeoutMs: number) {
23	    super(`attempt exceeded ${timeoutMs}ms timeout`);
24	    this.name = "AttemptTimeoutError";
25	  }
26	}
27	
28	/**
29	 * Thrown when every permitted attempt failed. Wraps the last underlying error
30	 * (a transport rejection, an {@link AttemptTimeoutError}, or a synthetic error
31	 * representing the last retryable HTTP status). `reason` distinguishes WHY
32	 * retrying stopped: out of attempts, the retry budget ran dry, or the breaker
33	 * opened mid-flight.
34	 */
35	export class RetriesExhaustedError extends Error {
36	  readonly code = "retries_exhausted" as const;
37	  constructor(
38	    public readonly attempts: number,
39	    public readonly lastError: unknown,
40	    public readonly reason: "max_retries" | "retry_budget" | "circuit_open" = "max_retries",
41	  ) {
42	    super(`exhausted ${attempts} attempt(s) (${reason})`);
43	    this.name = "RetriesExhaustedError";
44	  }
45	}
46	
47	/**
48	 * Thrown when the request's deadline (absolute time budget) elapsed before a
49	 * successful response could be obtained , either before any attempt, or before a
50	 * needed retry could complete.
51	 */
52	export class DeadlineExceededError extends Error {
53	  readonly code = "deadline_exceeded" as const;
54	  constructor(
55	    public readonly deadline: number,
56	    public readonly now: number,
57	  ) {
58	    super(`deadline ${deadline} exceeded at ${now}`);
59	    this.name = "DeadlineExceededError";
60	  }
61	}
62	
63	/**
64	 * Thrown when the client's bulkhead is saturated: too many requests are already
65	 * in flight (above the adaptive concurrency limit / the hard bulkhead cap), so
66	 * this request is shed immediately without calling the transport.
67	 */
68	export class BulkheadRejectedError extends Error {
69	  readonly code = "bulkhead_rejected" as const;
70	  constructor(
71	    public readonly inFlight: number,
72	    public readonly limit: number,
73	  ) {
74	    super(`bulkhead rejected: ${inFlight} in flight >= limit ${limit}`);
75	    this.name = "BulkheadRejectedError";
76	  }
77	}
78

1	import type { Clock } from "./clock.js";
2	import type { CircuitState } from "./types.js";
3	
4	/**
5	 * A standard three-state circuit breaker.
6	 *
7	 *   closed     normal operation; requests flow. Consecutive failures are counted;
8	 *              reaching `failureThreshold` trips the breaker to `open`.
9	 *   open       requests are short-circuited (rejected without calling the
10	 *              transport). After `openDurationMs` has elapsed since it opened,
11	 *              the breaker permits a limited number of trial requests by moving
12	 *              to `half_open`.
13	 *   half_open  up to `halfOpenMaxAttempts` trial requests are admitted. The first
14	 *              SUCCESS closes the breaker and resets counters; any FAILURE
15	 *              re-opens it and restarts the open timer.
16	 *
17	 * Single-threaded usage model: Node executes one continuation at a time, so the
18	 * breaker does not lock , but it deliberately tracks how many half-open trials
19	 * are *in flight* so concurrent callers cannot all be admitted at once during
20	 * recovery.
21	 *
22	 * This class is provided complete. The resilient client consults it via
23	 * {@link tryAcquire} before each attempt and reports the result via
24	 * {@link onSuccess} / {@link onFailure}.
25	 */
26	export class CircuitBreaker {
27	  private state: CircuitState = "closed";
28	  /** Consecutive failures while closed. */
29	  private consecutiveFailures = 0;
30	  /** Epoch ms at which the breaker last opened. */
31	  private openedAt = 0;
32	  /** Half-open trials currently admitted but not yet settled. */
33	  private halfOpenInFlight = 0;
34	  /** Successful half-open trials so far in the current half-open window. */
35	  private halfOpenSuccesses = 0;
36	
37	  constructor(
38	    private readonly clock: Clock,
39	    private readonly failureThreshold: number,
40	    private readonly openDurationMs: number,
41	    private readonly halfOpenMaxAttempts: number,
42	  ) {}
43	
44	  /** Current externally-visible state (after applying any time-based transition). */
45	  currentState(): CircuitState {
46	    this.maybeHalfOpen();
47	    return this.state;
48	  }
49	
50	  /**
51	   * Ask permission to make an attempt.
52	   *
53	   *  - `closed`    -> always admitted.
54	   *  - `open`      -> if the open window has elapsed, transition to `half_open`
55	   *                   and admit a trial; otherwise refuse.
56	   *  - `half_open` -> admit only while fewer than `halfOpenMaxAttempts` trials
57	   *                   are in flight; otherwise refuse.
58	   *
59	   * Returns `true` when the caller may proceed to call the transport. When it
60	   * returns `true` in a half-open state it has reserved a trial slot, which the
61	   * caller MUST release by calling exactly one of {@link onSuccess} /
62	   * {@link onFailure}.
63	   */
64	  tryAcquire(): boolean {
65	    this.maybeHalfOpen();
66	
67	    if (this.state === "open") {
68	      return false;
69	    }
70	
71	    if (this.state === "half_open") {
72	      if (this.halfOpenInFlight >= this.halfOpenMaxAttempts) return false;
73	      this.halfOpenInFlight += 1;
74	      return true;
75	    }
76	
77	    // closed
78	    return true;
79	  }
80	
81	  /** Report a successful attempt. */
82	  onSuccess(): void {
83	    if (this.state === "half_open") {
84	      this.halfOpenInFlight = Math.max(0, this.halfOpenInFlight - 1);
85	      this.halfOpenSuccesses += 1;
86	      // A single success is enough to consider the dependency healthy again.
87	      this.close();
88	      return;
89	    }
90	    // closed: reset the failure streak.
91	    this.consecutiveFailures = 0;
92	  }
93	
94	  /** Report a failed attempt (transport reject, timeout, or retryable status). */
95	  onFailure(): void {
96	    if (this.state === "half_open") {
97	      this.halfOpenInFlight = Math.max(0, this.halfOpenInFlight - 1);
98	      // A failed trial re-opens the breaker and restarts the timer.
99	      this.open();
100	      return;
101	    }
102	    if (this.state === "open") {
103	      // Shouldn't normally happen (open refuses), but stay defensive.
104	      return;
105	    }
106	    // closed
107	    this.consecutiveFailures += 1;
108	    if (this.consecutiveFailures >= this.failureThreshold) {
109	      this.open();
110	    }
111	  }
112	
113	  private open(): void {
114	    this.state = "open";
115	    this.openedAt = this.clock.now();
116	    this.halfOpenInFlight = 0;
117	    this.halfOpenSuccesses = 0;
118	    this.consecutiveFailures = 0;
119	  }
120	
121	  private close(): void {
122	    this.state = "closed";
123	    this.consecutiveFailures = 0;
124	    this.halfOpenInFlight = 0;
125	    this.halfOpenSuccesses = 0;
126	  }
127	
128	  /** If open and the cool-down has elapsed, move to half_open. */
129	  private maybeHalfOpen(): void {
130	    if (this.state !== "open") return;
131	    if (this.clock.now() - this.openedAt >= this.openDurationMs) {
132	      this.state = "half_open";
133	      this.halfOpenInFlight = 0;
134	      this.halfOpenSuccesses = 0;
135	    }
136	  }
137	}
138

1	/**
2	 * Bulkhead isolation + AIMD adaptive concurrency.
3	 *
4	 * Two limits gate concurrency:
5	 *   - `maxConcurrency` , a HARD ceiling on simultaneous in-flight attempts (the
6	 *     bulkhead). Never exceeded.
7	 *   - an ADAPTIVE limit that moves by AIMD (additive-increase /
8	 *     multiplicative-decrease): each success nudges it up by `additiveIncrease`
9	 *     (capped at `maxConcurrency`); each failure/timeout multiplies it down by
10	 *     `decreaseFactor` (floored at `minLimit`). Under load this converges toward
11	 *     the concurrency the dependency can actually sustain.
12	 *
13	 * The effective admission limit at any instant is `min(adaptiveLimit,
14	 * maxConcurrency)`. `tryAcquire` admits only while `inFlight` is below it.
15	 *
16	 * Single-threaded model: Node runs one continuation at a time, so plain counters
17	 * are race-free. Provided complete , the client calls
18	 * `tryAcquire`/`recordSuccess`/`recordFailure`; it does not re-implement AIMD.
19	 */
20	export interface BulkheadOptions {
21	  /** Hard ceiling on simultaneous in-flight attempts. Integer >= 1. */
22	  maxConcurrency: number;
23	  /** Starting adaptive limit. Integer in [minLimit, maxConcurrency]. Default = maxConcurrency. */
24	  initialLimit?: number;
25	  /** Floor for the adaptive limit. Integer >= 1. Default 1. */
26	  minLimit?: number;
27	  /** Additive step applied to the adaptive limit on success. Default 1. */
28	  additiveIncrease?: number;
29	  /** Multiplicative factor (in (0,1)) applied on failure. Default 0.5. */
30	  decreaseFactor?: number;
31	}
32	
33	export class Bulkhead {
34	  private readonly maxConcurrency: number;
35	  private readonly minLimit: number;
36	  private readonly additiveIncrease: number;
37	  private readonly decreaseFactor: number;
38	  private limit: number;
39	  private inFlightCount = 0;
40	
41	  constructor(options: BulkheadOptions) {
42	    if (!Number.isInteger(options.maxConcurrency) || options.maxConcurrency < 1) {
43	      throw new Error("maxConcurrency must be an integer >= 1");
44	    }
45	    this.maxConcurrency = options.maxConcurrency;
46	    this.minLimit = Math.max(1, Math.floor(options.minLimit ?? 1));
47	    this.additiveIncrease = options.additiveIncrease ?? 1;
48	    this.decreaseFactor = options.decreaseFactor ?? 0.5;
49	    const init = options.initialLimit ?? options.maxConcurrency;
50	    this.limit = Math.min(this.maxConcurrency, Math.max(this.minLimit, Math.floor(init)));
51	  }
52	
53	  /** Current effective admission limit. */
54	  get effectiveLimit(): number {
55	    return Math.min(this.maxConcurrency, this.limit);
56	  }
57	
58	  /** Current adaptive limit (may exceed nothing; informational). */
59	  get adaptiveLimit(): number {
60	    return this.limit;
61	  }
62	
63	  get inFlight(): number {
64	    return this.inFlightCount;
65	  }
66	
67	  /** Try to reserve a slot. Returns true (and increments in-flight) iff admitted. */
68	  tryAcquire(): boolean {
69	    if (this.inFlightCount >= this.effectiveLimit) return false;
70	    this.inFlightCount += 1;
71	    return true;
72	  }
73	
74	  /** Release a slot after a SUCCESS and additively increase the adaptive limit. */
75	  recordSuccess(): void {
76	    this.inFlightCount = Math.max(0, this.inFlightCount - 1);
77	    this.limit = Math.min(this.maxConcurrency, this.limit + this.additiveIncrease);
78	  }
79	
80	  /** Release a slot after a FAILURE and multiplicatively decrease the adaptive limit. */
81	  recordFailure(): void {
82	    this.inFlightCount = Math.max(0, this.inFlightCount - 1);
83	    this.limit = Math.max(this.minLimit, Math.floor(this.limit * this.decreaseFactor));
84	  }
85	}
86

1	/**
2	 * A retry budget (token bucket over RETRIES), the classic guard against retry
3	 * storms (Envoy/Finagle style).
4	 *
5	 * The balance is capped at `maxTokens`. Every top-level request deposits `ratio`
6	 * tokens (so a steady request stream refills the budget at `ratio` retries per
7	 * request); every retry or hedge withdraws `1` token and is only permitted when
8	 * the balance is `>= 1`. With `ratio = 0.2` the system tolerates roughly one
9	 * retry per five requests in steady state , plus an initial cushion of
10	 * `maxTokens` so a brief early burst can still retry.
11	 *
12	 * A withdrawn token can be REFUNDED whole (`refund`) when the retry/hedge it
13	 * authorized turned out to be productive , i.e. the call ultimately succeeded , 
14	 * so that productive retries are not counted as storm load. Refunds are integer
15	 * and capped at `maxTokens` just like deposits.
16	 *
17	 * Provided complete , the client calls `deposit`/`tryWithdraw`/`refund`; it does
18	 * not re-implement budget accounting. Deterministic (no time, no RNG).
19	 */
20	export class RetryBudget {
21	  private balance: number;
22	  private readonly ratio: number;
23	  private readonly maxTokens: number;
24	
25	  constructor(ratio = 0.2, maxTokens = 10) {
26	    if (!(ratio >= 0)) throw new Error("retry budget ratio must be >= 0");
27	    if (!(maxTokens >= 0)) throw new Error("retry budget maxTokens must be >= 0");
28	    this.ratio = ratio;
29	    this.maxTokens = maxTokens;
30	    this.balance = maxTokens; // start with the full cushion
31	  }
32	
33	  /** Record a top-level request: deposit `ratio` tokens, capped at `maxTokens`. */
34	  deposit(): void {
35	    this.balance = Math.min(this.maxTokens, this.balance + this.ratio);
36	  }
37	
38	  /** Try to spend one retry token. Returns true (and debits) iff a retry is permitted. */
39	  tryWithdraw(): boolean {
40	    if (this.balance >= 1) {
41	      this.balance -= 1;
42	      return true;
43	    }
44	    return false;
45	  }
46	
47	  /**
48	   * Refund `n` whole tokens previously withdrawn for retries/hedges that turned
49	   * out to be productive (the call succeeded). Capped at `maxTokens`. `n` defaults
50	   * to 1 and must be a non-negative integer.
51	   */
52	  refund(n = 1): void {
53	    if (!Number.isInteger(n) || n < 0) throw new Error("refund count must be a non-negative integer");
54	    this.balance = Math.min(this.maxTokens, this.balance + n);
55	  }
56	
57	  /** Current balance (observability/tests). */
58	  get tokens(): number {
59	    return this.balance;
60	  }
61	}
62

1	/**
2	 * Exponential backoff with "full jitter".
3	 *
4	 * For retry attempt `n` (0-based: n=0 is the delay before the FIRST retry, i.e.
5	 * after the initial attempt failed), the uncapped exponential delay is
6	 * `base * 2^n`, capped at `maxDelay`. Full jitter then picks a uniform random
7	 * value in `[0, capped]` to avoid thundering-herd synchronization across many
8	 * clients retrying at once.
9	 *
10	 * The randomness is injected (`rng` returns a value in `[0, 1)`) so the delay is
11	 * deterministic in tests. This module is provided complete; the resilient client
12	 * uses {@link backoffDelay} to decide how long to sleep between attempts.
13	 */
14	export function backoffDelay(
15	  attempt: number,
16	  baseDelayMs: number,
17	  maxDelayMs: number,
18	  rng: () => number,
19	): number {
20	  if (attempt < 0) throw new Error("attempt must be >= 0");
21	  const exponential = baseDelayMs * 2 ** attempt;
22	  const capped = Math.min(exponential, maxDelayMs);
23	  // Full jitter: uniform in [0, capped].
24	  const jittered = rng() * capped;
25	  // Whole milliseconds; never negative.
26	  return Math.max(0, Math.floor(jittered));
27	}
28

1	/**
2	 * Injectable clock + sleep + RNG.
3	 *
4	 * The resilient client never reads `Date.now()`, `setTimeout`, or `Math.random`
5	 * directly. Routing all of that through a {@link Clock} keeps the retry/breaker
6	 * timing logic deterministic and unit-testable: tests supply a
7	 * {@link ManualClock} and advance it explicitly instead of waiting in real time.
8	 */
9	export interface Clock {
10	  /** Current time in epoch milliseconds. */
11	  now(): number;
12	  /**
13	   * Resolve after `ms` (logical) milliseconds. If a `signal` is supplied and it
14	   * aborts first, the timer is cancelled and the promise resolves early , so a
15	   * caller that loses a `Promise.race` does not leak a pending timer.
16	   */
17	  sleep(ms: number, signal?: AbortSignal): Promise<void>;
18	  /** Uniform random in [0, 1). Used for jitter. */
19	  random(): number;
20	}
21	
22	export const systemClock: Clock = {
23	  now: () => Date.now(),
24	  sleep: (ms, signal) =>
25	    new Promise((resolve) => {
26	      if (signal?.aborted) {
27	        resolve();
28	        return;
29	      }
30	      const timer = setTimeout(() => {
31	        signal?.removeEventListener("abort", onAbort);
32	        resolve();
33	      }, ms);
34	      const onAbort = (): void => {
35	        clearTimeout(timer);
36	        resolve();
37	      };
38	      signal?.addEventListener("abort", onAbort, { once: true });
39	    }),
40	  random: () => Math.random(),
41	};
42	
43	/**
44	 * A controllable clock for tests. `sleep` does not block real time: a sleeping
45	 * caller is parked until {@link ManualClock.advance} moves the clock past its
46	 * wake time, at which point it resolves on the microtask queue. Jitter is made
47	 * deterministic via an injectable RNG (defaults to always returning 0, i.e. no
48	 * jitter, which is the most test-friendly default).
49	 */
50	export class ManualClock implements Clock {
51	  private current: number;
52	  private rng: () => number;
53	  private waiters: Array<{ at: number; resolve: () => void }> = [];
54	
55	  constructor(start = 0, rng: () => number = () => 0) {
56	    this.current = start;
57	    this.rng = rng;
58	  }
59	
60	  now(): number {
61	    return this.current;
62	  }
63	
64	  random(): number {
65	    return this.rng();
66	  }
67	
68	  sleep(ms: number, signal?: AbortSignal): Promise<void> {
69	    if (ms <= 0 || signal?.aborted) return Promise.resolve();
70	    return new Promise<void>((resolve) => {
71	      const waiter = { at: this.current + ms, resolve };
72	      this.waiters.push(waiter);
73	      if (signal) {
74	        const onAbort = (): void => {
75	          // Drop the parked waiter and resolve early on abort.
76	          this.waiters = this.waiters.filter((w) => w !== waiter);
77	          resolve();
78	        };
79	        signal.addEventListener("abort", onAbort, { once: true });
80	      }
81	    });
82	  }
83	
84	  /** Advance logical time by `ms`, waking any sleepers whose deadline passed. */
85	  async advance(ms: number): Promise<void> {
86	    this.current += ms;
87	    const due = this.waiters.filter((w) => w.at <= this.current);
88	    this.waiters = this.waiters.filter((w) => w.at > this.current);
89	    for (const w of due) w.resolve();
90	    // Let woken continuations run before returning.
91	    await Promise.resolve();
92	  }
93	
94	  /** Number of callers currently parked in {@link sleep}. */
95	  pending(): number {
96	    return this.waiters.length;
97	  }
98	
99	  /**
100	   * The earliest wake time among parked sleepers, or `undefined` if none. Tests
101	   * use this to advance EXACTLY to the next event (no fixed-step overshoot), so
102	   * logical time is bit-for-bit reproducible.
103	   */
104	  nextWakeAt(): number | undefined {
105	    if (this.waiters.length === 0) return undefined;
106	    let min = Infinity;
107	    for (const w of this.waiters) if (w.at < min) min = w.at;
108	    return min;
109	  }
110	}
111

The file /home/user/app/src/resilientClient.ts has been updated successfully. (file state is current in your context , no need to Read it back)

> resilient-http@2.1.0 typecheck
> tsc -p tsconfig.json --noEmit


> resilient-http@2.1.0 test
> vitest run


 RUN  v1.6.0 /home/user/app

 ✓ test/circuitBreaker.test.ts  (7 tests) 4ms
 ✓ test/smoke.test.ts  (2 tests) 4ms
 ✓ test/bulkhead.test.ts  (3 tests) 2ms
 ✓ test/retryBudget.test.ts  (3 tests) 3ms

 Test Files  4 passed (4)
      Tests  15 passed (15)
   Start at  20:25:11
   Duration  1.03s (transform 187ms, setup 0ms, collect 248ms, tests 13ms, environment 1ms, prepare 320ms)