diff --git a/README.md b/README.md index e8b64df..3b8e8cb 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ SDK to access [Zenrows](https://www.zenrows.com/) API directly from Node.js. Zen - [Installation](#installation) - [Usage](#usage) + - [Retries](#retries) - [Adaptive Stealth Mode](#adaptive-stealth-mode) - [POST Requests](#post-requests) - [Extract](#extract) @@ -49,7 +50,7 @@ Start using the API by [creating your API Key](https://app.zenrows.com/register? The SDK uses the official [fetch api](https://nodejs.org/dist/latest-v18.x/docs/api/globals.html) for HTTP requests. The client's response will be a [`Response`](https://nodejs.org/dist/latest-v18.x/docs/api/globals.html#response). -It also uses [fetch-retry](https://github.com/jonbern/fetch-retry) to automatically retry failed requests (network errors and status codes 422, 503 and 504, except Extract's `422 no_data_extracted`). Retries are not active by default; you need to specify the number of retries, as shown below. It already includes an exponential back-off retry delay between failed requests. +The client retries failed requests by default. See [Retries](#retries) for what it retries and how long it waits. > `client.fetch()` is the primary method for the main page-scraping product. `client.get()` still works and is kept as a deprecated alias — new code should call `fetch()`. @@ -131,6 +132,44 @@ You can also pass optional parameters and headers; the list above is a reference Sending headers to the target URL will overwrite our defaults. Be careful when doing it and contact us if there is any problem. +### Retries + +`fetch()`, `get()` and `extract()` retry failed requests through [fetch-retry](https://github.com/jonbern/fetch-retry). `post()` never retries. `client.batch` and `client.crawl` keep their own retry rules, described in their sections. `prepareDomain()` and `preparedDomainStatus()` never retry: they send one request and reject on an error answer (see [Preparing a domain](#preparing-a-domain)). + +What the client retries: + +1. Status codes 429, 502, 503 and 504. The API does not charge for them. +2. A `422`, except one with `code: "no_data_extracted"`. That one is a final answer (see [Extract](#extract)). +3. A failure before the request reached the API: connection refused, a name that does not resolve, an unreachable network or host, or a connect timeout. +4. A connection that timed out while connecting. When Node tries several addresses, every try must fail this way or as in item 3. + +What the client never retries: + +1. Any other status, `500` included. +2. A connection reset or close after the request was sent, a response timeout, or an abort. The API may still finish that request and charge it. +3. An error in the request itself, such as an invalid header value. +4. Any status when you set `original_status`, and a status you list in `allowed_status_codes`. That status comes from the target site, not the API. A refused connection is still retried. + +Which calls the client retries: + +1. `fetch()`, `get()` and `extract()`: always. +2. `post()`: never. The client sends a `post()` once, whatever its headers, the status or the failure. The API forwards the body to the target, and the target may act on it twice. +3. The API does not dedupe a `post()` on an `Idempotency-Key` header, so the client does not retry on one. +4. If you need to retry a `post()`, write the retry yourself. + +How long the client waits between tries: + +1. When the API sends `Retry-After` in whole seconds, the client waits that long, at least 0.25 seconds and at most 30 seconds. `retryMaxWait` does not change this cap. The client ignores any other `Retry-After` format, such as an HTTP date. +2. Otherwise it waits a random time between 0.25 seconds and `max(0.25, min(retryMaxWait, 2^attempt))` seconds (full jitter). `attempt` counts from 0, so the first retry waits 0.25 to 1 second, the second 0.25 to 2 seconds. + +Options, checked when you build the client (an invalid value throws a `RangeError`): + +1. `retries`: retries after the first try, an integer of 0 or more. Default `2`. Set `retries: 0` to send every request once. +2. `retryMaxWait`: the longest one random wait may be, in seconds, greater than 0. Default `30`. +3. `retryBudget`: in seconds, greater than 0. Default `90`: no retry starts after 90 s. The client counts from the start of the first try, and includes the wait before the retry. The budget does not cut a running try short, so a call can take longer than the budget by one try. + +The client takes no `AbortSignal`, so you cannot cancel a call while it waits between tries. To bound a call's time, lower `retries` or `retryBudget`. + ### Adaptive Stealth Mode Set `mode: "auto"` to let Zenrows pick the request configuration for you — it starts with the cheapest viable setup and escalates to `js_render`/`premium_proxy` only when the target needs it, billing only for the configuration that succeeds. @@ -145,6 +184,8 @@ Compatible with `proxy_country`, `js_instructions`, and `custom_headers`. The SDK also offers POST requests by calling the `client.post` method. It can receive a new parameter `data` that represents the data sent in, for example, a form. +The client sends a `post()` once and never retries it, whatever `retries` says. See [Retries](#retries). + ```javascript const { ZenRows } = require("zenrows"); @@ -152,7 +193,7 @@ const apiKey = "YOUR-API-KEY"; const url = "https://httpbin.org/anything"; (async () => { - const client = new ZenRows(apiKey, { retries: 1 }); + const client = new ZenRows(apiKey); const response = await client.post( url, diff --git a/src/index.ts b/src/index.ts index 516ce8f..b00dea4 100644 --- a/src/index.ts +++ b/src/index.ts @@ -8,6 +8,14 @@ import { checkDomain, preparedDomainsRequest, } from "./extract/prepared-domains.js"; +import { + DEFAULT_RETRIES, + DEFAULT_RETRY_BUDGET_SECONDS, + DEFAULT_RETRY_MAX_WAIT_SECONDS, + isRetryEligible, + retryPolicy, + validateRetryOptions, +} from "./retry.js"; export * from "./batch/client.js"; export * from "./batch/errors.js"; @@ -24,6 +32,7 @@ export { type PreparedDomainStatus, ZenRowsPreparedDomainError, } from "./extract/prepared-domains.js"; +export { isNoDataExtracted } from "./retry.js"; const API_URL = "https://api.zenrows.com/v1/"; @@ -61,23 +70,22 @@ function warnFallbackDeprecated(): void { ); } -/** True when a response is Extract's 422 `no_data_extracted`: Extract knows this kind - * of page but found no data on it. The request is not charged and the client never - * retries it, since another try returns the same answer. Reads via `.clone()`, so the - * original response body is left unconsumed for the caller. */ -export async function isNoDataExtracted(response: Response): Promise { - if (response.status !== 422) return false; - try { - const body = (await response.clone().json()) as { code?: unknown }; - return typeof body?.code === "string" && body.code.toLowerCase() === "no_data_extracted"; - } catch { - return false; - } -} - interface ClientConfig { concurrency?: number; + /** + * Retries for `fetch()` and `extract()` after a 429, 502, 503 or 504, a 422 other than + * `no_data_extracted`, or a failure before the request reached the server. An integer of 0 or + * more; default 2; 0 turns retries off. A reset after the request was sent, a timeout or an + * abort is never retried: the server may still finish the request and charge it. `post()` is + * never retried, whatever its headers: the API does not dedupe on `Idempotency-Key`. No status + * is retried when the request sets `original_status`, and no status listed in + * `allowed_status_codes` is: that status comes from the target site. + */ retries?: number; + /** The longest one random wait may be, in seconds (greater than 0). Default 30. It never caps a `Retry-After`. */ + retryMaxWait?: number; + /** No retry starts after this many seconds since the first try started (greater than 0). Default 90. */ + retryBudget?: number; /** Config for `client.crawl`: its own `baseURL`, `retries` and `timeout`. */ crawl?: CrawlClientConfig; } @@ -119,35 +127,16 @@ export class ZenRows { readonly crawl: ZenRowsCrawlClient; constructor(apiKey: string, clientConfig: ClientConfig = {}) { + validateRetryOptions(clientConfig); this.apiKey = apiKey; this.clientConfig = clientConfig; this.batch = new ZenRowsBatchClient(apiKey); this.crawl = new ZenRowsCrawlClient(apiKey, clientConfig.crawl); - const retries = this.clientConfig.retries ?? 0; this.queue = fastq.promise(this, this.worker, this.clientConfig.concurrency ?? 5); - this.fetchWithRetry = fetchRetry(fetch, { - retryDelay: (attempt) => 2 ** attempt * 1000, - // retryOn: [422, 503, 504], - retryOn: async (attempt, error, response) => { - if (attempt >= retries) { - return false; - } - - if (error !== null || response?.status === 503 || response?.status === 504) { - return true; - } - - // A 422 is usually a failure to get the target's content and worth another - // try, except Extract's `no_data_extracted`, which is a final answer. - if (response?.status === 422) { - return !(await isNoDataExtracted(response)); - } - - return false; - }, - }); + // The retry policy is per request (see `worker`), so this only wraps fetch. + this.fetchWithRetry = fetchRetry(fetch); } /** @@ -354,7 +343,20 @@ export class ZenRows { } } - const response = await this.fetchWithRetry(`${API_URL}?${params.toString()}`, fetchOptions); + // A POST, or a body that cannot be sent again, goes out once: it is not safe to send twice. + const eligible = isRetryEligible(method, fetchOptions.body); + const { retryOn, retryDelay } = retryPolicy({ + retries: eligible ? this.clientConfig.retries ?? DEFAULT_RETRIES : 0, + // A status the target site sent (original_status, allowed_status_codes) is not retried. + params: config, + retryMaxWait: this.clientConfig.retryMaxWait ?? DEFAULT_RETRY_MAX_WAIT_SECONDS, + retryBudget: this.clientConfig.retryBudget ?? DEFAULT_RETRY_BUDGET_SECONDS, + }); + const response = await this.fetchWithRetry(`${API_URL}?${params.toString()}`, { + ...fetchOptions, + retryOn, + retryDelay, + }); return response; } diff --git a/src/retry.ts b/src/retry.ts new file mode 100644 index 0000000..54b0c66 --- /dev/null +++ b/src/retry.ts @@ -0,0 +1,234 @@ +// Retry policy for `fetch()` and `extract()`. `post()` goes through it too, but is never retried. +// The same policy ships in every ZenRows client: retry only answers that say "try again" and are +// never charged, retry only requests that are safe to send twice, back off with full jitter, and +// start no retry after the budget. A status the target site sent is never retried. + +/** Statuses retried: rate limited, or the gateway or its upstream could not answer. Never charged. */ +export const RETRYABLE_STATUSES: ReadonlySet = new Set([429, 502, 503, 504]); + +/** + * `error.cause.code` values that mean the request never reached the server: the connection was + * refused, the name did not resolve, the network or host was unreachable, or the connect timed out. + * Anything else (a reset or close after the request was written, a header or body timeout, an + * abort, a bad header value) is not retried: the server may have run the request already. + */ +const PRE_SEND_ERROR_CODES: ReadonlySet = new Set([ + "ECONNREFUSED", + "ENOTFOUND", + "EAI_AGAIN", + "ENETUNREACH", + "EHOSTUNREACH", + "UND_ERR_CONNECT_TIMEOUT", +]); + +/** + * The only methods the client retries. A POST, PUT, PATCH or DELETE is sent once, whatever its + * headers: the API does not dedupe on `Idempotency-Key`. It only forwards it to the target. + */ +const IDEMPOTENT_METHODS: ReadonlySet = new Set(["GET", "HEAD", "OPTIONS"]); + +export const DEFAULT_RETRIES = 2; +/** The longest one backoff wait may be, in seconds. */ +export const DEFAULT_RETRY_MAX_WAIT_SECONDS = 30; +/** No retry starts once the time spent plus the next wait would pass this, in seconds. */ +export const DEFAULT_RETRY_BUDGET_SECONDS = 90; + +const BACKOFF_MIN_MS = 250; +const BACKOFF_BASE_MS = 1000; +// A Retry-After is clamped to this range. The cap is fixed and does not depend on retryMaxWait. +const RETRY_AFTER_MIN_MS = 250; +const RETRY_AFTER_MAX_MS = 30_000; + +/** + * The request params that make a status the target site's own answer, not the API's. The client + * reads them in the form it sends them: `String(value)`, so an array goes out comma-separated. + */ +export interface TargetStatusParams { + /** When on (`true`, `"true"`, `1` or `"1"`), the API returns the target's status: none is retried. */ + original_status?: unknown; + /** Statuses, comma-separated or as an array, that the API passes through from the target. */ + allowed_status_codes?: unknown; +} + +/** + * Builds the test for "this status came from the target site", from the request params. A + * status from the target is never retried: another try asks the same site the same question. + */ +export function targetStatusRule(params: TargetStatusParams = {}): (status: number) => boolean { + const originalStatus = params.original_status; + if (originalStatus !== undefined && originalStatus !== null) { + const wire = String(originalStatus).trim().toLowerCase(); + if (wire === "true" || wire === "1") return () => true; + } + const allowed = new Set(); + const listed = params.allowed_status_codes; + if (listed !== undefined && listed !== null) { + for (const part of String(listed).split(",")) { + const code = part.trim(); + if (/^\d+$/.test(code)) allowed.add(Number(code)); + } + } + return (status) => allowed.has(status); +} + +export interface RetryPolicyOptions { + /** Retries after the first try. 0 turns retries off. */ + retries: number; + /** The request params `original_status` and `allowed_status_codes` (see `targetStatusRule`). */ + params?: TargetStatusParams; + /** The longest one jitter wait may be, in seconds. It never caps a `Retry-After`. */ + retryMaxWait: number; + /** No retry starts once the time spent plus the next wait would pass this, in seconds. */ + retryBudget: number; + /** Clock in milliseconds; tests replace it. */ + now?: () => number; + /** Uniform random number in [0, 1); tests replace it. */ + random?: () => number; +} + +/** Throws a `RangeError` for a retry option the policy cannot honour, instead of clamping it. */ +export function validateRetryOptions(options: { + retries?: unknown; + retryMaxWait?: unknown; + retryBudget?: unknown; +}): void { + const { retries, retryMaxWait, retryBudget } = options; + if (retries !== undefined && !(Number.isInteger(retries) && (retries as number) >= 0)) { + throw new RangeError(`retries must be an integer of 0 or more, got ${String(retries)}`); + } + for (const [name, value] of [ + ["retryMaxWait", retryMaxWait], + ["retryBudget", retryBudget], + ] as const) { + if (value !== undefined && !(Number.isFinite(value) && (value as number) > 0)) { + throw new RangeError( + `${name} must be a finite number of seconds greater than 0, got ${String(value)}`, + ); + } + } +} + +/** True when a response is Extract's 422 `no_data_extracted`: Extract knows this kind + * of page but found no data on it. The request is not charged and the client never + * retries it, since another try returns the same answer. Reads via `.clone()`, so the + * original response body is left unconsumed for the caller. */ +export async function isNoDataExtracted(response: Response): Promise { + if (response.status !== 422) return false; + try { + const body = (await response.clone().json()) as { code?: unknown }; + return typeof body?.code === "string" && body.code.toLowerCase() === "no_data_extracted"; + } catch { + return false; + } +} + +/** True when the body is none, a string or bytes, so it can be sent again. A stream cannot. */ +function isReplayableBody(body: unknown): boolean { + return ( + body === undefined || + body === null || + typeof body === "string" || + body instanceof ArrayBuffer || + ArrayBuffer.isView(body) + ); +} + +/** + * True when a request is safe to send more than once: a GET, HEAD or OPTIONS whose body is none, + * a string or bytes. Any other method, or a body such as a stream, is sent once. + */ +export function isRetryEligible(method: string, body: unknown): boolean { + return IDEMPOTENT_METHODS.has(method.toUpperCase()) && isReplayableBody(body); +} + +interface ErrorCause { + code?: unknown; + syscall?: unknown; + errors?: unknown; +} + +/** True when one error says the connection was never made: a listed code, or a connect timeout. */ +function isConnectFailure(cause: ErrorCause | null | undefined): boolean { + const code = cause?.code; + if (typeof code !== "string") return false; + return PRE_SEND_ERROR_CODES.has(code) || (code === "ETIMEDOUT" && cause?.syscall === "connect"); +} + +/** + * True when fetch failed before the request was written to the server (see `PRE_SEND_ERROR_CODES`). + * With happy eyeballs, Node tries each address family and throws an `AggregateError` with one + * error per try. Its own `code` is the first try's, so the inner errors decide: every one must + * be a connect failure. An `ETIMEDOUT` from any syscall but `connect` is a read timeout. + */ +function isPreSendFailure(error: unknown): boolean { + const cause = (error as { cause?: ErrorCause } | null)?.cause; + const inner = cause?.errors; + if (Array.isArray(inner)) { + return inner.length > 0 && inner.every((each) => isConnectFailure(each as ErrorCause)); + } + return isConnectFailure(cause); +} + +/** True when the answer says "try again": 429, 502, 503, 504, or a 422 other than `no_data_extracted`. */ +async function isRetryableResponse(response: Response): Promise { + if (RETRYABLE_STATUSES.has(response.status)) return true; + if (response.status === 422) return !(await isNoDataExtracted(response)); + return false; +} + +/** Milliseconds from a `Retry-After` in whole seconds, clamped to [0.25 s, 30 s]. Undefined otherwise (an HTTP date too). */ +export function retryAfterMs(response: Response | null): number | undefined { + const raw = response?.headers.get("Retry-After")?.trim(); + if (!raw || !/^\d+$/.test(raw)) return undefined; + return Math.min(Math.max(Number(raw) * 1000, RETRY_AFTER_MIN_MS), RETRY_AFTER_MAX_MS); +} + +/** Full jitter for retry `attempt` (0-based): uniform(0.25 s, max(0.25 s, min(maxWait, 1 s * 2^attempt))). */ +export function backoffMs(attempt: number, maxWaitMs: number, random: () => number): number { + const ceiling = Math.max(BACKOFF_MIN_MS, Math.min(maxWaitMs, BACKOFF_BASE_MS * 2 ** attempt)); + return BACKOFF_MIN_MS + random() * (ceiling - BACKOFF_MIN_MS); +} + +/** + * The `retryOn` and `retryDelay` pair for one request through fetch-retry. Build one per + * request: it remembers when the first try started and the wait it chose in `retryOn`, which + * fetch-retry then reads through `retryDelay`. + */ +export function retryPolicy(options: RetryPolicyOptions) { + const now = options.now ?? Date.now; + const random = options.random ?? Math.random; + const maxWaitMs = options.retryMaxWait * 1000; + const budgetMs = options.retryBudget * 1000; + const isTargetStatus = targetStatusRule(options.params); + const startedAt = now(); + let nextDelayMs = 0; + + return { + async retryOn( + attempt: number, + error: Error | null, + response: Response | null, + ): Promise { + if (attempt >= options.retries) return false; + if (error) { + if (!isPreSendFailure(error)) return false; + } else if ( + !response || + isTargetStatus(response.status) || + !(await isRetryableResponse(response)) + ) { + return false; + } + + const delay = retryAfterMs(response) ?? backoffMs(attempt, maxWaitMs, random); + if (now() - startedAt + delay > budgetMs) return false; + nextDelayMs = delay; + // The retried answer is dropped; free its connection instead of waiting for GC. + response?.body?.cancel().catch(() => {}); + return true; + }, + retryDelay(): number { + return nextDelayMs; + }, + }; +} diff --git a/tests/index.test.ts b/tests/index.test.ts index 7fa19d7..92952b3 100644 --- a/tests/index.test.ts +++ b/tests/index.test.ts @@ -25,12 +25,12 @@ describe("ZenRows Client Get", () => { expect(clientSpy).toHaveBeenCalledWith( `${apiUrl}?url=${encodeURIComponent(url)}&apikey=${apiKey}`, - { + expect.objectContaining({ method: "GET", headers: { "User-Agent": `zenrows/${packageJson.version} node`, }, - }, + }), ); expect(response.status).toBe(200); }); @@ -64,12 +64,12 @@ describe("ZenRows Client Get", () => { expect(clientSpy).toHaveBeenCalledWith( `${apiUrl}?url=${encodeURIComponent(url)}&apikey=${apiKey}&custom_headers=true`, - { + expect.objectContaining({ method: "GET", headers: { "User-Agent": "test", }, - }, + }), ); }); diff --git a/tests/retries.test.ts b/tests/retries.test.ts index 0f3782d..9f1ba40 100644 --- a/tests/retries.test.ts +++ b/tests/retries.test.ts @@ -1,82 +1,652 @@ +import { Readable } from "node:stream"; import { http, HttpResponse } from "msw"; -import { type Mock, beforeEach, describe, expect, test, vi } from "vitest"; //TODO(Nestor): Try to use globals instead of importing -import { ZenRows } from "../src"; +import { afterEach, beforeEach, describe, expect, test, vi } from "vitest"; +import { ZenRows, ZenRowsPreparedDomainError } from "../src"; +import { backoffMs, isRetryEligible, retryAfterMs, retryPolicy } from "../src/retry"; import { server } from "./_setup"; -describe("ZenRows Client with Retries", () => { - const apiKey = "API_KEY"; - const url = "https://example.com"; - const apiUrl = "https://api.zenrows.com/v1/"; - let client: ZenRows; +const apiUrl = "https://api.zenrows.com/v1/"; +const url = "https://example.com"; - beforeEach(() => { - client = new ZenRows(apiKey, { retries: 2 }); +/** A policy on a fake clock, so budget checks can be driven by hand. */ +function policyAt( + clock: { ms: number }, + overrides: Partial[0]> = {}, +) { + return retryPolicy({ + retries: 10, + retryMaxWait: 30, + retryBudget: 90, + now: () => clock.ms, + random: () => 0.5, + ...overrides, }); +} - test( - "should call fetch 3 times (initial plus 2 retries)", - { timeout: Number.POSITIVE_INFINITY }, - async () => { - const clientSpy = vi.spyOn(client, "fetchWithRetry"); - server.use( - http.get("https://api.zenrows.com/v1/", () => { - return HttpResponse.error(); - }), - ); +const status = (code: number, headers: Record = {}) => + new Response(null, { status: code, headers }); - try { - await client.get(url); - } catch (error) { - return; - } +/** The error undici throws: `TypeError('fetch failed')` with the cause carrying the code. */ +const fetchFailed = (code: string) => + new TypeError("fetch failed", { cause: Object.assign(new Error(code), { code }) }); + +/** A Node system error, as a socket raises it: a `code` and the `syscall` that failed. */ +const systemError = (code: string, syscall?: string) => + Object.assign(new Error(`${syscall ?? "?"} ${code}`), { code, syscall }); + +/** + * `fetch failed` caused by an `AggregateError`, as Node throws it when every address it tried + * failed. Two shapes: Node copies the first inner error's `code` onto the aggregate, and a + * runtime that does not leaves it off. The policy must judge both by the inner errors alone. + */ +const aggregateFailed = (errors: Error[]) => { + const withCode = Object.assign(new AggregateError(errors), { + code: (errors[0] as { code?: string } | undefined)?.code, + }); + return [ + new TypeError("fetch failed", { cause: withCode }), + new TypeError("fetch failed", { cause: new AggregateError(errors) }), + ]; +}; + +/** The highest value `Math.random()` can return. */ +const RANDOM_MAX = 1 - Number.EPSILON; + +describe("conformance vectors", () => { + test.each([ + ["5", 5000], + ["0", 250], + [" 7 ", 7000], + ["99999", 30_000], + ])("Retry-After %j waits %i ms", (value, expected) => { + expect(retryAfterMs(status(503, { "Retry-After": value }))).toBe(expected); + const policy = policyAt({ ms: 0 }); + return policy.retryOn(0, null, status(503, { "Retry-After": value })).then((retried) => { + expect(retried).toBe(true); + expect(policy.retryDelay()).toBe(expected); + }); + }); - expect(clientSpy).toHaveBeenCalledTimes(3); + test.each(["1.5", "-1", "1e3", "0x10", "", "abc", "Wed, 21 Oct 2026 07:28:00 GMT"])( + "Retry-After %j is ignored and the wait is jitter", + async (value) => { + expect(retryAfterMs(status(503, { "Retry-After": value }))).toBeUndefined(); + const policy = policyAt({ ms: 0 }, { random: () => 0 }); + await policy.retryOn(0, null, status(503, { "Retry-After": value })); + expect(policy.retryDelay()).toBe(250); }, ); - test( - "should throw error with status code 500", - { timeout: Number.POSITIVE_INFINITY }, - async () => { - const clientSpy = vi.spyOn(client, "get"); - server.use( - http.get("https://api.zenrows.com/v1/", () => { - return new HttpResponse(null, { status: 503 }); - }), - ); + test.each([ + { attempt: 0, maxWaitMs: 30_000, high: 1000 }, + { attempt: 1, maxWaitMs: 30_000, high: 2000 }, + { attempt: 2, maxWaitMs: 30_000, high: 4000 }, + { attempt: 5, maxWaitMs: 30_000, high: 30_000 }, + { attempt: 9, maxWaitMs: 30_000, high: 30_000 }, + { attempt: 4, maxWaitMs: 5000, high: 5000 }, + { attempt: 3, maxWaitMs: 100, high: 250 }, + ])( + "jitter for attempt $attempt with retryMaxWait $maxWaitMs ms is [250, $high] ms", + ({ attempt, maxWaitMs, high }) => { + expect(backoffMs(attempt, maxWaitMs, () => 0)).toBe(250); + expect(backoffMs(attempt, maxWaitMs, () => RANDOM_MAX)).toBeCloseTo(high, 6); + }, + ); + + test.each([429, 502, 503, 504])("a %i is retried", async (code) => { + expect(await policyAt({ ms: 0 }).retryOn(0, null, status(code))).toBe(true); + }); + + test.each([200, 400, 401, 402, 403, 404, 409, 500])("a %i is not retried", async (code) => { + expect(await policyAt({ ms: 0 }).retryOn(0, null, status(code))).toBe(false); + }); - try { - const response = await client.get(url); - expect(clientSpy).toHaveBeenCalledTimes(3); - expect(response.status).toBe(503); - } catch (error) { - return; + test("a 422 is retried unless its code is no_data_extracted", async () => { + const noData = Response.json({ code: "no_data_extracted" }, { status: 422 }); + expect(await policyAt({ ms: 0 }).retryOn(0, null, noData)).toBe(false); + expect(await noData.json()).toEqual({ code: "no_data_extracted" }); + + const other = Response.json({ code: "RESP001" }, { status: 422 }); + expect(await policyAt({ ms: 0 }).retryOn(0, null, other)).toBe(true); + const noBody = status(422); + expect(await policyAt({ ms: 0 }).retryOn(0, null, noBody)).toBe(true); + }); + + test.each(["ECONNREFUSED", "ENOTFOUND", "EAI_AGAIN", "ENETUNREACH", "EHOSTUNREACH"])( + "a failure before the request was sent (%s) is retried", + async (code) => { + expect(await policyAt({ ms: 0 }).retryOn(0, fetchFailed(code), null)).toBe(true); + }, + ); + + test("a connect timeout is retried", async () => { + const error = fetchFailed("UND_ERR_CONNECT_TIMEOUT"); + expect(await policyAt({ ms: 0 }).retryOn(0, error, null)).toBe(true); + }); + + test("a plain connect timeout (ETIMEDOUT from connect) is retried", async () => { + const error = new TypeError("fetch failed", { cause: systemError("ETIMEDOUT", "connect") }); + expect(await policyAt({ ms: 0 }).retryOn(0, error, null)).toBe(true); + }); + + test.each([ + ["from read", "read"], + ["with no syscall", undefined], + ])("a plain ETIMEDOUT %s is not retried", async (_label, syscall) => { + const error = new TypeError("fetch failed", { cause: systemError("ETIMEDOUT", syscall) }); + expect(await policyAt({ ms: 0 }).retryOn(0, error, null)).toBe(false); + }); + + describe("a happy-eyeballs AggregateError", () => { + const refused = () => systemError("ECONNREFUSED", "connect"); + const connectTimeout = () => systemError("ETIMEDOUT", "connect"); + + test.each([ + ["refused on every address", [refused(), refused()]], + ["a connect timeout on every address", [connectTimeout(), connectTimeout()]], + ["refused on one address, a connect timeout on the other", [refused(), connectTimeout()]], + ])("%s is retried", async (_label, errors) => { + for (const error of aggregateFailed(errors)) { + expect(await policyAt({ ms: 0 }).retryOn(0, error, null)).toBe(true); + } + }); + + test.each([ + ["a read timeout on one address", [systemError("ETIMEDOUT", "read"), refused()]], + ["a reset on one address", [refused(), systemError("ECONNRESET", "read")]], + ["no inner error", []], + ])("%s is not retried", async (_label, errors) => { + for (const error of aggregateFailed(errors)) { + expect(await policyAt({ ms: 0 }).retryOn(0, error, null)).toBe(false); + } + }); + }); + + test.each([ + ["a socket closed after the request was sent", fetchFailed("UND_ERR_SOCKET")], + ["a connection reset", fetchFailed("ECONNRESET")], + ["a response-header timeout", fetchFailed("UND_ERR_HEADERS_TIMEOUT")], + ["a body timeout", fetchFailed("UND_ERR_BODY_TIMEOUT")], + ["an abort", new DOMException("The operation was aborted.", "AbortError")], + ["a timeout signal", new DOMException("The operation timed out.", "TimeoutError")], + ["a TypeError with no cause", new TypeError('Headers.append: "a\\nb" is an invalid value.')], + ])("%s is not retried", async (_label, error) => { + expect(await policyAt({ ms: 0 }).retryOn(0, error, null)).toBe(false); + }); + + test("a GET, HEAD or OPTIONS with no body, a string or bytes is eligible", () => { + const bodies = [ + undefined, + null, + "", + "{}", + new Uint8Array([1]), + Buffer.from("a"), + new ArrayBuffer(1), + ]; + for (const method of ["GET", "HEAD", "OPTIONS", "get"]) { + for (const body of bodies) { + expect(isRetryEligible(method, body)).toBe(true); + } + } + }); + + test("a POST, PUT, PATCH or DELETE is never eligible, whatever its body", () => { + for (const method of ["POST", "PUT", "PATCH", "DELETE", "post"]) { + for (const body of [undefined, "{}", new Uint8Array([1])]) { + expect(isRetryEligible(method, body)).toBe(false); + } + } + }); + + test("a GET with a one-shot body (a stream, a reader or a generator) is not eligible", () => { + async function* generator() { + yield "a"; + } + const oneShot = [ + new ReadableStream(), + new ReadableStream().getReader(), + Readable.from(["a"]), + generator(), + ]; + for (const body of oneShot) { + expect(isRetryEligible("GET", body)).toBe(false); + } + }); +}); + +describe("a status from the target site is not retried", () => { + const retriedCodes = [429, 502, 503, 504, 422]; + + test.each([true, "true", "TRUE", " true ", 1, "1"])( + "original_status %j: no status is retried", + async (on) => { + for (const code of retriedCodes) { + const policy = policyAt({ ms: 0 }, { params: { original_status: on } }); + expect(await policy.retryOn(0, null, status(code))).toBe(false); } }, ); - test( - "should fail the first attempt and succeed the second", - { timeout: Number.POSITIVE_INFINITY }, - async () => { - server.use( - http.get( - "https://api.zenrows.com/v1/", - () => { - return new HttpResponse(null, { status: 503 }); - }, - { - once: true, + test.each([false, "false", 0, "0", "", null, undefined])( + "original_status %j: the usual statuses are retried", + async (off) => { + for (const code of retriedCodes) { + const policy = policyAt({ ms: 0 }, { params: { original_status: off } }); + expect(await policy.retryOn(0, null, status(code))).toBe(true); + } + }, + ); + + test.each([ + ["the string '503'", "503"], + ["a list with spaces", "404, 503"], + ["an array of strings", ["404", "503"]], + ["an array of numbers", [404, 503]], + ["a number", 503], + ])("allowed_status_codes as %s: a 503 is final, a 502 is retried", async (_label, listed) => { + const params = { allowed_status_codes: listed }; + expect(await policyAt({ ms: 0 }, { params }).retryOn(0, null, status(503))).toBe(false); + expect(await policyAt({ ms: 0 }, { params }).retryOn(0, null, status(502))).toBe(true); + }); + + test("allowed_status_codes '422,429': a listed 422 or 429 is final", async () => { + const params = { allowed_status_codes: "422,429" }; + const other = Response.json({ code: "RESP001" }, { status: 422 }); + expect(await policyAt({ ms: 0 }, { params }).retryOn(0, null, other)).toBe(false); + expect(await policyAt({ ms: 0 }, { params }).retryOn(0, null, status(429))).toBe(false); + expect(await policyAt({ ms: 0 }, { params }).retryOn(0, null, status(503))).toBe(true); + }); + + test.each([ + ["original_status", { original_status: true }], + ["allowed_status_codes", { allowed_status_codes: "503" }], + ])("a refused connection is still retried with %s", async (_label, params) => { + const policy = policyAt({ ms: 0 }, { params }); + expect(await policy.retryOn(0, fetchFailed("ECONNREFUSED"), null)).toBe(true); + }); +}); + +describe("retryPolicy", () => { + test("stops after `retries` retries", async () => { + const policy = policyAt({ ms: 0 }, { retries: 2 }); + expect(await policy.retryOn(0, null, status(503))).toBe(true); + expect(await policy.retryOn(1, null, status(503))).toBe(true); + expect(await policy.retryOn(2, null, status(503))).toBe(false); + }); + + test("retryMaxWait caps the jitter ceiling", async () => { + const policy = policyAt({ ms: 0 }, { retryMaxWait: 2, random: () => RANDOM_MAX }); + await policy.retryOn(5, null, status(503)); + expect(policy.retryDelay()).toBeLessThanOrEqual(2000); + }); + + test("retryMaxWait never caps a Retry-After", async () => { + const policy = policyAt({ ms: 0 }, { retryMaxWait: 5 }); + await policy.retryOn(0, null, status(503, { "Retry-After": "20" })); + expect(policy.retryDelay()).toBe(20_000); + }); + + test("starts no retry that would pass the budget", async () => { + const clock = { ms: 0 }; + const policy = policyAt(clock, { retryBudget: 10 }); + clock.ms = 3000; + expect(await policy.retryOn(0, null, status(503, { "Retry-After": "7" }))).toBe(true); + clock.ms = 3001; + expect(await policy.retryOn(1, null, status(503, { "Retry-After": "7" }))).toBe(false); + }); + + test("cancels the body of a response it retries, and leaves a final one unread", async () => { + const cancelled: string[] = []; + const response = (name: string, code: number) => + new Response( + new ReadableStream({ + cancel() { + cancelled.push(name); }, - ), + }), + { status: code }, ); - try { - const response = await client.get(url); - expect(response).toBe(200); - } catch (error) { - return; - } + const policy = policyAt({ ms: 0 }, { retries: 1 }); + expect(await policy.retryOn(0, null, response("retried", 503))).toBe(true); + expect(await policy.retryOn(1, null, response("last", 503))).toBe(false); + expect(await policy.retryOn(0, null, response("final", 400))).toBe(false); + await new Promise((resolve) => setTimeout(resolve, 0)); + + expect(cancelled).toEqual(["retried"]); + }); +}); + +describe("ZenRows client retries", () => { + beforeEach(() => { + vi.useFakeTimers({ toFake: ["setTimeout", "Date"] }); + vi.spyOn(Math, "random").mockReturnValue(0); + }); + + afterEach(() => { + vi.useRealTimers(); + vi.restoreAllMocks(); + vi.unstubAllGlobals(); + }); + + test("a 503 is retried after a jittered wait, then succeeds", async () => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return calls === 1 ? new HttpResponse(null, { status: 503 }) : HttpResponse.json({ ok: 1 }); + }), + ); + + const pending = new ZenRows("API_KEY").fetch(url); + await vi.advanceTimersByTimeAsync(249); + expect(calls).toBe(1); + await vi.advanceTimersByTimeAsync(1); + const response = await pending; + + expect(response.status).toBe(200); + expect(calls).toBe(2); + }); + + test("waits the Retry-After the server sends", async () => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return calls === 1 + ? new HttpResponse(null, { status: 429, headers: { "Retry-After": "5" } }) + : HttpResponse.json({ ok: 1 }); + }), + ); + + const pending = new ZenRows("API_KEY").fetch(url); + await vi.advanceTimersByTimeAsync(4999); + expect(calls).toBe(1); + await vi.advanceTimersByTimeAsync(1); + + expect((await pending).status).toBe(200); + expect(calls).toBe(2); + }); + + test("waits a Retry-After of 20 s in full, even with retryMaxWait 5", async () => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return calls === 1 + ? new HttpResponse(null, { status: 503, headers: { "Retry-After": "20" } }) + : HttpResponse.json({ ok: 1 }); + }), + ); + + const pending = new ZenRows("API_KEY", { retryMaxWait: 5 }).fetch(url); + await vi.advanceTimersByTimeAsync(19_999); + expect(calls).toBe(1); + await vi.advanceTimersByTimeAsync(1); + + expect((await pending).status).toBe(200); + expect(calls).toBe(2); + }); + + test("returns the last answer once the budget would be passed", async () => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return new HttpResponse(null, { status: 503, headers: { "Retry-After": "20" } }); + }), + ); + + const pending = new ZenRows("API_KEY", { retries: 5, retryBudget: 45 }).fetch(url); + await vi.advanceTimersByTimeAsync(60_000); + const response = await pending; + + // Waits 20 s, then 20 s more (40 s spent); a third wait would end at 60 s, past 45 s. + expect(response.status).toBe(503); + expect(calls).toBe(3); + }); + + test("a 400 is not retried", async () => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return HttpResponse.json({ code: "REQS001" }, { status: 400 }); + }), + ); + + const response = await new ZenRows("API_KEY", { retries: 3 }).fetch(url); + + expect(response.status).toBe(400); + expect(calls).toBe(1); + }); + + test("a 422 other than no_data_extracted is retried, and the last one comes back readable", async () => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return HttpResponse.json({ code: "RESP001" }, { status: 422 }); + }), + ); + + const pending = new ZenRows("API_KEY").fetch(url); + await vi.advanceTimersByTimeAsync(1000); + const response = await pending; + + expect(calls).toBe(3); + expect(response.status).toBe(422); + expect(await response.json()).toEqual({ code: "RESP001" }); + }); + + test("a 500 is not retried", async () => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return new HttpResponse(null, { status: 500 }); + }), + ); + + const response = await new ZenRows("API_KEY", { retries: 3 }).fetch(url); + + expect(response.status).toBe(500); + expect(calls).toBe(1); + }); + + const retriedAnswers = [ + [503, () => new HttpResponse(null, { status: 503 })], + [429, () => new HttpResponse(null, { status: 429 })], + [422, () => HttpResponse.json({ code: "RESP001" }, { status: 422 })], + ] as const; + + describe.each([ + ["without an Idempotency-Key", {}], + ["with an Idempotency-Key", { "Idempotency-Key": "order-key" }], + ])("post() %s", (_label, headers: Record) => { + test.each(retriedAnswers)( + "is sent once on a %i, even with retries: 3", + async (code, answer) => { + const bodies: string[] = []; + server.use( + http.post(apiUrl, async ({ request }) => { + bodies.push(await request.text()); + return answer(); + }), + ); + + const pending = new ZenRows("API_KEY", { retries: 3 }).post( + url, + {}, + { headers, data: { order: 1 } }, + ); + await vi.advanceTimersByTimeAsync(60_000); + + expect((await pending).status).toBe(code); + expect(bodies).toEqual(['{"order":1}']); + }, + ); + }); + + test.each(retriedAnswers)("fetch() is still retried on a %i", async (code, answer) => { + let calls = 0; + server.use( + http.get(apiUrl, () => { + calls += 1; + return answer(); + }), + ); + + const pending = new ZenRows("API_KEY", { retries: 3 }).fetch(url); + await vi.advanceTimersByTimeAsync(60_000); + + expect((await pending).status).toBe(code); + expect(calls).toBe(4); + }); + + describe("a status from the target site", () => { + /** Answers every GET with `answer()` and records the query string of each request. */ + function recordGets(answer: () => Response) { + const queries: URLSearchParams[] = []; + server.use( + http.get(apiUrl, ({ request }) => { + queries.push(new URL(request.url).searchParams); + return answer(); + }), + ); + return queries; + } + + test.each(retriedAnswers)( + "fetch() with original_status: true is sent once on a %i", + async (code, answer) => { + const queries = recordGets(answer); + + const pending = new ZenRows("API_KEY", { retries: 3 }).fetch(url, { + original_status: true, + }); + await vi.advanceTimersByTimeAsync(60_000); + + expect((await pending).status).toBe(code); + expect(queries).toHaveLength(1); + expect(queries[0]?.get("original_status")).toBe("true"); + }, + ); + + test.each([ + ["false", { original_status: false }], + ["absent", {}], + ])("fetch() with original_status %s is retried on a 503", async (_label, config) => { + const queries = recordGets(() => new HttpResponse(null, { status: 503 })); + + const pending = new ZenRows("API_KEY", { retries: 3 }).fetch(url, config); + await vi.advanceTimersByTimeAsync(60_000); + + expect((await pending).status).toBe(503); + expect(queries).toHaveLength(4); + }); + + test("extract() with original_status: true is sent once on a 503", async () => { + const queries = recordGets(() => new HttpResponse(null, { status: 503 })); + + const pending = new ZenRows("API_KEY", { retries: 3 }).extract(url, { + original_status: true, + }); + await vi.advanceTimersByTimeAsync(60_000); + + expect((await pending).status).toBe(503); + expect(queries).toHaveLength(1); + }); + + test("allowed_status_codes '503': a 503 is sent once, a 502 is retried", async () => { + const client = new ZenRows("API_KEY", { retries: 3 }); + const config = { allowed_status_codes: "503" }; + + const once = recordGets(() => new HttpResponse(null, { status: 503 })); + const first = client.fetch(url, config); + await vi.advanceTimersByTimeAsync(60_000); + expect((await first).status).toBe(503); + expect(once).toHaveLength(1); + expect(once[0]?.get("allowed_status_codes")).toBe("503"); + + const retried = recordGets(() => new HttpResponse(null, { status: 502 })); + const second = client.fetch(url, config); + await vi.advanceTimersByTimeAsync(60_000); + expect((await second).status).toBe(502); + expect(retried).toHaveLength(4); + }); + }); +}); + +describe("prepared-domain calls are never retried", () => { + // Their 429 (`out_of_allowance`) and 503 (`overloaded`, `disabled`) are answers to read, not + // to resend: the caller gets them on the first attempt, even with retries on. + const preparedDomains = "https://api.zenrows.com/v1/extract/prepared-domains"; + + beforeEach(() => { + vi.useFakeTimers({ toFake: ["setTimeout", "Date"] }); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + + const calls = [ + ["prepareDomain", (client: ZenRows) => client.prepareDomain("www.example.com", [url])], + ["preparedDomainStatus", (client: ZenRows) => client.preparedDomainStatus("www.example.com")], + ] as const; + + for (const [name, call] of calls) { + for (const code of [429, 503]) { + test(`${name}() rejects on the first ${code}, with the default and with retries: 5`, async () => { + let requests = 0; + const answer = () => { + requests += 1; + return HttpResponse.json({ code: "x" }, { status: code }); + }; + server.use( + http.post(preparedDomains, answer), + http.get(`${preparedDomains}/:domain`, answer), + ); + + for (const client of [new ZenRows("API_KEY"), new ZenRows("API_KEY", { retries: 5 })]) { + requests = 0; + const pending = call(client).catch((error: unknown) => error); + await vi.advanceTimersByTimeAsync(60_000); + const error = await pending; + + expect(error).toBeInstanceOf(ZenRowsPreparedDomainError); + expect((error as ZenRowsPreparedDomainError).status).toBe(code); + expect(requests).toBe(1); + } + }); + } + } +}); + +describe("retry options are validated", () => { + test.each([-1, 1.5, Number.NaN, Number.POSITIVE_INFINITY, "2"])( + "retries %s is rejected", + (retries) => { + expect(() => new ZenRows("API_KEY", { retries: retries as number })).toThrow( + /retries must be an integer of 0 or more/, + ); }, ); + + test.each(["retryMaxWait", "retryBudget"] as const)("%s must be finite and above 0", (name) => { + for (const value of [0, -1, Number.NaN, Number.POSITIVE_INFINITY, "5"]) { + expect(() => new ZenRows("API_KEY", { [name]: value as number })).toThrow( + new RegExp(`${name} must be a finite number of seconds greater than 0`), + ); + } + }); + + test("valid values and defaults are accepted", () => { + expect(() => new ZenRows("API_KEY")).not.toThrow(); + expect( + () => new ZenRows("API_KEY", { retries: 0, retryMaxWait: 0.1, retryBudget: 1 }), + ).not.toThrow(); + }); }); diff --git a/tests/retry-count.test.ts b/tests/retry-count.test.ts index 61fc964..51b0227 100644 --- a/tests/retry-count.test.ts +++ b/tests/retry-count.test.ts @@ -1,29 +1,40 @@ import { http, HttpResponse } from "msw"; -import { describe, expect, test } from "vitest"; +import { afterEach, beforeEach, describe, expect, test, vi } from "vitest"; import { ZenRows } from "../src"; import { server } from "./_setup"; -describe("retryOn off-by-one fix", () => { +describe("retry count", () => { + beforeEach(() => { + vi.useFakeTimers({ toFake: ["setTimeout", "Date"] }); + }); + + afterEach(() => { + vi.useRealTimers(); + }); + test.each([ + { retries: undefined, expectedCalls: 3 }, { retries: 0, expectedCalls: 1 }, { retries: 1, expectedCalls: 2 }, { retries: 2, expectedCalls: 3 }, ])( - "retries: $retries should make exactly $expectedCalls fetch call(s) on 422", + "retries: $retries makes exactly $expectedCalls call(s) on a 503", async ({ retries, expectedCalls }) => { let fetchCount = 0; server.use( http.get("https://api.zenrows.com/v1/", () => { fetchCount++; - return new HttpResponse("Could Not Get Content", { status: 422 }); + return new HttpResponse("Service Unavailable", { status: 503 }); }), ); - const client = new ZenRows("API_KEY", { retries }); - const response = await client.get("https://example.com"); + const client = new ZenRows("API_KEY", retries === undefined ? {} : { retries }); + const pending = client.get("https://example.com"); + await vi.advanceTimersByTimeAsync(10_000); + const response = await pending; - expect(response.status).toBe(422); + expect(response.status).toBe(503); expect(fetchCount).toBe(expectedCalls); }, ); diff --git a/tests/retry-network.test.ts b/tests/retry-network.test.ts new file mode 100644 index 0000000..7949ab3 --- /dev/null +++ b/tests/retry-network.test.ts @@ -0,0 +1,293 @@ +// Retry decisions against a real node:http server through Node's real fetch, so each failure +// reaches the policy in the exact shape undici produces. The request mocks are off in this file. +import { lookup } from "node:dns/promises"; +import http from "node:http"; +import type { AddressInfo, Socket } from "node:net"; +import { afterAll, afterEach, beforeAll, describe, expect, test, vi } from "vitest"; +import { ZenRows } from "../src"; +import { server as mockServer } from "./_setup"; + +const API_URL = "https://api.zenrows.com/v1/"; + +let realFetch: typeof fetch; + +beforeAll(() => { + // The shared setup turns the request mocks on; turn them off here so fetch is Node's own. + mockServer.close(); + realFetch = globalThis.fetch; +}); + +afterAll(() => { + // Leave the mocks as the shared setup expects to find them when it closes them. + mockServer.listen(); +}); + +afterEach(() => { + vi.restoreAllMocks(); + vi.unstubAllGlobals(); +}); + +interface TestServer { + origin: string; + /** Requests the server read in full, by method. */ + requests: string[]; + /** Sockets the server still holds open. */ + sockets: Set; + close: () => Promise; +} + +async function startServer( + handle: (req: http.IncomingMessage, res: http.ServerResponse) => void, +): Promise { + const requests: string[] = []; + const sockets = new Set(); + const httpServer = http.createServer((req, res) => { + req.resume(); + req.on("end", () => { + requests.push(req.method ?? ""); + handle(req, res); + }); + }); + httpServer.on("connection", (socket) => { + sockets.add(socket); + socket.on("close", () => sockets.delete(socket)); + }); + await new Promise((resolve) => httpServer.listen(0, "127.0.0.1", resolve)); + const { port } = httpServer.address() as AddressInfo; + return { + origin: `http://127.0.0.1:${port}`, + requests, + sockets, + close: () => + new Promise((resolve) => { + for (const socket of sockets) socket.destroy(); + httpServer.close(() => resolve()); + }), + }; +} + +/** A port nothing listens on: bind one, then close it. */ +async function closedPortOrigin(): Promise { + const probe = http.createServer(); + await new Promise((resolve) => probe.listen(0, "127.0.0.1", resolve)); + const { port } = probe.address() as AddressInfo; + await new Promise((resolve) => probe.close(() => resolve())); + return `http://127.0.0.1:${port}`; +} + +/** + * Sends the client's API calls to `origin` through real fetch, with extra fetch options, and + * counts the calls. Must run before the client is built, since the client wraps fetch then. + */ +function routeApiTo(origin: string, extra: Record = {}) { + const calls = { count: 0 }; + vi.stubGlobal("fetch", (input: string | URL, init?: RequestInit) => { + calls.count += 1; + return realFetch(String(input).replace(API_URL, `${origin}/v1/`), { ...init, ...extra }); + }); + return calls; +} + +/** undici's Agent, taken from Node's own global dispatcher: Node bundles undici but does not export it. */ +async function undiciAgent(origin: string): Promise object> { + await (await realFetch(`${origin}/warm-up`)).arrayBuffer(); + const globals = globalThis as unknown as Record; + const dispatcher = + globals[Symbol.for("undici.globalDispatcher.2")] ?? + globals[Symbol.for("undici.globalDispatcher.1")]; + if (!dispatcher) throw new Error("Node's fetch exposes no global dispatcher"); + return dispatcher.constructor as new ( + options: object, + ) => object; +} + +const fastRandom = () => vi.spyOn(Math, "random").mockReturnValue(0); + +/** True when `localhost` resolves to an IPv4 and an IPv6 address, as Node's fetch looks it up. */ +const localhostHasBothFamilies = await lookup("localhost", { all: true }).then( + (addresses) => new Set(addresses.map((each) => each.family)).size === 2, + () => false, +); + +describe("failures before the request was sent are retried", () => { + test("a refused connection is retried `retries` times, then throws", async () => { + fastRandom(); + const calls = routeApiTo(await closedPortOrigin()); + + const error = await new ZenRows("API_KEY").fetch("https://example.com").catch((e) => e); + + expect(error).toBeInstanceOf(TypeError); + expect(error.cause.code).toBe("ECONNREFUSED"); + expect(calls.count).toBe(3); + }); + + // `localhost` resolves to ::1 and 127.0.0.1 on most hosts. Node then tries both (happy + // eyeballs) and the cause is an AggregateError with one ECONNREFUSED per address. + test.skipIf(!localhostHasBothFamilies)( + "a refused connection to localhost, on both address families, is retried", + async () => { + fastRandom(); + const port = new URL(await closedPortOrigin()).port; + const calls = routeApiTo(`http://localhost:${port}`); + + const error = await new ZenRows("API_KEY", { retries: 2 }) + .fetch("https://example.com") + .catch((e) => e); + + expect(error).toBeInstanceOf(TypeError); + expect(error.cause).toBeInstanceOf(AggregateError); + const innerCodes = error.cause.errors.map((inner: { code: string }) => inner.code); + expect(innerCodes.length).toBeGreaterThanOrEqual(2); + expect(new Set(innerCodes)).toEqual(new Set(["ECONNREFUSED"])); + expect(calls.count).toBe(3); + }, + ); +}); + +describe("original_status: true", () => { + test("a refused connection is still retried", async () => { + fastRandom(); + const calls = routeApiTo(await closedPortOrigin()); + + const error = await new ZenRows("API_KEY") + .fetch("https://example.com", { original_status: true }) + .catch((e) => e); + + expect(error.cause.code).toBe("ECONNREFUSED"); + expect(calls.count).toBe(3); + }); + + test("a 503 from the server is sent once", async () => { + fastRandom(); + const server = await startServer((_req, res) => { + res.writeHead(503); + res.end(); + }); + routeApiTo(server.origin); + + const response = await new ZenRows("API_KEY").fetch("https://example.com", { + original_status: true, + }); + await server.close(); + + expect(response.status).toBe(503); + expect(server.requests).toEqual(["GET"]); + }); +}); + +describe("post() is sent once, even on a refused connection", () => { + test.each([ + ["without an Idempotency-Key", {}], + ["with an Idempotency-Key", { "Idempotency-Key": "k1" }], + ])("%s", async (_label, headers: Record) => { + fastRandom(); + const calls = routeApiTo(await closedPortOrigin()); + + const error = await new ZenRows("API_KEY", { retries: 3 }) + .post("https://example.com", {}, { headers, data: { a: 1 } }) + .catch((e) => e); + + expect(error.cause.code).toBe("ECONNREFUSED"); + expect(calls.count).toBe(1); + }); +}); + +describe("failures after the request was sent are not retried", () => { + test("a socket closed after the server read the request", async () => { + fastRandom(); + const server = await startServer((req) => req.socket.destroy()); + routeApiTo(server.origin); + + const error = await new ZenRows("API_KEY").fetch("https://example.com").catch((e) => e); + await server.close(); + + expect(error).toBeInstanceOf(TypeError); + expect(error.cause.code).toBe("UND_ERR_SOCKET"); + expect(server.requests).toEqual(["GET"]); + }); + + test("a socket closed after a post() with an Idempotency-Key was read", async () => { + fastRandom(); + const server = await startServer((req) => req.socket.destroy()); + routeApiTo(server.origin); + + const error = await new ZenRows("API_KEY") + .post("https://example.com", {}, { headers: { "Idempotency-Key": "k1" }, data: { a: 1 } }) + .catch((e) => e); + await server.close(); + + expect(error.cause.code).toBe("UND_ERR_SOCKET"); + expect(server.requests).toEqual(["POST"]); + }); + + test("headers that arrive after undici's headersTimeout", async () => { + fastRandom(); + const server = await startServer((req, res) => { + if (req.url === "/warm-up") return res.end(); + setTimeout(() => res.end("late"), 1000).unref(); + }); + const Agent = await undiciAgent(server.origin); + server.requests.length = 0; + routeApiTo(server.origin, { dispatcher: new Agent({ headersTimeout: 200 }) }); + + const error = await new ZenRows("API_KEY").fetch("https://example.com").catch((e) => e); + await server.close(); + + expect(error).toBeInstanceOf(TypeError); + expect(error.cause.code).toBe("UND_ERR_HEADERS_TIMEOUT"); + expect(server.requests).toEqual(["GET"]); + }); + + test("an invalid header value throws at once, with no retry", async () => { + fastRandom(); + const server = await startServer((_req, res) => res.end("ok")); + const calls = routeApiTo(server.origin); + + const error = await new ZenRows("API_KEY") + .fetch("https://example.com", {}, { headers: { "X-Bad": "a\nb" } }) + .catch((e) => e); + await server.close(); + + expect(error).toBeInstanceOf(TypeError); + expect(error.cause).toBeUndefined(); + expect(calls.count).toBe(1); + expect(server.requests).toEqual([]); + }); +}); + +describe("retried answers", () => { + // An answer the client drops without reading holds its connection until garbage collection. + // The server counts answers that have neither finished nor lost their connection. That count + // does not depend on the Node version, unlike open sockets. Node 20 and 22 drain a cancelled + // body and keep the socket for undici's keep-alive window, and buffer a small body in full. + test("release their connection: no answer stays held after retried 503s with large bodies", async () => { + fastRandom(); + // Larger than undici buffers on its own, so an unread body really holds the connection. + const largeBody = Buffer.alloc(8 * 1024 * 1024, "x"); + let held = 0; + const server = await startServer((_req, res) => { + held += 1; + res.on("close", () => { + held -= 1; + }); + res.writeHead(503, { "Content-Type": "text/plain", "Content-Length": largeBody.length }); + res.end(largeBody); + }); + routeApiTo(server.origin); + + const client = new ZenRows("API_KEY"); + const responses = await Promise.all( + Array.from({ length: 5 }, () => client.fetch("https://example.com")), + ); + for (const response of responses) await response.body?.cancel(); + const deadline = Date.now() + 2000; + while (held > 0 && Date.now() < deadline) { + await new Promise((resolve) => setTimeout(resolve, 20)); + } + const stillHeld = held; + await server.close(); + + expect(server.requests).toHaveLength(15); + expect(stillHeld).toBe(0); + }); +});