A worker-thrown error loses its class at the boundary, so every worker-backed provider's retryable/permanent classification is re-guessed from the message

[sroussey]

What

WorkerServerBase.postError flattened a thrown value to { message, name, stack } before posting it to the main thread, and rehydrateWorkerError rebuilt a bare Error from those three fields. Nothing else crossed.

$ sed -n '56,67p' packages/util/src/worker/scrubStack.ts   # before this change
export function rehydrateWorkerError(data: unknown): Error {
  …
  const payload = data as { message?: string; name?: string; stack?: string };
  …
  return Object.assign(new Error(payload.message ?? asText(data)), {
    name: payload.name ?? "Error",
    …
  });
}

A run function registered worker-backed reaches AiJob.execute's catch through that path — AiProviderRegistry.registerAsWorkerRunFn → WorkerManager.callWorkerRunFunction → rehydrateWorkerError — so classifyProviderError received a plain Error and every instanceof in it was false:

if (err instanceof PermanentJobError || err instanceof RetryableJobError || …) return err;
if (err instanceof ProviderUnsupportedFeatureError || err instanceof ImageGenerationContentPolicyError)
  return new PermanentJobError(err.message);
if (err instanceof ImageGenerationProviderError)
  return err.retryable ? new RetryableJobError(…) : new PermanentJobError(…);

The classification a provider had just made was therefore discarded, and the error fell through to the message/status heuristics below it — landing on the PermanentJobError default whenever the message carried no HTTP-shaped status and no network keyword.

That is also what the HFT_NULL_PROCESSOR: message prefix is working around: it is a retryable classification encoded into the one field that does survive the trip.

Why it matters, in both directions

• A worker-side rate limit or transient 5xx arrives as an unlabelled failure and is failed permanently — the retry the provider asked for never happens.
• The reverse is worse and was reproducible: a ImageGenerationContentPolicyError (retryable = false) whose provider reason happens to contain "timed out" fell through to message.includes("timed out") and was classified RetryableJobError. A refusal that cannot succeed was re-run on the attempt budget.

Two adjacent facts that make it a structural gap rather than one bad branch:

• JobError.retryable was write-only. RetryableJobError's constructor sets it and nothing in the tree ever read it; every consumer branched on instanceof RetryableJobError. Two representations of one fact, and the one that is documented as the control was inert.
• Providers declare retryability three different ways: a JobError subclass (deepseek, chrome-ai, anthropic), a retryable field on a non-JobError class (the four image-generation providers, through ImageGenerationErrors.ts), and a magic message prefix (huggingface-transformers). Only the first two are even in principle recoverable at a boundary that keeps name and message.

Fix

Landed on claude/upbeat-feynman-59uare (ddf6923f2). One field, retryable, read through declaredRetryability / isRetryableError (packages/job-queue/src/job/Retryability.ts) and carried across the worker boundary by workerErrorPayload / rehydrateWorkerError. Unclassified stays permanent. Details and the reinstatement proof are in the comment below.

Not the finding as originally framed

The original report was that retryable is "consulted at exactly one call site, AiJob.ts:107, while six cloud providers classify errors". The single call site was real; the diagnosis was not. AiJob.execute is the only funnel — both DirectExecutionStrategy and QueuedExecutionStrategy call it — so an in-process error's classification was never discarded, and every downstream consumer read the resulting JobError subclass. The leak is the worker serialization boundary, and it is invisible from a grep of retryable.

Measured on origin/main @ 2d36880 (0.6.0).

[sroussey]

Landed on claude/upbeat-feynman-59uare — ddf6923f2.

Investigation, before the fix

Where retryable is set — two places, not six providers:

• packages/job-queue/src/job/JobError.ts:11 (JobError.retryable = false) and :50 (RetryableJobError sets true).
• packages/ai/src/errors/ImageGenerationErrors.ts:13,29,44 — ProviderUnsupportedFeatureError false, ImageGenerationContentPolicyError false, ImageGenerationProviderError true.

Providers throw those classes; none sets the flag itself. Four touch the AI error classes (google-gemini, huggingface-inference, openai, xai — image generation only); five more classify by throwing a JobError subclass directly (anthropic, deepseek, chrome-ai, tf-mediapipe, huggingface-transformers). DeepSeekToolChoiceNotHonoredError extends RetryableJobError and says so in its own JSDoc.

Where it is read — one site, and the count was right: AiJob.ts:107. The fact itself was read in four more places via instanceof: JobQueueWorker.ts:863, FetchUrlTask.ts:318, AiJob.ts:93-95, and the queue storages persisting the class name through jobErrorPersistedCode.

Which call sites should be consulting it and were not — this is where the filed finding is wrong, and where a real defect is:

• Retry/backoff paths: there is exactly one, JobQueueWorker.ts:863. Neither AI execution strategy retries — QueuedExecutionStrategy says so in its own docstring — so there was no second retry path to route.
• StreamProcessor: does not classify errors. Nothing to do.
• FallbackTaskRunner: falls back on any error by design. A permanent 400 on model A is precisely when you want model B, so retryability is the wrong discriminator there.
• StructuredGenerationTask: its loop retries on schema mismatch and lets provider throws propagate. No widening there either.
• AiJob.execute is the only funnel, and both DirectExecutionStrategy and QueuedExecutionStrategy call it. So for an error thrown in this process, the single call site was correct and sufficient: it is the translation boundary into the JobError hierarchy the rest of the system reads.

What is actually broken is the worker boundary, which a grep for retryable cannot see. WorkerServerBase.postError → {message, name, stack} → rehydrateWorkerError → bare Error. Class gone, flag gone. So every instanceof in classifyProviderError was false for a worker-backed provider, and the classification fell through to the message heuristics — which is also why huggingface-transformers encodes a retryable verdict as the HFT_NULL_PROCESSOR: message prefix: message is one of the two fields that survive.

The fix — a mechanism: one field, carried

• packages/job-queue/src/job/Retryability.ts (new) — declaredRetryability(error): boolean | undefined and isRetryableError(error): boolean. Three-valued, because "said no" and "said nothing" have to be told apart by a caller that falls back to heuristics. Unclassified is false everywhere.
• packages/util/src/worker/scrubStack.ts:63-119 — WorkerErrorPayload + workerErrorPayload() (the serializer postError at WorkerServerBase.ts:277 now uses) carry a declared boolean; rehydrateWorkerError restores it. An absent flag stays absent rather than becoming false, so a worker that predates this is not made to look like it answered.
• packages/ai/src/job/AiJob.ts:147-165 — reads the declaration instead of testing three AI error classes by name. Placed after the abort checks and before the heuristics: an abort is a different question, and a stated answer beats a guess made from a string. AiJob.ts:122 also recognises a worker-thrown AbortSignalJobError by name, which does survive the trip.
• packages/job-queue/src/job/JobQueueWorker.ts:1356-1365 — normalizeError preserves a declared flag instead of wrapping every non-JobError as permanent; :866 asks isRetryableError rather than testing for RetryableJobError. retryDate still comes only from a real RetryableJobError (:893).
• packages/job-queue/src/job/JobError.ts:11-17 — retryable documented as the single source of truth; it was settable and unread before.

Behaviour change, stated

A failure that declares retryable: true and previously reached the main thread stripped of its class is now retried instead of failing permanently. That is the defect being fixed. The reverse also lands: an error declaring retryable: false whose message contains "timed out" is no longer re-run by the heuristic that read it as transient. An error that declares nothing changes class in neither direction. It is in the commit message, which is what the generated changelogs are built from.

Not touched, deliberately: the HFT_NULL_PROCESSOR: prefix now has a better expression available but removing it means changing a provider path I cannot exercise here. Worth a follow-up. Same for code, which the worker payload still drops — a worker error persists today as error_code = "Error" rather than its own code, which is a different fact from this one.

Verification

$ bunx vitest run packages/job-queue/src/job/__tests__/Retryability.test.ts \
      packages/util/src/worker/__tests__/workerErrorPayload.test.ts \
      packages/test/src/test/ai/AiJob_classifyProviderError.test.ts
 Test Files  3 passed (3)
      Tests  23 passed (23)        exit 0, 11.1s

$ bun scripts/test.ts job-queue queue vitest unit
 Test Files  19 passed (19)     Tests  241 passed | 3 skipped     exit 0, 30.9s
$ bun scripts/test.ts util vitest unit
 Test Files  62 passed (62)     Tests  864 passed | 4 expected fail | 12 skipped   exit 0, 76.1s
$ bun scripts/test.ts ai vitest unit
 Test Files  50 passed (50)     Tests  533 passed                 exit 0, 59.0s
$ bun scripts/test.ts task scripts vitest unit
 Test Files  87 passed (87)     Tests  1346 passed | 24 skipped   exit 0, 103.7s

$ bun run lint             # exit 0, 26.5s
$ bun run build:types      # 43 successful, 43 total, exit 0, 28.2s
$ bun run format-check     # All matched files use the correct format, exit 0
$ bun run typecheck:tests  # exit 0, 4.2s
$ bun run typecheck:budget # OK (40 packages within budget), exit 0, 39.6s
$ bun scripts/test.ts --check-sections   # every test file reachable, exit 0

Reinstatement proof

Defect 1 — drop the flag at the boundary (workerErrorPayload stops carrying it):

 × carries a declared `retryable` across the boundary        expected undefined to be true
 × carries a declared `retryable: false` as the claim it is  expected undefined to be false
 × keeps a provider's retryable classification retryable     expected PermanentJobError… to be an instance of RetryableJobError
 × keeps a RetryableJobError thrown in the worker retryable  expected PermanentJobError… to be an instance of RetryableJobError
 × does not let a message heuristic overturn a stated permanent answer
                                                             expected RetryableJobError: Timeout calling TEST_P… to not be an instance of RetryableJobError
 Tests  5 failed | 12 passed (17)

That last line is the bug in one assertion: with the flag dropped, a content-policy refusal whose reason reads "the review timed out" comes back retryable.

Defect 2 — keep the flag on the wire but ignore the declaration in classifyProviderError:

 × maps retryable ImageGenerationProviderError to RetryableJobError    (the pre-existing test)
 × keeps a provider's retryable classification retryable
 × keeps a RetryableJobError thrown in the worker retryable
 × does not let a message heuristic overturn a stated permanent answer
 Tests  4 failed | 6 passed (10)

Restored:

$ bunx vitest run packages/test/src/test/ai/AiJob_classifyProviderError.test.ts \
      packages/util/src/worker/__tests__/workerErrorPayload.test.ts
 Test Files  2 passed (2)     Tests  17 passed (17)

No PR opened, per instructions.

---

Generated by Claude Code