Skip to content

feat(api-keys): stop storing API key plaintext (DW-1012) - #2172

Merged
tobydigz merged 4 commits into
mainfrom
feat/dw-1012/stop-storing-plaintext
Nov 6, 2026
Merged

tobydigz merged 4 commits into
mainfrom
feat/dw-1012/stop-storing-plaintext

Conversation

@tobydigz

@tobydigz tobydigz commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Part 6 of 7 for DW-1005 (DW-1012). Keys authenticate by secret_hash and fusillade dispatches by key id, so the stored plaintext copies are now only a leak risk. Nothing reads or writes them any more; DW-1013 (stacked on this) drops them.

Merge gates

Merge only when both hold in production:

What changes

  • API keys: the secret of a new or rotated key is returned once and only its hash is kept. api_keys.secret stays NOT NULL for the previous release; new rows take a column default ('unset-' || gen_random_uuid(), unique and not a key), so no code names the column.
  • Fusillade: realtime, flex, background and batch work stores no key; request_templates_g2.api_key defaults to ''. Dispatch always uses the key id; a row without one carries no credential and the loopback refuses it.
  • Reveal: works only from the wrapped pending secret. A pending key without a usable wrap returns 500 and stays revealable; rotating it fixes it.
  • Reconcile: no longer wraps from plaintext; still re-wraps old wraps and clears orphaned ones. dwctl_api_key_pending_reveals_unrecoverable replaces ..._needing_plaintext; dwctl_api_key_reveals_total drops its source label. Update any dashboards or alerts on those.
  • Migrations (dwctl and fusillade 20261105105806): the two column defaults. Metadata-only.
  • Existing plaintext stays until DW-1013.

Rollout

Previous-release pods read both columns as non-null. They only ever see the defaults for new rows, which they store but never use (dispatch by key id, reveal from the wrap). Rolling back to the previous release is safe.

Testing

cargo nextest run -p dwctl -p fusillade -p fusillade-core -p fusillade-arsenal --all-features (3757 passed) and just lint rust pass locally.

Keys authenticate by secret_hash and fusillade dispatches by key id, so
the plaintext copies are dead weight and a leak risk. Nothing now reads
or writes api_keys.secret or fusillade's stored api_key: the secret of a
new or rotated key is returned once and only its hash is kept, and
realtime, flex, background and batch work stores no key.

The columns stay NOT NULL for the previous release, which still reads
them during the rollout. New rows take column defaults instead of values
written by the application (a unique placeholder that is not a key, and
''), so this release also survives the next release dropping them.
Existing plaintext stays until then.

Without plaintext, an issued key's reveal works only from its wrapped
pending secret. The reconcile no longer wraps from plaintext: it
re-wraps old wraps and clears orphaned ones, and reports pending keys
that cannot be revealed in dwctl_api_key_pending_reveals_unrecoverable
(replacing ..._needing_plaintext). A dispatch without a key id carries
no credential and is refused by the loopback.
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Deploying control-layer with  Cloudflare Pages  Cloudflare Pages

Latest commit: 569b2d6
Status: ✅  Deploy successful!
Preview URL: https://f1f4baa9.control-layer.pages.dev
Branch Preview URL: https://feat-dw-1012-stop-storing-pl.control-layer.pages.dev

View logs

@tobydigz
tobydigz added this pull request to stack #2174 November 5, 2026 13:27
@tobydigz
tobydigz marked this pull request as ready for review November 5, 2026 13:28
@cubic-dev-ai

cubic-dev-ai Bot commented Nov 5, 2026

Copy link
Copy Markdown

Running ultrareview automatically — this PR appears to change authentication, authorization, or input validation, where a missed bug can expose data or grant the wrong access, so a deeper multi-pass review is worth running. I'll post findings when complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultrareview ran automatically — this PR appears to change authentication, authorization, or input validation, where a missed bug can expose data or grant the wrong access, so a deeper multi-pass review is worth running.

All reported issues were addressed across 120 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread fusillade-core/src/request/query.rs Resolved
Comment thread dwctl/src/pending_secrets.rs Resolved
…arly (DW-1012)

Review follow-ups. A pending key with no usable wrap now gets 409 with a
"rotate the key" message instead of 500, and the reveal stays unspent.
Without a keystore, rotating a pending key spends its reveal: nothing can
hold the new secret, and an older pod must not reveal the stored plaintext
of the rotated-away one.

A dispatch without a key id is refused before it is sent, with its own
failure signal, rather than reaching the loopback unauthenticated. The
daemon tests that dispatched to a mock now name their batch's key, with
the mock standing in for this process's /ai.

ApiKeyDBResponse's Debug no longer prints the create-time secret.
The reference cache stamps visible_at with the database clock and
arrivals with this process's; the tests now wait until this clock has
passed an entry's visible_at, so a database clock running slightly
ahead no longer turns a hit into a miss. The internal-credential tests
poll the playground and the probe with deadlines that load cannot
reach, and the pool-retry test retries opening its one connection,
which can exceed the 50 ms acquire timeout under test load.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 18 files (changes from recent commits).

Requires human review: Security-sensitive change to API-key secret handling: new keys store only hashes, fusillade drops plaintext, plus DB column defaults and rollout merge gates. Touches production auth/secrets and needs human approval.

View guided diff | Turn on auto-fix | Re-trigger cubic

…l reporting (DW-1012)

Review follow-ups. ApiKeys::create now returns a CreatedApiKey (key plus
secret) instead of an optional secret on every key response, so reads
cannot carry one and the "no secret returned" branches go away.

The unrecoverable-reveal count no longer includes valid wraps under an
older wrap key, which a rolling wrap-key rotation produces and which
reveal fine. A failed reveal now tells the holder to rotate the key,
which they can do in every configuration.

docs/migrations.md records the rollback floor: from 16.5.0, roll back
no further than 16.3.0.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 35 files (changes from recent commits).

Requires human review: Security-sensitive API-key change: plaintext secrets are no longer stored, fusillade drops stored keys, with DB defaults and production rollout gates. Requires human approval.

View guided diff | Turn on auto-fix | Re-trigger cubic

@tobydigz
tobydigz added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit dde6a1c Nov 6, 2026
28 checks passed
@tobydigz
tobydigz deleted the feat/dw-1012/stop-storing-plaintext branch November 6, 2026 10:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants