Skip to content

feat(fusillade)!: drop the plaintext API key columns (DW-1013) - #2173

Merged
tobydigz merged 7 commits into
mainfrom
feat/dw-1013/drop-plaintext-columns
Nov 6, 2026
Merged

tobydigz merged 7 commits into
mainfrom
feat/dw-1013/drop-plaintext-columns

Conversation

@tobydigz

@tobydigz tobydigz commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Part 7 of 7 for DW-1005 (DW-1013). Merge only after the release containing #2172 (16.5.0) is fully deployed.

What changes

  • dwctl migration 20261105123212_drop_api_keys_secret: secret_hash SET NOT NULL (instant with the validated check), drop the check, drop api_keys.secret, refresh stale column and index comments.
  • fusillade migration 20261105123212_drop_stored_api_keys: recreate active_request_templates, request_templates_all and request_templates without api_key, then drop request_templates_g2.api_key.
  • fusillade migration 20261105143830_drop_batches_api_key: drops batches.api_key on its own, so its lock isn't held with the template locks.
  • fusillade-core (breaking): api_key removed from RequestTemplate and every request/batch input type; create_flex/create_background require the API key id. RequestData.api_key stays as the in-memory per-dispatch credential.
  • Retained response snapshots stop carrying api_key; older ones still decode.
  • docs/migrations.md: from 16.6.0, roll back no further than 16.5.0.

Rollout

16.5.0 neither reads nor writes these columns, so its pods keep working while the migrations run.

Not removed

  • DROP COLUMN doesn't rewrite rows: old bytes stay in table pages, WAL and backups until rows are rewritten (VACUUM FULL / pg_repack) and backups expire.
  • Retained-response payloads written before 16.5.0 keep template.api_key until their daily partitions are retired (retained_response_retirement_enabled). Rewriting that table (~706 GB) to scrub them costs far more than retirement.

Testing

Targeted tests for the touched areas, the down-migration test, and just lint rust pass locally; CI runs the full suite.

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Deploying control-layer with  Cloudflare Pages  Cloudflare Pages

Latest commit: b1c76de
Status: ✅  Deploy successful!
Preview URL: https://cb6c2e60.control-layer.pages.dev
Branch Preview URL: https://feat-dw-1013-drop-plaintext.control-layer.pages.dev

View logs

@tobydigz
tobydigz added this pull request to stack #2174 November 5, 2026 13:27
@tobydigz
tobydigz marked this pull request as ready for review November 5, 2026 13:28
@cubic-dev-ai

cubic-dev-ai Bot commented Nov 5, 2026

Copy link
Copy Markdown

Running ultrareview automatically — this PR appears to change a database schema or migrate data, where a missed bug can corrupt or lose records, so a deeper multi-pass review is worth running. I'll post findings when complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ultrareview ran automatically — this PR appears to change a database schema or migrate data, where a missed bug can corrupt or lose records, so a deeper multi-pass review is worth running.

All reported issues were addressed across 36 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

@tobydigz
tobydigz force-pushed the feat/dw-1013/drop-plaintext-columns branch 2 times, most recently from 4cf36fd to c3ddb9b Compare November 5, 2026 17:04
Base automatically changed from feat/dw-1012/stop-storing-plaintext to main November 6, 2026 09:51
The previous release neither reads nor writes api_keys.secret or
fusillade's stored api_key, so dropping them removes the last stored
copies of key secrets. secret_hash becomes NOT NULL, which its validated
check makes instant; the check is then redundant.

The fusillade views that project api_key are recreated without it in
the same migration. Retained response snapshots stop carrying an
api_key; ones that still do decode as before.

Rolling back past the previous release breaks key creation and
fusillade writes (docs/migrations.md).
Fusillade stores no key secret and dispatches by API key id, so the
api_key fields on its inputs were ignored. Removing them stops callers
passing secrets that go nowhere.

Also scrubs key secrets from retained-response payloads written by
earlier releases, in a resumable batched no-transaction migration, and
drops batches.api_key in its own migration so its lock is not held with
the template locks. Migration comments now say what DROP COLUMN does and
does not remove, and stale column and index comments are replaced. Adds
a test for the down migrations.

BREAKING CHANGE: fusillade-core removes the `api_key` field from
RequestTemplate, RequestTemplateInput, BatchInput, BackgroundBatchInput,
CreateRealtimeInput, PersistCompletedRealtimeInput, CreateFlexInput and
CreateBackgroundInput. Callers stop setting it; a RequestProcessor sets
each dispatch's credential in RequestData::api_key.
Require the dispatch key id at fusillade's storage boundary: create_flex and
create_background take it in their inputs, the _as variants are folded in,
and the middleware fails a queued request early when it cannot resolve one.
The retained-payload scrub seeds its keyset with sentinels so each batch is
a plain index seek, and its test covers several partitions and a resumed run.
@tobydigz
tobydigz force-pushed the feat/dw-1013/drop-plaintext-columns branch from c3ddb9b to 19f7bd0 Compare November 6, 2026 09:51
@tobydigz
tobydigz removed this pull request from stack #2174 November 6, 2026 10:23
Rewriting retained_response_objects (706 GB, 339M rows) costs far more than
retiring its partitions, so payloads written before 16.5.0 keep
template.api_key until their partitions are retired.
@tobydigz tobydigz changed the title feat(api-keys): drop the plaintext API key columns (DW-1013) feat(fusillade)!: drop the plaintext API key columns (DW-1013) Nov 6, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 67 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="dwctl/migrations/20261105123212_drop_api_keys_secret.sql">

<violation number="1" location="dwctl/migrations/20261105123212_drop_api_keys_secret.sql:21">
P3: This column comment says secret_hash is "the only stored form of a key's secret", but `pending_secret` stores a keystore-wrapped copy of the secret in the same table. Reword it to describe secret_hash as the lookup form.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread fusillade/README.md Resolved
Comment thread fusillade/README.md Resolved
Comment thread fusillade/README.md Resolved
Comment thread dwctl/src/sync/onwards_config/mod.rs Resolved
The basic example dispatches unauthenticated with the default daemon, a
background request's key id must name an existing key, a missing secret
hash is unreachable in key sync, and secret_hash is the lookup form, not
the only stored form (pending_secret holds a wrapped copy).

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Requires human review: Drops production API key secret columns and Fusillade stored api_key fields via two migrations plus breaking fusillade-core type changes; requires human sign-off as it removes live schema data and alters the dispatch request path.

View guided diff | Turn on auto-fix | Re-trigger cubic

…et column (DW-1013)

A new onwards_config test looked keys up by the plaintext column, and its
sync query's offline data predates secret_hash becoming NOT NULL. 16.6.0
has shipped, so this lands as 17.0.0; the rollback row says so.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Requires human review: Drops plaintext API key columns and Fusillade stored api_key fields plus breaking fusillade-core type changes; requires human sign-off for production data deletion, request-path changes, and rollout sequencing.

View guided diff | Turn on auto-fix | Re-trigger cubic

@tobydigz
tobydigz added this pull request to the merge queue Nov 6, 2026
@tobydigz
tobydigz removed this pull request from the merge queue due to a manual request Nov 6, 2026
@tobydigz
tobydigz added this pull request to the merge queue Nov 6, 2026
Merged via the queue into main with commit e3c4de0 Nov 6, 2026
28 checks passed
@tobydigz
tobydigz deleted the feat/dw-1013/drop-plaintext-columns branch November 6, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants