Repository navigation
feat(fusillade)!: drop the plaintext API key columns (DW-1013) - #2173
Conversation
Deploying control-layer with
|
| Latest commit: |
b1c76de
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://cb6c2e60.control-layer.pages.dev |
| Branch Preview URL: | https://feat-dw-1013-drop-plaintext.control-layer.pages.dev |
|
Running ultrareview automatically — this PR appears to change a database schema or migrate data, where a missed bug can corrupt or lose records, so a deeper multi-pass review is worth running. I'll post findings when complete. |
There was a problem hiding this comment.
Ultrareview ran automatically — this PR appears to change a database schema or migrate data, where a missed bug can corrupt or lose records, so a deeper multi-pass review is worth running.
All reported issues were addressed across 36 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
4cf36fd to
c3ddb9b
Compare
The previous release neither reads nor writes api_keys.secret or fusillade's stored api_key, so dropping them removes the last stored copies of key secrets. secret_hash becomes NOT NULL, which its validated check makes instant; the check is then redundant. The fusillade views that project api_key are recreated without it in the same migration. Retained response snapshots stop carrying an api_key; ones that still do decode as before. Rolling back past the previous release breaks key creation and fusillade writes (docs/migrations.md).
Fusillade stores no key secret and dispatches by API key id, so the api_key fields on its inputs were ignored. Removing them stops callers passing secrets that go nowhere. Also scrubs key secrets from retained-response payloads written by earlier releases, in a resumable batched no-transaction migration, and drops batches.api_key in its own migration so its lock is not held with the template locks. Migration comments now say what DROP COLUMN does and does not remove, and stale column and index comments are replaced. Adds a test for the down migrations. BREAKING CHANGE: fusillade-core removes the `api_key` field from RequestTemplate, RequestTemplateInput, BatchInput, BackgroundBatchInput, CreateRealtimeInput, PersistCompletedRealtimeInput, CreateFlexInput and CreateBackgroundInput. Callers stop setting it; a RequestProcessor sets each dispatch's credential in RequestData::api_key.
Require the dispatch key id at fusillade's storage boundary: create_flex and create_background take it in their inputs, the _as variants are folded in, and the middleware fails a queued request early when it cannot resolve one. The retained-payload scrub seeds its keyset with sentinels so each batch is a plain index seek, and its test covers several partitions and a resumed run.
c3ddb9b to
19f7bd0
Compare
Rewriting retained_response_objects (706 GB, 339M rows) costs far more than retiring its partitions, so payloads written before 16.5.0 keep template.api_key until their partitions are retired.
There was a problem hiding this comment.
1 issue found across 67 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="dwctl/migrations/20261105123212_drop_api_keys_secret.sql">
<violation number="1" location="dwctl/migrations/20261105123212_drop_api_keys_secret.sql:21">
P3: This column comment says secret_hash is "the only stored form of a key's secret", but `pending_secret` stores a keystore-wrapped copy of the secret in the same table. Reword it to describe secret_hash as the lookup form.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
The basic example dispatches unauthenticated with the default daemon, a background request's key id must name an existing key, a missing secret hash is unreachable in key sync, and secret_hash is the lookup form, not the only stored form (pending_secret holds a wrapped copy).
There was a problem hiding this comment.
0 issues found across 3 files (changes from recent commits).
Requires human review: Drops production API key secret columns and Fusillade stored api_key fields via two migrations plus breaking fusillade-core type changes; requires human sign-off as it removes live schema data and alters the dispatch request path.
View guided diff | Turn on auto-fix | Re-trigger cubic
…et column (DW-1013) A new onwards_config test looked keys up by the plaintext column, and its sync query's offline data predates secret_hash becoming NOT NULL. 16.6.0 has shipped, so this lands as 17.0.0; the rollback row says so.
There was a problem hiding this comment.
0 issues found across 3 files (changes from recent commits).
Requires human review: Drops plaintext API key columns and Fusillade stored api_key fields plus breaking fusillade-core type changes; requires human sign-off for production data deletion, request-path changes, and rollout sequencing.
View guided diff | Turn on auto-fix | Re-trigger cubic
Part 7 of 7 for DW-1005 (DW-1013). Merge only after the release containing #2172 (16.5.0) is fully deployed.
What changes
20261105123212_drop_api_keys_secret:secret_hash SET NOT NULL(instant with the validated check), drop the check, dropapi_keys.secret, refresh stale column and index comments.20261105123212_drop_stored_api_keys: recreateactive_request_templates,request_templates_allandrequest_templateswithoutapi_key, then droprequest_templates_g2.api_key.20261105143830_drop_batches_api_key: dropsbatches.api_keyon its own, so its lock isn't held with the template locks.api_keyremoved fromRequestTemplateand every request/batch input type;create_flex/create_backgroundrequire the API key id.RequestData.api_keystays as the in-memory per-dispatch credential.api_key; older ones still decode.docs/migrations.md: from 16.6.0, roll back no further than 16.5.0.Rollout
16.5.0 neither reads nor writes these columns, so its pods keep working while the migrations run.
Not removed
DROP COLUMNdoesn't rewrite rows: old bytes stay in table pages, WAL and backups until rows are rewritten (VACUUM FULL/pg_repack) and backups expire.template.api_keyuntil their daily partitions are retired (retained_response_retirement_enabled). Rewriting that table (~706 GB) to scrub them costs far more than retirement.Testing
Targeted tests for the touched areas, the down-migration test, and
just lint rustpass locally; CI runs the full suite.