Skip to content

fix: Reconcile terminating objects that lost the replicator label selector - #922

Open
dlotterman wants to merge 1 commit into
datum-cloud:mainfrom
dlotterman:fix/replicator-terminating-unlabeled-secret
Open

dlotterman wants to merge 1 commit into
datum-cloud:mainfrom
dlotterman:fix/replicator-terminating-unlabeled-secret

Conversation

@dlotterman

@dlotterman dlotterman commented Nov 18, 2026 •

Copy link
Copy Markdown

Problem

Secrets referenced by a SecurityPolicy can get stuck Terminating forever, held by the finalizer gateway.networking.datumapis.com/gateway-resource-replicator.

Observed on production (several Secrets stuck for days) and reproduced with a minimal case:

  1. Create a Secret and a SecurityPolicy (spec.basicAuth.users -> that Secret). The Secret gets the networking.datumapis.com/gateway-sync label and the replicator finalizer.
  2. Delete the SecurityPolicy, then delete the Secret.
  3. The Secret stays Terminating indefinitely: no labels, finalizer still present. datumctl delete secret blocks.

Controls: deleting the Secret first (policy still present) works; a Secret never referenced by a policy has no finalizer and deletes cleanly.

Cause

  • On SecurityPolicy deletion, reconcileReferencedSecretLabels -> gcSecretSyncLabels strips the gateway-sync label from Secrets that are no longer referenced, but leaves the replicator finalizer on them.
  • typedEnqueueRequestForGVK drops every upstream event whose object does not match the configured label selector, so the later delete of the now-unlabeled Secret is never enqueued.
  • Reconcile / finalizeResource (which removes the finalizer) therefore never runs for it.

Fix

typedEnqueueRequestForGVK now also enqueues objects that are terminating and still carry the replicator finalizer, regardless of the label selector. Reconcile does not check labels, so the existing finalizeResource path runs: it deletes the downstream copy and anchor and removes the finalizer.

Secrets already stuck should also clear once this ships, since the controller re-lists objects on start and the terminating ones now pass the filter (not verified; I could only test with fake clients).

Verification

  • New test TestTypedEnqueueRequestForGVKEnqueuesTerminatingObjectWithFinalizer: fails without the change, passes with it. It also asserts that a terminating object with a foreign finalizer and a live unlabeled object stay filtered.
  • go test ./internal/controller/ passes.
  • Not run: make lint, e2e.

Alternative / larger fix (not done here)

This change fixes the stuck-forever symptom narrowly. A more thorough fix would be for gcSecretSyncLabels to finalize the Secret in the same step it removes the label: delete the downstream mirror and anchor and remove the replicator finalizer. That would also address a related issue I noticed but did not verify: when the label is stripped from a live Secret, the Secret leaves the watch filter too, so its downstream mirror may be left behind. It changes controller lifecycle behavior more, so I left it for maintainer preference. Happy to rework the PR in that direction.

@dlotterman
dlotterman requested a review from a team as a code owner November 18, 2026 20:48
…ector

gcSecretSyncLabels removes the gateway-sync label from Secrets once no
SecurityPolicy references them, but the replicator finalizer stays on the
Secret. The upstream watch filters events by the label selector, so a
later delete of that Secret never reaches finalizeResource and the
Secret stays Terminating forever.

Also enqueue objects that are terminating and still hold the replicator
finalizer, regardless of the label selector.
@dlotterman
dlotterman force-pushed the fix/replicator-terminating-unlabeled-secret branch from 2c2a82c to 7829f8b Compare November 18, 2026 20:49
@dlotterman

Copy link
Copy Markdown
Author

I think this is RFR / RFM

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

Finalizer handling is correct and the change adds no new cache or watch cost; the new test fails without the fix.

  • The predicate is relaxed only for objects that are terminating and still hold the replicator finalizer. Reconcile has no selector check, so it goes through finalizeResource, which calls finalize (deletes the downstream copy and anchor) before removeFinalizer. The finalizer is never dropped before downstream cleanup, so no replicated Secrets are orphaned on the edge.
  • Nothing widens: the informer already lists and watches every Secret cluster-wide, the selector was only a filter inside the map func. The extra cost is one enqueue per terminating Secret that still carries the finalizer.
  • Label re-add race is benign: a terminating object cannot be revived, and the finalize path ignores labels. Reconcile does its own fresh Get, so a stale event cannot finalize a live object.
  • Tamper check: removing && !isTerminatingWithReplicatorFinalizer(obj) makes TestTypedEnqueueRequestForGVKEnqueuesTerminatingObjectWithFinalizer fail (expected 1, actual 0). It is a handler-level unit test only, not an end-to-end test with a real Secret.

Follow-ups:

  • Add an end-to-end case (SecurityPolicy deleted first, then the Secret) asserting the finalizer is gone and the downstream Secret deleted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants