Repository navigation
fix: Reconcile terminating objects that lost the replicator label selector - #922
Open
dlotterman wants to merge 1 commit into
Open
dlotterman wants to merge 1 commit into
dlotterman wants to merge 1 commit into
Conversation
…ector gcSecretSyncLabels removes the gateway-sync label from Secrets once no SecurityPolicy references them, but the replicator finalizer stays on the Secret. The upstream watch filters events by the label selector, so a later delete of that Secret never reaches finalizeResource and the Secret stays Terminating forever. Also enqueue objects that are terminating and still hold the replicator finalizer, regardless of the label selector.
dlotterman
force-pushed
the
fix/replicator-terminating-unlabeled-secret
branch
from
November 18, 2026 20:49
2c2a82c to
7829f8b
Compare
Author
|
I think this is RFR / RFM |
scotwells
approved these changes
Nov 18, 2026
ecv
approved these changes
Nov 18, 2026
ecv
left a comment
Contributor
There was a problem hiding this comment.
Approve.
Finalizer handling is correct and the change adds no new cache or watch cost; the new test fails without the fix.
- The predicate is relaxed only for objects that are terminating and still hold the replicator finalizer. Reconcile has no selector check, so it goes through
finalizeResource, which callsfinalize(deletes the downstream copy and anchor) beforeremoveFinalizer. The finalizer is never dropped before downstream cleanup, so no replicated Secrets are orphaned on the edge. - Nothing widens: the informer already lists and watches every Secret cluster-wide, the selector was only a filter inside the map func. The extra cost is one enqueue per terminating Secret that still carries the finalizer.
- Label re-add race is benign: a terminating object cannot be revived, and the finalize path ignores labels. Reconcile does its own fresh Get, so a stale event cannot finalize a live object.
- Tamper check: removing
&& !isTerminatingWithReplicatorFinalizer(obj)makesTestTypedEnqueueRequestForGVKEnqueuesTerminatingObjectWithFinalizerfail (expected 1, actual 0). It is a handler-level unit test only, not an end-to-end test with a real Secret.
Follow-ups:
- Add an end-to-end case (SecurityPolicy deleted first, then the Secret) asserting the finalizer is gone and the downstream Secret deleted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Secrets referenced by a
SecurityPolicycan get stuckTerminatingforever, held by the finalizergateway.networking.datumapis.com/gateway-resource-replicator.Observed on production (several Secrets stuck for days) and reproduced with a minimal case:
SecurityPolicy(spec.basicAuth.users-> that Secret). The Secret gets thenetworking.datumapis.com/gateway-synclabel and the replicator finalizer.SecurityPolicy, then delete the Secret.Terminatingindefinitely: no labels, finalizer still present.datumctl delete secretblocks.Controls: deleting the Secret first (policy still present) works; a Secret never referenced by a policy has no finalizer and deletes cleanly.
Cause
SecurityPolicydeletion,reconcileReferencedSecretLabels->gcSecretSyncLabelsstrips thegateway-synclabel from Secrets that are no longer referenced, but leaves the replicator finalizer on them.typedEnqueueRequestForGVKdrops every upstream event whose object does not match the configured label selector, so the later delete of the now-unlabeled Secret is never enqueued.Reconcile/finalizeResource(which removes the finalizer) therefore never runs for it.Fix
typedEnqueueRequestForGVKnow also enqueues objects that are terminating and still carry the replicator finalizer, regardless of the label selector.Reconciledoes not check labels, so the existingfinalizeResourcepath runs: it deletes the downstream copy and anchor and removes the finalizer.Secrets already stuck should also clear once this ships, since the controller re-lists objects on start and the terminating ones now pass the filter (not verified; I could only test with fake clients).
Verification
TestTypedEnqueueRequestForGVKEnqueuesTerminatingObjectWithFinalizer: fails without the change, passes with it. It also asserts that a terminating object with a foreign finalizer and a live unlabeled object stay filtered.go test ./internal/controller/passes.make lint, e2e.Alternative / larger fix (not done here)
This change fixes the stuck-forever symptom narrowly. A more thorough fix would be for
gcSecretSyncLabelsto finalize the Secret in the same step it removes the label: delete the downstream mirror and anchor and remove the replicator finalizer. That would also address a related issue I noticed but did not verify: when the label is stripped from a live Secret, the Secret leaves the watch filter too, so its downstream mirror may be left behind. It changes controller lifecycle behavior more, so I left it for maintainer preference. Happy to rework the PR in that direction.