fix: reconcile starvation around configcheck pods and pipeline event debouncing#261
Merged
aa1ex merged 4 commits intoJul 16, 2026
Merged
Conversation
…uster aggregators
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The e2e suite flakes with
Timed out after 120swaiting for an agent config secret (seen on reruns of #253/#257/#258 and reproducible on a v1.32.11 kind cluster). Tracing it surfaced four independent defects, all starving the serial (MaxConcurrentReconciles=1) reconcile workers:unable to create new content in namespace ... because it is being terminated), sogetCheckResultholds the worker for the fullConfigCheckTimeout(300s); one CI run showed 30 failed reconciles plus two back-to-back 300s stalls.ClusterVectorAggregators from e2e kept pointingresourceNamespaceat deleted namespaces, spammingfailed to create or update Secret: namespaces ... not found.reconcileWithDelay) resets its ticker on every incoming event, so events arriving faster than the 10s delay postpone the flush indefinitely and agents stop being notified at all.envreferencing a missing secret →CreateContainerConfigError; the configcheck pod inherits the CR's env) stays Pending forever, andgetCheckResultonly reacted toModifiedevents withSucceeded/Failedphases — every reconcile burned the full 300s. The same watch loop also ignored the initial syntheticAddedevents andDeletedevents, and treated a closed watch channel as success, letting an unvalidated config through.Fixes (one commit per defect)
configcheck.Runreturns a newErrConfigcheckSkippedsentinel when the target namespace is terminating or gone; all call sites treat it as a no-op skip.spec.resourceNamespace) is terminating or gone, via a sharedk8s.NamespaceIsTerminatinghelper; the podmonitor e2e now deletes its cluster-scoped aggregators inAfterAll.reconcileWithDelaycoalesces events and flushes everydelayinstead of resetting the ticker per event, so notifications survive continuous load.getCheckResultfails fast with the container's waiting reason when the pod is unstartable (CreateContainerConfigError,InvalidImageName) — the CR gets a Failed status carrying the actual cause within seconds; it also handlesAdded, errors onDeleted, and errors on a closed watch channel instead of reporting success.Verification
configcheck/namespace_test.go,configcheck/watch_test.go,utils/k8s/namespace_test.go,cmd/manager/main_test.go);make testgreen.VectorAggregatorwhoseenvreferences a missing secret reachesconfigCheckResult: falsewith reasonconfigcheck pod cannot start, CreateContainerConfigError: secret "no-such-secret" not foundin ~25s; before the fix its configcheck pod sat inCreateContainerConfigErroruntil the 300s timeout on every reconcile.Behavior note: an unstartable configcheck pod is now reported as a validation failure (Failed status with reason) instead of a silent timeout-and-requeue loop; fixing the CR or creating the missing secret re-triggers reconciliation as usual.
Follow-up intentionally left out: raising
MaxConcurrentReconcilesfor the Vector controllers — serialization amplifies any single stall and deserves its own discussion.