Repository navigation
fix: wake deployment and DNS publisher only on Instance changes they read - #653
Merged
Merged
Conversation
…read An Instance and its network claim are written many times while it starts and stops. The WorkloadDeployment controller and the internal DNS publisher woke on every one of those writes, so a single workload restart caused a few dozen reconciles that did nothing, and restarting a workload about once a minute was enough to trip the reconcile storm alert. Filter Instance and NetworkInterfaceClaim update events down to the fields each reconciler reads. Creates, deletes and periodic resyncs still pass.
ecv
approved these changes
Nov 13, 2026
ecv
left a comment
Contributor
There was a problem hiding this comment.
Approve.
I traced every Instance and claim field each reconciler reads and found no missed wake; the filter is sound and the tests guard it, with one gap below.
- WorkloadDeployment reads from an Instance:
Status.Suspended,Status.Controller(ObservedTemplateHash), conditions Ready/Programmed/QuotaGranted/ReferencedDataReady by status, Available by status+reason+message (firstStartFailure), plus Spec.Location/Controller/SchedulingGates/TemplateHash, labels, deletionTimestamp and ownerRefs. Spec changes bump generation (Instance has a status subresource,instance_types.go:1161), so all of it is covered. No annotation of the Instance is read after create, so dropping annotation-only updates is safe. - DNS publisher reads Ready and RuntimeReady by status, interface name/ref/addresses/Allocated/Programmed,
Status.DNS, the location label, spec interfaces, UID, deletionTimestamp. All covered;ExternalAddressesandAssignmentsare not read, so leaving them out is right. - Claim predicate covers Bound/Allocated/Prepared by status+reason+message (what
networkInterfaceClaimSatisfiedandnetworkInterfaceClaimRejectionread), plus ownerRefs, deletionTimestamp and UID (networkInterfaceClaimHoldingFor). - Edge cases: a type-assertion failure passes, Generic events pass (nil
GenericFunc), Create/Delete pass, same-resourceVersion resyncs pass.go test -race -count=1onTestInstanceChanged*andTestNetworkInterfaceClaimChanged*passes at 19dc061. The full package fails locally only because envtest's etcd is absent. - Other watchers are unaffected: predicates attach per handler (
forked_controller.goappendsown.predicates/forInput.predicatesto that watch only), and the Instance informer is shared. Instance, quota and referenced-data controllers still see every event. - Guard check: I removed each guard once and reran. Dropping Status.Controller, addresses, ownerRefs, claim Prepared, Status.DNS or Suspended each fails a test. Dropping
InstanceNetworkInterfaceProgrammedfromdnsInterfacesChangedpasses, so that guard is uncaught.
Follow-ups:
- Add a
watch_predicates_test.gocase where only an interface's Programmed condition flips, so the guard above is pinned. - The recovery claim is weaker than it reads:
SyncPeriodis unset (cmd/main.goonly sets a transform), so the resync is controller-runtime's default of about 10h. The DNS publisher self-requeues every 10-15m, so it is fine. The WorkloadDeployment reconciler requeues only while waiting on a claim, so a field added to its reads without updating the predicate would sit up to ~10h. Say that in the comment, or add a test that fails whenreconcileInstanceGatesreads a condition the predicate lacks.
kevwilliams
approved these changes
Nov 13, 2026
kevwilliams
left a comment
Contributor
There was a problem hiding this comment.
Checked every Instance and claim field read by rollout gating, the Parallel policy, claim gating and DNS eligibility; the filters cover each one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Scot Schuchert-Wells · project thread
Related: datum-cloud/enhancements#921
What
While an Instance starts or stops, it and its network interface claim get written many times by different controllers (finalizers, interface status, runtime conditions, DNS status). The WorkloadDeployment controller and the internal DNS publisher woke on every one of those writes. One workload restart caused about 35 to 40 deployment reconciles in a few seconds, and most of them did nothing. Restarting a single workload about once a minute was enough to trip
ComputeControllerReconcileStorm, even though nothing was looping.This change filters Instance and NetworkInterfaceClaim update events down to the fields each reconciler actually reads:
status.dns.Creates, deletes and periodic cache resyncs still pass, so a missed event is still picked up eventually.
Testing