Skip to content

fix: wake deployment and DNS publisher only on Instance changes they read - #653

Merged
scotwells merged 1 commit into
mainfrom
fix/deployment-instance-event-filter
Nov 13, 2026
Merged

scotwells merged 1 commit into
mainfrom
fix/deployment-instance-event-filter

Conversation

@scotwells

Copy link
Copy Markdown
Contributor

Requested by Scot Schuchert-Wells · project thread

Related: datum-cloud/enhancements#921

What

While an Instance starts or stops, it and its network interface claim get written many times by different controllers (finalizers, interface status, runtime conditions, DNS status). The WorkloadDeployment controller and the internal DNS publisher woke on every one of those writes. One workload restart caused about 35 to 40 deployment reconciles in a few seconds, and most of them did nothing. Restarting a single workload about once a minute was enough to trip ComputeControllerReconcileStorm, even though nothing was looping.

This change filters Instance and NetworkInterfaceClaim update events down to the fields each reconciler actually reads:

  • WorkloadDeployment: Instance spec (through generation), labels, owners, deletion, suspension, observed template hash, the status of Ready, Programmed, QuotaGranted and ReferencedDataReady, and the full Available condition. On claims, the Bound, Allocated and Prepared conditions plus labels, owners and deletion.
  • Internal DNS publisher: Instance spec, labels, deletion, Ready and RuntimeReady status, each interface's reference, addresses and Allocated/Programmed status, and status.dns.

Creates, deletes and periodic cache resyncs still pass, so a missed event is still picked up eventually.

Testing

  • Unit tests for each filter, covering both the fields that should wake a reconciler and the fields that shouldn't.
  • The existing controller envtests pass unchanged.
  • After merge: restart a staging workload and confirm the per-restart deployment reconcile count drops and rollouts still finish.

…read

An Instance and its network claim are written many times while it starts
and stops. The WorkloadDeployment controller and the internal DNS publisher
woke on every one of those writes, so a single workload restart caused a
few dozen reconciles that did nothing, and restarting a workload about once
a minute was enough to trip the reconcile storm alert.

Filter Instance and NetworkInterfaceClaim update events down to the fields
each reconciler reads. Creates, deletes and periodic resyncs still pass.
@scotwells
scotwells marked this pull request as ready for review November 13, 2026 01:46
@scotwells
scotwells requested a review from a team as a code owner November 13, 2026 01:46
@scotwells
scotwells requested review from ecv and kevwilliams November 13, 2026 01:46
@scotwells scotwells self-assigned this Nov 13, 2026

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

I traced every Instance and claim field each reconciler reads and found no missed wake; the filter is sound and the tests guard it, with one gap below.

  • WorkloadDeployment reads from an Instance: Status.Suspended, Status.Controller (ObservedTemplateHash), conditions Ready/Programmed/QuotaGranted/ReferencedDataReady by status, Available by status+reason+message (firstStartFailure), plus Spec.Location/Controller/SchedulingGates/TemplateHash, labels, deletionTimestamp and ownerRefs. Spec changes bump generation (Instance has a status subresource, instance_types.go:1161), so all of it is covered. No annotation of the Instance is read after create, so dropping annotation-only updates is safe.
  • DNS publisher reads Ready and RuntimeReady by status, interface name/ref/addresses/Allocated/Programmed, Status.DNS, the location label, spec interfaces, UID, deletionTimestamp. All covered; ExternalAddresses and Assignments are not read, so leaving them out is right.
  • Claim predicate covers Bound/Allocated/Prepared by status+reason+message (what networkInterfaceClaimSatisfied and networkInterfaceClaimRejection read), plus ownerRefs, deletionTimestamp and UID (networkInterfaceClaimHoldingFor).
  • Edge cases: a type-assertion failure passes, Generic events pass (nil GenericFunc), Create/Delete pass, same-resourceVersion resyncs pass. go test -race -count=1 on TestInstanceChanged* and TestNetworkInterfaceClaimChanged* passes at 19dc061. The full package fails locally only because envtest's etcd is absent.
  • Other watchers are unaffected: predicates attach per handler (forked_controller.go appends own.predicates / forInput.predicates to that watch only), and the Instance informer is shared. Instance, quota and referenced-data controllers still see every event.
  • Guard check: I removed each guard once and reran. Dropping Status.Controller, addresses, ownerRefs, claim Prepared, Status.DNS or Suspended each fails a test. Dropping InstanceNetworkInterfaceProgrammed from dnsInterfacesChanged passes, so that guard is uncaught.

Follow-ups:

  • Add a watch_predicates_test.go case where only an interface's Programmed condition flips, so the guard above is pinned.
  • The recovery claim is weaker than it reads: SyncPeriod is unset (cmd/main.go only sets a transform), so the resync is controller-runtime's default of about 10h. The DNS publisher self-requeues every 10-15m, so it is fine. The WorkloadDeployment reconciler requeues only while waiting on a claim, so a field added to its reads without updating the predicate would sit up to ~10h. Say that in the comment, or add a test that fails when reconcileInstanceGates reads a condition the predicate lacks.

@kevwilliams kevwilliams left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked every Instance and claim field read by rollout gating, the Parallel policy, claim gating and DNS eligibility; the filters cover each one.

@scotwells
scotwells merged commit 84e9865 into main Nov 13, 2026
14 checks passed
@scotwells
scotwells deleted the fix/deployment-instance-event-filter branch November 13, 2026 02:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants