Skip to content

fix: Keep the manager up when a standby takes the lease - #536

Merged
vvoytovych merged 2 commits into
mainfrom
fix/multicluster-runtime-takeover-exit
Nov 18, 2026
Merged

vvoytovych merged 2 commits into
mainfrom
fix/multicluster-runtime-takeover-exit

Conversation

@vvoytovych

Copy link
Copy Markdown
Contributor

Summary

When a standby manager takes the lease, it can exit with code 1, and Milo then refuses DNSRecordSet writes. This change requires a multicluster-runtime version that has the fix.

  • A standby engages every project, but its controllers do not start until it leads. Each controller keeps one pending watch for each project.
  • When a project is deleted while the manager is the standby, the project's cache stops, but its pending watch stays.
  • multicluster-runtime v0.25.2 starts that watch when the standby takes the lease. The watch fails on the stopped cache, the controller does not start, and the manager exits 1 (kubernetes-sigs/multicluster-runtime#199).
 on takeover, for each pending watch
+  if the project was removed before the controller started
+    skip the watch
   start the watch

The fix is merged upstream and backported to release-0.25 (#205), but no release has it yet. So I require the release-0.25 commit, and I will move to v0.25.3 when it is released:

-	sigs.k8s.io/multicluster-runtime v0.25.2
+	sigs.k8s.io/multicluster-runtime v0.25.3-0.20261116153018-6d88a2667465

Evidence

The new test runs two managers in envtest. One project is deleted while the second manager is the standby. Then the leader frees the lease, and the test requires the new leader to reconcile the project that remains.

two managers, two projects
delete project A while manager 2 is the standby
manager 1 frees the lease
expect manager 2 to reconcile the DNSZone of project B
  • Before (v0.25.2): the test fails.

    the new leader never reconciled the DNSZone of the project that remains: its controller did not start, so the manager stopped
    a manager stopped with an error: handler {…} was not added to shared informer because it has stopped already
    --- FAIL: TestStandbyTakesTheLeaseAfterAProjectIsDeleted (74.97s)
    
  • After (this version): the test passes three times out of three, in about 15 s each. The rest of the unit tests pass, and task lint reports 0 issues.

These runs are a local tamper test on envtest 1.37.0, because the pinned 1.36.0 is not on my machine. CI on the branch is the verification.

The test commit also changes the replica harness in cmd/nonleader_test.go. The harness takes a project name, shares the zone replicator setup, and logs a manager's error when the test ends. TestSingleReplicaWithoutLeaderElection keeps its behaviour.

In production

The manager exited this way three times in production. The clusters whose watch failed were clusters that the standby had removed, except one cluster, at a gap in the standby's log. One time, the standby was the only ready pod, and Milo refused hundreds of DNSRecordSet writes.

The read that tells this change works, at the next lease takeover (a node upgrade or a roll):

  • the manager does not restart with exit code 1:

    max by (pod) (kube_pod_container_status_last_terminated_exitcode{namespace="datum-dns-system",pod=~"dns-operator-controller-manager-.*",container="manager"})
    
  • the mutating webhook refuses no writes:

    sum by (rejection_code, error_type) (increase(apiserver_admission_webhook_rejection_count{name="mdnsrecordset.kb.io"}[1m]))
    

Merge Danger

Door: two-way

A revert returns the module to v0.25.2.

Blast Radius: controllers

Over v0.25.2, the new version changes 7 lines in one Go file of the library: how a controller starts the watch of a cluster. A watch whose cluster was removed before the controller started is skipped. A watch whose cluster is still engaged starts as before.

Related to #252

A standby engages every project, so each of its controllers keeps a
pending watch per project until it leads. When a project is deleted
first, its watch stays and its cache stops. The test deletes one of two
projects on a standby, frees the lease, and requires the new leader to
reconcile the project that remains.

It fails on multicluster-runtime v0.25.2, whose controller starts the
deleted project's source on the stopped cache and fails to start.

The replica harness takes a project name, shares the zone replicator
setup, and logs a manager's error when the test ends.
A standby engages every project, so each of its controllers keeps a
pending watch for each project until it leads. multicluster-runtime
v0.25.2 starts the pending watch of a project that was deleted first,
on a cache that has stopped. The controller then fails to start, and
the manager exits 1. On 2026-11-09 this refused 419 record set writes.

Require multicluster-runtime at release-0.25 6d88a2667, which skips the
source of a cluster that was disengaged before the controller started
(kubernetes-sigs/multicluster-runtime#199, backported in #205). Over
v0.25.2 it changes one file, pkg/controller/controller.go. Move to
v0.25.3 when it is released.

The takeover test fails on v0.25.2 and passes on this version.
@vvoytovych vvoytovych self-assigned this Nov 18, 2026
@vvoytovych
vvoytovych marked this pull request as ready for review November 18, 2026 11:05
@vvoytovych
vvoytovych requested a review from a team as a code owner November 18, 2026 11:05

@andrewb3000 andrewb3000 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked

  • go.mod:26: I could not read upstream offline, so I did not check that 6d88a2667465 is on release-0.25 and differs from v0.25.2 only in the 7 lines described. The pseudo-version's base (v0.25.2), the unchanged go.mod hash in go.sum, and green CI are consistent with that. Open a follow-up to move to v0.25.3 once it is tagged.
  • cmd/takeover_test.go:24: The regression test runs in the existing test workflow (test.yml via ci.yaml). It passed on this head, and it fails on v0.25.2 according to the description, so it guards the fix.

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

The pin is the tip of upstream release-0.25, which is the merge of kubernetes-sigs/multicluster-runtime#205. Over v0.25.2 it changes only the 7 lines in pkg/controller/controller.go; the rest is CI workflow bumps and upstream tests.

  • The fix returns early from a pending watch when the cluster's engage context is already done, so a project removed while this replica was the standby no longer fails the controller's start. A project still engaged starts its watch as before.
  • The check and the source start are not atomic. A project deleted in the instant between them still fails the start the old way, but that window is far narrower than "any deletion during standby".
  • The new takeover test covers the production sequence: engage both, delete one, free the lease, require the survivor reconciled. CI is green on it.

Follow-ups:

  • Move to v0.25.3 once it is tagged; a pseudo-version on a release branch gets no Renovate release-note trail.

@vvoytovych
vvoytovych merged commit 1e41460 into main Nov 18, 2026
17 checks passed
@vvoytovych
vvoytovych deleted the fix/multicluster-runtime-takeover-exit branch November 18, 2026 11:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants