Repository navigation
fix: Keep the manager up when a standby takes the lease - #536
Merged
Merged
Conversation
A standby engages every project, so each of its controllers keeps a pending watch per project until it leads. When a project is deleted first, its watch stays and its cache stops. The test deletes one of two projects on a standby, frees the lease, and requires the new leader to reconcile the project that remains. It fails on multicluster-runtime v0.25.2, whose controller starts the deleted project's source on the stopped cache and fails to start. The replica harness takes a project name, shares the zone replicator setup, and logs a manager's error when the test ends.
A standby engages every project, so each of its controllers keeps a pending watch for each project until it leads. multicluster-runtime v0.25.2 starts the pending watch of a project that was deleted first, on a cache that has stopped. The controller then fails to start, and the manager exits 1. On 2026-11-09 this refused 419 record set writes. Require multicluster-runtime at release-0.25 6d88a2667, which skips the source of a cluster that was disengaged before the controller started (kubernetes-sigs/multicluster-runtime#199, backported in #205). Over v0.25.2 it changes one file, pkg/controller/controller.go. Move to v0.25.3 when it is released. The takeover test fails on v0.25.2 and passes on this version.
andrewb3000
approved these changes
Nov 18, 2026
andrewb3000
left a comment
Contributor
There was a problem hiding this comment.
Checked
go.mod:26: I could not read upstream offline, so I did not check that 6d88a2667465 is on release-0.25 and differs from v0.25.2 only in the 7 lines described. The pseudo-version's base (v0.25.2), the unchanged go.mod hash in go.sum, and green CI are consistent with that. Open a follow-up to move to v0.25.3 once it is tagged.cmd/takeover_test.go:24: The regression test runs in the existing test workflow (test.yml via ci.yaml). It passed on this head, and it fails on v0.25.2 according to the description, so it guards the fix.
ecv
approved these changes
Nov 18, 2026
ecv
left a comment
Contributor
There was a problem hiding this comment.
Approve.
The pin is the tip of upstream release-0.25, which is the merge of kubernetes-sigs/multicluster-runtime#205. Over v0.25.2 it changes only the 7 lines in pkg/controller/controller.go; the rest is CI workflow bumps and upstream tests.
- The fix returns early from a pending watch when the cluster's engage context is already done, so a project removed while this replica was the standby no longer fails the controller's start. A project still engaged starts its watch as before.
- The check and the source start are not atomic. A project deleted in the instant between them still fails the start the old way, but that window is far narrower than "any deletion during standby".
- The new takeover test covers the production sequence: engage both, delete one, free the lease, require the survivor reconciled. CI is green on it.
Follow-ups:
- Move to v0.25.3 once it is tagged; a pseudo-version on a release branch gets no Renovate release-note trail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a standby manager takes the lease, it can exit with code 1, and Milo then refuses DNSRecordSet writes. This change requires a multicluster-runtime version that has the fix.
The fix is merged upstream and backported to
release-0.25(#205), but no release has it yet. So I require therelease-0.25commit, and I will move to v0.25.3 when it is released:Evidence
The new test runs two managers in envtest. One project is deleted while the second manager is the standby. Then the leader frees the lease, and the test requires the new leader to reconcile the project that remains.
Before (v0.25.2): the test fails.
After (this version): the test passes three times out of three, in about 15 s each. The rest of the unit tests pass, and
task lintreports 0 issues.These runs are a local tamper test on envtest 1.37.0, because the pinned 1.36.0 is not on my machine. CI on the branch is the verification.
The test commit also changes the replica harness in
cmd/nonleader_test.go. The harness takes a project name, shares the zone replicator setup, and logs a manager's error when the test ends.TestSingleReplicaWithoutLeaderElectionkeeps its behaviour.In production
The manager exited this way three times in production. The clusters whose watch failed were clusters that the standby had removed, except one cluster, at a gap in the standby's log. One time, the standby was the only ready pod, and Milo refused hundreds of DNSRecordSet writes.
The read that tells this change works, at the next lease takeover (a node upgrade or a roll):
the manager does not restart with exit code 1:
the mutating webhook refuses no writes:
Merge Danger
Door: two-way
A revert returns the module to v0.25.2.
Blast Radius: controllers
Over v0.25.2, the new version changes 7 lines in one Go file of the library: how a controller starts the watch of a cluster. A watch whose cluster was removed before the controller started is skipped. A watch whose cluster is still engaged starts as before.
Related to #252