Skip to content

test(mcpserver): wait for the server before rotating its certificate - #921

Open
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/vlniui5ker6m
Open

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/vlniui5ker6m

Conversation

@polylane

@polylane polylane Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

The required CI check on the default branch failed on a test that starts the MCP server and replaces its certificate at the same time. The test replaced the certificate before the server finished loading the first one, so the server exited at startup and nothing answered, and the run waited 20 seconds before it failed. The test helper now waits until the server answers before any test can rotate the certificate. The failing run reported "loading serving certificate: tls: failed to find any PEM data in certificate input", a startup error that only the certificate load produces, and 2 of 40 runs failed on the unmodified tree. Waiting for a successful health response removes the window, and the assertions about which certificate is served stay the same.

Production impact at e3791a0: Test helper waits for the MCP server before rotation

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:26). Recorded for commit e3791a0f775e8755c3fd23eadb4c489dcff1087d.

Full analysis
sequenceDiagram
    participant T as Test
    participant S as serve goroutine
    participant F as tls.crt on disk
    T->>S: start serving
    Note over T,S: before: the helper returned here
    T->>F: truncate and write cert 2
    S->>F: certwatcher reads the files
    F-->>S: empty PEM
    S-->>T: loading serving certificate error
    Note over T,S: after: the helper waits for a 200 from /healthz, then the test rotates
Loading

Production impact

The change edits one Go test file, internal/mcpserver/tls_test.go. A _test.go file is excluded from the manager and extension-server builds, and the repository graph lists no deploy target for this repository, so no deployed resource changes behavior on merge. I checked the certificate load path in internal/mcpserver/server.go and the watcher poll interval of 10s against the test's 20s rotation window.

Checked and not affected (1)
  • network-services-operator

Observability

  • the test now fails with "serve returned before it answered: " when the server exits at startup, instead of a 20 second "the rotated certificate was never served"
  • CI keeps publishing the unit and envtest results of make test-suite as the required check

Polylane assessed this pull request at e3791a0.

Causal chain
  • Signal (alert): CI failing: CI on datum-cloud/network-services-operator@main
  • Surfacing site: repository github repo_17ad4f7cf001j4is53ayrp27 (datum-cloud/network-services-operator) at datum-cloud/network-services-operator:internal/mcpserver/tls_test.go#TestPicksUpARotatedCertificateWithoutARestart
  • Mechanism: The test rewrites tls.crt and tls.key right after it starts serve on a goroutine. serve reads the same files in certwatcher.New, sees the truncated certificate, returns "loading serving certificate: tls: failed to find any PEM data in certificate input", and the test finds no listener for 20s.
  • Producer: repository datum-cloud/network-services-operator, at internal/mcpserver/tls_test.go#startServing
  • Trigger: Merge of PR fix: Close webhook connections before stopping #919 (7f6caf1) pushed to main and ran the required CI check, which executes make test-suite
  • Cadence check: Failure takes the full 20s rotationTimeout; the certificate watcher poll interval is 10s
  • Last observed: 2026-11-18T17:07:44Z
  • Producer evidence:
    • internal/mcpserver/tls_test.go#startServing returns listener.Addr() as soon as the serve goroutine starts, before serve loads the certificate
    • Failing log: "loading serving certificate: tls: failed to find any PEM data in certificate input", a message only certwatcher.New (startup) produces
    • Reproducer: 2 of 40 runs of the rotation test failed on the unmodified tree at 7f6caf1
    • writeServingCertificate uses os.WriteFile, which truncates tls.crt to zero bytes before it writes the rotated certificate
1 file changed (+47/-6)
  • internal/mcpserver/tls_test.go: modified, +47/-6

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane
polylane Bot requested a review from a team as a code owner November 18, 2026 17:33
@polylane polylane Bot added the polylane label Nov 18, 2026
@polylane
polylane Bot requested a review from a team November 18, 2026 17:33
@polylane polylane Bot added the severity:medium Polylane autofix severity: medium label Nov 18, 2026

@ecv ecv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve.

The helper now waits for a 200 from /healthz before any test can rewrite the certificate files, and fails fast if serve returns first, which closes the startup race behind "failed to find any PEM data". The assertions on which certificate is served are unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:medium Polylane autofix severity: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants