Skip to content

fix(compute): always report the backends stage in deploy output - #656

Open
polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/7r2h3n5teszf
Open

polylane[bot] wants to merge 1 commit into
mainfrom
polylane/autofix/7r2h3n5teszf

Conversation

@polylane

@polylane polylane Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

On a redeploy of a workload whose URL is already live, datumctl compute deploy finished without printing its backends line, so the Datum website's rollout gate failed even though the deploy was healthy. The deploy output now always reports the backends stage for a workload that has a backend service, so the gate reads the line the tool documents. The tool's own output documentation lists the backends line as part of a successful publish, but the code printed it only once a backend was healthy. A live URL can briefly report no healthy member right after an instance is replaced, which dropped the line.

Production impact at 0948d3f: Deploy CLI prints one extra progress line; no serving path changes

Fix walkthrough

Watch the fix walkthrough

Watch the fix explained (1:32). Recorded for commit 0948d3f539e61bdcfc83a49cf1b3a38fd2cb417e.

Full analysis
flowchart TD
  A["Prior deploy left the URL live: edge programmed, certificate issued"] --> B["New push: datumctl compute deploy runs the replacement rollout"]
  B --> C["Watch prints Rollout complete"]
  C --> D["url.Wait polls one time"]
  D --> E{"Info.Live: URL + edge programmed + certificate"}
  E -->|true| F["Fix acts here: print the backends stage whenever a NetworkService backs the URL"]
  F --> G["Output: Backends n healthy, Edge programmed, Certificate issued"]
  G --> H["Workflow grep for Backends .*healthy matches and the gate passes"]
Loading
Detection evidence

Affected: int_17ad4b457001o98datdn4tkm

Out of scope
  • The datum.net rollout gate still greps the deploy log rather than the platform state; it now finds the line the tool documents, so it needs no change here.

Production impact

This change only affects stdout of the datumctl compute deploy plugin, distributed as the ghcr.io/datum-cloud/compute image; it does not touch the live-ness decision, the objects the tool writes, or any workload status. I checked the compute repository's only deploy edge (the ghcr.io/datum-cloud/compute image repository) and found no serving resource whose behavior changes.

Checked and not affected (1)
  • datum-cloud/compute

Observability

  • datumctl compute deploy stdout: the Backends <n> healthy across <cities> row now always appears for a workload that has a NetworkService, and the healthy count is the platform's own value.

Polylane assessed this pull request at 0948d3f.

Causal chain
  • Signal (log template): ::error::deploy output has no 'Backends .*healthy'
  • Surfacing site: datum-cloud/datum.net Deploy Datum Compute workflow, deploy job at datum-cloud/datum.net:.github/workflows/deploy-datum.yml#Check the rollout status
  • Mechanism: The step greps the deploy log for 'Backends .*healthy'. On a redeploy of an already-live URL the compute deploy returns on its first publish poll (edge already programmed, certificate already issued) while the NetworkService status still reports 0 healthy members, so the conditional backends row is never printed and the grep finds nothing.
  • Producer: datumctl compute deploy plugin (datum-cloud/compute url publish wait), at datum-cloud/compute:internal/cmd/compute/url/publish.go#progress.check
  • Trigger: Redeploy of the datum-net-gp-staging workload with its URL already live from the previous deploy
  • What happens to the failed unit today: datumctl compute deploy declares the URL before the rollout and then waits until the URL is live; Live() does not require a healthy backend.
  • Cadence check: 2 of about 20 recent main runs failed (run 63 on 2026-11-09 and run 82 on 2026-11-13); the others passed.
  • Last observed: 2026-11-13T09:12:03Z
  • Why the fix is still needed while the signal is quiet: The signal needs a narrow race, so it has not recurred since 2026-11-13; nothing merged since then changed publish.go or the workflow, and the producer code still omitted the row for a live URL with no healthy member before this fix.
  • Blast radius: 0 other resource(s), 0 other tenant(s); data at risk: none named
  • Producer evidence:
    • internal/cmd/compute/url/publish.go#progress.check guarded the backends row with 'if info.Backends.Healthy > 0', and lookup.go#Info.Live decides the wait can stop from URL + EdgeProgrammed + certificate alone.
    • GitHub Actions run 43398603202, job 131795571727 (datum.net run 82, head 67a256e7): the deploy log printed 'Rollout complete in 24s.', 'Edge programmed' and 'Certificate issued' at 2026-11-13T09:12:03Z with no backends row, and the step failed.
    • internal/cmd/compute/url/lookup_test.go:130 and state_test.go:73 assert Live() stays true with a degraded backend set, so Live() intentionally ignores backend health.
    • Reproducer: a Publish against an already-live URL with 0 healthy members prints only Edge and Certificate on the original code; the added publish_test fails there and passes with the fix.
2 files changed (+46/-2)
  • internal/cmd/compute/url/publish.go: modified, +19/-2
  • internal/cmd/compute/url/publish_test.go: modified, +27/-0

View autofix


Generated by Polylane. You can ask follow-ups by mentioning @polylane in a comment. If you close this pull request without merging, add a comment that says why. Polylane reads it to make its next fixes better.

…y member

Co-authored-by: polylane[bot] <277585245+polylane[bot]@users.noreply.github.com>
@polylane
polylane Bot requested a review from a team as a code owner November 16, 2026 05:28
@polylane polylane Bot added polylane severity:low Polylane autofix severity: low labels Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

polylane severity:low Polylane autofix severity: low

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant