[Visual Test] Golden image is completely black, causing compare_visual_golden.sh to refuse comparison

[MichaelFisher1997]

The automated visual test workflow failed at the golden-image comparison step. The screenshot capture itself works correctly, but the comparison never runs because the baseline image is invalid.

What failed

scripts/compare_visual_golden.sh exits 1 with:

Golden screenshot is effectively black (mean 0); refusing an invalid visual comparison

The check at the top of the script (require_non_black_image) aborts before any RMSE comparison:

# scripts/compare_visual_golden.sh:22-28
require_non_black_image() {
    local image=$1
    local label=$2
    local mean
    mean=$(magick "$image" -colorspace RGB -format '%[fx:mean]' info:)
    if awk -v value="$mean" 'BEGIN { exit !(value <= 0.0001) }'; then
        printf '%s is effectively black (mean %s); refusing an invalid visual comparison\n' "$label" "$mean" >&2
        exit 1
    fi
}

Verified locally with ImageMagick:

$ magick docs/visual-test/golden/menu.png -colorspace RGB -format 'mean=%[fx:mean] max=%[fx:maxima] min=%[fx:minima]\n' info:
mean=0 max=0 min=0

The committed golden docs/visual-test/golden/menu.png (332 bytes, 1920x1080, 1-bit grayscale) is a fully-black PNG.

Why this trips the workflow

.github/workflows/visual-test.yml:99-105 runs the comparison unconditionally whenever screenshot.png exists:

- name: Compare against golden image
  id: golden_diff
  if: steps.check_screenshot.outputs.screenshot_exists == 'true'
  run: |
    nix shell nixpkgs#imagemagick -c bash scripts/compare_visual_golden.sh screenshot.png docs/visual-test/golden/menu.png visual-diff.png
  env:
    VISUAL_DIFF_RMSE_TOLERANCE: "0.015"

The script's pre-check refuses to compare against any image whose mean pixel value is <= 0.0001 (effectively black). The committed golden hits that guard, so the job exits 1 and failure() becomes true, which triggers the diagnose step (Run opencode failure diagnosis, line 148).

A locally produced screenshot.png (rendered with the same Lavapipe + Weston + ZIGCRAFT_SAFE_RENDER=1 setup the workflow uses) renders the menu UI correctly and has mean=0.00889345 — well above the guard — so the failure is not on the capture side, only on the baseline.

Additional observation

build-output.log was not present in the workspace when the diagnose agent ran (only weston.log, an empty logs/ directory, and the existing zig-out/ from prior builds). That points at the Run menu screenshot capture step in .github/workflows/visual-test.yml:73-87 failing before the tee in .github/actions/run-with-log/action.yml:33 could create the log file, or the log having been cleaned up between steps. Either way, the actual terminal failure the workflow surfaces is the golden-image guard above, not anything inside the screenshot capture pipeline.

Suggested fix

Regenerate the baseline so it is not all-black:

1. Run the capture command locally (matches the workflow):

   mkdir -p /tmp/runtime-runner && chmod 700 /tmp/runtime-runner
   export XDG_RUNTIME_DIR=/tmp/runtime-runner WAYLAND_DISPLAY=headless \
          ZIGCRAFT_SAFE_RENDER=1 ZIGCRAFT_SMOKE_FRAMES=5 \
          VK_ICD_FILENAMES=$(nix build --no-link --print-out-paths nixpkgs#mesa.drivers)/share/vulkan/icd.d/lvp_icd.x86_64.json \
          VK_LAYER_PATH=$(nix build --no-link --print-out-paths nixpkgs#vulkan-validation-layers)/share/vulkan/explicit_layer.d \
          VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
          VK_LAYER_SETTINGS_PATH=$PWD/.github/vulkan/vk_layer_settings.txt
   devenv shell --profile graphics -- weston --socket=headless --backend=headless-backend.so --width=1280 --height=720 &
   devenv shell --profile graphics -- zig build run -Dscreenshot-path=docs/visual-test/golden/menu.png -Dskip-present=true

2. Confirm the regenerated PNG is non-black (magick ... -format '%[fx:mean]' info: returns a positive value).
3. Commit the new docs/visual-test/golden/menu.png.

If the regenerated golden is intentionally a blank output (e.g., ZIGCRAFT_SAFE_RENDER=1 + ZIGCRAFT_SKIP_WORLD_UPDATE already disables world passes and the menu preview can't seed before the capture timeout), then the test needs a different shape — either a UI-only render path that fills the frame, or skip the regression test under safe-render mode and add a non-safe-render variant for golden regeneration.

Reference

• Workflow run: https://github.com/OpenStaticFish/ZigCraft/actions/runs/30613255573
• Failure point: scripts/compare_visual_golden.sh:23-29 (require_non_black_image guard)
• Compare step: .github/workflows/visual-test.yml:99-105
• Black baseline: docs/visual-test/golden/menu.png (332 B, all pixels == 0)

[github-actions]

Analysis

Classification: Bug (matches the bug, hotfix, visual-test labels; no automated-audit label). The diagnosis in the body is correct and verified against the tree.

Validation

• scripts/compare_visual_golden.sh:24-33 require_non_black_image guard confirmed.
• .github/workflows/visual-test.yml:99-105 unconditional compare step confirmed.
• docs/visual-test/golden/menu.png is exactly as described: 332 B, 1920x1080, 1-bit grayscale, IDAT payload all zeros → fully black.

Important context the issue doesn't surface — docs/visual-test/README.md:5-9 explicitly says:

The tracked golden/menu.png is known to be black and is therefore intentionally not a valid baseline. ... Until a reviewed candidate is promoted, CI should fail loudly rather than treating an all-black image as a valid visual result.

So the workflow is behaving exactly as designed; the failure is a known/unresolved state, not a regression introduced by this run.

Related open items that gate the fix

• Issue [Visual Test] Menu screenshot RMSE 0.0437 exceeds 0.015 tolerance after g_pipeline inherits alphaToCoverageEnable #911 (OPEN) — g_pipeline inherits alphaToCoverageEnable = VK_TRUE despite g_pass.frag only declaring vec3 outNormal (VUID-08891), plus a rasterizationSamples mismatch on the water pipeline vs the MSAA HDR render pass (VUID-06853). Until that's fixed, any non-safe-render golden regenerated today will still exceed VISUAL_DIFF_RMSE_TOLERANCE=0.015 because Lavapipe's output is undefined at the same vkQueueSubmit step. [Visual Test] Menu screenshot RMSE 0.0437 exceeds 0.015 tolerance after g_pipeline inherits alphaToCoverageEnable #911 must land first.
• PR Black golden image fails the visual-test comparison. #963 ("Black golden image fails the visual-test comparison.") is already open but only touches devenv.lock (drops git-hooks). It does not regenerate menu.png, so merging it won't make the workflow green.

Scope / sub-task recommendation
The issue is small but has two distinct prerequisite tracks. I'd split it:

1. Unblock the rendered menu — land the [Visual Test] Menu screenshot RMSE 0.0437 exceeds 0.015 tolerance after g_pipeline inherits alphaToCoverageEnable #911 fixes (pipeline_specialized.zig:130-131 G-buffer multisampling reset, and water_system.zig:281 + rhi_init_deinit.zig:99 / rhi_frame_orchestration.zig:129 sample-count plumbing) so a Lavapipe capture is deterministic and matches what a real GPU produces.
2. Promote a real golden — capture in non-ZIGCRAFT_SAFE_RENDER mode (the author's caveat is correct: under ZIGCRAFT_SAFE_RENDER=1 world passes are off, so the menu can't seed before the capture timeout and the result is meaningless), validate mean > 0 via ImageMagick, get reviewer sign-off, then replace docs/visual-test/golden/menu.png and supersede Black golden image fails the visual-test comparison. #963.

Closing #962 with only the existing PR #963 (lockfile-only) would re-confirm the README's "fail loudly" intent but wouldn't move the workflow forward.

Information not blocking: none — the failure point, root cause, and remediation path are all clear from the issue body plus the README and the linked run. No additional logs/repro needed.

[github-actions]

Comment posted to issue #962: #962 (comment)

opencode session  |  github run