fix(layout): end a leaf at its start once the visibility retry budget is spent

[tag-und-nacht]

Summary

Entering the minibuffer with vertico (M-x) while the selected window was small (a freshly started frame, or one tiled to half the screen) froze the GUI at 100% CPU: no key, C-g or Esc was accepted and the process had to be killed. -Q did not reproduce; the trigger is a completion UI growing the minibuffer so the selected window (here the dashboard) shrinks under point, plus a display-row-collapsing property between the visible rows and point and scroll-conservatively > 0.

A sample of the hung process showed the evaluator thread inside layout_frame_rust_for_purpose_inner, re-laying the same leaf forever with nothing logged (the leaf loop bypasses the 12-retry FrameLayoutCoordinator). Two defects combined:

1. Unbounded leaf retry. BufferSourceRetryPlan::viewport_resolution turned a NeedMoreMeasurement decision with no visibility retries left into a PlaceRelativeToPoint request; the engine resolved it to the start it already had, zeroed the budget "to stop the chain", and issued a Retry anyway. GNU bounds this search: try_scrolling gives up past scroll_max lines (src/xdisp.c:19445-19446, emacs-31.0.90) and, after the recenter: start has been laid out, redisplay_window re-recenters at most twice before done: (xdisp.c:21360-21398). ViewportDecision::resolve_with_budget now owns that rule, keyed on the typed ViewportAttemptStart the rows were produced from: a spent budget ends the leaf at a semantic start; a measurement probe, which is never publishable, still falls back to the policy placement, committed with no retries left. The old table also finished on a probe start when a placement request arrived with no budget; that path now places instead.

2. Placement scanned the wrong buffer. Context::redisplay_start_before_point_by_display_rows walked the window's buffer while the minibuffer was current; the invisible-newline check in current_screen_line_start_with_truncation probes Lisp positions in the current buffer, so every probe past the minibuffer's end signalled args-out-of-range, the placement returned None, and the fallback was the unchanged start that fed defect 1. GNU reaches recenter: inside redisplay_window after set_buffer_internal_1 (XBUFFER (w->contents)) (xdisp.c:20532-20535); the helper now selects the window's buffer for the scan and restores the caller's (the with_frame_display_context idiom).

Declared divergence: GNU answers SCROLLING_FAILED with recenter: (xdisp.c:21097-21108) and always ends with a start placed around point. When core display motion cannot improve on the start already laid out, the leaf now ends there and point may be off-window for that redisplay instead of looping. The same terminal rule now applies to the passes whose budget is forced to zero on purpose (scroll replay and one-row transient windows in window_render.rs, which previously contradicted their own comments by placing).

Tests (red first)

• xdisp::tests::redisplay_start_before_point_scans_the_window_buffer_not_the_current_one: None before, Some(CharPos0(4500)) after; also pins that the caller's buffer is restored.
• engine::tests::point_placement_with_a_foreign_current_buffer_terminates_and_shows_point: reproduces the report (foreign current buffer, hidden span, scroll-conservatively 30); hung before both fixes.
• engine::tests::spent_visibility_budget_ends_the_leaf_at_its_start: pins defect 1 alone with a thirty-row display string that the producer lays out as rows while screen-line motion counts it as one line, so every placement clamps back to the window start. Runs the layout on a helper thread; under the old decision table it fails on its 120 s timeout, with the fix it finishes in 0.5 s.
• viewport_resolution::tests::spent_budget_* and budget_left_*: the decision table per attempt start and budget.

Verification

• cargo fmt --check clean; cargo clippy -p neomacs-layout-engine --tests and -p neovm-core with no errors.
• cargo test -p neomacs-layout-engine --lib and the core xdisp::/indent::/window:: tests with --no-fail-fast, baselined against main on the same machine: every test that failed only on this branch passes in isolation; the remaining failures are the run-to-run flaky set main shows here too (font/glyph/display_row tests).
• GUI (macOS, the reporter's config with vertico, dashboard, Rectangle left-half): after cargo xtask --release, M-x opens both in the fresh frame and after tiling, a heartbeat timer keeps firing, and C-g returns to the dashboard. No failed to converge in the log.

Not run: Linux, TTY.

🤖 Generated with Claude Code

https://claude.ai/code/session_01B3wJN81Y4miZnPBNY9o1Xv

[tag-und-nacht]

I have borrowed my son's computer (x86_64 architecture) and installed latest Ubuntu LTS in a partition. Will test on that now.

(Only 32 GB of RAM, so still not enough computing power)

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[eval-exec]

Reviewed the diff and the corresponding GNU Emacs xdisp.c buffer-selection and recenter paths. Selecting the window buffer during display motion addresses the incorrect property lookup, and distinguishing semantic starts from measurement probes makes the exhausted-budget handling appropriate for preventing this retry loop. I support this bounded freeze fix with the declared point-visibility limitation. A follow-up should reconcile screen-line motion with the actual rows produced by multiline display replacements so fallback placement can reliably bring point into view. Review scope: source inspection; I have not independently run the regression tests.

[eval-exec]

Independently reproduced the underlying layout hang on Linux using this PR’s regression fixtures against local main (5038ae19c), after studying GNU Emacs’s window-buffer selection and redisplay recovery paths in src/xdisp.c.

Configuration	Result
Regression tests only, without production fixes	Core motion test returned None instead of Some(CharPos0(4500)). Both layout regression tests kept running at approximately one CPU core each; I stopped them after 35 seconds.
Window-buffer selection fix only	Both foreign-current-buffer regressions passed. The separate exhausted-budget regression still exceeded a 5-second timeout.
Both fixes applied	All 11 selected layout tests passed, including both layout regressions and the viewport-resolution unit tests.

A debugger sample during the hang showed the layout path repeatedly reaching redisplay_start_before_point_by_display_rows / scan_screen_line_motion_target. The isolated runs confirm two independent problems: querying display properties in the wrong current buffer, and retrying without progress after exhausting the visibility budget.

Scope: this reproduces the engine-level failure on Linux, not the complete interactive Vertico GUI scenario.

For the longer-term follow-up, display motion and rendering should share display-row traversal semantics, especially for multiline display replacement strings. Regression coverage should require both bounded completion and point visibility; the documented possibility of finishing with point off-screen remains a separate issue.

Thanks for the focused regression fixtures!