xen: let dom0 grow into its boot-time extra memory

[alexandermerritt]

Part of https://linear.app/edera/issue/ENG-99

Motivation

At boot, Linux on Xen reserves address ranges beyond a domain's memory that have no memory behind them yet (the "extra memory" regions; on a PVH dom0 they cover the host RAM that dom0 was not given). Today all of them go to the pool used for grant and foreign mappings, so dom0's balloon starts empty. Growing dom0 at runtime then falls back to memory hotplug, which adds memory at the first free physical address. On a PVH dom0 that is the host's real address map, where the range may belong to a device.

We want dom0 to grow once at startup, for nodes that need more dom0 memory than the default, safely and quickly. This PR provides the mechanism, not yet connecting it to how customers can request this at instance startup, at reboot, or when running the installer.

Changes

• Keep dom0's growth room on the balloon. Before the mapping pool takes the extra memory, dom0 keeps the part it may grow into: the gap between Xen's ceiling for dom0 (dom0_mem=...,max:) and its current size. The mapping pool keeps the rest, always at least one 128 MiB section. Every region gives up the same fraction, so growth is spread across NUMA nodes. Nothing changes for domUs or for a dom0 booted without a max:.
• Park that room as 2 MiB blocks. The balloon's 2 MiB grow path only works on blocks, which until now were only created by shrinking a domain first. Parking the boot-time room as blocks lets dom0's first growth ask Xen for 2 MiB at a time.

dom0's size does not change at boot; it grows only when its balloon target is raised.

Results

64 GiB host, PVH dom0, dom0_mem=1G+35%,max:1G+50% (dom0 22.9 GiB, ceiling 32.2 GiB). A systemd unit raised dom0's target once, about 12 s after boot, to use all of the room (9.37 GiB):

┌────────────────────────┬─────────────┬────────────┐
│ xen.balloon_superpages │    Time     │    Rate    │
├────────────────────────┼─────────────┼────────────┤
│ Y (default, 2 boots)   │ 0.44-0.45 s │ ~21 GiB/s  │
├────────────────────────┼─────────────┼────────────┤
│ N (4 KiB pages)        │ 4.53 s      │ 2.07 GiB/s │
└────────────────────────┴─────────────┴────────────┘

• About 10x faster with 2 MiB blocks, matching our zone measurements of the same balloon path.
• The balloon kept exactly the expected share, 99.99% of it as blocks.
• dom0 grew by exactly the requested amount and stopped at Xen's ceiling, with no stalls and no memory hotplug.

Not yet tested: PV dom0 and multi-node NUMA hosts.

[alexandermerritt]

We added a similar patch for zones' ballooning, but need a separate one for dom0. "block" is a contiguous run of 512 KiB pages.

The 2 MiB zone balloon series speeds up regrowth: when a zone shrinks, whole free 2 MiB runs are kept as blocks, and growing back asks Xen for those blocks as 2 MiB pieces.

Dom0 never shrinks before it grows. Its growth goes into the boot-time room, which arrives in the balloon as single pages, so it would always take the 4 KiB path. This patch parks that room as 2 MiB blocks from the start.

[alexandermerritt]

I am working on a second commit, both need to get in together to work. Still running some tests now before review needed.

[alexandermerritt]

Attached are a test script that claude wrote to demonstrate the performance of these changes at early boot of a physical machine, and a systemd unit file that runs at early startup (reboot your machine) to record the performance.

If you append xen.balloon_superpages=0 to the kernel command line, then it will fall back to 4KiB pages and re-run after reboot. This is how the above numbers were captured.

dom0growtest.py

unit file dom0growtest.service

[Unit]
Description=ENG-99 test: grow dom0 into its balloon room once at boot
Documentation=file:///home/alexm/edera/dom0growtest.py
# Only on kernels with a balloon driver; the script itself refuses an empty room.
ConditionPathExists=/sys/devices/system/xen_memory/xen_memory0
# Where the protect grow step will sit: after the customer's setting can
# exist (cloud-init) and xl works (xenstored), before anything sizes itself
# from dom0's memory.
After=local-fs.target proc-xen.mount xenstored.service cloud-init.service
Before=protect-daemon.service protect-meminfo.service containerd.service kubelet.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/python3 -I /home/alexm/edera/dom0growtest.py grow --label boot

[Install]
WantedBy=multi-user.target

[github-actions]

Backport to edera/6.18-lts: proposed in #34

Checks and builds passed.

Run log

[alexandermerritt]

register the OOM deflate notifier only in domUs

this was backported with #31