Add free GPU memory limits to runtime profiles

[Baiju Meswani (baijumeswani)]

Why

Runtime profiles can already use total GPU memory and whether the GPU is integrated. Total memory does not tell us how much memory is currently free. A profile sized for a device's capacity may not be a good choice when other work is using that memory.

What changed

• Profiles can now set optional minimum_free_device_memory_bytes and maximum_free_device_memory_bytes. Both bounds are inclusive. They work alongside the existing total-memory bounds and is_integrated condition.
• The runtime uses the free-memory value it already reads from CUDA when it creates a Model. Profiles without free-memory bounds keep their previous behavior; if no profile matches, the base configuration is used.
• The C++ loader and Python builder reject invalid ranges and profiles that could match the same device. They account for the fact that free memory cannot exceed total memory.
• The host/CUDA add-on interface version moves to 9 because the configuration layout changed. The binaries must be built from matching sources.

Checks

• 33 focused C++ configuration tests passed.
• 211 Python builder-config tests passed.
• The changed C++ files passed the local clang-format check.

Free GPU memory is a snapshot, not a reservation. On systems where the CPU and GPU share memory, it is not a measure of free OS memory. Profile authors should still choose conservative settings and check host-memory headroom.

[Copilot]

Copilot review overview

🟢 Approval recommended

The C++ and Python implementations agree, handle physical memory constraints correctly, and have focused coverage.

Review effort: Balanced
Findings: None

What changed in this PR

Adds free GPU memory constraints to runtime-profile selection while preserving legacy behavior.

Changes:

• Adds inclusive free-memory bounds and physical-range validation.
• Applies CUDA-reported free memory during profile selection.
• Updates ABI version, documentation, and focused tests.

File	Description
src/​config.h	Adds free-memory eligibility fields and device facts.
src/​config.cpp	Parses, validates, and matches free-memory bounds.
src/​models/​runtime_profiles.cpp	Passes CUDA free memory into profile selection.
src/​smartptrs.h	Bumps the add-on interface version.
src/​python/​py/​models/​builder_config.py	Mirrors C++ validation and overlap rules.
test/​cpp/​engine/​config_tests.cpp	Tests matching, validation, and physical constraints.
test/​cpp/​engine/​engine_test_doubles.h	Supports configurable free memory in tests.
test/​python/​builder/​test_builder_config.py	Tests Python validation and overlap detection.
docs/​model_package.md	Documents runtime semantics and constraints.
docs/​ModelBuilderConfiguration.md	Documents builder validation behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Tianlei Wu (tianleiwu)]

Overall this looks correct and well tested. The inclusive bounds, the "free <= total" feasibility check in both the C++ EligibilitiesIntersect and the Python overlap check, and the C++/Python validation parity all check out. I traced the boundary cases in the new tests (e.g. free 80/81 and total max 150 vs. free min 160) and they behave as the tests expect. Free memory comes from cudaMemGetInfo on the same device used for total memory, and kDeviceInterfaceVersion is the only place the add-on handshake is checked, so the bump to 9 is sufficient.

No blocking issues. Two small non-blocking suggestions are inline:

• Selection is now environment-dependent, but nothing reports which facts were used.
• A short note on the std::nullopt free-memory semantics.

Minor test-coverage idea: add one case combining is_integrated with free-memory bounds. The current overlap tests cover free bounds and the integrated flag separately, and the combined AND-of-three-conditions path in EligibilitiesIntersect / MatchesEligibility is not exercised together.