cli & replayer: Add --device-pci-vendor and --device-pci-device options - #313
Draft
louzt wants to merge 1 commit into
Draft
cli & replayer: Add --device-pci-vendor and --device-pci-device options#313louzt wants to merge 1 commit into
louzt wants to merge 1 commit into
Conversation
Pins shader compilation to a specific GPU on hybrid systems by matching PCI vendorID / deviceID during vkEnumeratePhysicalDevices. Falls back to --device-index, then gpus.front() when no filter matches. The prctl(PR_SET_PDEATHSIG) part of PR ValveSoftware#305 already landed as 3efdda8; this is the device-filter follow-up invited in the same review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft PR. Open for review; not requesting merge yet.
What this PR does
Adds
--device-pci-vendor <vendorID_hex>and--device-pci-device <deviceID_hex>to
fossilize-replay, with a matchingExternalReplayer::Optionspair so themulti-process path can forward them. Selection walks the existing
vkEnumeratePhysicalDevicesenumeration, logsvendorIDanddeviceIDforeach candidate, picks the first GPU whose vendor matches and whose device
matches (if
--device-pci-deviceis set), and falls back to--device-indexand finally
gpus.front().On no-match, emits
LOGW("No GPU matched --device-pci-vendor 0x%x%s; ...")socallers learn the filter dropped out rather than silently landing on the wrong
GPU.
Windows
cmdlineusessnprintf+0x%xso the values round-trip as hexthrough the CLI parser; Linux argv uses
sprintfto match the existingindex_namepattern above it.Origin and relationship to PR #305
PR #305 bundled two unrelated concerns:
prctl(PR_SET_PDEATHSIG)to killorphan replay workers, and PCI-based GPU selection. The prctl part landed
on master as
3efdda8(squashed and stripped down by HansKristian-Work).Per the same review comment, device filtering was invited as a separate PR.
This is that PR. No code from #305 is carried over other than the design idea.
What does NOT conflict with subsequent work
a6a44aa(Start adding a validation tool for GPU key invariance) —different concern. That tool tests cache key invariance across Vulkan
implementations; this PR changes runtime device selection. No overlap.
96ae9b6,5b23748, etc.) — orthogonal extensiongating, not GPU selection.
f0270fe,39004ff,5f95aa9) — replay-time DBformat; this PR is pre-init-device.
2583ae3,,c774839`) — new tools, nointeraction with device enumeration.
Where the responsibility actually lives
Selection of dGPU vs iGPU for shader compilation is typically the caller's
job:
DXVK_FILTER_DEVICE_NAMEenv var for matching bydevice name (see
doitsujin/dxvk).PV_FORCE_USE_HWD-styleenv vars that influence which Vulkan device the wrapper selects.
fossilize-validate-cache-roundtriptool (
a6a44aa) with this flag to pin the cache test to a specific ICD.fossilize-replayis the right place for the flag because it's the lowestlayer that does enumeration; consumers that already decide the device
externally can ignore it, and consumers that don't can use it directly.
This PR provides the primitive, not the policy.
Out of scope
--device-pci-domainand--device-pci-busfor fullVK_EXT_pci_bus_infomatching — would help systems with multiple GPUs of the same vendorID +
deviceID. Happy to add if asked; not in this PR to keep it atomic.
pci_filter_matches()predicate helper onVulkanDevice— inlined hereto avoid scope creep. Easy to extract later if more callers need it.
synthetic test catalogue and 7-case matrix from the local branch are
available if you want them landed separately.
Validation
Tested on a 2-GPU hybrid laptop (AMD Renoir iGPU + NVIDIA RTX 3050 Mobile
dGPU), Vulkan 1.4.341, Mesa RADV RENOIR + NVIDIA proprietary 550.163.01,
against a real Steam pipeline cache (
/tmp/steam_pipeline_cache.snapshot.foz,83 MB, produced by
VK_LAYER_VALVE_steam_fossilize_32— i.e. the actualproducer of foz files that this tool replays).
The host enumerates 3 physical-device candidates:
Without filter the loader picks
gpus.front()(the AMD iGPU). Forshader-compilation workloads the dGPU is the correct target — shader
caches keyed on the iGPU are unusable on the dGPU and vice versa.
With
--device-pci-vendor 0x10dethe loader picks the NVIDIA dGPU:With
--device-pci-vendor 0x1002it picks the AMD iGPU (matchingthe vendorID but ignoring deviceID since not specified).
With
--device-pci-vendor 0x10de --device-pci-device 0x25a2it picksthe specific NVIDIA dGPU.
With
--device-pci-vendor 0xDEAD(no match) it logs the warning andfalls back to default:
Hex round-trip verified:
strtoul("0x10de", NULL, 0)produces0x10de,which matches the
vendorIDbyte-for-byte in the LOGI stream. The Windowsmulti-process path uses
snprintf(..., "0x%x", ...)so the same valuesurvives the cmdline round-trip.
Multi-process path (
--num-threads > 1) confirmed to forward both flagsthrough
argvon the same host: the childslave-processinvocationreceives
--device-pci-vendor 0x10deverbatim.Concrete consumer path (Steam Linux Runtime)
The foz file used above was produced by Steam's pipeline-cache layer
(
VK_LAYER_VALVE_steam_fossilize_32). On Steam Linux Runtime 3.0(
pressure-vessel) the launch script setsPV_FORCE_USE_HWDand theSteamLinuxRuntime_*env vars before invoking the game. For shadercompilation targets this PR lets the replayer's caller pin the dGPU
unambiguously by vendor, regardless of how the loader orders the ICDs
on a given hybrid host — which is otherwise up to the Mesa and NVIDIA
ICD init order and can change across driver updates.
Mesa CI use case
Mesa CI recently added
fossilize-validate-cache-roundtrip(a6a44aa)which validates cache-key invariance across Vulkan implementations. With
this flag Mesa CI can pin each roundtrip run to a specific vendor
(NVIDIA vs RADV vs Intel vs Lavapipe) without depending on the
enumeration order of the runner image.
Known limitation (out of scope here)
Hybrid hosts with two GPUs sharing
vendorID+deviceID(rare butreal — e.g. multi-GPU workstations with twin RTX cards) need
--device-pci-domain/--device-pci-busviaVK_EXT_pci_bus_infotodisambiguate. Happy to add as a follow-up PR if requested; not included
here to keep scope atomic.
Diff