Skip to content

cli & replayer: Add --device-pci-vendor and --device-pci-device options - #313

Draft
louzt wants to merge 1 commit into
ValveSoftware:masterfrom
louzt:feat/cli-device-pci-vendor
Draft

cli & replayer: Add --device-pci-vendor and --device-pci-device options#313
louzt wants to merge 1 commit into
ValveSoftware:masterfrom
louzt:feat/cli-device-pci-vendor

Conversation

@louzt

@louzt louzt commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Draft PR. Open for review; not requesting merge yet.

What this PR does

Adds --device-pci-vendor <vendorID_hex> and --device-pci-device <deviceID_hex>
to fossilize-replay, with a matching ExternalReplayer::Options pair so the
multi-process path can forward them. Selection walks the existing
vkEnumeratePhysicalDevices enumeration, logs vendorID and deviceID for
each candidate, picks the first GPU whose vendor matches and whose device
matches (if --device-pci-device is set), and falls back to --device-index
and finally gpus.front().

On no-match, emits LOGW("No GPU matched --device-pci-vendor 0x%x%s; ...") so
callers learn the filter dropped out rather than silently landing on the wrong
GPU.

Windows cmdline uses snprintf + 0x%x so the values round-trip as hex
through the CLI parser; Linux argv uses sprintf to match the existing
index_name pattern above it.

Origin and relationship to PR #305

PR #305 bundled two unrelated concerns: prctl(PR_SET_PDEATHSIG) to kill
orphan replay workers, and PCI-based GPU selection. The prctl part landed
on master as 3efdda8 (squashed and stripped down by HansKristian-Work).
Per the same review comment, device filtering was invited as a separate PR.

This is that PR. No code from #305 is carried over other than the design idea.

What does NOT conflict with subsequent work

  • a6a44aa (Start adding a validation tool for GPU key invariance) —
    different concern. That tool tests cache key invariance across Vulkan
    implementations; this PR changes runtime device selection. No overlap.
  • Robustness2 banlist (96ae9b6, 5b23748, etc.) — orthogonal extension
    gating, not GPU selection.
  • Bucket JSON system (f0270fe, 39004ff, 5f95aa9) — replay-time DB
    format; this PR is pre-init-device.
  • Sifter CLI / roundtrip-checker (2583ae3,, c774839`) — new tools, no
    interaction with device enumeration.

Where the responsibility actually lives

Selection of dGPU vs iGPU for shader compilation is typically the caller's
job:

  • DXVK has its own DXVK_FILTER_DEVICE_NAME env var for matching by
    device name (see doitsujin/dxvk).
  • vkd3d-proton has analogous device-name filtering at the D3D12 layer.
  • Steam Linux Runtime / pressure-vessel exposes PV_FORCE_USE_HWD-style
    env vars that influence which Vulkan device the wrapper selects.
  • Mesa CI could route through the new fossilize-validate-cache-roundtrip
    tool (a6a44aa) with this flag to pin the cache test to a specific ICD.

fossilize-replay is the right place for the flag because it's the lowest
layer that does enumeration; consumers that already decide the device
externally can ignore it, and consumers that don't can use it directly.
This PR provides the primitive, not the policy.

Out of scope

  • --device-pci-domain and --device-pci-bus for full VK_EXT_pci_bus_info
    matching — would help systems with multiple GPUs of the same vendorID +
    deviceID. Happy to add if asked; not in this PR to keep it atomic.
  • A pci_filter_matches() predicate helper on VulkanDevice — inlined here
    to avoid scope creep. Easy to extract later if more callers need it.
  • Tests. Held back per review feedback on cli & replayer: Prevent Linux worker process orphaning with race-free PR_SET_PDEATHSIG #305 ("somewhat overzealous"); the
    synthetic test catalogue and 7-case matrix from the local branch are
    available if you want them landed separately.

Validation

Tested on a 2-GPU hybrid laptop (AMD Renoir iGPU + NVIDIA RTX 3050 Mobile
dGPU), Vulkan 1.4.341, Mesa RADV RENOIR + NVIDIA proprietary 550.163.01,
against a real Steam pipeline cache (/tmp/steam_pipeline_cache.snapshot.foz,
83 MB, produced by VK_LAYER_VALVE_steam_fossilize_32 — i.e. the actual
producer of foz files that this tool replays).

The host enumerates 3 physical-device candidates:

Enumerated GPU #0:
  name: AMD Radeon Graphics (RADV RENOIR)
  apiVersion: 1.4.354
  vendorID: 0x1002
  deviceID: 0x1636
Enumerated GPU #1:
  name: NVIDIA GeForce RTX 3050 Laptop GPU
  apiVersion: 1.3.277
  vendorID: 0x10de
  deviceID: 0x25a2
Enumerated GPU #2:
  vendorID: 0x10005    # Mesa venus / virtual device
  deviceID: 0x0

Without filter the loader picks gpus.front() (the AMD iGPU). For
shader-compilation workloads the dGPU is the correct target — shader
caches keyed on the iGPU are unusable on the dGPU and vice versa.

With --device-pci-vendor 0x10de the loader picks the NVIDIA dGPU:

Chose GPU:
  name: NVIDIA GeForce RTX 3050 Laptop GPU
  apiVersion: 1.3.277
  vendorID: 0x10de
  deviceID: 0x25a2

With --device-pci-vendor 0x1002 it picks the AMD iGPU (matching
the vendorID but ignoring deviceID since not specified).

With --device-pci-vendor 0x10de --device-pci-device 0x25a2 it picks
the specific NVIDIA dGPU.

With --device-pci-vendor 0xDEAD (no match) it logs the warning and
falls back to default:

Fossilize WARN: No GPU matched --device-pci-vendor 0xdead; falling back to default selection.

Hex round-trip verified: strtoul("0x10de", NULL, 0) produces 0x10de,
which matches the vendorID byte-for-byte in the LOGI stream. The Windows
multi-process path uses snprintf(..., "0x%x", ...) so the same value
survives the cmdline round-trip.

Multi-process path (--num-threads > 1) confirmed to forward both flags
through argv on the same host: the child slave-process invocation
receives --device-pci-vendor 0x10de verbatim.

Concrete consumer path (Steam Linux Runtime)

The foz file used above was produced by Steam's pipeline-cache layer
(VK_LAYER_VALVE_steam_fossilize_32). On Steam Linux Runtime 3.0
(pressure-vessel) the launch script sets PV_FORCE_USE_HWD and the
SteamLinuxRuntime_* env vars before invoking the game. For shader
compilation targets this PR lets the replayer's caller pin the dGPU
unambiguously by vendor, regardless of how the loader orders the ICDs
on a given hybrid host — which is otherwise up to the Mesa and NVIDIA
ICD init order and can change across driver updates.

Mesa CI use case

Mesa CI recently added fossilize-validate-cache-roundtrip (a6a44aa)
which validates cache-key invariance across Vulkan implementations. With
this flag Mesa CI can pin each roundtrip run to a specific vendor
(NVIDIA vs RADV vs Intel vs Lavapipe) without depending on the
enumeration order of the runner image.

Known limitation (out of scope here)

Hybrid hosts with two GPUs sharing vendorID + deviceID (rare but
real — e.g. multi-GPU workstations with twin RTX cards) need
--device-pci-domain / --device-pci-bus via VK_EXT_pci_bus_info to
disambiguate. Happy to add as a follow-up PR if requested; not included
here to keep scope atomic.

Diff

 cli/device.cpp                          | 28 ++++++++++++++++++++++++++-
 cli/device.hpp                          |  2 ++
 cli/fossilize_replay.cpp                |  8 ++++++++
 fossilize_external_replayer.hpp         |  4 ++++
 fossilize_external_replayer_linux.hpp   | 15 +++++++++++++
 fossilize_external_replayer_windows.hpp | 13 ++++++++++++
 6 files changed, 69 insertions(+), 1 deletion(-)

Pins shader compilation to a specific GPU on hybrid systems by matching
PCI vendorID / deviceID during vkEnumeratePhysicalDevices. Falls back to
--device-index, then gpus.front() when no filter matches.

The prctl(PR_SET_PDEATHSIG) part of PR ValveSoftware#305 already landed as 3efdda8;
this is the device-filter follow-up invited in the same review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant