Conversation
Signed-off-by: David Bourdeau <42303109+dbourdea@users.noreply.github.com>
There was a problem hiding this comment.
🟢 Approval recommended
The change is scoped, test-covered, and integrates cleanly with the existing GPU telemetry freshness contract, with only a minor doc-comment mismatch noted in review comments.
Pull request overview
Adds Linux AMD GPU utilization telemetry to the nvpair-node-info service by reading the amdgpu gpu_busy_percent sysfs counter and joining samples to detected adapters via PCI address, so existing PAIR telemetry/scheduling can consume utilization on AMD hosts without adding a new engine adapter.
Changes:
- Add AMD PCI-based
statsKeygeneration and sysfs-based utilization sampling (gpu_busy_percent) on Linux. - Combine AMD utilization sampling with existing NVIDIA
nvidia-smisampling and include AMD adapters on mixed-vendor hosts. - Add Linux sysfs-fixture unit tests, update
nvpair-node-infodocumentation, and bumpnvpair-node-infoversion to0.14.0.
File summaries
| File | Description |
|---|---|
| services/versions.json | Bumps nvpair-node-info component version to 0.14.0. |
| services/nvpair-node-info/stats_linux.go | Extends Linux sampling pass to include AMD utilization and adjusts NVIDIA “unavailable” log behavior. |
| services/nvpair-node-info/README.md | Documents AMD sysfs utilization behavior and freshness/zero-omission implications. |
| services/nvpair-node-info/gpu_linux.go | Adds AMD PCI-based join key generation and includes AMD adapters on mixed-vendor hosts. |
| services/nvpair-node-info/amd_linux.go | Implements PCI normalization/vendor check and reads gpu_busy_percent for utilization. |
| services/nvpair-node-info/amd_linux_test.go | Adds fixture-based tests for AMD keying and utilization parsing/availability behavior. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| name = card.DeviceInfo.Product.Name | ||
| } | ||
| gpus = append(gpus, GPUInfo{Name: name}) | ||
| gpus = append(gpus, GPUInfo{Name: name, statsKey: amdStatsKey(amdPCIRoot, card.Address)}) |
|
Thanks for the PR - can we merge the AMD GPU code into gpu_linux.go to be consistent with out |
Signed-off-by: David Bourdeau <42303109+dbourdea@users.noreply.github.com>
|
Implemented the requested source layout update in 5279297.
Validation on an isolated Ubuntu Linux checkout with Go 1.26.0: cd services/nvpair-node-info
GOMAXPROCS=4 go test -race ./...Passed. Thank you. This should now match the repository's Linux GPU source layout for the next release. |
Description
Linux AMD adapters can already execute GPU-backed inference through PAIR's existing Ollama integration, but node-info does not provide their live GPU utilization. This change reads the amdgpu
gpu_busy_percentsysfs counter and matches it to detected AMD adapters by PCI address, so utilization is available to PAIR's existing telemetry and scheduling path.An isolated two-node experiment on Ryzen AI Max+ 395 / Radeon 8060S systems verified Ollama 0.33.3 native ROCm gfx1151 execution, all 29 layers of Qwen3 0.6B offloaded, discovery, PIN pairing, model inventory and concurrent independent requests on distinct nodes. Inference worked with configuration only; this patch adds utilization reporting, not a new engine adapter.
Detailed findings, six reproducible figures, sanitized measurements, failure cases and setup notes: public experiment repository and white paper DOI.
Scope
Validation
On Ubuntu 26.04.1, Go 1.26.0, AMD gfx1151:
All 13 service executables built. Ordinary
go test ./...passed in shared, engine-manager, ui-broker (including relay/workloadstore), ollama-proxy, lmstudio-proxy, scheduler, scanner and cluster-manager components. Saved logs are in the linked evidence repository. Windows amd64 and macOS arm64 node-info cross-compilation with CGO disabled passed; these are build checks only.Manual two-host coverage included streaming and non-streaming Ollama/OpenAI APIs, inventories, adoption and managed start, cancellation, real discovery/pairing, distinct concurrent destinations, new-request routing after engine loss, seven successful requests during whole-peer loss, and recovery without re-pairing. Final rebuilt node-info returned live AMD utilization with memory omitted and exited cleanly.
Repository checks:
node scripts/spdx-headers.mjspassed (873 checked, zero missing/review/unclassified),git diff --checkpassed, and the exported patch applies to upstream commit13b68115fa2c9c1d94f1ead1358f8d5a527cfecf.The complete cross-process
services/testssuite and Electron checks were not run. No runtime validation on real NVIDIA, Windows AMD, Apple or mixed-GPU hardware is claimed.Risk
The patch follows the existing global GPU freshness and omitted-zero conventions; per-adapter freshness and UMA-aware memory reporting remain future schema work. Mixed-vendor behavior is covered by preserving existing map entries in fixture tests, not real mixed hardware.
Separately, the experiment observed that abruptly killing an upstream engine after first content ended the stream without a completion marker while its workload remained
runningin a later snapshot. Normal client cancellation correctly becamefailed. This pre-existing proxy terminal-reporting path is documented in the paper and is outside this focused telemetry change; new requests still routed to a surviving peer. PAIR did not replay or migrate a partial answer.Timing is deliberately bounded: five warmed calls per route, same local engine, 128 generated tokens each. The difference between median wall time outside the engine was about 0.53 ms. This is not a large-model performance or production reliability claim.
Checklist
Codex assisted with source inspection, implementation, test execution and documentation. The accompanying report preserves evidence and explicitly limits conclusions to the measured configuration.