ci: bump build-rocm-wheels to torch 2.13, fix stale-line version resolution - #1237
Open
marcusr-amd wants to merge 1 commit into
Open
ci: bump build-rocm-wheels to torch 2.13, fix stale-line version resolution#1237marcusr-amd wants to merge 1 commit into
marcusr-amd wants to merge 1 commit into
Conversation
…otgun
Bumps requirements/build/rocm.txt and the wheel-build resolve step from
torch 2.12.0/torchvision 0.27.0 to 2.13.0/0.28.0.
Also fixes a latent bug in the resolve step uncovered while doing this:
`uv pip compile constraints.in` (with a bare `torch==2.12.*`/`2.13.*`
spec) was silently resolving to the frozen ROCm 7.15 nightly line
(`+rocm7.15.0a20260728`, unchanged since 2026-07-28) instead of the
live ROCm 10.1 line (`+rocm10.1.0aYYYYMMDD`, daily builds) that every
other consumer of this index (e.g. rocm-scripts' constraints.in) is
on. Root cause: PEP 440 compares local-version segments as opaque
strings once a segment contains a letter, so "rocm7" sorts ABOVE
"rocm10" (digit '7' > '1' at the first differing position) -- uv's
"pick the highest version" resolution is fooled into preferring the
stale line forever, for as long as both lines coexist on the index.
Confirmed directly:
>>> from packaging.version import Version
>>> Version("2.12.0+rocm7.15.0a20260728") > Version("2.12.0+rocm10.1.0a20260821")
True
This explains why the latest published prototype wheel
(vllm-0.26.1rc1.dev1574+g50604a385, built 2026-08-21 by this exact
workflow) resolved torch==2.12.0+rocm7.15.0a20260728 -- a month-old
ROCm nightly -- despite PYTORCH_INDEX_URL already pointing at the live
whl-multi-arch index. rocm[devel,libraries] then cascaded to the
matching 7.15.0a20260728 build via torch's own pinned metadata
dependency, so the whole build silently ran against five-week-old
ROCm.
Fix: resolve the newest build on the live `+rocm10.` local-version
line directly by scraping the index for the newest torch-2.13.x
filename matching that prefix, instead of trusting uv's version
comparison across both lines. torchvision/torchaudio derivation is
otherwise unchanged (still inherits the resolved +rocm local suffix).
See AIInfo context/rocm-nightly-install.md ("version-maximization
footgun") for the general form of this issue across the ROCm nightly
ecosystem.
NOTE: the currently-published vllm wheel (built against torch 2.12)
hard-pins `Requires-Dist: torch<2.13.0,>=2.12.0` in its own metadata,
so installing torch 2.13 alongside it is a hard resolver conflict
today. This PR must merge and produce a new wheel build before
rocm-scripts' regression-vllm job can be pointed at torch 2.13.
Signed-off-by: Marcus Rosen <marcus.rosen@amd.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bumps
requirements/build/rocm.txtand the wheel-build resolve step from torch 2.12.0/torchvision 0.27.0 to 2.13.0/0.28.0.Rationale
This is in response to delays in fixing https://amd-hub.atlassian.net/browse/ROCM-29466 issues on torch 2.12. This enables us to move forward and test recent rocm nightly builds again.
Bug found along the way
While making this change I found that the resolve step's
uv pip compile constraints.in(with a baretorch==2.12.*/2.13.*spec) was silently resolving to the frozen ROCm 7.15 nightly line (+rocm7.15.0a20260728, unchanged since 2026-07-28) instead of the live ROCm 10.1 line (+rocm10.1.0aYYYYMMDD, daily builds) that every other consumer of this index — e.g. rocm-scripts'constraints.in— is on.Root cause: PEP 440 compares local-version segments as opaque strings once a segment contains a letter, so
"rocm7"sorts above"rocm10"(the digit'7'beats'1'at the first differing position).uv's "pick the highest version" resolution gets fooled into preferring the stale line for as long as both lines coexist on the index. Confirmed directly:Impact: this explains why the latest published prototype wheel (
vllm-0.26.1rc1.dev1574+g50604a385, built 2026-08-21 by this exact workflow) resolvedtorch==2.12.0+rocm7.15.0a20260728— a month-old ROCm nightly — despitePYTORCH_INDEX_URLalready pointing at the livewhl-multi-archindex.rocm[devel,libraries]then cascaded to the matching7.15.0a20260728build via torch's own pinned metadata dependency, so the whole build silently ran against five-week-old ROCm.Fix: resolve the newest build on the live
+rocm10.local-version line directly, by scraping the index for the newesttorch-2.13.xfilename matching that prefix, instead of trustinguv's version comparison across both lines. torchvision/torchaudio derivation is otherwise unchanged (still inherits the resolved+rocmlocal suffix).See AIInfo
context/rocm-nightly-install.mdfor the general form of this issue across the ROCm nightly ecosystem.Note
The currently-published vllm wheel (built against torch 2.12) hard-pins
Requires-Dist: torch<2.13.0,>=2.12.0in its own metadata, so installing torch 2.13 alongside it is a hard resolver conflict today. This PR must merge and produce a new wheel build before rocm-scripts'regression-vllmjob can be pointed at torch 2.13.