Repository navigation
fix(shared): enforce typed shared-memory alignment - #1345
Merged
Merged
Conversation
Signed-off-by: Yunus Mutlu <mutluyunus708@gmail.com>
YunusMutlu
force-pushed
the
fix/shared-memory-alignment
branch
from
October 13, 2026 11:41
db617fa to
9966a04
Compare
Compute the required storage alignment of a shared array and of the dynamic pool in mir-lower, from mir_type_abi_align (the Rust ABI raise) and llvm_type_size_align (the LLVM natural layout), and error only when both are unknown. The dialect-side helper could not classify the llvm.array leaf the importer emits for opaque descriptors, so a SharedArray or DynamicSharedArray of them stopped compiling; they compile again at align 8. The error now carries the use site. Drop the power-of-two re-checks from lowering: the two op verifiers own that rule and run before conversion. Restore the private alignment helper in dialect-mir cast.rs so the dialect keeps no public alignment oracle. Add llvm.array lowering tests for the static element and the dynamic pointee, and a unit test for the combined helper. Signed-off-by: nihalpasham <nihalp@nvidia.com>
The two verifier tests now check the diagnostic text, including the offending value, for 3 and for 0 on both ops, and the module sits below the MirExternSharedOp impl it also tests. The MirExternSharedOp attribute table names extern_byte_offset, the attribute the op declares. Signed-off-by: nihalpasham <nihalp@nvidia.com>
Two kernels touch an under-requested shared array before a smaller pad, so the array lands behind the pad in the PTX: SharedArray<u64, 2, 1> and a repr(C, align(16)) element at natural alignment. Before the typed raise both launches failed with CUDA error 716, misaligned address. The host checks the values and reads sharedmem.ptx for the .align 8 and .align 16 declarations. Signed-off-by: nihalpasham <nihalp@nvidia.com>
State on SharedArray, DynamicSharedArray and the book page that ALIGN is 0 or a non-zero power of two (non-zero for the dynamic pool), that storage is never declared below align_of::<T>(), that the dynamic pool takes its strongest typed use combined with the launch contract, and that natural-alignment aggregates now declare their natural .align. Signed-off-by: nihalpasham <nihalp@nvidia.com>
A one-page figure for the book and the pull request, in the style of the tile-libraries direct-linkage figure. Signed-off-by: nihalpasham <nihalp@nvidia.com>
Collaborator
|
/ok to test bbea3a2 |
Collaborator
|
Thanks, this is the right route: verifier arms for the request and a typed raise where the storage is declared. Four things on top:
At |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Static and dynamic shared memory now carry a checked, typed alignment contract. The MIR verifier rejects an
ALIGNthat is not a non-zero power of two, with the source position, and lowering declares every static__shared_mem_Nglobal and the per-function__dynamic_smempool at the larger of the requested alignment and the element type's ABI alignment. Before, the declared alignment came only from the request, or from an exporter fallback that printed 8 for everything it did not know; ptxas then packed under-declared arrays behind smaller statics and the typed stores faulted.What we fixed
MirSharedAllocOpandMirExternSharedOpverify that an explicit alignment is a non-zero power of two; the error names the use site instead of surfacing as an llc failure.max(ALIGN, abi_align(T)), computed in mir-lower byrequired_storage_alignfrom the Rust ABI raise (mir_type_abi_align) and the LLVM natural layout (llvm_type_size_align), sorepr(align(N))aggregates and LLVM leaf elements such asTmaDescriptorare both covered.DynamicSharedArrayuses, then combined with the launch-contract alignment as before.SharedArray<bool, N>compiles: the exporter fallback used to emitalign 0fori1arrays and llc rejected it.Alignment contract
SharedArray<T, N, ALIGN>:ALIGNis 0 (natural) or a non-zero power of two. Storage is never declared belowalign_of::<T>(); a largerALIGNis honored.DynamicSharedArray<T, ALIGN>:ALIGNis a non-zero power of two. The kernel's pool is declared at the strongest typed requirement across itsget,get_rawandoffsetuses and the launch contract. The alignment of anoffset(bytes)result remains the caller's responsibility.f16now declare their natural.align(4 for[f32; N], 2 forf16) where the exporter used to print 8.Implementation
ops/memory.rs: verifier arms on both ops, tests asserting the diagnostic text; the attribute table namesextern_byte_offsetandextern_alignment.convert/types/layout.rs:required_storage_align(ctx, mir_ty, llvm_ty)ismaxof the two layout helpers andNoneonly when both are unknown.convert_shared_alloc_dcand the dynamic pre-scan inlowering.rsuse it and report an unknown element with the op's location. The power-of-two re-checks in lowering are gone; the verifiers own the rule.cast.rshelper it was, becausedialect-mircannot see LLVM leaf types.i641 to 8 (static and dynamic), an align-32 tuple 1 to 32, the maximum over two pointees, and anllvm.array<16 x i64>element and pointee to 8.sharedmem: two kernels that fault on main plus a PTX declaration check.Verification
Checked at
bbea3a2c6(nightly-2026-10-11, RTX 5090, sm_120a):cargo test -p dialect-mir(70) andcargo test -p mir-lower(327 + 147) pass; clippy and rustdoc with warnings as errors on dialect-mir, mir-lower and cuda-device,cargo oxide fmt --check, the 18 guards and the lock check are clean.sharedmemsource built with main's backend fails Test 3 withDriverError(716, "misaligned address")and declares.align 1and.align 8; this branch passes all five tests with.align 8and.align 16, memcheck and synccheck clean.SharedArray<TmaDescriptor, 1>andDynamicSharedArray::<TmaDescriptor, 128>compile atalign 8, as on main; they did not compile on the submitted head.ALIGN = 3(static and dynamic) andDynamicSharedArray::<T, 0>fail in the verifier with the source position; on main they reach llc asalign 3andalign 0.shared_slice_unsize, whose[f32; 128]tile moves from.align 8to the correct.align 4and still passes.Caveats
ALIGNbelowalign_of::<T>()is raised silently, as nvcc's__align__and Rust'srepr(align)do; it is not an error.TmaDescriptorisrepr(C, align(64))but the importer translates it to a plain[16 x u64], so its shared storage is declaredalign 8, as on main. That translation gap is separate from this change.align 8, and LLVM then stops formingld/st.shared.v2on them. No in-tree example is affected.Closes #1344.