Add Rust cooperative foundation and complete Load/Store

.github/workflows/rust-host.yml

@@ -0,0 +1,43 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+name: Rust host checks
+
+on:
+  push:
+    branches:
+      - main
+      - "pull-request/[0-9]+"
+    paths:
+      - "rust/**"
+      - ".github/workflows/rust-host.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  host:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: rust
+        shell: bash --noprofile --norc -euo pipefail {0}
+    env:
+      RUSTDOCFLAGS: -D warnings
+    steps:
+      - uses: actions/checkout@v6
+        with:
+          persist-credentials: false
+      - name: Check the pinned Rust workspace
+        run: |
+          rustup toolchain install --no-self-update
+          rustup show active-toolchain
+          cargo fmt --all -- --check
+          cargo test --workspace --locked
+          cargo clippy --workspace --all-targets --locked -- -D warnings
+          cargo doc --workspace --no-deps --locked

docs/index.rst

@@ -7,6 +7,7 @@ CUDA Core Compute Libraries
 
    cccl/index
    python/index
+   rust/index
    contributors/index
    maintainers/index
    infrastructure/index
@@ -18,6 +19,8 @@ make CUDA C++ and Python more delightful.
 
 - :doc:`Python Libraries <python/index>`
 
+- :doc:`Rust Development <rust/index>`
+
 - :doc:`Contributor Docs <contributors/index>`
 
 - :doc:`Maintainer Docs <maintainers/index>`

docs/rust/groups.rst

@@ -0,0 +1,107 @@
+.. Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+.. SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+Hierarchy and group queries
+===========================
+
+``cuda-coop`` describes the current CUDA launch without requiring dimensions in
+the Rust source. ``this_thread()``, ``this_warp()``, ``this_block()``,
+``this_cluster()`` and ``this_grid()`` return typed group descriptors.
+``ThreadHierarchy::new()`` and ``ThreadHierarchy::current()`` provide the same
+physical groups through ``thread()``, ``warp()``, ``block()``, ``cluster()`` and
+``grid()``. ``Hierarchy`` aliases ``ThreadHierarchy``.
+
+Queries run in an Oxide-compiled kernel with the workspace's pinned
+``1.99.1`` toolchain. The device dependency is pinned in Cargo; there
+is no backend-selection feature. Plain Cargo commands need the workspace's
+unstable-feature configuration; see :doc:`index` for standalone crate setup.
+Host payload tests require no CUDA installation.
+The host provider remains independently usable on Rust 1.85.
+
+Rank and count
+--------------
+
+``group.rank()`` and ``group.count()`` default to the Thread level. Rust selects
+an explicit level with ``rank_at::<coop::Warp>()`` or
+``count_at::<coop::Warp>()``. An inner level queries constituents within the
+group; an outer level queries that group within its parent. Equal levels have
+rank zero and count one. Physical queries use x-fastest linearization within
+each level, with blocks ordered before their constituent threads or warps.
+
+For example, ``this_block().rank()`` is the current thread's linear rank in the
+block, while ``this_block().rank_at::<coop::Grid>()`` is the block's linear rank
+in the grid. The result is ``u64`` when either scope or query level is Grid,
+and ``u32`` otherwise. Grid arithmetic promotes coordinates before multiplying.
+``rank_as::<i16>()`` and ``count_as::<i16>()`` select one of the eight fixed-width
+integer types; ``rank_at_as::<i16, coop::Warp>()`` also selects a level.
+Conversions follow integer ``as`` semantics, so callers must choose a sufficient
+width to avoid truncation.
+
+A physical warp always counts 32 lanes. Block warp counts round up when the
+last warp is partially occupied. On known targets below sm_90, cluster queries
+describe one block per cluster.
+On sm_90 and newer they describe the launch's cluster; an ordinary, unclustered
+launch still has one block per cluster. An unknown target retains the sm_90
+minimum for these queries. No query allocates storage or synchronizes threads.
+Copying a descriptor preserves the
+scope, and subsequent queries describe the thread executing them.
+
+Mapped groups
+-------------
+
+``this_warp().group_by::<8>()`` partitions a physical warp into groups of eight
+lanes. ``this_block().group_by::<2>()`` partitions a block into groups of two
+**physical warps**, each containing 64 threads. The default is exhaustive:
+partition size must divide the parent's unit count. Both forms require complete
+physical warps in the enclosing block.
+
+``group_by_non_exhaustive::<N>()`` excludes leftover units. Its ``is_member()``
+query returns ``1u8`` for members and ``0u8`` for excluded threads. Counts remain
+defined for nonmembers; rank queries on a nonmember trap. Lane groups accept
+only Thread and Warp levels; warp groups accept Thread, Warp and Block levels.
+Mapped groups cannot be partitioned again.
+
+Sizes must be positive compile-time constants. Invalid static sizes fail during
+compilation. Checks that depend on the actual block dimensions trap on device
+when violated. These descriptors and checks provide no proof of collective
+participation, uniform control flow, or scratch ownership. The Load/Store API
+accepts power-of-two lane groups of widths 1, 2, 4, 8, 16 and 32 with Direct, Striped, Vectorize or
+Transpose and implicit storage. Other mapped groups remain query-only. Canonical Warp-in-Block
+and logical-group-in-Warp ranks preserve group uniformity in checked operations;
+ordinary lane arithmetic does not receive that guarantee.
+
+Run the native query checks
+---------------------------
+
+The standalone ``rust/cuda-coop/tests/device_queries`` package contains the
+complete native program. It verifies all physical scope/level pairs with
+multidimensional blocks and grids, a partial final warp, a clustered launch,
+exhaustive and non-exhaustive mappings, and integer conversions. Raw CUDA
+coordinates select output locations independently of the queries being checked.
+The same program checks payload initialization, safe access and copying at
+item counts 1 and 4. It uses the
+same pinned public Oxide dependencies as the library. Keeping this native
+package separate lets ordinary workspace tests run without CUDA host libraries.
+
+With the matching ``cargo-oxide`` compiler installed, a CUDA toolkit, and an
+sm_120 GPU, run from the CCCL checkout:
+
+.. code-block:: shell
+
+   cd rust/cuda-coop/tests/device_queries
+   CUDA_HOME=/usr/local/cuda-13.3 cargo oxide run --arch sm_120 --materialize-cubin
+   compute-sanitizer --tool memcheck --error-exitcode 1 target/release/device_queries
+
+The five rejection modes run in separate processes because a device trap makes
+its CUDA context unusable:
+
+.. code-block:: shell
+
+   for case in 0 1 2 3 4; do
+     target/release/device_queries "$case"
+   done
+
+They check partial physical warps, an indivisible exhaustive partition, a
+partition larger than its parent, and a rank query by an excluded lane. The
+last mode indexes an unwritten payload slot. The program reports success only
+when CUDA returns the expected device-trap error.

docs/rust/index.rst

@@ -0,0 +1,121 @@
+.. Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. ALL RIGHTS RESERVED.
+.. SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+``cuda.coop`` in Rust
+=====================
+
+Write group-first cooperative operations in an ordinary generic Rust kernel.
+This kernel copies partial block tiles through a reusable helper in another
+crate. The item count stays a const generic from the host launch through the
+kernel and helper:
+
+.. literalinclude:: ../../rust/cuda-coop/examples/load_store/src/main.rs
+   :language: rust
+   :lines: 12-35
+   :dedent: 4
+
+The helper uses the following calls for a partial tile. ``ThreadData`` infers its
+element type from ``input``. Load fills the payload in place; Store writes the
+valid prefix and leaves the rest of the destination unchanged:
+
+.. literalinclude:: ../../rust/cuda-coop/examples/load_store/helper/src/lib.rs
+   :language: rust
+   :lines: 55-73
+   :dedent: 8
+
+The complete :doc:`load_store` example includes the helper, imports, host
+allocation and launch. It runs Direct, Striped, Vectorize, Transpose,
+WarpTranspose and WarpTransposeTimesliced with one and four items per thread.
+These listings come directly from that executable source.
+
+Start with a kernel
+-------------------
+
+Use ``use cuda_coop as coop;``. A ``ThreadData::<_, ITEMS_PER_THREAD>`` holds
+each thread's values and tracks which slots have been initialized. The group
+argument selects the participating threads. ``DisjointSlice`` with
+``CollectiveTiles<ITEMS_PER_THREAD>`` supplies collective output ownership
+without giving every lane an overlapping mutable Rust slice.
+
+For a block tile, calculate its size as ``block.count() * ITEMS_PER_THREAD``,
+its element offset from ``block.rank_at::<coop::Grid>()``, and its valid count
+as the remaining input length capped at the tile size. The kernel above shows
+that idiom, including a short final tile. Counts and offsets describe the whole
+group's tile, and must be uniform across that group.
+
+The current public operations are Load and Store, with the supported groups,
+algorithms and numeric types described in :doc:`load_store`. Additional
+primitive families are under development. The crate is unpublished and its
+interface can change.
+
+Build and run
+-------------
+
+Use the Oxide revision selected by the Cargo manifests and the pinned Rust
+1.99.1 toolchain. The example's initial native configuration is Linux, CUDA 13.3 and an
+NVIDIA GPU. From a complete CCCL checkout with the matching Oxide installed:
+
+.. code-block:: shell
+
+   cd rust/cuda-coop/examples/load_store
+   CUDA_HOME=/usr/local/cuda-13.3 CUDA_PATH=/usr/local/cuda-13.3 \
+     cargo oxide run --arch sm_120 --materialize-cubin
+
+Run plain Cargo commands from within ``rust/`` so they use its configuration.
+For a standalone crate, ``cargo oxide`` enables the unstable Rust features
+required by ``cuda-coop`` and ``cuda-device``. Plain ``cargo build``, ``test``,
+``clippy`` and ``doc`` also need this setting in that crate's
+``.cargo/config.toml``:
+
+.. code-block:: toml
+
+   [env]
+   RUSTC_BOOTSTRAP = "1"
+
+Choose the architecture for your GPU. The kernel's launch contract declares
+its exact block shape. Preparing a launch checks that shape and uses the
+compiler's inferred shared-memory requirement. The consumer needs no C++ shim,
+provider build script, manual linker inputs or scratch-byte calculation.
+
+How compilation works
+---------------------
+
+``cargo oxide`` discovers the concrete Load/Store uses, including those inside
+generic helpers from another crate. The dependency's host provider specializes
+CUB for their types, item counts, algorithms and launch geometry. NVRTC returns
+device LTO-IR and the actual C++ temporary-storage layouts. Oxide checks
+participation and resource use, plans shared storage and reuse synchronization,
+and links the device code into the final CUBIN.
+
+The same build supplies launch geometry and resource facts to the typed host
+launcher. Borrow checking remains useful, but collective participation and
+scratch completion also require the compiler's checks. See :doc:`provider` for
+the request format, native ABI and provider development workflow.
+
+Generic Rust API
+----------------
+
+The public API and downstream helpers use Rust types, traits and const
+generics. Element types are inferred from typed producers, and ordinary calls
+infer the algorithm from their options. Supported element types and algorithms
+are deliberately constrained by the API's traits.
+
+Each concrete native specialization gets a generated device C ABI wrapper at
+build time. This is an implementation boundary, not a finite set of functions
+that users select or declare. The user's source stays generic Rust.
+
+Read next
+---------
+
+.. toctree::
+   :maxdepth: 1
+
+   payload
+   groups
+   load_store
+   provider
+
+Generate the API reference with ``cargo doc --workspace --no-deps`` from
+``rust/``; ``cuda_coop`` contains the public item documentation. The Rust
+workspace is separate from CCCL's CMake build, and existing C++ users do not
+need a Rust toolchain.