[bjf-frz]

Main processes an entire SiLU-and-Mul input scalarly when any input or output address is misaligned. This PR adds a vector path for sufficiently large misaligned contiguous inputs while retaining the original arithmetic and a small-input scalar path. On B300, BF16 [8611, 5632] with gate/up offsets (1, 3) measured 115.832 µs on main and 64.196 µs on this PR; details and scope are below.

Required human contributor answers

The author's public answers in #160, “Required human contributor answers” remain the intent for this follow-up; this PR improves that merged operator rather than introducing a new framework or changing its arithmetic contract.

1. What outcome are you trying to achieve?

Reuse the author's answer in #160, question 1: develop native CUDA operators with a complete, reusable build, integration and validation path. This follow-up improves the first operator's supported misaligned-input path.

2. What is the highest-leverage variable for that outcome?

Reuse #160, question 2: activation memory movement, kernel launches and efficient parallel computation while preserving the numerical contract. Here the changed variable is device latency for independently shifted contiguous inputs.

3. How will this work move that variable rather than create busywork?

Reuse #160, question 3: vectorized fused computation, preserved rounding, numerical checks and matched performance measurements. The concrete extension is safe aligned reads for misaligned inputs, with a measured scalar fallback where reconstruction costs more.

4. Who is the customer or user, and what do they actually need?

Reuse #160, question 4: the author developing native CUDA operators needs a complete implementation/build/validation path that can support subsequent operators.

PR necessity and bugfix evidence

The author explicitly requested a PR for this follow-up. It requires native load/dispatch changes; an existing configuration cannot enable vector processing of these inputs. This is a measured performance optimization, not a claim that the previous scalar implementation returned incorrect results.

Purpose

This PR improves the existing SiLU-and-Mul CUDA operator for contiguous inputs whose starting addresses are not 16-byte aligned. Main currently falls back to scalar processing for the entire input if gate, up or output is misaligned. The PR adds a vector path for sufficiently large misaligned inputs while retaining the scalar path below a size cutoff. Aligned inputs continue to use vector loads and stores. The operator keeps its existing output and arithmetic contract.

Inputs and relationship to vLLM

Ouro has separate gate_proj and up_proj linear layers. For example, Ouro-1.4B transforms hidden with shape [N, 2048] into two separate contiguous tensors gate and up, each with shape [N, 5632]. They need not be adjacent in memory. silu_and_mul(gate, up) directly consumes those two tensors and returns an output of the same shape:

out[i] = SiLU(gate[i]) * up[i]

The original vLLM operator used in the comparison accepts a single packed tensor. Adapting Ouro's separate tensors requires cat((gate, up), dim=-1), producing [N, 11264], with each row laid out as [gate_row | up_row]. If an upstream projection already produces that packed layout, the original vLLM kernel can consume it directly and does not need this concatenation. The performance tables below therefore report both vLLM modes.

Dispatch and launch strategy

The CUDA entry point validates the same shape, device and dtype, requires contiguous inputs, allocates a fresh output, and uses the current CUDA stream. Empty inputs return without a launch. Supported dtypes are BF16, FP16 and FP32; the native operator is inference-only.

launch<T> selects one of three paths:

Condition	Path	Processing
gate, up and output are all 16-byte aligned	Aligned	Direct 16-byte loads for gate/up, common vector computation and 16-byte stores; an incomplete tail is scalar.
Any address is misaligned and numel < threshold	Scalar	Each element is loaded, computed and stored as a scalar.
Any address is misaligned and numel >= threshold	Misaligned	Scalar output prefix, reconstructed 16-byte input vectors, common vector computation and 16-byte output stores, followed by a scalar tail.

The number of elements is the total across the contiguous input, rather than a separate dispatch for each row. The cutoff uses the input GPU's actual SM count:

elements_per_vector = 16 / sizeof(dtype)  # BF16/FP16: 8, FP32: 4
threshold = SM_count * 160 reference blocks/SM * 256 threads/block * elements_per_vector

For the 148-SM B300 used below, the cutoff is 48,496,640 BF16/FP16 elements or 24,248,320 FP32 elements. The 160 is a reference workload for the heuristic, not a number of blocks physically assigned to each SM. The real launch uses 256 threads per block and computes its grid separately, capped at 4096 blocks, or 16384 for sufficiently large FP32 work. This cutoff follows the requested B300-based heuristic; it has not been tuned or validated as optimal for other GPUs.

Misaligned vector processing

silu_and_mul_kernel handles a scalar output prefix when required to make the first vector store 16-byte aligned. A 16-byte-aligned output has prefix length zero. The middle handles one 16-byte logical vector per thread iteration; the remaining incomplete output tail is scalar, in the same kernel. These regions partition the work; there is no device-wide barrier between them.

For each middle vector, load_vector handles gate and up independently:

1. If the requested address is 16-byte aligned, load one uint4 directly.
2. Otherwise load the two adjacent aligned 16-byte chunks and select the requested values in registers, using word moves and, when a 2-byte shift remains, __funnelshift_r.
3. If either aligned chunk would cross the input view's start or end, load only its valid fragment scalarly. No read crosses the view boundary.

For example, with BF16 storage A and B both 16-byte aligned, suppose gate = A[2:] and up = B[6:], and the output is aligned. For logical elements 8–15, the vector body processes:

gate[8:16] = A[10:18]: load aligned A[8:16] and A[16:24], then select 10:18
up[8:16]   = B[14:22]: load aligned B[8:16] and B[16:24], then select 14:22
out[8:16]              : compute the 8 results and store one aligned 16-byte vector

Here indices are BF16 elements and the example assumes the displayed aligned chunks lie inside the input views. Gate and up can have different offsets; each is reconstructed separately. The selection is register work, not a separate global-memory copy or kernel launch. A 16-byte input chunk may overlap the chunk read for an adjacent logical vector.

Calculation and rounding

The vector body is shared by aligned and misaligned modes. A 16-byte vector contains 8 BF16/FP16 or 4 FP32 elements. For each element, SiLU is evaluated in FP32, rounded to the input dtype, and then multiplied by up, preserving the existing staged arithmetic. BF16/FP16 multiply the rounded activations and up in pairs using __hmul2. FP32 still uses two scalar FP32 multiplications per float2; its memory loads and stores remain 16-byte vectors. Scalar prefix, tail and fallback use the corresponding scalar calculation.

Measured examples

BF16 on one B300, CUDA Graph median latency in µs. The first two rows come from the November 4 model-width sweep; the last comes from the November 3 cutoff sweep. Offsets are gate/up storage offsets in elements.

shape	offsets	PR path	PR	Main	vLLM packed	vLLM + cat
[128, 5632]	(0, 0)	aligned vector	1.986	1.986	2.114	3.968
[128, 5632]	(1, 3)	scalar	3.735	3.733	2.050	5.313
[8611, 5632]	(1, 3)	misaligned vector	64.196	115.832	51.240	218.988

For the common Ouro examples, the PR and main have nearly the same latency; the smaller misaligned case remains scalar. In the 126-case November 4 sweep, the PR was faster than vLLM + cat in every measured case, but vLLM's already-packed input can be faster, as shown above. These are separate-input operator timings, not end-to-end model-throughput results. The complete test method and both 126-case matrices follow.

Test Plan

vllm-rlt Version / Commit: base 989921ac1d3897c5c43e27ec8cddbc91fa953194, tested PR head 21c706e646ae350ab20844ec24bd6dc63ed6e27b.

• Built the head for SM80 and SM103; ran on B300 SXM6 AC, 148 SMs. Python 3.12.3, torch 2.13.0+cu130, CUDA toolkit 13.4 and driver 610.43.02.
• On the final head, pre-commit run --all-files passed Ruff lint, Ruff formatting and CPU regression tests.
• On the final head, 41 selected GPU tests passed (152 deselected).
• In the benchmark below, all 126 finite random-input cases produce elementwise-equal values across all three implementations (torch.equal; signed zero is not distinguished). This is not an exhaustive numerical proof.
• Before the access-mode rename, the same GPU test suite passed 241 tests at fe71d62; Compute Sanitizer memcheck passed 24 cases with 0 errors. The subsequent source difference to the current head is the enum name and documentation only; current-head targeted tests are reported separately above.

Test Result: performance

Measured on one idle B300, with no other compute process detected on that device during the run. The other GPUs were in use. These are operator-level measurements, not model throughput. Each point reports the median of 5 shuffled rounds of unprofiled CUDA Graph timing, 32 calls/graph, 30 graph replays per round (8 at the threshold); each function has 10 warmups. Clocks were not locked. Inputs use fixed seed 42, gate sampled from normal(std=3), up from normal(std=1). Gate/up are separate contiguous allocations; offsets in the table are in elements. The GPU results are limited to this hardware and workload.

The three code versions are:

1. PR: 21c706e, the restored output-prefix implementation.
2. Main: upstream main 989921a, compiled from the same SiLU source in a separate operator namespace; misaligned inputs use scalar access.
3. vLLM: local vLLM 2b678ac739166bd0356883048673bbd1c34a3b0e, compiled directly from its original csrc/activation_kernels.cu. Its interface consumes one packed [gate | up] tensor, so it is shown in two timing modes:
   • vLLM packed: packing and output allocation are done before the timed graph. This measures the prepacked vLLM kernel and its packed input starts aligned; the original independent gate/up offsets no longer apply to this input.
   • vLLM + cat: torch.cat((gate, up), dim=-1), output allocation, and the vLLM kernel are inside the timed graph. CUDA Graph timings measure device work; host-side allocation overhead is not included. This column reflects adapting separate inputs to the vLLM interface.

Representative latency, in microseconds (lower is better):

Shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
1 × 5632	(1,3)	scalar	1.282	1.282	1.986	3.585
16 × 5632	(1,3)	scalar	1.474	1.474	1.986	3.970
64 × 5632	(1,3)	scalar	2.434	2.434	2.050	4.547
128 × 5632	(1,3)	scalar	3.732	3.729	2.114	5.314
256 × 5632	(1,3)	scalar	5.417	5.442	3.106	7.886
1024 × 5632	(1,3)	scalar	10.715	10.703	7.053	20.879
128 × 5632	(0,0)	aligned vec	1.988	1.986	2.074	3.970
128 × 5633	(1,3)	scalar	3.715	3.734	3.522	7.362
1024 × 5633	(1,3)	scalar	10.719	10.720	14.470	29.473

All rows above are BF16. For the 120 common-shape measurements (BF16/FP16/FP32, 1–1024 rows, widths 5632/5633, aligned and independently shifted inputs), the PR/main latency ratio has a median of approximately 1.00; the misaligned inputs are below the cutoff and use the scalar path. For the 60 aligned cases, the observed main/PR speedup ranges from 0.960× to 1.015×. Small differences at these microsecond times are workload-specific; no blanket aligned-path speedup is claimed.

The cutoff is easier to see with fixed width 5632 and neighboring row counts. In each dtype the first row is just below the cutoff and the second is just above it:

dtype / shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
BF16 8610 × 5632	(1,3)	scalar	116.207	116.201	51.196	219.484
BF16 8611 × 5632	(1,3)	misaligned vec	64.196	115.832	51.240	218.988
FP16 8610 × 5632	(1,3)	scalar	117.404	117.420	49.540	218.096
FP16 8611 × 5632	(1,3)	misaligned vec	63.284	118.460	49.452	217.633
FP32 4305 × 5632	(1,3)	scalar	67.632	67.640	43.206	141.276
FP32 4306 × 5632	(1,3)	misaligned vec	53.576	67.704	43.256	141.172

Immediately above the cutoff, the PR is 1.80× faster for BF16, 1.87× for FP16, and 1.26× for FP32 than main on these shapes. Below it, the PR and main remain on the scalar path and have nearly identical latencies. For the above-cutoff examples, the PR's five-round ranges were 64.171–64.220 µs (BF16), 63.244–63.292 µs (FP16), and 53.564–53.592 µs (FP32); main's corresponding ranges were 115.800–115.840, 118.436–118.484, and 67.688–67.720 µs. These are repetitions of one benchmark run, not independent multi-run confidence intervals.

The vLLM packed column can be faster: at BF16 [8611, 5632] it takes 51.240 µs versus the PR's 64.196 µs, while adapting separate inputs with cat takes 218.988 µs. The packed layout is different, so that comparison does not isolate the cost of independent misalignment. For width 5633 the local vLLM dispatch falls back to scalar because the hidden dimension does not divide its vector width; the RLT operator handles the separately allocated tensors as flat contiguous arrays. Short ops-only and full profiles confirm the scalar and vector branches and show a separate cat kernel for the final column; profiled timings are not used as latency evidence.

All 126 measured cases (CUDA Graph median, µs)

BF16

Shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
1 × 5632	(0,0)	aligned vec	1.570	1.545	1.986	3.202
2 × 5632	(0,0)	aligned vec	1.540	1.542	1.986	3.266
4 × 5632	(0,0)	aligned vec	1.602	1.602	1.986	3.266
8 × 5632	(0,0)	aligned vec	1.602	1.540	1.986	3.330
16 × 5632	(0,0)	aligned vec	1.579	1.602	1.986	3.394
32 × 5632	(0,0)	aligned vec	1.602	1.602	1.988	3.397
64 × 5632	(0,0)	aligned vec	1.732	1.730	2.050	3.526
128 × 5632	(0,0)	aligned vec	1.988	1.986	2.074	3.970
256 × 5632	(0,0)	aligned vec	2.562	2.561	3.085	5.500
1024 × 5632	(0,0)	aligned vec	6.186	6.170	7.046	13.120
1 × 5632	(1,3)	scalar	1.282	1.282	1.986	3.585
2 × 5632	(1,3)	scalar	1.284	1.282	1.986	3.650
4 × 5632	(1,3)	scalar	1.346	1.346	1.986	3.650
8 × 5632	(1,3)	scalar	1.355	1.353	1.986	3.713
16 × 5632	(1,3)	scalar	1.474	1.474	1.986	3.970
32 × 5632	(1,3)	scalar	1.794	1.794	1.988	4.067
64 × 5632	(1,3)	scalar	2.434	2.434	2.050	4.547
128 × 5632	(1,3)	scalar	3.732	3.729	2.114	5.314
256 × 5632	(1,3)	scalar	5.417	5.442	3.106	7.886
1024 × 5632	(1,3)	scalar	10.715	10.703	7.053	20.879
1 × 5633	(0,0)	aligned vec	1.922	1.922	3.395	4.643
2 × 5633	(0,0)	aligned vec	1.922	1.922	3.422	5.121
4 × 5633	(0,0)	aligned vec	1.922	1.922	3.423	4.997
8 × 5633	(0,0)	aligned vec	1.564	1.576	3.439	5.114
16 × 5633	(0,0)	aligned vec	1.602	1.602	3.458	5.375
32 × 5633	(0,0)	aligned vec	1.602	1.602	3.458	5.509
64 × 5633	(0,0)	aligned vec	1.794	1.781	3.458	6.019
128 × 5633	(0,0)	aligned vec	1.986	1.986	3.484	6.812
256 × 5633	(0,0)	aligned vec	2.524	2.522	5.121	9.663
1024 × 5633	(0,0)	aligned vec	6.170	6.173	14.473	27.249
1 × 5633	(1,3)	scalar	1.282	1.282	3.351	5.105
2 × 5633	(1,3)	scalar	1.282	1.284	3.455	5.778
4 × 5633	(1,3)	scalar	1.346	1.346	3.418	5.763
8 × 5633	(1,3)	scalar	1.353	1.353	3.458	5.815
16 × 5633	(1,3)	scalar	1.474	1.474	3.427	6.031
32 × 5633	(1,3)	scalar	1.803	1.796	3.458	6.168
64 × 5633	(1,3)	scalar	2.434	2.445	3.520	6.609
128 × 5633	(1,3)	scalar	3.715	3.734	3.522	7.362
256 × 5633	(1,3)	scalar	5.441	5.440	5.121	10.314
1024 × 5633	(1,3)	scalar	10.719	10.720	14.470	29.473
8610 × 5632	(1,3)	scalar	116.207	116.201	51.196	219.484
8611 × 5632	(1,3)	misaligned vec	64.196	115.832	51.240	218.988

FP16

Shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
1 × 5632	(0,0)	aligned vec	1.540	1.547	1.922	3.200
2 × 5632	(0,0)	aligned vec	1.602	1.598	1.986	3.265
4 × 5632	(0,0)	aligned vec	1.602	1.577	1.986	3.266
8 × 5632	(0,0)	aligned vec	1.600	1.542	1.986	3.330
16 × 5632	(0,0)	aligned vec	1.602	1.602	1.986	3.330
32 × 5632	(0,0)	aligned vec	1.602	1.602	1.986	3.395
64 × 5632	(0,0)	aligned vec	1.734	1.730	2.007	3.522
128 × 5632	(0,0)	aligned vec	1.986	1.986	2.050	3.970
256 × 5632	(0,0)	aligned vec	2.529	2.562	3.012	5.421
1024 × 5632	(0,0)	aligned vec	6.086	6.087	6.784	12.820
1 × 5632	(1,3)	scalar	1.308	1.303	1.926	3.586
2 × 5632	(1,3)	scalar	1.284	1.284	1.986	3.648
4 × 5632	(1,3)	scalar	1.299	1.282	1.986	3.650
8 × 5632	(1,3)	scalar	1.359	1.359	1.986	3.707
16 × 5632	(1,3)	scalar	1.474	1.474	1.986	3.934
32 × 5632	(1,3)	scalar	1.794	1.794	1.986	4.063
64 × 5632	(1,3)	scalar	2.434	2.434	2.010	4.545
128 × 5632	(1,3)	scalar	3.718	3.740	2.050	5.313
256 × 5632	(1,3)	scalar	5.440	5.441	3.015	7.799
1024 × 5632	(1,3)	scalar	10.694	10.699	6.787	20.589
1 × 5633	(0,0)	aligned vec	1.922	1.922	3.266	4.547
2 × 5633	(0,0)	aligned vec	1.922	1.922	3.364	5.006
4 × 5633	(0,0)	aligned vec	1.922	1.922	3.330	4.931
8 × 5633	(0,0)	aligned vec	1.602	1.602	3.394	4.998
16 × 5633	(0,0)	aligned vec	1.602	1.602	3.394	5.270
32 × 5633	(0,0)	aligned vec	1.602	1.602	3.394	5.445
64 × 5633	(0,0)	aligned vec	1.730	1.739	3.443	5.958
128 × 5633	(0,0)	aligned vec	1.986	1.986	3.436	6.735
256 × 5633	(0,0)	aligned vec	2.501	2.500	5.055	9.587
1024 × 5633	(0,0)	aligned vec	6.091	6.095	14.267	27.010
1 × 5633	(1,3)	scalar	1.282	1.282	3.330	4.991
2 × 5633	(1,3)	scalar	1.282	1.284	3.340	5.712
4 × 5633	(1,3)	scalar	1.346	1.320	3.394	5.697
8 × 5633	(1,3)	scalar	1.410	1.410	3.377	5.761
16 × 5633	(1,3)	scalar	1.474	1.474	3.394	5.955
32 × 5633	(1,3)	scalar	1.794	1.794	3.394	6.019
64 × 5633	(1,3)	scalar	2.435	2.434	3.411	6.535
128 × 5633	(1,3)	scalar	3.765	3.776	3.458	7.290
256 × 5633	(1,3)	scalar	5.441	5.441	5.057	10.209
1024 × 5633	(1,3)	scalar	10.691	10.690	14.244	29.209
8610 × 5632	(1,3)	scalar	117.404	117.420	49.540	218.096
8611 × 5632	(1,3)	misaligned vec	63.284	118.460	49.452	217.633

FP32

Shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
1 × 5632	(0,0)	aligned vec	1.346	1.346	2.182	3.458
2 × 5632	(0,0)	aligned vec	1.346	1.346	2.242	3.459
4 × 5632	(0,0)	aligned vec	1.346	1.346	2.242	3.650
8 × 5632	(0,0)	aligned vec	1.346	1.346	2.242	3.650
16 × 5632	(0,0)	aligned vec	1.361	1.380	2.243	3.754
32 × 5632	(0,0)	aligned vec	1.476	1.476	2.244	3.906
64 × 5632	(0,0)	aligned vec	1.602	1.619	2.306	4.226
128 × 5632	(0,0)	aligned vec	1.986	1.986	2.307	4.800
256 × 5632	(0,0)	aligned vec	2.690	2.690	2.882	6.617
1024 × 5632	(0,0)	aligned vec	6.856	6.732	7.808	22.994
1 × 5632	(1,3)	scalar	1.282	1.282	2.242	3.718
2 × 5632	(1,3)	scalar	1.282	1.282	2.242	4.034
4 × 5632	(1,3)	scalar	1.301	1.284	2.242	4.034
8 × 5632	(1,3)	scalar	1.376	1.357	2.241	4.034
16 × 5632	(1,3)	scalar	1.474	1.474	2.242	4.293
32 × 5632	(1,3)	scalar	1.794	1.794	2.242	4.417
64 × 5632	(1,3)	scalar	2.437	2.443	2.259	4.878
128 × 5632	(1,3)	scalar	3.714	3.714	2.306	5.635
256 × 5632	(1,3)	scalar	5.379	5.379	2.883	7.489
1024 × 5632	(1,3)	scalar	18.208	18.058	7.820	33.839
1 × 5633	(0,0)	aligned vec	1.668	1.668	3.217	4.673
2 × 5633	(0,0)	aligned vec	1.666	1.666	3.266	5.105
4 × 5633	(0,0)	aligned vec	1.346	1.346	3.266	4.928
8 × 5633	(0,0)	aligned vec	1.346	1.359	3.266	5.164
16 × 5633	(0,0)	aligned vec	1.404	1.348	3.268	5.288
32 × 5633	(0,0)	aligned vec	1.474	1.474	3.330	5.684
64 × 5633	(0,0)	aligned vec	1.607	1.604	3.302	6.019
128 × 5633	(0,0)	aligned vec	1.986	1.986	3.394	6.688
256 × 5633	(0,0)	aligned vec	2.690	2.691	4.822	9.538
1024 × 5633	(0,0)	aligned vec	6.785	6.720	13.509	31.203
1 × 5633	(1,3)	scalar	1.282	1.282	3.212	5.060
2 × 5633	(1,3)	scalar	1.282	1.282	3.253	5.552
4 × 5633	(1,3)	scalar	1.282	1.301	3.266	5.521
8 × 5633	(1,3)	scalar	1.404	1.357	3.266	5.571
16 × 5633	(1,3)	scalar	1.476	1.474	3.266	5.824
32 × 5633	(1,3)	scalar	1.794	1.794	3.267	5.953
64 × 5633	(1,3)	scalar	2.455	2.444	3.330	6.429
128 × 5633	(1,3)	scalar	3.714	3.714	3.393	7.169
256 × 5633	(1,3)	scalar	5.382	5.379	4.820	10.117
1024 × 5633	(1,3)	scalar	18.093	18.179	13.505	40.361
4305 × 5632	(1,3)	scalar	67.632	67.640	43.206	141.276
4306 × 5632	(1,3)	misaligned vec	53.576	67.704	43.256	141.172

The benchmark reuses the repository's benchmarks/kernels/silu_and_mul.py timing and profiling helpers. The standalone comparison script, all five timing samples including eager timings, native build metadata, profiles, and GPU process-monitor log are retained in the local ignored artifacts/silu-unified-20261103/ directory; they are not committed. The displayed table includes all measured shapes.

Additional model-width measurements (2026-11-04)

The operator was rerun at FFN intermediate widths read from the local model configurations: Ouro-1.4B 5632, Nanbeige4.2-3B 10752, and Qwen3.8-27B text 17408. These are synthetic operator inputs using real configuration widths, not runs of those models; using a model's width does not mean this PR integrates the operator into that model. In this PR's Ouro code, gate_proj and up_proj are independent linear layers and return separate tensors before silu_and_mul.

On an otherwise idle B300 with no other compute process detected on that GPU, 126 cases were measured: BF16 rows 1/4/16/64/128/256/512/1024/2048/4096/8192; FP16 and FP32 rows 1/16/128/1024/4096; each at all three widths with separately allocated aligned (0, 0) and intentionally shifted (1, 3) input offsets. Offsets are in elements. The same PR, main and original vLLM commits listed above were used. Five shuffled CUDA Graph rounds, 32 calls per graph, 30/10/5 replays depending on size; 10 warmups per function. Results are medians in µs. As above, vLLM packed excludes packing and uses a packed input; vLLM + cat includes the GPU work to pack the separate inputs. Output allocation happens during graph capture, so Python/host allocator time is not measured.

The PR was faster than vLLM + cat in all 126/126 cases here, with a 1.41–4.29× ratio of median latencies. That is restricted to this GPU, matrix and separate-input contract; a model that already produces packed gate/up can use the vLLM packed path without cat. In the shifted subset, the PR was faster than vLLM packed in 25/63 cases, emphasizing that the two input layouts must not be conflated. All 126 cases matched elementwise across backends using torch.equal on finite random input (signed zeros not distinguished).

Representative BF16 aligned inputs:

Shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
1 × 5632	(0,0)	aligned vec	1.474	1.474	1.922	3.006
16 × 5632	(0,0)	aligned vec	1.542	1.540	1.950	3.360
128 × 5632	(0,0)	aligned vec	1.986	1.986	2.114	3.968
1024 × 5632	(0,0)	aligned vec	6.209	6.196	7.095	13.175
4096 × 5632	(0,0)	aligned vec	23.155	23.184	26.182	53.693
128 × 10752	(0,0)	aligned vec	2.562	2.561	3.202	5.635
1024 × 10752	(0,0)	aligned vec	10.307	10.294	12.163	26.128
4096 × 10752	(0,0)	aligned vec	42.243	42.272	52.346	106.035
128 × 17408	(0,0)	aligned vec	3.202	3.202	4.292	7.679
1024 × 17408	(0,0)	aligned vec	18.112	17.888	35.296	55.997
4096 × 17408	(0,0)	aligned vec	66.528	66.656	135.130	222.330

Representative BF16 shifted inputs:

Shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
128 × 5632	(1,3)	scalar	3.735	3.733	2.050	5.313
1024 × 5632	(1,3)	scalar	10.709	10.713	7.103	20.926
4096 × 5632	(1,3)	scalar	56.778	56.800	26.291	108.106
8192 × 5632	(1,3)	scalar	111.702	111.639	48.873	208.985
128 × 10752	(1,3)	scalar	5.317	5.316	3.202	7.823
1024 × 10752	(1,3)	scalar	17.082	17.046	12.170	52.954
8192 × 10752	(1,3)	misaligned vec	113.325	209.165	101.709	402.406
128 × 17408	(1,3)	scalar	6.532	6.535	4.297	10.918
1024 × 17408	(1,3)	scalar	44.768	44.266	34.541	98.832
4096 × 17408	(1,3)	misaligned vec	92.550	168.813	135.322	379.437

For BF16 Ouro width 5632, even [8192, 5632] is below the B300 cutoff of 48,496,640 elements; the shifted input therefore uses the scalar path in both PR and main. Its advantage over vLLM + cat is not a speedup introduced by this PR. At BF16 [8192, 10752] with offsets (1, 3), the PR enters the misaligned vector path and measures 113.325 µs versus main's 209.165 µs. Five-round ranges for that case are 113.286–113.363 µs and 209.165–209.204 µs, respectively. Representative ops-only/full profiles confirm the extra cat kernel and the selected PR/main paths; profiling was separate from timing.

All 126 additional model-width cases (CUDA Graph median, µs)

Ouro-1.4B width 5632

dtype / shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
BF16 1 × 5632	(0,0)	aligned vec	1.474	1.474	1.922	3.006
BF16 4 × 5632	(0,0)	aligned vec	1.474	1.474	1.922	3.138
BF16 16 × 5632	(0,0)	aligned vec	1.542	1.540	1.950	3.360
BF16 64 × 5632	(0,0)	aligned vec	1.771	1.749	1.986	3.570
BF16 128 × 5632	(0,0)	aligned vec	1.986	1.986	2.114	3.968
BF16 256 × 5632	(0,0)	aligned vec	2.562	2.562	3.138	5.443
BF16 512 × 5632	(0,0)	aligned vec	3.714	3.714	4.612	8.384
BF16 1024 × 5632	(0,0)	aligned vec	6.209	6.196	7.095	13.175
BF16 2048 × 5632	(0,0)	aligned vec	10.701	10.733	12.374	26.889
BF16 4096 × 5632	(0,0)	aligned vec	23.155	23.184	26.182	53.693
BF16 8192 × 5632	(0,0)	aligned vec	44.224	44.090	49.027	104.845
BF16 1 × 5632	(1,3)	scalar	1.282	1.282	1.922	3.394
BF16 4 × 5632	(1,3)	scalar	1.305	1.284	1.922	3.521
BF16 16 × 5632	(1,3)	scalar	1.474	1.474	1.986	3.778
BF16 64 × 5632	(1,3)	scalar	2.449	2.439	1.988	4.436
BF16 128 × 5632	(1,3)	scalar	3.735	3.733	2.050	5.313
BF16 256 × 5632	(1,3)	scalar	5.441	5.440	3.137	7.808
BF16 512 × 5632	(1,3)	scalar	7.456	7.440	4.637	12.333
BF16 1024 × 5632	(1,3)	scalar	10.709	10.713	7.103	20.926
BF16 2048 × 5632	(1,3)	scalar	17.808	17.910	12.368	55.235
BF16 4096 × 5632	(1,3)	scalar	56.778	56.800	26.291	108.106
BF16 8192 × 5632	(1,3)	scalar	111.702	111.639	48.873	208.985
FP16 1 × 5632	(0,0)	aligned vec	1.474	1.474	1.862	2.946
FP16 16 × 5632	(0,0)	aligned vec	1.553	1.596	1.935	3.330
FP16 128 × 5632	(0,0)	aligned vec	1.986	1.986	2.050	3.908
FP16 1024 × 5632	(0,0)	aligned vec	6.123	6.126	6.812	12.874
FP16 4096 × 5632	(0,0)	aligned vec	23.053	22.970	25.290	52.694
FP16 1 × 5632	(1,3)	scalar	1.282	1.282	1.860	3.393
FP16 16 × 5632	(1,3)	scalar	1.474	1.474	1.943	3.776
FP16 128 × 5632	(1,3)	scalar	3.762	3.766	2.054	5.313
FP16 1024 × 5632	(1,3)	scalar	10.714	10.711	6.799	20.631
FP16 4096 × 5632	(1,3)	scalar	57.123	57.174	25.450	107.331
FP32 1 × 5632	(0,0)	aligned vec	1.282	1.282	2.114	3.340
FP32 16 × 5632	(0,0)	aligned vec	1.346	1.346	2.187	3.650
FP32 128 × 5632	(0,0)	aligned vec	1.935	1.933	2.306	4.865
FP32 1024 × 5632	(0,0)	aligned vec	6.784	6.773	7.871	22.815
FP32 4096 × 5632	(0,0)	aligned vec	40.692	40.627	41.385	97.558
FP32 1 × 5632	(1,3)	scalar	1.282	1.282	2.114	3.586
FP32 16 × 5632	(1,3)	scalar	1.474	1.474	2.189	4.118
FP32 128 × 5632	(1,3)	scalar	3.733	3.729	2.306	5.635
FP32 1024 × 5632	(1,3)	scalar	18.262	18.250	7.851	33.719
FP32 4096 × 5632	(1,3)	scalar	65.507	65.437	41.353	135.002

Nanbeige4.2-3B width 10752

dtype / shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
BF16 1 × 10752	(0,0)	aligned vec	1.474	1.474	2.947	4.288
BF16 4 × 10752	(0,0)	aligned vec	1.538	1.538	3.012	4.353
BF16 16 × 10752	(0,0)	aligned vec	1.602	1.602	3.074	4.609
BF16 64 × 10752	(0,0)	aligned vec	1.986	1.986	3.137	5.060
BF16 128 × 10752	(0,0)	aligned vec	2.562	2.561	3.202	5.635
BF16 256 × 10752	(0,0)	aligned vec	3.681	3.671	4.447	8.071
BF16 512 × 10752	(0,0)	aligned vec	5.955	5.955	7.364	13.457
BF16 1024 × 10752	(0,0)	aligned vec	10.307	10.294	12.163	26.128
BF16 2048 × 10752	(0,0)	aligned vec	22.164	22.116	27.904	54.176
BF16 4096 × 10752	(0,0)	aligned vec	42.243	42.272	52.346	106.035
BF16 8192 × 10752	(0,0)	aligned vec	81.702	81.702	101.760	208.755
BF16 1 × 10752	(1,3)	scalar	1.282	1.282	2.946	4.480
BF16 4 × 10752	(1,3)	scalar	1.346	1.346	3.012	4.739
BF16 16 × 10752	(1,3)	scalar	1.794	1.794	3.074	5.134
BF16 64 × 10752	(1,3)	scalar	3.597	3.602	3.138	6.388
BF16 128 × 10752	(1,3)	scalar	5.317	5.316	3.202	7.823
BF16 256 × 10752	(1,3)	scalar	7.178	7.173	4.428	11.853
BF16 512 × 10752	(1,3)	scalar	10.382	10.395	7.374	20.771
BF16 1024 × 10752	(1,3)	scalar	17.082	17.046	12.170	52.954
BF16 2048 × 10752	(1,3)	scalar	55.206	55.245	28.093	106.093
BF16 4096 × 10752	(1,3)	scalar	105.690	105.606	52.451	205.373
BF16 8192 × 10752	(1,3)	misaligned vec	113.325	209.165	101.709	402.406
FP16 1 × 10752	(0,0)	aligned vec	1.474	1.474	2.992	4.226
FP16 16 × 10752	(0,0)	aligned vec	1.602	1.602	3.010	4.569
FP16 128 × 10752	(0,0)	aligned vec	2.526	2.553	3.170	5.639
FP16 1024 × 10752	(0,0)	aligned vec	10.134	10.128	11.658	25.616
FP16 4096 × 10752	(0,0)	aligned vec	42.016	41.955	50.301	103.501
FP16 1 × 10752	(1,3)	scalar	1.282	1.282	2.945	4.421
FP16 16 × 10752	(1,3)	scalar	1.794	1.794	3.011	5.121
FP16 128 × 10752	(1,3)	scalar	5.339	5.343	3.138	7.814
FP16 1024 × 10752	(1,3)	scalar	17.021	16.995	11.663	52.387
FP16 4096 × 10752	(1,3)	scalar	107.773	107.760	50.477	203.389
FP32 1 × 10752	(0,0)	aligned vec	1.269	1.282	2.822	4.035
FP32 16 × 10752	(0,0)	aligned vec	1.474	1.474	2.885	4.481
FP32 128 × 10752	(0,0)	aligned vec	2.643	2.632	3.010	6.859
FP32 1024 × 10752	(0,0)	aligned vec	20.198	20.218	21.814	46.659
FP32 4096 × 10752	(0,0)	aligned vec	76.355	76.477	76.924	185.840
FP32 1 × 10752	(1,3)	scalar	1.282	1.282	2.882	4.289
FP32 16 × 10752	(1,3)	scalar	1.794	1.794	2.886	4.990
FP32 128 × 10752	(1,3)	scalar	5.377	5.377	3.010	7.722
FP32 1024 × 10752	(1,3)	scalar	35.674	35.629	21.735	67.622
FP32 4096 × 10752	(1,3)	misaligned vec	91.629	116.678	76.944	250.336

Qwen3.8-27B text width 17408

dtype / shape	gate/up offsets	PR path	PR	Main	vLLM packed	vLLM + cat
BF16 1 × 17408	(0,0)	aligned vec	1.474	1.474	4.100	5.376
BF16 4 × 17408	(0,0)	aligned vec	1.538	1.538	4.162	5.635
BF16 16 × 17408	(0,0)	aligned vec	1.604	1.606	4.163	5.888
BF16 64 × 17408	(0,0)	aligned vec	2.306	2.306	4.226	6.527
BF16 128 × 17408	(0,0)	aligned vec	3.202	3.202	4.292	7.679
BF16 256 × 17408	(0,0)	aligned vec	5.012	5.008	8.837	14.392
BF16 512 × 17408	(0,0)	aligned vec	8.746	8.730	16.595	25.580
BF16 1024 × 17408	(0,0)	aligned vec	18.112	17.888	35.296	55.997
BF16 2048 × 17408	(0,0)	aligned vec	34.586	34.550	69.440	111.459
BF16 4096 × 17408	(0,0)	aligned vec	66.528	66.656	135.130	222.330
BF16 8192 × 17408	(0,0)	aligned vec	130.727	131.021	268.403	442.406
BF16 1 × 17408	(1,3)	scalar	1.282	1.282	4.098	5.504
BF16 4 × 17408	(1,3)	scalar	1.410	1.410	4.100	6.023
BF16 16 × 17408	(1,3)	scalar	2.179	2.179	4.162	6.615
BF16 64 × 17408	(1,3)	scalar	5.025	5.050	4.227	8.514
BF16 128 × 17408	(1,3)	scalar	6.532	6.535	4.297	10.918
BF16 256 × 17408	(1,3)	scalar	8.910	8.912	8.831	20.209
BF16 512 × 17408	(1,3)	scalar	14.158	14.154	16.593	40.731
BF16 1024 × 17408	(1,3)	scalar	44.768	44.266	34.541	98.832
BF16 2048 × 17408	(1,3)	scalar	87.600	87.402	69.414	193.907
BF16 4096 × 17408	(1,3)	misaligned vec	92.550	168.813	135.322	379.437
BF16 8192 × 17408	(1,3)	misaligned vec	180.877	338.797	268.083	752.698
FP16 1 × 17408	(0,0)	aligned vec	1.474	1.474	4.034	5.257
FP16 16 × 17408	(0,0)	aligned vec	1.604	1.604	4.098	5.824
FP16 128 × 17408	(0,0)	aligned vec	3.202	3.202	4.226	7.561
FP16 1024 × 17408	(0,0)	aligned vec	17.878	17.795	33.696	55.546
FP16 4096 × 17408	(0,0)	aligned vec	66.324	66.477	132.161	218.925
FP16 1 × 17408	(1,3)	scalar	1.282	1.282	4.034	5.441
FP16 16 × 17408	(1,3)	scalar	2.178	2.179	4.098	6.532
FP16 128 × 17408	(1,3)	scalar	6.540	6.542	4.227	10.923
FP16 1024 × 17408	(1,3)	scalar	43.907	44.406	34.251	98.467
FP16 4096 × 17408	(1,3)	misaligned vec	90.925	173.165	132.147	376.531
FP32 1 × 17408	(0,0)	aligned vec	1.346	1.346	3.847	5.257
FP32 16 × 17408	(0,0)	aligned vec	1.538	1.538	3.968	6.098
FP32 128 × 17408	(0,0)	aligned vec	3.462	3.471	4.098	9.793
FP32 1024 × 17408	(0,0)	aligned vec	31.796	31.776	33.555	76.598
FP32 4096 × 17408	(0,0)	aligned vec	123.187	123.613	124.775	301.395
FP32 1 × 17408	(1,3)	scalar	1.282	1.282	3.906	5.387
FP32 16 × 17408	(1,3)	scalar	2.178	2.178	3.962	6.530
FP32 128 × 17408	(1,3)	scalar	6.532	6.536	4.098	10.812
FP32 1024 × 17408	(1,3)	scalar	51.555	51.555	33.338	106.599
FP32 4096 × 17408	(1,3)	misaligned vec	144.801	184.115	124.826	402.560

The exact comparison script, per-round CUDA Graph and eager timings, model-config hashes, profiles, and GPU process-monitor log are retained locally under ignored artifacts/silu-unified-20261103/realistic-shapes-* files; the complete median matrix is included above.

AI assistance was used for implementation, measurements and author self-review.