feat(ds4): add external ROCTX phase markers

[cheese-cakee]

Summary

• add default-off semantic ROCTX ranges for external ROCm tracing
• cover monolithic and heterogeneous prefill, speculative decode, and leaf layer ranges
• label exact prefill, decode, and verification explicitly across local and remote target shards
• dynamically load ROCTX only when enabled and keep non-HIP builds as no-ops
• preserve rocprof's process-global ROCTX interposition while retaining the direct libroctx64.so fallback

This replaces the superseded historical in-process HIP-event profiler on current main. The old implementation recorded on a null/default stream and synchronized events, which could fence unrelated streams and distort the two-owner overlap being measured. This rewrite intentionally performs no internal GPU timing.

Behavior contract

DFLASH_DS4_ROCTX=1 enables balanced semantic ranges for correlation with an external rocprof kernel/copy trace. The marker layer:

• uses no HIP/CUDA events
• performs no stream or device synchronization
• calculates no elapsed time
• remains silent and does not load ROCTX when unset or false
• reports a one-time diagnostic when explicitly enabled but ROCTX cannot be loaded
• closes incomplete library handles when required symbols are missing
• balances successful pushes through RAII, including early-return paths
• avoids recursive layer-range double counting by marking leaf ranges only
• carries a validated value-type semantic phase through both target-shard IPC transports
• drains safely framed payloads before rejecting missing, extra, or invalid phase metadata

ROCTX span lengths are host-side semantic intervals. They are not GPU kernel-duration measurements.

Demonstrated loader defect and fix

Real rocprofv3 validation found that the previous handle-scoped dlopen("libroctx64.so") plus dlsym(handle, ...) path bypassed rocprof's preloaded ROCTX interposer: a normally linked ROCTX smoke test traced, while the equivalent dynamic-loader smoke test and server did not.

Commit e028bb22b136695bf4ef89f1677ed4670eae95ee adds a regression that fails with the old lookup, then prefers a complete process-global roctxRangePushA/roctxRangePop pair through dlopen(nullptr). It preserves the original library fallback and does not change Windows behavior or add synchronization.

Current-main build and test validation

Validated as a no-commit synthetic merge with upstream/main 4fa8937c69aa68658a14ec3cab16e1da22886dd0.

• strict C++17 -Wall -Wextra -Werror ROCTX/protocol test: passed
• ASan/UBSan ROCTX/protocol test: passed
• full HIP/gfx1151 RelWithDebInfo build: passed
• full HIP CTest: 364/364 passed
• full CUDA/sm_86 RelWithDebInfo build with CUDA 12.0.140: passed
• final CUDA-configured host-only ROCTX regression: passed
• malformed target-shard phase followed by a valid request: passed for missing, extra, and invalid values
• IPC serialization/value validation and next-request synchronization tests: passed
• git diff --check: passed
• fresh independent final review of the three-file fix: no actionable findings
• review follow-up 0aec3941088ae348df62c8a5fc8d9eedbb4599fd re-executes the real loader regression in an isolated child process so neither DFLASH_DS4_ROCTX nor the process-scoped callback cache leaks into the parent suite; HIP-defined strict, HIP ASan/UBSan, non-HIP strict, and the CUDA-configured repository test pass
• wait hardening 7c87686bad02bb1a1c5b96c2986ad37d3c27fa63 bounds the child to approximately five seconds with WNOHANG, terminates and reaps only that owned child on timeout, and reports signal termination explicitly; the same strict, sanitizer, non-HIP, and repository tests pass

Representative commands:

cmake -S . -B build-pr554-hip-gfx1151 -G Ninja \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DGGML_HIP=ON -DGGML_CUDA=OFF \
  -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DDFLASH27B_TESTS=ON \
  -DDFLASH27B_ENABLE_BSA=OFF
cmake --build build-pr554-hip-gfx1151 -j 4
ctest --test-dir build-pr554-hip-gfx1151 --output-on-failure --timeout 180

cmake -S . -B build-pr554-cuda-sm86 -G Ninja \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo -DGGML_CUDA=ON -DGGML_HIP=OFF \
  -DCMAKE_CUDA_ARCHITECTURES=86 -DDFLASH27B_TESTS=ON \
  -DDFLASH27B_ENABLE_BSA=ON
cmake --build build-pr554-cuda-sm86 -j 4
ctest --test-dir build-pr554-cuda-sm86 \
  -R '^test_deepseek4_roctx$' --output-on-failure --timeout 60

Real gfx1151 rocprof capture

Environment:

• Radeon 8060S, gfx1151
• ROCm 7.2.4 / HIP 7.2.53211
• rocprofv3 1.1.0
• final HIP dflash_server SHA-256: 2e545e9d4205773c679bb88b2f1058bac9768f241331d63d816023130fdbacd6

The enabled speculative request captured 21 balanced marker intervals:

• one ds4.prefill mode=exact enclosing 11 exact leaf ds4.layer_range intervals
• one ds4.spec_decode mode=batched enclosing 8 verify leaf ds4.layer_range intervals

A separate target-only request captured 27 intervals:

• one exact prefill plus 11 exact leaf layer ranges
• 15 mode=decode leaf layer ranges

All intervals had sane ordering and intelligible request nesting. Unset and explicit DFLASH_DS4_ROCTX=0 real-request controls produced no DS4 marker rows and no marker CSV.

Profiler command form:

rocprofv3 --marker-trace --minimum-output-data 0 -f csv json \
  -d <output-directory> -o trace -- env DFLASH_DS4_ROCTX=1 \
  <dflash_server model and request configuration>

Alternating disabled/enabled perturbation

The comparison used fresh processes, cache slots disabled, the same binary/models/prompt/placement, two warmups per mode, then five measured off,on pairs.

Mode	Server total median	MAD	Range
Disabled	3165.8 ms	2.4 ms	3160.4-3204.6 ms
Enabled	3188.6 ms	13.4 ms	3157.5-3217.0 ms

Measured median change: +0.72% in this bounded sample.

Correctness parity was exact across all ten measured requests:

• identical generated-content SHA-256: 39bcfb8d666171e8f833154fef4a6f70918ffb7458e4c8b32813b865e94abb67
• identical (prompt, completion, total) token counts: (11, 28, 39)
• acceptance rate 1.0 throughout

Clocks were not administratively pinned; alternating pairs were used to limit temporal drift. This result is instrumentation-perturbation evidence for this setup, not a general performance claim.

Remaining merge gates

• repository CI on exact head 7c87686bad02bb1a1c5b96c2986ad37d3c27fa63
• maintainer review/approval

Disclosed hardware follow-ups

• rerun the full local CUDA runtime suite after the RTX 3090 device handle is restored
• capture a real mixed CUDA/HIP target-shard trace after that device is restored; protocol-level phase transport is already covered

These unavailable local cells are not assumed to pass and are not presented as evidence. They are follow-up hardware coverage rather than blockers for this HIP-only marker-loader correction: the CUDA build and CUDA-configured host regression pass, while exact-head repository CI remains the cross-platform merge gate.

The PR is locally hardware-qualified for the available gfx1151 paths, but it is not genuinely merge-ready until exact-head CI and maintainer approval complete.

Stack

Independent current-main PR. It does not depend on #548 or the #569/#575/#577 performance stack. #575 may provide a useful external rocprof collection wrapper, but it is not a code prerequisite.

[cubic-dev-ai]

All reported issues were addressed across 10 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 6 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 5 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 10 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 1 file (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>