refactor(moe): extract hybrid decode orchestration

[zihaomu]

Summary

This PR does one thing:

Move the shared Hybrid MoE decode schedule out of OffloadMoELayer and into a
reusable HybridDecodeExecutor.

It does not change LRU policy, expert placement, expert kernels, numerical
behavior, or CUDA/ROCm synchronization.

This is a stacked Draft PR on top of #378. The refactor itself is commit d5fa332.

Depends on #378. Related to #350.

1. What happened before this PR

OffloadMoELayer is a model layer, but its _decode_hybrid() method also owned the
entire CPU/GPU execution schedule:

router output
    |
    v
clone raw expert IDs
    |
run capped-fetch LRU and rewrite IDs to GPU slots / -1
    |
split routes into CPU work and GPU work
    |
submit CPU expert computation
    |
copy missing experts to the GPU and run the GPU expert kernel
    |
wait for CPU computation
    |
GPU partial + CPU partial

This mixed four different responsibilities in one model class:

1. Cache policy — which experts stay on the GPU and which routes overflow.
2. Hybrid scheduling — how CPU and GPU work are ordered and overlapped.
3. Platform synchronization — how CUDA or ROCm connects GPU stream progress to
   CPU worker progress.
4. Expert math — which model/quantization-specific GPU kernel runs.

Only the fourth responsibility belongs to the model layer.

The practical problem was that supporting another model or device meant touching a
method that knew about all four concerns. In particular, AMD support appeared to need
its own Hybrid implementation even though AMD only differs in the low-level
synchronization mechanism.

2. The refactor principle

The refactor separates policy, orchestration, platform mechanism, and
model-specific computation:

                         HybridDecodeExecutor
                    owns the shared execution order
                         /                  \
                        /                    \
                       v                      v
             OffloadMoeCache             CpuMoeExecutor
             cache/LRU policy            CPU compute and
                                         CUDA/ROCm synchronization
                       ^
                       |
              GPU expert callback
                       |
                OffloadMoELayer
          model and quantization-specific math

The new executor is deliberately small. It knows how to split routes, overlap CPU and
GPU work, and merge their outputs. It does not contain CUDA checks, ROCm checks, cache
replacement logic, or quantization dispatch.

3. What changed in the code

New: python/freetoken/moe/hybrid_decode.py

This file introduces:

• HybridDecodeRequest: the four inputs needed by Hybrid decode;
• small protocols for the cache, CPU executor, and GPU expert callback;
• HybridDecodeExecutor.decode(): the extracted route/scheduling algorithm.

Changed: python/freetoken/layers/moe.py

OffloadMoELayer._decode_hybrid() is removed.

When the decode target is Hybrid, the layer now delegates:

HybridDecodeExecutor.decode(request, gpu_expert_runner)

The layer supplies _run_cached_decode_experts as the callback. That callback keeps
using the existing _expert_gemm quantization dispatch and current cache views.

In other words, the model layer still decides how GPU experts are computed, but no
longer decides how CPU and GPU execution is scheduled.

Changed: python/freetoken/moe/offload_cache.py

When the Engine attaches a CPU executor for Hybrid mode, the cache creates one
HybridDecodeExecutor and stores it beside the CPU executor.

This is temporary ownership chosen to preserve the current Engine initialization and
lifetime order. Moving runtime ownership is intentionally left for a later PR.

New tests and proposal

• tests/moe/test_hybrid_decode_executor.py verifies the extracted contract without a
  GPU;
• docs/proposals/0001-hybrid-decode-executor.md records the longer-term design and
  follow-up boundaries.

4. How decode works after this PR

The runtime algorithm is unchanged; it now lives in one reusable executor.

For every Hybrid decode layer:

1. Save the router's original expert IDs.
2. Ask OffloadMoeCache.ensure_experts_hybrid() to apply the existing LRU and
   capped-fetch policy.
3. Interpret the rewritten IDs:
   • a non-negative value is a GPU cache slot;
   • -1 means that route must run on the CPU.
4. Build complementary CPU and GPU routes.
5. Submit CPU expert work first.
6. Copy selected missing experts and run the GPU expert callback.
7. Wait for CPU work.
8. Add the GPU and CPU partial results.

The overlap order remains:

cache decision
    -> CPU submit
        -> GPU cache copy
            -> GPU expert kernel
                -> CPU wait
                    -> add partial results

Submitting CPU work before the GPU copy/kernel allows CPU GEMV to overlap with PCIe
traffic and GPU execution. Setting FREETOKEN_HYBRID_OVERLAP=0 keeps the existing
debug/measurement mode by moving the CPU wait before the GPU work.

5. How routes are split

For example, suppose the router selects experts [11, 42] with weights [0.7, 0.3].
The existing cache policy places expert 11 in GPU slot 3 and sends expert 42 to the
CPU:

original expert IDs   [11, 42]
cache result          [ 3, -1]

CPU expert IDs        [-1, 42]   # the CPU kernel skips -1
GPU slot IDs          [ 3,  0]   # slot 0 is safe because its weight is zero
GPU weights           [0.7, 0.0]

The CPU receives only CPU routes. The GPU receives only GPU routes. Their masks are
complementary, so each routed contribution is computed exactly once and the two
outputs can be added directly.

This PR also preserves two graph-related invariants:

• topk_ids is still rewritten in place;
• an all-GPU step still submits and waits for the empty CPU side, keeping the captured
  execution structure independent of the cache-hit pattern.

6. Why this works for both CUDA and ROCm

The shared algorithm above does not depend on a GPU vendor.

The vendor-specific boundary is CpuMoeExecutor.decode_submit() /
decode_sync(), which this PR does not change:

Platform	Existing mechanism below the boundary
CUDA	mapped-pinned flags and stream memory operations, with the existing host-callback fallback
ROCm	HIP signal memory and graph batch-memory-op nodes from #378, with graph disabled when its capability probe fails

HybridDecodeExecutor calls the same two methods on either platform. It neither knows
nor needs to know which mechanism implements them.

For the same reason, AMD does not need a separate LRU. OffloadMoeCache already makes
the platform-independent placement decision and produces the same GPU-slot / -1
contract for CUDA and ROCm. Only synchronization differs, and that is below the new
executor boundary.

7. What did not change

Area	Status in this PR
LRU and capped-fetch decisions	unchanged
Cache slot layout and -1 overflow marker	unchanged
CPU-before-GPU overlap order	unchanged
BF16/NVFP4/MXFP4/DS-FP4/Q4_0 kernels	unchanged
Quantization dispatch in _expert_gemm	unchanged
CUDA synchronization	unchanged
ROCm graph handshake and fallback	unchanged
Final result (gpu_partial + cpu_partial)	unchanged

This PR is a responsibility move, not a new Hybrid algorithm.

8. Validation

Run in a PyTorch 2.11 / ROCm 7.14 container without a GPU attached:

HybridDecodeExecutor contract tests       7 passed
ROCm CPU/Hybrid graph-safety tests       14 passed
Adjacent Hybrid fetch tests               3 passed
CPU MoE GPU-dependent tests              30 skipped (no GPU attached)
Python compilation and diff checks        passed

The new contract tests cover:

• overlapped and serialized execution order;
• complementary CPU/GPU route masks;
• exact ID and weight rewriting;
• all-GPU fixed submit/wait behavior;
• stats recording;
• cache/executor wiring;
• delegation from OffloadMoELayer.

CUDA and ROCm hardware performance/regression runs are still required before moving
this PR out of Draft. The target is no measurable decode-throughput regression.

9. What reviewers need to decide

The implementation is correct if these four statements hold:

1. HybridDecodeExecutor is the right owner for shared route splitting and CPU/GPU
   scheduling.
2. OffloadMoELayer should expose only the model-specific GPU expert callback.
3. CUDA/ROCm synchronization should remain hidden behind CpuMoeExecutor.
4. Temporary ownership beside OffloadMoeCache.cpu_executor is acceptable until a
   later OffloadMoeRuntime ownership refactor.

[zihaomu]

Rebased this proposal onto the refreshed CPU/Hybrid ROCm graph-safety PR (#378, head 94f7724) and ported the extraction to the current upstream MoE architecture.

The conflict in layers/moe.py was resolved semantically: the current moe_strategy and quant-method dispatch remain intact, while Hybrid route splitting / CPU-GPU overlap / result merging move into HybridDecodeExecutor. OffloadMoELayer now supplies only the narrow cached-GPU-expert callback. No legacy MoE implementation was restored.

Validation:

• Hybrid executor contracts + ROCm graph-safety tests: 21 passed
• adjacent Hybrid fetch tests: 3 passed
• CPU MoE suites: 30 GPU-dependent cases collected/skipped in the no-GPU container
• Python compile and git diff --check pass

New head: d5fa33231edd2328370fe7a7968b00701147c082.