Scale Residual Streams Wider · Push Model Intelligence Further
xHC expands the residual state to N=16 streams, reads all streams, and sparsely updates only k=4 active streams.
- 2026.11 We released SimpleHC, which simplifies HC architecture and execution, reducing additional training overhead by 62–76% relative to mHC while maintaining comparable model quality on 15B and 30B MoE models. See the technical report.
- 2026.10 We released the training code, reimplemented using Megatron-LM, along with efficient Triton kernels for xHC to support further community research. See the training curves.
- 2026.09 🎉 xHC has been accepted to NeurIPS 2026!
- 2026.07 We released the xHC technical report on arXiv.
Using the training code provided here, we observe lower training loss than baseline and mHC for an 18B MoE on private data. We also reproduce these gains for a 2.5B MoE on public Dolma3 data.
These runs use full experimental schedules; xHC denotes the efficient backend. Curves show lightly EMA-smoothed loss over faint raw values, using the same 50-step EMA span for all methods and both model sizes. The view covers 10%–100% of training with cropped y-axes; no outlier filtering is applied.
Hyper-Connections (HC) expand the Transformer residual stream into
We show that directly scaling mHC beyond
- Information supply: one write-back vector must provide new information to an increasing number of streams, making additional streams progressively redundant.
-
Residual-mapping cost: generating a dense
$N\times N$ mapping from an$NC$ -dimensional state scales as$O(N^3C)$ .
Scaling mHC from N=4 to N=16 brings diminishing gains at rapidly increasing cost; xHC changes this trade-off.
xHC (Expanded Hyper-Connections) is the first HC-family method to achieve meaningful expansion beyond
xHC combines two complementary designs:
- Temporal feature augmentation enriches MLP write-back with multi-scale local features produced by causal depthwise convolutions over neighboring tokens. This supplies a more diverse information basis as the number of residual streams grows.
-
Sparse residual-stream updates preserve dense read access to all
$N$ streams while routing residual mixing and write-back through only$k$ active streams. The dominant residual-mapping generation cost is reduced from$O(N^3C)$ to$O(k^3C)$ .
For practical deployment, xHC-Flash shares full-state operations across consecutive sublayers and reconstructs later sublayer inputs with lightweight corrections. It substantially reduces memory traffic while preserving nearly all of the performance gains of full xHC.
![]() |
![]() |
![]() |
| Training loss | Average downstream score | Benchmark-level gains |
Across 12 language understanding, reasoning, code, and Chinese benchmarks, xHC improves the average score over mHC by 4.0 points at 18B and 3.1 points at 28B. At 28B, xHC outperforms mHC on 11 of the 12 evaluated benchmarks.
xHC traces a consistently lower loss curve than mHC and the vanilla residual baseline across training compute.
- On the 18B MoE model, xHC reaches a final training loss of 1.758, compared with 1.776 for mHC and 1.799 for the vanilla baseline.
- To reach the same loss as xHC, the vanilla and mHC baselines require
$1.50\times$ and$1.19\times$ as much training compute, respectively. - Increasing
$N$ from 4 to 16 improves loss by 0.012 in xHC with only 4% additional training FLOPs. The same expansion in mHC improves loss by only 0.006 while increasing training FLOPs by 32%. - xHC remains effective under Muon, indicating that its gains are not specific to AdamW.
Estimated per-token memory traffic, amortized per Transformer sublayer. Here,
| Method | I/O per sublayer | Validation loss |
|---|---|---|
| mHC ( |
2.004 | |
| xHC | 1.983 | |
| xHC-Flash | 1.983 | |
| xHC-Flash-4sub | 1.984 |
xHC-Flash-4sub approaches the memory traffic of mHC at
Our implementation uses fused forward and backward kernels to reduce intermediate tensor materialization and kernel-launch overhead.
- On the 18B MoE model, our fused mHC implementation adds approximately 15% training overhead over the vanilla baseline, while xHC-Flash-4sub adds approximately 11% on top of mHC. Pipeline-communication overlap is disabled in these measurements to isolate the computational overhead of residual-stream operations.
- For inference prefill at 2K tokens, mHC and xHC-Flash-4sub add 11.4% and 12.9% overhead over the vanilla baseline, respectively. The total prefill latency of xHC-Flash-4sub is only 1.3% higher than that of mHC.
Default xHC configuration
| Setting | Value |
|---|---|
| Expansion streams | |
| Active streams | |
| Fixed / routed active streams | 2 / 2 |
| Temporal augmentation | MLP side only |
| Causal convolution kernels | |
| Residual mixing | Sinkhorn-normalized over active streams |
Training code is in Megatron-LM/, configurations in configs/, and launch scripts
in scripts/. The release includes baseline, mHC, xHC, and efficient full xHC.
The efficient xHC backend uses custom Triton kernels to fuse connection operations in the forward and backward passes.
xHC-Flash is not included.
The supplied recipes use a reference 1,000-step training budget, which may differ from those used in the paper, as the token budgets are not disclosed in the paper.
Adjust the training length and learning-rate schedule for your experiments.
Implementation: mHC/xHC, Transformer layer integration, and efficient backend. Triton kernels: fused xHC operations and Gram–Schmidt.
docker build -t xhc:latest .
docker run --rm -it --gpus all --network host --ipc host \
-v "$PWD:/workspace" \
-v /path/to/data:/data:ro \
-v /path/to/tokenizer:/tokenizer:ro \
-w /workspace xhc:latest bashThe default Docker build targets Blackwell GPUs. For Hopper, build with
--build-arg TORCH_CUDA_ARCH_LIST=9.0 --build-arg NVTE_CUDA_ARCHS=90.
Inside the container:
cp configs/runtime.env.example configs/runtime.env
cp configs/data_mixture.json.example configs/data_mixture.jsonSet the indexed .bin/.idx dataset prefixes in configs/data_mixture.json.
In configs/runtime.env, set DATA_ARGS=/workspace/configs/data_mixture.json
and TOKENIZER_DIR=/tokenizer. Preprocess the data with the same tokenizer used
for training. Data and tokenizer files are supplied separately.
For public-data experiments, use Dolma3 with the Qwen2 tokenizer. This public dataset differs from the private training data used in the paper. Run inside the training container:
pip install zstandard
python scripts/prepare_dolma3.py --output data/dolma3 --workers 16
export DATA_ARGS="$PWD/data/dolma3/data_args.json"
export TOKENIZER_DIR="$PWD/data/dolma3/tokenizer"The script downloads the pinned public snapshot (about 111 GB compressed), appends EOS to each document, and builds deterministic train/validation/test indices and the data mixture. Re-running resumes completed groups. Use the same prepared data for all model variants; preparation uses CPUs and needs no GPU.
Choose one script from scripts/:
| Model | baseline | mHC | xHC | efficient xHC |
|---|---|---|---|---|
| 2.5B | train_2.5B_baseline.sh |
train_2.5B_mHC.sh |
train_2.5B_xHC.sh |
train_2.5B_xHC_efficient.sh |
| 10B | train_10B_baseline.sh |
train_10B_mHC.sh |
train_10B_xHC.sh |
train_10B_xHC_efficient.sh |
| 18B | train_18B_baseline.sh |
train_18B_mHC.sh |
train_18B_xHC.sh |
train_18B_xHC_efficient.sh |
bash scripts/train_10B_xHC_efficient.shFor 32-GPU training, run the same command on each of four allocated 8-GPU nodes:
export NNODES=4 GPUS=8 NODE_RANK=0 # Set NODE_RANK=0..3 on the respective nodes.
export MASTER_ADDR=192.0.2.10 MASTER_PORT=29500 # Replace with node 0's IP.
export EXP_NAME=10B_xHC_efficient
bash scripts/train_10B_xHC_efficient.shAdjust expert parallelism and communication to suit your hardware by setting
EP and MOE_DISPATCHER before launching, for example
export EP=4 MOE_DISPATCHER=hybridep. Supported backends are alltoall, hybridep,
and deepep; the latter two require their corresponding dependencies.
If GPU memory is insufficient, reduce the micro-batch size with export MBS=1.
Checkpoints are saved every 500 steps under experiments/<EXP_NAME>/checkpoints/;
TensorBoard logs are in experiments/tensorboard/<EXP_NAME>/. The same command
resumes the experiment. Use a separate experiment name for each concurrent run.
The efficient xHC implementation improves training efficiency but does not represent the highest achievable efficiency; there is room for further optimization.
@misc{zhang2026xhcexpandedhyperconnections,
title = {xHC: Expanded Hyper-Connections},
author = {Xiangdong Zhang and Xiaohan Qin and Sunan Zou and Tuo Dai and
Xiaoming Shi and Huaijin Wu and Yebin Yang and Zhuo Xia and
Shaofeng Zhang and Lin Yao and Yuliang Liu and Yu Cheng and
Junchi Yan},
year = {2026},
eprint = {2607.14530},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2607.14530}
}For questions about the paper, please contact zhangxiangdong@sjtu.edu.cn.
Our contributions are licensed under Apache 2.0. Third-party code retains its original licenses; see Megatron-LM/LICENSE and individual file notices.







