Skip to content
aHapBeanPublic

About

[Tech Report, NeurIPS 2026] Expanded Hyper-Connections

Resources

Stars

78 stars

Watchers

0 watching

Forks

Repository files navigation

xHC: Expanded Hyper-Connections

Scale Residual Streams Wider · Push Model Intelligence Further

arXiv  |  Technical Report  |  Citation

xHC architecture overview

xHC expands the residual state to N=16 streams, reads all streams, and sparsely updates only k=4 active streams.

News

  • 2026.11 We released SimpleHC, which simplifies HC architecture and execution, reducing additional training overhead by 62–76% relative to mHC while maintaining comparable model quality on 15B and 30B MoE models. See the technical report.
  • 2026.10 We released the training code, reimplemented using Megatron-LM, along with efficient Triton kernels for xHC to support further community research. See the training curves.
  • 2026.09 🎉 xHC has been accepted to NeurIPS 2026!
  • 2026.07 We released the xHC technical report on arXiv.

Training Reproduction

Using the training code provided here, we observe lower training loss than baseline and mHC for an 18B MoE on private data. We also reproduce these gains for a 2.5B MoE on public Dolma3 data.

Training LM loss for baseline, mHC and efficient xHC on 18B private-data and 2.5B Dolma3 experiments

These runs use full experimental schedules; xHC denotes the efficient backend. Curves show lightly EMA-smoothed loss over faint raw values, using the same 50-step EMA span for all methods and both model sizes. The view covers 10%–100% of training with cropped y-axes; no outlier filtering is applied.

Overview

Hyper-Connections (HC) expand the Transformer residual stream into $N$ parallel streams, introducing a memory-scaling axis complementary to model width and depth. Manifold-Constrained Hyper-Connections (mHC) stabilize this formulation at scale, but existing HC-family methods typically stop at $N=4$.

We show that directly scaling mHC beyond $N=4$ runs into two bottlenecks:

  • Information supply: one write-back vector must provide new information to an increasing number of streams, making additional streams progressively redundant.
  • Residual-mapping cost: generating a dense $N\times N$ mapping from an $NC$-dimensional state scales as $O(N^3C)$.

Expansion efficiency comparison between mHC and xHC

Scaling mHC from N=4 to N=16 brings diminishing gains at rapidly increasing cost; xHC changes this trade-off.

xHC (Expanded Hyper-Connections) is the first HC-family method to achieve meaningful expansion beyond $N=4$. Its main configuration uses $N=16$ residual streams and updates only $k=4$ streams at each sublayer.

Method

xHC combines two complementary designs:

  1. Temporal feature augmentation enriches MLP write-back with multi-scale local features produced by causal depthwise convolutions over neighboring tokens. This supplies a more diverse information basis as the number of residual streams grows.
  2. Sparse residual-stream updates preserve dense read access to all $N$ streams while routing residual mixing and write-back through only $k$ active streams. The dominant residual-mapping generation cost is reduced from $O(N^3C)$ to $O(k^3C)$.

For practical deployment, xHC-Flash shares full-state operations across consecutive sublayers and reconstructs later sublayer inputs with lightweight corrections. It substantially reduces memory traffic while preserving nearly all of the performance gains of full xHC.

Results

Broad Gains at 18B Scale

18B training loss 18B average downstream score 18B benchmark gains
Training loss Average downstream score Benchmark-level gains

Downstream evaluation on 18B and 28B MoE models

Across 12 language understanding, reasoning, code, and Chinese benchmarks, xHC improves the average score over mHC by 4.0 points at 18B and 3.1 points at 28B. At 28B, xHC outperforms mHC on 11 of the 12 evaluated benchmarks.

Compute Efficiency

Scaling-law comparison

xHC traces a consistently lower loss curve than mHC and the vanilla residual baseline across training compute.

  • On the 18B MoE model, xHC reaches a final training loss of 1.758, compared with 1.776 for mHC and 1.799 for the vanilla baseline.
  • To reach the same loss as xHC, the vanilla and mHC baselines require $1.50\times$ and $1.19\times$ as much training compute, respectively.
  • Increasing $N$ from 4 to 16 improves loss by 0.012 in xHC with only 4% additional training FLOPs. The same expansion in mHC improves loss by only 0.006 while increasing training FLOPs by 32%.
  • xHC remains effective under Muon, indicating that its gains are not specific to AdamW.

Memory Efficiency

Estimated per-token memory traffic, amortized per Transformer sublayer. Here, $C$ is the hidden dimension, and validation loss is measured in the 10B ablation setting.

Method I/O per sublayer Validation loss
mHC ($N=4$) $34C$ 2.004
xHC $73.5C$ 1.983
xHC-Flash $51C$ 1.983
xHC-Flash-4sub $40C$ 1.984

xHC-Flash-4sub approaches the memory traffic of mHC at $N=4$ while retaining a clear performance advantage.

Practical Runtime

Our implementation uses fused forward and backward kernels to reduce intermediate tensor materialization and kernel-launch overhead.

  • On the 18B MoE model, our fused mHC implementation adds approximately 15% training overhead over the vanilla baseline, while xHC-Flash-4sub adds approximately 11% on top of mHC. Pipeline-communication overlap is disabled in these measurements to isolate the computational overhead of residual-stream operations.
  • For inference prefill at 2K tokens, mHC and xHC-Flash-4sub add 11.4% and 12.9% overhead over the vanilla baseline, respectively. The total prefill latency of xHC-Flash-4sub is only 1.3% higher than that of mHC.
Default xHC configuration
Setting Value
Expansion streams $N=16$
Active streams $k=4$
Fixed / routed active streams 2 / 2
Temporal augmentation MLP side only
Causal convolution kernels ${4,8,12}$
Residual mixing Sinkhorn-normalized over active streams

Training

Training code is in Megatron-LM/, configurations in configs/, and launch scripts in scripts/. The release includes baseline, mHC, xHC, and efficient full xHC. The efficient xHC backend uses custom Triton kernels to fuse connection operations in the forward and backward passes. xHC-Flash is not included. The supplied recipes use a reference 1,000-step training budget, which may differ from those used in the paper, as the token budgets are not disclosed in the paper. Adjust the training length and learning-rate schedule for your experiments.

Implementation: mHC/xHC, Transformer layer integration, and efficient backend. Triton kernels: fused xHC operations and Gram–Schmidt.

1. Prepare the environment

docker build -t xhc:latest .
docker run --rm -it --gpus all --network host --ipc host \
  -v "$PWD:/workspace" \
  -v /path/to/data:/data:ro \
  -v /path/to/tokenizer:/tokenizer:ro \
  -w /workspace xhc:latest bash

The default Docker build targets Blackwell GPUs. For Hopper, build with --build-arg TORCH_CUDA_ARCH_LIST=9.0 --build-arg NVTE_CUDA_ARCHS=90.

2. Set the data paths

Inside the container:

cp configs/runtime.env.example configs/runtime.env
cp configs/data_mixture.json.example configs/data_mixture.json

Set the indexed .bin/.idx dataset prefixes in configs/data_mixture.json. In configs/runtime.env, set DATA_ARGS=/workspace/configs/data_mixture.json and TOKENIZER_DIR=/tokenizer. Preprocess the data with the same tokenizer used for training. Data and tokenizer files are supplied separately.

Open-source data setup

For public-data experiments, use Dolma3 with the Qwen2 tokenizer. This public dataset differs from the private training data used in the paper. Run inside the training container:

pip install zstandard
python scripts/prepare_dolma3.py --output data/dolma3 --workers 16
export DATA_ARGS="$PWD/data/dolma3/data_args.json"
export TOKENIZER_DIR="$PWD/data/dolma3/tokenizer"

The script downloads the pinned public snapshot (about 111 GB compressed), appends EOS to each document, and builds deterministic train/validation/test indices and the data mixture. Re-running resumes completed groups. Use the same prepared data for all model variants; preparation uses CPUs and needs no GPU.

3. Launch training

Choose one script from scripts/:

Model baseline mHC xHC efficient xHC
2.5B train_2.5B_baseline.sh train_2.5B_mHC.sh train_2.5B_xHC.sh train_2.5B_xHC_efficient.sh
10B train_10B_baseline.sh train_10B_mHC.sh train_10B_xHC.sh train_10B_xHC_efficient.sh
18B train_18B_baseline.sh train_18B_mHC.sh train_18B_xHC.sh train_18B_xHC_efficient.sh
bash scripts/train_10B_xHC_efficient.sh

For 32-GPU training, run the same command on each of four allocated 8-GPU nodes:

export NNODES=4 GPUS=8 NODE_RANK=0  # Set NODE_RANK=0..3 on the respective nodes.
export MASTER_ADDR=192.0.2.10 MASTER_PORT=29500  # Replace with node 0's IP.
export EXP_NAME=10B_xHC_efficient
bash scripts/train_10B_xHC_efficient.sh

Adjust expert parallelism and communication to suit your hardware by setting EP and MOE_DISPATCHER before launching, for example export EP=4 MOE_DISPATCHER=hybridep. Supported backends are alltoall, hybridep, and deepep; the latter two require their corresponding dependencies. If GPU memory is insufficient, reduce the micro-batch size with export MBS=1.

Checkpoints are saved every 500 steps under experiments/<EXP_NAME>/checkpoints/; TensorBoard logs are in experiments/tensorboard/<EXP_NAME>/. The same command resumes the experiment. Use a separate experiment name for each concurrent run.

The efficient xHC implementation improves training efficiency but does not represent the highest achievable efficiency; there is room for further optimization.

Citation

@misc{zhang2026xhcexpandedhyperconnections,
  title         = {xHC: Expanded Hyper-Connections},
  author        = {Xiangdong Zhang and Xiaohan Qin and Sunan Zou and Tuo Dai and
                   Xiaoming Shi and Huaijin Wu and Yebin Yang and Zhuo Xia and
                   Shaofeng Zhang and Lin Yao and Yuliang Liu and Yu Cheng and
                   Junchi Yan},
  year          = {2026},
  eprint        = {2607.14530},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2607.14530}
}

Contact

For questions about the paper, please contact zhangxiangdong@sjtu.edu.cn.

License

Our contributions are licensed under Apache 2.0. Third-party code retains its original licenses; see Megatron-LM/LICENSE and individual file notices.

About

[Tech Report, NeurIPS 2026] Expanded Hyper-Connections

Resources

Stars

78 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages