Skip to content
dev-bozPublic

About

Make MoE experts from different models talk to each other: a learned translator reads one frozen MoE's expert activations and feeds them to another. Toy-scale experiments, leak-audited.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Repository files navigation

BLaMMoE

Bridged by Latent Attention Model Mixture of Experts

Make MoE experts from different models talk to each other.

Take two mixture-of-experts language models that were trained separately and work fine on their own. Freeze them both. Then train a translator that reads one model's expert activations as it runs and feeds what it reads into the other model, both ways, a token at a time. If it works, each model can draw on the other's experts where its own are weak, and two existing models become a different one without retraining either.

What I'm after:

  • Off-the-shelf models stay as they are, and each still works fine alone.
  • The translator allows, and ideally encourages, free expert-to-expert transfer in the middle of the models, not just one model handing the other its last layer.
  • The aim is a different model, not necessarily a better one.

So far it's all toy MoEs: a code-heavy model and a prose-heavy one, with the same tokenizer or with different ones. Every trained translator has to pass a float64 leak audit, so neither model can peek at text it hasn't reached yet. Where it stands:

  • Two-way transfer works. Each model takes its weak domain from the other, both at once.
  • Left to itself, the translator uses the latest layer it's offered. With the last layer offered, that turns into a swap of last layers, and each model comes to depend on the other. Without the last layer it gets about half the gain with a small fraction of the dependence. That's the best shape so far.
  • Between models with different tokenizers, only a little gets across. Crossing tokenizers is the main gap.

The next step up is a small real-MoE pilot. docs/HANDOFF.md has the current state and next steps.

Where it started

The first idea was the one at the top: let the native experts inside off-the-shelf MoEs talk to each other through a shared latent bus, so two models' experts become one big addressable pool (say 128 from one and 64 from another, 192 in all), with a learned Latent Attention Model doing the bridging. The original idea doc, with the prior art and the risks as I saw them then, is docs/idea-and-prior-art.md.

After review I narrowed it. The Phase 1 spec asked a sharper question, the Micro-Alignment Problem: can fine-grained MoE experts from independently trained checkpoints be aligned into a shared cross-model latent space without catastrophic interference? It scoped Phase 1 to non-destructive transmission of a single expert's own output between two models, with router training, task gains and more than two models left for later. The toy programme showed that much can be done (an expert's output carried across without breaking either model), and then went back to the broader goal above.

Related work

The closest recent work is Sparks of Weight-Space Programming (Takehi, Teoh, Zhang and Khattab, MIT CSAIL, 2 November 2026). In it a frontier LLM reads a network's raw weights as text and writes new ones. It names what a network was trained on, repairs a broken policy, and edits 0.016% of Qwen3.5-0.8B to lift MATH-500 from 8% to 48%. One of their future directions is the problem BLaMMoE goes after:

True architectural and model merging: Today, merging models relies on averaging weights, which only works for models with identical architectures. An LLM that understands the weight space could instead graft a circuit, such as a reasoning module, from one model into a completely different one, without retraining.

BLaMMoE comes at that from the other side:

Weight-space programming BLaMMoE
What gets read Weights, as text Expert activations, live, a token at a time
Who reads A frontier LLM, prompted A small translator, trained
The models One, edited Two, frozen, run together
How a skill moves Graft it in (future work) Each model draws on the other's experts as they run

They also found that "editing weights could remove an obstacle that hid an ability, but it could not add an ability the model lacked". That's the gap BLaMMoE aims at. In the toy runs a code-heavy model gets prose it doesn't have from a prose-heavy partner, and neither model changes. Across different architectures and tokenizers it's still weak (see "Where it stands" above), so that's the part to watch. I think the two approaches could fit together: a model that reads weights could choose which experts to connect, and a working bridge could show which circuits are worth grafting.

Earlier prior art (Mixture of Thoughts, Symphony-MoE, GraphMoE, vec2vec and others) is in docs/idea-and-prior-art.md.

Repo layout

blammoe/            importable package (only tested modules exist — no empty stubs)
  config.py         experiment config + YAML loader            [built]
  models/registry.py concrete checkpoint specs (VERIFY tags)   [built]
  hooks.py          expert-FFN capture primitive (WP1)         [built]
  align/            Procrustes / lstsq / ridge fitters (WP4)   [built]
  metrics/          CKA + mutual k-NN + residual battery (WP5) [built]
  data/pooling.py   char-span pooling for tokenizer mismatch   [built]
  adapters/         Tx/Rx tanh-gated cross-attention (WP6)     [built]
  toy/              toy MoE pair + transmission POC (WP-T)     [built]
configs/            one YAML per rung
docs/               work plan + design docs (agent hand-off)
papers/             source papers as markdown, local only (see below)
scripts/            one-off utilities (e.g. download_papers.py)
tests/              pytest — every built module has a test here
data/  results/     git-ignored; activations & experiment outputs

Quickstart

The code is device-agnostic: everything runs on CPU, and on CUDA when there is a GPU. The toy runs in the write-ups each take minutes per translator on one data-centre-class GPU. The float64 leak audits run on CPU.

# CPU-only (tests, the CPU diagnostics, the audits)
uv venv --python 3.14 .venv
uv pip install --python .venv --index-url https://download.pytorch.org/whl/cpu torch
uv pip install --python .venv pytest pyyaml numpy
CUDA_VISIBLE_DEVICES="" PYTHONPATH=. .venv/bin/python -m pytest -q

# Full stack (CUDA torch, transformers, datasets)
uv venv && uv pip install -e ".[dev]"

Each results doc ends with a Reproduce section with the exact commands. The toy runs need the source papers in papers/ (see below).

Status

Start with docs/HANDOFF.md: current state, how to reproduce, next steps.

Built and tested (pytest green, 119 tests): config loader, model registry, expert-capture hook, alignment fitters (WP4), metric battery + Rung-1 gate (WP5), char-span pooling (WP2), Tx/Rx gated cross-attention adapters (WP6), device selection, bootstrap statistics, a hardware probe, and a toy end-to-end programme:

  • One-way expert transmission between two different toy MoEs: docs/toy-poc-results.md.
  • Two-way lockstep fusion with translator-chosen experts, a tokenizer-leak audit and leak-proof results: docs/toy-fusion-results.md.
  • Current numbers: the seed sweep (3 cross-family + 3 same-design pairs, CIs, causal expert attribution): docs/toy-sweep-results.md.
  • Complementary models (code-heavy A, prose-heavy B, translator trained on ordinary text only, same-domain control partners): docs/toy-complement-results.md.
  • The allow/force dial, part 1 (a growing share of forced recall on the complementary models): docs/toy-dial-results.md.
  • The dial, part 2 (the partner-win reward, and the force dial with fair answer weighting): docs/toy-dial2-results.md.
  • "Different, not better" and why A never gets prose (CPU diagnostics on the trained translators): docs/toy-different-results.md.
  • A with a prose-heavy char partner (same architecture as A): the expert-bus gap is not the tokenizer: docs/toy-char-pair-results.md.
  • Outside review (Codex gpt-6-astra) of the char-pair result, and the attraction / wrong-partner split of "toward": docs/review-2026-10-31.md.
  • Controls for the char pair (new translator seed, swapped slots, a partner from another init): the one-way pattern holds: docs/toy-char-ctrl-results.md.
  • Which channel carries prose into A: the MoE-sum signal and the residual stream do (2 seeds of 3), the per-expert bus doesn't: docs/toy-char-channel-results.md.
  • Start A's gates open: prose goes into A through the expert bus after all (the one-way pattern was the zero gate start): docs/toy-char-gate-results.md.
  • After the gate start: A reads A''s last layer as a set; without the last link it gets half the prose with a fifth of the dependence; from BPE B it gets only a little: docs/toy-char-gate2-results.md.
  • Both sides open: two-way transfer at once; on links 0-2 it keeps half the gain with little partner dependence; A-B unchanged: docs/toy-char-gate3-results.md.
  • Links 0-2 both open holds at a second translator seed and with other-init partners; with links 0-1 only, A gets no prose; link dropout cuts dependence but doesn't spread A's prose off the latest link: docs/toy-char-gate4-results.md.

The real-model work packages (WP1 loader, WP2 collection, WP3, WP7-WP11) are specified in the work plan and intentionally not stubbed.

Source papers

papers/ holds markdown copies of the 30 arXiv papers and 3 GitHub repos behind the spec (list in docs/papers-index.md). It isn't in the repo, because most of them don't allow redistribution. scripts/download_papers.py fetches them again. The toy models use those papers as their prose corpus, so a re-fetch that converts a PDF slightly differently gives a slightly different corpus, and toy numbers won't match the write-ups exactly.

Licence

MIT, see LICENSE. This covers the code and docs in this repo, not the third-party papers.

About

Make MoE experts from different models talk to each other: a learned translator reads one frozen MoE's expert activations and feeds them to another. Toy-scale experiments, leak-audited.

Topics

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages