Repository navigation
llama: support tied Hadamard output weights (contract version 2) - #257
Merged
Merged
Conversation
A Hadamard-folded model with tied embeddings stores its latent token
embedding once and uses it for both row lookup and the output projection.
Until now the packer had to ship a byte-identical copy as output.weight so
the output matmul went through the normal folded-weight path.
This adds GGUF Hadamard contract version 2 with prism.hadamard.tied_output:
- Loader: version 2 requires tied_output=true and no output.weight, and
version 1 forbids the flag. The output projection gets its forward
transform from the embedding's block size and signs, bound to whichever
tensor the output actually uses (a separate allocation when placed on
another device).
- Graph verifier: rejects a latent lookup table consumed by a matmul
without a registered forward transform.
- Converter: accepts manifest schema 3 with tied_output=true, a tied HF
config and no separate head, and writes contract version 2. Schema 3
makes older converters reject the manifest instead of ignoring the flag.
Version-1 files are unchanged, and a runtime without this change refuses
version 2 at load ("unsupported prism.hadamard.version: 2").
Checked on a 2B PQ2_0 band, rewritten to version 2 with the duplicate head
removed (135 MB smaller): 8x512 perplexity chunks give KLD and top-1
agreement against the original at the logit-file floor on both Metal and
CPU (max KLD 5.0e-5 / 4.2e-5, top-1 100%), matching the parent commit's
own result on the original file.
github-actions
Bot
added
documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
GGUF Hadamard contract version 2, with
prism.hadamard.tied_output, so a Hadamard-folded model with tied embeddings can store its latent token embedding once and use it for both lookup and the output projection. Until now a tied pack had to carry a byte-identical copy of the embedding asoutput.weight.tied_output=trueand nooutput.weight; version 1 forbids the flag. The output projection gets its forward transform from the embedding's block size and signs, bound to whichever tensor the output actually uses, which can be a separate allocation on another device.tied_output=true, a tied HF config, no separate head) and writes contract version 2. Schema 3 makes older converters reject the manifest instead of silently ignoring the flag.docs/development/hadamard-tied-output.mddescribes the contract.Why
Our export tooling now omits the duplicate head for tied models and emits schema 3. Without this change those files can't be loaded; with it, the tied 2B band is 135 MB smaller, with identical output.
Compatibility
Version-1 files load exactly as before. A runtime without this change refuses a version-2 file at load with
unsupported prism.hadamard.version: 2, so version-2 files should only be published once a release carries this.How it was checked
A 2B PQ2_0 band rewritten to version 2, with the duplicate
output.weightremoved and data repacked, compared against the original file. 8 x 512-token perplexity chunks, KL divergence against saved logits:-dev none)Both rows sit at the saved-logit floor (the second row is the no-change control), so the tied file matches the duplicate-head file and version-1 behaviour is unchanged. The release build b10685 refuses the version-2 file cleanly. Not checked: CUDA, Vulkan, SYCL and HIP; the 4B; vision.