Repository navigation
llama: avoid graph rebuilds for unchanged layer capture - #364
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Repeated calls to
llama_set_embeddings_layer_inpwith the current value unnecessarily invalidate the scheduler, forcing the next decode to reserve its compute graph again. Return early when the capture flag is unchanged; real enable/disable transitions still request a reserve, and an already-pending reserve remains pending.The CUDAFast DFlash2 resident works around this at the caller by tracking whether capture is enabled. This applies the same principle at the library boundary. Source: CUDAFast submission and prefill investigation.
Validation: assertions-enabled CPU build on Apple Silicon. A linked public-API probe compared repeated setters with a reference that only sets on transitions, across eight three-token decodes, two captured layers, and enable/disable/re-enable transitions. The unchanged library performed eight scheduler reservations versus three in the reference; this change performs three. All logits and captured layer inputs were finite and bit-identical. Removing this guard reproduced the extra reservations. This measures eliminated work, not a CUDA or end-to-end throughput gain. CUDA hardware validation remains outstanding.