Ingest decoder prompts in batch-sized chunks - #196
Open
james-333i wants to merge 2 commits into
Open
Conversation
The open-ended dependency range resolves llama.swift to releases wrapping current llama.cpp builds, where the Llama trait no longer compiles: llama_sampler_init_penalties regained its leading n_vocab parameter, and llama_model_params replaced use_mmap and use_mlock with a llama_load_mode enum. Pass the vocabulary size at all three penalties call sites and set load_mode to LLAMA_LOAD_MODE_MMAP, matching the previous mmap-only behavior. Verified against llama.swift 2.10549.0 with the full live test suite.
Prompts longer than the batch capacity (512 tokens by default) threw insufficientMemory before generation started, so multi-turn conversations failed as soon as the rendered chat history crossed the batch size, regardless of how much memory was actually available. Feed decoder-only prompts through llama_decode in batch-sized chunks with absolute positions, requesting logits only for the final token. Generation positions now derive from the full prompt length rather than the last batch's token count. Encoder models keep the single-batch requirement, and a prompt that cannot fit in the context window now fails with a new promptExceedsContextWindow error instead of a misleading memory error. Adds a live test generating from a prompt several times the batch size.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prompts longer than
n_batchfailed to decode, which surfaced as generation errors after a few exchanges of context growth. This feeds the prompt tollama_decodein batch-sized chunks and reports a prompt that exceeds the context window as a typed error.Includes the build-fix commit from #193 as its base.