Add an assistant prefill option to LlamaLanguageModel - #198
Open
james-333i wants to merge 2 commits into
Open
Conversation
The open-ended dependency range resolves llama.swift to releases wrapping current llama.cpp builds, where the Llama trait no longer compiles: llama_sampler_init_penalties regained its leading n_vocab parameter, and llama_model_params replaced use_mmap and use_mlock with a llama_load_mode enum. Pass the vocabulary size at all three penalties call sites and set load_mode to LLAMA_LOAD_MODE_MMAP, matching the previous mmap-only behavior. Verified against llama.swift 2.10549.0 with the full live test suite.
llama_chat_apply_template cannot express chat template kwargs, so there is no way to pass switches like Qwen's enable_thinking. Models that reason by default then spend their entire token budget inside a think block with no way to turn it off. Add assistantPrefill to CustomGenerationOptions. The text is appended after the assistant header of the rendered prompt and the model continues from it, so prefilling an empty think block suppresses reasoning output, matching how llama.cpp itself implements a zero reasoning budget.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
assistantPrefillto the custom generation options, appended after the rendered assistant header so the model continues from supplied text. The motivating case is suppressing default reasoning output on models whose chat template offers no switch for it, such as prefilling an empty think block on Qwen, while leaving thinking available to callers who want it.Includes the build-fix commit from #193 as its base.