Fix constrained sampling ignoring the allowed-token set - #197
Open
james-333i wants to merge 2 commits into
Open
Conversation
The open-ended dependency range resolves llama.swift to releases wrapping current llama.cpp builds, where the Llama trait no longer compiles: llama_sampler_init_penalties regained its leading n_vocab parameter, and llama_model_params replaced use_mmap and use_mlock with a llama_load_mode enum. Pass the vocabulary size at all three penalties call sites and set load_mode to LLAMA_LOAD_MODE_MMAP, matching the previous mmap-only behavior. Verified against llama.swift 2.10549.0 with the full live test suite.
LlamaTokenBackend.sample(from:) masked disallowed tokens by writing negative infinity into the buffer returned by llama_get_logits, then called llama_sampler_sample. The context can refresh that buffer when the sampler fetches logits, so the mask is dropped and sampling runs over the full vocabulary. Constrained JSON generation then emits tokens outside the allowed set and fails. Build a llama_token_data_array containing only the allowed tokens and their logits, apply the sampler chain to it with llama_sampler_apply, and return the selected candidate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Structured generation masked logits in the buffer returned by
llama_get_logits, but the context refreshes that buffer, so the sampler drew from the full vocabulary and produced invalid JSON. This applies the constraint throughllama_token_data_arrayandllama_sampler_applyso masked tokens cannot be sampled.Includes the build-fix commit from #193 as its base.