Deliver streamed tokens live instead of after generation completes - #189
Closed
james-333i wants to merge 145 commits into
Closed
Deliver streamed tokens live instead of after generation completes#189james-333i wants to merge 145 commits into
james-333i wants to merge 145 commits into
Conversation
* Core ML: apply chat template * Cache compiled models * Update Core ML respond and streamResponse implementations * Require Core ML models to be compiled * Add chat template and tool spec handlers to CoreMLLanguageModel initializer * Add documentation comments to CoreMLLanguageModel * Throw specialized error in CoreMLLanguageModel initializer * Rename variable * Set property type to any Tokenizer * Re-add #if CoreML conditional --------- Co-authored-by: Mattt Zmuda <mattt@me.com>
* Implement guided generation for OllamaLanguageModel * Add test coverage for Ollama guided generation * Incorporate feedback from review
* Restructure usage section of README * Create separate Providers section and reorganize subheadings * Remove now outdated note about Ollama not supporting guided generation * Bump version to 0.7.0
…huggingface#129) * Filter injected tool definitions by includesSchemaInInstructions * Fixed synthesized Foundation Models transcript instructions to include tool definitions * Add test coverage for tool definitions in instructions * Fix race in response streaming tests
…huggingface#126) * Simplify LanguageModelSession state management - Fix race condition when streaming responses. * Replace OSAllocatedUnfairLock with Locked * Re-enable observation for LanguageModelSession.isResponding and transcript * Replace os_unfair_lock and pthread_mutex_t with NSLock
* Move Locked to new Shared subdirectory * Rename Locked.access to Locked.withLock * Add concise documentation comments to Locked * Move StructuredGeneration to Shared directory * Adopt Locked in MLXLanguageModel * Adopt Locked in StructuredGeneration * Don't shadow inFlight in closure * Replace Important with Note * Don't shadow state in closure
…e#125) * Fix format mismatch in OpenAI responses API * Rewrite expressions to improve clarity * Add unit tests for OpenAI tool calling formatting --------- Co-authored-by: Mattt Zmuda <mattt@me.com>
* Expand test coverage for ToolExecutionDelegate * Add tests for Character extensions * Add tests for generated content convertibility * Add tests for dynamic schema generation * Add tests for instructions * Add tests for generation guides * Add tests for feedback * Make defaultInstructionsAndPromptRepresentationsUseJSONString resilient to guardrail violations * Add tests for JSONDecoder extension * Add tests for prompt * Add tests for transcript * Add tests for URLSession extension * Make defaultInstructionsAndPromptRepresentationsUseJSONString resilient to key reordering
* Add UnavailableReason for MLXLanguageModel * Distinguish between not loaded and load failure * Refactor model state / context caching * Use removeFromCache to explicitly unload models when checking availability
… race (huggingface#134) * Serialize Linux URLSession request paths to mitigate _MultiHandle race * Incorporate feedback from review * Replace withLock instance method with top-level withLinuxRequestLock helper * Fix Linux compiler bug around generic returning lock helper * More workarounds for Linux compiler bugs * Incorporate feedback from review * AsyncHTTPClient support for linux (huggingface#143) * deps: add trait-based import of AsyncHTTPClient * feat: implement transparent AsyncHTTPClient wrapper * deps: conditionally include trait in EventSource * Increase HTTP request timeout from 60 to 180 seconds --------- Co-authored-by: Leonhard Solbach <49833472+KotlinFactory@users.noreply.github.com> * Incorporate feedback from review * swift format -i -r . --------- Co-authored-by: Jonas Stoehr <jonass@dev.jsje.de> Co-authored-by: Leonhard Solbach <49833472+KotlinFactory@users.noreply.github.com>
…ce#147) * Implement safe MLX KV reuse with scoped GPU cache eviction * Incorporate feedback from review * Incorporate feedback from review * Incorporate feedback from second round of review
…ce#148) * Update CI workflow to run Linux directly on container * Attempt to resolve hashFiles issue * Add --traits AsyncHTTPClient to test matrix * Resolve warning about try with non-throwing expression * Temporarily disable failing case in CI matrix
* Add additionalContext support to MLXLanguageModel * Fix merge conflict resolution * Incorporate feedback from review * Add userInputProcessing property to MLX custom generation options * Group KV cache generation options into struct * Improve ergnomics of resize processor at call site * Fix compiler errors due to JSONValue ambiguity * Update expectation for MLX image processing test * Incorporate feedback from review --------- Co-authored-by: Mattt Zmuda <mattt@me.com>
* Update README * Switch to Apache 2 license * Update code examples to use from: instead of branch: main
* Bump mlx-swift-lm to 3.x (3.31.3) * Less comments Co-authored-by: Pedro Cuenca <pedro@huggingface.co> --------- Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Temporarily remove Swift 6.1 from the CI matrix. mlx-swift-lm 3.x upgraded Swift version, which fails to compile with strict concurrency for SDKs earlier than 26. An alternative would be to test the "MLX" trait separately on just Xcode 26, and keep the full matrix for the rest of traits. * Use NIOCore to restore Linux compatibility. Graph resolution changed after the bump to mlx-swift-lm 3 and it looks like NIOFoundationCompat is no longer there.
* Restore CI matrix * Temporarily disable cache in CI * Upgrade swift-syntax for mlx-swift-lm 3.31.4 to be resolved
* Add the thinking content type. Sonnet 5 would start a “thinking” content block that was not properly recognized. This adds support for decoding the block. Signed-off-by: Taylor Lineman <git@actuallytaylor.com> * Update Sources/AnyLanguageModel/Models/AnthropicLanguageModel.swift Co-authored-by: Pedro Cuenca <pedro@huggingface.co> --------- Signed-off-by: Taylor Lineman <git@actuallytaylor.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
In preparation for Dependabot
* chore: enable Dependabot weekly GitHub Actions bumps * Monthly updates --------- Co-authored-by: hf-dependantbot-rollout[bot] <285970069+hf-dependantbot-rollout[bot]@users.noreply.github.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Bumps the actions group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [actions/cache](https://github.com/actions/cache). Updates `actions/checkout` from 6.0.3 to 7.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@df4cb1c...9c091bb) Updates `actions/cache` from 5.1.0 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@caa2961...55cc834) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/cache dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The open-ended dependency range resolves llama.swift to releases wrapping current llama.cpp builds, where the Llama trait no longer compiles: llama_sampler_init_penalties regained its leading n_vocab parameter, and llama_model_params replaced use_mmap and use_mlock with a llama_load_mode enum. Pass the vocabulary size at all three penalties call sites and set load_mode to LLAMA_LOAD_MODE_MMAP, matching the previous mmap-only behavior. Verified against llama.swift 2.10549.0 with the full live test suite.
Prompts longer than the batch capacity (512 tokens by default) threw insufficientMemory before generation started, so multi-turn conversations failed as soon as the rendered chat history crossed the batch size, regardless of how much memory was actually available. Feed decoder-only prompts through llama_decode in batch-sized chunks with absolute positions, requesting logits only for the final token. Generation positions now derive from the full prompt length rather than the last batch's token count. Encoder models keep the single-batch requirement, and a prompt that cannot fit in the context window now fails with a new promptExceedsContextWindow error instead of a misleading memory error. Adds a live test generating from a prompt several times the batch size.
streamResponse consumed an inner AsyncThrowingStream whose builder ran the entire generation loop synchronously on the consuming task, so every snapshot buffered and arrived in one burst after generation finished. Yield snapshots directly from the generation loop on the streaming task, and check for task cancellation between tokens so an abandoned stream stops decoding promptly.
james-333i
force-pushed
the
fix/llama-token-streaming
branch
from
August 27, 2026 19:03
9bd082c to
30ce38f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
streamResponseran the whole generation loop before the consuming task received anything, so tokens arrived in one burst at the end. This yields each token as it is sampled.Includes the chunked prompt ingestion commit it builds on, and the build-fix commit from #PENALTIES as its base.