Enable Metal GPU offload for llama.cpp models by default - #188
Closed
james-333i wants to merge 144 commits into
Closed
Enable Metal GPU offload for llama.cpp models by default#188james-333i wants to merge 144 commits into
james-333i wants to merge 144 commits into
Conversation
* Core ML: apply chat template * Cache compiled models * Update Core ML respond and streamResponse implementations * Require Core ML models to be compiled * Add chat template and tool spec handlers to CoreMLLanguageModel initializer * Add documentation comments to CoreMLLanguageModel * Throw specialized error in CoreMLLanguageModel initializer * Rename variable * Set property type to any Tokenizer * Re-add #if CoreML conditional --------- Co-authored-by: Mattt Zmuda <mattt@me.com>
* Serialize CoreML tests * Ensure CoreML test model is compiled * Cache loaded model in CoreML tests * Update TokenBackend APIs to be async Update existing call site usage * Implement CoreML guided generation * Add mask caching * Add CoreML guided generation tests * Fix new build failures * Incorporate feedback from review * Downgrade mlx-swift-lm to 2.29.3 See ml-explore/mlx-swift-lm#94 * Disable top-k/top-p/min-p warpers for CoreML guided generation to prevent masked logits from producing invalid integer tokens * Strip the prompt at the token level to avoid issues with normalization or whitespace differences in decoded strings * Throw CoreMLLanguageModelError.structuredStreamingUnsupported instead of fatalError * Strip prompt tokens in streaming response * Remove unused buildEndTokens method * Simplify sample method to avoid per‑step sorting and extra Set copies while keeping the mask cache effective * Relax structured generation test expectations * Remove MaskCacheKey/maskCache and always builds the mask tensor per call to avoid hashing large token sets
* Implement guided generation for OllamaLanguageModel * Add test coverage for Ollama guided generation * Incorporate feedback from review
* Restructure usage section of README * Create separate Providers section and reorganize subheadings * Remove now outdated note about Ollama not supporting guided generation * Bump version to 0.7.0
…huggingface#129) * Filter injected tool definitions by includesSchemaInInstructions * Fixed synthesized Foundation Models transcript instructions to include tool definitions * Add test coverage for tool definitions in instructions * Fix race in response streaming tests
…huggingface#126) * Simplify LanguageModelSession state management - Fix race condition when streaming responses. * Replace OSAllocatedUnfairLock with Locked * Re-enable observation for LanguageModelSession.isResponding and transcript * Replace os_unfair_lock and pthread_mutex_t with NSLock
* Move Locked to new Shared subdirectory * Rename Locked.access to Locked.withLock * Add concise documentation comments to Locked * Move StructuredGeneration to Shared directory * Adopt Locked in MLXLanguageModel * Adopt Locked in StructuredGeneration * Don't shadow inFlight in closure * Replace Important with Note * Don't shadow state in closure
…e#125) * Fix format mismatch in OpenAI responses API * Rewrite expressions to improve clarity * Add unit tests for OpenAI tool calling formatting --------- Co-authored-by: Mattt Zmuda <mattt@me.com>
* Expand test coverage for ToolExecutionDelegate * Add tests for Character extensions * Add tests for generated content convertibility * Add tests for dynamic schema generation * Add tests for instructions * Add tests for generation guides * Add tests for feedback * Make defaultInstructionsAndPromptRepresentationsUseJSONString resilient to guardrail violations * Add tests for JSONDecoder extension * Add tests for prompt * Add tests for transcript * Add tests for URLSession extension * Make defaultInstructionsAndPromptRepresentationsUseJSONString resilient to key reordering
* Add UnavailableReason for MLXLanguageModel * Distinguish between not loaded and load failure * Refactor model state / context caching * Use removeFromCache to explicitly unload models when checking availability
… race (huggingface#134) * Serialize Linux URLSession request paths to mitigate _MultiHandle race * Incorporate feedback from review * Replace withLock instance method with top-level withLinuxRequestLock helper * Fix Linux compiler bug around generic returning lock helper * More workarounds for Linux compiler bugs * Incorporate feedback from review * AsyncHTTPClient support for linux (huggingface#143) * deps: add trait-based import of AsyncHTTPClient * feat: implement transparent AsyncHTTPClient wrapper * deps: conditionally include trait in EventSource * Increase HTTP request timeout from 60 to 180 seconds --------- Co-authored-by: Leonhard Solbach <49833472+KotlinFactory@users.noreply.github.com> * Incorporate feedback from review * swift format -i -r . --------- Co-authored-by: Jonas Stoehr <jonass@dev.jsje.de> Co-authored-by: Leonhard Solbach <49833472+KotlinFactory@users.noreply.github.com>
…ce#147) * Implement safe MLX KV reuse with scoped GPU cache eviction * Incorporate feedback from review * Incorporate feedback from review * Incorporate feedback from second round of review
…ce#148) * Update CI workflow to run Linux directly on container * Attempt to resolve hashFiles issue * Add --traits AsyncHTTPClient to test matrix * Resolve warning about try with non-throwing expression * Temporarily disable failing case in CI matrix
* Add additionalContext support to MLXLanguageModel * Fix merge conflict resolution * Incorporate feedback from review * Add userInputProcessing property to MLX custom generation options * Group KV cache generation options into struct * Improve ergnomics of resize processor at call site * Fix compiler errors due to JSONValue ambiguity * Update expectation for MLX image processing test * Incorporate feedback from review --------- Co-authored-by: Mattt Zmuda <mattt@me.com>
* Update README * Switch to Apache 2 license * Update code examples to use from: instead of branch: main
* Bump mlx-swift-lm to 3.x (3.31.3) * Less comments Co-authored-by: Pedro Cuenca <pedro@huggingface.co> --------- Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Temporarily remove Swift 6.1 from the CI matrix. mlx-swift-lm 3.x upgraded Swift version, which fails to compile with strict concurrency for SDKs earlier than 26. An alternative would be to test the "MLX" trait separately on just Xcode 26, and keep the full matrix for the rest of traits. * Use NIOCore to restore Linux compatibility. Graph resolution changed after the bump to mlx-swift-lm 3 and it looks like NIOFoundationCompat is no longer there.
* Restore CI matrix * Temporarily disable cache in CI * Upgrade swift-syntax for mlx-swift-lm 3.31.4 to be resolved
* Add the thinking content type. Sonnet 5 would start a “thinking” content block that was not properly recognized. This adds support for decoding the block. Signed-off-by: Taylor Lineman <git@actuallytaylor.com> * Update Sources/AnyLanguageModel/Models/AnthropicLanguageModel.swift Co-authored-by: Pedro Cuenca <pedro@huggingface.co> --------- Signed-off-by: Taylor Lineman <git@actuallytaylor.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
In preparation for Dependabot
* chore: enable Dependabot weekly GitHub Actions bumps * Monthly updates --------- Co-authored-by: hf-dependantbot-rollout[bot] <285970069+hf-dependantbot-rollout[bot]@users.noreply.github.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Bumps the actions group with 2 updates: [actions/checkout](https://github.com/actions/checkout) and [actions/cache](https://github.com/actions/cache). Updates `actions/checkout` from 6.0.3 to 7.0.0 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@df4cb1c...9c091bb) Updates `actions/cache` from 5.1.0 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@caa2961...55cc834) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/cache dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
The open-ended dependency range resolves llama.swift to releases wrapping current llama.cpp builds, where the Llama trait no longer compiles: llama_sampler_init_penalties regained its leading n_vocab parameter, and llama_model_params replaced use_mmap and use_mlock with a llama_load_mode enum. Pass the vocabulary size at all three penalties call sites and set load_mode to LLAMA_LOAD_MODE_MMAP, matching the previous mmap-only behavior. Verified against llama.swift 2.10549.0 with the full live test suite.
LlamaLanguageModel forced n_gpu_layers = 0 with a note about Metal GPU issues. That predates the current packaging: llama.swift now wraps the official llama.cpp XCFramework, which is built with GGML_METAL=ON and GGML_METAL_EMBED_LIBRARY=ON, so the Metal backend and its shader library ship inside the binary. With full offload, the live test suite passes and runs the suite about 4x faster on Apple silicon (16.1s to 4.1s for 25 tests with a 135M Q8_0 model), with about 20x less CPU time. Add a gpuLayers parameter to the initializer, defaulting to all layers on real hardware and to CPU-only in the simulator, where Metal execution remains unreliable for llama.cpp. Pass 0 to keep the previous CPU-only behavior.
james-333i
force-pushed
the
feat/llama-gpu-offload
branch
from
August 27, 2026 19:03
5d48e68 to
7622737
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
n_gpu_layerswas fixed at zero, so every GGUF model ran on CPU even on devices with a capable GPU. This offloads all layers by default on device and keeps CPU execution on the simulator, and it adds agpuLayersinitializer parameter for callers that want explicit control. Verified on an iPhone 17 Pro with full layer assignment to Metal.Includes the build-fix commit from #PENALTIES as its base.