Enable Metal GPU offload for llama.cpp models by default - #194
Open
james-333i wants to merge 2 commits into
Open
Enable Metal GPU offload for llama.cpp models by default#194james-333i wants to merge 2 commits into
james-333i wants to merge 2 commits into
Conversation
The open-ended dependency range resolves llama.swift to releases wrapping current llama.cpp builds, where the Llama trait no longer compiles: llama_sampler_init_penalties regained its leading n_vocab parameter, and llama_model_params replaced use_mmap and use_mlock with a llama_load_mode enum. Pass the vocabulary size at all three penalties call sites and set load_mode to LLAMA_LOAD_MODE_MMAP, matching the previous mmap-only behavior. Verified against llama.swift 2.10549.0 with the full live test suite.
LlamaLanguageModel forced n_gpu_layers = 0 with a note about Metal GPU issues. That predates the current packaging: llama.swift now wraps the official llama.cpp XCFramework, which is built with GGML_METAL=ON and GGML_METAL_EMBED_LIBRARY=ON, so the Metal backend and its shader library ship inside the binary. With full offload, the live test suite passes and runs the suite about 4x faster on Apple silicon (16.1s to 4.1s for 25 tests with a 135M Q8_0 model), with about 20x less CPU time. Add a gpuLayers parameter to the initializer, defaulting to all layers on real hardware and to CPU-only in the simulator, where Metal execution remains unreliable for llama.cpp. Pass 0 to keep the previous CPU-only behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
n_gpu_layerswas fixed at zero, so every GGUF model ran on CPU even on devices with a capable GPU. This offloads all layers by default on device and keeps CPU execution on the simulator, and it adds agpuLayersinitializer parameter for callers that want explicit control. Verified on an iPhone 17 Pro with full layer assignment to Metal.Includes the build-fix commit from #193 as its base.