Add Ideogram support and improve BF16 dequantization handling - #459
Open
molbal wants to merge 49 commits into
Open
Add Ideogram support and improve BF16 dequantization handling#459molbal wants to merge 49 commits into
molbal wants to merge 49 commits into
Conversation
|
Great! I successfully ran it, but my device doesn't support bf16; it gets converted to fp32 computation, which makes it very slow. Can you make it run on my device in fp16? [INFO] got prompt |
Author
|
Hi @yu234567 - try now. It should work better now, can you verify please? |
Thank you so much, it worked! |
|
classifies Qwen3-VL-8B-Instruct q4_0 quant as _k quant and errors out ? is this expected or what quantization are you running for the te |
- Updated IMG_ARCH_LIST to include 'krea2'. - Introduced ModelKrea2 class with architecture details and tensor handling. - Enhanced convert_file function to support quantization types. - Added tools/convert_krea2_gguf.py for batch conversion of Krea-2 models to multiple GGUF quant levels.
…ation support limitations
…tensors and skip 0-dim scalars during GGUF conversion
Author
Fixed that was my bad |
Feature/molbal dynamic gguf
Add DynamicVRAM-aware GGUF loading and node variants. loader.py: attach mmap-backed file slices, preserve custom GGUF quant configs, and switch to dynamic handling (including BF16/quant paths). nodes.py: add DynamicVRAM loaders and GGUFModelPatcherDynamic, plus helpers to load Unet/CLIP via DynamicVRAM. quant_ops.py: implement GGML quantized tensor layout for Comfy's QuantizedTensor system. tools/convert.py: preserve safetensors metadata, introduce keys_noquant, and avoid quantizing specified small tensors. Add documentation pages for the new Dynamic VRAM loader nodes.
Add Dynamic VRAM support for GGUF loaders
## `lcpp.patch` changes (llama.cpp)
All additions follow the existing pattern used for the other image-model architectures (Flux, SD3, Aura, LTXV, HyVid, Wan, HiDream, Cosmos, Lumina2) --five/six standard insertion points plus two architecture-specific refinements (one a required correctness fix, one an optional quality tuning knob).
### Standard architecture registration (5 places in `src/llama.cpp`)
1. **`enum llm_arch`**: add `LLM_ARCH_KREA2,`
2. **`LLM_ARCH_NAMES` map**: add `{ LLM_ARCH_KREA2, "krea2" },`
3. **`LLM_TENSOR_NAMES` map**: add `{ LLM_ARCH_KREA2, {}},`
4. **`llm_load_hparams`**, "disable LLM metadata for image models" switch: add
`case LLM_ARCH_KREA2:` alongside the other image archs (skips LLM-style
hparam parsing, since this isn't a language model).
5. **`llama_model_quantize_internal`**, "rules for image models" section: add
a `LLM_ARCH_KREA2` block excluding the small/non-repeating tensors from
quantization (kept at their original F32/F16 precision):
```cpp
if (model.arch == LLM_ARCH_KREA2) {
image_model = true;
quantize &= name.find("first.") == std::string::npos;
quantize &= name.find("last.") == std::string::npos;
quantize &= name.find("tproj.") == std::string::npos;
quantize &= name.find("tmlp.") == std::string::npos;
quantize &= name.find("txtmlp.") == std::string::npos;
quantize &= name.find("txtfusion.projector.") == std::string::npos;
}
```
### The one architecture-specific fix: `txtfusion.projector.weight`
**Symptom:** ComfyUI failed to load the *quantized* GGUF (worked fine for the
BF16 intermediate file) with:
```
File ".../comfy/model_detection.py", line 917, in detect_unet_config
dit_config["txtlayers"] = state_dict['{}txtfusion.projector.weight'.format(key_prefix)].shape[1]
IndexError: tuple index out of range
```
**Root cause:** `txtfusion.projector.weight` has shape `(1, 12)` in the original checkpoint. When `llama-quantize` loads this into an actual `ggml_tensor` and writes it back out via `gguf_add_tensor()`, the tensor's dimensionality is derived from `ggml_n_dims()`, which scans the `ne[]` array from the highest index downward and drops trailing 1s. Since GGML's `ne` order is the reverse of the original torch shape, `(1, 12)` becomes `ne = [12, 1, 1, 1]` -- and `ne[1] == 1` gets trimmed away, collapsing the
tensor to 1D. The BF16 GGUF written directly by `convert.py`'s Python `gguf` library does *not* have this problem (it just serializes the given shape faithfully); the collapse is specific to the C++ `llama-quantize` step.
This is the same class of bug already handled for other architectures (SD3's `pos_embed`, AuraFlow's `positional_encoding`/`register_tokens`, Wan's `.modulation`/`img_emb.emb_pos`), all of which use a helper added to GGML for exactly this purpose: `gguf_set_tensor_ndim()`.
**Fix**, added to the same tensor-writing loop in `llama_model_quantize_internal` where the other archs' fixes live (right after the `LLM_ARCH_WAN` block):
```cpp
// Krea2's txtfusion.projector.weight has shape (1, 12) -- the leading
// dim of 1 becomes a trailing `ne` entry and gets truncated by
// ggml_n_dims() unless corrected explicitly.
if (model.arch == LLM_ARCH_KREA2) {
const std::string name = ggml_get_name(tensor);
if (name == "txtfusion.projector.weight" && tensor->ne[1] == 1) {
const int n_dim = 2;
gguf_set_tensor_ndim(ctx_outs[i_split], "txtfusion.projector.weight", n_dim);
LLAMA_LOG_INFO("\n%s: Correcting txtfusion.projector.weight shape for Krea2: [key:%s]\n", __func__, tensor->name);
}
}
```
Note the difference from the SD3/Aura/Wan precedents: those check `tensor->ne[2] == 1` and restore `n_dim = 3` because their tensors were originally 3D (e.g. `(1, H, W)`). Krea2's `txtfusion.projector.weight` is only 2D (`(1, 12)`), so the check is `tensor->ne[1] == 1` and the restored dimension is `n_dim = 2`.
**No other Krea2 tensors needed this fix.** All other tensors are either purely 1D, or 2D with no leading-1 dimension (verified against actual shapes pulled from the checkpoint).
### Optional refinement: `to_v` attention quant-type rules
llama.cpp's `img_tensor_get_type()` has a block of special-case rules that give the attention value projection ("to_v") preferential K-quant subtype treatment at certain `ftype`s, already applied to the other image archs (Flux, SD3, etc. -- matched via names like `.to_v.weight`, `.attn.w1v.weight`, `.attn.w2v.weight`). Krea2's equivalent tensor, `.attn.wv.weight`, wasn't originally in that match list. Added:
```cpp
if ( // Rules for to_v attention
(name.find("attn_v.weight") != std::string::npos) ||
(name.find(".to_v.weight") != std::string::npos) ||
(name.find(".v.weight") != std::string::npos) ||
(name.find(".attn.w1v.weight") != std::string::npos) ||
(name.find(".attn.w2v.weight") != std::string::npos) ||
+ (name.find(".attn.wv.weight") != std::string::npos) ||
(name.find("_attn.v_proj.weight") != std::string::npos)
){
```
Unlike the `txtfusion.projector.weight` fix above, this isn't a correctness issue -- without it, Krea2 quantizes and runs fine, `.attn.wv.weight` just gets the generic type-selection path instead of the to_v-specific one. It's a size/quality tuning knob, initially left out to keep the first working version minimal, then added and verified once the base support was confirmed stable.
Updated read_tensors.py to support tensor comparison and improved shape checking.
Enhance tensor reading and comparison functionality
Update tensor handling for Krea2 architecture
Added key_matches function to match tensor names against patterns and validate key patterns for model architectures.
Implement key_matches function for tensor name validation
…ization defaults in conversion script
Add icon.png to the repository and update pyproject.toml's [tool.comfy] Icon entry to point to the raw GitHub URL for the icon (refs/heads/main). This enables ComfyUI to display the package icon.
Introduce TARGET_SIZE-based GGUF quantization and progress reporting. Added planning and conversion logic (tools/convert.py) to select per-tensor quant types to meet a target size, CLI --max-size-mb support, and unit tests for targeted quantization. Propagate progress callbacks into safetensors/pt readers and GGUF loaders (loader.py, nodes.py) and add GGUFLoadProgress to coordinate ComfyUI progress bars for multi-file loads. Make ops compatible with mixed Q8_CR/Q4_0 GGUFs by materializing GGML weights before mixed-precision loading (ops.py). Update README and register a TargetedQuantizationGGUF node; include node.zip and tests.
Introduce inject_qwen3vl_detection_markers in loader.py to inject zero-shaped visual sentinel tensors (handles MiniMax H3 special-case for Qwen3-VL-32B) so detect_te_model() correctly recognizes Qwen3-VL GGUFs. Replace inline detection-injection logic in gguf_clip_loader with a call to the new helper and adjust logging. Update README to list MiniMax H3 and Qwen3-VL-32B GGUF entries. Add tests (tests/test_targeted_quantization.py) to dynamically load the loader and verify the injected marker shapes and model-type detection behavior.
Add device-aware Q8_CR quantization and allow selecting the Q8 baseline for TARGET_SIZE. Highlights: - CLI/node options: --target-size-q8-type (Q8_CR|Q8_0) and --quantization-device (auto|cpu|cuda). Nodes exposed corresponding UI fields and tooltips. - plan_target_size_quantization now supports a selectable Q8 baseline and downgrades core matrices to Q5_0 before Q4_0. - Device resolution and CUDA-aware quantization: resolve_quantization_device, _can_use_cuda_q8_cr, and quantize_int8_convrot now support a device parameter and CPU/CUDA fallback when VRAM is insufficient or OOM occurs. handle_tensors uses these to serialize I8 tensors and scales correctly. - Tests updated/added to cover TARGET_SIZE Q8_0 baseline and Q8_CR device behavior (including CUDA fallback). Adjusted expectations to Q5_0 for intermediate reductions. - Documentation added/updated: docs/k-quant-inference.md, AGENTS.md, .github SKILL, and README clarifications about quantization-device and TARGET_SIZE behavior. - Minor .gitignore/.idea entries added. These changes enable faster native INT8 ConvRot conversion on CUDA when available while providing safe CPU fallbacks and a selectable Q8 baseline for size-targeted quantization.
Add full GGUF LoRA import and fusion support, Gemma 4 tokenizer/SD mapping, and a convert_state_dict refactor. - README: document Gemma 4 GGUFs and LoRA fusion/cache workflow. - loader.py: add GEMMA4_SD_MAP, Gemma4 tokenizer JSON recreation and loader, and Gemma4 loading path adjustments. - lora.py: new module to read GGUF LoRA adapters, validate tensors, convert factors to torch, fuse deltas, and produce cache keys. - nodes.py: add nodes to import GGUF LoRAs and to fuse multiple GGUF LoRAs into Q8_CR caches; helper mapping/resolution functions; register nodes. - tools/convert.py: split conversion logic so an in-memory state_dict can be converted via convert_state_dict; convert_file now delegates. - tests: add unit tests for GGUF LoRA import/fusion and Gemma4 mapping/tokenizer behavior. These changes enable loading standard GGUF adapters, fusing adapter deltas into checkpoints with deterministic cache keys, and improved Gemma4 compatibility.
Add support for fusing direct LoRA factor files (.safetensors) alongside GGUF adapters by introducing load_safetensors_lora and load_lora, and refactor target-building into _build_targets. Enable node and CLI options to accept .safetensors or .gguf LoRA paths and per-adapter strengths (--lora / --lora-strength, lora_paths / lora_strengths). Detect scaled FP8 source tensors, apply their scale before fusion, and emit fused tensors as float16. Add unit tests covering safetensors fusion, FP8 behavior, and converter merging, and update README and validation messages accordingly.
Add end-to-end support for MiniMax H3 video VAE GGUF files: converter, loader node, tests, and docs. tools/convert.py: introduce ModelMinimaxH3VAE, a preserve_nd_shapes flag, and logic to flatten >4D tensors for GGUF storage while recording original shapes. nodes.py: add VAELoaderGGUF node that enforces arch="minimax_h3_vae" and Q8_CR (int8_convrot) requirement, dequantizes GGUF tensors, and builds a static VAE. loader.py: register the new arch in IMG_ARCH_LIST. README: document conversion and loader usage. Tests added to detect and verify conversion/shape and quantization behavior.
Replace gguf-based dtype selection with dtype inferred from torch.Tensor(value).dtype when calling dequantize_tensor, and remove the now-unused gguf import. Update the unit test to import dequantize_tensor, delete tensor_type from the GGMLTensor, materialize it to a torch.Tensor to get its shape, and clean up temporary objects. This avoids depending on gguf quantization enums and handles GGMLTensor instances that may not expose tensor_type.
ChrisColeTech
added a commit
to ChrisColeTech/ComfyUI-GGUF-Loader
that referenced
this pull request
Aug 12, 2026
Cherry-pick production fixes from city96/ComfyUI-GGUF open PRs: - city96#472 dequant device-constant cache (major LTX/sampling speedup) - city96#433 IQ1/IQ2/IQ3 torch dequant (extra TE quants) - city96#470 QK-norm .scale→.weight (silent NaN/black Flux-compat) - city96#467 dequant bare nn.Parameters (LTX learnable_registers) - city96#392 lumina2/zimage pad token shape fix - city96#456/city96#468 GGMLTensor dtype + dequantize() for core cast path - city96#461 WeightAdapter-aware move_patch_to_device - city96#469 force_patch on partial load/unload - city96#440/city96#436 mistral3 TE, city96#438 qwen35, qwen2 allowlist - city96#473 partial: Qwen3-VL deepstack mmproj map for MiniMax H3 TE Skipped mega/draft rewrites (city96#459, full city96#473 LazyGGUFReader, city96#336 Triton). See PR_BACKPORT.md for the full matrix.
Adds a new Fuse LoRAs & Load node that accepts safetensors or GGUF source models, fuses enabled Power LoRA rows into a content-addressed Q8_CR GGUF cache, and returns the loaded MODEL. This includes a compact Power LoRA UI adapted from rgthree-comfy, cache behavior and docs updates, a third-party notices file, and focused tests covering source detection, fixed cache paths, and Q8_CR source rejection.
Remove the Power LoRA frontend, FuseAndLoadQ8CRLoras node, and related web docs/third-party notices. Rework lora.py to: materialize scaled INT8 source weights (including ConvRot unrotation), parse LoKr direct-factor safetensors, map Comfy model LoRA targets, and handle sliced/qkv targets. Update fusion to apply multiple targets, support LoKr (kron) fusion, and warn on missing targets. Call INT8 materialization during conversion and add/adjust tests to cover INT8, LoKr and mapping behavior. Update README to remove obsolete UI docs.
Introduce streaming conversion for .safetensors to reduce peak RAM/VRAM by processing one tensor at a time and staging GGUF payloads to disk. Adds convert_safetensors_streamed and _streamed_safetensors_layout in tools/convert.py, including FP8 dtype handling, per-tensor FP8 scale application, and on-the-fly LoRA fusion using a new fuse_target_entries_into_tensor helper (extracted from lora.py). Expose a streamed BOOLEAN input in nodes.py and a --streamed CLI flag; update README to document streamed mode. Tests added to validate FP8 streamed layout and LoRA merging without loading the full state dict. Also fixes progress reporting offsets during per-tensor conversion. Overall this enables safetensors->GGUF conversion with much lower memory use while preserving FP8 and LoRA semantics.
Introduce show_progress and verbose flags to handle_tensors and use them to gate per-tensor tqdm output. Make per-tensor tqdm optional so streamed conversions can suppress noisy inner progress and logging. In convert_safetensors_streamed, add a single aggregated tqdm (streamed_progress) when no progress_callback is provided, update it per-tensor, and ensure it is closed in finally. Small cosmetic changes: conditional tqdm.write calls and clearer per-tensor logging formatting. This reduces duplicated/verbose output during streamed conversions and lets callers control verbosity and progress display. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Constructing a torch.Tensor from an inference-mode tensor raises "Inference tensors do not track version counter". Read the dtype through the base-class descriptor instead — no copy, no version-counter access. Hit in practice via low_vram_patch_estimate_vram on bias keys when a LoRA patches biases. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a dependency-free local web dashboard to run tools/convert.py. New files: tools/conversion_webui.py (ThreadingHTTPServer job queue, input validation, per-job subprocess, streaming logs, cancellation), tools/conversion_webui.html (single-file UI), DESIGN.md and PRODUCT.md (design & product notes). README.md updated with usage (python tools\conversion_webui.py, --port, --no-browser). Dashboard validates paths, supports quant/target modes, LoRA fusion, streamed safetensors, and runs conversions serially on localhost:8189 by default.
Add support for LTX 2.5 transformer and latent spatial/temporal upscalers. Updates README with conversion/installation notes. Extend loader architecture list to include 'ltxv_upscaler' and add a new ComfyUI node LTXVLatentUpscaleModelLoaderGGUF (nodes.py) that loads upscaler GGUFs and validates metadata. Add Conv3d handling in GGMLOps (ops.py) so bfloat16 Conv3d weights are cast correctly. Introduce ModelLTXVUpsampler in tools/convert.py, include it in arch_list, refine quantization rules to avoid low-bit conv quant for upscalers, and flush streamed GGUF payloads to disk during conversion. Add unit tests for LTX 2.5 detection and conversion and adjust a minimax test dtype mismatch.
Add support for MiniMax Music 3: update README with conversion instructions and required ComfyUI commit; extend loader.py to recognize 'minimax_music3' in image and text architecture lists; add ModelMiniMaxMusic3DiT and ModelMiniMaxMusic3TextEncoder in tools/convert.py (protect convolution/rotary buffers via keys_hiprec and keep text embeddings BF16 via keys_noquant), update arch_list and strip_prefix logic, and prevent quantizing hiprec keys in handle_tensors. Add unit tests in tests/test_targeted_quantization.py to validate detection and Q8_CR conversion behavior.
…mode
Fix GGMLTensor.dtype crash on inference-mode tensors ("Inference tensors do not track version counter")
|
Nice work @molbal! Any particular reason for |
Author
|
Thanks @joeblowma! nodes.zip is created by comfy-cli when I push the nodes to Comfy Registry, but I forgot to remove it. (I have since added a github action to do it, but it seems I forgot to remove the zip and accidentally committed it) |
Author
|
But anyways my fork grew out of proportions and for it to be merged back I would need to create a separate fork without all the changes I did (e.g. new node names to avoid conflict, changed readme, etc) |
Ensure force_patch_weights is set when partially loading/unloading mo…
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This adds support for Ideogram GGUF models.
What Changed
ideogramto the supported image GGUF architectures.Notes
Tested on Windows 11, Python version: 3.12.11 (main, Jul 23 2025, 00:32:20) [MSC v.1944 64 bit (AMD64)] [INFO] Total VRAM 8192 MB, total RAM 48394 MB
[INFO] pytorch version: 2.12.0+cu130
[INFO] Set vram state to: LOW_VRAM
[INFO] Device: cuda:0 NVIDIA GeForce RTX 3080 Laptop GPU
Tested with Q4_0 gguf from https://huggingface.co/leejet/ideogram-4-GGUF
Other GGUF quant types still use the existing dequant paths.