Support current GenAI composite model exports - #2663
Open
Tianlei Wu (tianleiwu) wants to merge 1 commit into
Open
Tianlei Wu (tianleiwu) wants to merge 1 commit into
Tianlei Wu (tianleiwu) wants to merge 1 commit into
Conversation
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical external-data handling issues remain unresolved, along with a requested regression-test gap.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
Updates ONNX Runtime GenAI exports for current composite models and external data.
Changes:
- Handles removed legacy GenAI modules.
- Copies and remaps multiple external-data files.
- Adds regression tests for module handling and external-data behavior.
File summaries
| File | Summary |
|---|---|
test/passes/onnx/test_model_builder.py |
Tests legacy module handling. |
test/passes/onnx/test_common.py |
Tests multi-file external data and hardlink behavior. |
olive/passes/onnx/model_builder.py |
Handles absent legacy GenAI modules. |
olive/passes/onnx/common.py |
Materializes and remaps external-data files; critical collision and hardlink issues remain. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+468
to
+469
| if new_external_file_name in used_names: | ||
| new_external_file_name = f"{new_model_path.stem}.{index}.{new_external_file_name}" |
| new_external_file_name = f"{new_model_path.name}.data" | ||
| # copy the external data file to the new location | ||
| hardlink_copy_file(external_file_path, new_model_path.parent / new_external_file_name) | ||
| shutil.copy2(external_file_path, new_model_path.parent / new_external_file_name) |
Comment on lines
+500
to
+501
| if exc.name != "onnxruntime_genai.models.quantized_model": | ||
| raise |
This was referenced Sep 11, 2026
Baiju Meswani (baijumeswani)
pushed a commit
to microsoft/onnxruntime-genai
that referenced
this pull request
Sep 12, 2026
## Summary Adds end-to-end builder and runtime support for INT4 paged KV caches and INT4 DFlash2 export. A Qwen3.8-27B model can now be exported as a directly servable INT4 target with per-channel INT4 KV and a quantized DFlash2 drafter, without editing `genai_config.json` or post-processing the ONNX graphs. ## Changes ### INT4 paged KV cache - Adds `int4_per_tensor` and `int4_per_channel` static-scale schemes, plus CUDA `int4_per_token` dynamic scaling. - Stores two INT4 values per byte and carries bit width, scale granularity, and quantization metadata through builder output and runtime cache allocation. - Supports dense and sparse hybrid layer IDs, scalar or per-channel calibrated scales, and per-token FP16 scale caches. - Extends synthetic model generation plus C++ and Python coverage for INT4 paged attention. ### Directly servable exports - Emits paged-engine settings including `max_scheduled_tokens`, `num_blocks`, and `gpu_utilization_factor` directly into `genai_config.json`. - Supports `exclude_mtp` and `use_device_allocator_for_initializers` and emits `ep.cuda.fpa_intb_gemm=1` for prepacked exports. - Rejects incompatible or no-op prepack configurations instead of advertising an optimization that was not applied. - Keeps Qwen GatedDeltaNet gate projections out of INT4 quantization and fixes sparse KV scale-layer mapping. ### INT4 DFlash2 - Adds `dflash2_precision=int4` to quantize the drafter body and LM head. - Keeps the BF16 drafter body in plain blockwise INT4 because the fpA-intB backend requires FP16 activations; the FP16 drafter LM head remains prepack eligible. - Preserves target/drafter shared-initializer metadata and configures block drafting for greedy generation by default. ## End-to-end validation Built Qwen3.8-27B from its Hugging Face target and DFlash2 checkpoints with an INT4 target, per-channel INT4 paged KV, and a seven-token INT4 drafter. The Olive-produced target and drafter payloads are byte-identical to a direct GenAI builder reference, and the final materialized artifact reloads both ONNX graphs with external data. | Prompt tokens | Batch | Drafts | TTFT (ms) | Target forward (ms) | Throughput (tok/s) | Peak GPU (MiB) | Acceptance | |---:|---:|---:|---:|---:|---:|---:|---:| | 512 | 1 | 7 | 368.66 | 25.594 | 325.66 | 26,540.875 | 99.72% | | 8,192 | 4 | 7 | 2,360.90 | 42.806 | 378.28 | 28,278.875 | 94.52% | | 32,768 | 4 | 7 | 8,812.43 | 123.894 | 209.97 | 29,302.875 | 100.00% | | 32,768 | 4 | 0 | 8,520.84 | 39.095 | 107.00 | 25,516.938 | N/A | At 32K prompt and batch 4, DFlash2 provides 1.962x throughput for an additional 3,785.94 MiB of peak GPU memory. | Evaluation | Direct reference | This export | Changed generations | McNemar p-value | |---|---:|---:|---:|---:| | MMLU-Pro | 663/800 (82.875%) | 663/800 (82.875%) | 0/800 | 1.0 | | GPQA Diamond | 146/198 (73.737%) | 146/198 (73.737%) | 0/198 | 1.0 | Both evaluations used fixed four-shard routing and same-process direct-reference repeats. DFlash2 recorded zero failures, disables, admission misses, and standard fallback steps. ## Tests - `test/python/builder`: 667 passed, 4 skipped, 790 deselected - Focused INT4 KV and DFlash2 suite: 122 passed - Quantized KV suite after formatting: 75 passed - clang-format 20 dry run passes for all affected C++ files - Exact `og.Engine` load and generation validated with ORT 1.30.0 and the CUDA INT4 KV runtime enabled ## Downstream validation - microsoft/Olive#2663 adds modern GenAI compatibility and preserves multi-file external-data artifacts during materialization. - microsoft/olive-recipes#615 publishes the single verified optimized recipe and its latency, memory, MMLU-Pro, and GPQA results. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
test_resave_model_preserves_multiple_external_files: 1 passedtest_model_builder_skips_removed_legacy_quantized_model: 1 passedonnx.loadModuleNotFoundErrorstill propagatesCPY001header findings) and Ruff format check passThe focused tests use
--confcutdir=test/passes/onnxin the export environment to avoid the repository's unrelated optional top-levelpeftfixture.