Add native VibeVoice ASR support - #730
Open
justinchuby wants to merge 13 commits into
Open
justinchuby wants to merge 13 commits into
justinchuby wants to merge 13 commits into
Conversation
Performance Comparison
|
justinchuby
commented
Sep 4, 2026
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new arch-validation test uses a huggingface_hub.hf_hub_download(local_dir=...) call that can break the same older Hub environments the PR explicitly targets for compatibility.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds first-class support for exporting and validating the original offline VibeVoice-ASR checkpoint as a staged ONNX package (dual cached waveform encoders → connectors → embedding replacement → cached Qwen2 decoder), plus host-side orchestration and ONNX Runtime GenAI advisory artifacts. This extends Mobius’ speech model coverage while keeping ASR routing fail-closed and distinct from VibeVoice TTS / streaming ASR.
Changes:
- Introduces
VibeVoiceASRForConditionalGeneration,VibeVoiceASRConfig, andVibeVoiceASRTaskto export the five-stage offline ASR pipeline. - Adds host contracts (
VibeVoiceASRProcessor,VibeVoiceASRHost) and ORT GenAI export outputs (preprocessor_config.json+ advisory compatibility metadata). - Updates Transformers builder/config loading to pin the ASR revision by default and to dispatch VibeVoice architectures explicitly (fail-closed), with expanded synthetic + arch-validation tests and documentation.
File summaries
| File | Description |
|---|---|
| tests/weight_alignment_test.py | Adds checkpoint-index census + routing coverage for the pinned VibeVoice-ASR tensor set. |
| tests/model_coverage_test.py | Documents why L4/L5 real-weight coverage is not claimed for the large BF16 checkpoint. |
| tests/build_graph/speech_test.py | Registers expected component topology for vibevoice-asr graph builds. |
| tests/arch_validation_test.py | Makes HF config loading tolerant of older Hub installs and adds coverage for that path. |
| tests/_test_configs.py | Adds a tiny VibeVoiceForASRTraining config for fast graph-level validation. |
| src/mobius/tasks/_vibevoice_asr.py | New task exporting the five offline ASR ONNX components and cache ABI. |
| src/mobius/tasks/init.py | Exposes and registers the new vibevoice-asr task. |
| src/mobius/models/vibevoice.py | Extends tokenizer encoder conv stack to support a final-chunk flag for source-equivalent padding. |
| src/mobius/models/vibevoice_asr.py | New staged offline ASR model + weight routing for both original and converted namespaces. |
| src/mobius/models/vibevoice_asr_test.py | Synthetic parity tests vs pinned Transformers VibeVoice-ASR implementation. |
| src/mobius/models/init.py | Exports VibeVoiceASRForConditionalGeneration from the models package. |
| src/mobius/integrations/vibevoice_asr.py | New host-side orchestration: normalization, chunk/cache handling, prompting, diarization parsing. |
| src/mobius/integrations/vibevoice_asr_test.py | Unit tests for processor/host behavior and deterministic orchestration contracts. |
| src/mobius/integrations/transformers/_config_resolver.py | Makes strict Hub error handling tolerant of older huggingface_hub layouts. |
| src/mobius/integrations/transformers/_config_resolver_test.py | Adds test coverage for Hub installs missing the errors module. |
| src/mobius/integrations/transformers/_builder.py | Pins microsoft/VibeVoice-ASR revision by default and enforces explicit VibeVoice architecture dispatch. |
| src/mobius/integrations/transformers/_builder_test.py | Adds fail-closed dispatch tests and validates pinned ASR raw-config build path. |
| src/mobius/integrations/onnx_genai/auto_export.py | Detects VibeVoice-ASR packages and writes processor + advisory runtime contracts. |
| src/mobius/integrations/onnx_genai/auto_export_test.py | Verifies ORT GenAI export emits processor contract + advisory compatibility for ASR. |
| src/mobius/_registry.py | Registers VibeVoiceForASRTraining as a distinct fallback architecture key and pins its test model id/revision. |
| src/mobius/_configs/vibevoice_asr.py | New config extraction for offline ASR (dual tokenizers + Qwen2 decoder) with validation. |
| src/mobius/_configs/init.py | Exposes VibeVoiceASRConfig from the configs package. |
| docs/vibevoice-asr.md | New documentation for staged offline ASR package boundaries, host contract, and evidence levels. |
| docs/model-catalog.md | Adds catalog entry linking VibeVoiceForASRTraining to the offline ASR docs. |
Review details
- Files reviewed: 24/24 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
Add fail-closed routing, staged ASR graphs, host processor contract, and source-backed coverage for microsoft/VibeVoice-ASR. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Preserve final speech frames, require an unambiguous architecture, and build the source-compatible prompt and token layout. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Preserve source prompt branches and clipping behavior while restoring the VibeVoice TTS fixture in weight alignment coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Match source clipping epsilon behavior and retain already aligned ASR parameters for direct package loading. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Carry final-chunk state into each encoder convolution and use JSON-aware diarization extraction for source-style assistant responses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Treat the strict dataclass validation exception as optional so all Transformers builds remain compatible with supported Hub releases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Keep both normal and raw Transformers config loading compatible with older supported huggingface_hub releases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Emit the source tokenizer’s object-reference speech tokens and keep architecture validation compatible with older Hub releases. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Render source-owned Qwen turns directly so compatible base tokenizers cannot inject an extra system prompt. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Preserve terminal-frame semantics for variable-length batches and pin default ASR builds to the validated checkpoint revision. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Rebase the offline ASR support onto the merged realtime VibeVoice work, preserve both routes, publish the ASR contract through generated model documentation, and use the supported Hugging Face cache API in the checkpoint-index test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a361a630-548c-41b4-9e11-43ed5a681542 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-vibevoice-offline-asr
branch
from
September 14, 2026 19:26
f166a78 to
c5c496f
Compare
Replace legacy training-checkpoint support with the official inference-native VibeVoice-ASR-HF release. Route all 901 native tensors without exclusions and reject the legacy repository before any Hub access. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a361a630-548c-41b4-9e11-43ed5a681542 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
changed the title
Require the native VibeVoice ASR architecture declaration before selecting the staged ASR graph. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a361a630-548c-41b4-9e11-43ed5a681542 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds staged offline ASR support exclusively for
microsoft/VibeVoice-ASR-HF@f22241c2062b3b25272bf117397e03d73381037a(vibevoice_asr/VibeVoiceAsrForConditionalGeneration). The legacymicrosoft/VibeVoice-ASRtraining checkpoint is explicitly rejected before download; it is not aliased.\n\nThe five-stage package routes all 901 native checkpoint tensors with no ignored weights. Processor/tokenizer assets and runtime metadata use the native HF repository. L4/L5 real-weight transcription remains unverified due to GPU capacity.