feat(speech): 1:1 BaoCut speech pipeline replication - forced alignment, VAD, diarization, LLM repair loops (macOS phase)

[Rog3rLuo]

Problem Statement

Word-level caption timing in OpenScreen currently comes from whisper.cpp's DTW token timestamps, which drift ±80–150 ms. The karaoke caption style built on top of it therefore highlights characters noticeably off-beat, there is no speaker awareness, and there are no quality gates over the AI stages. The reference product in this space — BaoCut (macOS) — implements the mature route: Silero VAD → ASR → dedicated forced aligner → speaker diarization → validation-gated LLM loops, with a machine-readable CLI contract and a quality-gate system. A full reverse-engineering pass (binary statics + dynamic CLI runs + open-doc analysis, see Further Notes) verified every model in that route is open-source and already cached locally. Today our users get approximate karaoke and no speakers; they should get reference-grade word timing, speaker awareness and quality gates.

Solution

Replicate BaoCut's speech-pipeline behavior, 1:1, on macOS first, using its exact open-source models and its verified protocols:

Silero VAD → ASR (whisper.cpp kept for text; MOSS-Transcribe-Diarize ported as the integrated transcribe+diarize engine) → Qwen3-ForcedAligner for word timestamps (DTW demoted to fallback) → Pyannote segmentation + WeSpeaker voiceprint clustering for speaker labels → LLM stages (polish / translate+align / refine) driven by BaoCut's agent task protocol (file-v1) with deterministic validation and keep-previous-on-failure semantics → a four-level quality gate (check) with named, repairable findings.

Acceptance is behavioral equivalence against golden reference outputs produced by BaoCut's own CLI — and the golden material already exists: BaoCut's 1.6 GB model cache and a real test project (RE-Test.bcut, bcutTranscript 0.4 with words[]{id,t0,t1,text,sp} and speakers{s1:{name,hue}}) are on this machine and were verified first-hand. Later phases adapt/extend to other models and platforms; this spec covers the macOS phase only.

User Stories

1. As a video creator, I want word timestamps accurate to ≈80 ms from a dedicated forced aligner, so that the karaoke highlight lands on the syllable being spoken instead of drifting.
2. As a video creator, I want whisper's DTW timestamps to remain as an automatic fallback, so that a failed or unavailable aligner never leaves me without captions.
3. As a video creator, I want Silero VAD to split my recording into clean speech regions before recognition, so that chunk boundaries never cut words and timings stay monotonic.
4. As a video creator, I want speaker labels attached to word atoms (sp: "s1") and a speaker registry (speakers: {s1: {name, hue}}), so that multi-person recordings become structured transcripts with stable per-speaker colors.
5. As a video creator, I want voiceprint clustering (Pyannote segmentation + WeSpeaker embeddings) with an optional expected-speaker count, so that I can steer diarization on interviews.
6. As a video creator, I want to disable speaker detection entirely, so that single-person recordings do not pay for it.
7. As a video creator, I want the pipeline to run fully locally with open-weight models, so that my recordings never leave my machine.
8. As a video creator, I want models downloaded from hf/modelscope/cdn mirrors with per-file SHA-256, resumable partials and a dependency-aware bundle (an ASR download pulls its aligner+VAD companions), so that one click gets a complete working engine set.
9. As a video creator, I want the integrated MOSS engine to produce transcription, timing and speaker labels in one pass, so that long recordings are processed without stitching separate stages.
10. As a video creator, I want an LLM polish stage that fixes typos, adds punctuation and re-segments by semantics, with engine-side normalization (ASCII punctuation adjacent to CJK → full-width; CJK/Latin spacing), so that my raw transcript reads like professionally typeset subtitles.
11. As a video creator, I want a bilingual glossary / canonical-terms brief (with locked terms) feeding ASR hotwords and translation, so that product names like "BaoCut" are not misheard as "BAO Cut".
12. As a video creator, I want translated subtitles aligned to the source word stream as monotonic blocks (one-to-one or many-to-one), so that translated karaoke highlights stay truthful to the audio.
13. As a video creator, I want alignment repair proposals validated deterministically before being applied — and the previous alignment kept when validation fails — so that an LLM can never make my captions worse.
14. As a video creator, I want a quality check that returns blockers / warnings / omitted / lossy with named, repairable findings and targeted repair commands, so that I always know whether my project is export-ready.
15. As a video creator, I want stage fingerprints (media hash, per-stage input fingerprints, word-list fingerprint), so that stale AI stages are detected instead of silently reused.
16. As a video creator, I want line-width budgets (CJK ≈16 display cells, Latin ≈42 chars, budget = floor(display-seconds × reading speed)) enforced when translation cuts lines, so that subtitles stay readable at speed.
17. As a caption editor, I want words[] to remain the persistent text/time truth that captions, karaoke, exports and the timeline all read from, so that no two views of my project ever disagree.
18. As a caption editor, I want word-anchored editing (--at-word-style anchors surviving cuts and re-timing), so that my edits survive pipeline re-runs.
19. As a caption editor, I want a punctuation-hiding option, so that CJK subtitles can be shown without terminal punctuation clutter.
20. As a maintainer, I want the aligner/VAD/segmentation engines recorded per transcript as provenance, so that I can tell which engine produced any historical transcript.
21. As a maintainer, I want golden equivalence tests that compare our pipeline against BaoCut's own transcript.json on the same audio, so that "1:1 replication" is a measurable gate, not an opinion.
22. As a maintainer, I want AI stages exposed as inspectable long jobs with JSONL events (progress, batch-dispatch, explicit cancellable model-wait, done) and stdin cancel, so that long runs are observable and cancellable.
23. As a maintainer, I want the LLM stages to work in agent mode — the pipeline writes file-based task packages (file-v1: contracts + payloads) that any agent can claim and answer — so that the editor works without any API key and remains operable by coding agents.
24. As a maintainer, I want hard failure instead of silent fallback (foreign architecture rejected for transcription, missing model offline → immediate error, remote node unreachable → explicit failure), so that degraded results can never masquerade as good ones.
25. As a maintainer, I want a self-describing pipeline contract (machine-readable command/flag/event spec with a version handshake), so that external tooling and the app stay in lockstep with the engine.
26. As a video creator, I want an explicit refine offer after a fast run (refineOffer with expected benefit and cost), so that I decide whether to spend time on refinement rounds — the pipeline never silently escalates.

Implementation Decisions

• Route replication, not code replication. BaoCut's inference runtime is closed-source; its route, models, data structures and CLI contract are fully visible (its own open layer is only docs/skills — the CLI main repo and app are closed). We replicate the route with the same open models; acceptance is behavioral equivalence against golden outputs.
• macOS phase. New engines target macOS only; whisper.cpp + DTW remains the pipeline on other platforms until a later cross-platform phase.
• Model set (verified open-source):
  • Qwen/Qwen3-ForcedAligner-0.6B (Apache-2.0, official): audio + known text → word [t0, t1] in one non-autoregressive pass; 5000-class timestamp head, 80 ms per class. Primary word-timing engine.
  • OpenMOSS-Team/MOSS-Transcribe-Diarize (Apache-2.0): 0.9B Whisper-encoder + Qwen-style decoder; one pass yields transcription, timestamps and [Sxx] speaker labels; hotword-promptable. Integrated engine; ported last (largest item).
  • Silero VAD v6 (MIT); Pyannote segmentation + WeSpeaker ResNet34 embeddings (≈32 MB speaker package) for non-MOSS engines.
  • ASR text for non-MOSS engines remains whisper.cpp (turbo tier). Qwen3-ASR MLX variants are catalogued but not required for the gate.
• Runtime anchor: BaoCut parity first. BaoCut runs all local models through pmetal-mlx-rs-0.25.8 (MLX Rust bindings) + a 102 MB mlx.metallib, plus direct CoreML bindings for the WhisperKit models. Our macOS engines use MLX-via-Rust the same way; an ONNX Runtime export remains the future cross-platform option, not this phase's target. A small spike (already approved) still measures DTW vs aligner timing error on a real recording before full integration.
• Golden capture is unblocked. BaoCut's 1.6 GB model cache and the RE-Test.bcut golden project already exist locally (verified: bcutTranscript 0.4, 19-word words[], speakers{s1:{name,hue}}, stages{asr, asrLayout}). Golden runs need no new downloads; additional clips only extend the fixture set.
• Word/speaker contract (corrected against measured data). Word atoms are {id, t0, t1, text, sp} where sp is the speaker label (e.g. "s1") — not a space flag. Speaker registry entries carry {name, hue} (hue drives per-speaker caption coloring). Ids are segment-scoped (g1.0, g1.1, …). Edge cases from real data must be handled: degenerate first-word spans (t1: 0.01), ASR mishearings mitigated by the hotword/canonical-terms mechanism.
• Agent task protocol (file-v1) is the primary LLM integration — no API key required. The pipeline writes task packages into the project (tasks/t-*/: task.json + contracts/<kind>.md full prompts + payload paths) and pauses pending answers; any agent claims via claim → read contract → write answer → submit --next. Adopted wholesale, including:
  • 17 call kinds (analysis, speaker-repair, polish, polish-retry, punct-repair, seam-repair, segment-repair, segment, segment-index, chapters, chapters-outline, translate-brief, translate, align, align-edges, align-rewrite, cleanup), retry rounds reusing the base kind.
  • Lease semantics: per-call leases (≤900 s), one-worker-one-lease, --next chaining that merges accept + next claim, late-submit first-valid-answer-wins, call-reused replay after producer restart.
  • Lint budget: 3 tries per callId+lease; third changed submission force-passes (engine checks still apply); byte-identical resubmissions are rejected without consuming budget; closed set of lint problem codes (document-wrapped, document-truncated, document-oversize, surface-artifact, missing-id, empty-translation, duplicate-id, unknown-id, frozen-modified, source-drift, glossary-missing, paragraph-move, sentence-oversize, paragraph-oversize, range-invalid, align-edge-ordinal, align-edge-text, align-content-drift, align-over-hard, align-illegal-seam, context-invalid) — anything else is a warning, not a retry.
  • Hedging: when a claim exceeds ~2× the batch median (in-flight last item counts 1×), dispatch a hedge copy; first answer through the full quality gate wins.
  • Worker-pool sizing formulas: translate ceil(words/880) (rows mode) or ceil(words/2200); align adds ≤40 items/page and a 16000-unit deterministic complexity budget; polish ≈2200 core words/page; hard concurrency cap default 3, adaptive 3–16 from median delivered latency; worker tiering — align and its repairs use the high-reasoning tier.
  • Mixed-protocol lesson: some kinds are always the other protocol (polish-retry is JSON even in file-v1 mode); kind filters must not strand such calls.
  • Our existing provider-mode LLM plumbing remains as the alternative execution mode; the two modes produce identical artifacts.
• Quality gate. check returns four arrays — blockers (exit-fail: polish-fallback, source-language-mismatch, target-language-mismatch, translation-placeholder, translation-source-copy, translation-duplicate-collapse), warnings (polish-false-sentence-end, polish-surface-artifact, align-row-deficit with one auto row-repair round capped at 40 rows, partial-translation, align hotspot codes capped at 40/class), omitted, lossy. Each finding carries its targeted repair command; a project must never enter translation with polish-fallback unresolved; lossy exports require explicit user acceptance. The editor gets a quality panel over the same data.
• Data truth boundaries (three layers, mapped onto our architecture). words[] (+ stage fingerprints, trans, transAlign) is the persistent text/time truth and is never hand-edited; the display projection (our derived caption view + settings) is derived, agent/user-editable; page-style edit overlays are CAS writes carrying base values that auto-invalidate when the projection moves. Append-only history (every write path creates a version; restore = new version) replaces destructive undo where pipelines are involved.
• Dual clocks and word anchors. Source-time cuts vs output-time clips/elements stay separate layers with list-read mappings (never derive one clock from the other); visual elements attach to words (--at-word anchors). We already run raw vs output time for speed regions — this decision formalizes the same discipline and adopts BaoCut's pattern of a shared render-contract golden-vector file consumed by every implementation (ours: TS preview, Rust compositor).
• Fingerprints and caches. Media sha256, per-stage input fingerprints, word-list fingerprints decide fresh/stale per stage; exports stamp per-cue baselines so small edits re-render patches (PAD 0.4 s / JOIN 0.8 s merge windows) — applied to our export path as document-revision-scoped caches.
• Line budgets and normalization (concrete rules). CJK/J/K ≈16 display cells per line, Latin ≈42 chars; budget = floor(display-seconds × target-language reading speed) with punctuation reading-free and Latin chars inside CJK ≈ ½ unit. On acceptance, ASCII ,;: adjacent to CJK becomes full-width (numbers/times/URLs exempt) and CJK/Latin boundaries get spacing normalization — downstream consumers see the normalized text byte-for-byte.
• Model management. New models join the existing catalog (pinned repos + per-file SHA-256 + resumable partials + single-flight verify/rename + hf/modelscope/cdn ordering) extended with dependency bundles (an ASR download pulls its aligner + VAD companions — BaoCut's qwen3-asr-0.6b is 1.62 GB on disk, not the nominal 680 MB, precisely because of this) and --offline failing fast with invalid_arg on a missing package. Supply-chain note: BaoCut funnels everything through one personal namespace (aufklarer/) — we prefer official upstream repos and pin converted repos by hash.
• Schema. AxcutDocument bumps to v8: words[] gain sp (speaker) + stable segment-scoped ids; transcripts gain speakers {id: {name, hue}} and engine provenance {aligner, vad, segmentation}; alignment metadata (blocks, confidence, metrics) stored beside translations; stage fingerprints stored per AI stage. Derived caption cue shape is unchanged.
• Render path unchanged. The karaoke spotlight and the three rasterizers already shipped stay as-is; this pipeline feeds them accurate timings and speaker ids. The 16-style registry from BaoCut's bundle (captions/registry.json: layers/channels/tunables + scripts[latin,cjk] + 3 aspect ratios) is recorded as reference for the future style-registry extension.
• Order of delivery. VAD → forced aligner → speakers (Pyannote+WeSpeaker, delivers the speaker feature without MOSS) → MOSS port → agent-task protocol + LLM loops → quality gate command. DTW fallback stays live at every step, so each stage ships value independently.

Testing Decisions

• A good test asserts external behavior only: audio (and options) in → transcript/artifacts out; or task package in → linted answer accepted/rejected per the closed code set. No test reaches into engine internals.
• Golden equivalence harness (the 1:1 gate). A script-level harness — modeled on the existing real round-trip harness for the whisper helper — runs our pipeline and BaoCut's golden transcript.json on the same WAV and compares field-by-field: text exact; t0/t1 within one frame (≈33 ms at 30 fps); sp labels equal up to anonymous-id permutation; engine provenance recorded. Golden fixtures start from the existing RE-Test.bcut project and grow with additional clips; the harness is env-gated (like the existing download smoke test) because it needs local models.
• Primary seam: the main-process speech pipeline boundary. Everything new hangs off the single STT service contract that already backs the renderer's transcription API (the stt:* IPC layer): its result schema is extended with speakers, engine provenance, stage fingerprints and alignment metadata. Renderer, caption derivation and export consume it unchanged — one new seam, not many.
• Agent-protocol seam. The task-queue machinery is tested at its file boundary: fixture task packages → claim/submit round-trips → lint decisions against the closed code set, with the hedging and lease invariants unit-tested (no live LLM).
• Existing pure seams continue unchanged: caption cue derivation and scene description already have speed-drift, style and karaoke-emission pins; they must stay green byte-for-byte while consuming the new fields.
• Engine unit tests mirror the compositor's pattern: pure CPU tests for alignment evaluation, word↔text matching (the strict in-order matcher shared with karaoke), VAD segmentation math, fingerprint computation, and normalization rules (punctuation width, CJK/Latin spacing); platform-pinned so CI (Linux) skips macOS-engine tests instead of failing.
• Prior art: the whisper helper round-trip script (real binary, real WAV, invariant checks), the env-gated download smoke test, vitest suites colocated with source, model-catalog unit tests, and BaoCut's own contract-first pattern (machine-readable spec + golden vectors shared across implementations).

Out of Scope

• Windows/Linux support for the new engines (later phase; whisper.cpp + DTW remains the cross-platform pipeline).
• Replicating BaoCut's BCF project format, timeline renderer, Studio web/WASM preview, overlay templates, animations, waveform/spectrum caches, its cloud worker/agent orchestration, or its 6-way NLE project exports (Premiere/Resolve/FCP/Shotcut/Kdenlive/CapCut).
• OCR screentext extraction, yt-dlp URL import, TTS, audio DSP beyond VAD.
• Automatic real-name speaker identification (LLM proposes; a human confirms).
• Remote/cloud ASR providers and worker pairing (noted as a future option only).
• Telemetry of any kind (BaoCut has optional telemetry endpoints; we have none and add none).
• Traditional→Simplified normalization of transcripts (explicitly deferred by the product owner).

Further Notes

• Evidence base (all first-hand, reproducible): the reverse-engineering workspace at ~/Downloads/BaoCut-逆向拆解工作区/ — REPORT.md (v1.1.4 build 51 summary + feature matrix), analysis/command-surface.md (2022-line machine-readable command contract, 51 commands/189 entries, specVersion 1.36.0), analysis/spec.json (237 KB), analysis/runtime-analysis.md (end-to-end measured runs: 6.2 s audio → 21.09 s transcribe → 19 word atoms → SRT/VTT/JSON/MD exports), analysis/binary-analysis.md, and docs-analysis.md (protocol deep-dive: agent task protocol, truth boundaries, dual clocks, quality-gate encodings). BaoCut is a renamed continuation of VoiceInk (GUI module VoiceInkKit); its open layer is docs/skills only — the engine repo is closed.
• Corrections this spec makes against measured data (superseding earlier string-level inference): Word.sp is the speaker label, not a space flag; the primary runtime is pmetal-mlx-rs (MLX), with CoreML reserved for the WhisperKit model variants; the LLM stages need no API key in agent mode (file-v1 task protocol), provider mode being an alternative; golden capture needs no downloads (1.6 GB cache + golden project already local).
• Known epistemic limits, per the owner's no-guessing rule: BaoCut's internal numeric thresholds that are neither env-configurable nor observable from artifacts (e.g. exact clustering distance beyond SPEAKER_MERGE_THRESHOLD) remain approximations; the behavioral-equivalence gate is the substitute guarantee. BaoCut's own defects are explicitly non-goals: failure half-states without rollback, env checks blocking unrelated commands, infinite no-key waits, inconsistent API styles, 88%-missing English localization, minOS mismatch between GUI and CLI, duplicated dependencies.
• Licensing & product constraints. All referenced models are Apache-2.0 or MIT; redistribution via our catalog complies with the free-forever constraint. Full local stack ≈3–5 GB disk.
• Relation to the existing roadmap: folds in the planned Silero-VAD item (A2), CJK caption-quality items (C2 line budgets / C4 normalization — now with BaoCut's exact rules), and the quality-gate item (D2 — now with BaoCut's four-level encoding). Karaoke itself is shipped and user-accepted; this spec replaces its timing source and adds speakers.
• The owner has approved: use of the local model cache and BaoCut CLI for golden capture, the delivery order above, and the agent-task protocol as the primary LLM integration.