feat(video): support muxing an audio track in make_mp4_writer - #56
Merged
Conversation
Adds optional audio_path/audio_codec to make_mp4_writer (imageio-ffmpeg forwards them to ffmpeg as a second input) and a write_stereo_wav helper that produces the 16-bit stereo WAV the writer consumes. Groundwork for video models that generate audio alongside frames (e.g. MiniMax H3). Callers must pre-trim audio to the video duration: imageio-ffmpeg passes no -shortest, so the container duration is the max of the streams. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lstein
requested review from
JPPhoto,
Pfannkuchensack and
blessedcoolant
as code owners
August 7, 2026 18:10
Three findings from review of the audio-mux helper, none of which changed behavior for the existing callers: - write_stereo_wav accepted an empty (2, 0) array and wrote a zero-frame WAV. ffmpeg treats that as valid input and mixes it into an MP4 with no audio stream and no diagnostic — the same silent failure the audio_path existence check was added to convert into a named error. Reject it explicitly. - The float -> int16 conversion truncated toward zero rather than rounding (0.99999 encoded as 32766, 1.9 LSB as 1), costing up to a full LSB and biasing the signal toward silence. Round instead. The existing clipping test could not catch this because clip() pins full scale exactly; a separate in-range test now covers it. - test_audio_codec_is_forwarded_not_defaulted assumed the ffmpeg binary has libmp3lame. That holds for imageio-ffmpeg's bundled builds but not necessarily for one supplied via IMAGEIO_FFMPEG_EXE; skip when absent. Verified against the full stack: tests/app/invocations/test_video_concat_audio.py, test_video_frame_extract_range_audio.py and tests/app/util/ all pass at minimax-h3/14-vae-working-memory with this helper change applied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Port of lstein#138 to the InvokeAI-7 tree. Mechanical replay of the original commits onto v7
main(4a48a37967);openapi.json/schema.tsregenerated and verified drift-free at every level, and thedependencies.pyauto-merge verified by hand. All changes target the backend and the existingfrontend/webapp -webv2is untouched (H3 support in the new UI is future work). Review history and test notes live on the original PR.Summary
Groundwork for MiniMax H3 (stacked PR 1 of 5): teaches the shared MP4 writer to mux an audio track, so video models that jointly generate audio (H3 produces 32 kHz stereo alongside frames) can save it inside the video container — no new asset type, service, or dependency needed.
make_mp4_writergains optionalaudio_path/audio_codec="aac"kwargs. imageio 2.37 / imageio-ffmpeg 0.6 already forward these to ffmpeg (-i <audio> -acodec <codec>); AAC-LC is the browser-safe choice for MP4.write_stereo_wavhelper: float PCM(2, n)in [-1, 1] → 16-bit stereo WAV. Conversion runs in float64 (fp16 input would round 1.0·32767 up to 32770 and wrap peaks to -32768); NaN becomes silence; out-of-range clips.audio_pathis existence-checked at construction: ffmpeg spawns lazily on the first frame append, so a missing file otherwise surfaces as an unexplained BrokenPipeError mid-encode — or, for tiny clips, silently produces no output.Two caveats documented in the module docstring for future callers:
-shortest, so container duration = max(stream durations). Callers must pre-trim audio to the video duration.audio_codec="copy"for a PCM WAV — PCM-in-MP4 doesn't play in browsers.Existing callers (
wan_latents_to_video,video_concat,video_frame_extract_range) are unaffected: the no-audio path builds the identical writer call as before.Testing
uv run pytest tests/app/util/test_video_encoding.py— 8 passed. New cases: aac stream present + container duration not stretched; codec forwarding pinned with a non-default codec (mp3) since ffmpeg's MP4 default is already aac; fast-fail on missing audio file; fp16/NaN safety; WAV format/clipping; no-audio output has no audio stream.h264+aac (LC), 32000 Hz, stereo, duration exactly matches frames/fps.🤖 Generated with Claude Code