fix: NVENC and QSV cannot encode 4:2:2 (#74) - #76
Merged
Conversation
…g pick A 4:2:2 source killed every NVENC job on pre-Blackwell hardware (#74): zero frames, "YUV422P not supported / No capable devices found". Reported with a CineForm yuv422p10le capture on an RTX 4070 Super. An encoder's declared pix_fmt list is compiled into ffmpeg, but NVENC's real capabilities are queried from the driver at avcodec_open2. A recent ffmpeg built against NVENC SDK 13 advertises yuv422p on h264_nvenc for Blackwell's 4:2:2 support, so negotiation picks it for a 4:2:2 source and every earlier card rejects it. ffmpeg cannot negotiate its way out of this, because the list it negotiates against is wrong for the hardware in the machine. forced_pix_fmt therefore takes the format the pipeline will actually hand the encoder (VideoJob::encoder_input_pix_fmt: the output conversion when one is selected, the pipe format otherwise) and picks an encodable one. The NVENC/QSV arm is deliberately conditional where HuffYUV's and AMF's are not. Pinning NVENC unconditionally the way AMF is pinned would flatten a 10-bit 4:2:0 source to 8-bit for everyone it already serves correctly, so it fires only on 4:2:2, 4:4:4, or >8-bit into an H.264 encoder (neither family has a 10-bit H.264 mode). A 4:2:0 job emits no -pix_fmt at all, as before. HEVC keeps the depth via p010le; only the chroma has to go. QSV is included preventively rather than on a report — hevc_qsv advertises y210le on builds whose hardware may not have it. VideoToolbox is excluded: it never advertises a mode it lacks, so forcing a format would only discard chroma it could have kept. The substitution is logged, since it changes the output and should not also be invisible. Custom FFmpeg Arguments still win, which is the escape hatch for a card that really does have the mode. Neither encoder exists in CI or on macOS, so this rests on unit tests over the emitted arguments. Note the functional probe cannot catch this class at all: it encodes one yuv420p frame, so it correctly reports NVENC as available — the device works, only the format doesn't.
The worker half of #74 keeps the job running, but silently changing someone's output is only acceptable if they can see it coming and choose otherwise. A 4:2:0 10-bit output format. The only 10-bit layout NVENC, QSV and AMF can encode, so it is how a 10-bit source keeps its grading through a GPU encoder by choice rather than by the guard's fallback. Verified end to end: a 10-bit 4:2:2 source comes out yuv420p10le, profile High 10. A warning under the dropdown naming the substitute and the reason, before the job runs. It covers both routes into #74 — a 4:2:2 source at "Match source", and an explicitly chosen 4:2:2 output — and stays silent for anything encodable, for VideoToolbox and AMF, and until a file is loaded. It is a second implementation of the worker's decision, and if the two disagree the interface promises one thing while the encode does another, which is worse than either being wrong alone. Both sides are pinned to the same table of cases: "substitutions match the worker, case for case" against test_nvenc_cannot_be_handed_422 and its neighbours. pixelFormatChromaLayout is likewise a coarse twin of ChromaClass, following the precedent already set by pixelFormatBitDepth rather than inventing a new one. A second help dialog explaining what the app does to colour, as opposed to the existing one explaining what the formats are and which to pick: the pipe source normalising upward, filters converting down and back per pass, UI thresholds being in 8-bit units and rescaled to the clip depth, the output conversion dithering, the Y4M pipe stripping SAR and colour tags so they must be re-stamped, and the encoder having the last word. A distinct icon rather than a second info_outline, asserted, because two identical adjacent buttons read as one control repeated.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #74.
The bug
A 4:2:2 source killed every NVENC job on pre-Blackwell hardware — zero frames,
YUV422P not supported/No capable devices found. Reported with a CineFormyuv422p10lecapture on an RTX 4070 Super at the default "Match source" colour format.An encoder's declared pix_fmt list is compiled into ffmpeg, but NVENC's real capabilities are queried from the driver at
avcodec_open2. A recent ffmpeg built against NVENC SDK 13 advertisesyuv422ponh264_nvencfor Blackwell's 4:2:2 support, so negotiation picks it — and every earlier card rejects it. ffmpeg cannot negotiate its way out of this, because the list it negotiates against is wrong for the hardware in the machine.Same class as the AMF bug fixed under #51, whose reasoning is written verbatim above the AMF arm. NVENC was simply never given the same treatment.
The fix
VideoCodec::forced_pix_fmtnow takes the format the pipeline will actually hand the encoder (VideoJob::encoder_input_pix_fmt— the output conversion when one is selected, the pipe format otherwise) and picks an encodable one.The NVENC/QSV arm is conditional where HuffYUV's and AMF's are not. Pinning NVENC unconditionally would flatten a 10-bit 4:2:0 source to 8-bit for everyone it already serves correctly, so it fires only on 4:2:2, 4:4:4, or >8-bit into an H.264 encoder. A 4:2:0 job emits no
-pix_fmtat all, exactly as before. HEVC keeps the depth viap010le; only the chroma has to go.QSV is included preventively, not on a report —
hevc_qsvadvertisesy210leon builds whose hardware may not have it. VideoToolbox is excluded: it never advertises a mode it lacks.The substitution is logged, and Custom FFmpeg Arguments still win — the escape hatch for a card that really does have the mode.
User-facing half
yuv420p10le, profile High 10.Risk worth flagging
The warning is a second implementation of the worker's decision. If the two drift the interface promises one thing while the encode does another — worse than either being wrong alone. Both sides are pinned to the same table of cases (
substitutions match the worker, case for caseagainsttest_nvenc_cannot_be_handed_422and neighbours), andCLAUDE.mdrecords it as change-one-change-both.Testing
20 new tests. Neither NVENC nor QSV exists in CI or on macOS, so — as with AMF — this rests on unit tests over the emitted arguments plus reporter confirmation. The functional probe cannot catch this class at all: it encodes one
yuv420pframe, so it correctly reports NVENC as available. The device works; only the format doesn't.The reported case is a test verbatim. Locally: full
cargo testgreen,flutter test --exclude-tags heavy766 passing, and the new heavy format assertion run against a real encode.