Skip to content

fix: NVENC and QSV cannot encode 4:2:2 (#74) - #76

Merged
StuartCameronCode merged 2 commits into
mainfrom
fix/nvenc-cannot-encode-422
Aug 19, 2026
Merged

fix: NVENC and QSV cannot encode 4:2:2 (#74)#76
StuartCameronCode merged 2 commits into
mainfrom
fix/nvenc-cannot-encode-422

Conversation

@StuartCameronCode

Copy link
Copy Markdown
Owner

Fixes #74.

The bug

A 4:2:2 source killed every NVENC job on pre-Blackwell hardware — zero frames, YUV422P not supported / No capable devices found. Reported with a CineForm yuv422p10le capture on an RTX 4070 Super at the default "Match source" colour format.

An encoder's declared pix_fmt list is compiled into ffmpeg, but NVENC's real capabilities are queried from the driver at avcodec_open2. A recent ffmpeg built against NVENC SDK 13 advertises yuv422p on h264_nvenc for Blackwell's 4:2:2 support, so negotiation picks it — and every earlier card rejects it. ffmpeg cannot negotiate its way out of this, because the list it negotiates against is wrong for the hardware in the machine.

Same class as the AMF bug fixed under #51, whose reasoning is written verbatim above the AMF arm. NVENC was simply never given the same treatment.

The fix

VideoCodec::forced_pix_fmt now takes the format the pipeline will actually hand the encoder (VideoJob::encoder_input_pix_fmt — the output conversion when one is selected, the pipe format otherwise) and picks an encodable one.

The NVENC/QSV arm is conditional where HuffYUV's and AMF's are not. Pinning NVENC unconditionally would flatten a 10-bit 4:2:0 source to 8-bit for everyone it already serves correctly, so it fires only on 4:2:2, 4:4:4, or >8-bit into an H.264 encoder. A 4:2:0 job emits no -pix_fmt at all, exactly as before. HEVC keeps the depth via p010le; only the chroma has to go.

QSV is included preventively, not on a report — hevc_qsv advertises y210le on builds whose hardware may not have it. VideoToolbox is excluded: it never advertises a mode it lacks.

The substitution is logged, and Custom FFmpeg Arguments still win — the escape hatch for a card that really does have the mode.

User-facing half

  • A 4:2:0 10-bit output format — the only 10-bit layout NVENC, QSV and AMF can encode, so a 10-bit source can keep its grading on a GPU encoder by choice rather than by fallback. Verified end to end: yuv420p10le, profile High 10.
  • A warning under the dropdown naming the substitute and the reason, before the job runs. Covers both routes in (a 4:2:2 source at "Match source", an explicitly chosen 4:2:2 output), silent for everything encodable and until a file is loaded.
  • A second help dialog explaining what the app does to colour — pipe source normalising upward, per-pass depth conversion, UI thresholds rescaled from 8-bit units, dithered output conversion, Y4M stripping SAR and colour tags so they must be re-stamped, and the encoder having the last word.

Risk worth flagging

The warning is a second implementation of the worker's decision. If the two drift the interface promises one thing while the encode does another — worse than either being wrong alone. Both sides are pinned to the same table of cases (substitutions match the worker, case for case against test_nvenc_cannot_be_handed_422 and neighbours), and CLAUDE.md records it as change-one-change-both.

Testing

20 new tests. Neither NVENC nor QSV exists in CI or on macOS, so — as with AMF — this rests on unit tests over the emitted arguments plus reporter confirmation. The functional probe cannot catch this class at all: it encodes one yuv420p frame, so it correctly reports NVENC as available. The device works; only the format doesn't.

The reported case is a test verbatim. Locally: full cargo test green, flutter test --exclude-tags heavy 766 passing, and the new heavy format assertion run against a real encode.

…g pick

A 4:2:2 source killed every NVENC job on pre-Blackwell hardware (#74): zero
frames, "YUV422P not supported / No capable devices found". Reported with a
CineForm yuv422p10le capture on an RTX 4070 Super.

An encoder's declared pix_fmt list is compiled into ffmpeg, but NVENC's real
capabilities are queried from the driver at avcodec_open2. A recent ffmpeg
built against NVENC SDK 13 advertises yuv422p on h264_nvenc for Blackwell's
4:2:2 support, so negotiation picks it for a 4:2:2 source and every earlier
card rejects it. ffmpeg cannot negotiate its way out of this, because the list
it negotiates against is wrong for the hardware in the machine.

forced_pix_fmt therefore takes the format the pipeline will actually hand the
encoder (VideoJob::encoder_input_pix_fmt: the output conversion when one is
selected, the pipe format otherwise) and picks an encodable one.

The NVENC/QSV arm is deliberately conditional where HuffYUV's and AMF's are
not. Pinning NVENC unconditionally the way AMF is pinned would flatten a
10-bit 4:2:0 source to 8-bit for everyone it already serves correctly, so it
fires only on 4:2:2, 4:4:4, or >8-bit into an H.264 encoder (neither family
has a 10-bit H.264 mode). A 4:2:0 job emits no -pix_fmt at all, as before.
HEVC keeps the depth via p010le; only the chroma has to go.

QSV is included preventively rather than on a report — hevc_qsv advertises
y210le on builds whose hardware may not have it. VideoToolbox is excluded: it
never advertises a mode it lacks, so forcing a format would only discard
chroma it could have kept.

The substitution is logged, since it changes the output and should not also be
invisible. Custom FFmpeg Arguments still win, which is the escape hatch for a
card that really does have the mode.

Neither encoder exists in CI or on macOS, so this rests on unit tests over the
emitted arguments. Note the functional probe cannot catch this class at all:
it encodes one yuv420p frame, so it correctly reports NVENC as available — the
device works, only the format doesn't.
The worker half of #74 keeps the job running, but silently changing someone's
output is only acceptable if they can see it coming and choose otherwise.

A 4:2:0 10-bit output format. The only 10-bit layout NVENC, QSV and AMF can
encode, so it is how a 10-bit source keeps its grading through a GPU encoder
by choice rather than by the guard's fallback. Verified end to end: a 10-bit
4:2:2 source comes out yuv420p10le, profile High 10.

A warning under the dropdown naming the substitute and the reason, before the
job runs. It covers both routes into #74 — a 4:2:2 source at "Match source",
and an explicitly chosen 4:2:2 output — and stays silent for anything
encodable, for VideoToolbox and AMF, and until a file is loaded.

It is a second implementation of the worker's decision, and if the two disagree
the interface promises one thing while the encode does another, which is worse
than either being wrong alone. Both sides are pinned to the same table of
cases: "substitutions match the worker, case for case" against
test_nvenc_cannot_be_handed_422 and its neighbours. pixelFormatChromaLayout is
likewise a coarse twin of ChromaClass, following the precedent already set by
pixelFormatBitDepth rather than inventing a new one.

A second help dialog explaining what the app does to colour, as opposed to the
existing one explaining what the formats are and which to pick: the pipe source
normalising upward, filters converting down and back per pass, UI thresholds
being in 8-bit units and rescaled to the clip depth, the output conversion
dithering, the Y4M pipe stripping SAR and colour tags so they must be
re-stamped, and the encoder having the last word. A distinct icon rather than a
second info_outline, asserted, because two identical adjacent buttons read as
one control repeated.
@StuartCameronCode
StuartCameronCode merged commit 29dd1c3 into main Aug 19, 2026
4 checks passed
@StuartCameronCode
StuartCameronCode deleted the fix/nvenc-cannot-encode-422 branch August 19, 2026 00:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NVENC Errors when using YUV422P Video Input File

1 participant