Live on the software path decodes its scrub still from the DVR packet ring - #545
Merged
Merged
Conversation
The software path holds the whole timeshift window as demuxed packets, each with its pts and a keyframe flag, but nothing in the engine has ever turned packets into a picture. This is the first half: which packets a still at a given time needs, as a pure function of the index. Two shapes decide its rules. Packets are stored in decode order, so with B-frames the frame at the target sits behind the first packet that reaches it, which is what the reorder tail pays for. And a live scrub routinely aims a fraction past the newest packet, so a target beyond the end clamps to it instead of answering nil, which would blink the preview out at exactly the edge the viewer sits on most. Bounded in packets and in seconds, so a stream whose keyframes are minutes apart refuses rather than decoding for minutes. Only the window the span can cover is copied out under the lock: a 30 minute ring holds ~150k entries and a held scrub asks for a still every 80 ms. Part of #544. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F
…packet ring (#544) The live scrub preview has always been a SegmentCache feature: it is gated on `nativeVideoSession != nil`, which a software session never has. So every channel the box cannot decode in hardware scrubbed against an empty card, and on an ATSC tuner that is all of them, because MPEG-2 has no hardware decoder on Apple TV. There was no fallback either: a host can pair the VOD arm with a second demuxer, but a live source is forward-only and cannot be seeked twice. What the path did have is the whole timeshift window as demuxed packets, each with its pts and a keyframe flag, and no image consumer anywhere in the engine. `liveScrubThumbnail` grows a second arm that decodes out of it: the same buffer the scrubber seeks within, so a still and a commit cannot name two different moments, and the same `sessionStartPts` conversion the DVR rewind uses. `SoftwareStillExtractor` drives a real `SoftwareVideoDecoder` rather than a minimal one of its own, because the still should be the picture the renderer would show. Broadcast is where that matters: interlaced MPEG-2 at a non-square sample aspect is the normal case on a tuner, and the deinterlace and the SAR resolution behind it are hardened here already. It decodes single-threaded (new `decodesSingleThreaded`, off everywhere else): a still run is one short GOP decoded once, where frame-level threading only adds output delay and a second worker pool. It runs on its own queue, off the demux and feed loops, so a preview frame never costs playback a packet. Measured on the harness, `play --live --sw --dvr-window 120 --host-calls still` against the paced raw-TS origin, three aims per run: before after stills hit 0 of 3 3 of 3 decode MISS in 0 ms 20 to 61 ms picture none the second it was asked for The seed burns its own second into the frame, so the file is the verdict and not the count: asked for 14.85 s it returns the frame marked 14, asked for 25.11 s the one marked 25, while playback stayed at the edge throughout. Paired with `seekback` in one run, the rewind still lands and the run verdicts OK. Part of #544. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F
Two hazards from the first cut, both found reading it back rather than from a failing run, which is why they are worth naming. The collector kept every frame the run decoded so it could pick afterwards, and those buffers come out of the decoder's own pool. A long GOP would therefore hold the pool empty against the next run, and the bound that allows 900 packets is exactly the bound that makes it possible. It now keeps the two candidates it can actually return: the newest frame at or before the target, and the oldest as the fallback the live edge needs. Teardown ran on the main actor while a decode could be in flight on the still queue. The decoder's lock makes that survivable rather than safe, so the close now goes onto the still queue itself and lands after whatever was running. Re-measured unchanged: 3 of 3 hit, 16 to 52 ms, and the frame asked for at 15.09 s still comes back marked 15. Full suite green. Part of #544. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F
…eview findings (#544) The worst one was not a crash, it was the diagnostic log. `flush()` tears down the deinterlace filter graph, and the still extractor flushes before every run, so on interlaced content every single still rebuilt a Metal pipeline and a full-resolution hwframes pool AND emitted an unconditional `[Deinterlace] engaged` line. A held scrub asks about sixteen times a second, and a host's ring buffer is 300 lines, so half a minute of scrubbing would overwrite the entire log a live playback report depends on, on App Store builds. `flush` grows a `resetFilterGraph` parameter and the still path keeps the graph: a run decodes a full GOP and returns the frame at its target, so the filter has context from this position by the time that frame is made. Measured on an interlaced MPEG-2 fixture, three stills: three graph builds before, one per session after. Three more, each with a failure it actually has: - **Superseded requests ran to completion.** The queue is serial and nothing cancelled, so it took work faster than it retired it: the card fell further behind the thumb with every request and kept decoding past the commit. A newest-wins ticket drops a request that was superseded while it waited. - **A truncated window read as the end of the ring.** `stillRunSpan` could not tell "the ring ran out" from "the caller's window ran out", so a long-GOP high-frame-rate channel could get a still from up to a GOP before the time it asked for, presented as the answer. It now takes `indexReachesEnd` and refuses rather than clamping, with two tests on the distinction. - **The image could alias a pool buffer.** At 1:1 the CGImage was handed back as VideoToolbox made it, and VideoToolbox documents it as backed by the pixel buffer, which returns to the decoder's pool straight afterwards. It is always drawn into an owned bitmap now. Plus: the still path allocates through `trackedPacketAlloc`, so it is visible to the leak instrument like every other packet path; the drill's third aim really is the live edge rather than the playhead; and the collector's fallback comment now names the case it actually covers (the eviction race, not the edge). Verified against a broadcast-shaped fixture this time, MPEG-2 704x480 SAR 10:11 interlaced tff with AC-3, which is what a tuner serves: 3 of 3 stills, 37 to 70 ms, returned 320x240 (so the sample aspect is honoured, not stretched 320x218) and showing the second each one asked for. Full suite green on both runners. Part of #544. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The live scrub preview has always been a
SegmentCachefeature. It is gated onnativeVideoSession != nil, which a software session never has, so every channel the box cannot decode in hardware scrubbed against an empty card. On an ATSC tuner that is all of them: MPEG-2 has no hardware decoder on Apple TV, so an HDHomeRun channel lands onSoftwarePlaybackHostevery time. The route is not the cause, the codec is.There was no fallback either. A host can pair the VOD arm with
makeFrameExtractor(), whose second demuxer seeks a file happily, which is why software VOD still shows stills. A live source is forward-only, so that fallback does not exist on live.What the path did have is its whole timeshift window as demuxed packets, each carrying its pts and a keyframe flag, and no image consumer anywhere in the engine.
What this does
liveScrubThumbnailgrows a second arm. It reads the same buffer the scrubber seeks within, through the samesessionStartPtsconversion the DVR rewind uses, so a still and a commit can never name two different moments.PacketRingBuffer.stillRunSpanis the selection, pure and tested: from the newest video keyframe at or before the target forward to the first video packet reaching it, plus a reorder tail. Packets are stored in decode order, so with B-frames the frame at the target sits behind the first packet that reaches it. A target past the newest packet clamps to it rather than answering nil, because a live scrub routinely aims a fraction past the edge and a nil there would blink the card out at exactly the place a live viewer sits. Bounded in packets and in seconds, so a stream whose keyframes are minutes apart refuses instead of decoding for minutes. Only the window the span can cover is copied under the lock: a 30 minute ring holds ~150k entries and a held scrub asks every 80 ms.SoftwareStillExtractordrives a realSoftwareVideoDecoder, not a minimal one of its own, so the still is the picture the renderer would show. Broadcast is where that matters: interlaced MPEG-2 at a non-square sample aspect is the normal case on a tuner, and the deinterlace and SAR resolution behind it are hardened here already. It decodes single-threaded (decodesSingleThreaded, off everywhere else), because a still run is one short GOP decoded once and frame threading only adds output delay and a second worker pool. It runs on its own queue, off the demux and feed loops, so a preview frame never costs playback a packet.aetherctl play --host-calls stillis the harness: three aims per run, each written to a PNG, with hit or MISS and the decode time.Measured
play --live --sw --dvr-window 120 --host-calls stillagainst the paced raw-TS origin:The seed burns its own second into the frame, so the file is the verdict rather than the count: asked for 14.85 s it returns the frame marked
14, asked for 25.11 s the one marked25, while playback stayed at the edge throughout. Paired withseekbackin one run the rewind still lands and the run verdicts OK. Full suite green on both runners (3010 swift-testing, 0 XCTest failures).Host side
None. Sodalite already calls
liveScrubThumbnailon every live scrub tick and takes nil for an answer, so the card fills as soon as the second arm answers.Not covered
Live on the
nativeRemoteHLSbypass (AVPlayer holds those bytes, nothing engine-side to decode) and a VOD source forced to software for being forward-only (the extractor cannot seek it either, and the software VOD spool is a forward FIFO with no history).Part of #544. The reporter's tuner is the first real device this meets.
🤖 Generated with Claude Code
https://claude.ai/code/session_015PM3xUJB6ZQyqnmGK1fp6F