fix: prevent AVAssetWriter -16364 (InvalidTimestamp) and harden instant/pause recording paths

[richiemcilroy]

Problem

Two users on 0.5.8 hit the same mid-recording failure:

Task mux-video failed: Video muxer stopped accepting frames at frame 8342:
Failed to encode video frame: WriterFailed/The operation could not be completed
(frame #8339, ts=285.513558066s)

The recording aborts with an error dialog and the display track is unusable (no moov atom is ever written).

Root cause

Uploaded logs contain the full NSError: AVFoundationErrorDomain -11800 with underlying NSOSStatusErrorDomain -16364 — CoreMedia's InvalidTimestamp (duplicate/backwards PTS, reported asynchronously a few appends late). MP4Encoder::queue_video_frame enforced monotonicity on nanosecond-precision Durations while the writer receives microsecond-truncated PTS; two frames inside the same microsecond pass the guard and collapse into duplicate writer PTS. The trigger is a stall-recovery burst (the log shows a ~15s system-wide stall right before failure). Reached by macOS studio recordings with camera active (non-fragmented AVFoundationMp4Muxer), the camera track writer, and camera-only recordings. Not a 0.5.8 regression — the path is byte-identical since 0.4.7x-era code.

Fixes

• enc-avfoundation: quantize video PTS to whole microseconds before the monotonic tie correction — the guard now operates in the units the writer sees; emitted PTS are strictly increasing integral microseconds with non-overlapping extents.
• enc-avfoundation: hold the pending frame across pause() instead of flushing with nominal duration, which put the first post-resume frame inside the flushed sample's extent (the sporadic overlapping-extents writer-failure shape). Stop-while-paused still flushes via finish_start; a container-duration assertion proves the muxed timeline is unchanged.
• enc-avfoundation: shift a held frame's deferred offset when a pause gap is consumed (found by adversarial review): a tie-bumped frame held across a pause carried a stale offset snapshot that, applied on append after resume, overwrote the gap-adjusted timestamp_offset and silently shifted all later timestamps.
• recording (macOS): disk-exhaustion guard for all AVAssetWriter modes, not just instant. Previously a studio or camera recording filling the disk killed the writer on a failed async write (file unrecoverable); now the encoder thread stops while the writer is alive so finish() preserves the recording.
• mediafoundation-ffmpeg (Windows): strictly monotonic muxer PTS in stream ticks. A muxer-path audit found the same unit-mismatch class: MF stamps samples in 100ns ticks, the stream time base is ~333× coarser, and nothing guarded the re-quantization — surfacing as dropped packets in the hardware encoder path. One guard at the writer-visible unit closes it for every MF consumer.
• Diagnosability: QueueFrameError::WriterFailed and all four fatal-message sites now debug-format the NSError, so dialogs and logs carry the code and NSUnderlyingError instead of "The operation could not be completed".

Audits (documented, no action needed)

A full audit of every muxer/encoder path for the unit-mismatch class found all other shipping paths safe: the ffmpeg H264 stack guards in tick space (normalize_input_pts) with warn-and-continue containment; the OOP muxer's guard runs in the same tick unit that crosses the process boundary; all audio encoders funnel through one guarded base; win_segmented{,_camera}.rs contain the defect but are dead code. Error-propagation audit: only the display/screen pipeline is fatal to a recording; mic/camera/system-audio failures degrade.

Test coverage (real encoders, real files, wired into CI)

• enc-avfoundation (macOS CI, real AVAssetWriter): same-microsecond pair bumped apart, same-microsecond bursts survive, pause/resume with resume-tie keeps extents disjoint + container duration intact, stop-while-paused flushes the held frame. Queue calls use a writer-ready retry helper mirroring production so paravirtualized runners (no hardware VideoToolbox) can't flake the suite.
• enc-ffmpeg (all platforms): stall-recovery burst with same-microsecond timestamps, exact duplicate, and backwards blip must encode every frame with strictly monotonic PTS and survive the production remux + decode probe.
• cap-recording lib (all platforms): SharedPauseState excision/no-frame-pause/accumulated-cycles/backwards-resume coverage.
• instant_mode_scenarios (all platforms, newly wired into sync-tests.yml): full-pipeline pause/resume excision on both tracks and stall-burst A/V alignment, ending in validate_instant_recording uploadability. Four rotted tests repaired (segments only cut at keyframes; the harness now marks I-frames at the segment cadence and compares assembled media durations, not manifest estimates).
• Hardware harnesses (real screen/mic on developer Macs): hardware_instant_recording gated to macOS (it broke every non-macOS test build via ungated imports) and extended with a real pause/resume cycle; new hardware_studio_recording exercises the exact field-failure path (non-fragmented AVFoundation display writer) with pause/resume and per-segment validation. sync-tests.yml now compile-checks all test targets so harnesses can't rot again.

CI repairs (sync-tests was failing on every branch, including main's nightlies)

• cap-rendering notch golden tests: the windows-2022 runner image update broke WARP compositing with no repo change (passing Aug 4, failing since Aug 5, blocking every PR). Shape assertions stay at full strength on any adapter that renders (hardware everywhere, Ubuntu's lavapipe); a software adapter that fails render sanity skips loudly.
• Sync matrix: above real-device delivery rates, bounded encoder-overload drops are tolerated but every muxed pts is verified against the nearest sent timestamp (timestamp bugs still fail; runner throughput doesn't). Heavy over-delivery cases get a 0.25s relative tolerance on top of the drift tracker's designed 0.1s wall-clock re-pinning. A per-frame emission-lateness guard skips cases the runner stalled through (>0.1s), the invisible-to-end-lag shape that failed the plain 30fps case with a 0.307s error.

Verification — full board green

Final run: 27 checks, 0 failed — A/V Sync Tests green on macOS, Windows, and Ubuntu (the first fully green sync-tests run on any branch since Aug 4), Clippy -D warnings green on macOS and Windows, Format/Typecheck/Biome/CodeQL green. macOS ran cap-enc-avfoundation 44/44 against the real AVAssetWriter, instant_mode_scenarios 61/61, lib 248/248; the matrix passed with one case skipped by its own pre-existing lag guard. Locally: 41-case matrix green with zero skips, cap-enc-ffmpeg 50/50, cap-recording --lib 237/237, instant_mode_scenarios 61/61. Property check across 200 randomized stall/burst scenarios × 20k frames: pre-fix 243,989 duplicate writer-PTS pairs, post-fix 0.

Follow-up candidates (out of scope)

• Durable salvage of a non-fragmented AVAssetWriter after a non-timestamp writer failure (movieFragmentInterval or segment rotation).
• Video manifest total_duration under-reports a tail ending between keyframes (bookkeeping estimate only; media is complete).
• Dead code removal: win_segmented{,_camera}.rs, WindowsOOPFragmentedM4SMuxer.

  

Greptile Summary

The PR hardens macOS and Windows recording timestamp handling, preserves pending AVFoundation frames across pauses, expands low-disk protection, improves native writer diagnostics, and adds extensive regression and hardware-oriented coverage.

• Quantizes AVFoundation video timestamps before monotonic correction and coordinates deferred offsets across pause gaps.
• Enforces writer-visible monotonic timestamps in the Media Foundation muxer.
• Extends AVFoundation disk-space checks to studio and camera recording paths.
• Adds pause/resume, stall-recovery, container-validity, and CI compile coverage.

Confidence Score: 5/5

The PR appears safe to merge, with no concrete changed-code defect identified.

The timestamp normalization, pause-offset coordination, low-disk shutdown paths, and expanded test coverage are consistent with their production callers and lifecycle behavior.

Important Files Changed

Filename	Overview
crates/enc-avfoundation/src/mp4.rs	Aligns monotonic correction with AVAssetWriter’s microsecond timescale and preserves pending-frame timing across pause/resume without an identified actionable regression.
crates/mediafoundation-ffmpeg/src/h264.rs	Adds stream-tick PTS/DTS normalization for Media Foundation output; current no-B-frame configuration supports the shared timestamp assignment.
crates/recording/src/output_pipeline/macos.rs	Expands periodic disk-space protection and improves NSError diagnostics across AVFoundation screen and camera writers.
crates/recording/src/output_pipeline/core.rs	Adds focused tests for pause excision, repeated cycles, and anomalous resumed timestamps without changing production behavior.
crates/enc-ffmpeg/src/mux/segmented_stream.rs	Adds end-to-end regression coverage for duplicate, backward, and same-microsecond recovery-burst timestamps.
crates/recording/tests/instant_mode_scenarios.rs	Adds full-pipeline pause and stall scenarios while repairing segment tests to respect keyframe boundaries and actual assembled-media durations.
.github/workflows/sync-tests.yml	Runs instant-mode scenarios and compile-checks all recording test harnesses across the CI matrix.

_{Reviews (1): Last reviewed commit: "test(recording): add real-hardware studi..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Desktop Recording Capture Pipeline
• Knowledge Base — Export and Rendering Pipeline

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.