Accept-workflow and honest reporting: adopt what Playwright, VCR and SimpleCov already proved

[pftg]

Evidence from a lightning-demo teardown (.ai/lightning-demos.md) — every mechanism below was executed on a real machine, not read about. Transcripts are in that file.

Two failure moments drove this, both measured on real customers:

• M1 — the miss. A green bar on a deliberately broken page.
• M2 — the accept. The most frequent daily action, and we have no verb for it — grep -rni "def accept|approve|update_baseline" lib/ returns nothing, there are no rake tasks and no exe/.

#255 and #257 fixed the silence (we now warn loudly and the docs are true). This issue is about the shape.

---

1. Write the baseline on a miss — and fail anyway

Playwright's default missing mode, verified live:

Error: A snapshot doesn't exist at /.../homepage-darwin.png, writing actual.
  1 failed

toMatchSnapshot.ts:211 returns a literal false — the test fails while the file is written. Second run: 1 passed.

The user never has to learn how to record; recording happened. They get a red bar plus a file on disk, which is the honest signal: I could not verify this, here is what I saw, look at it and re-run.

This needs a decision, because it changes a default. Our fail_if_new defaults to ENV["CI"], so locally a new screenshot passes. Playwright's position is that a snapshot with no baseline has not been checked, so it is red. I think Playwright is right and both our customer personas were burned exactly here — but it makes "first run always passes" (README Quick Start) no longer true, and every existing user adding a test would see one new red bar. Wants 4-eyes before implementing.

The trap, stated by the researcher: writing-on-missing is safe because the run fails. Adopt the write without keeping the failure and we ship M1 as a feature.

2. rake snap_diff:accept — re-run, don't copy

Three independent tools converged on this shape: Playwright --last-failed -u, BackstopJS backstop approve (45 lines, promotes only failed_diff_*), Lost Pixel's update mode. RSpec's example_status_persistence_file_path is the Ruby-native precedent our users already know.

The property worth stealing: it re-runs the test rather than copying the previous run's artifact, so you cannot promote a baseline that no longer reproduces. Add FILTER= (BackstopJS has --filter) and narrate before acting.

The local equivalent of Percy's "approve" is git add of a file the tool just regenerated and re-verified for you. The verb is missing; the concept already fits.

3. Rewrite both messages (VCR + WebMock + SimpleCov)

VCR's unhandled-request error, produced live, has four properties we have none of: it restates the effective configuration back at you (:record => :none, :match_requests_on), enumerates every escape route including "if you're surprised, here's how to debug", and uses versioned doc links (?v=6-4-0) that cannot point at docs for an API you do not have.

WebMock goes further and emits the fix, pre-filled from this run's real values, rather than describing it.

Design rule, and it is why RECORD_SCREENSHOTS rotted for years: never print a command in an error message that is not generated from live state.

SimpleCov's threshold failure earns every one of its four lines — what it is, what was required, where to look first, and a distinct exit code 2 for "the tool ran fine, your content failed the gate".

4. Labelled, relative, ordered paths

Expected: tests/vis.spec.js-snapshots/homepage-darwin.png
Received: test-results/.../homepage-actual.png
Diff:     test-results/.../homepage-diff.png

Ours prints four unlabelled absolute paths and the reader guesses. Playwright omits Diff: when there is none — the block is honest about what exists.

5. A third state in the summary line

Argos names the no-baseline state orphan — not a pass, not a failure. Ours should count it:

14 screenshots verified, 0 changed, 1 new (not verified)

You cannot read that and believe the page was checked. Highest value-per-byte item here.

---

The anti-lesson, from the market leader

Percy exits 0 when it silently does nothing:

$ npx @percy/cli exec -- echo hi
[percy] Skipping visual tests
[percy] Error: Missing Percy token
[percy] Command "echo hi" exited with status: 0
$ echo $?
0

It prints the word Error, says it is skipping every visual test, and exits successfully. A CI job that loses its token goes green forever. That is our M1 in its purest form, shipped by the biggest name in the category.

Contrast Chromatic, where every route to a green bar without a real comparison has its own named, branch-scoped flag — --auto-accept-changes, --exit-zero-on-changes, --skip. You cannot land there by accident, and a reviewer reading CI config sees it.

Rule: misconfiguration must be a non-zero exit.

---

Cost

Items 1–5 are one config key, one rake task, two rewritten methods and a counter. No subsystem. Target 2.1; items 4 and 5 are cheap enough to consider for 2.0 if the release slips.

Deliberately not stealing: Percy/Chromatic/Argos dashboards (need a service), TurboSnap dependency tracing (four flags to configure one feature), jest-image-snapshot's terminal image embedding (two options and a terminal allow-list for something a labelled path does for everyone), a .snap_diff_todo.yml (our accepted state is already PNGs in git — more reviewable than a YAML exclusion list).

Also worth a follow-up: stale-baseline detection (jest-image-snapshot's OutdatedSnapshotReporter, ~20 lines). First version must report only, never delete — obsolescence is only knowable from a full run, and deleting after a filtered run destroys baselines.

[pftg]

Reversed — the evidence killed the recommendation

A four-judge panel voted 4/4 FLIP WITH CONDITIONS. Community research run in parallel then contradicted it, and the research wins.

We would have been a minority of one. Jest, AVA, Vitest, testthat and jest-image-snapshot all chose CI-only failure. Playwright is the sole always-fail, and it is greenfield with no migration to perform. Our current fail_if_new = !ENV["CI"].nil? is already the majority position.

The screenshot-specific argument is decisive. A locally-generated screenshot baseline is often worthless: Playwright names baselines -darwin.png precisely because fonts and antialiasing differ across OS (playwright#22337). So "fail locally so the developer records a baseline" produces a baseline that fails CI anyway. Every judge reasoned from text-snapshot precedent, where the artifact is portable. Ours is not.

Precedent, from a tool that shipped exactly this three months ago: testthat 3.3.0 (2025-11-13) made new snapshots fail on CI after a four-year deliberation (#1461), and already has an open request to disable it from a Posit maintainer with many visual snapshots (#2320). It also broke a downstream integration that relied on "new screenshot never fails" (#2293).

And write-and-fail interacts badly with retries. playwright#38046: the re-run passes, so any CI with retries: 1 marks it flaky and swallows the signal.

What actually fixes the reported bug

Schloerke's compromise from testthat#2320 — fail only if the project already has ≥1 committed baseline — does not rescue it either. Our persona had committed zero baselines, so every variant says "bootstrapping, pass."

The three green-suites-that-compared-nothing had three different causes, and only one was a missing baseline. The other two — zero system tests ran, and an inherited GIT_DIR redirect — are invisible to any per-assertion rule. The one signal that catches all three is a count:

14 verified, 0 changed, 1 new (not verified)

0 verified is the tell. This is the dissenting argument from the quality-gate judge, and the evidence now supports the dissent over the vote.

Revised scope

1. Do not flip the global default.
2. Ship the summary line — catches all three causes.
3. Fix The new-screenshot error prints a path that will never exist in CI #260 (check_base_screenshot raises before capture_screenshot) — a bug either way.
4. Let explicit config outrank the env sniff — fail_if_new = false must beat CI=true; today the direction is backwards. Precedent: insta#924, and jest#12288 where Jest read argv.ci instead of detected CI state for an entire major version.

~15 lines, no breaking change, instead of a default flip.

Deferred to 2.1, not rejected

VCR-shaped record: modes — :once (default, permissive) / :none (strict) / :all (bulk re-record). Ruby-native and config-shaped, which removes the objection that killed the Playwright-style ladder ("they have a runner, we don't"). It meets the maintainer condition of a 3-for-1 exchange: deletes fail_if_new and pending_if_new, and gives bulk-accept for free — retiring the proposed rake snap_diff:accept permanently. VCR's :once already implements Schloerke's rule, measured live: no cassette → records and passes; cassette exists and something new appears → raises.

[pftg]

Milestoned 2.0.0 — but only the additive half

This issue contains both non-breaking and breaking work. Splitting it here so nobody implements the wrong half.

Ships in 2.0.0 (additive, nothing removed):

• Honest reporting — N verified, N changed, N new (not verified) — done, feat: the summary line counts what was verified, not just what was compared #261
• Labelled failure paths with units and the applied threshold (lane in flight)
• record: modes added alongside the existing settings. fail_if_new, pending_if_new and fail_on_difference all keep working; setting one warns that record: supersedes it, naming 2.1
• git add-based accept documented as the workflow it already is

Deferred to 2.1.0 (breaking):

• Delete fail_if_new, pending_if_new, fail_on_difference

That is the 3-for-1 exchange: three booleans out, one named mode setting in. It cannot happen in 2.0 because ADR-010 says 2.0 deletes nothing.

Already decided, do not reopen without new evidence (see the reversal comment above): the missing-baseline default is not flipped. Community precedent is CI-only failure — Jest, AVA, Vitest, testthat, jest-image-snapshot — and a locally-recorded screenshot baseline is often worthless across OS, which is why Playwright stamps the platform into filenames. record: gives users the strict mode as an explicit choice instead.