Adopt SimpleCov's run-visibility model: staleness, an env convention, and an off-vs-broken signal

[pftg]

Scheduled for after v2.0 and v2.1. Filed now so the reasoning is not lost.

Why SimpleCov specifically

It is the closest Ruby analogue this project has: a gem that instruments a test run, accumulates results across processes, writes an HTML report, and has to be switchable off. It has solved both of the problems we solved recently — and it solved them years earlier, with far more users exercising the edges.

We have just shipped our own versions of both:

• fix: the HTML report survives Rails' fork-parallel test runs (#258) #266 — fork-parallel report merging. Workers write JSON fragments to a pid-keyed temp directory on Rails' run_cleanup_hook; the parent merges them in Minitest.after_run and removes the directory. Feature detection needed two doors because Bundler.require loads us before ActiveSupport::TestCase exists.
• The summary line is invisible by default — it ships inside the HTML reporter #269 / A disabled screenshot still counts as an assertion #270 — the summary must print whether or not a reporter is registered, and a disabled screenshot must not count as an assertion.

The question is not whether ours works — it is measured and guarded. It is whether SimpleCov already knows something about the edges we have not hit yet.

What to look at

Result merging:

• SimpleCov.use_merging, merge_timeout, and SimpleCov::ResultMerger — how results are keyed, where they are stored, how staleness is decided, and what happens to a fragment from a worker that died.
• How it decides a run is "complete" versus "one process of several". We use the parent's after_run; what does it use, and does it handle runners we do not?
• SimpleCov.at_exit / at_fork and how it hooks parallel runners (parallel_tests, Knapsack, CI shards). Our fix: the HTML report survives Rails' fork-parallel test runs (#258) #266 solves Rails fork-parallelism only — separate-process runners still get last-writer-wins, documented but unfixed.
• Whether merged output can be misleading, and what it does about it.

Enable/disable:

• SimpleCov.start semantics, SIMPLECOV_NO_DEFAULTS, --no-cov-style switches, and what happens when it is loaded but not started.
• minimum_coverage failing with exit code 2 — a distinct code for "the tool ran fine, your content failed the gate", which we noted in research/competitive-landscape.md and have not adopted.
• How it reports "nothing was measured", if at all. Our equivalent is the NOTHING WAS VERIFIED line, and we should know whether they concluded something different.

Triage question this must answer

Does any of it actually impact us, or is it interesting and inapplicable? A well-evidenced "no, our shape is different and here is why" is a perfectly good outcome and should be written down so it is not re-asked. Specifically: SimpleCov merges coverage data (mergeable by construction — union of line hits), while we merge assertion records and image paths. That difference may make most of their design irrelevant, or may be exactly why their staleness handling matters more than ours does.

Method: read the source, not blog posts. Cite file:line. Add findings to research/competitive-landscape.md, which already has SimpleCov entries for its threshold failure and its 🎨 Colorblind report toggle.

[pftg]

Researched. SimpleCov is the right analogue, and the mapping is more useful than expected: we already built the hard half.

What we already have

SimpleCov	snap_diff
.resultset.json per worker	pid-keyed JSON fragments (#266)
SimpleCov.collate merges them	merge_parallel_fragments! in the parent
command_name disambiguates workers	fragments are pid-keyed

Cross-process merging is the part SimpleCov spent years on, and #266 solved it independently for the same reason (Rails' parallelize(workers:) forks, and Minitest skips after_run in children).

What is missing, in order of impact

1. .last_run.json — no staleness signal

SimpleCov writes .last_run.json after every run and the next step can read it. snap_diff persists nothing between runs; only the HTML report is written, and it is overwritten.

For a suite that runs visual checks rarely — which is the whole reason this issue exists — the question is not "did this run pass". It is "how stale is my visual coverage?" Nobody can answer that today. A developer sees green from a smoke tier and has no way to know that the screenshots outside it were last verified three weeks ago.

Proposal: .snap_diff_last_run.json — {verified, changed, new, at, names}. That enables the line nothing can print today:

[snap_diff] 12 screenshots not verified since 2026-08-04.

This also fixes a failure mode in the hand-maintained-tier approach (see below): rename a test and it silently leaves the gate, with nothing reporting the loss. A staleness list surfaces exactly that.

2. No COVERAGE=1-style convention

Disabling requires writing Ruby: SnapDiff.config.enabled = false. SimpleCov's ecosystem converged on COVERAGE=1; we have no idiom, so every team invents its own env var. A documented SNAP_DIFF=0/1 costs one reader and makes every project's setup look the same.

3. minimum_coverage has only a partial analogue

record: :none fails on a missing baseline. There is no "N screenshots must have been verified this run" gate — the assertion nobody can currently make in CI.

Real usage this is grounded in

jetthoughts.github.io runs visual checks in hand-maintained tiers, selected by regexes over test names:

CRITICAL_TESTS = "/homepage|blog_index|blog_post$|course_|..."   # ~2 min
SMOKE_TESTS    = "/test_homepage$|test_blog_post$|..."           # ~60s

Their own comments record one test at ~196s, 44% of the suite, excluded by hand. The list is a maintenance surface with no feedback when it drifts.

Explicitly out of scope

Choosing which screenshots to run from a diff. The gem cannot know which code affects which page, and guessing would be worse than an explicit list.

Suggested order

1. SNAP_DIFF=0/1 convention — documentation, no new code path
2. Distinguish "off" from "compared nothing" — Disabled runs and silently-broken runs print the same summary #297
3. .snap_diff_last_run.json + the staleness line — the actual SimpleCov experience