Visual regression is expensive, so teams run it sometimes — like SimpleCov. The gem supports that (SnapDiff.config.enabled = false short-circuits assert_matches_screenshot, and #279 put the readiness block inside the same guard so it costs nothing either). What it does not support is knowing which mode you are in.
The defect
A run with visual checks deliberately OFF prints:
[snap_diff] 0 verified, 0 changed, 0 new (not verified). NOTHING WAS VERIFIED -- no screenshot was compared to a committed baseline.
A run with visual checks ON that compared nothing — the exact bug 2.0 exists to kill (#255, #256) — prints the same line.
Reproduced:
SnapDiff.config.enabled = false
SnapDiff::Reporting.send(:counts_summary)
# => "[snap_diff] 0 verified, 0 changed, 0 new (not verified). NOTHING WAS VERIFIED -- ..."
Two opposite situations, one message. Worse: developers who run with screenshots off see this constantly and learn to ignore it — which trains them to ignore it on the day it means something.
Proposal
- When
active? is false, say so instead: [snap_diff] visual checks OFF -- 14 assertions skipped. Enable with SNAP_DIFF=1.
- Count skipped assertions, so the size of what was not run is visible.
Both are cheap; the counter is one increment in the existing early return.
Why this matters more than it looks
Real usage from a consumer that runs the suite this way, jetthoughts.github.io: tiers are hand-maintained regexes of test names —
CRITICAL_TESTS = "/homepage|blog_index|blog_post$|visit_blog_post|course_|..."
SMOKE_TESTS = "/test_homepage$|test_blog_post$|test_not_found$|..."
That list is a maintenance surface with the same failure mode: rename a test and it silently drops out of the gate, with nothing reporting the loss. Their comments also record one test at ~196s being 44% of the suite and excluded by hand.
Making "what did I skip" visible is the smallest change that helps; it does not require the gem to know anything about test selection.
Out of scope
Selecting which screenshots to run from a diff. The gem cannot know which code affects which page, and guessing would be worse than the current explicit lists.
Visual regression is expensive, so teams run it sometimes — like SimpleCov. The gem supports that (
SnapDiff.config.enabled = falseshort-circuitsassert_matches_screenshot, and #279 put the readiness block inside the same guard so it costs nothing either). What it does not support is knowing which mode you are in.The defect
A run with visual checks deliberately OFF prints:
A run with visual checks ON that compared nothing — the exact bug 2.0 exists to kill (#255, #256) — prints the same line.
Reproduced:
Two opposite situations, one message. Worse: developers who run with screenshots off see this constantly and learn to ignore it — which trains them to ignore it on the day it means something.
Proposal
active?is false, say so instead:[snap_diff] visual checks OFF -- 14 assertions skipped. Enable with SNAP_DIFF=1.Both are cheap; the counter is one increment in the existing early return.
Why this matters more than it looks
Real usage from a consumer that runs the suite this way,
jetthoughts.github.io: tiers are hand-maintained regexes of test names —That list is a maintenance surface with the same failure mode: rename a test and it silently drops out of the gate, with nothing reporting the loss. Their comments also record one test at ~196s being 44% of the suite and excluded by hand.
Making "what did I skip" visible is the smallest change that helps; it does not require the gem to know anything about test selection.
Out of scope
Selecting which screenshots to run from a diff. The gem cannot know which code affects which page, and guessing would be worse than the current explicit lists.