Skip to content

Disabled runs and silently-broken runs print the same summary #297

Description

@pftg

Visual regression is expensive, so teams run it sometimes — like SimpleCov. The gem supports that (SnapDiff.config.enabled = false short-circuits assert_matches_screenshot, and #279 put the readiness block inside the same guard so it costs nothing either). What it does not support is knowing which mode you are in.

The defect

A run with visual checks deliberately OFF prints:

[snap_diff] 0 verified, 0 changed, 0 new (not verified). NOTHING WAS VERIFIED -- no screenshot was compared to a committed baseline.

A run with visual checks ON that compared nothing — the exact bug 2.0 exists to kill (#255, #256) — prints the same line.

Reproduced:

SnapDiff.config.enabled = false
SnapDiff::Reporting.send(:counts_summary)
# => "[snap_diff] 0 verified, 0 changed, 0 new (not verified). NOTHING WAS VERIFIED -- ..."

Two opposite situations, one message. Worse: developers who run with screenshots off see this constantly and learn to ignore it — which trains them to ignore it on the day it means something.

Proposal

  1. When active? is false, say so instead: [snap_diff] visual checks OFF -- 14 assertions skipped. Enable with SNAP_DIFF=1.
  2. Count skipped assertions, so the size of what was not run is visible.

Both are cheap; the counter is one increment in the existing early return.

Why this matters more than it looks

Real usage from a consumer that runs the suite this way, jetthoughts.github.io: tiers are hand-maintained regexes of test names —

CRITICAL_TESTS = "/homepage|blog_index|blog_post$|visit_blog_post|course_|..."
SMOKE_TESTS    = "/test_homepage$|test_blog_post$|test_not_found$|..."

That list is a maintenance surface with the same failure mode: rename a test and it silently drops out of the gate, with nothing reporting the loss. Their comments also record one test at ~196s being 44% of the suite and excluded by hand.

Making "what did I skip" visible is the smallest change that helps; it does not require the gem to know anything about test selection.

Out of scope

Selecting which screenshots to run from a diff. The gem cannot know which code affects which page, and guessing would be worse than the current explicit lists.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions