From 267375ed0117372ef6cb9a1bc9a4a0cc1baaa7bb Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 07:52:15 +0200 Subject: [PATCH 1/8] docs: 2.0.0 release readiness Audit of everything a 2.0.0 final would ship, and the fixes that did not need lib/ changes. CHANGELOG - A v2.0.0 entry written for someone upgrading from 1.15.1, not a diff of the betas. What to change (the version), what they will see (exact warning text), the five things that can actually break, and what 2.1 removes. Every claim verified against a real install; the beta sections stay as history. Version consistency - README, docs/UPGRADING.md: no more "beta"/"alpha"/"experiment" framing and no beta pins. Gemfile examples say `~> 2.0`. - Gem name: `capybara-screenshot-diff` is the one we tell people to install; `snap_diff-capybara` is a reserved identical mirror. Stated once in the README with the dual-install consequence, applied everywhere else. - Stale "3.0" references in the Rakefile and docs/architecture.md are now 2.1 (#247 fixed the user docs and missed these). Corrections to claims that were not true - docs/drivers.md promised that everything 2.1 removes "warns once per process naming 2.1". `driver: :auto` is silent whenever ruby-vips is present, and the `driver:` setting itself never warns at all even though 2.1 deletes it (`NoMethodError`). Both are now written down as silent, in drivers.md and UPGRADING.md, since a note is the only notice they can get. - README called ruby-vips "Optional". With neither ruby-vips nor chunky_png installed, comparisons raise `Wrong adapter nil. Available adapters: []`. Says so now. - Setup examples no longer teach `driver: :vips`, a line users have to delete for 2.1. Gem hygiene - gemspec: summary/description that describe what the gem does, the rubygems metadata links (source, changelog, bug tracker, docs), and docs/docker-testing.md dropped from the package (it documents bin/dtest, which is not packaged). Dead bindir/executables removed -- the allow-list never matched exe/. - README's links to CONTRIBUTING.md and docker-testing.md are absolute, so they resolve from inside the gem too. - test/unit/gemspec_packaging_test.rb pins the packaged file list: both Bundler.require entry files present (this broke twice), consumer docs in, contributor docs and build files out, capybara the only runtime dependency. Verified it fails when an entry file is unpackaged. - *.gem is gitignored. Release process - The GitHub Release body linked to blob/main on a repo whose default branch is master -- 404 on every release so far. Links to the tag now. - docs/RELEASE_PREP.md was a stale v1.15.1 checklist. It is now a runbook for how releases actually happen: what the workflow does step by step, the trusted-publisher prerequisite for BOTH gem names, prereleases, post-release verification, and what to do when a run fails halfway. - CONTRIBUTING.md pointed at the wrong version.rb and recommended `rake release`, which publishes only one of the two gem names. Verified with real installs on ruby 4.0.6: 1.15.1 -> this master via path:, a canonical-names setup, `Bundler.require` under each gem name from the built .gem, and the dual-install guard with both gems installed. No lib/ changes. Version not bumped. --- .github/workflows/release.yml | 4 +- .gitignore | 1 + CHANGELOG.md | 145 ++++++++++++++++++++++++++++ CONTRIBUTING.md | 24 +++-- README.md | 21 ++-- Rakefile | 8 +- capybara-screenshot-diff.gemspec | 17 +++- docs/RELEASE_PREP.md | 97 +++++++++++++------ docs/UPGRADING.md | 42 +++++--- docs/architecture.md | 4 +- docs/drivers.md | 6 +- docs/snapdiff.md | 3 +- test/unit/gemspec_packaging_test.rb | 47 +++++++++ 13 files changed, 338 insertions(+), 81 deletions(-) create mode 100644 test/unit/gemspec_packaging_test.rb diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index da99c0dc..0c07cc4f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -90,8 +90,8 @@ jobs: body: | ## What's Changed - See [CHANGELOG.md](https://github.com/${{ github.repository }}/blob/main/CHANGELOG.md#v${{ github.event.inputs.version }}) for full details. + See [CHANGELOG.md](https://github.com/${{ github.repository }}/blob/v${{ github.event.inputs.version }}/CHANGELOG.md) for full details. - **Upgrade Guide:** [docs/UPGRADING.md](https://github.com/${{ github.repository }}/blob/main/docs/UPGRADING.md) + **Upgrade Guide:** [docs/UPGRADING.md](https://github.com/${{ github.repository }}/blob/v${{ github.event.inputs.version }}/docs/UPGRADING.md) env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} diff --git a/.gitignore b/.gitignore index d2a6596e..ca07113a 100644 --- a/.gitignore +++ b/.gitignore @@ -10,6 +10,7 @@ /gemfiles/*.lock /gems.locked /pkg/ +*.gem /spec/reports/ /tmp/ /vendor/sigs/ diff --git a/CHANGELOG.md b/CHANGELOG.md index ad54e1dc..8c39e31f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,151 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [v2.0.0] - unreleased + +**The transitional release.** Everything you run today keeps working. 2.0 makes +`SnapDiff` the canonical namespace, keeps the entire v1 API alongside it as +same-object aliases, and warns — once per process — about what **2.1** removes. +There is no 3.0; 2.1 is the cleanup. + +The sections below the divider are the prerelease notes (alpha1 → beta3) and are +kept as history. This entry is the one to read if you are coming from **1.15.1**. + +### Upgrading from 1.15.1: change the version, run your suite + +```ruby +gem "capybara-screenshot-diff", "~> 2.0" +``` + +That is the whole migration. `screenshot`, `assert_matches_screenshot` and +`capture_screenshot` are unchanged. Your `Capybara::Screenshot::Diff.configure` +block, every `Capybara::Screenshot.*` / `Capybara::Screenshot::Diff.*` setting, and +every legacy constant still resolve — to the *same objects* the new names resolve +to. Baselines are unchanged: capture, encoding, file naming and the `png` default are +the same code as 1.15.1, moved — 2.0 will not rewrite a baseline you already committed. + +Rolling back is a Gemfile edit: pin `"~> 1.15"` and `bundle update`. + +### What you will see in your test output + +One migration notice, the first time the process touches a v1 API: + +```text +[snap_diff deprecation] This process uses the v1 `Capybara::Screenshot*` / +`CapybaraScreenshotDiff*` API. It still works in 2.0 and is REMOVED in 2.1 -- +see docs/UPGRADING.md for the SnapDiff replacements. Silence with +`SnapDiff.silence_deprecations = true` or SNAP_DIFF_SILENCE_DEPRECATIONS=1. +(shown once per process) +``` + +Plus one line per *lazily shimmed* legacy constant you reference, naming your call +site: + +```text +[snap_diff deprecation] `Capybara::Screenshot::Diff::ImageCompare` is deprecated +(constant); use `SnapDiff::Comparison` instead. (called from test/test_helper.rb:12) +``` + +Requiring the gem, the DSL, settings accessors and the eagerly-defined constants +(the error classes, `::VERSION`, `Os`, `Region`, `Reporters::Default`, +`LOADED_DRIVERS`, `AVAILABLE_DRIVERS`) are **silent by design** — +[docs/UPGRADING.md](docs/UPGRADING.md#deprecation-warnings) lists exactly which +names warn and which do not. Silence everything with +`SnapDiff.silence_deprecations = true` or `SNAP_DIFF_SILENCE_DEPRECATIONS=1`. + +### The five things that can actually break + +Everything else is source-compatible. These are not: + +1. **Error class names printed in output are now `SnapDiff::…`.** The class objects + are identical, so `rescue CapybaraScreenshotDiff::ExpectationNotMet` still + catches them — but a CI job that greps the *old* class name out of test output + needs updating. +2. **`defined?` / `const_defined?` on lazily shimmed legacy names returns + `false`/`nil`.** They resolve through `const_missing`, which those checks never + trigger. Move feature detection to the `SnapDiff::` name. Names in the + silent-by-design list are real constants and are unaffected. +3. **Reopening `module Capybara::Screenshot::Diff::Drivers`** (the historical + custom-driver monkey-patch) defines a fresh, empty module that shadows the shim. + Define custom drivers under `SnapDiff::Drivers` instead — and `BaseDriver` is a + mixin now: `class MyDriver < BaseDriver` becomes `include SnapDiff::Driver`. +4. **Stubbing `Capybara::Screenshot::Diff::AVAILABLE_DRIVERS` no longer works.** It + is an eager alias; the gem reads `SnapDiff::Drivers::AVAILABLE_DRIVERS`. Stubbing + the legacy name rebinds the alias only, so the test passes for the wrong reason. +5. **`SnapDiff::Config::MAPPING` is gone**, split into `SnapDiff::Config::SETTINGS` + (setting names) and the `@api private` `SnapDiff::LegacyShims::CONFIG_MAPPING`. + +### Act now: what 2.1 removes + +2.1 deletes the v1 namespace trees, the ChunkyPNG driver, `shift_distance_limit`, +and the whole driver abstraction — **libvips becomes the only backend**. Removing +public API in a minor is a deliberate departure from strict semver; publishing the +contract one release ahead is the mitigation. 2.0 warns once per process for each of: + +| You will hear about it when you… | Do this in 2.0 | +|---|---| +| select `driver: :chunky_png` | add `gem "ruby-vips"` and drop the option | +| run `driver: :auto` **without ruby-vips** — nothing in your setup says `chunky_png`, so this warning is the only sign 2.1 will break the process | install libvips + `ruby-vips` | +| set `shift_distance_limit` | use `median_filter_window_size`, `tolerance` or `color_distance_limit` | +| read `SnapDiff::Drivers.loaded` / `.available` | require `ruby-vips` instead of branching on a detected list | +| `include SnapDiff::Driver` in your own driver | nothing — custom drivers have no migration path | + +Two removals 2.0 cannot warn about, so they are written down instead: **`driver:` +as a setting goes away entirely** (`SnapDiff.config.driver = :vips` and +`Capybara::Screenshot::Diff.driver = :vips` raise `NoMethodError` on 2.1 — delete +the line, one backend needs no selection), and the legacy `LOADED_DRIVERS` / +`AVAILABLE_DRIVERS` constants are plain aliases with nothing to hook. + +### Added +- **`SnapDiff` is the canonical namespace** — the implementation lives in + `lib/snap_diff/`. Configuration (`SnapDiff::Config`, one storage behind every + settings surface), errors (`SnapDiff::Error` and friends), `SnapDiff::Region`, + `SnapDiff::Reporters::Default`, `SnapDiff.session`, `SnapDiff::Reporting.register`, + and integration requires under `snap_diff/integrations/…`. Start here for new + code: [docs/snapdiff.md](docs/snapdiff.md) +- **One consolidated config object** — all 27 settings on `SnapDiff.config`, via + `SnapDiff.configure { |config| … }`. Old and new surfaces share one storage, so a + write through either is visible through the other +- **`SnapDiff::Error` is the catch-all** the docs promise: every error the gem + raises inherits it, including `WindowSizeMismatchError` and `DualInstallError` +- **Deprecation warnings name your call site**, so migration is warning-driven + rather than grep-driven +- **Dual-install guard** — installing both `capybara-screenshot-diff` and + `snap_diff-capybara` raises `SnapDiff::DualInstallError` at require time instead of + silently loading files from whichever gem activated first +- **The gem is also published as + [`snap_diff-capybara`](https://rubygems.org/gems/snap_diff-capybara)** — identical + content and version, matching this repository. `capybara-screenshot-diff` remains + the name to install; the mirror reserves the forward-looking one. **Install one, + never both.** +- New documentation shipped inside the gem: + [docs/snapdiff.md](docs/snapdiff.md) (canonical API) and + [docs/UPGRADING.md](docs/UPGRADING.md) (every renamed constant, which names warn, + rollback) + +### Changed +- **No `activesupport` at runtime.** 1.x required `active_support/core_ext/…` + without declaring the dependency, so a non-Rails install could fail to load. 2.0 + requires nothing beyond `capybara` +- The images-holder struct is now `SnapDiff::Comparison::Images`, ending the + two-classes-one-name collision with the comparator +- The packaged gem is an explicit allow-list — `lib/`, `docs/`, `README.md`, + `LICENSE.txt`, `CHANGELOG.md`. 1.x shipped `Rakefile`, `gems.rb` and the gemspec, + and omitted the README + +### Fixed +- Annotation color constants resolve under a bare `require "snap_diff"`; a differing + comparison previously raised `NameError` there +- `require "snap_diff/integrations/…"` loads the full `SnapDiff` surface, and + `gem "snap_diff-capybara"` works with `Bundler.require` +- Reporter failure warnings use one brand and name the failing reporter class + +### Unchanged +- Ruby 3.2+, Capybara `>= 2, < 4`, the `screenshot` / `assert_matches_screenshot` + DSL, every capture and comparison option, baseline file names and formats + +--- + ## [v2.0.0.beta3] - 2026-08-23 Fixes the canonical `SnapDiff` entry points, which were incomplete in beta2. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 581f276e..dc658049 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -152,17 +152,15 @@ Include: To release a new version: -1. Update the version number in [lib/capybara/screenshot/diff/version.rb](lib/capybara/screenshot/diff/version.rb) +1. Update the version number in [lib/snap_diff/version.rb](lib/snap_diff/version.rb) — the + only place it lives; the gemspec and the legacy + `lib/capybara/screenshot/diff/version.rb` both read it 2. Update [CHANGELOG.md](CHANGELOG.md) with the new version and date -3. Create a GitHub Release: - - Go to [Actions → Release](https://github.com/snap-diff/snap_diff-capybara/actions/workflows/release.yml) - - Click **Run workflow**, enter the version number - - The workflow will: test → tag → publish to RubyGems → create GitHub Release - -Or manually: - -```bash -bundle exec rake release -``` - -This creates a git tag, pushes commits and tags, and pushes the `.gem` file to [rubygems.org](https://rubygems.org). +3. Go to [Actions → Release](https://github.com/snap-diff/snap_diff-capybara/actions/workflows/release.yml), + click **Run workflow** and enter the version number. The workflow verifies the version + against `lib/`, tests, tags, publishes **both** gem names, and creates the GitHub Release. + +**Do not use `rake release`.** It is inherited from `bundler/gem_tasks` and publishes only +`capybara-screenshot-diff`, skipping the `snap_diff-capybara` mirror — the two gems must +never diverge in version. The full runbook, including the trusted-publishing prerequisites +and what to do when a run fails halfway, is in [docs/RELEASE_PREP.md](docs/RELEASE_PREP.md). diff --git a/README.md b/README.md index 909e337a..b9757ae0 100644 --- a/README.md +++ b/README.md @@ -11,9 +11,11 @@ Stop shipping UI bugs. Take screenshots in your Capybara tests, commit baselines **Why this gem?** Baselines live in git — review UI changes in pull requests like you review code. Runs offline, works in CI, zero vendor lock-in. Unlike Percy/Chromatic (paid SaaS), nothing to sign up for. Unlike BackstopJS, no Node required. -> **2.0 experiment (beta):** the gem is moving to a `SnapDiff` canonical namespace. Opt in with `gem "capybara-screenshot-diff", "2.0.0.beta3"` (or the latest 2.0.0 prerelease; prereleases are never installed by default — normal installs stay on 1.x). Legacy names keep working; the first legacy API a process touches prints one migration notice (lazily shimmed constants also warn once each — see [which names warn](docs/UPGRADING.md#deprecation-warnings)), silenceable via `SnapDiff.silence_deprecations = true` or `SNAP_DIFF_SILENCE_DEPRECATIONS=1`. Writing new code? Start from [SnapDiff — the canonical API](docs/snapdiff.md), which uses canonical names only. Migrating an existing suite? See the [upgrade guide](docs/UPGRADING.md). Share feedback on [#166](https://github.com/snap-diff/snap_diff-capybara/issues/166). +> **2.0 is the transitional release.** The gem's canonical namespace is now `SnapDiff`. Upgrading from 1.x is a version bump — every legacy `Capybara::Screenshot::Diff` / `CapybaraScreenshotDiff` name still resolves to the same object and keeps working. The first legacy API a process touches prints one migration notice (lazily shimmed constants also warn once each — see [which names warn](docs/UPGRADING.md#deprecation-warnings)), silenceable via `SnapDiff.silence_deprecations = true` or `SNAP_DIFF_SILENCE_DEPRECATIONS=1`. > -> Starting with the 2.0 prereleases the gem is also published as [`snap_diff-capybara`](https://rubygems.org/gems/snap_diff-capybara) — identical content and versions under the forward-looking name, matching this repository. Install either; don't install both. +> **2.1 removes what 2.0 warns about**: the legacy namespaces, the ChunkyPNG driver, `shift_distance_limit`, the `driver:` setting and the driver abstraction — libvips becomes the only backend. There is no 3.0. Writing new code? Start from [SnapDiff — the canonical API](docs/snapdiff.md), which uses canonical names only. Migrating an existing suite? See the [upgrade guide](docs/UPGRADING.md). +> +> **Two gem names, one gem.** `capybara-screenshot-diff` is the name to install. The same content is also published as [`snap_diff-capybara`](https://rubygems.org/gems/snap_diff-capybara) — identical version, forward-looking name matching this repository — so that name is reserved and resolvable. **Install one, never both**: with both in a Gemfile the gem raises `SnapDiff::DualInstallError` at require time. ## Quick Start (5 minutes) @@ -22,9 +24,13 @@ Stop shipping UI bugs. Take screenshots in your Capybara tests, commit baselines ```ruby # Gemfile gem 'capybara-screenshot-diff' -gem 'ruby-vips' # Optional: 10x faster comparisons +gem 'ruby-vips' # The image backend. Needs libvips — see Installation below ``` +The gem ships no image backend of its own. Add `ruby-vips` (recommended, and the only +backend from 2.1 on) or `chunky_png` (pure Ruby, no system library, removed in 2.1) — with +neither, comparisons raise `Wrong adapter nil. Available adapters: []`. + ```ruby # test/test_helper.rb require 'capybara_screenshot_diff/minitest' @@ -186,7 +192,7 @@ Set `window_size` for consistent dimensions and use `perceptual_threshold: 2.0`
Will this slow down my tests? -Comparisons add ~50ms per image with VIPS. Without `ruby-vips`, ChunkyPNG is used (slower but no system dependency). `stability_time_limit` adds wait time — keep it low (0.1-0.5s) or use `disable_animations` instead. +Comparisons add ~50ms per image with VIPS. If you add `chunky_png` to your Gemfile instead, it is used as a pure-Ruby fallback (slower, no system dependency, and removed in 2.1). `stability_time_limit` adds wait time — keep it low (0.1-0.5s) or use `disable_animations` instead.
@@ -197,11 +203,12 @@ Comparisons add ~50ms per image with VIPS. Without `ruby-vips`, ChunkyPNG is use ## Installation -**Requirements:** Ruby 3.2+. Rails 7.1+ for Rails integration; non-Rails projects supported via `CapybaraScreenshotDiff.serve()`. For the `:vips` driver: [libvips 8.9+](https://libvips.github.io/libvips/install.html). On macOS: `brew install vips`. On Ubuntu: `apt-get install libvips-dev`. +**Requirements:** Ruby 3.2+, Capybara 2–3. Rails 7.1+ for Rails integration; non-Rails projects supported via `CapybaraScreenshotDiff.serve()`. For the `:vips` driver (recommended, and the only backend from 2.1 on): [libvips 8.9+](https://libvips.github.io/libvips/install.html). On macOS: `brew install vips`. On Ubuntu: `apt-get install libvips-dev`. ## Docs - [SnapDiff — the canonical API](docs/snapdiff.md) — setup, config, object map, custom drivers & reporters, canonical names only +- [Upgrading](docs/UPGRADING.md) — 1.x → 2.0, every renamed constant, which names warn, what 2.1 removes, rollback - [Framework Setup](docs/framework-setup.md) — Minitest, RSpec, Cucumber - [CI & Non-Rails Integration](docs/ci-integration.md) — GitHub Actions, reusable action, static sites, baseline updates - [Configuration Reference](docs/configuration.md) — all options explained @@ -211,11 +218,11 @@ Comparisons add ~50ms per image with VIPS. Without `ruby-vips`, ChunkyPNG is use ## Development -After checking out the repo, run `bin/setup` then `rake test`. See [Docker Testing](docs/docker-testing.md) for reproducible CI-matching test runs. +After checking out the repo, run `bin/setup` then `rake test`. See [Docker Testing](https://github.com/snap-diff/snap_diff-capybara/blob/master/docs/docker-testing.md) for reproducible CI-matching test runs. ## Contributing -See [CONTRIBUTING.md](CONTRIBUTING.md) +See [CONTRIBUTING.md](https://github.com/snap-diff/snap_diff-capybara/blob/master/CONTRIBUTING.md) ## License diff --git a/Rakefile b/Rakefile index c0c705ed..3562df55 100644 --- a/Rakefile +++ b/Rakefile @@ -5,13 +5,13 @@ require "rake/testtask" task default: :test -# THE 3.0 SPLIT. +# THE 2.1 SPLIT. # # test/legacy/ holds every test whose SUBJECT is the v1 compatibility surface # -- the old Capybara::Screenshot / CapybaraScreenshotDiff namespaces, their # deprecation warnings, and the gates that keep lib/capybara* alias-only. # Those tests guard the v1 contract for the whole 2.x line, so they stay and -# stay green; in 3.0 they are deleted by the same commit that deletes what +# stay green; in 2.1 they are deleted by the same commit that deletes what # they test: # # git rm -r lib/capybara* lib/capybara_screenshot_diff.rb \ @@ -23,7 +23,7 @@ task default: :test # # `rake test` -- everything, today's gate. # `rake test:canonical` -- exactly what must still pass once test/legacy and -# the v1 trees are gone. THE 3.0 GATE. +# the v1 trees are gone. THE 2.1 GATE. # `rake test:unit` -- unit-sized tests; test/legacy is unit-sized too # (legacy/ marks lifetime, not kind), so it is in. LEGACY_SURFACE_TESTS = "test/legacy/**/*_test.rb" @@ -34,7 +34,7 @@ Rake::TestTask.new(:test) do |t| t.test_files = FileList["test/**/*_test.rb"] end -desc "Run every test that must survive the 3.0 deletion of the v1 surface" +desc "Run every test that must survive the 2.1 deletion of the v1 surface" Rake::TestTask.new("test:canonical") do |t| t.libs << "test" t.libs << "lib" diff --git a/capybara-screenshot-diff.gemspec b/capybara-screenshot-diff.gemspec index fea3dbc2..efa095d5 100644 --- a/capybara-screenshot-diff.gemspec +++ b/capybara-screenshot-diff.gemspec @@ -9,20 +9,27 @@ Gem::Specification.new do |spec| spec.version = SnapDiff::VERSION spec.authors = ["Uwe Kubosch"] spec.email = ["uwe@kubosch.no"] - spec.summary = "Track your GUI changes with diff assertions" - spec.description = "Save screen shots and track changes with graphical diff" + spec.summary = "Visual regression testing for Capybara — screenshot diffs in your test suite" + spec.description = "Take screenshots in your Capybara tests, commit the baselines to git, " \ + "and let your suite fail on unintended visual changes. Runs offline, no SaaS." spec.homepage = "https://github.com/snap-diff/snap_diff-capybara" spec.required_ruby_version = ">= 3.2" spec.license = "MIT" spec.metadata["allowed_push_host"] = "https://rubygems.org/" + spec.metadata["source_code_uri"] = spec.homepage + spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/master/CHANGELOG.md" + spec.metadata["bug_tracker_uri"] = "#{spec.homepage}/issues" + spec.metadata["documentation_uri"] = "#{spec.homepage}/blob/master/docs/snapdiff.md" # Allow-list: everything a consumer needs at runtime plus the shipped docs. # Build/dev files (gems.rb, Rakefile, the gemspec itself, tests, CI) stay out. spec.files = `git ls-files -z`.split("\x0") .grep(%r{\A(lib/|docs/|README\.md\z|LICENSE\.txt\z|CHANGELOG\.md\z)}) - .grep_v(%r{\Adocs/RELEASE_PREP\.md\z}) # maintainer-only, not user documentation + # Contributor docs: they describe releasing this gem and running its own suite + # via bin/dtest, which is not packaged. Nothing a consumer can act on. + .grep_v(%r{\Adocs/(RELEASE_PREP|docker-testing)\.md\z}) - spec.bindir = "exe" - spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) } + # No executables: the allow-list above never matches exe/, so bindir and + # executables would always be empty. spec.require_paths = ["lib"] spec.add_development_dependency "actionpack", ">= 7.1", "< 9" diff --git a/docs/RELEASE_PREP.md b/docs/RELEASE_PREP.md index a50b1ddc..2b4087bc 100644 --- a/docs/RELEASE_PREP.md +++ b/docs/RELEASE_PREP.md @@ -1,44 +1,81 @@ -# Release Preparation — v1.15.1 +# Release runbook -## Summary +Maintainer-only. This file is deliberately excluded from the packaged gem. -HIGH-severity VipsDriver resize fix (retina halving could enlarge screenshots -and corrupt baselines) plus the driver contract tests that caught it. +Releases are **one `workflow_dispatch`**. There is no `rake release`, no manual +`gem push`, and no local credential. Everything below either happens in CI or has to +be committed to `master` before you dispatch. -## Release Checklist +## What the Release workflow does -### Pre-Release +[`.github/workflows/release.yml`](../.github/workflows/release.yml), dispatched from +[Actions → Release](https://github.com/snap-diff/snap_diff-capybara/actions/workflows/release.yml) +with a single `version` input (e.g. `2.0.0`, or `2.1.0.beta1`): -- [x] Update version to `1.15.1` -- [x] Run tests: `bundle exec rake test:unit` (376 runs, 0 failures) -- [x] Update CHANGELOG.md +1. **Verify version** — reads `Capybara::Screenshot::Diff::VERSION` out of `lib/` and + fails if it does not equal the input. This is the guard that makes the dispatch + safe: the version lives in `lib/snap_diff/version.rb` and must already be on + `master`. +2. **Test** — `bundle exec rake test:unit` on Ruby 4.0. +3. **Tag** — creates and pushes `v`. Idempotent: it skips if the tag already + exists at HEAD and fails loudly if it exists at a *different* commit. A failed run + can be re-dispatched without cleanup. +4. **Publish `capybara-screenshot-diff`** — `rubygems/release-gem@v1`, via RubyGems + trusted publishing (OIDC). No API key is stored anywhere. +5. **Publish the `snap_diff-capybara` mirror** — the same gemspec is loaded, renamed + in memory, built and pushed. The mirror gemspec is *generated in CI, never + committed*, so local `gem build` and the `gemspec` directive in `gems.rb` stay + unambiguous. It reuses the credential `release-gem` already set up, which covers + any gem whose rubygems.org settings trust this repo + workflow. +6. **GitHub Release** — body links to the CHANGELOG and upgrade guide **at the tag**. + Prerelease status is auto-detected from the tag, so `v2.1.0.beta1` lands as + *Pre-release* and never displaces the *Latest* badge. -### Release (One Click) +### Prerequisites that live outside this repo -1. Push to GitHub -2. Go to [Actions → Release](https://github.com/snap-diff/snap_diff-capybara/actions/workflows/release.yml) -3. Click **Run workflow**, enter `1.15.1` -4. Workflow will: test → tag → publish to RubyGems → create GitHub Release +- **Both** gem names must trust this repo + `release.yml` as a trusted publisher on + rubygems.org — `capybara-screenshot-diff` **and** `snap_diff-capybara`. If only one + does, step 4 or 5 fails after the tag is already pushed; re-dispatch after fixing. +- Branch protection on `master` must allow the workflow's tag push. -### Post-Release +## Before you dispatch -- [ ] Verify on [RubyGems](https://rubygems.org/gems/capybara-screenshot-diff) -- [ ] Verify GitHub Release created +- [ ] `lib/snap_diff/version.rb` bumped to the exact version you will type into the + workflow. Nothing else holds a version — the gemspec, the legacy + `capybara/screenshot/diff/version.rb` and the mirror gemspec all read it. +- [ ] `CHANGELOG.md` has a section for this version with a real date (not + `unreleased`), written for someone upgrading from the last **stable** release + rather than from the previous prerelease. +- [ ] Docs carry no stale version pins — `README.md`, `docs/UPGRADING.md`, + `docs/snapdiff.md`. Grep for the previous version string. +- [ ] `mise x ruby@4.0.6 -- bundle exec rake test` (full suite, both gates) and + `mise x ruby@4.0.6 -- bundle exec standardrb` are green. +- [ ] CI is green on `master` at the commit you are releasing — the workflow only + runs `test:unit`, which is a subset. +- [ ] `gem build capybara-screenshot-diff.gemspec` and inspect the file list if + anything touched the gemspec allow-list. `*.gem` is gitignored, but delete the + artifact anyway — a stale one in the working tree is confusing. -## What Changed +### Prereleases -### Added -- `SnapDiff::Config` — flat, additive consolidation of all 27 settings - (`SnapDiff.config` / `SnapDiff.configure`); old accessors stay canonical +Nothing special to configure. Use a prerelease version string (`2.1.0.beta1`) in +`version.rb` and in the dispatch input. RubyGems never resolves a prerelease by +default, and GitHub marks the release *Pre-release* on its own. -### Fixed -- `pending_if_new` no longer converts real teardown/after-hook failures into - pending tests (Minitest defers to `after_teardown`; RSpec uses `append_after`; - known residual for consumer `append_after` hooks documented) -- `BacktraceFilter` custom `lib_directory` matches on a path boundary +## After -### Internal -- Guard tests for failure masking and skip-area/VCS-baseline regressions; - two files merged into sole consumers (constant paths preserved) +- [ ] Both gems visible and at the same version: + [capybara-screenshot-diff](https://rubygems.org/gems/capybara-screenshot-diff), + [snap_diff-capybara](https://rubygems.org/gems/snap_diff-capybara). A version + published under only one name is the failure mode to watch for — the two gems + ship identical files and the dual-install guard assumes they never diverge. +- [ ] GitHub Release created, with the right Latest/Pre-release status. +- [ ] `gem install capybara-screenshot-diff -v ` in a scratch dir resolves. -See [CHANGELOG.md](../CHANGELOG.md) for full details. +## If a run fails halfway + +Re-dispatch the same version. The tag step is idempotent and `gem push` rejects a +duplicate version, so the only real hazard is a mirror push that failed for a +credential reason — fix the trusted-publisher settings and re-dispatch. **Never** +retag: the workflow refuses to move an existing tag and you should not do it by hand +either. diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index e7b623e0..f3137a38 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -1,12 +1,12 @@ # Upgrading -## Upgrading to v2.0 (alpha) +## Upgrading to v2.0 ### Overview Version 2.0 introduces a new canonical namespace (`SnapDiff`) for cleaner, more discoverable code. The public DSL remains unchanged — your existing `screenshot` and `assert_matches_screenshot` calls work without modification. This guide covers the optional migration path for settings and the new namespace. -**Status:** `2.0.0.beta3` is an opt-in prerelease. RubyGems never installs prereleases by default resolution — normal `bundle update` keeps you on the 1.x line. The final 2.0.0 ships only after adopter feedback; please report anything surprising on [#166](https://github.com/snap-diff/snap_diff-capybara/issues/166). +**Status:** 2.0 is the **transitional** release — the v1 API and the canonical `SnapDiff` API both work. **2.1 removes** everything 2.0 warns about (the legacy namespaces, the ChunkyPNG driver, `shift_distance_limit`, the `driver:` setting and the driver abstraction). There is no 3.0. Migrating on 2.0 is optional; doing it before 2.1 is not. **Estimated upgrade time:** 5–15 minutes (most users need only the Gemfile pin) @@ -14,17 +14,20 @@ Version 2.0 introduces a new canonical namespace (`SnapDiff`) for cleaner, more [SnapDiff — the canonical API](snapdiff.md): the same setup, configuration, and extension points with canonical names only, no legacy shapes to unlearn. -**Breaking changes:** None for the DSL; one migration notice per process plus a deprecation warning per legacy constant you reference (both suppressible), plus two known alpha caveats (see below) +**Breaking changes:** None for the DSL; one migration notice per process plus a deprecation warning per legacy constant you reference (both suppressible), plus the [known caveats](#known-caveats) below --- ### The Short Version (Most Users) ```ruby -# In your Gemfile — the exact prerelease version is required to opt in -gem "capybara-screenshot-diff", "2.0.0.beta3" # or the latest 2.0.0 prerelease +# In your Gemfile +gem "capybara-screenshot-diff", "~> 2.0" ``` +The same content is also published as `snap_diff-capybara`. Install **one** — with both +in a Gemfile the gem raises `SnapDiff::DualInstallError` at require time. + ```bash bundle install bundle exec rake test @@ -114,7 +117,6 @@ Capybara::Screenshot::Diff.configure do |screenshot, diff| screenshot.window_size = [1280, 1024] screenshot.blur_active_element = false diff.tolerance = 0.0005 - diff.driver = :vips end ``` @@ -125,7 +127,6 @@ SnapDiff.start do |screenshot, diff| screenshot.window_size = [1280, 1024] screenshot.blur_active_element = false diff.tolerance = 0.0005 - diff.driver = :vips end ``` @@ -136,7 +137,6 @@ SnapDiff.configure do |config| config.window_size = [1280, 1024] config.blur_active_element = false config.tolerance = 0.0005 - config.driver = :vips end ``` @@ -215,6 +215,16 @@ warns once per process per subject, through the same channel and the same silenc | read `SnapDiff::Drivers.available` | driver detection | require `ruby-vips` instead of branching on a detected list | | `include SnapDiff::Driver` in your own driver class | the driver mixin | nothing — see below | +> **The one removal on this list that 2.0 cannot warn you about: the `driver:` setting +> itself.** `SnapDiff.config.driver = :vips` (and the legacy +> `Capybara::Screenshot::Diff.driver = :vips`, and the per-screenshot `driver:` override) +> is **silent** in 2.0 and raises `NoMethodError: undefined method 'driver='` in 2.1. +> Warning on it would fire on the recommended configuration, so this note is the warning: +> **delete the line.** With libvips the only backend there is nothing to select, and the +> default just works. The same goes for `driver: :auto` on a machine that *has* +> `ruby-vips` — the `:auto` warning above only fires when `:auto` actually falls back to +> ChunkyPNG, because that is the case where 2.1 stops the process comparing at all. + ``` [snap_diff deprecation] `driver: :auto` selected chunky_png because libvips is not available in this process. The chunky_png driver is REMOVED in 2.1, when libvips (the `ruby-vips` gem) becomes required -- install it now, or this setup stops comparing on 2.1. See docs/drivers.md. Silence with `SnapDiff.silence_deprecations = true` or SNAP_DIFF_SILENCE_DEPRECATIONS=1. (shown once per process) (called from /app/test/test_helper.rb:12) ``` @@ -223,7 +233,7 @@ warns once per process per subject, through the same channel and the same silenc mixin, the `SnapDiff::Drivers.loaded` registry, `SnapDiff::Drivers.available` / `SnapDiff::Utils.detect_available_drivers`, and selecting a driver by name. Nothing replaces them, and this guide is not going to pretend otherwise — if you maintain a third-party driver, -say so on [#166](https://github.com/snap-diff/snap_diff-capybara/issues/166) before 2.1 ships. +say so on [the issue tracker](https://github.com/snap-diff/snap_diff-capybara/issues) before 2.1 ships. Three spots on the same chopping block stay silent: the legacy `Capybara::Screenshot::Diff::LOADED_DRIVERS` / `::AVAILABLE_DRIVERS` aliases are plain constants @@ -276,9 +286,10 @@ export SNAP_DIFF_SILENCE_DEPRECATIONS=1 --- -### Known Alpha Caveats +### Known Caveats -Two deliberate consequences of the lazy shim design — both flagged for feedback on [#166](https://github.com/snap-diff/snap_diff-capybara/issues/166): +Two deliberate consequences of the lazy shim design. Both go away at 2.1, when the shimmed +names go away with them: 1. **`defined?` / `const_defined?` on lazily-shimmed legacy names returns `false`/`nil`.** The shims resolve via `const_missing`, which those checks never trigger. Feature detection like `defined?(Capybara::Screenshot::Diff::ImageCompare)` must move to the `SnapDiff::` name. Everything in [Silent by design](#silent-by-design) is unaffected — those names are real constants, so `defined?`, `const_defined?` and `rescue` all behave as they always did. @@ -331,12 +342,15 @@ All settings and baselines are compatible with v1.x. Simply pin your Gemfile bac ### Summary Checklist -- [ ] Pin `gem "capybara-screenshot-diff", "2.0.0.beta3"` (or the latest 2.0.0 prerelease) in your Gemfile +- [ ] Pin `gem "capybara-screenshot-diff", "~> 2.0"` in your Gemfile - [ ] Run `bundle install` - [ ] Run your test suite to verify no regressions -- [ ] (Optional) Migrate config to the `SnapDiff` namespace +- [ ] Read the warnings it prints — each one names something 2.1 removes +- [ ] Add `gem "ruby-vips"` if you are not already on it (2.1 makes libvips the only backend) +- [ ] Drop `driver:` from your config — it is silent in 2.0 and gone in 2.1 +- [ ] (Optional, but do it before 2.1) Migrate config and constants to the `SnapDiff` namespace - [ ] (Optional) Silence deprecation warnings if not ready to migrate -- [ ] Report anything surprising on [#166](https://github.com/snap-diff/snap_diff-capybara/issues/166) +- [ ] Report anything surprising on [the issue tracker](https://github.com/snap-diff/snap_diff-capybara/issues) --- diff --git a/docs/architecture.md b/docs/architecture.md index dc864310..84521b1c 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -127,7 +127,7 @@ Drivers abstract image processing operations. Shared default behavior lives in t **Auto-detection:** `SnapDiff::Drivers.detect_available` tries to load `:vips` first (via `ruby-vips` gem), then `:chunky_png`. The `:auto` driver mode picks the first available. `Utils.detect_available_drivers` is the older name and one-lines into it. -**Registry (ADR-008 step 5b):** `SnapDiff::Drivers.loaded` is the canonical driver-class cache — a `name => class` hash filled lazily by `Utils.find_driver_class_for`, and the registration point for custom drivers (the legacy `Capybara::Screenshot::Diff::LOADED_DRIVERS` is an eager same-object alias, so registrations through either land in the same hash). `SnapDiff::Drivers.available` is the canonical read API for the detected list, and since the 3.0-readiness pass the value lives with it, as `SnapDiff::Drivers::AVAILABLE_DRIVERS` — that constant is now the published stubbing point, and the legacy `Capybara::Screenshot::Diff::AVAILABLE_DRIVERS` is an eager same-object alias of it. `SnapDiff::Drivers.for` resolves an options hash to a driver instance. See [Custom drivers](snapdiff.md#custom-drivers). +**Registry (ADR-008 step 5b):** `SnapDiff::Drivers.loaded` is the canonical driver-class cache — a `name => class` hash filled lazily by `Utils.find_driver_class_for`, and the registration point for custom drivers (the legacy `Capybara::Screenshot::Diff::LOADED_DRIVERS` is an eager same-object alias, so registrations through either land in the same hash). `SnapDiff::Drivers.available` is the canonical read API for the detected list, and since the 2.1-readiness pass the value lives with it, as `SnapDiff::Drivers::AVAILABLE_DRIVERS` — that constant is now the published stubbing point, and the legacy `Capybara::Screenshot::Diff::AVAILABLE_DRIVERS` is an eager same-object alias of it. `SnapDiff::Drivers.for` resolves an options hash to a driver instance. See [Custom drivers](snapdiff.md#custom-drivers). ### 6. Difference Region Detection @@ -223,7 +223,7 @@ Since ADR-008 step 1 the storage ownership is inverted from the original v2 cons The legacy `Capybara::Screenshot.*` / `Capybara::Screenshot::Diff.*` accessors are thin delegators generated from `SnapDiff::LegacyShims::CONFIG_MAPPING` (both singleton and instance methods, matching what `mattr_accessor` used to define) that forward to that one object. One storage, two views — a write through either surface is visible through the other structurally, not by synchronization. -Since the 3.0-readiness pass, `lib/snap_diff/legacy_shims.rb` is the single file that holds the v1 surface as code: the `const_missing` forwarders, `CONFIG_MAPPING` and its generator, the derived forwarders (`Screenshot.active?`, `Diff.configure`, `Diff.default_options`, …) and `SnapDiff.start`. `Config` itself names nothing from the v1 namespaces — it declares its settings in `Config::SETTINGS`, and `LegacyShims::CONFIG_MAPPING` says which legacy holder each one is exposed on (an invariant pinned by `snap_diff_config_test.rb`). `lib/capybara/screenshot/diff/config_legacy.rb` remains at the old path as a pair of requires. +Since the 2.1-readiness pass, `lib/snap_diff/legacy_shims.rb` is the single file that holds the v1 surface as code: the `const_missing` forwarders, `CONFIG_MAPPING` and its generator, the derived forwarders (`Screenshot.active?`, `Diff.configure`, `Diff.default_options`, …) and `SnapDiff.start`. `Config` itself names nothing from the v1 namespaces — it declares its settings in `Config::SETTINGS`, and `LegacyShims::CONFIG_MAPPING` says which legacy holder each one is exposed on (an invariant pinned by `snap_diff_config_test.rb`). `lib/capybara/screenshot/diff/config_legacy.rb` remains at the old path as a pair of requires. The two legacy views are organized into two namespaces: diff --git a/docs/drivers.md b/docs/drivers.md index f4faede0..b9608177 100644 --- a/docs/drivers.md +++ b/docs/drivers.md @@ -9,13 +9,15 @@ ## Removed in 2.1: everything on this page except VIPS 2.1 makes **libvips the only backend**. 2.0 is the transitional release — all of the -following still works, and warns once per process naming 2.1. Silence the warnings with +following still works. Most of it warns once per process naming 2.1; the rows marked +*silent* cannot warn, and this table is their notice. Silence the warnings with `SnapDiff.silence_deprecations = true` or `SNAP_DIFF_SILENCE_DEPRECATIONS=1`. | Removed in 2.1 | What to do in 2.0 | |---|---| | the `:chunky_png` driver | add `gem "ruby-vips"` to your Gemfile and drop `driver: :chunky_png` | -| `driver: :auto` (and the `:auto` default) | with one backend there is nothing to choose; install `ruby-vips` and the default just works | +| the `driver:` setting itself — `SnapDiff.config.driver =`, the legacy `Capybara::Screenshot::Diff.driver =`, and the per-screenshot `driver:` override (**silent**: on 2.1 they raise `NoMethodError`) | delete the line; one backend needs no selection | +| `driver: :auto` (and the `:auto` default) — warns **only when `:auto` actually falls back to ChunkyPNG**, i.e. when `ruby-vips` is missing; **silent** otherwise | with one backend there is nothing to choose; install `ruby-vips` and the default just works | | `shift_distance_limit` | ChunkyPNG-only. Use `median_filter_window_size`, `tolerance` or `color_distance_limit` — see [Configuration](configuration.md#allowed-shift-distance) | | `SnapDiff::Driver` (the custom-driver mixin) | nothing — see below | | `SnapDiff::Drivers.loaded` (the registry) | nothing — see below | diff --git a/docs/snapdiff.md b/docs/snapdiff.md index d431afe6..c74fdccd 100644 --- a/docs/snapdiff.md +++ b/docs/snapdiff.md @@ -104,7 +104,6 @@ All 27 settings live on one flat object, `SnapDiff.config` (a `SnapDiff::Config` SnapDiff.configure do |config| config.window_size = [1280, 1024] config.tolerance = 0.0005 - config.driver = :vips config.save_path = "doc/screenshots" end @@ -270,7 +269,7 @@ SnapDiff::Reporting.finalize! > by name. In 2.0 all of it still works and warns once per process (silence with > `SnapDiff.silence_deprecations = true` or `SNAP_DIFF_SILENCE_DEPRECATIONS=1`). Nothing > here migrates to a 2.1 shape — there is no 2.1 shape. If you maintain a driver, say so on -> [#166](https://github.com/snap-diff/snap_diff-capybara/issues/166) before 2.1 ships. +> [the issue tracker](https://github.com/snap-diff/snap_diff-capybara/issues) before 2.1 ships. A driver is a plain object that does the image work. Include `SnapDiff::Driver` for the shared defaults, then implement the operations the comparison engine calls: diff --git a/test/unit/gemspec_packaging_test.rb b/test/unit/gemspec_packaging_test.rb new file mode 100644 index 00000000..e8bbee26 --- /dev/null +++ b/test/unit/gemspec_packaging_test.rb @@ -0,0 +1,47 @@ +# frozen_string_literal: true + +require "test_helper" + +# The gemspec's `spec.files` allow-list is the one part of the release that no +# other test looks at, and it has shipped broken twice: a missing +# Bundler.require entry file makes `gem "..."` load NOTHING and fail later with +# a confusing NameError, far from the cause. These assertions are about the +# PACKAGED file list, not about the working tree. +class GemspecPackagingTest < ActiveSupport::TestCase + GEMSPEC_PATH = File.expand_path("../../capybara-screenshot-diff.gemspec", __dir__) + + def spec + @spec ||= Gem::Specification.load(GEMSPEC_PATH) + end + + # Bundler requires the gem's OWN NAME. Each published name needs a file that + # matches it, or `Bundler.require` silently no-ops for that name. + test "ships a Bundler.require entry file for both published gem names" do + assert_includes spec.files, "lib/capybara-screenshot-diff.rb" + assert_includes spec.files, "lib/snap_diff-capybara.rb" + end + + test "ships the files rubygems.org and a consumer read" do + ["README.md", "LICENSE.txt", "CHANGELOG.md", "docs/UPGRADING.md", "docs/snapdiff.md"] + .each { |file| assert_includes spec.files, file } + end + + # Contributor-only docs describe releasing this gem and bin/dtest, neither of + # which a consumer has. + test "ships no contributor-only docs" do + refute_includes spec.files, "docs/RELEASE_PREP.md" + refute_includes spec.files, "docs/docker-testing.md" + end + + test "ships no build or development files" do + cruft = spec.files.grep_v(%r{\A(lib/|docs/|README\.md\z|LICENSE\.txt\z|CHANGELOG\.md\z)}) + assert_empty cruft + end + + # Every driver is an optional require. Adding a runtime dependency here is a + # decision, not an accident -- 2.1 makes ruby-vips one deliberately. + test "declares capybara as its only runtime dependency" do + runtime = spec.dependencies.select { |dependency| dependency.type == :runtime } + assert_equal ["capybara"], runtime.map(&:name) + end +end From c06db9a0af3f18b96e1b22f64d2d35a453d7d883 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 07:56:39 +0200 Subject: [PATCH 2/8] docs: SnapDiff::Error is the base class for errors the gem defines, not every error it raises docs/snapdiff.md's object map said "Base class for every error this gem raises". It is not: a missing image backend raises a bare RuntimeError ("Wrong adapter nil. Available adapters: []", reproduced on a bundle with neither ruby-vips nor chunky_png) and StableScreenshoter raises ArgumentError. Verified the four defined errors -- ExpectationNotMet, UnstableImage, WindowSizeMismatchError, DualInstallError -- do all inherit SnapDiff::Error, so the useful half of the promise holds and is now the one being made. --- CHANGELOG.md | 7 +++++-- docs/snapdiff.md | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c39e31f..a9cfbe6b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -110,8 +110,11 @@ the line, one backend needs no selection), and the legacy `LOADED_DRIVERS` / - **One consolidated config object** — all 27 settings on `SnapDiff.config`, via `SnapDiff.configure { |config| … }`. Old and new surfaces share one storage, so a write through either is visible through the other -- **`SnapDiff::Error` is the catch-all** the docs promise: every error the gem - raises inherits it, including `WindowSizeMismatchError` and `DualInstallError` +- **`SnapDiff::Error` is the base class for every error the gem defines** — + `ExpectationNotMet`, `UnstableImage`, `WindowSizeMismatchError` and + `DualInstallError` all inherit it, so one `rescue SnapDiff::Error` covers them. + (Misuse still surfaces as plain Ruby: `ArgumentError` for bad arguments, + `RuntimeError` when no image backend is installed.) - **Deprecation warnings name your call site**, so migration is warning-driven rather than grep-driven - **Dual-install guard** — installing both `capybara-screenshot-diff` and diff --git a/docs/snapdiff.md b/docs/snapdiff.md index c74fdccd..57fc812f 100644 --- a/docs/snapdiff.md +++ b/docs/snapdiff.md @@ -160,7 +160,7 @@ integration require; a few objects need their own require, noted below. | `SnapDiff::Region` | Bounding box value object — `from_edge_coordinates`, `to_edge_coordinates` | | `SnapDiff::DSL` | `screenshot`, `assert_matches_screenshot`, `capture_screenshot`, groups/sections | | `SnapDiff::Minitest::Assertions` | Minitest wiring (`snap_diff/integrations/minitest`) | -| `SnapDiff::Error` | Base class for every error this gem raises | +| `SnapDiff::Error` | Base class for every error this gem *defines* — one `rescue SnapDiff::Error` covers all of the rows below. Misuse still surfaces as plain Ruby: bad arguments raise `ArgumentError`, and a missing image backend raises `RuntimeError` | | `SnapDiff::ExpectationNotMet` | A screenshot did not match its baseline | | `SnapDiff::UnstableImage` | No stable capture within `stability_time_limit` / `wait` | | `SnapDiff::WindowSizeMismatchError` | Browser window is not the configured `window_size` | From 9b1434689266dab182b99432bf5018e3726d05a8 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 07:57:29 +0200 Subject: [PATCH 3/8] docs: tag protection, not branch protection, is what gates the release tag push Branch protection rules do not govern tag pushes; tag protection rules (or rulesets) do. The runbook prerequisite now names the right control. --- docs/RELEASE_PREP.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/RELEASE_PREP.md b/docs/RELEASE_PREP.md index 2b4087bc..d05afb4c 100644 --- a/docs/RELEASE_PREP.md +++ b/docs/RELEASE_PREP.md @@ -36,7 +36,8 @@ with a single `version` input (e.g. `2.0.0`, or `2.1.0.beta1`): - **Both** gem names must trust this repo + `release.yml` as a trusted publisher on rubygems.org — `capybara-screenshot-diff` **and** `snap_diff-capybara`. If only one does, step 4 or 5 fails after the tag is already pushed; re-dispatch after fixing. -- Branch protection on `master` must allow the workflow's tag push. +- If tag protection rules are ever added to this repo, `v*` must allow + `github-actions[bot]` to push — that is how step 3 creates the tag. ## Before you dispatch From 3914c3d5ba870509811aec3c2846b49076c46dbc Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 07:58:48 +0200 Subject: [PATCH 4/8] docs: the per-screenshot driver: override dies quietly, the config setting dies loudly Both were lumped together as "raises NoMethodError on 2.1". Only the config setting does. Per-screenshot options are a free-form hash, so on 2.1 `screenshot "index", driver: :vips` is inert and nothing tells you the line is dead -- #249's own upgrade note spells out the split. Grep-for-it advice added, since that is the only signal a user gets. --- CHANGELOG.md | 9 +++++---- docs/UPGRADING.md | 19 +++++++++++-------- docs/drivers.md | 3 ++- 3 files changed, 18 insertions(+), 13 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index a9cfbe6b..87eb9ef3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -95,10 +95,11 @@ contract one release ahead is the mitigation. 2.0 warns once per process for eac | `include SnapDiff::Driver` in your own driver | nothing — custom drivers have no migration path | Two removals 2.0 cannot warn about, so they are written down instead: **`driver:` -as a setting goes away entirely** (`SnapDiff.config.driver = :vips` and -`Capybara::Screenshot::Diff.driver = :vips` raise `NoMethodError` on 2.1 — delete -the line, one backend needs no selection), and the legacy `LOADED_DRIVERS` / -`AVAILABLE_DRIVERS` constants are plain aliases with nothing to hook. +as a setting goes away entirely** — `SnapDiff.config.driver = :vips` and +`Capybara::Screenshot::Diff.driver = :vips` raise `NoMethodError` on 2.1, and the +per-screenshot `screenshot "x", driver: :vips` is silently ignored there. Delete +both; one backend needs no selection. The legacy `LOADED_DRIVERS` / +`AVAILABLE_DRIVERS` constants are also plain aliases with nothing to hook. ### Added - **`SnapDiff` is the canonical namespace** — the implementation lives in diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index f3137a38..b9fd5f65 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -216,14 +216,17 @@ warns once per process per subject, through the same channel and the same silenc | `include SnapDiff::Driver` in your own driver class | the driver mixin | nothing — see below | > **The one removal on this list that 2.0 cannot warn you about: the `driver:` setting -> itself.** `SnapDiff.config.driver = :vips` (and the legacy -> `Capybara::Screenshot::Diff.driver = :vips`, and the per-screenshot `driver:` override) -> is **silent** in 2.0 and raises `NoMethodError: undefined method 'driver='` in 2.1. -> Warning on it would fire on the recommended configuration, so this note is the warning: -> **delete the line.** With libvips the only backend there is nothing to select, and the -> default just works. The same goes for `driver: :auto` on a machine that *has* -> `ruby-vips` — the `:auto` warning above only fires when `:auto` actually falls back to -> ChunkyPNG, because that is the case where 2.1 stops the process comparing at all. +> itself.** `SnapDiff.config.driver = :vips` and the legacy +> `Capybara::Screenshot::Diff.driver = :vips` are **silent** in 2.0 and raise +> `NoMethodError: undefined method 'driver='` in 2.1, at config time before any test runs. +> The per-screenshot form — `screenshot "index", driver: :vips` — is silent in 2.0 **and** +> in 2.1: per-screenshot options are a free-form hash, so an unknown key is simply inert. +> Warning on any of this would fire on the recommended configuration, so this note is the +> warning: **delete the line, and grep for the per-screenshot one.** With libvips the only +> backend there is nothing to select, and the default just works. The same goes for +> `driver: :auto` on a machine that *has* `ruby-vips` — the `:auto` warning above only +> fires when `:auto` actually falls back to ChunkyPNG, because that is the case where 2.1 +> stops the process comparing at all. ``` [snap_diff deprecation] `driver: :auto` selected chunky_png because libvips is not available in this process. The chunky_png driver is REMOVED in 2.1, when libvips (the `ruby-vips` gem) becomes required -- install it now, or this setup stops comparing on 2.1. See docs/drivers.md. Silence with `SnapDiff.silence_deprecations = true` or SNAP_DIFF_SILENCE_DEPRECATIONS=1. (shown once per process) (called from /app/test/test_helper.rb:12) diff --git a/docs/drivers.md b/docs/drivers.md index b9608177..69792949 100644 --- a/docs/drivers.md +++ b/docs/drivers.md @@ -16,7 +16,8 @@ following still works. Most of it warns once per process naming 2.1; the rows ma | Removed in 2.1 | What to do in 2.0 | |---|---| | the `:chunky_png` driver | add `gem "ruby-vips"` to your Gemfile and drop `driver: :chunky_png` | -| the `driver:` setting itself — `SnapDiff.config.driver =`, the legacy `Capybara::Screenshot::Diff.driver =`, and the per-screenshot `driver:` override (**silent**: on 2.1 they raise `NoMethodError`) | delete the line; one backend needs no selection | +| the `driver:` setting itself — `SnapDiff.config.driver =` and the legacy `Capybara::Screenshot::Diff.driver =` (**silent in 2.0**; on 2.1 they raise `NoMethodError` at config time) | delete the line; one backend needs no selection | +| the per-screenshot `driver:` override — `screenshot "index", driver: :vips` (**silent in 2.0 *and* 2.1**: per-screenshot options are a free-form hash, so an unknown key is simply inert) | delete the option, and grep for it — nothing will tell you the line is dead | | `driver: :auto` (and the `:auto` default) — warns **only when `:auto` actually falls back to ChunkyPNG**, i.e. when `ruby-vips` is missing; **silent** otherwise | with one backend there is nothing to choose; install `ruby-vips` and the default just works | | `shift_distance_limit` | ChunkyPNG-only. Use `median_filter_window_size`, `tolerance` or `color_distance_limit` — see [Configuration](configuration.md#allowed-shift-distance) | | `SnapDiff::Driver` (the custom-driver mixin) | nothing — see below | From 6e5938d7942152df57af6d098e4c1a32a2f50af6 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 08:13:40 +0200 Subject: [PATCH 5/8] docs: pin versions, and stop telling people to delete baselines Second pass, from customer-persona findings. Each verified here before acting; two of the four reported items turned out to be artifacts of the PUBLISHED beta3 rather than of master, and are handled as such. Baselines (the oldest bug in the tracker: #5 and #6 in 2018, #133 in 2024) - README told users to "delete the baseline and re-run" in two places. It cannot work. `Vcs.checkout_vcs` (lib/snap_diff/vcs.rb:24) resolves every baseline with `git show HEAD:`, and `ScreenshotMatcher#check_base_screenshot` calls it before `need_to_compare?` tests `base_path.exist?` -- so a committed baseline is fetched from HEAD no matter what the working tree says, and `rm` changes nothing. - New first-class "Accepting an intentional change" section: the mechanism, the commit that actually accepts it, and the surprising part -- staging is not enough, so no local run goes green until you commit. The FAQ answer now says the same thing instead of the opposite. - Deliberately does NOT document RECORD_SCREENSHOTS. It is printed by our own error message (screenshot_matcher.rb:73) but read nowhere in lib/; a separate lane is implementing it, and it should be documented once it works, not before. Version pinning - `gem "snap_diff-capybara"` unpinned installs 0.0.1 -- a placeholder whose entire payload is one README and zero Ruby files (verified by fetching and unpacking it), so the user gets an immediate LoadError. And unpinned `gem "capybara-screenshot-diff"` resolves to 1.15.1, not to the 2.0 the surrounding prose is selling. Every install instruction now pins, and the README says plainly that the mirror name is not the one to reach for. CHANGELOG, all verified - Failure messages leaked a libvips pointer struct via the comparison metadata; `to_h` excludes `diff_mask` since #234, which landed after the beta3 tag, so 2.0.0 final is the fix. - Known limitation: fork-parallel runs write no HTML report. Workers accumulate assertions per process; the report is written from `Minitest.after_run` in the parent (integrations/minitest.rb:69), which never sees them. Artifacts and pass/fail are unaffected. - A note for anyone sitting on a prerelease: beta3's deprecation channel was incomplete, so its silence is not evidence of being migrated. Constants - `Capybara::Screenshot::Os` -> `SnapDiff::Os` was in no rename table. Gemspec - rubygems_mfa_required. The four URI fields were added in the first commit. --- CHANGELOG.md | 16 ++++++++++++ README.md | 42 ++++++++++++++++++++++++++++---- capybara-screenshot-diff.gemspec | 1 + docs/UPGRADING.md | 11 +++++++-- docs/migration-guide.md | 2 +- 5 files changed, 64 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 87eb9ef3..5919ea9f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,11 @@ site: (constant); use `SnapDiff::Comparison` instead. (called from test/test_helper.rb:12) ``` +> **On a 2.0.0 prerelease? Upgrade, do not trust its silence.** In `2.0.0.beta3` the +> deprecation channel was incomplete: a v1-only suite got **no** warnings at all, and +> the driver-half removal warnings did not exist yet. Silence on a beta is not evidence +> that you are migrated. + Requiring the gem, the DSL, settings accessors and the eagerly-defined constants (the error classes, `::VERSION`, `Os`, `Region`, `Reporters::Default`, `LOADED_DRIVERS`, `AVAILABLE_DRIVERS`) are **silent by design** — @@ -146,8 +151,19 @@ both; one backend needs no selection. The legacy `LOADED_DRIVERS` / comparison previously raised `NameError` there - `require "snap_diff/integrations/…"` loads the full `SnapDiff` surface, and `gem "snap_diff-capybara"` works with `Bundler.require` +- **Failure messages no longer dump a libvips pointer struct.** The comparison + metadata carried the raw `diff_mask` image into the error text + (`"diff_mask":{"ptr":{}…}`); it is excluded now, leaving the metrics - Reporter failure warnings use one brand and name the failing reporter class +### Known limitations +- **Fork-based parallel tests produce no HTML report.** Under Minitest's forked + parallel executor (`parallelize(workers: N)`, the Rails default), each worker + accumulates its assertions in its own process, while the report is written from + `Minitest.after_run` in the parent — which never sees them. Pass/fail is correct + and the diff image artifacts are still written; only the HTML report is missing. + Fixed in 2.1 + ### Unchanged - Ruby 3.2+, Capybara `>= 2, < 4`, the `screenshot` / `assert_matches_screenshot` DSL, every capture and comparison option, baseline file names and formats diff --git a/README.md b/README.md index b9757ae0..8c5a0b4e 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ Stop shipping UI bugs. Take screenshots in your Capybara tests, commit baselines > > **2.1 removes what 2.0 warns about**: the legacy namespaces, the ChunkyPNG driver, `shift_distance_limit`, the `driver:` setting and the driver abstraction — libvips becomes the only backend. There is no 3.0. Writing new code? Start from [SnapDiff — the canonical API](docs/snapdiff.md), which uses canonical names only. Migrating an existing suite? See the [upgrade guide](docs/UPGRADING.md). > -> **Two gem names, one gem.** `capybara-screenshot-diff` is the name to install. The same content is also published as [`snap_diff-capybara`](https://rubygems.org/gems/snap_diff-capybara) — identical version, forward-looking name matching this repository — so that name is reserved and resolvable. **Install one, never both**: with both in a Gemfile the gem raises `SnapDiff::DualInstallError` at require time. +> **Two gem names, one gem — install `capybara-screenshot-diff`.** From 2.0.0 on, the identical content is also published as [`snap_diff-capybara`](https://rubygems.org/gems/snap_diff-capybara), the forward-looking name matching this repository. Do not reach for it yet: that name's only non-prerelease before 2.0.0 is a `0.0.1` placeholder containing a README and no Ruby files, so an unpinned `gem "snap_diff-capybara"` installs an empty gem and fails with `LoadError`. **Always pin the version**, and **install one name, never both** — with both in a Gemfile the gem raises `SnapDiff::DualInstallError` at require time. ## Quick Start (5 minutes) @@ -23,8 +23,8 @@ Stop shipping UI bugs. Take screenshots in your Capybara tests, commit baselines ```ruby # Gemfile -gem 'capybara-screenshot-diff' -gem 'ruby-vips' # The image backend. Needs libvips — see Installation below +gem 'capybara-screenshot-diff', '~> 2.0' # pin: unpinned resolves to the 1.x line +gem 'ruby-vips' # The image backend. Needs libvips — see Installation below ``` The gem ships no image backend of its own. Add `ruby-vips` (recommended, and the only @@ -107,7 +107,7 @@ Screenshot does not match for 'homepage': ({"area_size":1250,"region":[0,19,199,83],"max_color_distance":42.5}) ``` -Open `doc/screenshots/homepage.diff.png` to see exactly what changed. If the change is intentional, delete the baseline and re-run to update it. +Open `doc/screenshots/homepage.diff.png` to see exactly what changed. If the change is intentional, see [Accepting an intentional change](#accepting-an-intentional-change). | File | Description | |------|-------------| @@ -115,6 +115,38 @@ Open `doc/screenshots/homepage.diff.png` to see exactly what changed. If the cha | `homepage.diff.png` | Visual diff with changes highlighted in red | | `homepage.heatmap.diff.png` | Heatmap of pixel differences | +## Accepting an intentional change + +**Baselines are read from git, not from your working directory.** Every comparison runs +`git show HEAD:` for the baseline, so a screenshot that is committed is the one you +are compared against — no matter what the file on disk says. + +That makes the obvious move the wrong one: **deleting the baseline file does nothing.** The +gem fetches the committed copy from `HEAD` and the test fails exactly as before. + +Accepting a change is therefore a **commit**, not a file operation. The run writes its new +capture to the baseline path, so `git status` shows the baseline as modified — review it and +commit it: + +```bash +git status # doc/screenshots/homepage.png is modified +git diff --stat doc/screenshots/ + +# Look at homepage.diff.png. If the change is what you wanted: +git add doc/screenshots/homepage.png +git commit -m "chore: update homepage baseline" + +bundle exec rake test # now green — HEAD holds the new baseline +``` + +> **Staging is not enough.** `git add` alone does not move `HEAD`, so a staged-but-uncommitted +> baseline is still compared against the old committed one. You cannot get a green local run +> until you commit. That is by design: the baseline under review in a pull request is exactly +> the baseline the suite uses. + +Reviewing the change is what the pull request is for — the updated `.png` shows up as an image +diff next to the code that caused it. + ## Web UI for Reviewing Screenshot Changes Add one line to get an interactive dashboard for reviewing all screenshot differences: @@ -174,7 +206,7 @@ Yes. First run saves baselines and always passes. Run tests again to compare aga
How do I update baselines after intentional UI changes? -Delete the baseline file and re-run tests: `rm doc/screenshots/homepage.png && bundle exec rake test`. Or update all: `rm -rf doc/screenshots/ && bundle exec rake test`. +**Not by deleting the file** — baselines are read from git (`git show HEAD:`), so `rm` has no effect on what you are compared against. Commit the new capture instead: `git add doc/screenshots/homepage.png && git commit`. See [Accepting an intentional change](#accepting-an-intentional-change).
diff --git a/capybara-screenshot-diff.gemspec b/capybara-screenshot-diff.gemspec index efa095d5..01853679 100644 --- a/capybara-screenshot-diff.gemspec +++ b/capybara-screenshot-diff.gemspec @@ -20,6 +20,7 @@ Gem::Specification.new do |spec| spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/master/CHANGELOG.md" spec.metadata["bug_tracker_uri"] = "#{spec.homepage}/issues" spec.metadata["documentation_uri"] = "#{spec.homepage}/blob/master/docs/snapdiff.md" + spec.metadata["rubygems_mfa_required"] = "true" # Allow-list: everything a consumer needs at runtime plus the shipped docs. # Build/dev files (gems.rb, Rakefile, the gemspec itself, tests, CI) stay out. spec.files = `git ls-files -z`.split("\x0") diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index b9fd5f65..8e777e85 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -25,8 +25,14 @@ with canonical names only, no legacy shapes to unlearn. gem "capybara-screenshot-diff", "~> 2.0" ``` -The same content is also published as `snap_diff-capybara`. Install **one** — with both -in a Gemfile the gem raises `SnapDiff::DualInstallError` at require time. +**Pin the version.** An unpinned `gem "capybara-screenshot-diff"` resolves to the 1.x line, +not to 2.0. + +The same content is also published as `snap_diff-capybara` from 2.0.0 on, but that name's +only earlier non-prerelease is a `0.0.1` placeholder with no Ruby files in it — unpinned, it +installs an empty gem and raises `LoadError`. Stay on `capybara-screenshot-diff`, and install +**one** name: with both in a Gemfile the gem raises `SnapDiff::DualInstallError` at require +time. ```bash bundle install @@ -48,6 +54,7 @@ The implementation now lives in `lib/snap_diff/` under the `SnapDiff` namespace. | `Capybara::Screenshot::Diff::ImageCompare` | `SnapDiff::Comparison` | | `Capybara::Screenshot::Diff::Difference` | `SnapDiff::ComparisonResult` | | `Capybara::Screenshot::Diff::Drivers::BaseDriver` | `SnapDiff::Driver` (now a mixin — see below) | +| `Capybara::Screenshot::Os` | `SnapDiff::Os` | | `CapybaraScreenshotDiff::SnapManager` / `::Snap` | `SnapDiff::SnapManager` / `SnapDiff::Snap` | | `CapybaraScreenshotDiff::RED_RGBA` / `::ORANGE_RGBA` | `SnapDiff::RED_RGBA` / `SnapDiff::ORANGE_RGBA` | | `CapybaraScreenshotDiff::Minitest::Assertions` | `SnapDiff::Minitest::Assertions` | diff --git a/docs/migration-guide.md b/docs/migration-guide.md index 2859aedc..3d56e0a4 100644 --- a/docs/migration-guide.md +++ b/docs/migration-guide.md @@ -36,7 +36,7 @@ end **After (capybara-screenshot-diff):** ```ruby # Gemfile -gem 'capybara-screenshot-diff' +gem 'capybara-screenshot-diff', '~> 2.0' # test helper require 'capybara_screenshot_diff/minitest' From e4e7154074b91e74f23e2406797679936ad16473 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 08:46:36 +0200 Subject: [PATCH 6/8] docs: stop teaching two commands that do not work Two customer personas independently followed the docs and got a green bar on a page they had deliberately broken. `rake test` does not run `test/system/` in a Rails app. The Quick Start told users to run it, so step 1 produced `0 runs` and no baselines -- which reads as a pass. The example is a Rails system test; the command now matches it, with a callout, because "0 runs" is the single easiest way to believe visual testing is working when nothing is running. `RECORD_SCREENSHOTS=1` appeared in three user-facing docs for a feature that has never existed in `lib/` -- it is this repository's own test-suite convention, read by `test/test_helper.rb`. The user-facing copies are replaced with the flow that actually works: run the suite, which rewrites every changed baseline in place, then `git add` and commit. The contributor page keeps it and now says plainly that it is not a library feature. --- README.md | 13 +++++++++---- docs/ci-integration.md | 11 +++++++---- docs/docker-testing.md | 6 ++++++ docs/migration-guide.md | 10 +++++++--- 4 files changed, 29 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index 8c5a0b4e..5934067c 100644 --- a/README.md +++ b/README.md @@ -59,16 +59,21 @@ Then run these steps in order: ```bash # Step 1: Save baselines (first run always passes) -bundle exec rake test +bin/rails test:system # Step 2: Commit baselines to git git add doc/screenshots/ git commit -m "chore: add screenshot baselines" # Step 3: Now comparisons work — change your UI and re-run -bundle exec rake test +bin/rails test:system ``` +> **Run the task that actually runs system tests.** In a Rails app, `rake test` +> and `rails test` skip `test/system/` — you get `0 runs` and no baselines, which +> looks like a pass. Use `rails test:system` (or `rails test test/system`). +> Outside Rails, run whatever task loads your Capybara tests. + After Step 1, you'll see: ```text doc/screenshots/ @@ -136,7 +141,7 @@ git diff --stat doc/screenshots/ git add doc/screenshots/homepage.png git commit -m "chore: update homepage baseline" -bundle exec rake test # now green — HEAD holds the new baseline +bin/rails test:system # now green — HEAD holds the new baseline ``` > **Staging is not enough.** `git add` alone does not move `HEAD`, so a staged-but-uncommitted @@ -230,7 +235,7 @@ Comparisons add ~50ms per image with VIPS. If you add `chunky_png` to your Gemfi
Debug mode -`DEBUG=1 bundle exec rake test` keeps `.diff.png` files for inspection. +`DEBUG=1 bin/rails test:system` keeps `.diff.png` files for inspection.
## Installation diff --git a/docs/ci-integration.md b/docs/ci-integration.md index 65d0806f..f61270ab 100644 --- a/docs/ci-integration.md +++ b/docs/ci-integration.md @@ -184,10 +184,13 @@ jobs: ## Update Baselines in CI -When intentional UI changes are made, baselines need to be re-recorded. You can do this locally: +When intentional UI changes are made, baselines need to be re-recorded. Baselines are +read from git, so accepting a change is a commit — the failing run has already written +the new capture to the baseline path: ```bash -RECORD_SCREENSHOTS=1 bundle exec rake test +bin/rails test:system # fails, and rewrites the changed baselines +git status # review what moved git add test/fixtures/screenshots/ git commit -m "chore: update screenshot baselines" ``` @@ -226,8 +229,8 @@ jobs: cache-apt-packages: true - name: Record new baselines - run: RECORD_SCREENSHOTS=1 bundle exec rake test - continue-on-error: true + run: bin/rails test:system + continue-on-error: true # the run fails by design; it rewrites the baselines - name: Commit updated baselines run: | diff --git a/docs/docker-testing.md b/docs/docker-testing.md index 5dd861fd..f782e0a8 100644 --- a/docs/docker-testing.md +++ b/docs/docker-testing.md @@ -13,6 +13,12 @@ This builds a Docker image with Chrome and runs the test suite against three Cap ## Recording baseline screenshots +> **This page is for contributors to this gem, not for users of it.** `bin/dtest` and +> `RECORD_SCREENSHOTS` are this repository's own test harness. `RECORD_SCREENSHOTS` is +> read by `test/test_helper.rb` — it is **not** a feature of the library and does +> nothing in your application. To accept a change in your own app, see +> [Accepting an intentional change](../README.md#accepting-an-intentional-change). + Screenshot baselines are committed to the repo and compared against during tests. When you set up the project for the first time, or after upgrading the browser/driver, you need to re-record them: ```bash diff --git a/docs/migration-guide.md b/docs/migration-guide.md index 3d56e0a4..68371017 100644 --- a/docs/migration-guide.md +++ b/docs/migration-guide.md @@ -246,9 +246,13 @@ end ### "My baselines are on Percy/Chromatic servers" -You'll need to take fresh screenshots. Either: -- Visit each page and capture manually -- Run tests with `RECORD_SCREENSHOTS=1` to generate all baselines at once +You'll need to take fresh screenshots. Run your system tests once — every screenshot +is written to its baseline path, whether or not it matched — then commit them: + +```bash +bin/rails test:system +git add doc/screenshots/ && git commit -m "chore: record baselines" +``` ### "I had hundreds of BackstopJS scenarios" From 658cdbb1a55c2074e0f2fa068f985942ee6c20db Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 09:30:31 +0200 Subject: [PATCH 7/8] docs: make the Quick Start something a new user can actually run A reviewer built a stock Rails 8.1 app, ran the documented Quick Start end to end, and none of it worked. Fixes, each verified against a scratch app or against lib/: - `gem "capybara-screenshot-diff", "~> 2.0"` does not resolve. rubygems has 1.15.1 and 2.0.0.alpha1..beta3 and no final 2.x, and Bundler never picks a prerelease from a plain requirement, so `bundle install` fails with `Could not find gem 'capybara-screenshot-diff (~> 2.0)'`. All five install snippets now pin `2.0.0.beta3` and say why. RELEASE_PREP gains the step that swaps them back to `~> 2.0` as part of the 2.0.0 push, so the good pin lands with the release rather than before it. - The Quick Start taught the API 2.1 deletes, silently. `require "capybara_screenshot_diff/minitest"` + `include CapybaraScreenshotDiff::Minitest::Assertions` print nothing: they are eager aliases, so `const_missing` never fires. The Quick Start now starts on canonical `SnapDiff`, and README/CHANGELOG/snapdiff.md say which doors actually warn (config accessors, `include`, `default_options`, `const_missing`) and which cannot. - The CI "Record new baselines" job could not record a new baseline. `check_base_screenshot` runs before `capture_screenshot` and `fail_if_new` is true whenever `ENV["CI"]` is set, so a new screenshot raises before anything is written and the commit step finds nothing. The job now clears `CI` for that step. - `bundle exec rake test` / `rails test` swept out of the three CI workflows, "The Short Version", and the historical upgrade sections: in a Rails app they skip `test/system/` and report `0 runs`. - The delete-the-baselines block in UPGRADING replaced with the commit workflow the README documents. - `docs/drivers.md` told you to delete the `driver:` setting on one screen and to add it on another; same for configuration.md, migration-guide.md. - `rescue SnapDiff::Error` does not catch a failed assertion under the framework integrations -- Minitest converts it to `Minitest::Assertion` and RSpec to `ExpectationNotMetError`. Said so. - "2.0 will not rewrite a baseline you already committed" contradicted the README and reality: a failing run does rewrite the baseline path. Reworded to what is meant (no re-encoding) plus what actually happens. Also, all verified in a scratch app: the `git add test/fixtures/screenshots/` path was never the default (`doc/screenshots` is); the example failure output showed a `max_color_distance` key the vips path never emits; the artifact table listed three of five files; `application_system_test_case.rb` omitted both `require "test_helper"` and `driven_by` (without the latter the same page captures at 2800x1610 instead of 1400x1257); `homepage_test.rb` omitted `require "application_system_test_case"` and raised NameError as printed; and `DEBUG=1` never had anything to do with keeping `.diff.png` files. Docs only. rake test:unit 610/0, standardrb clean. --- CHANGELOG.md | 29 +++++++++++++----- README.md | 67 ++++++++++++++++++++++++++++++++--------- docs/RELEASE_PREP.md | 8 +++++ docs/UPGRADING.md | 36 +++++++++++++--------- docs/ci-integration.md | 34 ++++++++++++++++----- docs/configuration.md | 6 ++-- docs/drivers.md | 11 ++++--- docs/framework-setup.md | 13 ++++++-- docs/migration-guide.md | 27 +++++++++++------ docs/snapdiff.md | 34 ++++++++++++++++++--- 10 files changed, 197 insertions(+), 68 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5919ea9f..d1024fb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,21 +18,30 @@ kept as history. This entry is the one to read if you are coming from **1.15.1** ### Upgrading from 1.15.1: change the version, run your suite ```ruby -gem "capybara-screenshot-diff", "~> 2.0" +gem "capybara-screenshot-diff", "2.0.0.beta3" # current 2.0 prerelease; 2.0.0 final is not out yet ``` +Pin the exact prerelease until 2.0.0 ships: Bundler never resolves a prerelease from a plain +requirement, so `"~> 2.0"` fails with `Could not find gem 'capybara-screenshot-diff (~> 2.0)'`. +From 2.0.0 on, `"~> 2.0"` is the pin. + That is the whole migration. `screenshot`, `assert_matches_screenshot` and `capture_screenshot` are unchanged. Your `Capybara::Screenshot::Diff.configure` block, every `Capybara::Screenshot.*` / `Capybara::Screenshot::Diff.*` setting, and every legacy constant still resolve — to the *same objects* the new names resolve to. Baselines are unchanged: capture, encoding, file naming and the `png` default are -the same code as 1.15.1, moved — 2.0 will not rewrite a baseline you already committed. +the same code as 1.15.1, moved — upgrading does not re-encode or invalidate a baseline you +already committed, and a matching screenshot stays byte-identical. (Unchanged from 1.x: a +screenshot that *differs* is written to its baseline path, so a failing run leaves +`doc/screenshots/` dirty. That is how you accept a change — review the diff and commit.) Rolling back is a Gemfile edit: pin `"~> 1.15"` and `bundle update`. ### What you will see in your test output -One migration notice, the first time the process touches a v1 API: +One migration notice, the first time the process goes through a v1 door that *can* be +hooked — a legacy config accessor, `include Capybara::Screenshot[::Diff]`, +`Capybara::Screenshot::Diff.default_options`, or a lazily shimmed legacy constant: ```text [snap_diff deprecation] This process uses the v1 `Capybara::Screenshot*` / @@ -50,10 +59,16 @@ site: (constant); use `SnapDiff::Comparison` instead. (called from test/test_helper.rb:12) ``` -> **On a 2.0.0 prerelease? Upgrade, do not trust its silence.** In `2.0.0.beta3` the -> deprecation channel was incomplete: a v1-only suite got **no** warnings at all, and -> the driver-half removal warnings did not exist yet. Silence on a beta is not evidence -> that you are migrated. +> **Silence is not evidence that you are migrated — on any 2.0 build.** In `2.0.0.beta3` +> the deprecation channel was incomplete: a v1-only suite got **no** warnings at all, and +> the driver-half removal warnings did not exist yet. 2.0.0 fixes the config-accessor, +> `include`, `default_options` and `const_missing` doors, but **a suite whose only contact +> with the v1 API is `require "capybara_screenshot_diff/minitest"` + +> `include CapybaraScreenshotDiff::Minitest::Assertions` still prints nothing** — those +> names are eager aliases, so there is no `const_missing` to hook. That setup is removed in +> 2.1 all the same. Do not use warning output as a migration checklist; use +> [docs/UPGRADING.md](docs/UPGRADING.md#deprecation-warnings), which lists what warns and +> what cannot. Requiring the gem, the DSL, settings accessors and the eagerly-defined constants (the error classes, `::VERSION`, `Os`, `Region`, `Reporters::Default`, diff --git a/README.md b/README.md index 5934067c..28e94b4e 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ Stop shipping UI bugs. Take screenshots in your Capybara tests, commit baselines **Why this gem?** Baselines live in git — review UI changes in pull requests like you review code. Runs offline, works in CI, zero vendor lock-in. Unlike Percy/Chromatic (paid SaaS), nothing to sign up for. Unlike BackstopJS, no Node required. -> **2.0 is the transitional release.** The gem's canonical namespace is now `SnapDiff`. Upgrading from 1.x is a version bump — every legacy `Capybara::Screenshot::Diff` / `CapybaraScreenshotDiff` name still resolves to the same object and keeps working. The first legacy API a process touches prints one migration notice (lazily shimmed constants also warn once each — see [which names warn](docs/UPGRADING.md#deprecation-warnings)), silenceable via `SnapDiff.silence_deprecations = true` or `SNAP_DIFF_SILENCE_DEPRECATIONS=1`. +> **2.0 is the transitional release.** The gem's canonical namespace is now `SnapDiff`. Upgrading from 1.x is a version bump — every legacy `Capybara::Screenshot::Diff` / `CapybaraScreenshotDiff` name still resolves to the same object and keeps working. A legacy **config accessor**, an `include Capybara::Screenshot[::Diff]`, `Diff.default_options`, or a lazily shimmed legacy constant prints one migration notice per process (shimmed constants also warn once each). **The legacy integration require is not one of those doors** — `require "capybara_screenshot_diff/minitest"` plus `include CapybaraScreenshotDiff::Minitest::Assertions` is silent by design, because those names are eager aliases with no `const_missing` to hook. See [which names warn](docs/UPGRADING.md#deprecation-warnings). Silence the ones that do via `SnapDiff.silence_deprecations = true` or `SNAP_DIFF_SILENCE_DEPRECATIONS=1`. > > **2.1 removes what 2.0 warns about**: the legacy namespaces, the ChunkyPNG driver, `shift_distance_limit`, the `driver:` setting and the driver abstraction — libvips becomes the only backend. There is no 3.0. Writing new code? Start from [SnapDiff — the canonical API](docs/snapdiff.md), which uses canonical names only. Migrating an existing suite? See the [upgrade guide](docs/UPGRADING.md). > @@ -23,28 +23,38 @@ Stop shipping UI bugs. Take screenshots in your Capybara tests, commit baselines ```ruby # Gemfile -gem 'capybara-screenshot-diff', '~> 2.0' # pin: unpinned resolves to the 1.x line -gem 'ruby-vips' # The image backend. Needs libvips — see Installation below +gem 'capybara-screenshot-diff', '2.0.0.beta3' # current 2.0 prerelease; 2.0.0 final is not out yet +gem 'ruby-vips' # The image backend. Needs libvips — see Installation below ``` +Pin the exact prerelease. Bundler never resolves a prerelease from a plain requirement, so +`'~> 2.0'` fails with `Could not find gem 'capybara-screenshot-diff (~> 2.0)'` until 2.0.0 +ships. Once it does, `'~> 2.0'` is the pin to use. + The gem ships no image backend of its own. Add `ruby-vips` (recommended, and the only backend from 2.1 on) or `chunky_png` (pure Ruby, no system library, removed in 2.1) — with neither, comparisons raise `Wrong adapter nil. Available adapters: []`. ```ruby # test/test_helper.rb -require 'capybara_screenshot_diff/minitest' +require "snap_diff/integrations/minitest" ``` ```ruby # test/application_system_test_case.rb +require "test_helper" + class ApplicationSystemTestCase < ActionDispatch::SystemTestCase - include CapybaraScreenshotDiff::Minitest::Assertions + driven_by :selenium, using: :headless_chrome, screen_size: [1400, 1400] + + include SnapDiff::Minitest::Assertions end ``` ```ruby # test/system/homepage_test.rb +require "application_system_test_case" + class HomepageTest < ApplicationSystemTestCase test "homepage" do visit "/" @@ -53,6 +63,19 @@ class HomepageTest < ApplicationSystemTestCase end ``` +> **Pin the browser.** `driven_by` is not optional decoration in a pixel-diffing suite. +> Rails falls back to a *visible* browser at whatever size and pixel ratio the machine +> gives it: the same page that captures as **1400x1257** with the line above captures as +> **2800x1610** without it, and every comparison then fails with `Dimensions have changed`. +> Both `require` lines matter too — Rails does not autoload `test/`, so dropping either +> one raises `NameError: uninitialized constant`. + +`SnapDiff::Minitest::Assertions` already includes `SnapDiff::DSL`, so no separate include is +needed. The legacy `require "capybara_screenshot_diff/minitest"` + +`include CapybaraScreenshotDiff::Minitest::Assertions` still work and resolve to these same +objects — 2.1 removes them, so new suites should start here. See +[SnapDiff — the canonical API](docs/snapdiff.md). + (`screenshot` still works as a shorthand, and is safe to override in your own helpers — the gem no longer calls it internally.) Then run these steps in order: @@ -92,13 +115,14 @@ snap_diff_report.html If you skip Step 2 and push to CI, the build will fail — `fail_if_new` is `true` by default in CI. -For RSpec, Cucumber, or non-Rails setup, see [Framework Setup](docs/framework-setup.md). +For RSpec, Cucumber, or non-Rails setup, see [SnapDiff — the canonical API](docs/snapdiff.md#quick-start) +(or [Framework Setup](docs/framework-setup.md) for the same wiring in legacy names). ### For Non-Rails Projects (Hugo, Jekyll, Static Sites) ```ruby -require 'capybara_screenshot_diff/static' -CapybaraScreenshotDiff.serve("_site") # or "public", "build", "dist" +require "snap_diff/static" +SnapDiff.serve("_site") # or "public", "build", "dist" ``` Then commit baselines to git just like Rails. [Full setup](docs/ci-integration.md#non-rails-projects-hugo-jekyll-static-sites). @@ -108,18 +132,27 @@ Then commit baselines to git just like Rails. [Full setup](docs/ci-integration.m The test fails with a clear message and generates diff files: ```text -Screenshot does not match for 'homepage': -({"area_size":1250,"region":[0,19,199,83],"max_color_distance":42.5}) +Screenshot does not match for 'homepage': ({"area_size":41520.0,"region":[8.0,8.0,1392.0,38.0]}) +doc/screenshots/homepage.png +doc/screenshots/homepage.base.diff.png +doc/screenshots/homepage.diff.png +doc/screenshots/homepage.heatmap.diff.png ``` Open `doc/screenshots/homepage.diff.png` to see exactly what changed. If the change is intentional, see [Accepting an intentional change](#accepting-an-intentional-change). +A failing run leaves five files behind — the rewritten baseline plus four artifacts: + | File | Description | |------|-------------| -| `homepage.png` | Committed baseline | -| `homepage.diff.png` | Visual diff with changes highlighted in red | +| `homepage.png` | Baseline path — **rewritten** with the new capture (`git status` shows it modified) | +| `homepage.base.png` | The committed baseline, checked out of `HEAD` for the comparison | +| `homepage.diff.png` | The new capture, with changed regions highlighted | +| `homepage.base.diff.png` | The old baseline, with the same regions highlighted | | `homepage.heatmap.diff.png` | Heatmap of pixel differences | +Only `homepage.png` is committed; the `.gitignore` above keeps the other four out. + ## Accepting an intentional change **Baselines are read from git, not from your working directory.** Every comparison runs @@ -235,12 +268,18 @@ Comparisons add ~50ms per image with VIPS. If you add `chunky_png` to your Gemfi
Debug mode -`DEBUG=1 bin/rails test:system` keeps `.diff.png` files for inspection. +You do not need a flag to keep the diff images — a failing run leaves `.diff.png`, +`.base.diff.png`, `.heatmap.diff.png` and `.base.png` on disk and nothing in the gem +deletes them (`SnapManager#cleanup!` is this repository's own test-harness call, not +something your suite runs). + +`DEBUG=1` does one thing: it makes the HTML reporter print why it skipped an assertion +instead of failing quietly — useful when `snap_diff_report.html` is missing entries.
## Installation -**Requirements:** Ruby 3.2+, Capybara 2–3. Rails 7.1+ for Rails integration; non-Rails projects supported via `CapybaraScreenshotDiff.serve()`. For the `:vips` driver (recommended, and the only backend from 2.1 on): [libvips 8.9+](https://libvips.github.io/libvips/install.html). On macOS: `brew install vips`. On Ubuntu: `apt-get install libvips-dev`. +**Requirements:** Ruby 3.2+, Capybara 2–3. Rails 7.1+ for Rails integration; non-Rails projects supported via `SnapDiff.serve()`. For the `:vips` driver (recommended, and the only backend from 2.1 on): [libvips 8.9+](https://libvips.github.io/libvips/install.html). On macOS: `brew install vips`. On Ubuntu: `apt-get install libvips-dev`. ## Docs diff --git a/docs/RELEASE_PREP.md b/docs/RELEASE_PREP.md index d05afb4c..a21101c2 100644 --- a/docs/RELEASE_PREP.md +++ b/docs/RELEASE_PREP.md @@ -49,6 +49,14 @@ with a single `version` input (e.g. `2.0.0`, or `2.1.0.beta1`): rather than from the previous prerelease. - [ ] Docs carry no stale version pins — `README.md`, `docs/UPGRADING.md`, `docs/snapdiff.md`. Grep for the previous version string. +- [ ] **Releasing 2.0.0: swap every install snippet from the `2.0.0.beta3` pin to + `"~> 2.0"`,** and delete the "final is not out yet" clauses next to them. Until + 2.0.0 exists on rubygems, `gem "capybara-screenshot-diff", "~> 2.0"` hard-fails + `bundle install` (Bundler never resolves a prerelease from a plain requirement), so + the good pin has to land *with* the release, not before it. The snippets live in + `README.md`, `CHANGELOG.md`, `docs/UPGRADING.md` (×2) and `docs/migration-guide.md` — + re-grep rather than trusting that list: + `grep -rn 'capybara-screenshot-diff.\{0,4\}2\.0' README.md CHANGELOG.md docs/` - [ ] `mise x ruby@4.0.6 -- bundle exec rake test` (full suite, both gates) and `mise x ruby@4.0.6 -- bundle exec standardrb` are green. - [ ] CI is green on `master` at the commit you are releasing — the workflow only diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index 8e777e85..41f54eab 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -22,11 +22,13 @@ with canonical names only, no legacy shapes to unlearn. ```ruby # In your Gemfile -gem "capybara-screenshot-diff", "~> 2.0" +gem "capybara-screenshot-diff", "2.0.0.beta3" # current 2.0 prerelease; 2.0.0 final is not out yet ``` -**Pin the version.** An unpinned `gem "capybara-screenshot-diff"` resolves to the 1.x line, -not to 2.0. +**Pin the exact prerelease.** An unpinned `gem "capybara-screenshot-diff"` resolves to the 1.x +line, and `"~> 2.0"` does not resolve at all — Bundler never picks a prerelease from a plain +requirement, so it fails with `Could not find gem 'capybara-screenshot-diff (~> 2.0)'`. Once +2.0.0 ships, `"~> 2.0"` is the pin to use. The same content is also published as `snap_diff-capybara` from 2.0.0 on, but that name's only earlier non-prerelease is a `0.0.1` placeholder with no Ruby files in it — unpinned, it @@ -36,9 +38,13 @@ time. ```bash bundle install -bundle exec rake test +bin/rails test:system # the task that actually runs your screenshot tests ``` +> **Not `rake test` / `rails test`.** In a Rails app those skip `test/system/`, so you get +> `0 runs, 0 assertions, 0 failures` — a clean-looking pass that compared nothing. Outside +> Rails, run whatever task loads your Capybara tests. + **That's it.** Your existing code works unchanged. The old namespaces (`Capybara::Screenshot::Diff`, `CapybaraScreenshotDiff`) are shimmed with deprecation warnings; the new one (`SnapDiff`) is available if you want to modernize. --- @@ -352,9 +358,9 @@ All settings and baselines are compatible with v1.x. Simply pin your Gemfile bac ### Summary Checklist -- [ ] Pin `gem "capybara-screenshot-diff", "~> 2.0"` in your Gemfile +- [ ] Pin `gem "capybara-screenshot-diff", "2.0.0.beta3"` in your Gemfile (`"~> 2.0"` once 2.0.0 ships) - [ ] Run `bundle install` -- [ ] Run your test suite to verify no regressions +- [ ] Run your system tests (`bin/rails test:system`, not `rake test`) to verify no regressions - [ ] Read the warnings it prints — each one names something 2.1 removes - [ ] Add `gem "ruby-vips"` if you are not already on it (2.1 makes libvips the only backend) - [ ] Drop `driver:` from your config — it is silent in 2.0 and gone in 2.1 @@ -383,7 +389,7 @@ gem 'capybara-screenshot-diff', '~> 1.13.0' ```bash bundle update capybara-screenshot-diff -bundle exec rake test +bin/rails test:system # `rake test` skips test/system/ — 0 runs, no comparison ``` **That's it!** Existing `screenshot` calls work unchanged. New methods available if needed. @@ -468,7 +474,7 @@ gem 'capybara-screenshot-diff', '~> 1.12.0' ```bash bundle update capybara-screenshot-diff -bundle exec rake test # Verify tests still pass +bin/rails test:system # Verify tests still pass (`rake test` skips test/system/) ``` **That's it!** The zero-config setup still works out of the box. Your existing screenshot comparisons will continue to work with v1.12.0. @@ -673,7 +679,6 @@ Use the new `Diff.configure` block: Capybara::Screenshot::Diff.configure do |screenshot, diff| screenshot.window_size = [1280, 1024] screenshot.stability_time_limit = 1 - diff.driver = :vips diff.tolerance = 0.0005 end ``` @@ -724,7 +729,7 @@ bundle update capybara-screenshot-diff ### Step 3: Run Tests ```bash -bundle exec rake test +bin/rails test:system # `rake test` skips test/system/ — 0 runs, no comparison ``` ### Step 4: Check for New Screenshot Failures @@ -778,13 +783,14 @@ Capybara::Screenshot.hide_caret = false Then re-record baselines with the new defaults (recommended): ```bash -# Delete old baselines -rm doc/screenshots/*.png +# Do NOT delete the baselines — they are read from git (`git show HEAD:`), so +# removing the files changes nothing. Re-recording is a commit. -# Run tests to generate new baselines -bundle exec rake test +# The run fails and rewrites every changed baseline in place +bin/rails test:system -# Commit new baselines +# Review the diffs, then commit the new baselines +git status git add doc/screenshots/ git commit -m "Re-record baselines with v1.12.0 defaults" ``` diff --git a/docs/ci-integration.md b/docs/ci-integration.md index f61270ab..ac296b4c 100644 --- a/docs/ci-integration.md +++ b/docs/ci-integration.md @@ -58,7 +58,7 @@ jobs: bundler-cache: true - name: Run tests - run: bundle exec rake test + run: bin/rails test:system # not `rake test` — it skips test/system/ - name: Upload screenshot reports if: failure() @@ -68,6 +68,10 @@ jobs: pr-comment: 'true' ``` +The three workflows on this page all run `bin/rails test:system`, because `rake test` and +`rails test` skip `test/system/` and report `0 runs` — a green CI job that compared nothing. +Not on Rails? Substitute whatever task loads your Capybara tests. + That's it. On failure, this will: - Upload diff images + HTML report as artifacts - Post a PR comment with links to the inline report and full artifact download @@ -129,7 +133,7 @@ jobs: ruby-version: '4.0' cache-apt-packages: true - - run: bundle exec rake test + - run: bin/rails test:system # not `rake test` — it skips test/system/ - uses: snap-diff/snap_diff-capybara/.github/actions/upload-screenshots@master if: failure() @@ -160,7 +164,7 @@ jobs: run: sudo apt-get install -y libvips-dev - name: Run tests - run: bundle exec rake test + run: bin/rails test:system # not `rake test` — it skips test/system/ - name: Upload screenshot report if: failure() @@ -191,10 +195,14 @@ the new capture to the baseline path: ```bash bin/rails test:system # fails, and rewrites the changed baselines git status # review what moved -git add test/fixtures/screenshots/ +git add doc/screenshots/ # the default save_path; adjust if you changed it git commit -m "chore: update screenshot baselines" ``` +With the [recommended `.gitignore`](../README.md#quick-start-5-minutes) in place, `git add +doc/screenshots/` stages only the baselines — the `.diff.png` / `.base.png` artifacts are +ignored. + Or add a workflow that maintainers can trigger manually:
@@ -228,24 +236,36 @@ jobs: ruby-version: '4.0' cache-apt-packages: true - - name: Record new baselines + - name: Record baselines run: bin/rails test:system continue-on-error: true # the run fails by design; it rewrites the baselines + env: + CI: "" # see below — without this, a NEW screenshot is never written - name: Commit updated baselines run: | git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" - git add test/fixtures/ doc/screenshots/ + git add doc/screenshots/ git diff --staged --quiet || git commit -m "chore: update screenshot baselines" git push ```
+> **`CI: ""` is what makes this job able to record a *new* baseline.** `fail_if_new` +> defaults to `true` whenever `ENV["CI"]` is set and non-empty, and the check runs +> **before** the capture ([`screenshot_matcher.rb`](https://github.com/snap-diff/snap_diff-capybara/blob/master/lib/snap_diff/screenshot_matcher.rb) +> — `check_base_screenshot` precedes `capture_screenshot`). So on a stock GitHub Actions +> runner a screenshot with no committed baseline raises `No existing screenshot found for +> …` and **nothing is written to disk** — the commit step then finds nothing to commit, +> whatever the error message suggests. Clearing `CI` for this one step (or setting +> `SnapDiff.config.fail_if_new = false`) lets both new and changed baselines land. +> *Changed* baselines are rewritten either way; only new ones need this. + **How it works:** 1. Go to Actions → "Update Screenshot Baselines" → "Run workflow" 2. Enter the branch name (e.g. your PR branch) -3. The workflow records new baselines, commits, and pushes +3. The workflow records new and changed baselines, commits, and pushes [← Back to README](../README.md) diff --git a/docs/configuration.md b/docs/configuration.md index 2406f5ed..b8af96d1 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -11,7 +11,6 @@ SnapDiff.configure do |config| config.stability_time_limit = 1 config.blur_active_element = true config.hide_caret = true - config.driver = :vips config.tolerance = 0.0005 config.color_distance_limit = 15 end @@ -26,12 +25,15 @@ Capybara::Screenshot::Diff.configure do |screenshot, diff| screenshot.stability_time_limit = 1 screenshot.blur_active_element = true screenshot.hide_caret = true - diff.driver = :vips diff.tolerance = 0.0005 diff.color_distance_limit = 15 end ``` +> **`driver:` is deliberately absent from both examples.** The setting is removed in 2.1 +> (`NoMethodError` at config time) and 2.0 cannot warn about it. Add `gem "ruby-vips"` and +> leave the selection alone — see [Drivers](drivers.md#removed-in-21-everything-on-this-page-except-vips). + `SnapDiff::Config` **is** the storage; the legacy accessors are thin delegators onto it. There is one source of truth, so a write through either surface is visible through the other — mixing them is safe, and you can migrate a suite one line at a time: diff --git a/docs/drivers.md b/docs/drivers.md index 69792949..42fbb8cc 100644 --- a/docs/drivers.md +++ b/docs/drivers.md @@ -97,12 +97,13 @@ There are several options to setup active driver: `:auto`, `:chunky_png` and `:v ## Enable VIPS image processing [Vips](https://www.rubydoc.info/gems/ruby-vips/Vips/Image) driver provides a faster comparison, -and could be enabled by adding `ruby-vips` to `Gemfile`. +and is enabled by adding `ruby-vips` to your `Gemfile` (plus the libvips system package). +That is the whole setup — with `ruby-vips` installed the default `:auto` already picks it. -If need to setup explicitly Vips driver, there are several ways to do this: - -* Globally: `Capybara::Screenshot::Diff.driver = :vips` -* Per screenshot option: `screenshot 'index', driver: :vips` +**Do not select it explicitly.** Both forms are on the 2.1 removal list at the top of this page: +`Capybara::Screenshot::Diff.driver = :vips` / `SnapDiff.config.driver = :vips` raises +`NoMethodError` at config time on 2.1, and the per-screenshot `screenshot 'index', driver: :vips` +becomes inert. Neither warns in 2.0 — delete the line and grep for the per-screenshot form. With enabled VIPS there are new alternatives to process differences, which are easier to find and support. For example, `shift_distance_limit` is a very heavy operation. Instead, use `median_filter_window_size`. diff --git a/docs/framework-setup.md b/docs/framework-setup.md index 392f1f8b..d54b772b 100644 --- a/docs/framework-setup.md +++ b/docs/framework-setup.md @@ -26,12 +26,16 @@ For Minitest, need to require `capybara_screenshot_diff/minitest`. In your test class, include the `CapybaraScreenshotDiff::Minitest::Assertions` module: ```ruby +# test/application_system_test_case.rb +require 'test_helper' require 'capybara_screenshot_diff/minitest' class ApplicationSystemTestCase < ActionDispatch::SystemTestCase - # Make the Capybara & Capybara Screenshot Diff DSLs available in tests - include CapybaraScreenshotDiff::DSL - # Make `assert_*` methods behave like Minitest assertions + # Pin the browser: window size and pixel ratio are inputs to every comparison + driven_by :selenium, using: :headless_chrome, screen_size: [1400, 1400] + + # Make `assert_*` methods behave like Minitest assertions. + # This already includes CapybaraScreenshotDiff::DSL — a separate include is not needed. include CapybaraScreenshotDiff::Minitest::Assertions def test_my_feature @@ -41,6 +45,9 @@ class ApplicationSystemTestCase < ActionDispatch::SystemTestCase end ``` +Run it with `bin/rails test:system` — `rake test` / `rails test` skip `test/system/` and +report `0 runs`. + ## RSpec To use the screenshot capturing and change detection features in your tests, diff --git a/docs/migration-guide.md b/docs/migration-guide.md index 68371017..8172ab0e 100644 --- a/docs/migration-guide.md +++ b/docs/migration-guide.md @@ -35,15 +35,19 @@ end **After (capybara-screenshot-diff):** ```ruby -# Gemfile -gem 'capybara-screenshot-diff', '~> 2.0' +# Gemfile — pin the exact prerelease; '~> 2.0' does not resolve until 2.0.0 ships +gem 'capybara-screenshot-diff', '2.0.0.beta3' # test helper -require 'capybara_screenshot_diff/minitest' +require "snap_diff/integrations/minitest" + +# test class — test/application_system_test_case.rb +require "test_helper" -# test class class ApplicationSystemTestCase < ActionDispatch::SystemTestCase - include CapybaraScreenshotDiff::Minitest::Assertions + driven_by :selenium, using: :headless_chrome, screen_size: [1400, 1400] + + include SnapDiff::Minitest::Assertions test "homepage" do visit '/' @@ -60,7 +64,7 @@ end | First run | Uploads to Percy | Saves locally, passes automatically | | CI setup | `PERCY_TOKEN` env var | GitHub Action (3 lines) | | Diff review | Percy dashboard | `snap_diff_report.html` or PR artifacts | -| Update baselines | Percy's "Approve" button | Delete file, re-run tests, commit | +| Update baselines | Percy's "Approve" button | Re-run tests (the run rewrites the baseline), review, commit — **not** delete: baselines are read from git | | Snapshot limits | Paid plan dependent | Unlimited | | Parallel builds | Built-in | Thread-safe with t-locals + mutex | @@ -77,7 +81,7 @@ end - uses: snap-diff/snap_diff-capybara/.github/actions/setup-ruby-and-dependencies@master with: ruby-version: '4.0' -- run: bundle exec rake test +- run: bin/rails test:system # not `rake test` — it skips test/system/ - uses: snap-diff/snap_diff-capybara/.github/actions/upload-screenshots@master if: failure() with: @@ -209,7 +213,7 @@ end **After (capybara-screenshot-diff in CI):** ```yaml - uses: snap-diff/snap_diff-capybara/.github/actions/setup-ruby-and-dependencies@master -- run: bundle exec rake test +- run: bin/rails test:system # not `rake test` — it skips test/system/ - uses: snap-diff/snap_diff-capybara/.github/actions/upload-screenshots@master if: failure() with: @@ -266,11 +270,14 @@ screenshot 'step2' ### "My tests are slow now" -Use the VIPS driver for ~50ms comparisons per image: +Use libvips for ~50ms comparisons per image — installing the gem is the whole setup, and from +2.1 it is the only backend: ```ruby +# Gemfile gem 'ruby-vips' -Capybara::Screenshot::Diff.driver = :vips ``` +Do **not** add `driver = :vips`: that setting is removed in 2.1 and raises `NoMethodError` +there, silently, with nothing in 2.0 to warn you. See [Drivers](drivers.md). ### "The diffs look different from what I'm used to" diff --git a/docs/snapdiff.md b/docs/snapdiff.md index 57fc812f..02dd9b14 100644 --- a/docs/snapdiff.md +++ b/docs/snapdiff.md @@ -5,10 +5,12 @@ reference: setup, configuration, the object map, and the extension points — al names only. The legacy `Capybara::Screenshot::Diff` / `CapybaraScreenshotDiff` names still work — they resolve -to the same objects — and the rest of the docs still teach them. The first legacy API a process -touches prints one migration notice; on top of that, *lazily shimmed* constants warn once each. -Some legacy names are silent by design. [UPGRADING.md](UPGRADING.md#deprecation-warnings) lists -exactly which is which. Nothing here replaces a working setup — it is what you write for **new** code. +to the same objects — and the rest of the docs still teach them. A legacy config accessor, +`include`, `default_options` call, or lazily shimmed constant prints one migration notice per +process; *lazily shimmed* constants warn once each on top of that. Other legacy names — the +integration requires and `CapybaraScreenshotDiff::Minitest::Assertions` among them — are silent +by design, so a quiet suite is not a migrated one. +[UPGRADING.md](UPGRADING.md#deprecation-warnings) lists exactly which is which. Nothing here replaces a working setup — it is what you write for **new** code. For migrating an existing suite, see [UPGRADING.md](UPGRADING.md). ## Quick start @@ -22,12 +24,20 @@ require "snap_diff/integrations/minitest" ```ruby # test/application_system_test_case.rb +require "test_helper" + class ApplicationSystemTestCase < ActionDispatch::SystemTestCase + # Window size and pixel ratio are inputs to every comparison — pin them. + driven_by :selenium, using: :headless_chrome, screen_size: [1400, 1400] + include SnapDiff::Minitest::Assertions # brings in SnapDiff::DSL too end ``` ```ruby +# test/system/homepage_test.rb +require "application_system_test_case" + class HomepageTest < ApplicationSystemTestCase test "homepage" do visit "/" @@ -36,6 +46,9 @@ class HomepageTest < ApplicationSystemTestCase end ``` +Run it with `bin/rails test:system`. `rake test` and `rails test` skip `test/system/` +entirely — `0 runs` and no baselines, which reads exactly like a pass. + `SnapDiff::Minitest::Assertions` already includes `SnapDiff::DSL`, so a separate `include SnapDiff::DSL` is not needed (it is harmless if you have it). @@ -160,7 +173,7 @@ integration require; a few objects need their own require, noted below. | `SnapDiff::Region` | Bounding box value object — `from_edge_coordinates`, `to_edge_coordinates` | | `SnapDiff::DSL` | `screenshot`, `assert_matches_screenshot`, `capture_screenshot`, groups/sections | | `SnapDiff::Minitest::Assertions` | Minitest wiring (`snap_diff/integrations/minitest`) | -| `SnapDiff::Error` | Base class for every error this gem *defines* — one `rescue SnapDiff::Error` covers all of the rows below. Misuse still surfaces as plain Ruby: bad arguments raise `ArgumentError`, and a missing image backend raises `RuntimeError` | +| `SnapDiff::Error` | Base class for every error this gem *defines* — see the caveat below. Misuse still surfaces as plain Ruby: bad arguments raise `ArgumentError`, and a missing image backend raises `RuntimeError` | | `SnapDiff::ExpectationNotMet` | A screenshot did not match its baseline | | `SnapDiff::UnstableImage` | No stable capture within `stability_time_limit` / `wait` | | `SnapDiff::WindowSizeMismatchError` | Browser window is not the configured `window_size` | @@ -175,6 +188,17 @@ integration require; a few objects need their own require, noted below. | `SnapDiff::Capture::Viewport` | Per-capture viewport check seam (`require "snap_diff/capture/viewport"`) | | `SnapDiff.serve` | Point Capybara at a static site directory (`require "snap_diff/static"`) | +> **`rescue SnapDiff::Error` does not catch a failed assertion under the framework +> integrations.** `SnapDiff::ExpectationNotMet` — the row you most want to rescue — is +> converted before it reaches your code: the Minitest integration re-raises it as +> `Minitest::Assertion` ([`integrations/minitest.rb`](../lib/snap_diff/integrations/minitest.rb)), +> and the RSpec integration as `RSpec::Expectations::ExpectationNotMetError` +> ([`integrations/rspec.rb`](../lib/snap_diff/integrations/rspec.rb)). That conversion is the +> point — it is what makes a mismatch a test failure rather than an error. You see a raw +> `SnapDiff::ExpectationNotMet` only when you drive `SnapDiff::DSL` yourself, with a bare +> `include SnapDiff::DSL` and no integration. `UnstableImage` and `WindowSizeMismatchError` +> are not converted and do arrive as `SnapDiff::Error`. + ## Compare two images without a browser Works on anything on disk — rendered PDFs, generated charts, CI artifacts: From f544cc6b1f273f7c271e395e324a820ee4a1f578 Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Mon, 24 Aug 2026 09:44:44 +0200 Subject: [PATCH 8/8] docs: three residuals from the verifier pass - bug_report template told reporters to run `rake test`, which runs zero system tests in a Rails app -- the same trap this branch exists to remove. - drivers.md said an unknown per-screenshot `driver:` key is "simply inert". It is validated and raises; only the deprecation warning is absent. - drivers.md said `:auto` and `:chunky_png` each warn once per process. `:auto` is silent when ruby-vips resolves; only `:chunky_png` warns. --- .github/ISSUE_TEMPLATE/bug_report.md | 2 +- docs/drivers.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 2958c4ad..1b4195fb 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -53,7 +53,7 @@ If applicable, add the `.diff.png` or `.heatmap.diff.png` files to help explain If applicable, run with `DEBUG=1` and paste the output: ``` -DEBUG=1 bundle exec rake test +bin/rails test:system # or whatever task runs your Capybara tests ``` ## Additional context diff --git a/docs/drivers.md b/docs/drivers.md index 42fbb8cc..610e3d48 100644 --- a/docs/drivers.md +++ b/docs/drivers.md @@ -17,7 +17,7 @@ following still works. Most of it warns once per process naming 2.1; the rows ma |---|---| | the `:chunky_png` driver | add `gem "ruby-vips"` to your Gemfile and drop `driver: :chunky_png` | | the `driver:` setting itself — `SnapDiff.config.driver =` and the legacy `Capybara::Screenshot::Diff.driver =` (**silent in 2.0**; on 2.1 they raise `NoMethodError` at config time) | delete the line; one backend needs no selection | -| the per-screenshot `driver:` override — `screenshot "index", driver: :vips` (**silent in 2.0 *and* 2.1**: per-screenshot options are a free-form hash, so an unknown key is simply inert) | delete the option, and grep for it — nothing will tell you the line is dead | +| the per-screenshot `driver:` override — `screenshot "index", driver: :vips` (**no deprecation warning in 2.0** -- but the key is validated, not inert: an unknown driver raises `RuntimeError: Wrong adapter ...`. 2.1 rejects the key outright) | delete the option, and grep for it — nothing will tell you the line is dead | | `driver: :auto` (and the `:auto` default) — warns **only when `:auto` actually falls back to ChunkyPNG**, i.e. when `ruby-vips` is missing; **silent** otherwise | with one backend there is nothing to choose; install `ruby-vips` and the default just works | | `shift_distance_limit` | ChunkyPNG-only. Use `median_filter_window_size`, `tolerance` or `color_distance_limit` — see [Configuration](configuration.md#allowed-shift-distance) | | `SnapDiff::Driver` (the custom-driver mixin) | nothing — see below | @@ -90,7 +90,7 @@ There are several options to setup active driver: `:auto`, `:chunky_png` and `:v * `:auto` - will try to load `:vips` if there is gem `ruby-vips`, in other cases will load `:chunky_png` * `:chunky_png` and `:vips` will load correspondent driver -> **2.1 keeps only `:vips`.** `:auto` and `:chunky_png` are removed; each warns once per +> **2.1 keeps only `:vips`.** `:auto` and `:chunky_png` are removed; `:chunky_png` warns once per > process in 2.0. If `:auto` is quietly running you on ChunkyPNG today (no `ruby-vips` > installed), the warning says so — that is the setup 2.1 breaks.