Instability is undiagnosable, so people add sleeps instead — and the wait is the dominant cost

[pftg]

Reported by the maintainer from real use on jetthoughts.github.io: 10-minute suites, dominated by waiting for the page to stabilise, plus the effort of locating the component that kept moving. The workaround was "sleeps, intensively, to avoid debugging as much as possible."

That workaround is the finding. The tool made diagnosis so expensive that buying silence with sleep was rational.

Measured cost

A DevX spike measured, per screenshot (macOS arm64, ruby 4.0.6, cuprite, 800x600, N=15):

Phase	chunky_png → vips
browser round-trip	~18 ms
decode + write	106 → 31 ms
baseline checkout (git)	12.6 ms
comparison	53 → 8 ms
stability_time_limit: 0.5	+660 ms

docs/configuration.md recommends 1–2 s. At 2 s, one screenshot costs about 2 s — an order of magnitude more than every other phase combined. Stability is the expense; the screenshots are not.

Why the wait cannot simply be shortened

stable_screenshoter.rb:63-70 is semantically correct — "unchanged for stability_time_limit" genuinely requires waiting that long:

loop do
  attempt_next_screenshot(snapshot)
  return true if attempt_successful?
  return false if timeout?(deadline_at)
  sleep(stability_time_limit)
end

So the cost is not a bug in the algorithm. The bug is that the user has no way to learn what value they actually need, and no way to find the one element that never settles. Both are information we already have and discard.

Fix 1 — say what moved, in the message

attempts_reporter.rb:35 builds a Comparison between consecutive attempts and asks different?. That comparison knows the changed region. We annotate the PNGs with it and then print only a list of file paths:

Could not get stable screenshot within 2s:
/…/home.attempt_01.png
/…/home.attempt_02.png

It should print the region, and the escape hatch that removes the need for a sleep:

Could not get stable screenshot for 'home' within 2s (5 attempts).
  The page kept changing in: [40,600,1200,680] (left,top,right,bottom edges)
    — 8% of the 800x600 image, changed between every attempt.
  If that area is a clock, animation, carousel or live counter, exclude it:
    assert_matches_screenshot "home", skip_area: [40,600,1200,680]

A user who can see which region is unstable masks it once. A user who cannot, sleeps. We already have skip_area; the failure message just never mentions it.

If several regions move, report them; if the same region moves every time, say so — that distinguishes an animation from genuine page churn.

Fix 2 — report how long stabilisation actually took

On success, we know the elapsed time and the attempt count and report neither. A user setting stability_time_limit: 2 has no idea their page settles in 150 ms. Surface it (in the summary line, in DEBUG, or on the first run) so the value can be tuned with evidence instead of guessed upward until the flakiness stops — which is how a 10-minute suite happens.

Fix 3 — the docs recommend the expensive value without saying what it costs

docs/configuration.md suggests 1–2 s with no indication that this is per screenshot and dominates the run. State the cost, and say to start low and raise only if instability appears.

Checklist

• Instability message names the unstable region(s) and suggests skip_area with the literal coordinates
• Distinguish "one region moves every attempt" from "different regions each time"
• Report actual stabilisation time and attempt count on success
• docs/configuration.md: state the per-screenshot cost of stability_time_limit; recommend starting low
• Never print a coordinate or command not derived from live state — the suggested skip_area must be the region actually measured
• Guards written as the USER's code, with a genuinely unstable fixture page
• Mutation-check each guard; a mutation that reds nothing is a finding

Related

#270 (a disabled screenshot still counts as an assertion) and #269 (the summary line is invisible by default) are the other two ways this gem currently fails to tell users what it did.

[pftg]

Unscheduled for now (maintainer, 2026-08-24) — recorded so the reasoning survives, to be kicked off after v2.0 and v2.1 ship.

Not blocked on evidence in the usual sense: the demand here is first-hand and measured. What is deferred is the work, not the case for it.

Where the case already stands, so a kickoff does not restart from zero:

• First-hand report: 10-minute suites on a real project, dominated by stabilisation waiting, with sleep adopted deliberately to avoid the cost of debugging.
• Measured: stability_time_limit: 0.5 costs ~660 ms per screenshot against ~18 ms for the browser round-trip and 8–53 ms for comparison. Docs recommend 1–2 s.
• The information needed for Fix 1 is already computed and discarded — attempts_reporter.rb:35 builds the comparison and knows the region.
• skip_area already exists; the failure message simply never mentions it.

What a kickoff should start with, in priority order — smallest change with the largest effect first:

1. Print the unstable region and the literal skip_area: line. Removes the reason to sleep.
2. Report actual stabilisation time on success, so the limit can be tuned down with evidence.
3. Correct the docs' cost guidance.

Nothing here needs new research. It needs a lane and a decision on release.

[pftg]

Pairs with #272. These two are one workflow, not two features: the tool should name the region that would not settle, and masking it should be instant. Today it names nothing (the region is computed at attempts_reporter.rb:35 and discarded) and masking costs 5s per non-matching selector. A user facing an unstable page therefore gets no diagnosis and a slow remedy — which is why the observed workaround was sleep. Kick these off together.

[pftg]

Delivered in #280 (df2d98e), on master.