Skip to content

Flaky: Test Drivers (selenium_headless, chunky_png) — one input row renders differently ~1 run in 5 #285

Description

@pftg

Split out of #284 so it does not sit in the release path.

BrowserScreenshotTest#test_compare_crops_only_when_other_part_is_not_working fails
intermittently on one matrix cell, Test Drivers (selenium_headless, chunky_png):

the change spans 4378 of 16800 px (26.06% of the 200x84 image)
  changed region: [0,19,199,41] (left,top,right,bottom edges)
  max color distance: 50.9
  judged against: tolerance 0.13, color_distance_limit 40

What is established

  • Base rate: 4 success / 1 failure over the last 12 Test runs on that cell. Every
    other Drivers cell is 100%.
  • Not environment drift. The failing job (97473161378) and a passing job at the same
    commit (97450518382) ran on the identical runner image 20260816.277.1, same
    runner version, same Ruby.
  • Not antialiasing. The changed region is 200x22 = 4400 px against 4378 px changed —
    the entire first input row and nothing else — at max colour distance 50.9. That is a
    materially different render of #first-field, not sub-pixel drift.
  • The fixture (test/fixtures/app/index-without-img.html) loads no remote resources;
    the font stack is local with fallbacks, so it is not the usual web-font race.

What is not established

The actual cause. A plausible shape is leftover focus/selection state on #first-field
from the test that does fill_in "First Field:", but that is unverified — and it
cannot be verified from CI artifacts until #284 is fixed, because the failure's own
evidence is overwritten before upload.

Blocked on #284

Fix the shared screenshot name first, then let this flake reproduce once with real
artifacts. Debugging it before that means guessing.

Also worth deciding

fail-fast turns this single flaky cell into three red cells (two siblings report
cancelled), so a run reports far more damage than occurred. Same mechanism hid the JRuby
lane for 15 runs (#283).

Scheduling

Not a 2.0.0 blocker: it predates the release, and chunky_png is deprecated in 2.0 and
removed in 2.1 — so this cell disappears on its own in 2.1. Filed against 2.1.0 so the
deletion is a deliberate outcome rather than an accidental one, and so nobody mistakes the
removal for a fix.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions