Skip to content

Skip pylint's primer tests, which race each other under pytest-xdist - #99

Merged
thomasjm merged 1 commit into
mainfrom
fix-pylint-primer-test-race
Aug 27, 2026
Merged

Skip pylint's primer tests, which race each other under pytest-xdist#99
thomasjm merged 1 commit into
mainfrom
fix-pylint-primer-test-race

Conversation

@thomasjm

Copy link
Copy Markdown
Contributor

build-sample-environments (Linux, x86_64-linux) failed on a cold cache with:

error: Cannot build '/nix/store/...-python3.11-pylint-3.3.7.drv'
  > FAILED tests/testutils/_primer/test_primer.py::TestPrimer::test_compare[message_changed]
  > AssertionError: assert '' == '🤖 **Effect o...mmit v2.14.2*'
  > ============ 1 failed, 996 passed, 250 skipped, 5 xfailed in 8.20s =============

It only shows up when pylint is built from source, so it stays hidden until a cache eviction and
then takes out sample environments on whatever branch happens to arrive first. It hit my
stable-Nixpkgs branch and looked like a regression there; it isn't.

Why it is flaky

Every test in that module funnels through one helper:

with patch("sys.argv", new_argv):
    Primer(PRIMER_DIRECTORY, PACKAGES_TO_PRIME_PATH).run()
with open(PRIMER_DIRECTORY / "comment.txt", encoding="utf8") as f:
    content = f.read()

PRIMER_DIRECTORY is a single fixed path, tests/.pylint_primer_tests/, shared by every
test_compare parametrization plus test_compare_batched and test_truncated_compare. nixpkgs runs
the suite under pytest-xdist, so those land on different workers in the same working directory and
overwrite each other's comment.txt. A worker that reads while another is mid-write gets an empty
file, which is exactly the assert '' == ... in the failure.

That also explains why it is specifically python3.11-pylint: the whole TestPrimer class is
skipif sys.version_info[:2] != (3, 11), so no other interpreter runs these at all.

The fix

Add the module to disabledTestPaths, alongside the two entries nixpkgs already disables there.
These tests cover pylint's own CI primer tooling -- comparing lint output between two git revisions
of upstream packages -- which is not something we ship or depend on.

Testing

Local rebuild of python311Packages.pylint from source, unpatched: 997 passed, 250 skipped,
5 xfailed
-- it passes here, which confirms the test is flaky rather than broken, and is why this
only ever shows up under CI's scheduling.

With the override: 991 passed, 250 skipped, 5 xfailed -- the 6 primer tests are gone and nothing
else changes. Verified the override reaches the package: disabledTestPaths goes from
["tests/benchmark" "tests/pyreverse/test_writer.py"] to that plus
"tests/testutils/_primer/test_primer.py", and the derivation hash changes.

@thomasjm
thomasjm force-pushed the fix-pylint-primer-test-race branch from 0a6cc64 to 801c6aa Compare August 27, 2026 10:13
@thomasjm
thomasjm merged commit 283daeb into main Aug 27, 2026
57 of 60 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant