Skip pylint's primer tests, which race each other under pytest-xdist - #99
Merged
Conversation
thomasjm
force-pushed
the
fix-pylint-primer-test-race
branch
from
August 27, 2026 10:13
0a6cc64 to
801c6aa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
build-sample-environments (Linux, x86_64-linux)failed on a cold cache with:It only shows up when pylint is built from source, so it stays hidden until a cache eviction and
then takes out sample environments on whatever branch happens to arrive first. It hit my
stable-Nixpkgs branch and looked like a regression there; it isn't.
Why it is flaky
Every test in that module funnels through one helper:
PRIMER_DIRECTORYis a single fixed path,tests/.pylint_primer_tests/, shared by everytest_compareparametrization plustest_compare_batchedandtest_truncated_compare. nixpkgs runsthe suite under
pytest-xdist, so those land on different workers in the same working directory andoverwrite each other's
comment.txt. A worker that reads while another is mid-write gets an emptyfile, which is exactly the
assert '' == ...in the failure.That also explains why it is specifically
python3.11-pylint: the wholeTestPrimerclass isskipif sys.version_info[:2] != (3, 11), so no other interpreter runs these at all.The fix
Add the module to
disabledTestPaths, alongside the two entries nixpkgs already disables there.These tests cover pylint's own CI primer tooling -- comparing lint output between two git revisions
of upstream packages -- which is not something we ship or depend on.
Testing
Local rebuild of
python311Packages.pylintfrom source, unpatched: 997 passed, 250 skipped,5 xfailed -- it passes here, which confirms the test is flaky rather than broken, and is why this
only ever shows up under CI's scheduling.
With the override: 991 passed, 250 skipped, 5 xfailed -- the 6 primer tests are gone and nothing
else changes. Verified the override reaches the package:
disabledTestPathsgoes from["tests/benchmark" "tests/pyreverse/test_writer.py"]to that plus"tests/testutils/_primer/test_primer.py", and the derivation hash changes.