test: isolate the settings store from the user's own and from other tests - #579
Open
FelipeDefensor wants to merge 1 commit into
Open
test: isolate the settings store from the user's own and from other tests#579FelipeDefensor wants to merge 1 commit into
FelipeDefensor wants to merge 1 commit into
Conversation
…ests The suite pointed the settings manager at QSettings(APP_NAME, "DesktopTests"), a real named store that is never cleared, and it did so from a module-scoped fixture that most tests do not request. Two things followed. Values written by a test outlived it. They leaked into later tests, into the other xdist workers -- which share the store file -- and into every later run, which is why `pytest -n auto` failed intermittently on tests/test_app.py and on test_file_with_measures_to_force_display while the same tests passed serially. On this machine the store had kept beat_timeline/display_measure_periodicity at 2 instead of its default of 4, plus 14722 recent-file entries pointing at deleted pytest tmpdirs. Tests that ran before any module requesting that fixture wrote to the developer's own settings instead: their real store had accumulated 298 recent-file entries from test runs. The store is now a throwaway ini file, one per xdist worker, installed session-wide and autouse so no test can reach the real store whichever fixtures it requests, and every setting a test writes is restored afterwards. use_test_settings keeps setting prioritise_performance. assert_open_failed compared a str to a Path, so it was always true; it only surfaced now because it also indexed a list that is legitimately empty once recent files no longer leak between tests.
Collaborator
|
related?: #574 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The suite pointed the settings manager at
QSettings(APP_NAME, "DesktopTests")— a real, named, persistent store that is never cleared — and it did so from a module-scoped fixture (use_test_settings) that most tests never request. Two problems follow.1. Settings leak out of the test that set them. Because the store is a real file, a
settings.set(...)outlives the test. It leaks into later tests, into the other xdist workers (which share the file), and into every subsequent run. This is the cause of the intermittentpytest -n autofailures intests/test_app.pyand intest_file_with_measures_to_force_displaythat pass when run serially.It is not theoretical — on my machine the store had kept:
editable.beat_timeline.display_measure_periodicity = 2, where the default is4. Every run started with the wrong value.private.recent_files.*entries pointing at long-deleted pytest tmpdirs, from runs going back many sessions (pytest-4,pytest-73/popen-gw3,pytest-129, …). 7.8 MB of accumulated state that every run read from.2. Tests wrote to the developer's own settings.
tiliaandtlsdon't depend onuse_test_settings, and the fixture never restores the previous store, so which store a test got depended on module order within the worker. Any test running before a module that requested the fixture used the real store — my owncom.tilia.Desktop Settings.plisthad accumulated 298 of its 1000 keys as recent-file entries from test runs.The change
.iniin the pytest tmp dir, one per xdist worker.CLAUDE.mdalready tells contributors to expect ("set the values you depend on explicitly inside each test").use_test_settingskeeps doing its other job (prioritise_performance).assert_open_failedneeded a fix to land this: it compared astrto aPath, so it was always true and had never asserted anything. It only surfaced now because it also indexedget_recent_files()[0], and that list is legitimately empty once recent files stop leaking between tests. Corrected to compare like-for-like; it passes, so a failed open is correctly not recorded as recent.Test plan
tests/test_settings_isolation.py— pins both properties: the store is not the user's own, and a setting written by one test is restored before the next. 4 pass here; 2 of them fail ondev.dev, one process writesdisplay_measure_periodicity = 7and a separate process then reads back7instead of the default4. With this change the second process reads4.pytest -n auto, twice: 1834 passed, 13 skipped, 2 xfailed, 2 xpassed.tests/test_app.py+tests/ui/timelines/beat/) run 5x under-n auto: 136 passed each time.Note for maintainers
The two stale stores are now unused but still on disk. On macOS they can be dropped with:
The real store keeps its test-written recent-file entries; clearing those is a user-facing action, so I have left it alone.
🤖 Generated with Claude Code