Skip to content

test: isolate the settings store from the user's own and from other tests - #579

Open
FelipeDefensor wants to merge 1 commit into
devfrom
fix/test-settings-isolation
Open

test: isolate the settings store from the user's own and from other tests#579
FelipeDefensor wants to merge 1 commit into
devfrom
fix/test-settings-isolation

Conversation

@FelipeDefensor

Copy link
Copy Markdown
Collaborator

Summary

The suite pointed the settings manager at QSettings(APP_NAME, "DesktopTests") — a real, named, persistent store that is never cleared — and it did so from a module-scoped fixture (use_test_settings) that most tests never request. Two problems follow.

1. Settings leak out of the test that set them. Because the store is a real file, a settings.set(...) outlives the test. It leaks into later tests, into the other xdist workers (which share the file), and into every subsequent run. This is the cause of the intermittent pytest -n auto failures in tests/test_app.py and in test_file_with_measures_to_force_display that pass when run serially.

It is not theoretical — on my machine the store had kept:

  • editable.beat_timeline.display_measure_periodicity = 2, where the default is 4. Every run started with the wrong value.
  • 14,722 private.recent_files.* entries pointing at long-deleted pytest tmpdirs, from runs going back many sessions (pytest-4, pytest-73/popen-gw3, pytest-129, …). 7.8 MB of accumulated state that every run read from.

2. Tests wrote to the developer's own settings. tilia and tls don't depend on use_test_settings, and the fixture never restores the previous store, so which store a test got depended on module order within the worker. Any test running before a module that requested the fixture used the real store — my own com.tilia.Desktop Settings.plist had accumulated 298 of its 1000 keys as recent-file entries from test runs.

The change

  • The store is a throwaway .ini in the pytest tmp dir, one per xdist worker.
  • Installed session-scoped and autouse, so no test can reach the real store whichever fixtures it requests.
  • Every setting a test writes is restored afterwards, so tests can rely on defaults — which is what CLAUDE.md already tells contributors to expect ("set the values you depend on explicitly inside each test").
  • use_test_settings keeps doing its other job (prioritise_performance).

assert_open_failed needed a fix to land this: it compared a str to a Path, so it was always true and had never asserted anything. It only surfaced now because it also indexed get_recent_files()[0], and that list is legitimately empty once recent files stop leaking between tests. Corrected to compare like-for-like; it passes, so a failed open is correctly not recorded as recent.

Test plan

  • tests/test_settings_isolation.py — pins both properties: the store is not the user's own, and a setting written by one test is restored before the next. 4 pass here; 2 of them fail on dev.
  • Leak across processes demonstrated directly: on dev, one process writes display_measure_periodicity = 7 and a separate process then reads back 7 instead of the default 4. With this change the second process reads 4.
  • Full suite green under pytest -n auto, twice: 1834 passed, 13 skipped, 2 xfailed, 2 xpassed.
  • Previously-flaky selection (tests/test_app.py + tests/ui/timelines/beat/) run 5x under -n auto: 136 passed each time.
  • Neither plist is touched by a full run any more — both key counts unchanged (14813 / 1000), and the real store's size and mtime are byte-identical before and after.

Note for maintainers

The two stale stores are now unused but still on disk. On macOS they can be dropped with:

defaults delete com.tilia.DesktopTests

The real store keeps its test-written recent-file entries; clearing those is a user-facing action, so I have left it alone.

🤖 Generated with Claude Code

…ests

The suite pointed the settings manager at QSettings(APP_NAME,
"DesktopTests"), a real named store that is never cleared, and it did so
from a module-scoped fixture that most tests do not request. Two things
followed.

Values written by a test outlived it. They leaked into later tests, into
the other xdist workers -- which share the store file -- and into every
later run, which is why `pytest -n auto` failed intermittently on
tests/test_app.py and on test_file_with_measures_to_force_display while
the same tests passed serially. On this machine the store had kept
beat_timeline/display_measure_periodicity at 2 instead of its default of
4, plus 14722 recent-file entries pointing at deleted pytest tmpdirs.

Tests that ran before any module requesting that fixture wrote to the
developer's own settings instead: their real store had accumulated 298
recent-file entries from test runs.

The store is now a throwaway ini file, one per xdist worker, installed
session-wide and autouse so no test can reach the real store whichever
fixtures it requests, and every setting a test writes is restored
afterwards. use_test_settings keeps setting prioritise_performance.

assert_open_failed compared a str to a Path, so it was always true; it
only surfaced now because it also indexed a list that is legitimately
empty once recent files no longer leak between tests.
@azfoo

azfoo commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

related?: #574

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants