Skip to content

fix(mqtt): retire session workers on reconnect - #44

Merged
QuiteYellow merged 1 commit into
QuiteYellow:mainfrom
Moballo-LLC:codex/issue-9-session-workers
Aug 18, 2026
Merged

fix(mqtt): retire session workers on reconnect#44
QuiteYellow merged 1 commit into
QuiteYellow:mainfrom
Moballo-LLC:codex/issue-9-session-workers

Conversation

@Jason-Morcos

@Jason-Morcos Jason-Morcos commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Fixes #9.

Each DTLS reconnect starts a new PollScheduler, KeepaliveTask, and ObserveRefreshTask, but those workers currently wait on the bridge-lifetime self.stop event. Clearing their references at teardown therefore leaves the old workers running against a closed session. That fits the long-run evidence in #9: orphaned six-hour OBSERVE refresh timers accumulate and eventually fire at a much shorter apparent cadence.

This change:

  • gives each connected session its own stop event;
  • signals that event whenever the session reader exits, including error exits;
  • lets bridge shutdown signal both the bridge event and the active session event, with a locked handoff so SIGTERM cannot miss a newly published event;
  • retires already-started workers if a later worker thread cannot start;
  • disables the retiring keepalive callbacks before shutdown, so an old session cannot change availability after a replacement connects;
  • waits up to two seconds total for all three workers and logs any worker that does not exit.

The regression tests cover consecutive sessions, reader failure, partial thread startup, shutdown while the reader remains joined, and shutdown just before the session-event handoff. They verify that reconnect does not poison the bridge-wide event, shutdown wakes active workers immediately, each session gets a distinct event, callbacks are retired, and all workers exit before teardown returns.

Validation:

  • SmartThings-Local: 258 tests on Python 3.14
  • dependency floor: 258 tests on Python 3.11 with the minimum declared versions
  • worker lifecycle test: 50 consecutive clean runs
  • current LocalThings main: 1,625 tests
  • wheel/sdist install checks, distribution, compile, and share-safety checks

This is one commit on current main, after #42, #43, and #45. It remains independent of #36.

@QuiteYellow

Copy link
Copy Markdown
Owner

The diagnosis is right and the per-session stop event is the fix. Orphaned workers waiting on a bridge-lifetime event explains the #9 evidence better than anything I had.

One change before I merge it.

The workers now wait only on session_stop, and nothing outside session_once's finally ever sets that. On SIGTERM, mqtt_demo/__main__.py:169 sets b.stop and publishes availability offline, but session_once is parked in sess.join() and the reader is still alive, so session_stop stays clear. The poll, ping and obsref threads keep working the device for the whole shutdown window. If a keepalive tick recovers in that window it calls _on_reachable, which publishes availability back online after the shutdown handler published offline.

Before this PR, self.stop.set() stopped all three immediately. That is the one behaviour I would not want to give up to fix #9.

Simplest fix is to keep a handle on the event, something like self._session_stop = session_stop, so the shutdown path can set it too. Having the workers wait on both events would also work if you prefer that shape.

Unrelated and non-blocking: the new keepalive.on_reachable = None in the finally now overlaps with the existing null-out in _maybe_force_reconnect at mqtt_demo/bridge.py:512-516. Two places doing the same defensive thing is fine, just noting it so it does not look accidental later.

CI is green and the worker-lifecycle test reads well, particularly the forced start failure.

@Jason-Morcos
Jason-Morcos force-pushed the codex/issue-9-session-workers branch from 8fe946d to 8c374e1 Compare August 18, 2026 18:13
@Jason-Morcos

Jason-Morcos commented Aug 18, 2026

Copy link
Copy Markdown
Contributor Author

Fixed in 8c374e1. The shutdown handler now calls request_stop(), which sets the bridge-wide event and the active per-session event before publishing offline. The session-event handoff is lock-protected and rechecks the bridge event, so a SIGTERM landing during that handoff cannot leave the workers running.

I added regressions for shutdown while sess.join() is still blocked and for a stop requested just before the session event is published. Both prove the poll, ping, and OBSERVE-refresh workers exit without poisoning reconnect state.

I left the callback null-outs in both places deliberately: _maybe_force_reconnect protects that specific force-close path, while the finally covers every other way a session can end.

@QuiteYellow

QuiteYellow commented Aug 18, 2026

Copy link
Copy Markdown
Owner

That closes it, thanks.

Fair enough on keeping both callback null-outs. That reasoning holds.

Merging.

@QuiteYellow
QuiteYellow merged commit 6b9a508 into QuiteYellow:main Aug 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker threads might not be getting stopped on reconnect

2 participants