feat(session): bring sessions back after a host reboot, not just after a server restart

[irisitymichaelgrundberg]

The problem

Codeman sessions survive a server restart because the tmux server outlives it. They do not survive a reboot of the host, because tmux dies with the machine. Picking yesterday's work back up means finding each conversation in history and resuming it by hand, one session at a time.

The docs already promise something close enough to be read as this. Q&A discussion #293, "Do my agents keep running when I close the browser or my laptop?", answers that codeman service install "also survives reboots". That is true of the server but not of the sessions. The systemd unit makes the same assumption explicit: "Agents keep running in tmux when the server restarts, so only signal the server itself" (src/service-installer.ts).

Here is what the first boot after a reboot actually does:

1. reconcileSessions() (src/tmux-manager.ts:2572) lists panes on the Codeman socket and finds none.
2. Every tracked session lands in dead, leaves this.sessions, and drops out of mux-sessions.json.
3. finalizeRestoredState() runs cleanupStaleSessions() (src/web/server.ts:2098), which prunes those sessions from state.json.
4. The board comes up empty, and every conversation has to be resumed by hand.

I prototyped this before filing, so the numbers are measured

The core is about 140 lines of production code. A prototype against master at a017e9a8 adds 536 lines across four files: 135 in src/web/server.ts, 5 in src/web/schemas.ts, a new 155-line module holding the decision logic as pure functions, and 242 lines of tests. Typecheck, lint, format:check and the full npm test suite pass, including 19 new tests.

Three things in the existing code make it that small.

The construction path already exists. restoreMuxSessions() (src/web/server.ts:2791) rebuilds a Session from state.json for each surviving mux session, with the owner re-resolved, the permission mode recomputed through resolveClaudeModeForUsername, and env overrides recovered. Restoring a reboot-killed session is that same block with the mux session omitted, so a pane gets created rather than attached.

Pane creation is not in the route handler. startInteractive() (src/session.ts:1942) calls _setupOrAttachMuxSession() (src/session.ts:1593), which calls mux.createSession() whenever the Session carries no mux session. POST /api/sessions (src/web/routes/session-routes.ts:873) never creates a pane at all, so nothing about it needs extracting or refactoring.

The resume hook is already declared. resumeSessionId is a Session constructor option documented as "Resume a previous Claude conversation (used after server reboot)" (src/session.ts:670). It already flows into the create options at src/session.ts:2074, and CreateSessionSchema already accepts it (src/web/schemas.ts:538).

The timing window is real. state.json is not pruned until finalizeRestoredState() runs on the line after restoreMuxSessions() (src/web/server.ts:2503-2504). Inside restoreMuxSessions, the dead sessions' persisted records are still readable, which is where the restore pass belongs.

What the prototype does

An opt-in setting, default off, turns on a restore pass inside restoreMuxSessions. For each eligible dead session it rebuilds the Session with resumeSessionId set to the persisted conversation id, then starts it. Four rules keep it honest:

• It restores only when the machine plausibly rebooted. The socket must hold no panes at all while state.json still lists sessions, and os.uptime() must put the host's boot after the newest persisted activity.
• It never revives a session the user killed. See the invariant section below for how that survives a reboot.
• A restored session comes back attached and idle. Nothing runs until its owner types.
• Respawn controllers and Ralph loops stay disarmed. Re-arming them at boot would start unattended agents spending tokens with nobody watching.

The invariants it ran into

COD-108 needs a durable form. The rule that an intentional kill or detach must never be auto-revived is enforced by an in-memory guard (src/tmux-manager.ts:2385), and memory does not survive a reboot. The durable stand-in is demoteOrRemoveSession() (src/state-store.ts:503). An unpinned kill deletes the record, so absence from state.json is itself the guard. A pinned kill demotes the record to status: 'stopped' and keeps it, so the restore pass must refuse that status or it would undo the kill.

Stats collection and the hook sweep sit behind the wrong gate. startStatsCollection is inside a block gated on alive.length > 0 || discovered.length > 0 (src/web/server.ts:3057). After a reboot both counts are zero, so restored sessions would get no stats and no workspace-hook install without a small branch for that case.

Session ids must stay hex. SAFE_MUX_NAME_PATTERN rejects anything else, and a rejected name makes startInteractive() fall back to a direct PTY without an obvious error.

The crash-loop breaker has to be respected. Any session persisted as respawn-blocked must be skipped, or restore hands a crash-looping pane a fresh start.

What a real implementation still needs

The prototype is deliberately incomplete. Five things remain:

1. The multi-user env clamp. clampEnvOverridesForOwner() lives inside the route module (src/web/routes/session-routes.ts:495) and strips privileged environment keys for a non-granted owner. The restore pass replays persisted envOverrides without it. This is the one piece that genuinely has to move somewhere both callers can reach, and I would rather you chose where.
2. The other CLIs. Only Claude sessions are restored. Codex, omp and the rest name their thread with an id in their own config object, which the prototype does not thread through.
3. Persisted per-session settings. Auto-compact, auto-clear, nice priority and the flicker filter are not re-applied on the rebuilt session.
4. A server-sent event. Rebuilt sessions emit no session_created, so an already-open browser tab will not see them until it refetches.
5. A checkbox. The setting exists in settings.json and in the Zod schema, so today it has to be edited by hand.

Remote and docker sessions are skipped on purpose. Both need another host or container to be up, which a freshly booted machine cannot promise, and the COD-108 watcher already owns the remote case.

Open questions

I would rather agree these before turning the prototype into a PR:

1. Should the setting be one global toggle, per case, or per session?
2. Should restore run automatically, or should the board offer a "restore N sessions from before the reboot" banner and wait for a click?
3. Should respawn and Ralph ever re-arm on their own, or always wait for the user?
4. Where should clampEnvOverridesForOwner live so that both the route and the restore pass can call it?

What this cannot do

A restored session is a new pane running a new process, so scrollback from before the reboot does not come back. The conversation continues and the terminal history does not, and the UI should say so rather than imply a full restore.

The reboot heuristic is the part I trust least. It correctly refuses an ordinary server restart and a hand-wiped tmux socket on a long-uptime host. It cannot tell a reboot from a crash that took tmux down inside the same window, and there is no equivalent of the remote case's has-session probe to confirm a session was alive when power was lost. That is the main reason the setting defaults to off.

Nothing here has been tested against a real reboot. Under vitest the tmux layer is an in-memory mock, so what the tests prove is that the construction path creates a session and threads the resume id.

Environment

Codeman 1.26.2, master at a017e9a8, installed from a git clone. Linux on WSL2, Claude Code sessions, tmux backend.

The prototype is pushed to my fork on the branch spike/reboot-restore, at https://github.com/irisitymichaelgrundberg/Codeman/tree/spike/reboot-restore. It is one commit against master, and I am happy to open it as a draft PR if you would rather review code than prose.

[Ark0N]

This is the most useful issue anyone has filed here: you read the restore path, found the timing window inside restoreMuxSessions() before finalizeRestoredState() prunes, and came back with the invariants instead of a patch. Answers to your four questions, so you can turn the spike into a PR.

1. One global toggle, not per case or per session. The decision a user is actually making is "this machine reboots and I want my board back", which they make once. A per-session field also has to survive the reboot in state.json and then be re-read by a pass whose whole job is deciding what to rebuild, which adds a second source of truth to the thing you correctly identified as the fragile part.

2. A banner, and that is the important one. You wrote that the reboot heuristic is the part you trust least, and the banner is what makes that acceptable rather than something to keep tightening. A wrong automatic restore spawns N CLI processes nobody asked for, in workspaces that may have moved on; a wrong banner is a line of text the user ignores. It also collapses most of your open work: with a click in front of it, the heuristic does not have to be right, it only has to be worth asking about, and the "restored sessions emit no session_created" gap goes away because the click is a user action the frontend is already standing in front of.

So for a first PR I would drop the setting entirely. No settings.json key, no Zod change, no checkbox, no migration. The board offers "Restore N sessions from before the reboot", the user clicks, and the restore runs. An always-on variant for headless boxes is a fine follow-up once the mechanism has been used against a few real reboots, and it can reuse the same pass.

3. Never re-arm respawn or Ralph. Your instinct is right and the PTY-exit breaker is the precedent: it resets only on an explicit {clearBreaker:true}, specifically so no automatic path can clear it. A restored session comes back attached, idle and disarmed, and the user re-arms what they want.

4. clampEnvOverridesForOwner moves to its own module under src/, something like src/session-env-clamp.ts, exporting ownerClampedEnvKeys() and the clamp. Not route-helpers.ts: the question it answers is about session privilege, not HTTP, and server.ts does not import that module today. There is already a precedent for a non-route session creator reaching for this: cron/cron-service.ts imports canUsernameRunPrivilegedCommands from user-store.ts directly and re-resolves the owner's grant at fire time. Keep the _clampEnvOverridesForOwner test hook re-exported from session-routes.ts so its existing test keeps passing, and re-resolve the grant at restore time rather than trusting anything persisted, for the same reason cron does.

Two things to add to your list:

• Skip a workspace that no longer exists. restoreMuxSessions() already does this for recovered mux sessions, deliberately, so a deleted repo with a surviving tmux session is not resurrected as an empty directory. A reboot restore needs the same check, and it is more likely to matter there.
• Install workspace hooks on the restored session. You found that startStatsCollection sits behind a gate that is false after a reboot; the hook install (applyWorkspaceHooks) is behind the same one, and a claude session without it goes silently blind: no stop/idle_prompt for respawn, no Approvals Inbox item, no red tab on a blocking dialog. That is the kind of degradation nobody reports as a bug, they just conclude the feature is flaky.

On scrollback: agreed, and say it in the UI. "Conversation restored, terminal history did not survive the reboot" is a sentence users accept; silently coming back with an empty pane reads as a broken restore.

Please do open it as a draft PR. Claude only for the first pass is the right scope, and the other CLIs' resume ids are a good second one.

[irisitymichaelgrundberg]

Thank you, all four answers are clear and I will build to them. The setting comes
out entirely, the board offers the restore behind a click, respawn and Ralph stay
disarmed, and clampEnvOverridesForOwner moves to src/session-env-clamp.ts with
the owner's grant re-resolved at restore time. I will rebase the spike first, since
it sits on a017e9a8 and master has moved 117 commits since.

Line references below are against master at 3248f350.

The banner leaves one question your reply does not cover

A click arrives seconds or hours after boot, and by then the data the restore needs
is gone. cleanupSessionsByIds() (src/state-store.ts:529) deletes each dead
session's record from state.json unless the session is pinned, and a
reboot-killed session is almost never pinned. The record holds the working
directory, the owner, the mode and the Claude conversation id, and the restore pass
cannot rebuild a session without them.

The banner itself needs none of that. It shows a count and a button, and the click
sends nothing back but an intent. What needs somewhere to live is the server-side
list of which sessions are restorable and what each one was, which I will call the
pending restore plan. The server either holds its own copy of those records from
boot until the click, or it stops the deletion so the records are still in
state.json when the click arrives. I see three ways to do that.

A. The server keeps the plan in memory. The boot pass builds the plan inside
restoreMuxSessions(), hands the records to a module-level singleton, and lets the
pruning run unchanged. src/web/approval-inbox.ts is the precedent, and its header
already documents the shape: in-memory only, dropped on a server restart, emit
callbacks injected by the server, no IO of its own. The cost is that a server
restart before the click destroys the plan, and because the records are already
pruned, it destroys the only route back to those conversations. An unattended
self-update is the case I have in mind.

B. The server writes the plan to a file. The boot pass writes the plan to
something like dataPath('reboot-restore-offer.json') before the pruning, and a
click or a decline deletes it. The plan then survives a server restart. The cost is a second
file holding copies of session records, which needs an expiry rule and an answer
for a session id that comes back to life.

C. The server keeps the records themselves. The boot pass adds its candidates
to a retain set that cleanupStaleSessions() honours, the records stay in
state.json with their real statuses, and the plan is derived from them on
demand. This does not put ghost tabs on the board, because getLightSessionsState()
(src/web/server.ts:2125) builds the session list from the live this.sessions
map. Durability across a server restart is the catch. The next boot re-runs the
heuristic, and if the user restored some sessions before that restart, the pass
sees live panes, decides no reboot happened, and prunes what is left. Making C
durable needs a marker field on the persisted record, so it becomes a schema change
and a migration.

Relying on the pin is the fourth thing one could try, and it does not work. Only
pinned records survive the pruning, most sessions are not pinned, and pinning is a
user-facing state I should not set on someone's behalf.

I lean to A for this PR. You asked for the smallest first change, A reuses a
pattern that is already in the tree with its tradeoffs written down, and B is a
clean upgrade later that does not disturb the restore pass. I will go with A unless
you would rather pay for B now.

Two of your additions move with the click

Both of the things you added sit in boot-time code that has finished running by the
time the user clicks, so the click path has to do them itself.

ensureHooksForRecoveredWorkspaces() iterates this.sessions and would cover a
restored session for free if the restore still happened at boot. Behind a banner it
does not, so the restore route calls applyWorkspaceHooks for the restored
workspace directly, still gated on workspaceHooksEnabled.

startStatsCollection (src/web/server.ts:3138) has the same problem from the
other side. The gate you pointed at is false after a reboot, so the collector never
starts, and by click time nothing will start it. The restore route starts it if it
is not already running.

The workspace-exists check is unaffected and belongs in the plan step, where it
also keeps a vanished repo out of the banner's count.

[Ark0N]

A, for this PR. And the reason A is acceptable belongs in the module header: its failure mode is today's behaviour, not data loss.

The pruned record is not the only route back to a conversation. The conversation is the transcript under ~/.claude/projects, and Codeman already resumes from that alone: the Welcome screen's Resume list and the Session Manager are built by unified-session-service.ts from those files, and resumeHistorySession() in terminal-ui.js creates a fresh session with resumeSessionId set to the row's conversation id. So if the server restarts between boot and the click, the user is back to exactly where they are now, resuming by hand one at a time. What the record held that the transcript does not (owner, name, env overrides, effort, lineage) is the convenience this feature adds, and that is what a lost plan costs. It never costs the conversation. Write that down the way approval-inbox.ts writes "items are in-memory only, a server restart drops them, the next prompt re-fires the hook".

On the unattended self-update: there is none. startUpdate is imported by system-routes.ts only, behind a click on POST /api/system/update, so the restart that eats the plan is a crash or a systemctl restart. Both rare, both costing only the convenience.

Why not C, concretely. cleanupStaleSessions() (server.ts:2167) no longer calls the store directly. It goes through tabLayouts.runStaleSessionCleanup() (tab-layout-service.ts:536), which builds its own candidate list from the store (!active && !pinned) and runs a layout transaction per owner before cleanupSessionsByIds() ever sees the ids. A retain set has to reach both, or the layout service prunes the refs of the sessions the store kept. Add the marker field for your multi-restart case and C is three files plus a persisted-record change, all inside the pruning path, which is the code I least want a first PR to touch.

B is the upgrade, later. Your plan module is pure and a file is just a second producer of the same plan, so B can arrive without disturbing the pass. Not now.

Rules A needs that are not on your list yet. Each one follows from the plan being built at boot and consumed hours later, when the board has moved on.

• Scope the plan by owner. In multi-user mode the banner count and the click go through canAccessOwned like every list endpoint, so a user sees and restores their own entries and an admin sees all. The grant re-resolves at click time, as agreed.
• Dedupe at click time against LIVE sessions, by conversation id. Between boot and the click the user may have resumed one of those conversations by hand from the Resume list. Skip a plan entry whose resolveResumeConversationId() matches a live session's claudeSessionId, and skip an id already in this.sessions (the spike does the second). Two panes on claude --resume <same id> is the failure this prevents.
• Take-then-act, like take() in approval-inbox.ts. Remove the entries from the plan synchronously at the top of the route, before the first await, and single-flight the route. The spike's this.sessions.has(saved.id) guard sits after await this.getClaudeModeConfig(), so a double-click, or two devices, can both pass it and create two panes on one conversation.
• Re-check the workspace at click time. The plan-time check gives the banner its count; the click-time check decides what gets a pane. A repo can be deleted in between.
• Call startStatsCollection() unconditionally. It clears and re-arms its own interval (tmux-manager.ts:2885), so the click path needs no "not already running" guard, and a wrong one would be worse than none.
• Broadcast session_created per rebuilt session through the same path POST /api/sessions uses. The clicking tab has the response, but every other open tab and phone needs the event, and _onSessionCreated() is an idempotent upsert, so POST-first and SSE-first both render one tab.
• The banner needs a Dismiss. It clears the plan. With A that costs exactly what a restart costs, and the Resume list is still there.

Your stopped rule holds, and I checked the half you did not mention. cleanupSessionsByIds() (state-store.ts:537) keeps a pinned record WITHOUT touching its status, so a pinned session the reboot killed still reads idle or busy and stays eligible, while a pinned user kill goes through demoteOrRemoveSession() and lands as stopped. Refusing stopped therefore restores the pinned sessions the user cares most about and never the killed ones. Keep reusing the old session id as the spike does (id: saved.id): the pinned record, subagent-parents, window states and the lifecycle log all key off it, and under A the unpinned record is gone, so there is nothing to collide with.

Line numbers are against master at bd286bf5, one wiki-only commit past yours, so the src/ lines match.

Open the draft when the rebase is done.