feat(sessions): durable reconnect with a per-session sequence, snapshot, and replay

[mmabrouk]

Context

A client that disconnects during a run has no cursor to resume from: the watch route sends change notices, and the client reloads the whole transcript. This is increment 4, part two, of the session-control design on PR #6495, stacked on the live relay in PR #6522. It gives every durable fact a database-defined per-session order, a snapshot that carries that order, and a replay that continues from it.

Changes

• Sequence: a cursor table on the analytics database (session_sequence_cursors) and a nullable sequence column on records, allocated in the same transaction as the record insert, so concurrent inserts get distinct increasing sequences per session and a retry keeps its sequence. Old rows keep null. Behind the history-writes switch; off means nothing changes.
• Durable events: when a record with a sequence commits, the six typed events (execution.started, execution.stopped, execution.failed, execution.lost, message.completed, tool.completed) are published as kind: event envelopes on the same stream the relay tails, after commit.
• Snapshot: GET /sessions/{id} returns the grouped shape with read.latest_sequence and read.history_complete, with a paged transcript.
• Replay: GET /sessions/{id}/events?after=N sends committed events after N in order, then follows live, subscribing before it queries so nothing is missed; a legacy session with null sequences loads in created-at order and is flagged incomplete.
• Client: the reader in @agenta/chat and @agenta/entities loads the snapshot, follows after its sequence, applies durable events idempotently, discards previews on reconnect and refetches, and ignores unknown event types. Storybook stories for the incomplete and reconnecting states.

Tests

• API sessions unit and integration suites on Postgres: 560 passed; @agenta/chat 645; @agenta/entities 1,469; type checks and lint clean; Storybook builds.
• Rebased onto the relay round 2 (7ae0792252); one test-file conflict resolved.
• Whole-PR review by an Opus agent: ship after fixes, six inline comments on the PR. The core invariants hold and are tested (same-transaction allocation, deadlock-free cursor upsert, retries keep their sequence, publish after commit, subscribe-before-query with a no-gap test, legacy sessions flagged incomplete, idempotent client reducer). Round 2 (44ed249812) renumbered the tracing migration to 006 on top of fix(sessions): settle executions after runner or sandbox loss #6501's 005 (with a one-head guard test) and scoped the cursor key, the unique index, and the read state by project. Re-review: ship as a merge set with fix(sessions): settle executions after runner or sandbox loss #6501, because 006 chains onto fix(sessions): settle executions after runner or sandbox loss #6501's 005 and Alembic fails on this branch alone. The setting is named AGENTA_SESSIONS_SEQUENCE_WRITES here and folds into AGENTA_SESSIONS_HISTORY_WRITES when this PR and PR feat(sessions): separate session history from tracing retention and gate immutable record writes #6517 merge, so there is one switch per increment. Live proof, first pass (relay stack on the merge set with fix(sessions): settle executions after runner or sandbox loss #6501): replay after N delivered the typed events in order with no duplicates (sequences 3, 6, 7, 9 after a snapshot at 1), but the client reducer demanded a contiguous chain and never applied them, because every record consumes a sequence and only the six typed events are relayed; a reconnect masked it by resetting the watermark. Round 3 in progress: every event envelope carries the session's current watermark, the catch-up ends with the watermark, and the reducer applies any newer event and advances to the watermark. Legacy session: pass (a session from before the branch loads with latest_sequence 0, history_complete false, eleven null-sequence records in creation order, and the reducer skips null sequences). Concurrent writes: pass (two sessions writing at once keep separate, strictly increasing counters, and each events connection carries only its own session). Reconnect: pass at the protocol level (the durable path advanced from 31 to 34 while the reader was gone; the reconnect after 34 delivered 37 and 39 with nothing repeated). Correction to the masking: a reconnect does not rescue the reducer either; what hides the defect is the unconditional transcript refetch on every connect, which reads the records query with a full contiguous sequence. The reducer fix in round 3 is therefore required, not optional. Round 3 is pushed (head 7c11118): the reducer applies any event newer than its watermark, so a sparse sequence such as 1, 3, 6, 7, 9 applies each event once, and duplicates and older events are dropped. The round 3 review verdict was ship with three small findings; round 4 (head bcc4d52) fixes them: no transcript refetch on a watermark-only advance, the two watermark meanings documented at each site, and the wire field described in the DTO and the zod schema. Scenario 1 re-ran on the proof stack with this head: pass. The second reader rendered tool progress within about a second of the sender, a reconnect after a gap delivered the sparse sequences 61 to 85 without loss or repeat, and the transcript refetch fired six times over about seventy seconds of a turn that wrote dozens of records, not once per write. The two Postgres-backed tests (replay and snapshot) run against the proof stack database: 3 passed, after their fixture was switched to the pytest_asyncio decorator.

Agent-generated, low weight. Not merged.

https://claude.ai/code/session_01GAqSs7fw6QRi2n1ZJ2tmAV

Merge notes: landing this after PR #6501

A throwaway merge of this branch with #6501 (for the live proof) conflicted in five files; the resolutions to repeat when this PR lands after #6501:

• apis/fastapi/sessions/models.py: keep this PR's live-frame fields on the ingest request and fix(sessions): settle executions after runner or sandbox loss #6501's cancel, command, execution, and settlement models.
• apis/fastapi/sessions/router.py: keep this PR's root router wiring and mount fix(sessions): settle executions after runner or sandbox loss #6501's control router.
• core/sessions/records/service.py: keep durable_events_from_records for replay together with fix(sessions): settle executions after runner or sandbox loss #6501's settled-execution terminal guard, late-record quarantine, and ending-written marker.
• dbs/postgres/sessions/records/dao.py: union the SQLAlchemy primitives; keep sequence allocation in the same analytics transaction as the insert; a quarantined record gets NO sequence (stored as evidence, never advances the replay watermark), a redelivered already-sequenced record keeps its sequence, and the quarantine filter applies to every transcript-facing reader (snapshot pages, replay after N, history completeness); a Postgres regression test covers it.
• engines/sandbox_agent/errors.ts: keep both error classifications.
  env.py and the env examples merge to the union. Migration heads after the merge: core 026, tracing 006. The merged tree passed ruff, the runner suite (2,751), and the API session suite (580 outside the migration-dependent set, which needs the migrated database).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
agenta-documentation	Ready	Preview	Sep 5, 2026 10:18am UTC

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

• release/.*

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Team

Run ID: b9d94330-30b0-43ea-bb80-4d59fea1028f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[mmabrouk]

Agent-generated review, low weight. One human should confirm every point below before acting on it.

Whole-PR review of the durable-reconnect unit (6 commits over feat/session-live-relay, 46 files). The design contract is docs/design/session-control-and-live-events/contracts/{persistence,events,public-api}.md.

Reading guide (5 lines):

1. Sequence allocation and idempotency: api/oss/src/dbs/postgres/sessions/records/dao.py _append_sequenced plus the migration.
2. Post-commit publish and the subscribe-before-query replay: records_worker.py, core/sessions/records/events.py, apis/fastapi/sessions/live_events.py.
3. Snapshot, read state, and watermark paging: router.py get_session_snapshot, dao.py get_read_state / get_records_page.
4. Client side: web/packages/agenta-chat/src/model/durableEvents.ts reducer and hooks/useSessionLivePreview.ts reconnect.
5. Coordination with PR #6517: history_incomplete is read defensively; its migration is on the core_oss chain, so no migration-file clash there.

Verdict: ship after fixes. The core invariants hold and are well tested (concurrent distinct sequences, retry does not consume a sequence, post-commit publish, subscribe-before-query no-gap with a direct test, legacy sessions flagged incomplete without backfill, idempotent client reducer, Fern for HTTP with EventSource as the SSE exception, Storybook present). One hard blocker: the migration revision id collides with feat/session-execution-watchdog. See inline comments.

[mmabrouk]

Agent-generated follow-up, low weight. Round 2 (head 44ed249812) resolves the P1 migration collision (renumbered to oss000000006 on 005, with a chain guard asserting one head 004->005->006) and project-scopes the cursor PK, the unique index (project_id, session_id, sequence), and the DAO upsert; 13 unit tests pass here. One caveat, not a blocker: this branch does not carry #6501's 005, so it must merge as a set with #6501 (alembic on this branch alone dangles on the missing 005). Postgres integration tests were skipped in my session (no reachable analytics DB); those DB assertions are author-claimed, not re-verified by me. Ship as a merge set with #6501.

[mmabrouk]

Round 3 review (commit 7c11118, "accept sparse durable event sequences") — verdict: ship.

The core fix is correct and well-tested. The client reducer no longer demands a contiguous chain: it applies any event with sequence > lastEventSequence, drops duplicates and out-of-order older events, and advances a separate latestSequence (reconnect cursor) via max(sequence, watermark). The server now bounds replay to the committed cursor and reports that cursor as the ready watermark, so a gappy tail (records committed without a typed event) still advances the client cursor.

Correctness checks (all pass):

• Sparse 1,3,6,7,9 applies each event exactly once (unit + integration tests assert [3,6,7,9], watermark 9).
• A duplicate or an older sequence is dropped (sequence <= lastEventSequence).
• Reconnect re-anchors to snapshot.read.latest_sequence in useSessionLivePreview, so no loss even if the in-memory cursor overshot; the host also re-reads the durable transcript on every gap.
• Legacy null-sequence path stays ignored in the reducer (unchanged behavior).
• The SessionSequenceCursorDBE.latest_sequence cursor and the record's sequence commit in the same transaction, so the replay snapshot (get_records_after) is consistent even though the cursor read and the rows read are separate statements: rows are bounded <= watermark.

Wire contract: the watermark producer change is in this same commit — no separate API change is pending. watermark is a required field on the durable-event envelope (DTO SessionDurableEventBase, ready-frame payload) and on the client zod schema. Old-server fallback is handled: the ready handler defaults watermark to the connection's after when the field is absent or malformed.

Tests run locally:

• pnpm --filter @agenta/chat test -> 61 files, 652 passed.
• pytest oss/tests/pytest/unit/sessions/ -> 505 passed, 41 skipped (Postgres-gated).
• The integration replay test test_records_replay_postgres.py skipped here (Postgres unreachable in the worktree). It is well-formed and asserts the exact sparse sequence against real Postgres; run it on a stack with the analytics DB before merge for full assurance.

Non-blocking findings (nits, posted inline):

1. Live-event watermark is the per-batch max; replay/ready watermark is the session cursor. Same field, two meanings. Safe (batch-max <= cursor, reconnect re-anchors), but a latent trap. A comment or alignment would help.
2. useSessionLivePreview triggers a full transcript re-fetch on any latestSequence change, including watermark-only advances that append nothing renderable. Key it off the events-length change instead.
3. Docs drift: the new required watermark wire field is not recorded in any design/contract doc in this diff (no docs/ files changed in the commit range). Add a follow-up doc note.

No blockers.

https://claude.ai/code/session_01GAqSs7fw6QRi2n1ZJ2tmAV

[mmabrouk]

Agent-generated review, low weight. One human should confirm each finding before acting on it.

Reviewed the complete 47-file delta at 664d1b7d208abbeda008a9095d216353625bff0d, including the existing reviews and comments.

Review summary

Verdict: do not merge yet. I found two blocking data-loss risks and one incomplete client integration.

P1: Durable event publication can trim unprocessed records

publish_durable_event calls XADD MAXLEN directly on the shared streams:records stream. If that stream exceeds the configured limit while the records consumer has pending work or lag, Redis can trim durable record entries that have not reached Postgres. The existing trim_live_stream helper deliberately guards the count trim behind the records consumer's acknowledgement frontier, but this new path bypasses that protection.

The loss is silent because a removed record never receives a database sequence, so history_complete cannot detect the missing input.

Smallest fix: append the relay envelope without MAXLEN, then use trim_live_stream, or move relay envelopes to a separate bounded stream. Add a test where the records group has lag and verify publication cannot cross its acknowledgement boundary.

Code: api/oss/src/core/sessions/records/streaming.py:276-281.

P1: Opposite multi-session batch orders can deadlock

The sequenced append_many path locks session cursor rows in input order inside one project transaction. Multiple records-worker replicas can therefore process batches [session A, session B] and [session B, session A] concurrently. Each transaction can lock its first cursor and wait for the other, causing Postgres to abort one as a deadlock.

The worker currently catches the append failure but still returns the message IDs collected before the write, so the rejected batch can then be acknowledged and permanently lost.

Smallest fix: group records by session and acquire cursor locks in one stable session order while preserving record order within each session. Add a Postgres test with two concurrent batches containing the same sessions in opposite order. Failed database batches must remain pending.

Code: api/oss/src/dbs/postgres/sessions/records/dao.py:130-140.

P2: The client does not use the snapshot-bound transcript

useSessionLivePreview fetches the snapshot and passes it to onDisconnect, but the caller ignores that argument and runs the old loadSessionMessages path. That path still calls the unbounded querySessionRecords; the new querySessionTranscript(throughSequence) function has no caller.

The production client therefore does not load a transcript fixed at snapshot.read.latest_sequence, still relies on legacy timestamp ordering, and never surfaces snapshot.read.history_complete=false. SessionHistoryNotice is only used by Storybook, so a partially preserved session with some records appears complete.

Smallest fix: hydrate through querySessionTranscript with the snapshot watermark, expose reconnecting and incomplete states from the hook, and render SessionHistoryNotice in the production chat surfaces. Add a hook-level reconnect test.

Code: web/packages/agenta-chat/src/hooks/useSessionLivePreview.ts:75-85, web/packages/agenta-chat/src/hooks/useAgentConversation.ts:704-714, and web/packages/agenta-entities/src/session/api/api.ts:98-140.

Verification

• git diff --check passed.
• All 25 changed Python files compiled successfully.
• Exact-head formatting, lint, secret scanning, and contribution checks passed.
• API, web, runner, and integration test jobs were skipped in the reported exact-head check run.
• I could not independently run pytest or pnpm tests because the temporary checkout had no installed dependencies.
• The known migration dependency remains: this branch's tracing revision 006 requires PR #6501's revision 005.

[github-actions]

Railway Preview Environment

Status	Destroyed (PR closed)

Updated at 2026-09-05T10:45:16.920Z

[mmabrouk]

🤖 The AI agent says:

This PR adds durable sequence cursors, snapshot recovery, and replay into the live client.

Focused simplification review using the new simplify skill: one non-blocking suggestion below. This is a static design/code review, not a rerun of the live release gate or a replacement for existing correctness reviews.

[mmabrouk]

🤖 The AI agent says:

Correctness follow-up: one additional replay failure reproduced below. I checked the existing review and am not duplicating its record-trimming, transaction-order, or snapshot-integration findings.

[mmabrouk]

Agent-generated Codex review, low weight.

CHANGES NEEDED

The replay handoff itself is correctly ordered: subscription precedes the Postgres replay, replay is bounded by the captured watermark, and ready is queued only after replay. The reducer also separates the reconnect cursor from the last accepted event sequence and avoids watermark-only refetches.

Four defects remain at this head: two paths can silently lose record history, the client does not actually consume the snapshot-bounded transcript, and the fixed reconnect loop can amplify Redis/Postgres failures. Details are inline.

The migration is additive and nullable, and flag-off preserves the old runtime path once the migration is present. The migration must precede this binary because the ORM maps records.sequence even when allocation is disabled. I did not repeat the resolved oss000000005 dependency thread.

Validation: reviewed 2146213306...fc02d8554b as one 47-file delta, read all ten resolved threads, and ran git diff --check. No test file was rerun; the readiness report already covers the focused replay/API/frontend tests, while these findings need missing concurrency and hook-integration coverage.

[mmabrouk]

Agent-generated Codex review, low weight.

CHANGES NEEDED

The rebase delta is scoped to durable reconnect and its five new base-integration/test files. The base's separate live stream, record-ingest MAXLEN 100000, isolated relay startup, and gap-rejecting reducer remain intact. Durable events no longer touch streams:records; cursor locks are stable and failed project batches are not acknowledged; reconnect delay resets only on ready.

The reconnect order is snapshot N, bounded transcript through N, awaited host callback, then SSE after N. That costs one snapshot request plus max(1, ceil(records/100)) transcript requests per open; a reconnect also triggers one immediate host records refresh before the delayed bounded hydration. The two cursor-handoff problems and the fresh-migration failure are inline.

Validation on dedicated fresh databases: core legacy/OSS and tracing legacy reached head; tracing OSS failed because revision oss000000005 is absent. API sessions: 550 passed, 4 failed, 9 warnings. Sequence: 0 passed, 3 failed, 3 warnings. Snapshot: 0 passed, 1 failed, 3 warnings. Replay: 0 passed, 2 failed, 3 warnings. All backend failures follow from the tracing migration failure. Focused live-preview hook: 2 passed. git diff --check passed.