Post-compartment cleanup synchronously reads all session parts, blocking OpenCode for 4.8–6.7 seconds

[null-axiom]

Environment

• @cortexkit/opencode-magic-context 0.41.4
• OpenCode Desktop 1.18.30, custom branch with additional performance diagnostics
• macOS, Electron's Node sidecar
• Large, long-lived sessions

Problem

After compartment publication, Magic Context's drop-queue cleanup reads every stored part in the session through synchronous SQLite. On large sessions, individual reads block the shared OpenCode sidecar for several seconds.

Observed symptoms include delayed session loading, output delivery, and Stop acknowledgment. These operations share the blocked backend; the evidence does not establish that this query explains every historical slowdown.

Measured evidence

Records from September 11, 2026; timestamps are EDT and identify record emission, not query start:

Timestamp	SQLite call duration	Returned rows
15:26:32.271	4,811.7 ms	101,919
15:27:44.091	6,702.1 ms	143,946

Both records identify the same SQL fingerprint and Magic Context call chain:

a96296ae25dd024a5600dbd5c8fd1edd8e09a47a1ac7b45e71127a853681f422

A separate live sidecar stack sample at 15:25:07 found 88 of 90 main-thread samples inside node::sqlite::StatementSync::All, including disk reads. That sample corroborates synchronous database blocking but does not independently identify the query.

Exact query

Installed bundle dist/index-0y2mcg5y.js:8470–8472:

SELECT message_id, data, time_updated
FROM part
WHERE session_id = ?
ORDER BY time_created ASC, id ASC

The query has no message-boundary restriction or page limit.

Caller chain

compartment publication
  → queueDropsForCompartmentalizedMessages
  → getRawSessionTagKeysThrough
  → readRawSessionMessages
  → readRawSessionMessagesFromSource
  → readRawSessionMessagesFromDb
  → synchronous full-session part query

Installed-bundle locations in dist/index-0y2mcg5y.js:

• queueDropsForCompartmentalizedMessages: 30673–30699
• getRawSessionTagKeysThrough: 9973–9974
• Raw reader/source: 9794–9804, 9954–9960
• Full-session part query: 8470–8479
• Publication callers: 37311, 37604, 37811

The recorded stack includes index-0y2mcg5y.js:8472:137 and :9960:40. The database wrapper in index-raatkvw5.js opens OpenCode's database read-only.

Both incremental Historian and recomp publication paths reach this cleanup. The incident logs do not distinguish which publication path triggered each read.

Expected behavior

Cleanup should avoid synchronously materializing the entire session's part history merely to determine tag ownership through a compartment boundary.

An existing paged reader appears relevant:

• Message pagination: 8495–8515
• Parts restricted to selected message IDs: 8518–8522
• Exposed as readRawSessionMessages.readPage: 9829

A possible correction is boundary-limited traversal with bounded hydration batches and event-loop yielding. Tool-invocation ownership must remain correct across page boundaries. Simply adding a row limit to the current query could omit required ownership information. This proposed correction has not been implemented or benchmarked.

Suggested regression coverage

• Identical tag/drop ownership to the existing reader, including tool calls spanning pages.
• No reading beyond the required message boundary.
• Bounded parts hydration on large sessions.
• Backend responsiveness during cleanup, not merely eventual query completion.

[null-axiom]

Follow-up: incremental message indexing also blocks the OpenCode sidecar

Further analysis identified a second blocking path in Magic Context 0.41.4: incremental message FTS/index maintenance.

This is separate from the full-session post-compartment cleanup query in the original report.

Exact query and measured impact

The following query executed 34 times in the captured records, totaling 69,958.73 ms of synchronous SQLite time:

SELECT message_id, data, time_updated
FROM part
WHERE session_id = ? AND message_id = ?
ORDER BY time_created ASC, id ASC

The SQL was verified against its SHA-256 fingerprint:

bd6aeef0c1fc72457e1e627549ad30c4c62fc658d6a613b225f2a6526cc944f1

Unlike the original cleanup query, this query selects parts for one message. The indexing caller and SQL are verified; the reason individual message lookups take seconds still requires query-plan/index investigation.

Installed source attribution

Package: @cortexkit/opencode-magic-context 0.41.4.

• SQL site: dist/index-0y2mcg5y.js:8749
• Message lookup wrapper: dist/index-0y2mcg5y.js:9942–9952
• Feature source marker: dist/index.js:21883, identifying src/features/magic-context/message-index-async.ts
• Scheduler execution: dist/index.js:21990–22023
• Message event trigger: dist/index.js:25895–25906

Recorded callsites include:

index-raatkvw5.js:86:106
index-0y2mcg5y.js:8749:156
index-0y2mcg5y.js:9952:40
index-raatkvw5.js:210:10
index-0y2mcg5y.js:9952:10
index.js:21999:61
index.js:21931:11

The logical path is:

message.updated for a user message or completed assistant message
  → scheduleIncrementalIndex
  → readRawSessionMessageById
  → synchronous parts lookup

The 100 ms debounce deduplicates pending work by session/message identity, not across different message IDs. Subsequent source-identity checks do not eliminate the initial database lookup. The diagnostic does not record message IDs, so 34 executions cannot be assumed to represent 34 distinct messages.

Scheduling this feature asynchronously does not make the underlying synchronous SQLite call nonblocking.

Broader captured blocking

The capture spans September 11, 2026, 19:21:49.184–19:51:43.636 UTC (approximately 30 minutes), on OpenCode Desktop 1.18.30 with additional performance diagnostics.

Attribution	Recorded calls	Total synchronous SQLite time
Original Magic Context full-session cleanup query	2	11.5138 s
Other queries with exclusively Magic Context callsites, including the indexing query above	93	117.17062 s
Mixed Magic Context/OpenCode callsites	1	0.07331 s
OpenCode-only callsites	4	7.07092 s
Total	100	135.82865 s

Approximately 128.68 seconds had exclusively Magic Context callsites. The 69.96-second indexing total is included in the second row, not additional to it. Recorded statement intervals did not overlap.

This establishes repeated blocking across the interval, not one uninterrupted freeze. Event-loop summaries in the same interval recorded a maximum sampling delay of 12.13 seconds; that measurement is not independently attributable to one query.

Observation limits

The SQLite diagnostic records only calls taking at least 50 ms, with caps of 10 records per minute and 100 records per process. This capture reached the total cap.

Consequently, these counts and durations are lower bounds, not all database work during the interval. Gaps between records do not prove the backend was idle or responsive.

Investigation and correction considerations

• Inspect the query plan and available indexes to establish why a single-message lookup takes seconds.
• Assess redundant event-driven reads and opportunities to reuse already-hydrated message data or coalesce indexing work without losing updates.
• Preserve eventual indexing correctness for completed messages and subsequent changes.
• Do not replace the single-message lookup with a full-session read.
• Validate backend responsiveness as well as indexing correctness.

[magic-alfonso]

Confirmed at source. After each OpenCode compartment publication, the drop-queue path called getRawSessionTagKeysThrough(), which called the full raw-session reader. That reader issued one synchronous, unbounded part query for the entire session before the ordinal boundary was applied in JavaScript, so parts after the newly published compartment were read and parsed unnecessarily on the sidecar main thread.

The fix changes this path to walk only through the published compartment end ordinal with the existing message-paged reader. Each page restricts its parts query to that page's message IDs, and the traversal yields to the event loop between pages. FIFO invocation state is retained across pages, including invocation/result pairs that straddle a page boundary, and result-only observations use the same nearest-prior persisted-owner fallback as the tagger. Publication callers collect the bounded key set before opening their write transaction, then commit the derived drop rows atomically with the compartment state. Incremental, full recomp (including its early-finalization path), Pi historian, and wrapup-through-incremental paths use the bounded flow.

Regression coverage compares the paged key set with the legacy unpaged result over more than two pages, verifies spanning tool-call ownership, caps parts hydrated per fixture batch, asserts the whole-session parts SQL is absent, and exercises a 100,000-part OpenCode-shaped SQLite session. On this machine the same timer-gap test measured 298.1 ms with the old whole-session read and 8.2 ms with the paged traversal (asserted at <=100 ms).

The Pi twin did not query SQLite, but its fallback provider did convert the full JSONL branch into RawMessage[] during this cleanup. It now exposes a bounded Pi page converter, so cleanup does not hydrate the rest of the branch. The Rust publication path queues drops store-side and does not traverse opencode.db, so no Rust change was needed.

One indexing clarification: time_updated was selected but was not an ordering key; the old query ordered by time_created, id. A (session_id, time_created) index can serve the session/time prefix but may still need tie sorting on id. The replacement parts query is bounded by selected message IDs and orders by message_id, time_created, id, matching an index-friendly (session_id, message_id, time_created, id) layout.

This will ship in the next patch.

[null-axiom]

Thanks, this addresses the full-session cleanup finding. Does the patch also address the separate incremental FTS/index-maintenance path in the follow-up?

I recorded 34 executions totaling 69,958.73 ms for:

SELECT message_id, data, time_updated
FROM part
WHERE session_id = ? AND message_id = ?
ORDER BY time_created ASC, id ASC

The caller was message-index-async.ts through scheduleIncrementalIndex → readRawSessionMessageById, rather than compartment cleanup.

Does this patch change that lookup’s query plan/index selection or its repeated execution? The paged cleanup correction alone appears to leave that path unchanged.

[magic-alfonso]

Thanks — that is a separate path and a real one. The query you captured comes from the incremental FTS indexer (message-index-async.ts → readRawSessionMessageById), which re-reads a message's parts on each message.updated event while a response streams. It is not touched by the paged cleanup fix above.

What we found (c353343a, master): on a 433k-part copy of our own opencode.db we could not reproduce the ~2 s per execution — EXPLAIN QUERY PLAN shows OpenCode's part(session_id, message_id) index serving the lookup on both the old and new shapes, so the plan itself was already index-bound here. What can produce your timing is planner statistics: with skewed sqlite_stat1 rows the planner can prefer the session_id prefix and walk the whole session. We reproduced that shape with an adversarial-stats fixture (170 ms → 0.05 ms per 34 reads after the change) and hardened the lookup with an explicit selectivity hint so the plan no longer depends on statistics. Separately, the indexer now coalesces the per-event re-reads to one read per message (8 → 1 in the streaming fixture), which removes most of the 34 executions regardless of plan.

Four other event-driven reads of opencode.db are catalogued as follow-ups with their bounds; none is on the transform hot path. If you can share EXPLAIN QUERY PLAN for that statement on your database (and whether ANALYZE has ever run on it), it would confirm which of the two mechanisms you were hitting. Ships in the same patch.