Skip to content

ARCHITECTURE.md

Latest commit

 

History

History
179 lines (136 loc) · 77 KB

ARCHITECTURE.md

File metadata and controls

179 lines (136 loc) · 77 KB
Raw

Architecture

All src/ paths are relative to packages/plugin/ (the published npm package). File locations live in STRUCTURE.md; this document explains how the pieces fit and — above all — the invariants that keep the Anthropic prompt cache stable. When in doubt about transform behavior, read "Transform pass mechanics" below before touching code.

Overview

Magic Context is an @opencode-ai/plugin (entry src/index.ts) that rewrites the message array and system prompt on every LLM call to keep a long session inside the context window without losing history. Core tenets:

  • Thin adapters, real logic separated. OpenCode-facing handlers live in src/plugin/; feature logic in src/hooks/magic-context/ (runtime), src/features/magic-context/ (services), src/tools/ (agent tools).
  • Durable SQLite state, never ephemeral — if storage is unavailable the plugin fails closed rather than silently letting the prompt grow past the provider limit. DB at ~/.local/share/cortexkit/magic-context/context.db (overrideable via MAGIC_CONTEXT_STORAGE_DIR), shared cross-harness (OpenCode + Pi) and across all projects on the machine; session-scoped tables carry a harness discriminator (including session_projects for session-to-project mappings backfilled on boot), project-scoped tables (memories, git commits) are shared.
  • Replay-everything for cache stability. Every persistent message mutation (reasoning clearing, structural-noise / placeholder / image / merged-assistant stripping, caveman compression, synthetic-todowrite, drop placeholders, and smart-drop/edit-marker compressions) is re-applied deterministically on EVERY transform pass — including defer passes — so the wire bytes stay byte-identical and the provider prompt cache survives.
  • Hidden subagents (historian, historian-editor, and dreamer) do the heavy LLM work out of band; the transform itself does no LLM calls.
  • Runtime SQLite backend (src/shared/sqlite.ts): bun:sqlite under Bun, node:sqlite (DatabaseSync) under Node (Pi) and Electron (Desktop). The non-Bun branch adds a savepoint-aware transaction() shim and readonly→readOnly mapping; otherwise identical. No native module, no prebuild.
  • Pi parity: packages/pi-plugin/ mirrors OpenCode semantics, importing shared core from @magic-context/core. Intentional divergences are tracked in packages/pi-plugin/PARITY.md. OMP is supported as a first-class harness via a four-rung detection ladder (process.title → host package name via realpath walk → ESM APP_NAME probe → launcher basename via piHarnessKindFromExecutable in packages/pi-plugin/src/pi-harness-kind.ts), logging the deciding rung at boot, with dedicated omp configuration and subagent resolver blocks (bin.omp). Provider prefixes are translated between canonical (OpenCode) and Pi configuration models via src/shared/harness-provider-map.ts at configuration read/write edges to keep shared model configurations portable. Pi implements session state inheritance on branch clone forks by copying filtered tags, compartments, and pending marker states, whereas OpenCode /fork does not yet inherit context state. Pi implements Last-Known-Good (LKG) replay on transient SQLite failures (packages/pi-plugin/src/pi-lkg.ts, sharing content-sensitive snapshot helpers with OpenCode to reuse SHA digests across stable prefixes and sign only changed suffixes in deferred capture), aligns pressure accounting using one prompt-tokens/usable-window snapshot driving the scheduler, historian trigger, transform logs, status, and footer (with no forward-pressure scaling factor), delivers Channel 2 nudges mid-run as steer (triggerTurn: true), tracks per-pass served-array digest ledgers for cache-bust attribution (packages/pi-plugin/src/served-array-ledger.ts), aborts the turn with a display-only retry entry plus ctx.abort() when a context handler throws (registerPiGuardedContext in packages/pi-plugin/src/pi-context-refusal.ts, wrapping both the main handler and the fail-closed hook, because Pi otherwise swallows hook exceptions and continues unprotected), and supports concurrent multi-session execution in embedded RPC hosts with shared Dreamer registration.
  • Rust module / subc integration. A harness-agnostic Rust workspace in crates/ re-implements the cache-stability transform and autonomous historian to run under the subconscious daemon (subc), communicating via the @cortexkit/subc-client library on the TypeScript side and wire v2 protocols on the Rust side. All host and script route-open sites pin consumerIdentity: null so daemon credentials in ambient SUBC_* environment variables are not inherited by independent hosts.
  • Experimental Rust runtime mode. Gated by transform_mode: "rust", routes the entire Magic Context transform pipeline for a project through the ck-mc Rust module over subc. The TypeScript layer serves as a coordinator that manages state sync, ordinal tracking, and Last Known Good (LKG) fallbacks (freezing served fallback representations across defer recovery to prevent per-blip double busts, releasing the freeze if LKG replay fails validation or hits 8 healthy passes or 16 new raw messages, deferring snapshot hashing behind exact accepted-prefix reuse bound to capture sequence proof, and committing priced LKG captures synchronously to durable storage before returning). Rust mode advances the OpenCode compaction marker from the module's materialized boundary through the TS CAS path, shrinking filterCompacted input arrays across passes. The module retains fingerprints on non-materializing plans to prevent restart double-busts, holds demoted signed assistant native vectors (native_reasoning_keep_mids) until priced passes, enforces typed protocol responses (rejecting decisionless wire responses with RustTransformProtocolError), and keys candidate mural caching by canonical image-support verdicts (modelKeyAcceptsImages) across same-model SDK refreshes. Cold-start execution accelerates by serializing only servable messages, priming boundary caches with durable tag token counts, and scaling deadlines with seed size (30s + 2ms/message, capped at 120s). Stuck transform historian waits in emergency are bounded to a shared 20s budget (TRANSFORM_HISTORIAN_FOLLOWUP_BUDGET) with typed state=stuck health reporting. Authority workspace activation is idempotent (get-or-create), typed non-retryable store constraint failures park the session (state_sync_non_retryable), and mirror drains use recency-guarded atomic transactions to prevent stale snapshot rollbacks, draining pages within a bounded page budget (drainMirrorPages / TRANSFORM_MEMORY_MIRROR_PAGE_BUDGET) and stamping the projection key only when the mirror drain completes to avoid adopting partial backlogs.

Layers

  • Bootstrap (src/index.ts): load config, register hidden agents + hooks + tools, start RPC server, dream-timer, auto-update checker; detect conflicting plugins (DCP / OMO / OpenCode auto-compaction) and disable the runtime if any is active. Begins the boot quiet period (packages/plugin/src/plugin/boot-quiet.ts) to defer background maintenance, and bounds plugin boot initialization to a 15-second server-wide budget (BOOT_SERVER_DEADLINE_MS in src/plugin/boot-deadline.ts) with per-phase timing attribution and a synchronously flushed boot: entering breadcrumb. Configures a 5s SQLite busy_timeout before first fence reads, avoids taking BEGIN IMMEDIATE during current-schema checks when no migrations are pending to eliminate dual-instance boot writer contention, arms loud fail-closed blocking if the hooks phase times out while adopting late-settling hooks immediately upon completion, and defers non-blocking background tasks (dream timer registration, announcement loading, RPC startup) past boot return.
  • Adapters (src/plugin/): hook wrappers, tool registry, RPC handlers, dream-timer lifecycle, per-session hook construction, and the rustToolBackends tool-routing layer when running in Rust mode.
  • Runtime (src/hooks/magic-context/): the transform pipeline, postprocess phase, event/command handlers, system-prompt injection, compartment runners, decay rendering, strip-and-replay, nudges, and m[0]/m[1] injection. Reasoning-variant flips (src/hooks/magic-context/hook-handlers.ts) check variantChangeBustsProviderCache (src/hooks/magic-context/sentinel.ts); Anthropic-family providers flush pending ops to ride the natural prompt cache bust, while cache-preserving models (Anthropic Fable 5.1, OpenAI GPT-6 Astra) and implicit-prefix providers defer flushes to avoid gratuitous busts. Intercepts Desktop slash-stripped commands before persistence at the chat.message hook seam (src/hooks/magic-context/stripped-command.ts), routing through command handlers and throwing a 204 suppression sentinel before LLM dispatch. Tool execution before dispatch intercepts dropped placeholder arguments via src/hooks/magic-context/dropped-input-guard.ts (and packages/pi-plugin/src/dropped-input-guard-pi.ts on Pi), throwing non-executable recovery errors when an argument matches copied placeholder shapes. When transform_mode is set to "rust", delegates execution to src/hooks/magic-context/rust-mode-transform.ts which synchronizes database state via src/hooks/magic-context/module-state-sync.ts, maps message ordinals via src/hooks/magic-context/module-wire.ts (with lifecycle-invalidated ordinal mapping instead of polling persisted state each pass), and falls back to LKG (Last Known Good) transform state replays via src/hooks/magic-context/lkg-slot.ts and src/hooks/magic-context/lkg-replay.ts on failures (freezing served fallback representations across defer passes until an authorized cache-busting pass adopts module output, or releasing the freeze when LKG replay cannot validate or exceeds healthy-pass / raw-tail limits), parking immediately on typed state_sync_non_retryable errors.
  • Feature services (src/features/magic-context/): storage, scheduler, tagger, memory, dreamer, git-commit + message FTS indexes, unified search, overflow detection, session-project mapping and backfill, migrations, clone-state copy helpers (src/features/magic-context/storage-clone.ts), project identity resolution (resolves git:<sha> or fallback dir:<md5-12> identifiers, caching directory fallbacks, utilizing a cooldown period for transient git errors, and identity merging via src/features/magic-context/storage-identity-merge.ts), domain authority management (src/features/magic-context/context-authority.ts, declaring authority transitions once per project transition on settled states, with authoritative source identity rebinding, mirror page prebinding so memory IDs survive round-trips, and bounded multi-page mirror drains via drainMirrorPages), and schema fence probes (src/features/magic-context/schema-fence-probe.ts).
  • Tools (src/tools/): ctx_reduce, ctx_expand, ctx_note, ctx_memory, ctx_search. Gating: when memory.enabled is false, ctx_memory is not registered (on Pi, it registers but refuses queries for memory-off projects), all memory-related guidance is stripped from the system prompt, and all memory-derived prompt injection surfaces (project-memory baseline/deltas, memory-update deltas, user-profile baseline in m[0] and deltas in m[1], and the mural) are suppressed across TS, Pi, and Rust module lanes (with configuration flips riding an immediate render-config HARD path).
  • Config + shared (src/config/, src/shared/): Zod config (deep-merge raw JSONC before validation; invalid leaves fall back to defaults with warnings, never disable the plugin). Recover unparseable magic-context.jsonc via deep-imported ESM jsonc-parser while maintaining prototype-pollution boundaries, distinguishing file-parse, file-io, and invalid-leaf warning classes and broadcasting parse failures loudly across OpenCode banners, Pi session-start notifications, and /ctx-status / status dialogs (src/shared/config-diagnostics.ts, src/shared/config-warning-surface.ts). Resolve status-only cache TTL text and provenance from live config and model (src/shared/cache-ttl-display.ts, src/shared/cache-ttl-seed.ts) while session rows are unsynced. Support first-class OMP configuration blocks (historian.omp, dreamer.omp, profile omp overlays). Model-selection profiles (src/config/profiles.ts) resolve named profile overlays with project > user > none precedence, transporting host-resolved historian and dreamer model chains into Rust-lane module requests. Top-level mural block config controls mural rendering parameters (graduated from experimental namespace). Project-tier trust boundaries (src/config/project-security.ts) strip unsafe fields from untrusted repository configs before merge — preventing escalation via sqlite PRAGMAs, hidden-agent reprogramming (prompt, tools), embedding destinations, profile definitions (profiles), or developer-only shadow transform settings. Project-level compaction thresholds are raise-only to prevent cloned repos from forcing extra historian cost. Test runs isolate config directories via pinned XDG_CONFIG_HOME (test-preload.ts) and fail closed on direct network provider construction. Also includes logger (with bounded 32 MiB rotation to one .1 predecessor), data paths, SQLite selector, harness id, RPC transport, conflict detector, test temp directory lifecycle management (src/shared/test-temp-dir.ts), tag-transcript primitive (shared with Pi), provider-id translation map (src/shared/harness-provider-map.ts), export-aware TUI runtime import specifiers mapping (src/shared/tui-runtime-specifiers.ts), prompt surface config resolution (src/shared/prompt-surface.ts, src/shared/prompt-surface-runtime.ts), window geometry derivation with output_reserve override support (src/shared/window-geometry.ts), light tool descriptions (src/tools/light-descriptions.ts), exit-abort listener coordinator (src/shared/exit-abort-registry.ts), and slow write-transaction timing attribution (src/shared/write-transaction-timing.ts).
  • TUI (src/tui/): sidebar + /ctx-status / /ctx-recomp dialogs, RPC-backed; shipped as raw TS via the ./tui export (not bundled into dist/index.js).
  • CLI (packages/cli/, separate @cortexkit/magic-context package): npx setup / doctor / migrate wizard, including session migration (doctor migrate-session with TS authority fencing), cross-harness OpenCode→Pi/OMP migration (doctor migrate with migration_pending crash recovery and module-managed project authority checks), pinned plugin schema fence validation against the live database (packages/cli/src/lib/opencode-plugin-schema-fence.ts), dual-grammar/dual-path log parsing in doctor, diagnostics, and issue bundles (packages/cli/src/lib/log-lines.ts, pinned to the subconscious golden fixture), keying recent errors on log level and reading the omp/ log path in doctor-omp, GitHub issue diagnostic bundle generation on failed --issue submissions (packages/cli/src/lib/github-issue.ts), and local embedding runtime checks for missing or broken onnxruntime-node native and onnxruntime-web WASM bindings (packages/cli/src/lib/embedding-runtime.ts). Database access adapters (packages/cli/src/lib/database-access.ts) bypass Bun Linux URI incompatibilities by utilizing the { create: false, readwrite: true } options object instead of file:// URIs when opening existing SQLite databases.
  • Rust Core & Store (crates/mc-core/, crates/mc-store/): Re-implement cache-stability transform/classification logic and durable SQLite schemas with row-version CAS in Rust. It implements a durable mc_reduce_command_ledger table (via migration 16) to provide command-id idempotency for ctx_reduce agent-drop requests, mc_project_mural_artifacts (via migration 49) for project-scoped host-rendered mural storage, persists raw deflated messages alongside chunk transcripts (via migration 50) for durable ctx_expand recovery, records memory mapping origins (via migration 51), persists compiled note metadata (via migration 52), and logs store-ahead migration outcomes without refusing startup on rollback.
  • Rust Tokenizer (crates/mc-tokenizer/): Tiktoken byte-BPE Claude token estimator.
  • Rust subc Module (crates/mc-module/): Integrate with the subc daemon, handle transform requests, route MCP facade requests (for ctx_memory, ctx_search, ctx_expand, and ctx_note), and orchestrate the autonomous historian. Use a bounded 60s lease-wait retry on store open during daemon-restart teardown overlap. Serve prompt guidance with support for "full" and "light" prompt surface presets (subject to a ratified 1825-token budget ceiling) as well as a "no_reduce" variant on reduce-less surfaces. Expose co-owned profile and feature epochs (PROFILE_EPOCH_CLAUDE_CODE_ANTHROPIC, TAGGER_FEATURE_EPOCH) during status checks to assert compatibility, folding profile-specific serializer epochs to coordinate hard transitions on format changes. Maintain a durable pass trace (mc_pass_trace table in mc-store) to audit session receive/complete/reject events separately from cache state. Use harness model codecs to translate between harness-specific JSON (OpenCode, Pi) and canonical CkWireMessage values, reject client-reported context limits below MIN_PLAUSIBLE_CONTEXT_LIMIT (1024) by falling back to 200,000, and use serializer profiles to gate per-consumer tail-reclaim capability (every shipping profile is now a full-array consumer — Claude Code joined them when the Thalamus peer retired its byte-splice at U0, so no profile suppresses tail mutations for a splice anymore) while absorbing covered system-role messages into the m0 baseline. It supports agent_drops.append for queuing tag drops, with support for server-side canonicalization of raw range/list drop strings and command-id idempotency checks, and enforces epoch-aware route channel tracking (via OpenedRoute / RouteHandle) to reject stale route executions under the wire v2 protocol. It orchestrates session.status, session.wrapup, and session.delete operations (utilizing structured status fields, machine-readable wrapup dispositions, and process-local per-session latches under a MAX_WRAPUP_REQUEST_BUDGET deadline, with session.delete atomically removing session-owned rows from SQLite tables), bounds inline and busy emergency historian follow-up waits to a shared 20s budget (TRANSFORM_HISTORIAN_FOLLOWUP_BUDGET), delivers oversize completed tool-arc components whole to the producer (crates/mc-module/src/historian_chunk.rs) when the head cap lands inside them (crates/mc-module/src/boundary.rs), holds demoted signed assistant native reasoning vectors (native_reasoning_keep_mids) until priced passes, excludes cache-preserving model variants from render identities, structures no-fire diagnostics inputs, tracks transform dispatch health metrics and heartbeat reporting (reporting typed state=stuck if the dispatch queue or heartbeat goes stale), manages LRU-bounded InFlight snapshot caching (MAX_IN_FLIGHT_SNAPSHOT_ENTRIES ceiling) to limit process-lifetime memory growth, pages tool_input_key_orders maps to avoid frame overruns, coordinates bootstrap state imports using StateImportCoordinator, persists consumed renderer shape classes in transition markers to scope split-coverage repairs, recombines reduced tool call/result shells into unified native tool parts after sidecar matching, serves module mural cue facades, composes host-fed mural artifacts into m0 baseline history (crates/mc-module/src/m0_compose.rs), evaluates allocator-oriented retained-size estimates for memory-budgeted holders (crates/mc-module/src/retained_size.rs), evaluates response-side Claude Code Channel-2 directives (PendingChannel2Directive), resolves trusted user-tier guidance overrides at the route boundary (crates/mc-module/src/config.rs), handles host-neutral window geometry wire structs (usable_soft, usable_hard, derivation), exports build provenance (ModuleManifest.provenance stamped from MC_BUILD_SHA, degrading non-canonical stamps to omission instead of boot panics under subc-protocol 0.17), and tracks per-pass transform output divergence and attribution across execution runs.

Transform pass mechanics

This is the heart of the system and the part most easily gotten wrong. A "transform pass" is one invocation of experimental.chat.messages.transform (src/hooks/magic-context/transform.ts), wrapped defensively in src/plugin/messages-transform.ts (transient SQLITE_BUSY → return messages unmodified so the prompt loop always proceeds). OpenCode fires it once per LLM round-trip (per step within a turn).

Pass lifecycle (in order)

  1. Resolve usage + scheduler decision (execute vs defer).
  2. Emergency overflow recovery if ≥95%.
  3. Compartment trigger check (off the in-memory args.messages tail — no opencode.db read steady-state); fire the historian async if eligible.
  4. Prepare compartment injection (decide m[0]/m[1] materialization).
  5. Tag messages; replay dropped-status, caveman, reasoning, placeholder, image strips.
  6. Compartment phase: inject the <session-history> (m[0]/m[1]) into message[0].
  7. Postprocess (transform-postprocess-phase.ts): the mutation gates — pending-op drain, heuristic cleanup, nudges, synthetic-todowrite, auto-search.

Pass taxonomy (every pass is exactly one)

  • SOFT+ (defer / cache_hit): nothing new. m[0] AND m[1] replay byte-identical; the entire system + m[0] + m[1] prefix stays cached. Only the conversation tail moves (where ctx_reduce/age drops land, themselves replayed deterministically). The steady state — most passes are this.
  • SOFT (cache-busting): m[1] re-renders (new compartments / memories / user-profile surface as deltas) while m[0] stays byte-identical. system + m[0] stays cached; the cache busts at the m[1] breakpoint. Driven by an execute pass, /ctx-flush, or a deferred-history drain.
  • HARD (m[0] fold): mustMaterialize fires → m[0] re-materializes, folding m[1] into the new decayed baseline and resetting m[1] to a placeholder. The whole prefix rebuilds — but "for free" because the provider cache key was already dead (see HARD triggers in "m[0]/m[1] cache layout"). Decay re-tiering happens ONLY on a HARD fold — a SOFT pass must never re-tier (that would change m[0] bytes).

The mutation gates (the part to get right)

Pending-op drain and heuristic cleanup are each gated by the same shape in transform-postprocess-phase.ts:

shouldApplyPendingOps / shouldRunHeuristics =
  (execute || materializationRequested || forceMaterialization || foldExecutedThisPass) // BUST clause: did a HARD fold land this pass?
  && (!compartmentRunning || emergencyBypassCompartmentGate)                            // VETO clause: is the historian mid-run?
  • BUST clause — only mutate (drop tools, run heuristics) on a pass that is already busting the prefix, so the mutation rides that one bust instead of causing its own. foldExecutedThisPass is true only after off-wire fold pre-execution reports that m[0] actually materialized; a mustMaterialize advisory by itself never opens mutation gates.
  • VETO clause — compartmentRunning — block mutation while the historian is summarizing the tail, so we don't change the bytes it's reading mid-run. Bypassed by emergencyBypassCompartmentGate.
  • emergencyBypassCompartmentGate bypasses the veto when forceMaterialization (≥85%) OR foldExecutedThisPass — i.e. a hard fold drains pending ops + runs heuristics even while the historian runs, because the prefix is busting regardless (see "drain into the known bust" invariant). This is safe per the disjoint-DB model below; both harness twins use the shared executed-fold predicate.

Load-bearing invariants (memorize these)

  1. A HARD bust means the prefix is already gone → drain EVERYTHING into it. Never "defer" a hard bust. This pass IS the fold; there is no later fold to wait for. Deferring the drain only produces a second, avoidable bust ~one turn later. (The compartmentRunning veto must therefore yield to a hard fold — the fold-exec bypass.)
  2. A defer (SOFT+) pass must replay byte-identical. Any first-application of a strip/drop on a defer pass changes tail bytes and busts the whole prefix after it. Watermark-gated strips (placeholders, images, stale-ctx_reduce) use a frozen-id replay pattern: detect-and-freeze the affected ids only on cache-busting passes, replay the frozen set on every pass. There is exactly ONE drop placeholder string, [dropped §N§], a pure function of tag id — never re-derive bytes from mutated content (that caused repeated cache catastrophes).
  3. Deferred work rides the next bust cycle; it never forces its own. Historian publishes, compaction-marker moves, and queued drops accumulate while m[1] replays frozen, and materialize together on the next genuine bust (execute / hard fold / flush). A historian publish does NOT bust the cache — between busts every pass is cache_hit.
  4. Automatic reclaim is ride-only, and every lane shares ONE bust permission. Age sweeps, heuristic cleanup, supersession and duplicate dedup never originate a bust: they land only on a pass that is already busting for another reason (a fold or refold, a published-history refresh into m[1], /ctx-flush, an agent ctx_reduce drop landing, or the ≥85% force band). A single per-pass permission decides whether the pass busts, and every mutation lane — reductions, m[1] refresh, heuristics, synthetic todo, sentinel first-application — consults that same permission; a veto that applies to one lane and not another is a defect (the 2026-09-07 ALF split bust: the age sweep bypassed the historian veto that held the m[1] refresh, so one threshold crossing became two priced busts). There is no mid-turn deferral: a tool loop is not a reason to hold an execute (the OpenCode detector never engaged and Pi's only measurable effect was withholding drops for hours on steered marathons), and Anthropic's incremental tool-loop caching makes a held mutation cost strictly more than one applied at first eligibility.

Disjoint-DB safety model

Mutating while the historian runs is safe because the two databases are disjoint on the read/write side:

  • The historian reads raw OpenCode messages from opencode.db (read-only) for its chunk.
  • Drops + heuristics mutate context.db (tags / pending_ops) and the in-memory outgoing wire only.
  • The historian's in-flight snapshot is validated by computeRawRangeFingerprint, which hashes raw content only (ids, part types, content lengths) — never tag/drop state — so a concurrent drop can't invalidate it.
  • Its post-publish queueDropsForCompartmentalizedMessages is idempotent against already-dropped tags.

m[0]/m[1] cache layout

The compacted history renders into TWO synthetic user-role message slots at the head, so the large stable prefix survives steady-state work. inject-compartments.ts (renderM0 / renderM1 / materializeM0 / mustMaterialize), mirrored in inject-compartments-pi.ts. Both slots prepend with synthetic: true parts so they don't count toward OpenCode's title-generation gate.

  • m[0] — cumulative baseline (frozen, like system[0]). Holds <project-docs> (root ARCHITECTURE.md + STRUCTURE.md), baseline <user-profile>, and the decay-rendered compartment history as of the last materialization. Does NOT change on routine turns.
  • m[1] — volatile delta. Holds everything added since the last m[0] materialization: new user-profile additions, new memories (via the maxMemoryId watermark), <memory-updates> supersede deltas, and the newest compartments at full tier. Renders a minimal placeholder when empty (never fully empty — Anthropic cache-breakpoint structure).

mustMaterialize (HARD fold) triggers — organized around the bust taxonomy so the trigger list and the m[0]/m[1] contract can never silently disagree:

  • Provider-side cache eviction (the cache is already dead, so folding is free): model/provider change (cachedM0ModelKey), system-prompt-hash change (cachedM0SystemHash), idle > TTL (cacheExpired, self-consuming via lastResponseTime > cachedM0MaterializedAt).
  • Genuine m[0] content change (baseline bytes differ): first render, cached_m1_missing, project_memory_epoch change (dashboard / external mutation), pending m[0] mutations (max_mutation_id — structural compartment delete/merge/recomp), upgrade-state change.
  • Deliberately NOT triggers (these are m[1] deltas — triggering would bust m[0] on routine background work and defeat the design): new compartment sequence, project_user_profile_version, maxMemoryId, project-docs-hash change (docs edits fold in on the next natural hard bust, never on their own), and tool-set-hash change (process-global, false positives).
  • Pressure backstop refold: on a cache-busting pass, if no natural HARD bust has arrived but m[1] has grown large — gated by the m[1]/m[0] size ratio (with a small-m[0] floor) OR an absolute m[1] token cap (~20% of history budget) OR a large memory-mutation count.
  • applyMarkersToState updates ALL state.cachedM0* fields post-materialize (guards against an infinite re-materialize loop). /ctx-flush is SOFT (drives m[1] refresh + heuristics, not an m[0] fold).

Memory mutations route through m[1], not the epoch. In-session ctx_memory mutations do NOT bump project_memory_epoch: additive writes surface via the maxMemoryId watermark; non-additive (update/archive/merge) record a memory_mutation_log row rendered as a <memory-updates> delta. Both reconcile into m[0] on the next natural hard bust. The epoch is bumped only by dashboard mutations and /ctx-session-upgrade migration (an external editor can't otherwise signal a running session).

Historian compartment flow (produce → store → render)

The long-history pipeline. Tiered compartments + deterministic decay renderer (replaced the v1 flat-compartment + LLM-compressor model).

  1. Trigger (compartment-trigger.ts): threshold-relative pressure (context_limit × execute_threshold × 5%, clamped 5k–50k), commit clusters, and TC-chunked unsummarized-tail size (≥ triggerBudget × 3), while protecting the live tail. Runs off the in-memory tail (zero opencode.db reads steady-state); hands the resolved boundary snapshot to the runner so the historian sees exactly what the fire decision saw. Drain reservations are governed by a session-scoped internal drain budget (reserveProtectedTailDrainTokens in storage-meta-persisted.ts) over rolling 10-minute clock-armed windows, logging explicit limiter spend and reset delays on skips. On the Rust/subc module leg, the trigger honors host-transported profile model chains (historian_model_chain), the skip discriminant (trigger_false, no_models, etc.) is written durably to HistorianDurableState.last_no_fire with structured raw-cause and TS-canonical taxonomy (HistorianNoFireCause, change-gated to avoid hot-path writes, and cleared on fire), and trigger_false details carry quantized measurements. The transform handler recognizes the mc-historian: session namespace as self-owned and returns an identity pass-through to prevent self-recursive loops.
  2. Produce (compartment-runner-incremental.ts): runs the historian subagent on the raw chunk above the last compartment boundary with a bounded prompt (no full state dump) — 4 rotating seed compartments + the last 6 persisted compartments + the project-memory block for fact dedup. Emits each compartment with 4 paraphrase tiers (p1 verbose → p4 anchor-only), an importance (decay-rate semantics), an episode_type, a <facts> block in the 5-category taxonomy, and an <events> block. On the Rust/subc side, the run uses an opt-in sampling temperature (omitted by default for reasoning model compatibility) and a generous max_output_tokens of 32k with a 600s await budget. Historian producer prompt assembly in both TypeScript and Rust carries content-language directives for output parity. Truncation is tracked via ProducerOutput.length_capped and validation rejections write last_failure with validation errors and length-cap hints. Producer session IDs (mc-historian:) include a bound-session hash to isolate concurrent subagent lineages and prevent TerminalRunMismatch collisions. During assembly, enforce a min_chunk_tokens substance floor to avoid spawning a producer on near-empty content, but bypass this floor when the session is in emergency (≥95% usage) or on verbatim-tail profiles where folding is the sole reclaim path (fold_is_only_reclaim is true).
  3. Parse + validate: parseCompartmentOutput + validateHistorianOutput (sparse-but-strictly-increasing ordinals for consumer legs, non-overlapping ranges, correct unprocessed_from). Live coverage validation absorbs mid-span system role ordinals safely without rejecting the chunk. Lenient tier extraction permits mismatched closing tags (e.g. <p1>...</p2>), terminating by any closing tier tag or next opener, with legacy migration v70 healing stranded compartments. Apply strict gap healing to absorb tool-only gaps while rejecting narrative-bearing gaps to trigger re-reads on repair attempts.
  4. Discard-last boundary healing: if the historian consumed to the chunk edge with weak lookahead, the last (lookahead-free) compartment is not persisted; the next run re-reads it at the head with full lookahead. Guarded by progress (k≥2) and emergency-disabled. Skips unanchored fact, observation, and primer promotion on the discarded tail to prevent double-storing when the range is re-processed.
  5. Store: publish transaction appends compartments with tier columns. Promotable facts promote to project memory (exact-dedup); user_observations stored only when dreamer.user_memories.enabled (privacy gate). Events → compartment_events. Compartment-chunk embeddings generated on publish (memory-gated). Side-channel writes (events, primers, observations) use a best-effort contract inside the publish transaction without blocking compartment progress on non-fatal failures. Publish defers a compaction-marker move (see Subsystems) and signals a deferred history refresh — it does NOT force a bust. When reattaching, the historian subscriptions are always re-drained from the start (persisted producer_cursor is killed to prevent dropping output). First published compartment after an empty bootstrap forces a HARD fold (since a SOFT delta is impossible without an existing boundary).
  6. Render (decay): decay-render.ts (shared OpenCode + Pi) picks one tier per compartment via decay-curve.ts: half-life H = H50·2^((I−50)/D)/max(p,0.10) (H50=24, D=25), log-cost tier boundaries [0.201,0.729,1.322,2.587], budget pressure p once per pass. Older / lower-importance / higher-pressure compartments demote oldest-first; past the archive boundary they render P4/self-close or drop. Self-tunes as the context window changes — no LLM call. Legacy (pre-v2) rows render P3 (if they carry a U: line) else P4.
  7. Recomp / upgrade: /ctx-recomp rebuilds compartment structure from raw history (emits NO facts — preserves curated memories). /ctx-session-upgrade runs full recomp + a once-per-project 9→5-category memory migration (active only, permanent untouched, bumps the epoch).
  8. Wrapup: /ctx-wrapup manually compacts older raw conversation history into compartments in token-capped chunks while keeping the newest N raw messages intact. Guided by resolveWrapupProtectedTailBoundary, the orchestrator runs in sequential loops under a session-specific wrapup lease, and skips unanchored fact, observation, and primer promotion on the final chunk. Under the Rust subc module, wrapup is driven via the session.wrapup operation (monitored by process-local per-session wrapup latches and bounded by MAX_WRAPUP_REQUEST_BUDGET), returning structured status fields (including coverage ordinals and row versions) and machine-readable wrapup dispositions (such as completed, nothing_to_compact, already_in_progress carrying round counts, or failed).

Protected-tail boundary

protected-tail-boundary.ts decides, per pass, which prefix of the raw tail is eligible for the historian and which suffix stays protected — from true-raw token sizes (not user-turn counts), so sparse-user-turn sessions can't deadlock the historian (#132). Boundary anchors at lastCompartmentEnd + 1; token target N capped at 0.40 × usable (ABS_CAP 96k); a live-prompt floor keeps it from crossing the newest meaningful user message below the derived force band. An emergency drain catch-up latch (usage-driven) bypasses the per-run token cap when pressure reaches that same band (≥85% at the default threshold, rising with threshold + 2 to 92% at the maximum threshold), draining continuously until usage falls back below the safe threshold. Open tool arcs (a tool invocation with no result in the window) only hold the boundary back when recent (≥ the size-walk start = the live window); a stale/interrupted open arc older than that is compactable — otherwise one dead running tool call at the eligible-head edge would freeze the historian indefinitely. Completed tool arcs (tool_use / tool_result pairs) are kept atomic: boundaries fence backward around completed tool arcs so an invocation and its result are never split across a compartment boundary or by terminal validation/discard-last. If the historian head cap lands inside the first completed tool component, the boundary logic (applyHeadCap in protected-tail-boundary.ts and crates/mc-module/src/boundary.rs) admits the leading completed component whole as an oversize atomic unit rather than fencing backward to an empty head, and the chunk reader (read-session-chunk.ts / crates/mc-module/src/historian_chunk.rs) and runner pre-flight deliver the entire component to the producer without clipping. Self-describing boundary diagnostics are attached to ProtectedTailBoundarySnapshot.diagnostics to explain no-op and wrapup decisions. The trigger/runner share a content-stable range fingerprint for cross-view staleness validation. For manual wrapup passes, resolveWrapupProtectedTailBoundary counts raw messages (including tools) rather than user turns to determine the keep watermark (default 20), protecting the newest tail segment while using the same tool-arc fencing and user-boundary snapping logic. On verbatim-tail profiles (where fold_is_only_reclaim is true, e.g. claude-code-anthropic), folding is the only reclaim path while the newest message is forwarded in full. Keep the newest message and its completed tool arc in the tail and out of the fold so that the live turn does not become a durable compartment boundary.

Memory, search & embeddings

  • Memories (memory/storage-memory.ts): project-scoped durable knowledge in the 5-category taxonomy (PROJECT_RULES / ARCHITECTURE / CONSTRAINTS / CONFIG_VALUES / NAMING), with FTS + vector side tables. ctx_memory exposes write/archive/update/merge/list; list is dreamer-only; primary agents may only mutate their own project's memories (workspace-shared categories aside). update accepts an optional category to recategorize a memory (src/tools/ctx-memory/tools.ts). Duplicate detection on update is scoped to the target (project, category, normalized_hash) inside a single BEGIN IMMEDIATE transaction, with an adapter-agnostic UNIQUE constraint failed fallback so bun:sqlite and node:sqlite report the same friendly duplicate error; the mutation log records the target category so <memory-updates> deltas render the recategorization on both the TypeScript and Rust module lanes. Writes are idempotent, and an embedding hash guard (saveEmbeddingIfHashMatches) ensures vectors are only saved if the memory content hasn't changed during the provider call.
  • Unified search (search.ts): one query embedding or text matching query dispatched across memories, raw message history (FTS via message-index.ts), indexed git commits, compartment-chunk embeddings, and session/smart notes. Hard-filters memories already visible in <session-history> and raw-message hits newer than the last compartment boundary (already in context). The raw-message index tracks failed incremental writes via a dirty floor (dirty_floor_ordinal), rewinding the async reconciler to cover gaps. Under Rust module search facades, queries return prose-matched hits over visible memories, durable compartment summaries, and notes, filtering current prompt-visible memory IDs and returning structured suppression diagnostics when hits are already visible in context.
  • Embeddings: vectors stored as plain SQLite BLOBs, scanned in-memory via Float32Array cosine (sqlite-vec rejected — bun:sqlite can't load extensions + write-amplification). Provider resolved per-project; a substitution guard rejects a served model that doesn't match the requested one. Compartment-chunk embedding is on-demand via /ctx-embed (auto-drains the active session once per process; resilient retry with circuit-break; chunk-window config folds into chunk identity so it doesn't invalidate memory/commit vectors). Candidate selection and shadow backfills enforce key and content-hash completeness across chunk transcript windows (src/features/magic-context/compartment-chunk-embedding.ts, src/features/magic-context/project-embedding-registry.ts), prioritizing missing windows over stale-hash rows, while persistent shadow backfill latches (src/features/magic-context/shadow-backfill-state.ts) deduplicate identical submissions for one hour to prevent resubmission loops when batches stall without candidate changes. Classified remote-provider and Synapse failure diagnostics (src/features/magic-context/memory/embedding-failure.ts, src/hooks/magic-context/format-embed-failure.ts) categorize substitution, HTTP, envelope, transport, empty-result, certification-refusal, and credential-required classes to surface actionable fixes (including Synapse recertification and fallback routing) in /ctx-embed summaries. When running tests, constructing network-capable embedding providers without explicit test factories fails closed (project-embedding-registry.ts). When using the local embedding provider, embedding.local_runtime ("auto" | "native" | "wasm", defaulting to "auto") selects WASM under Bun versions before 1.4.0 (avoiding Bun NAPI teardown crashes) and native under Node or Bun ≥1.4.0; if native loading fails (or optional Sharp loader fails), it falls back to lazy web evaluation (transformers-web-entry.ts) and onnxruntime-web (WASM); if both fail, local embeddings are permanently disabled for the process to prevent repeating import errors, and a warning is logged routing the user to the CLI doctor command. When the certified local synapse embedding provider is used (embedding.provider: "synapse"), the client communicates with the subc daemon using RPC endpoints (embed.batch and models.list). It supports model discovery, batching, polling, and daemon restart/retry mechanisms, falling back to a required fallback_provider (local, openai-compatible, or off) upon permanent initialization failure, and logging batch states to a synapse_batch_ledger table.
  • Workspaces (workspaces / workspace_members): a project belongs to at most one workspace; member sessions read the union of members' memories (repo-attributed), gated per-category by share_categories. A cached_m0_workspace_fingerprint (sorted identity+epoch+categories hash) detects membership/policy changes for a single hard fold on change. Workspace sharing is fail-closed (read visibility only); primary ownership gates ensure tools cannot mutate foreign workspace memories.

Dreamer

Background maintenance (V2: per-task cron scheduling). A process-wide 15-min timer evaluates each task's cron (dreamer/cron.ts) against task_schedule_state (per-(project,task) lastRunAt/next_due_at/schedule, reconciled from config each pass), runs due tasks through their activity gate (task-gates.ts), and serializes them by conflict-domain lease (task-registry.ts): the memory-mutating tasks (map-memories, verify, verify-broad, curate, render-mural, classify-memories, retrospective, promote-primers, refresh-primers) share one memory:<project> lease; others (maintain-docs, evaluate-smart-notes, review-user-memories) hold independent leases. Each lease uses a guarded write (runLeaseGuardedWrite via BEGIN IMMEDIATE) so read-then-write steps are atomic across processes. In embedded multi-session Pi hosts, Dreamer registration is shared across plugin instances and hands its timer to the most recently registered active session owner. Each task runs in its own ephemeral child session via task-executor.ts; resolve model per-task (override → dreamer → session-model last resort, historian-only). Gate child session retirement on prompt settlement (src/shared/child-session-teardown.ts): settled runs delete inline when privacy-sensitive or when subagents are not kept (keep_subagents), while unresolved prompts (timeouts or errors) are archived for immediate UI hiding and left to the shared age-gated sweep (src/features/magic-context/dreamer/retrospective-orphan-sweep.ts) to avoid racing detached OpenCode writers. Make manual runs (via /ctx-dream) wait up to 60 seconds (polling every 2 seconds) for a busy domain lease to free, while scheduled ticks exit immediately. lastRunAt advances only on success (so a failed run retries). Verifiable run progress tracks backlog probes per task on the registry, reports live task and count deltas at chunk boundaries over RPC/status surfaces, and surfaces backlog counts prior to manual /ctx-dream <task> runs. The memory-maintenance reader tasks (map-memories, verify, verify-broad) and classify-memories are non-agentic: the host renders one prompt, a locked read-only (or zero-tool) agent emits ONE XML manifest, and the host parses + applies the DB writes (with fail-closed parsing: a missing root tag strictly rejects the output instead of applying a truncated prefix). Dreamer run rows record structured failure facts (DreamRunFailureDetail with closed-vocabulary failure_class including provider_timeout, provider_error, empty_completion, no_models, child_aborted, parse_failed, unknown; last model and models tried, redacted 500-char provider errors, timeouts, and child session IDs) inside the existing tasks_json blob without requiring a migration, rendered in the dashboard Detail column and task banners with /ctx-dream printing failure details inline on OpenCode and Pi. Tasks:

  • map-memories — map each memory to its backing files (or a file-independent sentinel) so verify has a per-memory gate target; the locked read-only dreamer-memory-mapper agent reads code + path-seeds and emits a <map> manifest. Process directives in PROJECT_RULES and memories whose proposed paths fail validation are overridden to file-independent sentinels with mapping_origin='host_rejected_fallback' (migration v82 TS / v51 Rust) so mapping runs converge. Host-chunked, resumable; mostly a one-time backfill then a cheap trickle for new (unmapped) memories.
  • verify / verify-broad — the dreamer-memory-mapper agent re-checks mapped memories against code and emits a <verify> manifest (verified / update / archive), applied host-side via the cache-neutral mutation-log. Process directives in PROJECT_RULES are restricted to code-fact verification (memory-claim-safety.ts); host-enforced checks refuse directive update/archive verdicts and refuse content-loss updates (>50% text drop without consolidation="true"). Refusals log structured reason lines and count as non-failing skips on task progress. verify gates per-memory on each memory's own verified_at (a mapped file changed since that memory was checked); partial progress sticks, so a timed-out run banks what it checked. verify-broad ignores the change gate and re-checks the whole mapped pool in resumable cycles (persisting the open cycle timestamp in last_broad_run_at). File-independent + unmapped memories are excluded (curate + age decay own the file-independent ones). Provider outage completions (finish=stop assistant responses with zero reasoning and output tokens ≤32) are classified by DreamerProviderOutputFailureError (src/features/magic-context/dreamer/provider-output-failure.ts); consecutive identical provider-failure batches abort early so the scheduler hot-retries instead of recording incomplete progress. curate — whole-pool hygiene (consolidate / tighten / archive, agentic, gateless, weekly; rejects pseudo-tool completions, accepting completed tool-only runs with completed ctx_memory operations when terminal assistant text is omitted, reporting completed operation counts in progress). Cross-category merges are structurally rejected. verify-broad progress is reported in a dedicated progress field separate from errors.
  • classify-memories — score importance/scope/shareable as a zero-tool single-shot transform: the dreamer-classifier agent emits a <classify> manifest the host applies column-only via setMemoryClassification (cache-neutral). Stages on pool size (10/100): skip <10, full pool ≤100, new/changed (classified_at NULL) + stratified anchors >100. Shareability fail-closed via hasShareabilitySensitiveText.
  • retrospective — learn from user-friction: a cheap LLM gate over the project's new user messages (cross-session scan via dreamer/retrospective-raw-provider.ts, opencode.db read-only / Pi JSONL), then a ctx_search-only restricted child emits XML <learnings> that the host validates + routes (project memory / user-observation candidate) in dreamer/retrospective-learnings.ts. The scan is bounded per run (RETROSPECTIVE_MAX_SESSIONS_PER_RUN) and explicitly excludes hidden subagent sessions to prevent task prompts from triggering the friction gate.
  • Plus maintain-docs / promote-primers / refresh-primers / evaluate-smart-notes / review-user-memories / render-mural (config-gated). Background tasks never force a prompt-cache materialization — their writes ride the next natural bust. The dreamer.inject_docs configuration flag (default true) controls the injection of project documents (ARCHITECTURE.md and STRUCTURE.md) into the m[0] <project-docs> block. If set to false, the block is omitted and its hash stored as "". Standalone smart-note condition evaluation (evaluate-smart-notes) capability-gates against the fleet wake plane (wake.create via src/features/magic-context/smart-notes/wake-plane.ts), standing down only on affirmative catalog detection while failing open on daemon connection failures.

Other subsystems

  • Synthetic-todowrite: tool.execute.after captures todo state to last_todo_state (pure DB write). On a cache-busting pass, postprocess injects a synthetic tool_use/tool_result pair (call_id = mc_synthetic_todo_<sha256(state)[:16]}) into the latest assistant message, AFTER tagging so it's never dropped. Defer passes rebuild from the persisted state_json for byte-identity. To keep the prefix byte-stable across tail growth, a None-anchor synthetic todo pair is emitted immediately after m0/m1 and before the tail loop. On coverage folds where the synthetic-todo anchor is folded out of the tail, the pair's call_id and frozen bytes are kept while relocating its anchor_mid to the tail end (re-anchoring only rides coverage-advancing busts).
  • ctx_reduce nudges: Both channels use one persisted {U,T} instrument over the final rendered tail: T is eligible non-synthetic text/tool-I/O/file mass, U is the active, unprotected tagged subset; reasoning, signatures, dropped skeletons, and Channel 1's own reminder spans are excluded from both, while protected ctx_reduce exemplars (shared K=3 newest) remain T-only and excluded from U. Cache-busting passes refresh the baseline at the last byte-affecting postprocess point; defer passes add typed append and protection-boundary deltas or hold a generation-invalidated baseline. Agent-visible reminders count droppable tool outputs and reclaimable token mass rather than usage against a window size denominator. Channel 1 appends a replay-stable <system-reminder> to tool outputs, requires T ≥ 60k and U ≥ 25k, then bands on U/T at 0.20/0.40/0.60 (gentle/firm/urgent) with max(25k, 0.08 × T) re-fire cadence. Following a ctx_reduce call, Channel 1 enters compliance grace (recorded in session_meta.last_nudge_level), staying quiet until post-drop $U$ grows by max(25k, 0.08 × T) or upward band escalation occurs. Upward crossings render full copy once, while same-band cadence fires use calm sticky copy gated by a 5-real-user-turn floor between Channel 1 emissions. Reclaimable tool hints in Channel 1 skip coordination and control-plane tools (work, board, ask, task, todoread, todowrite, bash_status, bash_kill, and ctx_*), enforce an AGE_RECLAIM_MIN_TOKENS value floor (250 tokens), and order candidate tags by tool tier (T3 misc before T2 edit/search before T1 navigation) then by age. Channel 2 is the fourth band (U/T ≥ 0.75, U ≥ 50k, T ≥ 60k) and delivers a synthetic-user nudge after step-boundary revalidation (via steer with triggerTurn: true on Pi/OMP so in-flight and scheduled tools complete before injecting at the batch boundary); its lease CAS is token-bound and resets only after a coverage-advancing hard fold or measured U-collapse, with stuck claims reaped after ten minutes. Sidebar and /ctx-status read this same baseline, while Pi's footer reports honest prompt pressure against the scheduler's usable-context denominator without forward-pressure scaling factors. Both channels gate on ctx_reduce actually being in the session's tool allow-list. Tail hygiene last-writer signatures (tailHygieneStructuralSignature in tail-hygiene-walk.ts and packages/pi-plugin/src/tail-hygiene-walk-pi.ts) compute a structural size proxy via single-pass recursive traversal of the served message tree instead of copying and UTF-8 encoding every message twice per pass.
  • Smart-drops (opt-in): Gated by the smart_drops config flag. Operates on cache-busting transform passes to reclaim additional context by targeting obsolete tool outputs rather than just the oldest:
    • Spent control-plane outputs: Drops older todowrite (keeping the newest 1), protects the newest three ctx_reduce exemplars across every reclaim lane, drops older reducible arcs, removes zero-value meta tools (like bash_status/bash_kill), and removes dismissed ctx_note actions.
    • Superseded edits: For older edit/write calls to a file that has since been edited again, it compresses the tool call to edit_marker mode. This mode preserves the filePath parameter verbatim and a region-hint prefix of the diff (length 40), while replacing the bulky output with [dropped §N§], keeping the model aware of which files and regions it previously modified. Supersession enforces a newest-20 owner floor derived from persisted tag chronology (storage-tags.ts), withholding tags whose owner is in the newest 20 distinct message IDs to keep protection stable across provider projection contraction and re-expansion. Both types of drops deduplicate with age-based reclaims and freeze their drop mode (full, truncated, or edit_marker) in the tags.drop_mode column for deterministic replay on later passes. Truncated-mode tool shells replace all original argument keys with an inert marker { dropped: "[dropped §N§]" } across OpenCode, Pi, and Rust; tool dispatch verifies arguments via assertExecutableToolInput (src/hooks/magic-context/dropped-input-guard.ts / packages/pi-plugin/src/dropped-input-guard-pi.ts), throwing an actionable recovery error (ctx_expand original arguments) if a dropped placeholder is submitted. Two-pass age reclaim advances tool_reclaim_watermark only on an actual application opportunity, freezing during plain execute residency. Open tool arcs (pending/running tool calls without completed outputs or errored status) are excluded from reclaim, drop, truncate, or edit_marker selection via partHasCompletedResult (src/hooks/magic-context/tool-drop-target.ts), reclaiming closed arcs (completed string outputs or errored tool parts with status === "error") while clamping deep clones of message parts so OpenCode's live in-memory execution objects stay byte-identical.
  • Tiered emergency drop and reclaim episodes (derived force band: ≥85% at the default threshold, up to 92% at raised thresholds): target-headroom eviction down to fixedFloor + 0.30 × (ceiling − fixedFloor), tools oldest-first across tiers (T3 misc → T2 edit/search → T1 navigation), newest-20% recency reserve on T1/T2. floorTags (full active set, for floor accounting) vs tags (droppable candidates). Reclaim uses a continuous pressure-episode contract across OpenCode and Pi: last_emergency_input_sample acts as an episode latch admitting one non-empty emergency batch per force-pressure episode; zero-removal evaluations leave it armed. Fresh usage readings within the episode do not release the latch; dropping below the force band rearms it for future pressure, and independent provider-visible mutations rearm it so candidates ride already-priced busts. Heuristic cleanup, caveman compression, duplicate cleanup, and reasoning clearing adhere to the same episode discipline, admitting one originating application per continuous primary execute episode. Below 95%, the newest-20 dropped tool calls keep a [dropped §N§] skeleton (the tool_use survives, output replaced); older drops are fully removed. At provider-proven or estimated ≥95% pressure, TypeScript emergency selection yields both protected_tags and the T1/T2 recency reserve, fully removing selected completed arcs while retaining open arcs and the newest K=3 ctx_reduce exemplars; reasoning-adjacent tool arcs retain a paired skeleton (requiresToolArcSkeleton in heuristic-cleanup.ts and packages/pi-plugin/src/heuristic-cleanup-pi.ts) to prevent Anthropic from merging signed assistant turns. The target and non-empty-batch episode discipline do not change. At force pressure, a head cap landing inside the first completed arc admits the whole atomic component if it fits before the protected tail, rather than fencing back to an empty head.
  • Compaction markers: inject an OpenCode-compatible compaction boundary so filterCompacted stops at the historian's last compartment, shrinking the transform-input array. The marker move is deferred from historian publish into the next materializing pass (one bust covers both the <session-history> rebuild and the boundary advance); CAS-guarded, restart-safe.
  • Content stripping (strip-content.ts, caveman.ts, sentinel.ts): stateless strip functions + deterministic in-place sentinel replacement + persisted watermarks. Provider-aware: empty-content sentinels only stay empty for providers that accept them (modelAcceptsEmptyContent); others get a [dropped] placeholder (e.g. Copilot/Bedrock break tool adjacency on empty parts — #135). Merged-assistant reasoning neutralization tracks applied message IDs via merged_reasoning_stripped_ids in session_meta (with CoreState units twin in Rust), freezing the applied set on cache-busting passes and replaying on defer passes while keeping whitespace-only text parts sentinel-invisible, with frozen sentinel IDs surviving temporary transform absence across compaction marker advances until message.deleted. For Anthropic Fable 5.1 thinking-prefix binding mismatches (classified in overflow-detection.ts), the recovery arm strips bound reasoning from the targeted assistant (stripReasoningFromAssistantIds in strip-content.ts), recording the frozen ID (binding_mismatch:<id>) in merged_reasoning_stripped_ids for replay stability before clearing the armed target upon live turn completion. Trailing assistant blank decisions (TrailingBlankDecision) freeze while a message is newest (on every pass, capturing the last-live shape), becoming immutable once historical (findTrailingBlankDecisionCandidates / applyFrozenTrailingBlankDecisions), normalizing late-visible empty text blocks on non-newest assistant messages across race directions and cloning message part arrays before length-changing splices to avoid mutating live execution object graphs. Trailing blank classifications snapshot source shapes before sentinel insertion so keep decisions never manufacture absent suffixes; poisoned keeps demote to strip via CAS on cache-busting passes. The transform wrapper (src/plugin/messages-transform.ts) preserves user-terminated prompt tails (preserveUserTerminatedTail) when OpenCode concurrently appends a pending assistant shell mid-transform, re-anchoring the user message at the wire tail so the shell cannot turn into an illegal assistant prefill.
  • Message / git-commit indexes: Maintain an FTS5 raw-message index outside the search hot path (via async reconciliation + live message.updated events) and a HEAD-only non-merge git-commit corpus populated by the dream timer. Out-of-band orphan session sweeps (src/features/magic-context/message-index.ts) discover candidate sessions by unioning across all harness-scoped SESSION_SCOPED_TABLES against OpenCode's authoritative session table. Session deletions record pending cleanups in pending_session_cleanup (distinguishing Rust module deletions via :rust harness tags), retried on dream timer ticks (retryPendingSessionCleanups / retryPendingRustSessionCleanupsForProject); deleteSessionScopedRows protects session coordinates until module acknowledgement. For directories that are not git repositories or are empty (no commits yet), the sweep coordinator future-dates the last sweep time to park them on a 24-hour re-probe cooldown to avoid error log flooding.
  • Commit-detection utility: A unified detection helper (src/shared/commit-detection.ts) parses git commit hashes (7-12 hex chars) paired with commit-action verbs (commit, cherry-pick, merge, rebase) to detect commit boundaries consistently across the historian trigger, OpenCode note-nudge, and Pi note-nudge.
  • System-prompt injection (system-prompt-hash.ts): injects only the Magic Context guidance text + a frozen Today's date: line (per-session sticky, updated only on cache-busting passes). Guidance is concatenated into output.system[0] with a blank line separator rather than appended as a second array entry so wire serializers emit a single system role message compatible with strict chat templates (Qwen/vLLM/LiteLLM). Guidance is gated by ctx_reduce availability (resolved from the tools map of the session's first user message). While the verdict is provisional (before the first user message has been processed/persisted), the guidance block still renders using a fail-open default, but the computed system prompt hash is not written to the database to prevent cache-busting flips when the first user message freezes the verdict. Prompt surfaces resolve presets ("full" vs "light", using src/shared/prompt-surface.ts and src/tools/light-descriptions.ts for light guidance and tool descriptions under the ratified 1825-token budget ceiling). Adjunct blocks (<project-docs> / <user-profile>) are NOT here — they moved into m[0]/m[1] so the system prompt stays maximally cache-stable. Skipped entirely for OpenCode's internal title/summary/compaction agents and for hidden child sessions (detected by the magic-context- title prefix).
  • TUI ↔ server RPC: localhost server on an ephemeral port (published to session_meta); the TUI plugin reads all data via RPC (no direct SQLite, avoids lock contention). In Rust mode, status reads query the module directly and fail closed if module status is unavailable (avoiding stale context.db mirrors), with tagCountsAuthoritative: false distinguishing exact module tag totals from host active/dropped breakdowns. Status surfaces (/ctx-status, status dialogs, shared chat fallback) declare host-backend routing (Host backends → MODULE: ctx_memory, ctx_note; historian: module-side) in Rust mode via src/shared/rust-mode-status.ts.
  • Diagnostics & Redaction: The CLI doctor command, logger (src/shared/logger.ts, with 32 MiB bounded log rotation to one .1 predecessor, periodic stat checks every 64 flushes, and 0600 permissions), and RPC endpoints sanitize file paths and credentials (src/shared/redaction.ts) to prevent leaking secrets in logs/dumps. Diagnostic text and config value sanitization is unified across logger and CLI routines. Numeric redaction preserves non-secret scalars (like token counts or boolean flags) to keep diagnostic metrics readable while masking high-entropy strings. Dual-grammar (fleet structured prefix vs legacy bracketed format) and dual-path log parsing across doctor, diagnostics, issue bundles, and the desktop dashboard (packages/cli/src/lib/log-lines.ts, packages/dashboard/src-tauri/src/log_parser.rs), pinned to the subconscious golden fixture (packages/cli/src/lib/__fixtures__/log_format_golden.json), keying recent errors on log level when present, with OMP doctor reading the omp/ log path.
  • Assistant activity predicates: assistantAwaitingTools detects whether an OpenCode subagent still has a live tool-waiting run before Channel 2 delivery. The stronger notice guard also holds ignored messages while generation is unfinished or a real user prompt is unanswered. Both predicates distinguish real user input from synthetic user messages such as Channel 2 nudges.
  • Desktop slash-stripped command interception & notice holding: OpenCode Desktop strips the leading slash from registered plugin commands and forwards them as single-text-part prompts (ctx-status, ctx-wrapup, ctx-dream, etc.). Magic Context intercepts registered commands at the chat.message hook seam before persistence (src/hooks/magic-context/stripped-command.ts), validates arguments against the command registry union, executes through command.execute.before handlers (rendering TUI dialogs or ignored-message fallbacks), and throws an Effect-compatible 204 suppression sentinel to abort message persistence and the prompt loop before LLM dispatch. Passive status notices and Desktop startup notifications (conflict warnings, startup announcements, schema-fence warnings, and project-identity warnings) route through RPC toasts (sendStatusNotification in src/hooks/magic-context/send-session-notification.ts) rather than injecting noReply user-role chat rows. Explicit command fallback replies and fenced ignored-chat callers use guarded ignored-message delivery (sendIgnoredMessage), holding messages during in-flight generation or unanswered real prompts so plugin notices cannot become the latest user turn while OpenCode's runLoop evaluates its parent exit check, and consuming post-append rollback attempts even on HTTP 409 or missing row IDs to prevent notice storm replay loops.

Storage & migrations

storage-db.ts creates the schema and runs versioned migrations (migrations.ts, currently v1–v82). LATEST_SUPPORTED_VERSION is a schema fence — it MUST be bumped with every new migration (a unit test asserts it equals the highest migration), and a stale value makes the DB refuse to open after the migration applies. Open-time migrations guard against live-process schema corruption via enforceMigrationOnOpenGuard, refusing to migrate when live OpenCode or Pi processes hold the database (scoped conservatively across all live instances for the default shared context.db, and scoped strictly to same-data-directory PIDs for non-default paths). Current-schema migration checks no longer acquire BEGIN IMMEDIATE when no migrations are pending, avoiding contention behind writer instances under dual-instance boot. SQLite busy_timeout (5s) is configured before the first fence read on both backends (src/features/magic-context/storage-db.ts). Historian and wrapup lease releases in TypeScript and Pi are best-effort, catching SQLITE_BUSY errors on delete and letting orphaned rows expire on their TTL. For replay stability across compaction marker advances, stripped placeholder IDs are bounded (4096) and cleared on message.removed. Legacy open fences and column healing (ensureColumn() + healAllNullColumns(), defined in storage-schema-helpers.ts to prevent cycles between storage-db and migrations) backfill upgraded DBs before orphan-sweep index creation so pre-migration tables do not wedge on startup. Migration v71 rebuilds authority guard triggers to durable context_privilege_state state-table form, eliminating connection-local UDF dependencies (mc_privileged_writer) so non-registering SQLite connections execute guarded writes safely. Migration v78 adds the migration_pending recovery journal to track in-flight cross-harness session migrations (OpenCode → Pi/OMP) crash-safely without exposing raw session_id columns to clearSession(). Migration v81 persists Last Known Good transform snapshots in lkg_slots for crash-safe recovery across restarts. Migration v82 adds memory_verifications.mapping_origin to distinguish mapper independence from host-rejected fallback sentinels (mirrored in Rust store migration 51). In Rust mode, memory mirror projections and NULL-field repairs execute in atomic write transactions with SQL-level recency guards against host updated_at and classified_at to prevent stale module snapshots from rolling back newer host lifecycle or classification states, and authority prepare/mirror routines rebind stable source identities and prebind page tombstones so host memory IDs survive round-trips. New session-scoped tables must be added to clearSession(). A bulletproof MAGIC_CONTEXT_TEST_DATA_DIR guard keeps the test suite off the live DB (running bun test once migrated a live DB and fail-closed running binaries). SQLite binds must use SPREAD positional args, never the array form (bun:sqlite binds a lone array positionally; node:sqlite reads it as named params and throws). For branch forks in Pi, copy durable session state (compartments, tags, pending operations, and session metadata) to the new session via copySessionStateForClone() in src/features/magic-context/storage-clone.ts. Run this copy inside an immediate SQLite transaction, filtering and mapping message ordinals and tag composite keys to the copied branch entries, and clearing cached cache bytes to trigger fresh rematerialization. Slow SQLite write transactions (held for ≥1000ms) are logged post-commit with site and duration attribution (src/shared/write-transaction-timing.ts) across historian publish, recomp, lease-guarded writes, clear-session, marker-drain, and migration runs without affecting transaction success. Under Rust mc-store, cortexkit-store 0.2.0 logs a store-ahead migration outcome instead of a silent no-op, without refusing startup so older binary rollbacks remain safe.

Session modes

There are three effective runtime surfaces. Primary sessions and subagents use the normal context-management pipeline with different feature sets. Compaction-off mode keeps additive knowledge surfaces but removes Magic Context's context-window management. The primary reduce surface is additionally gated by the session's actual ctx_reduce tool availability.

Feature Primary sessions Subagents Compaction-off
Tag DB records ✓ ✓ Existing rows inert; no new rows
§N§ prefix injection + ctx_reduce tool ✓ when ctx_reduce is available ✓ when ctx_reduce is available ✗
Historian / compartments / decay / m[0]/m[1] ✓ ✗ Additive m[0]/m[1] only; no history
ctx_expand knowledge tool ✓ ✓ ✓
Channel 1 nudge ✓ when ctx_reduce is available ✓ when ctx_reduce is available ✗
Channel 2 nudge ✓ when ctx_reduce is available ✓ when ctx_reduce is available ✗
Synthetic-todowrite / auto-search ✓ ✗ Auto-search ✓; synthetic todo ✗
Heuristic tool drops at execute ✓ once/turn ✓ every execute pass ✗
85% force-materialize / 95% block ✓ ✗ (overflow path only) ✗; fail-closed blocking is inert
Caveman text compression opt-in ✗ ✗

Subagents run heuristic drops on every execute pass (no once-per-turn guard) because a long subagent run is effectively one parent turn and would otherwise starve; they have no provider-cache reuse to protect. In compaction-off mode, native compaction covers child sessions (verified against OpenCode v1.18.4), so subagents receive additive memory/docs injection but no Magic Context reclaim.

Compaction-off mode is boot-resolved from the user-level compaction.enabled setting and requires a restart. OpenCode's compaction.auto / compaction.prune are separate native settings in opencode.jsonc, not aliases for Magic Context's compaction.enabled in magic-context.jsonc. Disabling MC can expose history hidden solely by MC; the first turn after disabling may trigger one native compaction cycle on a long session. Marker cleanup is lazy per session. When turning the mode back on, /ctx-wrapup is the suggested catch-up path only when the historian is runnable.

Error handling

Fail closed when storage is unavailable: when the user enabled Magic Context, deterministic inoperability (schema fence, storage open/migration failure) keeps a minimal transform registered and throws a loud recovery error every primary pass instead of silently falling through to native compaction (fail_closed_blocking, default true; user-tier only). Blocking process diagnostics classify the actual process kind (OpenCode server, OpenCode instance (TUI/CLI), Pi, or generic process) with per-PID probe evidence (kind, start time, redacted cmdline) via normalizeFailClosedProcessKind and formatFailClosedBlockingProcesses in fail-closed-block.ts. Fail open in ordinary per-turn handlers (log and skip). Wrap the outer transform so transient SQLITE_BUSY/SQLITE_LOCKED never crash the prompt loop (#23). overflow-detection.ts parses provider context-overflow errors (Anthropic / OpenAI / Copilot) and persists the detected limit so later passes use the lower value; successful requests with input tokens exceeding the detected limit establish a model-scoped pressure floor and invalidate stale detected limits (clearDetectedContextLimit in storage-meta-persisted.ts), restoring catalog/default limits. For Anthropic Fable 5.1 models, it classifies HTTP 400 thinking-prefix binding mismatches (detectThinkingBindingMismatch) to arm per-session thinking recovery (armThinkingBindingRecovery in storage-meta-persisted.ts) and drop the LKG slot, converting account-enforced thinking wedges into a single-turn reasoning strip. Pi similarly recovers from Fable thinking-binding errors off message_end.errorMessage (packages/pi-plugin/src/provider-error-recovery-pi.ts) with a frozen entry-id strip and LKG slot drop. When a provider-proven emergency meets storage failure, OpenCode, Pi, and Rust mode all throw EmergencyFailClosedError (src/hooks/magic-context/emergency-fail-closed.ts) before any LKG or raw fallback can execute. Rust store failures surface in status and health as runtime_store_error even when trace persistence is busy. Pi supports Last-Known-Good (LKG) replay on transient SQLite failures (packages/pi-plugin/src/pi-lkg.ts), serving the previous pass's transformed bytes plus the raw tail while refusing on JSONL entry-id divergence. Block the request and abort the session fail-closed at ≥95% usage when the context limit is provider-proven and no fold materialized in the current pass (storing the origin in the emergency_recovery_origin column). Subagent model fallback (model-suggestion-retry.ts) iterates the chain on retryable failures; abort/timeout/context-overflow short-circuit. Hidden agents carry a steps/maxSteps cap and are aborted via session.abort on timeout so a weak local model can't loop forever (#154).

Tag identity

Each tags row is one taggable source-content unit (message, file, or tool). message/file tags key on (session_id, message_id) (synthetic content id). tool tags key on a COMPOSITE (session_id, callID, tool_owner_message_id) — because OpenCode reuses a callID counter per assistant turn, so the same read:32 recurs across turns; including the owning assistant message id gives each invocation its own row (migration v10). Owner derivation: invocation parts own themselves; result parts pop a FIFO of unpaired invocations; a result whose invocation was compacted away falls back to the nearest prior persisted owner. The same composite keying mirrors in the drop queue and heuristic cleanup so dropped keys match what the tagger persisted. Per-tag token counts (token_count / input_token_count / reasoning_token_count) are computed once on tag insert and summed for sidebar / boundary / nudge math (off the hot path).

On Pi, adapt fallback tags by mapping temporary message IDs to stable branch entry IDs when a pass succeeds. Re-probe negative fallback tag preflights after building the fingerprint map to safely catch tags committed by sibling processes. For token estimation cache safety, bypass stable-id cache equality for messages with custom prototypes, non-enumerable fields, or toJSON methods to ensure JSON serialization matches what the tokenizer reads.