Simplify memory core and protect snapshots and concurrent reads

.github/workflows/ci.yml

@@ -41,7 +41,8 @@ jobs:
       - run: python eval/run_tasks.py --verify-fixtures
       - run: python examples/quickstart.py
       - run: python examples/handoff_demo.py
-      - run: ruff check --select E9,F63,F7,F82 src tests eval scripts examples
+      - run: ruff check src tests eval scripts examples
+      - run: ruff format --check src
 
   mcp-versions:
     runs-on: ubuntu-latest

CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## 0.4.0rc2 — independent snapshots and a simpler core
+
+- Return detached entries from every public operation, including nested source,
+  metadata and history. Caller mutations cannot bypass revisions or re-embedding.
+- Serialize operations on each store object with a reentrant thread lock. Readers
+  cannot observe a partial write or a change that later rolls back.
+- Build startup context from one snapshot so another writer's correction cannot
+  mix a retired handoff with its replacement in the same response.
+- Count literal tokenizer markers as ordinary text, preserving recall, statistics
+  and rendered budgets for memories that quote those markers.
+- Separate record rules (`models.py`) and JSON persistence (`persistence.py`)
+  from memory operations. Preserve format 3, old IDs and migration support.
+- Replace `dedup_threshold` with explicit `deduplicate=False`; make the writing
+  agent keyword-only and document the Python compatibility changes.
+- Default new stores to offline hashing. Preserve existing backend configuration,
+  require semantic opt-in, and reject unknown configuration values.
+- Share display tags, avoid copying full histories just to count candidates,
+  and make evaluation output directories work on first use.
+- Add API ownership and concurrent-read regressions, an architecture walkthrough,
+  and project-local lint/format checks in CI.
+
+See [migration](docs/migration-v0.4.md) and [verification](docs/verification-rc2.md).
+The optional semantic-model and live-agent comparisons remain outside this
+candidate's verified scope.
+
 ## 0.4.0rc1 — release candidate
 
 - Generate UUID4 memory IDs and reject duplicate explicit IDs; retain legacy and caller-supplied identities.

CONTRIBUTING.md

@@ -10,6 +10,8 @@ AGENT_MEMORY_EMBEDDER=hashing python -m pytest -q -ra
 python eval/run_eval.py
 python eval/run_tasks.py --verify-fixtures
 python examples/handoff_demo.py
+ruff check src tests eval scripts examples
+ruff format --check src
 python -m build
 ```
 
@@ -21,4 +23,10 @@ Keep generated retrieval results current if their underlying behavior changes. K
 
 Core dependencies should stay small. MCP and model integrations remain optional. For persistence changes, cover multiple instances, malformed data, failed writes, legacy IDs and restart behavior. For tool changes, test the real MCP schema and stdio boundary.
 
+Keep client adapters thin, record rules in `models.py`, and file encoding in
+`persistence.py`. Store operations own revisions, retrieval and transactions.
+Return detached snapshots from public APIs; never expose live internal entries.
+Include same-object threaded readers as well as separate writer processes when
+changing transaction behavior. See [architecture](docs/architecture.md).
+
 Feature requests should include a concrete workflow and what currently fails. A benchmark showing where the current store stops working is more useful than a new backend in search of a workload.

README.md

@@ -8,22 +8,46 @@
 
 Keep architectural decisions, bug findings and session handoffs in one project store. Retrieve relevant notes within a text token budget. When a decision changes, retain its history and keep stale agents from overwriting the correction.
 
-For developers switching between coding agents or returning to a project after a break. It provides a Python library, CLI and local MCP server. No model API key is required; the lightweight installation uses offline lexical retrieval.
+Designed for developers switching between coding agents or returning to a project after a break, the engine provides a Python library, CLI and local MCP server. No model API key or database service is required. The default retriever matches words and character fragments; an optional local model adds semantic retrieval.
 
-**v0.4 release candidate:** new correction APIs and store format. Existing IDs remain intact. Read the [migration notes](docs/migration-v0.4.md) before upgrading a shared store.
+The workflow is simple: **recall context → work → save durable facts and a handoff**. Memories are explicitly written by you or your agent. This is not automatic learning from every conversation, and stored claims still need verification.
+
+**Status: v0.4.0rc2, a release candidate.** The instructions below install this checkout, not a published stable release. Existing format-3 stores need no conversion; Python callers should read the [migration notes](docs/migration-v0.4.md).
+
+Use this when several agents need selective retrieval and traceable corrections. If a few short, maintained Markdown files already solve your problem, keep them.
+
+## Architecture at a glance
+
+```mermaid
+flowchart LR
+    Clients[CLI / MCP / hooks] --> Context[Rendering and diagnostics]
+    Clients --> Store[MemoryStore]
+    Context --> Store
+    Store --> Records[Records and validation]
+    Store --> Embeddings[Embedding backend]
+    Store --> Disk[Atomic JSON persistence]
+```
+
+One Python package, one project store, one implementation of memory operations. All client adapters share the same core. The only required dependency is NumPy.
+
+The CLI, MCP server and hooks translate requests. Rendering builds the returned context. [`store.py`](src/agent_memory/store.py) coordinates memory operations, [`models.py`](src/agent_memory/models.py) defines records and validation, and [`persistence.py`](src/agent_memory/persistence.py) reads and atomically replaces the JSON file. These are modules, not separate services.
+
+The design prioritizes safe corrections and predictable behavior: stale edits are rejected when a revision is supplied, failed writes roll back, and returned Python records cannot mutate the store. The [architecture guide](docs/architecture.md) explains the write flow, locking and tradeoffs.
 
 ## Try it in five minutes
 
-Python 3.10+ and Git are required. From this checkout:
+You need Python 3.10+ and Git. On macOS/Linux:
 
 ```bash
+git clone https://github.com/Ninadnj/agent-memory-engine.git
+cd agent-memory-engine
 python -m venv .venv
 source .venv/bin/activate
 python -m pip install -e ".[mcp]"
 python examples/handoff_demo.py
 ```
 
-On Windows PowerShell, activate with `.venv\Scripts\Activate.ps1` instead.
+On Windows PowerShell, use `.venv\Scripts\Activate.ps1` instead of the `source` line. If your system uses `python3`, substitute it for `python`.
 
 The demo starts **two real MCP server processes** against a temporary store. Its output:
 
@@ -34,9 +58,11 @@ The demo starts **two real MCP server processes** against a temporary store. Its
 All checks passed. Two server processes; one temporary store; no model calls.
 ```
 
-This is a scripted protocol demonstration. It does not measure an LLM's ability to use memory.
+An expected stale-revision error may also appear in the logs: the demo deliberately attempts an outdated deletion and verifies that it is rejected. It uses a temporary store and makes no model calls. This tests the protocol, not an LLM's ability to use memory.
+
+### Use it in your own project
 
-To use your project's store:
+With the environment still active, **change into the Git project you want to remember**, then run:
 
 ```bash
 agent-memory --agent developer write "Bookings are stored in UTC; the UI converts to local time." --type decision
@@ -45,22 +71,34 @@ agent-memory doctor
 agent-memory recall "booking timezone UTC" --budget 200 --explain
 ```
 
-Inside a Git project, the default is `.agent_memory/store.json`. Outside one, it is `~/.agent_memory/store.json`. `--path` or `AGENT_MEMORY_PATH` selects an explicit store; use the same absolute path for clients that should share it. Keep memory files out of Git.
+The store is created on the first write. Inside a Git project, its default path is `.agent_memory/store.json` at the repository root. Outside Git, the default is `~/.agent_memory/store.json`.
+
+`--path` or `AGENT_MEMORY_PATH` selects an explicit store. Give every client the **same absolute path** when they should share memory. Add `.agent_memory/` to that project's `.gitignore`; memories can contain private project details.
 
 ## Correct a memory without losing the explanation
 
 ```python
 from agent_memory import HashingEmbedder, MemoryStore
 
 store = MemoryStore(".agent_memory/store.json", embedder=HashingEmbedder())
-old = store.write("Queue retries are limited to three attempts.", type="decision",
-                  source={"path": "queue/policy.py"}, agent="reviewer")
-new = store.supersede(old.id, "Queue retries are limited to five attempts.",
-                      expected_revision=old.revision, agent="implementer")
+old = store.write(
+    "Queue retries are limited to three attempts.",
+    type="decision",
+    source={"path": "queue/policy.py"},
+    agent="reviewer",
+)
+new = store.supersede(
+    old.id,
+    "Queue retries are limited to five attempts.",
+    expected_revision=old.revision,
+    agent="implementer",
+)
 assert store.get(old.id).superseded_by == new.id
 ```
 
-Use `update` for a correction to the same memory, `supersede` for a replacement decision, and `forget` for permanent deletion. MCP and CLI edits require the revision you read. Python accepts `expected_revision` for compatibility; **pass it when multiple callers can edit**. Updates retain the last 20 prior revisions. Forgetting removes the entry and its history from the current store.
+Use `update` to correct the same memory, `supersede` to replace it while retaining the old record, and `forget` to permanently delete it. MCP and CLI edits require the revision you read. In Python, `update` and `forget` allow the revision to be omitted; **always pass it when multiple callers can edit**. `supersede` requires it. The last 20 prior revisions are retained; forgetting removes the entry and its history from the current store.
+
+Python methods return independent snapshots. Editing their fields changes only that local copy; calling `save()` afterward does not persist those edits. Use `store.update(...)` or `store.supersede(...)`. An earlier snapshot keeps its original revision even after another caller updates the store.
 
 CLI equivalents, replacing `MEMORY_ID` and revision with values from `list`:
 
@@ -76,7 +114,7 @@ Sources can include a project-relative `path`, Git `commit`, `event`, or caller-
 
 ## Connect a coding agent
 
-See [client setup](docs/clients.md) for Claude Code, Codex CLI and Cursor. All use the same stdio command, `agent-memory-mcp`, and can share a store on the same machine.
+See [client setup](docs/clients.md) for Claude Code, Codex CLI and Cursor. Configure the absolute path to the installed `agent-memory-mcp` executable and the project store. The client launches the server over stdio; no HTTP service or port is needed. Agent use of the tools depends on the client and model—it is not guaranteed to happen automatically.
 
 | Task | MCP tool |
 | --- | --- |
@@ -91,33 +129,49 @@ See [client setup](docs/clients.md) for Claude Code, Codex CLI and Cursor. All u
 
 Optional Claude Code hooks inject fresh startup notes and record observed Git changes. Install with `agent-memory install-hooks`; enable per-prompt recall with `--with-prompt-recall`. Hooks are fallible observations and do not establish who authored a change.
 
-## What is guaranteed, and what is measured?
+## Correctness and limits
 
 - New IDs use `mem_` plus UUID4. Deletion cannot reset an ID counter. Unique explicit IDs and legacy `mem_0001` IDs remain usable; duplicate explicit IDs are rejected.
 - Mutations reload under an OS lock and replace the JSON file atomically. Failed writes roll back the local snapshot. This contract covers cooperating v0.4 processes on a local filesystem.
+- A per-store thread lock prevents concurrent calls from seeing partially applied changes. Returned records include independent copies of nested metadata and history.
+- Startup selects its handoff and relevant memories from one snapshot. A concurrent correction appears on the next call, rather than mixing old and replacement notes in one response.
 - Correction revisions reject stale edits. Superseded notes leave normal recall; old handoffs and worklogs leave startup context. Exact duplicates are skipped only within the same type and source; similar wording never silently merges a contradiction.
 - MCP/CLI context budgets include the returned text's labels, IDs and source references. `cl100k_base` is used when tiktoken is installed, otherwise counting is approximate. Client wrappers, tool schemas and full session usage are outside this budget. The Python store API budgets memory bodies only.
 
-The [retrieval benchmark](eval/results.md) is a small diagnostic: 14 memories and seven queries. Its paraphrase results expose the limits of lexical matching. It is not evidence of improved coding outcomes.
+The intended workload is **hundreds of memories per project**. Writes replace the whole JSON file; recall scores all stored vectors. There is no cloud sync, authorization layer or distributed-storage guarantee. Memory content may reach your coding agent's provider: treat it as fallible data, never executable instructions.
+
+## Verification and evidence
+
+[CI](https://github.com/Ninadnj/agent-memory-engine/actions/workflows/ci.yml) tests Linux, macOS and Windows, Python 3.10/3.12, both supported MCP SDK majors, packaging and clean installation. Tests cover revisions, concurrent access, rollback, legacy stores, token budgets and real MCP calls. See the [verification record](docs/verification-rc2.md) for results and scope.
 
-The [coding evaluation](docs/evaluation.md) adds **30 executable tasks across three synthetic projects**, with six calibration tasks and 24 test tasks. It compares no memory, curated Markdown and engine recall with the same external agent. A [bundled Codex CLI adapter](docs/evaluation.md#run-with-codex-cli) provides setup checks and a direct evaluation command. [Fixture validation](eval/fixture-validation.json) verifies all 30 graders. **The live-agent comparison is deferred for v0.4.0rc1; no live-agent performance result is published.** You can install, use and test the engine without running that comparison or supplying model credentials.
+- **Retrieval diagnostic:** [14 memories and seven queries](eval/results.md). Literal-query recall is 0.93; paraphrase recall is 0.43. This small benchmark exposes the offline retriever's limitations, not improved coding performance.
+- **Evaluation infrastructure:** [30 executable tasks](docs/evaluation.md) across three synthetic projects. All [task graders are validated](eval/fixture-validation.json); this is not an agent-success result.
+- **Still unverified in this candidate:** the optional semantic model, live coding-client sessions, and the live-agent comparison. No whole-session token-savings or coding-performance claim is made.
 
 ## Install options
 
+Run these from the cloned repository, with its environment active:
+
 | Installation | Contents |
 | --- | --- |
-| `pip install -e .` | Python library and CLI; NumPy only |
-| `pip install -e ".[mcp]"` | Adds the MCP server |
-| `pip install -e ".[real,mcp]"` | Adds optional sentence-transformers embeddings and tiktoken; first model use can download weights |
-| `pip install -e ".[dev]"` | Tests, MCP, exact tokenizer and build tools |
+| `python -m pip install -e .` | Python library and CLI; NumPy only |
+| `python -m pip install -e ".[mcp]"` | Adds the MCP server |
+| `python -m pip install -e ".[real,mcp]"` | Adds optional sentence-transformers embeddings and tiktoken; first model use can download weights |
+| `python -m pip install -e ".[dev]"` | Tests, MCP, exact tokenizer and build tools |
+
+New stores use offline hashing by default, even if semantic packages are installed. Set `AGENT_MEMORY_EMBEDDER=sentence-transformers` to opt into semantic retrieval, or pass an explicit embedder in Python. An unset value (or `auto`) reuses an existing v0.4 store's saved backend and model configuration. Unknown values fail with an actionable error. Configuration changes trigger re-embedding; immutable model revisions are recommended for reproducibility.
 
-Set `AGENT_MEMORY_EMBEDDER=hashing` for offline behavior. A new store otherwise prefers the semantic backend when installed. Existing v0.4 stores retain their embedding configuration unless explicitly overridden. Configuration changes trigger re-embedding; immutable model revisions are recommended for reproducibility.
+## Development and existing notes
 
-## Scope
+After installing the `dev` extra, run:
 
-This is a local JSON + NumPy store for small project memories. It has no network server, authorization layer, cloud sync or automatic truth checker. Memory content may reach your coding agent's provider. Treat it as fallible data and verify operational claims against code.
+```bash
+python -m pytest -q
+ruff check src tests eval scripts examples
+ruff format --check src
+```
 
-Keep maintained Markdown if a few short files already solve your problem. Use this engine when selective retrieval, cross-session handoffs and inspectable corrections justify the extra component. Vector database migrations and automatic LLM compaction are deferred until measurements justify them.
+See [Contributing](CONTRIBUTING.md) for the full checks. Keep changes small and backed by regression tests; a database migration or automatic summarizer needs a measured reason.
 
 The original Markdown convention remains in [scaffold/](scaffold/). Import existing notes with `python scripts/ingest_markdown.py path/to/notes --path .agent_memory/store.json`; oversized sections are split into bounded chunks.