From 2f87c71aba89c906fbf4dd8afe5e9ebfddc6f35d Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 21:53:31 +0000 Subject: [PATCH 01/10] =?UTF-8?q?curate:=20hone=20AGENTS.md=20back=20under?= =?UTF-8?q?=20its=206,000-token=20budget=20(6,747=20=E2=86=92=205,970)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The policy file has been over budget since the #79/#82/#80 rounds each added a bullet: 6,001 tokens on 2026-09-08, 6,747 at this branch point, against the 6,000 in .skills/context-budget. Loaded on every invocation, so the overrun is paid on every task. Class C throughout, plus one demotion. The rewrites drop words, never claims: prove-no-loss.sh --claims reports claims_dropped: 0, so every backticked identifier, issue reference and link target of the base file still exists, and each 81 warranted line was read against the destination that carries its reasoning (docs/CONVENTIONS.md for OOM/ACL/three-fates/dedupe keys, docs/STORAGE.md for the backend seam, docs/STREAMS.md for the compiled-in local default, docs/DEPLOYMENT.md for the guard and the co-core pin). The one move: Project Overview's command→fact diagram and the paragraph above it to docs/ARCHITECTURE.md, which is the founding-design doc and had 8.5k tokens of headroom. The rubric's own finding is that overviews in a policy file do not help an agent reach files faster; the identity line and the worker-first constraint stay. The one deletion: the four-line "Where the reasoning lives" mini-index, which listed four docs that ## Detail Docs already lists with fuller one-liners. Also this repo's first check-counts.sh pass: two rhetorical counts dropped to prose, nine judged in a new .skills/context-counts-ok. Seams 0/32 acked, two stale acknowledgements pruned on the tool's advice. 972 tests pass. Co-Authored-By: Claude Opus 5 (1M context) --- .skills/context-counts-ok | 25 ++++++ .skills/context-loss-ok | 97 +++++++++++++++++++- .skills/context-metrics.jsonl | 2 + .skills/context-seams-ok | 15 +++- .skills/context-token-counts | 12 +-- AGENTS.md | 164 +++++++++++++++------------------- docs/ARCHITECTURE.md | 11 +++ 7 files changed, 223 insertions(+), 103 deletions(-) create mode 100644 .skills/context-counts-ok diff --git a/.skills/context-counts-ok b/.skills/context-counts-ok new file mode 100644 index 0000000..5c4b7af --- /dev/null +++ b/.skills/context-counts-ok @@ -0,0 +1,25 @@ +# Counts judged legitimate — check-counts.sh --ack-file default. +# First pass, 2026-09-09: this check post-dates the repo's last curation, so +# every entry below is an existing count read for the first time. Two counts +# were dropped rather than warranted (form 2, the policy-file default): the +# "three ways a skew has already failed" now reads "the ways", and the logging +# stack's "one formatter, two installers" now reads "its formatter, its +# installers" — both numbers were rhetorical and neither is checkable here. +# +# --- enumerated: the counted things are named in the same clause or the list +# --- immediately under it, so the number is checkable by reading the section. +enumerated :: **two backends** (`local`, `gcs`) +enumerated :: Three that bite, each symptomless until it matters: +enumerated :: Two env files, and the boundary between them +enumerated :: **Two blob backends, one seam.** +enumerated :: the worker retries the two it meets at runtime +enumerated :: **Three stream kinds, three sets of rules.** +# The three contracts and their four documents are the four `docs/contracts/` +# rows in ## Detail Docs, one line down the same file. +enumerated :: **Three normative contracts bound the wire and the roadmap** +enumerated :: four documents under `docs/contracts/` +# +# --- stable: structural, and cannot change without the sentence being rewritten. +# XAUTOCLAIM's reply shape is Redis wire format, and the ≥7.0 floor beside it is +# what the sentence exists to state; a fourth element would rewrite both. +stable :: `XAUTOCLAIM`'s three-element reply diff --git a/.skills/context-loss-ok b/.skills/context-loss-ok index 474baaa..5d448d2 100644 --- a/.skills/context-loss-ok +++ b/.skills/context-loss-ok @@ -21,7 +21,6 @@ duplicate :: `/etc/systemd/system/`** — the installed unit is a copy, not a sy duplicate :: `daemon-reload` alone silently re-reads the old file and the mismatch has no duplicate :: symptom until a directive matters. **Refuses to start off `main`, or off duplicate :: unpushed commits** (#37, #48) — `scripts/check_main_checkout.sh`; -duplicate :: `REPLICATOR_ALLOW_ANY_CHECKOUT=1` overrides; a dev worker asks the same duplicate :: question at the writer (#52). retarget :: Full lifecycle table and the dev-server invocation: @@ -58,4 +57,98 @@ disproven :: loop (CannObserv/watcher#275). Every `BlobStore` call from a corout # --- The style bullets -> docs/STYLE.md ------------------------------------- # The block moved verbatim; this bold pseudo-heading became the `## General` # heading of its own section, which is a heading rename the move forced. -rename :: **General:** + +# --- The 2026-09-09 curation (AGENTS.md 6,747 -> 5,969 exact tokens). --- +# Two shapes only. Every `tighten` line is a class-C rewrite whose claims all survive: +# prove-no-loss.sh --claims reports claims_dropped: 0, so no backticked identifier, +# issue reference or link target of the base file was dropped -- only words. The +# reasoning each rewrite stopped repeating was read in its destination first: +# docs/CONVENTIONS.md (OOM, ACL, the three fates, the dedupe keys), docs/STORAGE.md +# (the backend seam, to_thread inside the shutdown budget), docs/STREAMS.md (the +# compiled-in `local` default), docs/DEPLOYMENT.md (guard verdicts, the co-core pin). +# Some entries are re-wrapped fragments rather than whole claims -- a rewrite moves +# line boundaries, and whole-line matching cannot see the text survived. +# +# The `duplicate` five are the 'Where the reasoning lives' mini-index, which listed +# four docs the ## Detail Docs index below it already lists with fuller one-liners. +tighten :: Python ≥3.12, uv, pytest, ruff. `ty` is available as a **non-gating** type checker (`uv run ty check`) — advisory only; no pre-commit or CI gate. +tighten :: Auth is ADC. Pin the current minor — `>=0.13.1,<0.14` — and raise the **patch** floor with every co-core feature the code starts depending on: the reasoning, and the three ways a skew has already failed, in [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). +tighten :: **The manifest is a source, not the artifact.** Nothing re-embeds it — no hook, no CI step — so editing a `description` there changes what the repo says and not what `codebase_context_search` returns. Re-run `codebase_context_index` in the same change, or the highest-authority answer an agent gets stays the stale one (#19 CR #17). +tighten :: the `BlobStore` protocol — **two backends now** (`local`, `gcs`), selected by +tighten :: `REPLICATOR_BLOB_BACKEND` and defaulting to `local` (#7); `src/api/` is the dev-only +tighten :: `/health` app; `src/core/` holds config, logging, and the consume path's failure +tighten :: vocabulary. `tests/` mirrors +tighten :: `src/`. Every module with the job it owns: +tighten :: **Single-VM setup.** Code committed to main is the deployed code. Replicator shares the VM with archiver, watcher, and notifier. +tighten :: The worker binds no port; 8041 is the dev API port and 8040 is reserved. **Redis +tighten :: is Archiver-operated** — Replicator is a client, never ships a broker, never +tighten :: claims ownership — and server **≥ 7.0** is Replicator-critical because +tighten :: `claim_stale` reads `XAUTOCLAIM`'s three-element reply. `scripts/check_redis_floor.sh` +tighten :: guards it as an `ExecStartPre`. Ports, neighbours, and the redis-py pin: +tighten :: Three things that bite, each of which has no symptom until it matters: +tighten :: Every deploy situation with its command, the guard's full verdict table, and the +tighten :: dev-server invocation: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). +tighten :: **The service must never load the repo `.env`.** Those PATs carry write access the +tighten :: worker has no use for, and a process whose job is fetching public URLs must not +tighten :: widen their blast radius. Anything the service needs goes in `/etc/replicator/.env`. +tighten :: Every variable, which file carries it, and the reasoning behind each default: +tighten :: Replicator is a **consumer** first. Follow the conventions co-core and the archiver producer established: +tighten :: `command_id:occurred_at`), so nothing an issuer waits on can collapse — storage +tighten :: identity and correlation identity are not interchangeable. `info_source_id` +tighten :: rides both and is **echoed, never read**, as is replicate's +tighten :: `info_item_rep_spec_id`: each `test_boundaries.py` carve-out is one field +tighten :: wide, and adding one edits the charter (#28, #29). +tighten :: `gs://`; `local` is the compiled-in default **by decision, not by schedule** — +tighten :: the deployment flipped on 2026-08-20 and the default did not. Every `BlobStore` +tighten :: call from a coroutine goes through `asyncio.to_thread` — which puts it in the +tighten :: unit's shutdown budget, not just the handler's. Per-backend retention, ceilings +tighten :: and failure classification: [docs/STORAGE.md](docs/STORAGE.md), which is the +tighten :: authority and worth reading before touching either store. +tighten :: - **Store, then publish — never the reverse.** A fact pointing at bytes that are +tighten :: not there is unrepairable by the consumer; stored bytes with no fact repair +tighten :: themselves on the reclaim. +tighten :: *before* it — as `XADD .dlq` then `XACK`, which is the form broker#2's +tighten :: ACL grants and is now observed rather than inferred (#79). Retry cadence is +tighten :: `REPLICATOR_CLAIM_MIN_IDLE_MS`; a failing *cycle* is `run_loop`'s problem, not +tighten :: the message's. +tighten :: ceiling, the consume path keeps reading, acking and reclaiming throughout, and +tighten :: nothing is dropped or dead-lettered. The third, `XGROUP CREATE … MKSTREAM`, is +tighten :: boot-only and does **not** retry: `ensure_group` re-raises anything but +tighten :: `BUSYGROUP`, so a first boot against a capped broker exits and systemd +tighten :: restarts. Verified against a scratch broker this repo spawns, never the shared +tighten :: one. Never answer an OOM with a client-level retry — a re-sent `XADD` the +tighten :: broker already applied publishes twice. +tighten :: `fetch_failed(handler_error)`. The cost is deliberate: a grant nobody fixes +tighten :: retries forever rather than reaching `.dlq`. +tighten :: handler — so losing them costs one TTL window of re-fetches and never +tighten :: correctness. Endorsed as bus state rather than a role blur, with broker#1's four +tighten :: answers and the ACL grant they imply, in +tighten :: - **The replicate loop writes for `gcs` (#29)** — create-if-absent, `blob_uri` +tighten :: never resolved as a path, writers keyed by alias, refusals before credentials, +tighten :: provider failures classified by HTTP status. Read +tighten :: [docs/CONVENTIONS.md](docs/CONVENTIONS.md) before touching that path. +tighten :: requires `--production` for the one combination the live worker consumes — a +tighten :: frame there is fetched for real. +tighten :: - **Three normative contracts bound the wire and the roadmap** — four documents, +tighten :: all under `docs/contracts/`, linked from sibling repos and indexed below. +tighten :: `tests/test_boundaries.py` enforces the charter in CI; change a +tighten :: charter and its tests together. +duplicate :: Where the reasoning lives: +duplicate :: - What each stream carries — [docs/STREAMS.md](docs/STREAMS.md) +duplicate :: - The rules common to all of them, and the `replicator:cmd:*` keyspace — [docs/CONVENTIONS.md](docs/CONVENTIONS.md) +duplicate :: - Blob paths, modes, and the retention sweep — [docs/STORAGE.md](docs/STORAGE.md) +duplicate :: - Fakeredis's divergences, the keys an integration run may touch, and why production `co-gcs-replication` is refused from every test — [docs/TESTING.md](docs/TESTING.md) +tighten :: The logging stack — one formatter, two installers, and the journald lines that +tighten :: are deliberately not JSON: [docs/STYLE.md](docs/STYLE.md). +tighten :: classes and functions, small focused functions, and tests mirroring source — +tighten :: each with its rationale and its ruff gate in [docs/STYLE.md](docs/STYLE.md). +tighten :: - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design and the module-by-module layout; read before changing one +tighten :: - [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule `AGENTS.md` states in a line +tighten :: - [docs/CONVENTIONS.md](docs/CONVENTIONS.md) — the co-core/Redis Streams rules common to every stream: idempotency, validation, DLQ, `claim_stale`; and the `replicator:cmd:*` keys, this service's only non-stream footprint (#80) +tighten :: - [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the three populations under `REPLICATOR_BLOB_DIR`, TTL and ceiling semantics +tighten :: - [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — VM topology, ports, the systemd unit's lifecycle, and the co-core pin +tighten :: - [docs/TESTING.md](docs/TESTING.md) — where fakeredis diverges from the live broker, which keys an integration run may create, and why production `co-gcs-replication` is unreachable from every test (#38) +tighten :: - [docs/SKILLS.md](docs/SKILLS.md) — vendored skill inventory, refresh procedure, and the doc-check sensitive-path list +tighten :: - [docs/SOCRATICODE.md](docs/SOCRATICODE.md) — the full tool table, the prefetch query, per-tool gotchas, and cross-repo search +tighten :: - [docs/contracts/content-fetch-issuer-reference.md](docs/contracts/content-fetch-issuer-reference.md) — its lookup half: the refusal list, the failure taxonomy, the silent conditions, trust posture +tighten :: - [docs/contracts/content-replicate-issuer-contract.md](docs/contracts/content-replicate-issuer-contract.md) — the replicate trust model and issuer obligations, settled ahead of the code (#34) diff --git a/.skills/context-metrics.jsonl b/.skills/context-metrics.jsonl index c7015a9..18a2299 100644 --- a/.skills/context-metrics.jsonl +++ b/.skills/context-metrics.jsonl @@ -13,3 +13,5 @@ {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 14840, "delta_days": 5, "delta_tokens": 0, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "1330c82", "seams": 0, "seams_acked": 28, "skill_commit": "d710691", "skill_version": "1.11", "tokens": 5985, "tokens_exact": true, "tokens_live": 104362, "top_section": "Bus Conventions", "top_section_share": 25, "ts": "2026-08-27"} {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 14876, "delta_days": 5, "delta_tokens": 14, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "214df18", "seams": 3, "seams_acked": 28, "skill_commit": "662de71", "skill_version": "1.12", "tokens": 5999, "tokens_exact": true, "tokens_live": 105677, "top_section": "Bus Conventions", "top_section_share": 25, "ts": "2026-09-01"} {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 14878, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": 7, "delta_tokens": 2, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": true, "repo": "replicator", "repo_commit": "c3f0e31", "seams": 3, "seams_acked": 28, "skill_commit": "f603abf", "skill_version": "1.16", "tokens": 6001, "tokens_exact": true, "tokens_live": 109141, "top_section": "Bus Conventions", "top_section_share": 25, "ts": "2026-09-08"} +{"actions": ["baseline:pre-curation"], "budget": 6000, "bytes": 16709, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": 1, "delta_tokens": 746, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 289, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": true, "repo": "replicator", "repo_commit": "8d63a77", "seams": null, "seams_acked": null, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 6747, "tokens_exact": true, "tokens_live": 116261, "top_section": "Bus Conventions", "top_section_share": 33, "ts": "2026-09-09"} +{"actions": ["tighten:Bus Conventions", "tighten:Detail Docs", "demote:Project Overview", "prune:Bus Conventions mini-index", "fix:count-precision"], "budget": 6000, "bytes": 14501, "claims_dropped": 0, "claims_warranted": 0, "counts": 0, "counts_acked": 9, "delta_days": 0, "delta_tokens": -777, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": "ok", "no_loss_warrants": 81, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "8d63a77", "seams": 0, "seams_acked": 32, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 5970, "tokens_exact": true, "tokens_live": 115702, "top_section": "Bus Conventions", "top_section_share": 30, "ts": "2026-09-09"} diff --git a/.skills/context-seams-ok b/.skills/context-seams-ok index 724f520..15795d9 100644 --- a/.skills/context-seams-ok +++ b/.skills/context-seams-ok @@ -61,7 +61,6 @@ docs/ARCHITECTURE.md :: What each stream carries, and the reasoning behind each docs/DEPLOYMENT.md :: The GCS test bucket — the opposite grant, on purpose # The GCS test bucket stayed in DEPLOYMENT.md's Infrastructure section through the # #67 split — this pointer still resolves to what it names. -docs/TESTING.md :: The bucket and the SA are provisioned (#50) # 2026-08-19, #67 CR #14. The new ENVIRONMENT.md preamble points back at # DEPLOYMENT.md for the lifecycle and the unit — neither moved in this split. docs/ENVIRONMENT.md :: and the unit itself are in [DEPLOYMENT.md](DEPLOYMENT.md). @@ -74,3 +73,17 @@ docs/SKILLS.md :: case, which is why the rule in `AGENTS.md` → `## Code Explor # Provenance heading from #69, unrelated to this run and the same shape as the # ten provenance-headings already judged legitimate above. docs/contracts/replicator-boundaries.md :: Asked and answered — the derived-sidecar question (#69, 2026-08-20) + +# --- judged during the 2026-09-09 curation ------------------------------ +# All three SKILLS.md hits point at the *arrangement* rather than at a block: +# `## Project Layout` and `## Detail Docs` both still exist and still hold the +# content each sentence describes — this run tightened their prose and moved the +# command→fact diagram to docs/ARCHITECTURE.md, neither of which is what these +# lines name. Verified by reading all three against the sections they point at. +docs/SKILLS.md :: top-level package is the change that leaves AGENTS.md's Project Layout stale, +docs/SKILLS.md :: AGENTS.md's Detail Docs list is a by-name inventory of `docs/*.md`, so a new +docs/SKILLS.md :: doc does need an AGENTS.md line the gate will not ask for. A blanket `docs/` +# Provenance heading from #79 in a doc this curation did not move anything into +# or out of; same shape as the eleven already judged legitimate above, and +# renaming it would break inbound anchors for no gain this run can claim. +docs/TESTING.md :: Testing against a broker at its cap (#79) diff --git a/.skills/context-token-counts b/.skills/context-token-counts index eaec6cc..62be866 100644 --- a/.skills/context-token-counts +++ b/.skills/context-token-counts @@ -3,18 +3,18 @@ # The offline estimators prefer a file's own last exact measurement to the # repo-wide figure in .skills/context-token-ratio, falling back to it for a # file never counted exactly or since drifted far from the size recorded here. -14878 6001 AGENTS.md -3753 1463 docs/ARCHITECTURE.md -10927 4251 docs/COMMANDS.md -13946 5164 docs/CONVENTIONS.md +14501 5970 AGENTS.md +4428 1681 docs/ARCHITECTURE.md +11213 4352 docs/COMMANDS.md +27269 9984 docs/CONVENTIONS.md 26650 9986 docs/DEPLOYMENT.md -21206 8005 docs/ENVIRONMENT.md +21446 8100 docs/ENVIRONMENT.md 22943 8612 docs/SKILLS.md 4977 1932 docs/SOCRATICODE.md 13176 4835 docs/STORAGE.md 24259 8747 docs/STREAMS.md 3759 1367 docs/STYLE.md -12139 4537 docs/TESTING.md +15909 5895 docs/TESTING.md 31972 11854 docs/contracts/content-fetch-issuer-contract.md 31801 11551 docs/contracts/content-fetch-issuer-reference.md 28660 10140 docs/contracts/content-replicate-issuer-contract.md diff --git a/AGENTS.md b/AGENTS.md index 086ffc6..d76b92e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,24 +6,18 @@ Be terse. Prefer fragments over full sentences. Skip filler and preamble. Sacrif Retrieval, fingerprinting, and temporary storage layer for the Cannabis Observer cluster. -Owns content fetching, temp storage, and fingerprinting — the network-bound, byte-handling work re-homed out of Watcher. Driven by **commands** on the Redis change bus; reports outcomes as **facts**. - -``` -content.fetch (command) → fetch → fingerprint → temp-store → blob_available (fact) - ↘ closed without bytes ───────────────→ fetch_failed (fact) -content.replicate (cmd) → guards → create-if-absent ────────→ replication_complete (fact) - ↘ refused / conflict ──────→ replication_failed (fact) -``` - **Worker-first.** Primary process = bus consumer (`src/worker/main.py`), not an HTTP API. The FastAPI app is a `/health` surface only, dev-only until a status endpoint is wanted. +The command → fact flow, and what each module owns: +[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). + ## Development Methodology TDD required. Red → Green → Refactor. No production code without a failing test first. ## Environment & Tooling -Python ≥3.12, uv, pytest, ruff. `ty` is available as a **non-gating** type checker (`uv run ty check`) — advisory only; no pre-commit or CI gate. +Python ≥3.12, uv, pytest, ruff. `ty` is a **non-gating** type checker (`uv run ty check`) — advisory, no pre-commit or CI gate. **co-core comes from the wheelhouse, not PyPI.** `co-core` / `co-core-aio` resolve from `./.wheelhouse`, mirrored from the private GCS index `gs://co-gcs-pypi` by `scripts/sync_wheelhouse.py` via `[tool.uv] find-links`. Run the sync **before** `uv sync` on a fresh clone or after a version bump: @@ -31,7 +25,7 @@ Python ≥3.12, uv, pytest, ruff. `ty` is available as a **non-gating** type che uv run --no-project --with 'google-cloud-storage>=2,<4' python scripts/sync_wheelhouse.py ``` -Auth is ADC. Pin the current minor — `>=0.13.1,<0.14` — and raise the **patch** floor with every co-core feature the code starts depending on: the reasoning, and the three ways a skew has already failed, in [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). +Auth is ADC. Pin the current minor — `>=0.13.1,<0.14` — and raise the **patch** floor with every co-core feature the code starts depending on; the ways a skew has already failed are in [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). ## Code Exploration Policy @@ -58,7 +52,7 @@ Full tool table, prefetch query, per-tool guidance, cross-repo search: ## Code Exploration Notes (repo-specific) -**The manifest is a source, not the artifact.** Nothing re-embeds it — no hook, no CI step — so editing a `description` there changes what the repo says and not what `codebase_context_search` returns. Re-run `codebase_context_index` in the same change, or the highest-authority answer an agent gets stays the stale one (#19 CR #17). +**The manifest is a source, not the artifact.** Nothing re-embeds it, so re-run `codebase_context_index` in the same change as a `description` edit — otherwise the highest-authority answer an agent gets stays the stale one (#19 CR #17). **`mcp-driver.mjs` lies twice — silently through the `skills/` symlink (skills#177), falsely from a worktree (skills#180).** Use `"$SOCRATICODE_DRIVER"`; disbelieve health findings outside the main checkout. Both in [docs/SKILLS.md](docs/SKILLS.md). @@ -67,23 +61,21 @@ Full tool table, prefetch query, per-tool guidance, cross-repo search: `src/worker/` is the primary process — the bus consumer, with the byte path, the failure fact, the retention sweep, the pacer, and the `content.fetch-policy` reader each behind their own seam. `src/storage/` is the content-addressed temp store behind -the `BlobStore` protocol — **two backends now** (`local`, `gcs`), selected by -`REPLICATOR_BLOB_BACKEND` and defaulting to `local` (#7); `src/api/` is the dev-only -`/health` app; `src/core/` holds config, logging, and the consume path's failure -vocabulary. `tests/` mirrors -`src/`. Every module with the job it owns: +the `BlobStore` protocol — **two backends** (`local`, `gcs`), selected by +`REPLICATOR_BLOB_BACKEND`, default `local` (#7). `src/api/` is the dev-only `/health` +app; `src/core/` holds config, logging, and the consume path's failure vocabulary; +`tests/` mirrors `src/`. Every module with the job it owns: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). ## Infrastructure -**Single-VM setup.** Code committed to main is the deployed code. Replicator shares the VM with archiver, watcher, and notifier. +**Single-VM setup.** Code committed to main is the deployed code; the VM is shared with archiver, watcher, and notifier. -The worker binds no port; 8041 is the dev API port and 8040 is reserved. **Redis -is Archiver-operated** — Replicator is a client, never ships a broker, never -claims ownership — and server **≥ 7.0** is Replicator-critical because -`claim_stale` reads `XAUTOCLAIM`'s three-element reply. `scripts/check_redis_floor.sh` -guards it as an `ExecStartPre`. Ports, neighbours, and the redis-py pin: -[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). +The worker binds no port; 8041 is the dev API port and 8040 is reserved. **Redis is +Archiver-operated** — Replicator is a client, never ships a broker — and server +**≥ 7.0** is Replicator-critical because `claim_stale` reads `XAUTOCLAIM`'s +three-element reply, guarded by `scripts/check_redis_floor.sh` as an `ExecStartPre`. +Ports, neighbours, and the redis-py pin: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). ## Server Lifecycle @@ -91,7 +83,7 @@ guards it as an `ExecStartPre`. Ports, neighbours, and the redis-py pin: && uv sync --frozen && sudo systemctl restart replicator` — `git push` instead of the pull when the merge happened here. -Three things that bite, each of which has no symptom until it matters: +Three that bite, each symptomless until it matters: - **The service refuses to start off `main`, or off unpushed commits** (#37, #48). `REPLICATOR_ALLOW_ANY_CHECKOUT=1` overrides; a dev worker asks the same question @@ -101,8 +93,8 @@ Three things that bite, each of which has no symptom until it matters: - **The daily skills-refresh hook commits without pushing**, which is one of the states the checkout guard refuses. Check `git status -sb` before a restart. -Every deploy situation with its command, the guard's full verdict table, and the -dev-server invocation: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). +Every deploy situation, the guard's verdict table, and the dev-server invocation: +[docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). ## Environment Variables @@ -112,39 +104,36 @@ convention: 1. **`/etc/replicator/.env`** — production config. **The only file `replicator.service` reads.** 2. **`.env`** (repo root, git-ignored) — dev/agent secrets, chiefly org-wide GitHub PATs. Never commit. -**The service must never load the repo `.env`.** Those PATs carry write access the -worker has no use for, and a process whose job is fetching public URLs must not -widen their blast radius. Anything the service needs goes in `/etc/replicator/.env`. +**The service must never load the repo `.env`.** Those PATs carry write access a +process whose job is fetching public URLs must not widen the blast radius of; +anything the service needs goes in `/etc/replicator/.env`. New settings take the `REPLICATOR_` prefix — the VM is shared, and the prefix is what keeps a sibling service from colliding. `BUILD_ID` is the one deliberate exception, stamped generically by the unit. For shell commands (dev only), load both — the snippet is under Common Commands. -Every variable, which file carries it, and the reasoning behind each default: +Every variable, which file carries it, and each default's reasoning: [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md). ## Bus Conventions -Replicator is a **consumer** first. Follow the conventions co-core and the archiver producer established: +Replicator is a **consumer** first — follow what co-core and the archiver producer established: - **At-least-once ⇒ idempotent.** The command dedupes on `command_id`; both facts are keyed per *occurrence* (`content_fingerprint:command_id`, - `command_id:occurred_at`), so nothing an issuer waits on can collapse — storage - identity and correlation identity are not interchangeable. `info_source_id` - rides both and is **echoed, never read**, as is replicate's - `info_item_rep_spec_id`: each `test_boundaries.py` carve-out is one field - wide, and adding one edits the charter (#28, #29). + `command_id:occurred_at`). `info_source_id` and replicate's + `info_item_rep_spec_id` are **echoed, never read** — each `test_boundaries.py` + carve-out is one field wide, and adding one edits the charter (#28, #29). - **Two blob backends, one seam.** `local` announces `file://` and `gcs` announces - `gs://`; `local` is the compiled-in default **by decision, not by schedule** — - the deployment flipped on 2026-08-20 and the default did not. Every `BlobStore` - call from a coroutine goes through `asyncio.to_thread` — which puts it in the - unit's shutdown budget, not just the handler's. Per-backend retention, ceilings - and failure classification: [docs/STORAGE.md](docs/STORAGE.md), which is the - authority and worth reading before touching either store. -- **Store, then publish — never the reverse.** A fact pointing at bytes that are - not there is unrepairable by the consumer; stored bytes with no fact repair - themselves on the reclaim. + `gs://`; `local` is the compiled-in default **by decision, not by schedule**. + Every `BlobStore` call from a coroutine goes through `asyncio.to_thread`, which + puts it in the unit's shutdown budget, not just the handler's. + [docs/STORAGE.md](docs/STORAGE.md) is the authority — read it before touching + either store. +- **Store, then publish — never the reverse.** A fact pointing at absent bytes is + unrepairable by the consumer; stored bytes with no fact repair themselves on the + reclaim. - **Read `count=1`.** `AsyncBusConsumer.read(count>1)` raises on a malformed frame *before* returning the well-formed ones, and `claim_stale` at `count>1` lets a poison entry jam recovery permanently. @@ -154,30 +143,25 @@ Replicator is a **consumer** first. Follow the conventions co-core and the archi `schema_version` first. - **Deterministic ⇒ DLQ; transient ⇒ retry; completed without bytes ⇒ fact + ack, no DLQ (#17).** `dead_letter` acks inside itself, so a fact is published - *before* it — as `XADD .dlq` then `XACK`, which is the form broker#2's - ACL grants and is now observed rather than inferred (#79). Retry cadence is - `REPLICATOR_CLAIM_MIN_IDLE_MS`; a failing *cycle* is `run_loop`'s problem, not - the message's. + *before* it — as `XADD .dlq` then `XACK`, the form broker#2's ACL grants + (#79). Retry cadence is `REPLICATOR_CLAIM_MIN_IDLE_MS`; a failing *cycle* is + `run_loop`'s problem, not the message's. - **A capped broker refuses only its `denyoom` commands, and the worker retries the two it meets at runtime (#79).** `XADD` and `SET` are refused and retried indefinitely — `OutOfMemoryError` is transient and exempt from the delivery - ceiling, the consume path keeps reading, acking and reclaiming throughout, and - nothing is dropped or dead-lettered. The third, `XGROUP CREATE … MKSTREAM`, is - boot-only and does **not** retry: `ensure_group` re-raises anything but - `BUSYGROUP`, so a first boot against a capped broker exits and systemd - restarts. Verified against a scratch broker this repo spawns, never the shared - one. Never answer an OOM with a client-level retry — a re-sent `XADD` the - broker already applied publishes twice. + ceiling — while the consume path reads, acks and reclaims throughout. The + third, `XGROUP CREATE … MKSTREAM`, is boot-only and does **not** retry: a first + boot against a capped broker exits and systemd restarts. Never answer an OOM + with a client-level retry, which republishes an `XADD` the broker already + applied. - **An ACL denial is transient too (#82).** `NoPermissionError` is the second `ResponseError` subclass in `_TRANSIENT_ERRORS`, so a grant broker#1's cutover got wrong backs off instead of closing valid commands with a terminal - `fetch_failed(handler_error)`. The cost is deliberate: a grant nobody fixes - retries forever rather than reaching `.dlq`. + `fetch_failed(handler_error)` — at the deliberate cost that a grant nobody + fixes retries forever. - **The `replicator:cmd:*` keys are the only non-stream keys on the broker (#80).** Per-stream dedupe — `SET NX EX` after a *completing* close, `EXISTS` before the - handler — so losing them costs one TTL window of re-fetches and never - correctness. Endorsed as bus state rather than a role blur, with broker#1's four - answers and the ACL grant they imply, in + handler — so losing them costs one TTL window of re-fetches, never correctness: [docs/CONVENTIONS.md](docs/CONVENTIONS.md#the-replicatorcmd-keys). - **Consumers must be idempotent; producers own the outbox.** Replicator has no DB — its durable record of intent is the consumer group's PEL. Do not add a @@ -187,24 +171,17 @@ Replicator is a **consumer** first. Follow the conventions co-core and the archi `content.blobs` and `content.artifacts` each carry both outcomes of their command; `content.fetch-policy` is read **groupless** — no group, no ack, no DLQ. -- **The replicate loop writes for `gcs` (#29)** — create-if-absent, `blob_uri` - never resolved as a path, writers keyed by alias, refusals before credentials, - provider failures classified by HTTP status. Read - [docs/CONVENTIONS.md](docs/CONVENTIONS.md) before touching that path. +- **The replicate loop writes for `gcs` (#29)** — create-if-absent, `blob_uri` never + resolved as a path, writers keyed by alias, refusals before credentials, provider + failures classified by HTTP status. Read + [docs/CONVENTIONS.md](docs/CONVENTIONS.md) first. - **Nothing but the seed script writes to `content.fetch`.** `scripts/seed_fetch.py` - requires `--production` for the one combination the live worker consumes — a - frame there is fetched for real. -- **Three normative contracts bound the wire and the roadmap** — four documents, - all under `docs/contracts/`, linked from sibling repos and indexed below. - `tests/test_boundaries.py` enforces the charter in CI; change a - charter and its tests together. - -Where the reasoning lives: - -- What each stream carries — [docs/STREAMS.md](docs/STREAMS.md) -- The rules common to all of them, and the `replicator:cmd:*` keyspace — [docs/CONVENTIONS.md](docs/CONVENTIONS.md) -- Blob paths, modes, and the retention sweep — [docs/STORAGE.md](docs/STORAGE.md) -- Fakeredis's divergences, the keys an integration run may touch, and why production `co-gcs-replication` is refused from every test — [docs/TESTING.md](docs/TESTING.md) + requires `--production` for the one combination the live worker consumes: a frame + there is fetched for real. +- **Three normative contracts bound the wire and the roadmap** — four documents + under `docs/contracts/`, linked from sibling repos and indexed below. + `tests/test_boundaries.py` enforces the charter in CI; change a charter and its + tests together. ## Common Commands @@ -259,31 +236,30 @@ logger = get_logger(__name__) ``` Entry points only: `configure_logging()` is called once inside the FastAPI `lifespan` or the worker's `run()`. Never in library modules. -The logging stack — one formatter, two installers, and the journald lines that -are deliberately not JSON: [docs/STYLE.md](docs/STYLE.md). - **Date & Time:** - All UTC - ISO 8601: `YYYY-MM-DDTHH:MM:SS.ffffffZ` (timestamps), `YYYY-MM-DD` (dates) **General:** imports at file top and explicit, docstrings on public modules, -classes and functions, small focused functions, and tests mirroring source — -each with its rationale and its ruff gate in [docs/STYLE.md](docs/STYLE.md). +classes and functions, small focused functions, and tests mirroring source. +Those with their rationale and ruff gate, plus the logging stack — its formatter, +its installers, and the journald lines deliberately not JSON: +[docs/STYLE.md](docs/STYLE.md). ## Detail Docs -- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design and the module-by-module layout; read before changing one -- [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule `AGENTS.md` states in a line -- [docs/CONVENTIONS.md](docs/CONVENTIONS.md) — the co-core/Redis Streams rules common to every stream: idempotency, validation, DLQ, `claim_stale`; and the `replicator:cmd:*` keys, this service's only non-stream footprint (#80) -- [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the three populations under `REPLICATOR_BLOB_DIR`, TTL and ceiling semantics -- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — VM topology, ports, the systemd unit's lifecycle, and the co-core pin +- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design, module by module; read before changing one +- [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule stated here in a line +- [docs/CONVENTIONS.md](docs/CONVENTIONS.md) — the rules common to every stream: idempotency, validation, DLQ, `claim_stale`; and the `replicator:cmd:*` keys (#80) +- [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the three populations under `REPLICATOR_BLOB_DIR`, TTL and ceilings +- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — VM topology, ports, the unit's lifecycle, the co-core pin - [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) — every variable either env file carries, and the boundary between them -- [docs/TESTING.md](docs/TESTING.md) — where fakeredis diverges from the live broker, which keys an integration run may create, and why production `co-gcs-replication` is unreachable from every test (#38) +- [docs/TESTING.md](docs/TESTING.md) — fakeredis's divergences, the keys an integration run may create, and why production `co-gcs-replication` is unreachable from every test (#38) - [docs/STYLE.md](docs/STYLE.md) — the logging stack: formatter, installers, and the non-JSON journald lines - [docs/COMMANDS.md](docs/COMMANDS.md) — every runnable command, with flags -- [docs/SKILLS.md](docs/SKILLS.md) — vendored skill inventory, refresh procedure, and the doc-check sensitive-path list -- [docs/SOCRATICODE.md](docs/SOCRATICODE.md) — the full tool table, the prefetch query, per-tool gotchas, and cross-repo search +- [docs/SKILLS.md](docs/SKILLS.md) — vendored skill inventory, refresh procedure, doc-check sensitive paths +- [docs/SOCRATICODE.md](docs/SOCRATICODE.md) — full tool table, prefetch query, per-tool gotchas, cross-repo search - [docs/contracts/content-fetch-issuer-contract.md](docs/contracts/content-fetch-issuer-contract.md) — what a `content.fetch` producer must do; normative, linked from issuer repos -- [docs/contracts/content-fetch-issuer-reference.md](docs/contracts/content-fetch-issuer-reference.md) — its lookup half: the refusal list, the failure taxonomy, the silent conditions, trust posture +- [docs/contracts/content-fetch-issuer-reference.md](docs/contracts/content-fetch-issuer-reference.md) — its lookup half: refusal list, failure taxonomy, silent conditions, trust posture - [docs/contracts/replicator-boundaries.md](docs/contracts/replicator-boundaries.md) — what Replicator may become; run its three tests against any proposed capability -- [docs/contracts/content-replicate-issuer-contract.md](docs/contracts/content-replicate-issuer-contract.md) — the replicate trust model and issuer obligations, settled ahead of the code (#34) +- [docs/contracts/content-replicate-issuer-contract.md](docs/contracts/content-replicate-issuer-contract.md) — the replicate trust model and issuer obligations (#34) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 07f80b0..3097d3e 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -8,6 +8,17 @@ bus stream carries is in [STREAMS.md](STREAMS.md), the rules common to all of them in [CONVENTIONS.md](CONVENTIONS.md), and blob-tree and retention rules in [STORAGE.md](STORAGE.md). +## What it owns, and what it emits + +Owns content fetching, temp storage, and fingerprinting — the network-bound, byte-handling work re-homed out of Watcher. Driven by **commands** on the Redis change bus; reports outcomes as **facts**. + +``` +content.fetch (command) → fetch → fingerprint → temp-store → blob_available (fact) + ↘ closed without bytes ───────────────→ fetch_failed (fact) +content.replicate (cmd) → guards → create-if-absent ────────→ replication_complete (fact) + ↘ refused / conflict ──────→ replication_failed (fact) +``` + ## Project Layout ``` From 087300b6b8db33df7cd8003c4f03081f374b6b93 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 21:53:37 +0000 Subject: [PATCH 02/10] curate: stamp the ledger row with the commit it describes Co-Authored-By: Claude Opus 5 (1M context) --- .skills/context-metrics.jsonl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.skills/context-metrics.jsonl b/.skills/context-metrics.jsonl index 18a2299..5c8b249 100644 --- a/.skills/context-metrics.jsonl +++ b/.skills/context-metrics.jsonl @@ -14,4 +14,4 @@ {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 14876, "delta_days": 5, "delta_tokens": 14, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "214df18", "seams": 3, "seams_acked": 28, "skill_commit": "662de71", "skill_version": "1.12", "tokens": 5999, "tokens_exact": true, "tokens_live": 105677, "top_section": "Bus Conventions", "top_section_share": 25, "ts": "2026-09-01"} {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 14878, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": 7, "delta_tokens": 2, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": true, "repo": "replicator", "repo_commit": "c3f0e31", "seams": 3, "seams_acked": 28, "skill_commit": "f603abf", "skill_version": "1.16", "tokens": 6001, "tokens_exact": true, "tokens_live": 109141, "top_section": "Bus Conventions", "top_section_share": 25, "ts": "2026-09-08"} {"actions": ["baseline:pre-curation"], "budget": 6000, "bytes": 16709, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": 1, "delta_tokens": 746, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 289, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": true, "repo": "replicator", "repo_commit": "8d63a77", "seams": null, "seams_acked": null, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 6747, "tokens_exact": true, "tokens_live": 116261, "top_section": "Bus Conventions", "top_section_share": 33, "ts": "2026-09-09"} -{"actions": ["tighten:Bus Conventions", "tighten:Detail Docs", "demote:Project Overview", "prune:Bus Conventions mini-index", "fix:count-precision"], "budget": 6000, "bytes": 14501, "claims_dropped": 0, "claims_warranted": 0, "counts": 0, "counts_acked": 9, "delta_days": 0, "delta_tokens": -777, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": "ok", "no_loss_warrants": 81, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "8d63a77", "seams": 0, "seams_acked": 32, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 5970, "tokens_exact": true, "tokens_live": 115702, "top_section": "Bus Conventions", "top_section_share": 30, "ts": "2026-09-09"} +{"actions": ["tighten:Bus Conventions", "tighten:Detail Docs", "demote:Project Overview", "prune:Bus Conventions mini-index", "fix:count-precision"], "budget": 6000, "bytes": 14501, "claims_dropped": 0, "claims_warranted": 0, "counts": 0, "counts_acked": 9, "delta_days": 0, "delta_tokens": -777, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": "ok", "no_loss_warrants": 81, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "2f87c71", "seams": 0, "seams_acked": 32, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 5970, "tokens_exact": true, "tokens_live": 115702, "top_section": "Bus Conventions", "top_section_share": 30, "ts": "2026-09-09"} From 4f718f556265d825f0a16f3209109d6561308339 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:00:49 +0000 Subject: [PATCH 03/10] =?UTF-8?q?curate:=20CR=2013=20=E2=80=94=20restore?= =?UTF-8?q?=20"never=20the=20shared=20one"=20to=20the=20capped-broker=20bu?= =?UTF-8?q?llet?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The one line in the policy file that says not to set maxmemory on a broker three services share. Losing it saved eight tokens and left an agent reading this bullet with no reason not to reproduce it against co-broker. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d76b92e..e80ea4c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -151,9 +151,9 @@ Replicator is a **consumer** first — follow what co-core and the archiver prod indefinitely — `OutOfMemoryError` is transient and exempt from the delivery ceiling — while the consume path reads, acks and reclaims throughout. The third, `XGROUP CREATE … MKSTREAM`, is boot-only and does **not** retry: a first - boot against a capped broker exits and systemd restarts. Never answer an OOM - with a client-level retry, which republishes an `XADD` the broker already - applied. + boot against a capped broker exits and systemd restarts. Verified against a + broker the tests spawn, **never the shared one**. Never answer an OOM with a + client-level retry, which republishes an `XADD` the broker already applied. - **An ACL denial is transient too (#82).** `NoPermissionError` is the second `ResponseError` subclass in `_TRANSIENT_ERRORS`, so a grant broker#1's cutover got wrong backs off instead of closing valid commands with a terminal From 90466894dd281021fcb8fc7f83ae337ac6327d95 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:00:58 +0000 Subject: [PATCH 04/10] =?UTF-8?q?curate:=20CR=2014=20=E2=80=94=20unfuse=20?= =?UTF-8?q?the=20.env=20boundary=20paragraph?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two claims, not one: the worker has no use for that access, and a fetcher of public URLs must not widen its blast radius. The tightened version dropped the first and stranded the second's preposition across a relative clause, in the paragraph that states a security boundary. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e80ea4c..09417fa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -104,9 +104,9 @@ convention: 1. **`/etc/replicator/.env`** — production config. **The only file `replicator.service` reads.** 2. **`.env`** (repo root, git-ignored) — dev/agent secrets, chiefly org-wide GitHub PATs. Never commit. -**The service must never load the repo `.env`.** Those PATs carry write access a -process whose job is fetching public URLs must not widen the blast radius of; -anything the service needs goes in `/etc/replicator/.env`. +**The service must never load the repo `.env`.** Those PATs carry write access the +worker has no use for, and a fetcher of public URLs must not widen their blast +radius. Anything the service needs goes in `/etc/replicator/.env`. New settings take the `REPLICATOR_` prefix — the VM is shared, and the prefix is what keeps a sibling service from colliding. `BUILD_ID` is the one deliberate From c8b1738dfa4840ecaa7832d04e9ab0f0095b1a15 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:01:14 +0000 Subject: [PATCH 05/10] =?UTF-8?q?curate:=20CR=2015=20=E2=80=94=20"here"=20?= =?UTF-8?q?in=20the=20STREAMS.md=20index=20row=20resolved=20to=20the=20wro?= =?UTF-8?q?ng=20file?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The row's subject is docs/STREAMS.md, so "one bullet per rule stated here" reads as a rule stated in STREAMS.md — and STREAMS.md's own opening line says `AGENTS.md`. Two tokens to stop the index contradicting the doc it indexes. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index 09417fa..fe2c6a1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -249,7 +249,7 @@ its installers, and the journald lines deliberately not JSON: ## Detail Docs - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design, module by module; read before changing one -- [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule stated here in a line +- [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule `AGENTS.md` states in a line - [docs/CONVENTIONS.md](docs/CONVENTIONS.md) — the rules common to every stream: idempotency, validation, DLQ, `claim_stale`; and the `replicator:cmd:*` keys (#80) - [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the three populations under `REPLICATOR_BLOB_DIR`, TTL and ceilings - [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — VM topology, ports, the unit's lifecycle, the co-core pin From 50b07f690219932c2437083b820daaa13938d2f8 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:01:14 +0000 Subject: [PATCH 06/10] =?UTF-8?q?curate:=20CR=2016=20=E2=80=94=20put=20the?= =?UTF-8?q?=20logging=20pointer=20back=20where=20logging=20is=20read?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Merging it into **General:** left the Logging block with no pointer and opened the merged sentence with a dangling "Those". The pointer belongs beside the two lines an agent reads when adding a logger; General keeps its own. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index fe2c6a1..eacc227 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -234,17 +234,15 @@ from src.core.logging import get_logger logger = get_logger(__name__) ``` -Entry points only: `configure_logging()` is called once inside the FastAPI `lifespan` or the worker's `run()`. Never in library modules. +Entry points only: `configure_logging()` is called once inside the FastAPI `lifespan` or the worker's `run()`. Never in library modules. The stack itself — formatter, installers, the journald lines deliberately not JSON — is in [docs/STYLE.md](docs/STYLE.md). **Date & Time:** - All UTC - ISO 8601: `YYYY-MM-DDTHH:MM:SS.ffffffZ` (timestamps), `YYYY-MM-DD` (dates) **General:** imports at file top and explicit, docstrings on public modules, -classes and functions, small focused functions, and tests mirroring source. -Those with their rationale and ruff gate, plus the logging stack — its formatter, -its installers, and the journald lines deliberately not JSON: -[docs/STYLE.md](docs/STYLE.md). +classes and functions, small focused functions, and tests mirroring source — each +with its rationale and ruff gate in [docs/STYLE.md](docs/STYLE.md). ## Detail Docs From f7dbd74128a8f8c1c26e440bf2d959d58d599d92 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:01:28 +0000 Subject: [PATCH 07/10] =?UTF-8?q?curate:=20CR=2017=20=E2=80=94=20the=20ind?= =?UTF-8?q?ex=20row=20did=20not=20describe=20what=20the=20demotion=20put?= =?UTF-8?q?=20there?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The command→fact flow moved into docs/ARCHITECTURE.md and the Detail Docs row still said "founding design, module by module". The index is how an agent reaches a doc, so a relocation the index does not name is harder to find than it was inline — which makes the demotion a loss rather than a move. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index eacc227..6dda796 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -246,7 +246,7 @@ with its rationale and ruff gate in [docs/STYLE.md](docs/STYLE.md). ## Detail Docs -- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design, module by module; read before changing one +- [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design, the command → fact flow, module by module; read before changing one - [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule `AGENTS.md` states in a line - [docs/CONVENTIONS.md](docs/CONVENTIONS.md) — the rules common to every stream: idempotency, validation, DLQ, `claim_stale`; and the `replicator:cmd:*` keys (#80) - [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the three populations under `REPLICATOR_BLOB_DIR`, TTL and ceilings From a19d77e792c6cf3dcfb606cf31a716b1539df008 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:01:28 +0000 Subject: [PATCH 08/10] =?UTF-8?q?curate:=20CR=2019,=2020=20=E2=80=94=20a?= =?UTF-8?q?=20contrast=20with=20no=20referent,=20and=20a=20doubled=20point?= =?UTF-8?q?er?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "by decision, not by schedule" kept the contrast and lost the evidence for it (the 2026-08-20 deployment flip), leaving a reader contrasting against a schedule they cannot see; "stays the compiled-in default deliberately (#7)" is the same claim, self-contained, and shorter. And the new overview pointer claimed the module map that Project Layout's pointer four sections down already claims — the shape the deleted mini-index was deleted for. Both pay for CR 13-17. Co-Authored-By: Claude Opus 5 (1M context) --- AGENTS.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6dda796..3a8b632 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,8 +8,7 @@ Retrieval, fingerprinting, and temporary storage layer for the Cannabis Observer **Worker-first.** Primary process = bus consumer (`src/worker/main.py`), not an HTTP API. The FastAPI app is a `/health` surface only, dev-only until a status endpoint is wanted. -The command → fact flow, and what each module owns: -[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). +The command → fact flow: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). ## Development Methodology @@ -126,7 +125,7 @@ Replicator is a **consumer** first — follow what co-core and the archiver prod `info_item_rep_spec_id` are **echoed, never read** — each `test_boundaries.py` carve-out is one field wide, and adding one edits the charter (#28, #29). - **Two blob backends, one seam.** `local` announces `file://` and `gcs` announces - `gs://`; `local` is the compiled-in default **by decision, not by schedule**. + `gs://`; `local` stays the compiled-in default deliberately (#7). Every `BlobStore` call from a coroutine goes through `asyncio.to_thread`, which puts it in the unit's shutdown budget, not just the handler's. [docs/STORAGE.md](docs/STORAGE.md) is the authority — read it before touching From 3aaa8cb1a2719057d08a2877b5713b88046684ac Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:04:05 +0000 Subject: [PATCH 09/10] =?UTF-8?q?curate:=20CR=2014,=2016,=2018=20follow-up?= =?UTF-8?q?=20=E2=80=94=20restore=20two=20base=20lines,=20pay=20for=20the?= =?UTF-8?q?=20additions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CR 14's rewrite and CR 16's appended pointer each replaced a line the base file had verbatim, which prove-no-loss reported as a fresh loss and which two warrants then had to cover. Restoring both is cheaper than warranting them: the .env paragraph is now byte-identical to base, and the logging pointer is its own line. The additions from CR 13-17 are paid for out of the Detail Docs rows and the manifest note, so the file ships at 5,988 exact against the 6,000 budget rather than the 6,017 the fixes alone would have cost. Co-Authored-By: Claude Opus 5 (1M context) --- .skills/context-loss-ok | 4 ---- .skills/context-token-counts | 2 +- AGENTS.md | 13 +++++++------ 3 files changed, 8 insertions(+), 11 deletions(-) diff --git a/.skills/context-loss-ok b/.skills/context-loss-ok index 5d448d2..7028143 100644 --- a/.skills/context-loss-ok +++ b/.skills/context-loss-ok @@ -88,9 +88,6 @@ tighten :: guards it as an `ExecStartPre`. Ports, neighbours, and the redis-py p tighten :: Three things that bite, each of which has no symptom until it matters: tighten :: Every deploy situation with its command, the guard's full verdict table, and the tighten :: dev-server invocation: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). -tighten :: **The service must never load the repo `.env`.** Those PATs carry write access the -tighten :: worker has no use for, and a process whose job is fetching public URLs must not -tighten :: widen their blast radius. Anything the service needs goes in `/etc/replicator/.env`. tighten :: Every variable, which file carries it, and the reasoning behind each default: tighten :: Replicator is a **consumer** first. Follow the conventions co-core and the archiver producer established: tighten :: `command_id:occurred_at`), so nothing an issuer waits on can collapse — storage @@ -143,7 +140,6 @@ tighten :: are deliberately not JSON: [docs/STYLE.md](docs/STYLE.md). tighten :: classes and functions, small focused functions, and tests mirroring source — tighten :: each with its rationale and its ruff gate in [docs/STYLE.md](docs/STYLE.md). tighten :: - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design and the module-by-module layout; read before changing one -tighten :: - [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule `AGENTS.md` states in a line tighten :: - [docs/CONVENTIONS.md](docs/CONVENTIONS.md) — the co-core/Redis Streams rules common to every stream: idempotency, validation, DLQ, `claim_stale`; and the `replicator:cmd:*` keys, this service's only non-stream footprint (#80) tighten :: - [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the three populations under `REPLICATOR_BLOB_DIR`, TTL and ceiling semantics tighten :: - [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — VM topology, ports, the systemd unit's lifecycle, and the co-core pin diff --git a/.skills/context-token-counts b/.skills/context-token-counts index 62be866..fb87157 100644 --- a/.skills/context-token-counts +++ b/.skills/context-token-counts @@ -3,7 +3,7 @@ # The offline estimators prefer a file's own last exact measurement to the # repo-wide figure in .skills/context-token-ratio, falling back to it for a # file never counted exactly or since drifted far from the size recorded here. -14501 5970 AGENTS.md +14508 5988 AGENTS.md 4428 1681 docs/ARCHITECTURE.md 11213 4352 docs/COMMANDS.md 27269 9984 docs/CONVENTIONS.md diff --git a/AGENTS.md b/AGENTS.md index 3a8b632..5963de2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,7 +51,7 @@ Full tool table, prefetch query, per-tool guidance, cross-repo search: ## Code Exploration Notes (repo-specific) -**The manifest is a source, not the artifact.** Nothing re-embeds it, so re-run `codebase_context_index` in the same change as a `description` edit — otherwise the highest-authority answer an agent gets stays the stale one (#19 CR #17). +**The manifest is a source, not the artifact.** Nothing re-embeds it, so re-run `codebase_context_index` in the same change as a `description` edit — otherwise the stalest answer is the one with the most authority (#19 CR #17). **`mcp-driver.mjs` lies twice — silently through the `skills/` symlink (skills#177), falsely from a worktree (skills#180).** Use `"$SOCRATICODE_DRIVER"`; disbelieve health findings outside the main checkout. Both in [docs/SKILLS.md](docs/SKILLS.md). @@ -104,8 +104,8 @@ convention: 2. **`.env`** (repo root, git-ignored) — dev/agent secrets, chiefly org-wide GitHub PATs. Never commit. **The service must never load the repo `.env`.** Those PATs carry write access the -worker has no use for, and a fetcher of public URLs must not widen their blast -radius. Anything the service needs goes in `/etc/replicator/.env`. +worker has no use for, and a process whose job is fetching public URLs must not +widen their blast radius. Anything the service needs goes in `/etc/replicator/.env`. New settings take the `REPLICATOR_` prefix — the VM is shared, and the prefix is what keeps a sibling service from colliding. `BUILD_ID` is the one deliberate @@ -233,7 +233,8 @@ from src.core.logging import get_logger logger = get_logger(__name__) ``` -Entry points only: `configure_logging()` is called once inside the FastAPI `lifespan` or the worker's `run()`. Never in library modules. The stack itself — formatter, installers, the journald lines deliberately not JSON — is in [docs/STYLE.md](docs/STYLE.md). +Entry points only: `configure_logging()` is called once inside the FastAPI `lifespan` or the worker's `run()`. Never in library modules. +The stack itself: [docs/STYLE.md](docs/STYLE.md). **Date & Time:** - All UTC @@ -248,10 +249,10 @@ with its rationale and ruff gate in [docs/STYLE.md](docs/STYLE.md). - [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) — founding design, the command → fact flow, module by module; read before changing one - [docs/STREAMS.md](docs/STREAMS.md) — what each stream carries, one bullet per rule `AGENTS.md` states in a line - [docs/CONVENTIONS.md](docs/CONVENTIONS.md) — the rules common to every stream: idempotency, validation, DLQ, `claim_stale`; and the `replicator:cmd:*` keys (#80) -- [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the three populations under `REPLICATOR_BLOB_DIR`, TTL and ceilings +- [docs/STORAGE.md](docs/STORAGE.md) — blob paths and modes, the populations under `REPLICATOR_BLOB_DIR`, TTL and ceilings - [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — VM topology, ports, the unit's lifecycle, the co-core pin - [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md) — every variable either env file carries, and the boundary between them -- [docs/TESTING.md](docs/TESTING.md) — fakeredis's divergences, the keys an integration run may create, and why production `co-gcs-replication` is unreachable from every test (#38) +- [docs/TESTING.md](docs/TESTING.md) — fakeredis's divergences, the keys an integration run may create, why production `co-gcs-replication` is unreachable (#38) - [docs/STYLE.md](docs/STYLE.md) — the logging stack: formatter, installers, and the non-JSON journald lines - [docs/COMMANDS.md](docs/COMMANDS.md) — every runnable command, with flags - [docs/SKILLS.md](docs/SKILLS.md) — vendored skill inventory, refresh procedure, doc-check sensitive paths From b67a35ce15520dc955024205f2bf857ad3cb4e74 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Wed, 9 Sep 2026 23:04:25 +0000 Subject: [PATCH 10/10] curate: rewrite this run's ledger row to the tree that ships (5,988) Phase 7's rule: a late fix rewrites the run's own row, and only across runs is the ledger append-only. CR 13-20 moved the count from 5,970 to 5,988 and the warrant count from 81 to 77, so the row now describes the commit it names. Co-Authored-By: Claude Opus 5 (1M context) --- .skills/context-metrics.jsonl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.skills/context-metrics.jsonl b/.skills/context-metrics.jsonl index 5c8b249..9758e2d 100644 --- a/.skills/context-metrics.jsonl +++ b/.skills/context-metrics.jsonl @@ -14,4 +14,4 @@ {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 14876, "delta_days": 5, "delta_tokens": 14, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "214df18", "seams": 3, "seams_acked": 28, "skill_commit": "662de71", "skill_version": "1.12", "tokens": 5999, "tokens_exact": true, "tokens_live": 105677, "top_section": "Bus Conventions", "top_section_share": 25, "ts": "2026-09-01"} {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 14878, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": 7, "delta_tokens": 2, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": true, "repo": "replicator", "repo_commit": "c3f0e31", "seams": 3, "seams_acked": 28, "skill_commit": "f603abf", "skill_version": "1.16", "tokens": 6001, "tokens_exact": true, "tokens_live": 109141, "top_section": "Bus Conventions", "top_section_share": 25, "ts": "2026-09-08"} {"actions": ["baseline:pre-curation"], "budget": 6000, "bytes": 16709, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": 1, "delta_tokens": 746, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 289, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": true, "repo": "replicator", "repo_commit": "8d63a77", "seams": null, "seams_acked": null, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 6747, "tokens_exact": true, "tokens_live": 116261, "top_section": "Bus Conventions", "top_section_share": 33, "ts": "2026-09-09"} -{"actions": ["tighten:Bus Conventions", "tighten:Detail Docs", "demote:Project Overview", "prune:Bus Conventions mini-index", "fix:count-precision"], "budget": 6000, "bytes": 14501, "claims_dropped": 0, "claims_warranted": 0, "counts": 0, "counts_acked": 9, "delta_days": 0, "delta_tokens": -777, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 265, "links_dead": 0, "links_dead_anchors": 0, "no_loss": "ok", "no_loss_warrants": 81, "note": null, "over_budget": false, "repo": "replicator", "repo_commit": "2f87c71", "seams": 0, "seams_acked": 32, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 5970, "tokens_exact": true, "tokens_live": 115702, "top_section": "Bus Conventions", "top_section_share": 30, "ts": "2026-09-09"} +{"actions": ["tighten:Bus Conventions", "tighten:Detail Docs", "demote:Project Overview", "prune:Bus Conventions mini-index", "fix:count-precision"], "budget": 6000, "bytes": 14508, "claims_dropped": 0, "claims_warranted": 0, "counts": 0, "counts_acked": 9, "delta_days": 0, "delta_tokens": -759, "docs_orphaned": 0, "docs_total": 15, "file": "AGENTS.md", "lines": 263, "links_dead": 0, "links_dead_anchors": 0, "no_loss": "ok", "no_loss_warrants": 77, "note": "rewritten in place after CR 13-20: eight review fixes on top of the curation, measured on the tree that ships", "over_budget": false, "repo": "replicator", "repo_commit": "3aaa8cb", "seams": 0, "seams_acked": 32, "skill_commit": "798803e", "skill_version": "1.16", "tokens": 5988, "tokens_exact": true, "tokens_live": 115702, "top_section": "Bus Conventions", "top_section_share": 30, "ts": "2026-09-09"}