diff --git a/.dockerignore b/.dockerignore index f1e6c865..bb5e81e2 100644 --- a/.dockerignore +++ b/.dockerignore @@ -3,6 +3,7 @@ certbot logs data +profiles *.log __pycache__ **/__pycache__ @@ -15,3 +16,41 @@ venv # Secrets — never in an image layer (compose injects these at runtime via env_file) .env .env.* + +# Bloat + secrets that COPY . . would otherwise bake into every image layer. +# backups/ is the sharp one: it holds full production pg_dumps AND a +# plaintext prod .env (SLACK_BOT_TOKEN_* per agent, POSTGRES_PASSWORD, +# SLACK_CONFIG_TOKEN/_REFRESH_TOKEN) that the .env/.env.* patterns above do +# not catch — it has no leading dot and sits two directories down. +backups +.venv-test +.venv* +tests +.notes +mutants +build +.hypothesis +.ruff_cache +.coverage +.mutmut-cache +# Created by ci.sh's mypy stage — large relative to the built image's /app +# tree, and not something any runtime process reads. +.mypy_cache +# A stray uv.lock (a `uv` command run without --no-project writes one) is a second, +# competing lockfile; requirements.lock is the only one the image installs. +uv.lock +.playwright-mcp +copi.egg-info +.superpowers +**/.env* +docs/specs +# The one-off implementation plans and their evidence subtrees — a large +# fraction of /app's tree for text no process reads. Nothing under src/, +# scripts/, templates/ or alembic/ opens a path below docs/; every reference +# is a comment or a docstring (`grep -rn 'docs/' src/ scripts/ templates/ +# alembic/`), which is the same reasoning that already excludes docs/specs +# above. docs/ itself is deliberately NOT excluded: docs/production-migration.md +# and docs/inbound-email.md are operator runbooks worth having next to the +# code in a shell inside the container. +docs/plans +docs/superpowers diff --git a/.gitignore b/.gitignore index 2c088065..a5696559 100644 --- a/.gitignore +++ b/.gitignore @@ -37,6 +37,7 @@ profiles/**/*.md # Sweep backups of agent memory (same prose as the .md files — never commit; # this repo is public and a blanket `git add -A` must not pick them up). profiles/**/*.pre-sweep +profiles/**/*.tmp profiles/*.log # Generated/scratch data and static asset bundles (not source) @@ -63,6 +64,17 @@ certbot/ .coverage htmlcov/ +# mypy's incremental cache (scripts/ci.sh's mypy stage, #27 I5) — 58 MB of +# per-module analysis state, no reason to ever commit it. +.mypy_cache/ + +# Stray uv-native lockfile. requirements.lock (pip-compile, hash-pinned) is +# the one actually installed by the Dockerfile and checked by ci.sh's +# freshness gate; `uv lock`/`uv sync` commands run against this repo would +# otherwise leave a second, competing lockfile that is a public-repo secret +# risk if it ever picks up an internal index URL or credentials (#27 I5). +uv.lock + # Generated by scripts/export_agent_roster.py (roster snapshot for host provisioning) data/agent_roster.json diff --git a/AGENT.md b/AGENT.md index a94b3384..e5919bdc 100644 --- a/AGENT.md +++ b/AGENT.md @@ -4,9 +4,9 @@ Python implementation of the CoPI researcher collaboration platform combined with the LabAgent multi-agent Slack system. ORCID OAuth, profile generation pipeline, profile editing UI, admin dashboard, and Slack-based AI agent simulation. -**GitHub:** https://github.com/andrewsu/coPI-python-opus +**GitHub:** https://github.com/SuLab/coPI.science **Target domain:** copi.science -**Pilot:** 10 labs at Scripps Research +**Pilot:** Scripps Research labs — the live roster and count are at **/admin/agents** (see CLAUDE.md "Adding New PIs"); the number changes as PIs are provisioned. ## What's In Scope @@ -14,14 +14,21 @@ Python implementation of the CoPI researcher collaboration platform combined wit - Profile ingestion pipeline (ORCID → PubMed → PMC → Claude Opus synthesis) - Profile review/editing web UI (FastAPI + Jinja2) - Admin dashboard (users, profiles, jobs, agent activity) -- Slack agent system (8 bots, simulation engine) +- Slack agent system (bot count tracked live at /admin/agents; simulation engine) ## What's Out of Scope - Matching engine (pairwise proposal generation) - Swipe interface -- Notifications (email) -- Daily digest + +## What Email Actually Does (in scope, built) + +`src/services/email.py`, `email_inbound.py`, `email_notifications.py` — +proposal-review emails, reply intake, unsubscribe/settings, and a periodic +status-overview digest (`check_and_send_status_overviews`, called from the worker's +~300 s poll loop in `src/worker/main.py`; each PI's own cadence comes from +`email_notification_preferences.frequency`, which defaults to weekly — +`daily` is one option on that ladder, not the schedule). ## Key Specs @@ -73,8 +80,8 @@ templates/ # Jinja2 HTML templates Decisions made autonomously during implementation are recorded here for human review. ### 2026-03-20: Admin impersonation endpoint location -**Decision:** Admin impersonation routes placed at `/api/admin/impersonate` (POST) and `/api/admin/impersonate/stop` (POST) rather than inside the `/admin` router prefix. -**Reason:** The impersonate stop button posts from any page (including non-admin pages when impersonating), so a clean `/api/admin/` prefix was clearer. Both routes still require is_admin verification. +**Decision:** Admin impersonation routes are `POST /admin/impersonate` and `POST /admin/impersonate/stop` (`src/routers/admin.py`, mounted under the `/admin` prefix in `src/main.py`). *(Corrected 2026-09: this entry originally described an `/api/`-prefixed design that was never what shipped — see issue #26 DOC-3.)* +**Reason:** The impersonate-stop button posts from any page (including non-admin pages when impersonating). Both routes still require is_admin verification. ### 2026-03-20: Login page GET /login serves both redirect and HTML **Decision:** `/login` GET route redirects directly to ORCID OAuth if not already logged in. The login.html page has its sign-in button also pointing to `/login` (which re-triggers the redirect). @@ -102,7 +109,7 @@ Decisions made autonomously during implementation are recorded here for human re **Reason:** Avoids Node.js build step in a Python project. Acceptable for pilot; switch to compiled Tailwind for production if performance matters. ### 2026-03-20: Profile markdown export -**Decision:** When a ResearcherProfile is saved/updated in the DB, automatically export it to `profiles/public/{lab}.md` if the user is one of the 8 pilot labs (matched by ORCID). +**Decision:** When a ResearcherProfile is saved/updated in the DB, automatically export it to `profiles/public/{lab}.md` (matched by ORCID). Written when there were 8 pilot labs; the roster is now whatever `AgentRegistry` holds — see `/admin/agents` for the live count, and `orcids.txt` (48 entries) for the seeding list. Do not treat any number in this file as the current roster size. **Reason:** Keeps the DB (source of truth) and filesystem (agent input) in sync without a separate sync step. ## Implementation Status @@ -115,11 +122,15 @@ Decisions made autonomously during implementation are recorded here for human re - [x] Admin dashboard - [x] Worker process - [x] Agent system (Slack bots, simulation engine) -- [x] Agent profiles (8 pilot labs, auto-generated structure) +- [x] Agent profiles (auto-generated structure; current roster at /admin/agents) - [x] Prompt files ## Pilot Lab ORCIDs +**Historical snapshot — the original pilot cohort, not the current roster.** The live roster is +`AgentRegistry` (see `/admin/agents`); the seeding list is `orcids.txt`, which now holds 48 ORCIDs. +This table is kept because the decisions above refer to it. + | PI | ORCID | |---|---| | Andrew Su | 0000-0002-9859-4104 | diff --git a/CLAUDE.md b/CLAUDE.md index c19aeca4..a73e681b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -17,6 +17,12 @@ docker compose exec -T -e TEST_DATABASE_URL=postgresql+asyncpg://copi:copi@postg app python -m pytest tests/ -v ``` +**Note (2026-09, #27 I3):** the command above works against the **dev** compose file, whose +`.:/app` bind mount supplies `tests/`. `tests/` is excluded from the built image by +`.dockerignore`, so the same command against a prod-built container reports +"file or directory not found: tests/". Run the suite on the host (`./scripts/ci.sh`) or with the +dev compose file. + The named database must already exist — the suite migrates it, it does not create it. Add a fresh scratch DB with `docker compose exec -T postgres createdb -U copi copi_xN`, and give concurrent @@ -72,9 +78,11 @@ docker compose $C --profile agent run -d --name agent-run agent python -m src.ag On resume the sim fetches Slack history for each bot in roster order before reaching turn 1. Slack throttles this hard — expect ~10 minutes of -`[] Rate limited, retrying in 10s (attempt 1/3)` before the first -`=== Turn 1 ===`. Repeated `attempt 1/3` (never `2/3`) means each call 429s once then -succeeds on retry — that is forward progress, not a hang. +`[] Rate limited, retrying in 10s (attempt 1/8, 10.0s/180.0s of wait +budget used)` before the first `=== Turn 1: ===`. Repeated `attempt 1/8` +(never `2/8`) means each call 429s once then succeeds on retry — that is forward +progress, not a hang (RC-3 on this branch raised the ceiling from 3 attempts to 8 and +added the 180s cumulative wait-budget figure logged alongside it). **Before restarting**, always save logs and rebuild containers: @@ -88,12 +96,36 @@ ls -t logs/run_*.log | tail -n +11 | xargs rm -f # 2. Stop the old container — GRACEFULLY. `docker rm -f` sends SIGKILL, which # skips the shutdown flush and permanently loses the in-flight turn's # messages (the DB, not Slack, is the durable store). `docker stop` sends -# SIGTERM; -t 30 leaves room for an in-flight LLM call to finish. +# SIGTERM; -t 30 leaves room for an in-flight LLM call to finish. One +# SIGTERM stops after the current turn and aborts Slack retry sleeps 20 s +# later; a SECOND signal aborts Slack immediately (the DB flush still +# runs); a THIRD terminates the process at once and can lose the flush -- +# never send a third unless the process is wedged. docker stop -t 30 agent-run docker rm agent-run -# 3. Rebuild app + worker (picks up code changes) -docker compose $C up -d --build app worker +# 3. Redeploy app + worker + grantbot against the migrated schema — via +# scripts/redeploy.sh, NOT a bare `up -d --build`. `depends_on: migrate: condition: +# service_completed_successfully` only orders container CREATION: on an +# already-running stack, an existing exited `migrate` container from the last +# deploy can satisfy that condition without being re-run against the freshly +# built image, so old code can keep serving requests against a schema the new +# migration hasn't applied yet (audit 2026-09-08 RC-6, #27 I2; audit 2026-09-10 +# R-4 added grantbot, which has the identical depends_on shape and was +# otherwise left running the old image). redeploy.sh builds +# migrate+app+worker+grantbot, STOPS app/worker/grantbot first, runs migrate +# and checks its exit code, only then starts the new app/worker/grantbot, then +# reloads nginx (the recreated app container gets a new IP — see the +# nginx-stale-upstream-ip memory note). `agent` is NOT part of this — it is a +# one-off with its own restart runbook above. It refuses to run unless both prod compose files are visible +# (via $COMPOSE_FILE or -f) and never passes an orphan-removal flag. The image +# runs as UID 10001, so profiles/ and data/ on the host must already be owned +# by 10001:10001 (never prompts/ — see docs/production-migration.md §10.8 and +# Part R.5 of docs/plans/2026-09-02-close-issues-20-27.md) or the services +# that mount them fail to write into their bind mounts — profiles/ and +# prompts/ are mounted on app/worker (and agent/grantbot); data/ is mounted +# only on agent/grantbot, not app/worker. +./scripts/redeploy.sh $C # 4. Rebuild the agent image too — prod bakes code into the image, so skipping # this silently runs whatever source was current at the last build. @@ -115,6 +147,18 @@ they can decide whether to restart.** Roster changes — activating/inactivating setting a new `slack_bot_token` in `AgentRegistry` — do NOT need a restart; they're picked up live by `_sync_roster_from_db`. +**One-time Slack-ts repair (legacy rows).** A workspace that predates the DB-primary +conversation model may have `agent_messages` rows with `slack_ts IS NULL`. Replies to +threads rooted on those rows are silently kept off Slack — `_slack_parent_ts` +(`src/agent/simulation.py`) returns `None` for a legacy root and callers skip the +mirror rather than guess a timestamp Slack never issued. Run +`docker compose exec -e PYTHONPATH=/app app python scripts/backfill_slack_ts.py --apply` once, before your +next restart, to ask Slack which timestamps actually exist and repair them (safe to +re-run; read-only against Slack otherwise). `docs/production-migration.md` §8 Step 8 +walks through this as an ordered step after the migration and before the app-code +deploy for a *fresh* migration; if your workspace is already at head and has never run +it, run it manually — nothing else will prompt you to. + ## Adding New PIs **The `AgentRegistry` table is the single source of truth for the agent roster.** @@ -139,7 +183,7 @@ Each agent needs an `AgentRegistry` row with a unique `agent_id` (lowercase last and `bot_name` (`{LastName}Bot`), created `status='pending'`. Self-service signups (`src/routers/agent_page.py`) and the backfill scripts both create these automatically. -**Last-name collisions:** If a last name is already taken (e.g., Chunlei Wu = `wu`), prefix with the first initial (e.g., Peng Wu = `pwu` / `PWuBot`). The web UI applies this logic automatically. +**Last-name collisions:** If a last name is already taken (e.g., Chunlei Wu = `wu`), prefix with the first initial (e.g., Peng Wu = `pwu` / `PWuBot`). If that prefixed id is *also* taken (a third same-initial namesake), append a numeric suffix to the prefixed candidate (e.g. `pwu2` / `PWu2Bot`). The web UI applies this logic automatically. ### 3. Provision the Slack bot + activate (admin UI) diff --git a/Dockerfile b/Dockerfile index c032e953..adfb6686 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,23 +1,68 @@ -FROM python:3.11-slim +FROM python:3.11-slim AS builder WORKDIR /app -# Install build dependencies +# Build-time only: gcc/libpq-dev compile any dependency that ships as an sdist +# for this platform/Python combination. Not present in the runtime image +# below — asyncpg itself needs none of this, it bundles its own wire protocol +# implementation rather than linking libpq. RUN apt-get update && apt-get install -y --no-install-recommends \ gcc \ libpq-dev \ && rm -rf /var/lib/apt/lists/* -# Install Python dependencies -COPY pyproject.toml . +COPY pyproject.toml requirements.lock ./ +RUN pip install --no-cache-dir --require-hashes -r requirements.lock COPY src/ src/ -RUN pip install --no-cache-dir . +# --no-build-isolation: build isolation would otherwise fetch a fresh, +# unhashed setuptools/wheel from PyPI at build time just to satisfy +# pyproject.toml's [build-system] requires; the base image's preinstalled +# setuptools/wheel already satisfy it. This local `pip install .` therefore +# is not hash-verified the way the `-r requirements.lock` install above is +# — accepted, since it installs only this repo's own source, not a +# third-party artifact off the network. +RUN pip install --no-cache-dir --no-deps --no-build-isolation . -# Copy source +FROM python:3.11-slim AS runtime + +WORKDIR /app + +# libpq5 only: the runtime client library a compiled wheel may dlopen. Nothing +# currently links it — asyncpg is pure-protocol — this is insurance for a +# future psycopg dependency. No compiler, no -dev headers, no build toolchain +# of any kind in this stage. Deliberately avoids naming the builder-stage +# packages here — the structural test in tests/unit/test_dockerfile_build.py +# asserts their names are absent from this section. +RUN apt-get update && apt-get install -y --no-install-recommends \ + libpq5 \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages +COPY --from=builder /usr/local/bin /usr/local/bin COPY . . -# Create directories for profiles and prompts -RUN mkdir -p profiles/public profiles/private prompts logs static +# Bake the bytecode cache while root still owns src/ — UID 10001 (set below) +# cannot write __pycache__ into root-owned src/, so without this every +# process start pays a first-import compile cost (~0.9s, measured). Must run +# AFTER src/ lands (COPY . . above) and BEFORE USER drops root. +RUN python -m compileall -q src + +# Fixed UID so it matches whatever the prod host chowns the bind-mounted +# profiles/data trees to — a plain chown target on the host, not a real host +# account. Ownership is scoped to the directories the runtime user actually +# writes to (profiles/data/logs); src/, templates/, alembic/, scripts/ and +# static/ stay root-owned and read-only to this user, so a compromised +# process cannot rewrite its own code. static/ is deliberately excluded: +# StaticFiles only ever reads it, nothing under src/ writes to it, so a write +# grant there would be a needless stored-XSS surface on assets served +# straight to the browser. +RUN groupadd --gid 10001 copi \ + && useradd --uid 10001 --gid 10001 --no-create-home --shell /usr/sbin/nologin copi \ + && mkdir -p profiles/public profiles/private profiles/memory data logs \ + && chown -R 10001:10001 profiles data logs +ENV HOME=/app + +USER 10001 EXPOSE 8000 diff --git a/README.md b/README.md index c4a7fe39..3b4f7760 100644 --- a/README.md +++ b/README.md @@ -5,9 +5,10 @@ discovers collaboration opportunities, shares resources, and explores research synergies with other lab agents in natural language. Promising ideas are escalated to PIs for human input. -Currently piloting with 14+ labs at Scripps Research, with multi-institution -expansion in progress. See `labbot-spec.md` for the full system specification -and `specs/` for component-level designs. +Piloting with Scripps Research labs, with multi-institution expansion in +progress — the current roster and count are at **/admin/agents**. See +`labbot-spec.md` for the full system specification and `specs/` for +component-level designs. ## Architecture @@ -59,6 +60,12 @@ All tests must pass before committing. ## Running the agent simulation +**Note:** the commands below are the dev shape (bare `docker compose`, reading +`docker-compose.yml`). For prod, every command needs +`-f docker-compose.prod.yml -f docker-compose.override.yml` — see "Running the +Agent Simulation" in `CLAUDE.md` for the full prod runbook, including the +mandatory rebuild-before-restart steps. + ```bash # Resume an existing run (no budget limit): docker compose --profile agent run -d --name agent-run agent \ @@ -85,8 +92,19 @@ docker compose --profile agent run -d --name agent-run agent \ python -m src.agent.main --budget 0 ``` -The `agent-run` container mounts source code but only loads modules at -startup — code changes affecting the running agent require a restart. +**One-time repair:** if this workspace predates the DB-primary conversation model and +has never run it, run +`docker compose exec -e PYTHONPATH=/app app python scripts/backfill_slack_ts.py --apply` once before your +next restart — legacy `agent_messages` rows with `slack_ts IS NULL` otherwise keep +Slack replies to their threads silently off Slack. See `docs/production-migration.md` +§8 and the fuller note in `CLAUDE.md`. + +Under prod compose, `agent-run` **bakes** the source into the image — a code +change requires rebuilding the agent image +(`docker compose ... --profile agent build agent`), not just a restart. See +"Running the Agent Simulation" in `CLAUDE.md` for the full command set, +including the `-f docker-compose.prod.yml -f docker-compose.override.yml` +flags every prod compose command needs. ## Adding new PIs @@ -94,10 +112,16 @@ startup — code changes affecting the running agent require a restart. `docker compose exec app python -m src.cli seed-profiles --file new_orcids.txt`. 2. Add an `AgentRegistry` row (`agent_id` = lowercase last name, `bot_name` = `{LastName}Bot`, `status='pending'`). For last-name collisions, prefix with - the first initial (e.g., `pwu` / `PWuBot`). -3. Create a Slack bot token per agent and add to env config. -4. Add to `PILOT_LABS` in `src/agent/simulation.py` and restart the - simulation. + the first initial (e.g., `pwu` / `PWuBot`). If that prefixed id is *also* + taken (a third same-initial namesake), append a numeric suffix to the + prefixed candidate (e.g. `pwu2` / `PWu2Bot`). +3. Provision the Slack bot and activate the agent from **/admin/agents** in + the web UI. `AgentRegistry` is the single source of truth for the roster — + there is no hardcoded roster list in `src/agent/simulation.py`, and no + `.env`/`config.py` edit. A running + simulation re-syncs from the DB every ~30s (`_sync_roster_from_db`), so + activating the agent goes live with no restart. See "Adding New PIs" in + `CLAUDE.md` for the full provisioning flow (including bulk provisioning). ## Repository layout diff --git a/alembic/versions/0001_initial.py b/alembic/versions/0001_initial.py index 68337508..f32ff98e 100644 --- a/alembic/versions/0001_initial.py +++ b/alembic/versions/0001_initial.py @@ -2,7 +2,6 @@ Revision ID: 0001 Revises: -Create Date: 2026-03-20 00:00:00.000000 """ diff --git a/alembic/versions/0002_add_llm_call_logs.py b/alembic/versions/0002_add_llm_call_logs.py index e465fa1d..16fb1506 100644 --- a/alembic/versions/0002_add_llm_call_logs.py +++ b/alembic/versions/0002_add_llm_call_logs.py @@ -2,7 +2,6 @@ Revision ID: 0002 Revises: 0001 -Create Date: 2026-03-22 00:00:00.000000 """ diff --git a/alembic/versions/0003_add_thread_decisions_and_update_agent_messages.py b/alembic/versions/0003_add_thread_decisions_and_update_agent_messages.py index 4226ceed..59690817 100644 --- a/alembic/versions/0003_add_thread_decisions_and_update_agent_messages.py +++ b/alembic/versions/0003_add_thread_decisions_and_update_agent_messages.py @@ -2,7 +2,6 @@ Revision ID: 0003 Revises: 0002 -Create Date: 2026-03-26 00:00:00.000000 """ diff --git a/alembic/versions/0004_add_agent_registry_and_proposal_reviews.py b/alembic/versions/0004_add_agent_registry_and_proposal_reviews.py index 18d299e2..10992437 100644 --- a/alembic/versions/0004_add_agent_registry_and_proposal_reviews.py +++ b/alembic/versions/0004_add_agent_registry_and_proposal_reviews.py @@ -2,7 +2,6 @@ Revision ID: 0004 Revises: 0003 -Create Date: 2026-03-27 00:00:00.000000 """ diff --git a/alembic/versions/0005_add_private_profile_columns.py b/alembic/versions/0005_add_private_profile_columns.py index e2680c7b..027a1c42 100644 --- a/alembic/versions/0005_add_private_profile_columns.py +++ b/alembic/versions/0005_add_private_profile_columns.py @@ -2,7 +2,6 @@ Revision ID: 0005 Revises: 0004 -Create Date: 2026-03-30 00:00:00.000000 """ diff --git a/alembic/versions/0006_add_delegate_slack_ids.py b/alembic/versions/0006_add_delegate_slack_ids.py index 758eb6cd..bbe847d7 100644 --- a/alembic/versions/0006_add_delegate_slack_ids.py +++ b/alembic/versions/0006_add_delegate_slack_ids.py @@ -2,7 +2,6 @@ Revision ID: 0006 Revises: 0005 -Create Date: 2026-04-01 00:00:00.000000 """ diff --git a/alembic/versions/0007_add_web_delegates.py b/alembic/versions/0007_add_web_delegates.py index 9b5b00c0..fddf8b23 100644 --- a/alembic/versions/0007_add_web_delegates.py +++ b/alembic/versions/0007_add_web_delegates.py @@ -2,7 +2,6 @@ Revision ID: 0007 Revises: 0006 -Create Date: 2026-04-03 00:00:00.000000 """ diff --git a/alembic/versions/0008_email_notifications.py b/alembic/versions/0008_email_notifications.py index 9051aefc..6aa8488a 100644 --- a/alembic/versions/0008_email_notifications.py +++ b/alembic/versions/0008_email_notifications.py @@ -2,7 +2,6 @@ Revision ID: 0008 Revises: 0007 -Create Date: 2026-04-03 00:00:00.000000 """ diff --git a/alembic/versions/0009_profile_revisions.py b/alembic/versions/0009_profile_revisions.py index 11df42f5..162442ed 100644 --- a/alembic/versions/0009_profile_revisions.py +++ b/alembic/versions/0009_profile_revisions.py @@ -2,7 +2,6 @@ Revision ID: 0009 Revises: 0008 -Create Date: 2026-04-04 00:00:00.000000 """ diff --git a/alembic/versions/0010_access_gate_and_waitlist.py b/alembic/versions/0010_access_gate_and_waitlist.py index 36c0ec61..ffc80d27 100644 --- a/alembic/versions/0010_access_gate_and_waitlist.py +++ b/alembic/versions/0010_access_gate_and_waitlist.py @@ -2,7 +2,6 @@ Revision ID: 0010 Revises: 0009 -Create Date: 2026-04-15 00:00:00.000000 """ diff --git a/alembic/versions/0011_channel_visibility_and_private_members.py b/alembic/versions/0011_channel_visibility_and_private_members.py index b3a800bb..9a941a8f 100644 --- a/alembic/versions/0011_channel_visibility_and_private_members.py +++ b/alembic/versions/0011_channel_visibility_and_private_members.py @@ -2,7 +2,6 @@ Revision ID: 0011 Revises: 0010 -Create Date: 2026-04-20 00:00:00.000000 """ diff --git a/alembic/versions/0012_grantbot_posted_foas.py b/alembic/versions/0012_grantbot_posted_foas.py index 08b574f8..8fbc7ad9 100644 --- a/alembic/versions/0012_grantbot_posted_foas.py +++ b/alembic/versions/0012_grantbot_posted_foas.py @@ -2,7 +2,6 @@ Revision ID: 0012 Revises: 0011 -Create Date: 2026-04-24 00:00:00.000000 """ diff --git a/alembic/versions/0013_drop_agent_registry_slack_app_token.py b/alembic/versions/0013_drop_agent_registry_slack_app_token.py index f391cd7b..8b65bd03 100644 --- a/alembic/versions/0013_drop_agent_registry_slack_app_token.py +++ b/alembic/versions/0013_drop_agent_registry_slack_app_token.py @@ -2,7 +2,6 @@ Revision ID: 0013 Revises: 0012 -Create Date: 2026-04-30 00:00:00.000000 """ diff --git a/alembic/versions/0014_add_last_login_at.py b/alembic/versions/0014_add_last_login_at.py index 32a3735d..62ab8d7d 100644 --- a/alembic/versions/0014_add_last_login_at.py +++ b/alembic/versions/0014_add_last_login_at.py @@ -2,7 +2,6 @@ Revision ID: 0014 Revises: 0013 -Create Date: 2026-05-13 00:00:00.000000 """ diff --git a/alembic/versions/0015_add_proposal_votes.py b/alembic/versions/0015_add_proposal_votes.py index 90a7e8c0..e44ad07c 100644 --- a/alembic/versions/0015_add_proposal_votes.py +++ b/alembic/versions/0015_add_proposal_votes.py @@ -2,7 +2,6 @@ Revision ID: 0015 Revises: 0014 -Create Date: 2026-06-07 00:00:00.000000 """ diff --git a/alembic/versions/0016_notification_categories.py b/alembic/versions/0016_notification_categories.py index a3cf126c..f19c09e8 100644 --- a/alembic/versions/0016_notification_categories.py +++ b/alembic/versions/0016_notification_categories.py @@ -2,7 +2,6 @@ Revision ID: 0016 Revises: 0015 -Create Date: 2026-06-10 00:00:00.000000 """ diff --git a/alembic/versions/0017_provisioning_tables.py b/alembic/versions/0017_provisioning_tables.py index b69b3c67..bbe0fa63 100644 --- a/alembic/versions/0017_provisioning_tables.py +++ b/alembic/versions/0017_provisioning_tables.py @@ -2,7 +2,6 @@ Revision ID: 0017 Revises: 0016 -Create Date: 2026-06-26 00:00:00.000000 """ diff --git a/alembic/versions/0018_allowlist_email_hint.py b/alembic/versions/0018_allowlist_email_hint.py index f191d17a..d8c31547 100644 --- a/alembic/versions/0018_allowlist_email_hint.py +++ b/alembic/versions/0018_allowlist_email_hint.py @@ -2,7 +2,6 @@ Revision ID: 0018 Revises: 0017 -Create Date: 2026-06-29 00:00:00.000000 """ diff --git a/alembic/versions/0019_agent_message_content.py b/alembic/versions/0019_agent_message_content.py index 17c52a37..1c48c514 100644 --- a/alembic/versions/0019_agent_message_content.py +++ b/alembic/versions/0019_agent_message_content.py @@ -2,7 +2,6 @@ Revision ID: 0019 Revises: 0018 -Create Date: 2026-07-20 00:00:00.000000 Makes the local DB the primary store for agent conversations: agent_messages now carries the message body and sender metadata (previously only in Slack + the diff --git a/alembic/versions/0020_pi_dm_messages.py b/alembic/versions/0020_pi_dm_messages.py index b2acd57d..4cca9935 100644 --- a/alembic/versions/0020_pi_dm_messages.py +++ b/alembic/versions/0020_pi_dm_messages.py @@ -2,7 +2,6 @@ Revision ID: 0020 Revises: 0019 -Create Date: 2026-07-20 00:00:00.000000 DMs never entered the shared message log, so they had no durable home. This table stores them so a PI can DM their bot (standing instructions, questions) diff --git a/alembic/versions/0021_inbox_cursor_created_at_indexes.py b/alembic/versions/0021_inbox_cursor_created_at_indexes.py index c2cc1ae5..2a73ebd0 100644 --- a/alembic/versions/0021_inbox_cursor_created_at_indexes.py +++ b/alembic/versions/0021_inbox_cursor_created_at_indexes.py @@ -2,15 +2,13 @@ Revision ID: 0021 Revises: 0020 -Create Date: 2026-07-25 00:00:00.000000 Both DB inbox pollers used to page over ``posted_at``, which is derived from the *writing process's* clock (float of its minted ts). That made inbound PI delivery depend on every writer's clock agreeing with the engine's to within the lookback window — fine on one host, silently lossy across hosts. They now page over ``created_at`` (``server_default=now()``, i.e. the single Postgres server's -clock), so these indexes back the new access path. See -.notes/db-conversations-residual-2026-07-24.md (R3). +clock), so these indexes back the new access path. """ from typing import Sequence, Union diff --git a/alembic/versions/0022_add_cohorts.py b/alembic/versions/0022_add_cohorts.py index 5a594efd..24bd7b07 100644 --- a/alembic/versions/0022_add_cohorts.py +++ b/alembic/versions/0022_add_cohorts.py @@ -2,17 +2,15 @@ Revision ID: 0022 Revises: 0021 -Create Date: 2026-07-30 00:00:00.000000 Renumbered from 0019 at merge time. The cohort branch was cut before main's db-primary work, so its original "0019" collided with 0019_agent_message_content: two revisions sharing an id resolve to whichever file sorts last, which silently skips the other while stamping the DB as fully migrated. Revision ids are assigned -at merge, never at branch. See .notes/cohort-system-v2.md §4.2 / §14 and the -alembic guard in scripts/ci.sh. +at merge, never at branch; see the alembic guard in scripts/ci.sh. Downgrades are idempotent (if_exists) so a rollback cannot wedge on an object that -a partially-applied upgrade never created. See v2 §14.4. +a partially-applied upgrade never created. """ from typing import Sequence, Union @@ -30,7 +28,7 @@ def upgrade() -> None: # A cohort is a named group of agents permitted to act on each other's - # activity during simulation. See .notes/cohort-system-v2.md. + # activity during simulation. op.create_table( "cohorts", sa.Column("id", UUID(as_uuid=True), primary_key=True), @@ -91,7 +89,7 @@ def upgrade() -> None: # and NO FK on cohort_id. `topology` snapshots the full cohort->members map # plus the active gate settings at run start and on every change, so a # completed simulation run stays attributable to the configuration that - # produced it (v2 §13.1). + # produced it. op.create_table( "cohort_audit_events", sa.Column("id", UUID(as_uuid=True), primary_key=True), diff --git a/alembic/versions/0023_profile_synthesis_provenance.py b/alembic/versions/0023_profile_synthesis_provenance.py index 889c15ed..82aa2678 100644 --- a/alembic/versions/0023_profile_synthesis_provenance.py +++ b/alembic/versions/0023_profile_synthesis_provenance.py @@ -2,7 +2,6 @@ Revision ID: 0023 Revises: 0022 -Create Date: 2026-07-31 00:00:00.000000 Two defects in src/services/profile_pipeline.py were invisible because the pipeline wrote down nothing about *how* a profile was produced: diff --git a/alembic/versions/0024_add_agent_role.py b/alembic/versions/0024_add_agent_role.py index f44e3e7a..dc659b29 100644 --- a/alembic/versions/0024_add_agent_role.py +++ b/alembic/versions/0024_add_agent_role.py @@ -2,12 +2,10 @@ Revision ID: 0024 Revises: 0023 -Create Date: 2026-08-05 00:00:00.000000 `role` selects per-role prompt overrides (prompts/roles/{role}/) and a per-role tool allow-list. Default 'pi_lab' == the pre-existing all-agents-identical behaviour, so this column is a no-op until an agent is explicitly reassigned. -See docs/specs/2026-08-05-hub-bot-customization-design.md. Downgrade is idempotent (if_exists) per the branch convention (0022/0023). """ diff --git a/alembic/versions/0025_publications_unique_user_pmid.py b/alembic/versions/0025_publications_unique_user_pmid.py new file mode 100644 index 00000000..89326b83 --- /dev/null +++ b/alembic/versions/0025_publications_unique_user_pmid.py @@ -0,0 +1,140 @@ +"""Deduplicate publications by (user_id, pmid) and add a unique constraint + +Revision ID: 0025 +Revises: 0024 + +publications has never had a uniqueness guarantee on (user_id, pmid) +(models/publication.py had no __table_args__; 0001_initial.py:121-122 created only +non-unique indexes). profile_pipeline.py builds the `pmids` list from an ORCID works +listing with no dedup (a work linked to two affiliation-groups, or a DOI-only work +whose resolved PMID matches one already in `pmids`, lists the same PMID twice) and +the `existing_pubs` lookup dict used to skip re-inserts was built once before the +insert loop and never updated inside it — so a PMID appearing twice in one ORCID +works listing inserted two Publication rows in a single pipeline run. Downstream +this silently: (a) made `scalar_one_or_none()` in the PMC-methods step raise +MultipleResultsFound, swallowed at a debug log; (b) duplicated citation lines in the +exported markdown; (c) inflated admin.py's publication counts; (d) was a multiplier +on the join `_load_publication_records` (simulation.py) re-runs on every roster +sync. The pipeline-side dedup landed separately. + +`pmid` stays nullable — a DOI-only publication has none, and Postgres unique +constraints treat NULL as distinct from every other NULL, so multiple no-PMID rows +for the same user remain legal. + +The dedup step below runs BEFORE the constraint is added: an existing deployment +can already have duplicate rows, and create_unique_constraint fails outright +against a table that violates it. + +The keeper is not chosen by ``p.id > p2.id`` (delete the higher id): +``Publication.id`` is ``default=uuid.uuid4`` (src/models/publication.py), +which is uncorrelated with insertion order or richness, so that rule could delete +a rich row (abstract, methods_text, doi, pmcid, journal, year, author_position) +and keep a title-only one. The keeper is now chosen deterministically by +``created_at ASC, id ASC`` (oldest row wins ties by id), and every nullable data +column on the doomed rows is COALESCE-merged into the keeper before the doomed +rows are deleted, so no data is lost regardless of which row the dedup happens +to keep by identity. + +Downgrade is idempotent (if_exists) per the 0022+ convention; it drops the +constraint only — the row deletions/merges from upgrade() are not (and cannot +be) undone. Because downgrade() cannot restore them, upgrade() prints the +deleted-row and merged-column counts so the migration log is the only trace of +what a given run changed. +""" + +from typing import Sequence, Union + +import sqlalchemy as sa + +from alembic import op + +revision: str = "0025" +down_revision: Union[str, None] = "0024" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + +#: Every nullable data column on Publication (src/models/publication.py) other than +#: the keys (id, user_id, pmid) and created_at itself. `title` is NOT NULL, so it is +#: never merged — the keeper's own title (already required) is left alone. +_MERGE_COLUMNS = ( + "pmcid", + "doi", + "abstract", + "journal", + "year", + "author_position", + "methods_text", +) + + +def upgrade() -> None: + # Keep the row with the earliest created_at (tie-broken by id) per (user_id, + # pmid); pmid IS NOT NULL so distinct-NULL rows (no-PMID publications) are + # never touched. Merge the doomed rows' data into the keeper before deleting + # them, since a doomed row can be richer than the one that happens to be kept + # by identity. RECORD both counts: downgrade() cannot put any of this back, so + # the migration log is the only trace of what a given run changed. + conn = op.get_bind() + dup_rows = conn.execute( + sa.text( + f""" + SELECT id, user_id, pmid, {", ".join(_MERGE_COLUMNS)} + FROM publications + WHERE pmid IS NOT NULL + AND (user_id, pmid) IN ( + SELECT user_id, pmid FROM publications + WHERE pmid IS NOT NULL + GROUP BY user_id, pmid + HAVING count(*) > 1 + ) + ORDER BY user_id, pmid, created_at ASC, id ASC + """ + ) + ).mappings().all() + + groups: dict[tuple, list] = {} + for row in dup_rows: + groups.setdefault((row["user_id"], row["pmid"]), []).append(row) + + deleted = 0 + merged_columns = 0 + for rows in groups.values(): + keeper, *doomed = rows # first row per group is the earliest (ORDER BY above) + updates = {} + for col in _MERGE_COLUMNS: + if keeper[col] is not None: + continue + for loser in doomed: + if loser[col] is not None: + updates[col] = loser[col] + merged_columns += 1 + break + if updates: + set_clause = ", ".join(f"{c} = :{c}" for c in updates) + conn.execute( + sa.text(f"UPDATE publications SET {set_clause} WHERE id = :id"), + {**updates, "id": keeper["id"]}, + ) + doomed_ids = [loser["id"] for loser in doomed] + if doomed_ids: + result = conn.execute( + sa.text("DELETE FROM publications WHERE id IN :ids").bindparams( + sa.bindparam("ids", expanding=True) + ), + {"ids": doomed_ids}, + ) + deleted += result.rowcount + + print( + f"0025: deleted {deleted} duplicate (user_id, pmid) publication rows, " + f"merged {merged_columns} column value(s) into their keepers" + ) + op.create_unique_constraint( + "uq_publications_user_pmid", "publications", ["user_id", "pmid"] + ) + + +def downgrade() -> None: + op.drop_constraint( + "uq_publications_user_pmid", "publications", type_="unique", if_exists=True + ) diff --git a/alembic/versions/0026_pcm_user_cascade.py b/alembic/versions/0026_pcm_user_cascade.py new file mode 100644 index 00000000..300fa110 --- /dev/null +++ b/alembic/versions/0026_pcm_user_cascade.py @@ -0,0 +1,97 @@ +"""Cascade-delete private_channel_members rows when their user is deleted + +Revision ID: 0026 +Revises: 0025 + +private_channel_members.user_id was ondelete="SET NULL" (0011). A role="pi" +membership row always has agent_id IS NULL (src/services/private_channels.py +never sets agent_id on a PI row), so a SET NULL on user_id collides with the +pcm_exactly_one_of_agent_or_user CHECK — the row would end up with BOTH +columns NULL, which is neither "bot" nor "pi". The DB raises a +CheckViolationError and the whole DELETE FROM users fails, making any PI who +was ever a private-channel member permanently undeletable (live-reproduced; +tests/integration/test_db_contract.py now asserts the delete succeeds). + +CASCADE is a pure behaviour change on an existing column, not a backfill: +every existing PI membership row already satisfies the CHECK (agent_id IS +NULL, user_id IS NOT NULL), so there is no data pre-step. added_by_user_id +stays ondelete="SET NULL" — nulling it never violates the CHECK (see the +test_deleting_added_by_user_is_safe contrast case). + +The FK is dropped and recreated under its ORIGINAL implicit name +(private_channel_members_user_id_fkey — 0011's op.create_table gave it no +explicit name, so Postgres assigned the default `__fkey`) +so the two directions of this migration are exact inverses and no other +tooling needs to learn a new constraint name. + +The constraint to drop is not assumed to be named `_FK` — a +`Base.metadata.create_all` bootstrap, a pg_dump/restore that renamed it, or a +hand-edit could leave prod's actual name different, and a hard-coded +`op.drop_constraint` with no `if_exists` would abort `alembic upgrade` mid +stop-the-world window with `constraint "..." does not exist`. Both upgrade() +and downgrade() resolve the real name from `pg_constraint` first (the single +foreign key on `private_channel_members.user_id`) and only fall back to `_FK` +as the expected-default label in the error message / when downgrade finds none +(if_exists-safe no-op). Either direction always recreates the FK under the +canonical `_FK` name, so the two directions stay exact inverses regardless of +what name upgrade() found on the way in. + +Downgrade is idempotent (if_exists) per the branch convention (0022+). +""" + +from typing import Sequence, Union + +import sqlalchemy as sa + +from alembic import op + +revision: str = "0026" +down_revision: Union[str, None] = "0025" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + +_FK = "private_channel_members_user_id_fkey" + + +def _resolve_user_id_fk(conn) -> str | None: + """The actual name of the (single-column) FK on private_channel_members.user_id, + or None if there isn't one.""" + return conn.execute( + sa.text( + """ + SELECT con.conname + FROM pg_constraint con + JOIN pg_attribute att + ON att.attrelid = con.conrelid AND att.attnum = ANY(con.conkey) + WHERE con.conrelid = 'private_channel_members'::regclass + AND con.contype = 'f' + AND att.attname = 'user_id' + AND cardinality(con.conkey) = 1 + """ + ) + ).scalar_one_or_none() + + +def upgrade() -> None: + conn = op.get_bind() + fk_name = _resolve_user_id_fk(conn) + if fk_name is None: + raise RuntimeError( + "No single-column foreign key found on private_channel_members.user_id " + f"(expected {_FK!r} by default). Cannot add the CASCADE behaviour without " + "knowing which constraint to drop and recreate — verify the column's " + "actual FK name on this database before re-running this migration." + ) + op.drop_constraint(fk_name, "private_channel_members", type_="foreignkey") + op.create_foreign_key( + _FK, "private_channel_members", "users", ["user_id"], ["id"], ondelete="CASCADE" + ) + + +def downgrade() -> None: + conn = op.get_bind() + fk_name = _resolve_user_id_fk(conn) or _FK + op.drop_constraint(fk_name, "private_channel_members", type_="foreignkey", if_exists=True) + op.create_foreign_key( + _FK, "private_channel_members", "users", ["user_id"], ["id"], ondelete="SET NULL" + ) diff --git a/alembic/versions/0027_fk_and_badge_indexes.py b/alembic/versions/0027_fk_and_badge_indexes.py new file mode 100644 index 00000000..5d51341c --- /dev/null +++ b/alembic/versions/0027_fk_and_badge_indexes.py @@ -0,0 +1,135 @@ +"""FK-target indexes and thread_decisions badge-count composites + +Revision ID: 0027 +Revises: 0026 + +Eighteen FK columns carry an ondelete rule but no index on the referencing +side (measured by cross-checking every model Index/UniqueConstraint/PK +against every alembic create_index/create_unique_constraint/create_table +constraint across alembic/versions/); every user-delete, +agent-delete or thread_decision-delete that cascades through one of these +walks a full table scan on the child instead of an index lookup. +thread_decisions also gets two composite indexes on (agent_a, outcome) and +(agent_b, outcome) — AgentBadgeMiddleware (src/main.py) runs one COUNT per +side per agent per request, and on a 300k-row synthetic thread_decisions table +the composite pair (a BitmapOr over both) measurably speeds up that lookup; +treat the exact magnitude as workload-dependent. + +Hand-written, not autogenerated: the ORM/alembic drift already present in +this schema (ten FK columns are indexed only by alembic DDL with no matching +model Index/UniqueConstraint) would make `alembic revision --autogenerate` +try to drop those pre-existing indexes. + +Every op.create_index/op.drop_index call below is written out literally, one +name per call (NOT looped over a tuple): +tests/unit/test_migration_checks.py's PLANNED_OBJECTS drift guard finds +index names with the regex `create_index\\(\\s*\\n?\\s*"([^"]+)"`, which only +matches a literal string in the call's first-argument position. A loop like +`for name, table, cols in _INDEXES: op.create_index(name, table, cols)` +makes the guard find zero names and pass vacuously — silently disabling the +one check that would catch a future PLANNED_OBJECTS/migration-file drift +here (verified: sabotaging 5 of the 20 PLANNED_OBJECTS entries with the loop +form still passes the drift guard; with the literal calls below it correctly +fails). + +Downgrade is idempotent (if_exists) per the branch convention (0022+). +""" + +from typing import Sequence, Union + +from alembic import op + +revision: str = "0027" +down_revision: Union[str, None] = "0026" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.create_index("ix_access_allowlist_added_by_user_id", "access_allowlist", ["added_by_user_id"]) + op.create_index("ix_agent_delegates_user_id", "agent_delegates", ["user_id"]) + op.create_index("ix_agent_delegates_invitation_id", "agent_delegates", ["invitation_id"]) + op.create_index("ix_agents_approved_by", "agents", ["approved_by"]) + op.create_index("ix_cohort_audit_events_actor_id", "cohort_audit_events", ["actor_id"]) + op.create_index("ix_cohort_memberships_added_by", "cohort_memberships", ["added_by"]) + op.create_index("ix_cohorts_created_by", "cohorts", ["created_by"]) + op.create_index( + "ix_delegate_invitations_invited_by_user_id", "delegate_invitations", ["invited_by_user_id"] + ) + op.create_index( + "ix_delegate_invitations_accepted_by_user_id", "delegate_invitations", ["accepted_by_user_id"] + ) + op.create_index( + "ix_email_notifications_thread_decision_id", "email_notifications", ["thread_decision_id"] + ) + op.create_index( + "ix_email_notifications_agent_registry_id", "email_notifications", ["agent_registry_id"] + ) + op.create_index("ix_private_channel_members_user_id", "private_channel_members", ["user_id"]) + op.create_index( + "ix_private_channel_members_added_by_user_id", + "private_channel_members", + ["added_by_user_id"], + ) + op.create_index( + "ix_profile_revisions_changed_by_user_id", "profile_revisions", ["changed_by_user_id"] + ) + op.create_index("ix_proposal_reviews_user_id", "proposal_reviews", ["user_id"]) + op.create_index("ix_proposal_reviews_delegate_user_id", "proposal_reviews", ["delegate_user_id"]) + op.create_index( + "ix_proposal_reviews_reviewed_by_user_id", "proposal_reviews", ["reviewed_by_user_id"] + ) + op.create_index( + "ix_slack_app_provisions_agent_registry_id", "slack_app_provisions", ["agent_registry_id"] + ) + op.create_index("ix_thread_decisions_agent_a_outcome", "thread_decisions", ["agent_a", "outcome"]) + op.create_index("ix_thread_decisions_agent_b_outcome", "thread_decisions", ["agent_b", "outcome"]) + + +def downgrade() -> None: + op.drop_index("ix_thread_decisions_agent_b_outcome", table_name="thread_decisions", if_exists=True) + op.drop_index("ix_thread_decisions_agent_a_outcome", table_name="thread_decisions", if_exists=True) + op.drop_index( + "ix_slack_app_provisions_agent_registry_id", + table_name="slack_app_provisions", + if_exists=True, + ) + op.drop_index( + "ix_proposal_reviews_reviewed_by_user_id", table_name="proposal_reviews", if_exists=True + ) + op.drop_index("ix_proposal_reviews_delegate_user_id", table_name="proposal_reviews", if_exists=True) + op.drop_index("ix_proposal_reviews_user_id", table_name="proposal_reviews", if_exists=True) + op.drop_index( + "ix_profile_revisions_changed_by_user_id", table_name="profile_revisions", if_exists=True + ) + op.drop_index( + "ix_private_channel_members_added_by_user_id", + table_name="private_channel_members", + if_exists=True, + ) + op.drop_index( + "ix_private_channel_members_user_id", table_name="private_channel_members", if_exists=True + ) + op.drop_index( + "ix_email_notifications_agent_registry_id", table_name="email_notifications", if_exists=True + ) + op.drop_index( + "ix_email_notifications_thread_decision_id", table_name="email_notifications", if_exists=True + ) + op.drop_index( + "ix_delegate_invitations_accepted_by_user_id", + table_name="delegate_invitations", + if_exists=True, + ) + op.drop_index( + "ix_delegate_invitations_invited_by_user_id", + table_name="delegate_invitations", + if_exists=True, + ) + op.drop_index("ix_cohorts_created_by", table_name="cohorts", if_exists=True) + op.drop_index("ix_cohort_memberships_added_by", table_name="cohort_memberships", if_exists=True) + op.drop_index("ix_cohort_audit_events_actor_id", table_name="cohort_audit_events", if_exists=True) + op.drop_index("ix_agents_approved_by", table_name="agents", if_exists=True) + op.drop_index("ix_agent_delegates_invitation_id", table_name="agent_delegates", if_exists=True) + op.drop_index("ix_agent_delegates_user_id", table_name="agent_delegates", if_exists=True) + op.drop_index("ix_access_allowlist_added_by_user_id", table_name="access_allowlist", if_exists=True) diff --git a/alembic/versions/0028_thread_reopen_state.py b/alembic/versions/0028_thread_reopen_state.py new file mode 100644 index 00000000..1d61b249 --- /dev/null +++ b/alembic/versions/0028_thread_reopen_state.py @@ -0,0 +1,52 @@ +"""Add reopened_at to thread_decisions (durable reopen/dedup state) + +Revision ID: 0028 +Revises: 0027 + +_sync_proposal_reviews_from_db reopens a thread whenever a PI submits a +rating=0 (reopen-with-guidance) review, and _reopen_thread reopens one when a +PI tags a closed thread directly (Slack or DB-native). Neither wrote anything +durable: on every restart, _rebuild_agent_state re-closed every thread with a +ThreadDecision row (having no way to tell a reopened one from a still-closed +one), and the only thing that put a reopened thread's ThreadState back was +_sync_proposal_reviews_from_db's own re-processing of the same rating=0 review +row on the next tick — which also re-minted a brand-new synthetic PI-guidance +AgentMessage row and a fresh reply-count budget, forever, once per restart. + +reopened_at lets the rebuild recognise a reopened-but-not-yet-reclosed thread +(via its LATEST ThreadDecision row), keep it out of _closed_thread_ids, and +restore its reply budget (message_count_offset) and PI guidance (pi_context) +in the same active-threads reconstruction every other still-open thread +already gets. It also lets the web-guidance reopen flow in +_sync_proposal_reviews_from_db recognise, via the message log content the +rebuild already reloaded, that a prior process already minted this thread's +synthetic PI-guidance row — so every process restores the ThreadState, but +no process re-mints the row. + +Nullable, no backfill: NULL means "never reopened", which is the correct +value for every existing row. + +Downgrade is idempotent (if_exists) per the 0022+ convention. +""" + +from typing import Sequence, Union + +import sqlalchemy as sa + +from alembic import op + +revision: str = "0028" +down_revision: Union[str, None] = "0027" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + "thread_decisions", + sa.Column("reopened_at", sa.DateTime(timezone=True), nullable=True), + ) + + +def downgrade() -> None: + op.drop_column("thread_decisions", "reopened_at", if_exists=True) diff --git a/alembic/versions/0029_pi_engagement_and_inbound_state.py b/alembic/versions/0029_pi_engagement_and_inbound_state.py new file mode 100644 index 00000000..5fa4f001 --- /dev/null +++ b/alembic/versions/0029_pi_engagement_and_inbound_state.py @@ -0,0 +1,96 @@ +"""Add thread_decisions.pi_engaged_at and agent_messages.pi_inbound_state + +Revision ID: 0029 +Revises: 0028 + +Two nullable markers. Both are nullable with no server default and no backfill: +NULL means "unknown", and every reader must treat unknown as the pre-0029 +behaviour. + +thread_decisions.pi_engaged_at +------------------------------- +_check_pi_proposal_review clears an agent's pending-proposal block when the +owning PI engages with the thread, but _persist_implicit_proposal_review can +only record that durably as a ProposalReview row, and proposal_reviews.user_id +is NOT NULL — so an agent whose AgentRegistry.user_id is NULL (a bulk-provisioned +or self-deleted-account agent) gets an in-memory-only clear, and +_rebuild_agent_state re-blocks the proposal on the next restart. + +Making ProposalReview.user_id (src/models/agent_registry.py) nullable instead +was rejected: it is ForeignKey("users.id", ondelete="CASCADE"), so deleting a +PI would cascade-delete the engine's own block-clearing markers. A timestamp +on the thread's own decision row has no CASCADE exposure to users at all +(thread_decisions cascades from simulation_runs only). + +NULL means "no PI engagement recorded", which is the correct value for every +existing row: the rebuild's re-block is exactly what those rows have always +produced. + +agent_messages.pi_inbound_state +--------------------------------- +The DB inbound poller (_poll_inbound_from_db) is moving its MessageLog append +AHEAD of _handle_pi_inbound_entry, so a handler failure can no longer lose the +PI's text. But the log entry's presence WAS the dedup key, so once the append +runs first the retry is skipped and the side effects are lost instead. Dedup +therefore has to read a marker that is distinct from the log entry. + +Three states, because two are not enough. A plain "handled_at" timestamp cannot +tell "no poller has claimed this row" from "the poller claimed it and its +handler failed", and the difference is load-bearing: _poll_channels appends a +Slack-origin PI message to the log itself and applies the same side effects +inline, then persists it as an agent_messages row that _poll_inbound_from_db +re-reads (it filters on simulation_run_id and created_at only). Under a +two-valued marker that row reads as unhandled and gets its side effects applied +a second time, including the @bot tag route — a duplicate reply on every tagged +Slack message. + + NULL unknown: no DB inbound poller has claimed this row. Every + pre-0029 row, every bot/agent-authored row, and every row some + other path put in the log. Readers fall back to today's + behaviour (dedup on MessageLog presence). + 'ingested' the poller appended this row's text to the log and has NOT yet + confirmed _handle_pi_inbound_entry succeeded. Re-run it. + 'handled' _handle_pi_inbound_entry returned without raising. Skip, and + advance the cursor. + +Written only by SimulationEngine._poll_inbound_from_db, only for is_bot=false +rows. Deliberately a plain String, not an enum type: an enum would need its own +CREATE TYPE (and a DROP TYPE in the downgrade) for a value set only one function +writes, and 0020's pi_dm_direction_enum is the only precedent for spending that. + +Sizing: both are ADD COLUMN ... NULL with no default, which Postgres 11+ applies +as a catalogue-only change (no table rewrite, no per-row work) under a brief +ACCESS EXCLUSIVE lock, regardless of agent_messages' size. + +Unlike 0026 there is no constraint whose name could have drifted, so no +pg_constraint resolution is needed here; the equivalent defence is the +if_exists=-guarded downgrade (the branch convention since 0022), so a +half-applied or hand-repaired database cannot abort the downgrade partway. +""" + +from typing import Sequence, Union + +import sqlalchemy as sa + +from alembic import op + +revision: str = "0029" +down_revision: Union[str, None] = "0028" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + "thread_decisions", + sa.Column("pi_engaged_at", sa.DateTime(timezone=True), nullable=True), + ) + op.add_column( + "agent_messages", + sa.Column("pi_inbound_state", sa.String(length=16), nullable=True), + ) + + +def downgrade() -> None: + op.drop_column("agent_messages", "pi_inbound_state", if_exists=True) + op.drop_column("thread_decisions", "pi_engaged_at", if_exists=True) diff --git a/alembic/versions/0030_pi_ownership_and_dm_handled.py b/alembic/versions/0030_pi_ownership_and_dm_handled.py new file mode 100644 index 00000000..e7ae198e --- /dev/null +++ b/alembic/versions/0030_pi_ownership_and_dm_handled.py @@ -0,0 +1,77 @@ +"""Add agent_messages.sender_user_id and pi_dm_messages.handled_at + +Revision ID: 0030 +Revises: 0029 + +Two columns, fixing two ownership gaps: + +``agent_messages.sender_user_id`` records who actually wrote a row (NULL for bot +rows and pre-migration rows, which cannot be recovered), so the engine can gate +PI-only side effects (clearing a pending-proposal block, setting ``pi_context``, +the @bot tag route) on the actual writer's owned agents instead of on thread +membership — previously any PI sharing a thread with another PI's agent could +trigger that agent's side effects. ``ON DELETE SET NULL``: deleting a PI's +account must not delete the historical record of what they said. + +``pi_dm_messages.handled_at`` is a durable, timestamp-based "has this inbound DM +been processed" marker, replacing three in-memory/cursor mechanisms that all +reset on restart and could each drop a PI message sent while ``agent-run`` was +down. Backfilled to ``created_at`` for existing inbound rows (already handled, +unrecoverable otherwise) so this migration does not replay the DM channel's +entire history; outbound rows are left NULL (unused). + +Sizing: the two ADD COLUMNs are catalogue-only. The FK and index on +``agent_messages.sender_user_id`` each scan/lock the full table once; both are +sub-second at current table sizes. If ``agent_messages`` grows very large, +split these into a NOT VALID FK + separate VALIDATE CONSTRAINT and a +CONCURRENTLY index build instead. + +Downgrade is idempotent (if_exists) per the 0022+ convention. +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from sqlalchemy.dialects import postgresql + +from alembic import op + +revision: str = "0030" +down_revision: Union[str, None] = "0029" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + +_SENDER_USER_ID_FK = "agent_messages_sender_user_id_fkey" + + +def upgrade() -> None: + op.add_column( + "agent_messages", + sa.Column("sender_user_id", postgresql.UUID(as_uuid=True), nullable=True), + ) + op.create_foreign_key( + _SENDER_USER_ID_FK, + "agent_messages", "users", ["sender_user_id"], ["id"], ondelete="SET NULL", + ) + op.create_index( + "ix_agent_messages_sender_user_id", "agent_messages", ["sender_user_id"] + ) + op.add_column( + "pi_dm_messages", + sa.Column("handled_at", sa.DateTime(timezone=True), nullable=True), + ) + op.execute( + "UPDATE pi_dm_messages SET handled_at = created_at " + "WHERE direction = 'inbound' AND handled_at IS NULL" + ) + + +def downgrade() -> None: + op.drop_column("pi_dm_messages", "handled_at", if_exists=True) + op.drop_index( + "ix_agent_messages_sender_user_id", table_name="agent_messages", if_exists=True + ) + op.drop_constraint( + _SENDER_USER_ID_FK, "agent_messages", type_="foreignkey", if_exists=True + ) + op.drop_column("agent_messages", "sender_user_id", if_exists=True) diff --git a/docker-compose.override.yml b/docker-compose.override.yml index 93632354..7960b50a 100644 --- a/docker-compose.override.yml +++ b/docker-compose.override.yml @@ -1,34 +1,62 @@ # Logging override — the EC2 instance role (copi-ec2-ses-role) lacks # logs:CreateLogStream, so the awslogs driver in docker-compose.prod.yml makes # every container fail to start with an AccessDeniedException. This forces the -# local json-file driver instead. -# -# Originally added 2026-05-26 when the instance role was swapped; that copy was -# never committed and was later deleted from the working tree, which is how the -# stack ended up running from the dev docker-compose.yml (no restart policy) and -# failed to come back after the 2026-08-06 host freeze. +# local json-file driver instead. This file MUST be passed alongside +# docker-compose.prod.yml on every prod compose invocation (see CLAUDE.md +# "Compose file set") — running prod off docker-compose.prod.yml alone +# recreates every service with the broken awslogs driver. # # Remove this file once the admin adds CloudWatch Logs perms to the role # (or restores the previous role with the SES policy attached). +# +# json-file with no options grows unbounded on disk -- max-size/max-file caps +# each service's log at 5 x 50m = 250 MiB. services: postgres: logging: driver: json-file + options: + max-size: "50m" + max-file: "5" app: logging: driver: json-file + options: + max-size: "50m" + max-file: "5" worker: logging: driver: json-file + options: + max-size: "50m" + max-file: "5" agent: logging: driver: json-file + options: + max-size: "50m" + max-file: "5" grantbot: logging: driver: json-file + options: + max-size: "50m" + max-file: "5" nginx: logging: driver: json-file + options: + max-size: "50m" + max-file: "5" certbot: logging: driver: json-file + options: + max-size: "50m" + max-file: "5" + migrate: + logging: + driver: json-file + options: + max-size: "50m" + max-file: "5" diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml index d2d514cb..3b605ddc 100644 --- a/docker-compose.prod.yml +++ b/docker-compose.prod.yml @@ -2,6 +2,10 @@ services: postgres: image: postgres:15 restart: unless-stopped + # Decision D24: no mem_limit/cpus here. A cgroup mem cap on a ~2.3 GB + # database also caps its page cache, and an OOM-killed backend restarts + # the whole cluster -- worse than the failure mode caps elsewhere guard + # against. environment: POSTGRES_USER: ${POSTGRES_USER:-copi} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env} @@ -22,10 +26,39 @@ services: awslogs-create-group: "true" awslogs-region: ${AWS_REGION:-us-east-2} + migrate: + build: + context: . + restart: "no" + mem_limit: 256m + cpus: 0.5 + command: ["python", "-m", "alembic", "upgrade", "head"] + env_file: .env + environment: + DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-copi}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-copi} + SECRET_KEY: ${SECRET_KEY:?Set a strong random SECRET_KEY in .env} + ENVIRONMENT: ${ENVIRONMENT:-production} + # Same knob run_migration.sh already reads (alembic/env.py:66) — a + # routine `alembic upgrade head` here should never need to wait long, + # but give it the same override surface as the gated runbook. + ALEMBIC_LOCK_TIMEOUT_MS: ${ALEMBIC_LOCK_TIMEOUT_MS:-10000} + depends_on: + postgres: + condition: service_healthy + logging: + driver: awslogs + options: + awslogs-group: /copi/migrate + tag: migrate + awslogs-create-group: "true" + awslogs-region: ${AWS_REGION:-us-east-2} + app: build: context: . restart: unless-stopped + mem_limit: 384m + cpus: 1.0 command: ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] expose: - "8000" @@ -42,6 +75,8 @@ services: depends_on: postgres: condition: service_healthy + migrate: + condition: service_completed_successfully healthcheck: test: ["CMD-SHELL", "python -c \"import urllib.request; urllib.request.urlopen(\\\"http://127.0.0.1:8000/api/health\\\")\""] interval: 30s @@ -60,6 +95,8 @@ services: build: context: . restart: unless-stopped + mem_limit: 512m + cpus: 0.5 command: ["python", "-m", "src.worker.main"] env_file: .env environment: @@ -70,9 +107,16 @@ services: ENVIRONMENT: ${ENVIRONMENT:-production} volumes: - ./profiles:/app/profiles + # No ./prompts bind mount here: prompts/ is baked into the image by the + # Dockerfile's `COPY . .`, and prompt text must never be edited on the + # host without a rebuild, so a bind mount would silently shadow the + # image's prompts with stale host-side text. A few other services do + # bind-mount prompts/ as a stated residual, not a pattern to extend. depends_on: postgres: condition: service_healthy + migrate: + condition: service_completed_successfully logging: driver: awslogs options: @@ -84,6 +128,18 @@ services: agent: build: context: . + # 768m is a measured ceiling with headroom over the agent process's real + # working set (peak usage well under half this limit across a full + # multi-agent sweep and hundreds of turns, with no growth over time — the + # turn loop does not leak). NOT raised (the host also runs the blackbird + # stack) and NOT left uncapped the way postgres is: uncapping moves the + # choice of OOM victim to the kernel, which on this host can pick + # postgres -- the durable store that is deliberately left uncapped. + mem_limit: 768m + cpus: 1.0 + # An OOM kill is SIGKILL and skips the shutdown flush that persists the + # in-flight turn; the runbook's `docker stop -t 30` relies on this. + stop_grace_period: 30s command: ["python", "-m", "src.agent.main"] env_file: .env environment: @@ -113,6 +169,8 @@ services: build: context: . restart: unless-stopped + mem_limit: 256m + cpus: 0.5 command: ["python", "-m", "src.agent.grantbot", "scheduler", "--run-hour", "8", "--max-per-channel", "1"] env_file: .env environment: @@ -128,6 +186,8 @@ services: depends_on: postgres: condition: service_healthy + migrate: + condition: service_completed_successfully logging: driver: awslogs options: @@ -139,6 +199,22 @@ services: nginx: image: nginx:1.27-alpine restart: unless-stopped + # Memory budget. This comment deliberately does not restate nginx.conf's + # shared-memory total as a number — that goes stale the moment a zone is + # added, split or resized. Instead it states only the terms nginx.conf + # cannot know, and tests/unit/test_nginx_config.py sums the zones from + # nginx.conf itself and checks the whole budget against mem_limit: + # nginx-mem-budget: zones + 48 workers x 1.0 MiB + 16 MiB headroom + # 48 is the worker count the image's stock `worker_processes auto` picks + # on the prod host (this is not ours to pin -- the mounted file is a + # conf.d snippet and worker_processes is a main-context directive). 1.0 + # MiB per worker is the measured RSS for master + workers idle, rounded + # up for proxy/request buffers under load; the 16 MiB flat headroom + # covers TLS handshake scratch and page-cache churn. 64m OOMs under load. + # 1.0 cpus so a hard CFS quota doesn't throttle TLS termination for all + # three vhosts even on an idle host. + mem_limit: 128m + cpus: 1.0 ports: - "80:80" - "443:443" @@ -180,6 +256,11 @@ services: certbot: image: certbot/certbot:latest restart: unless-stopped + # certbot's import graph alone exceeds 32 MiB; PID 1 is a shell loop, so + # the OOM killer takes the python child silently and renewals stop while + # the container still reports "running". + mem_limit: 128m + cpus: 0.1 volumes: - ./certbot/conf:/etc/letsencrypt - ./certbot/www:/var/www/certbot diff --git a/docker-compose.yml b/docker-compose.yml index cb89974c..9a65b2e8 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -17,6 +17,8 @@ services: app: build: . + # dev only: the prod image runs as UID 10001 (Dockerfile); the dev bind-mount is owned by the host user, so run as root here + user: "0:0" command: uvicorn src.main:app --host 0.0.0.0 --port 8000 --reload ports: - "8001:8000" @@ -31,6 +33,8 @@ services: worker: build: . + # dev only: the prod image runs as UID 10001 (Dockerfile); the dev bind-mount is owned by the host user, so run as root here + user: "0:0" command: python -m src.worker.main env_file: .env volumes: @@ -45,6 +49,8 @@ services: agent: build: . + # dev only: the prod image runs as UID 10001 (Dockerfile); the dev bind-mount is owned by the host user, so run as root here + user: "0:0" command: python -m src.agent.main # On SIGTERM the engine finishes the current turn and flushes buffered # messages to Postgres (the durable store). The 10s default can cut that @@ -64,6 +70,8 @@ services: grantbot: build: . + # dev only: the prod image runs as UID 10001 (Dockerfile); the dev bind-mount is owned by the host user, so run as root here + user: "0:0" command: python -m src.agent.grantbot scheduler --run-hour 8 --max-per-channel 1 env_file: .env volumes: diff --git a/docs/inbound-email.md b/docs/inbound-email.md index f06108fd..a98deb7b 100644 --- a/docs/inbound-email.md +++ b/docs/inbound-email.md @@ -81,7 +81,37 @@ Then, in this order: `_amazonses` TXT verification record if the domain was newly verified. 2. Attach the printed S3 policy to `copi-ec2-ses-role`. 3. Re-run `--check` until all layers are OK. -4. Set `ENABLE_INBOUND_EMAIL=true` in the prod `.env` and recreate BOTH the +4. **Prerequisite (land before this step, not after):** the worker must already claim its own + canonical-id writer slot (`WRITER_WORKER` in `src/agent/ids.py`, claimed in + `src/worker/main.py:main()`) — otherwise it mints PI-reply messages in the web app's residue + class and a same-microsecond collision silently drops one of the two messages (the + `uq_agent_messages_run_ts` conflict handler's resolution; unrecoverable now that the DB is the + only durable store). Landed as part of #21 V11 — if this line still says "not yet landed" when + you read it, stop and land it first. + + **Prerequisite (also land before this step):** the poller must survive a poison object and an + odd-but-legitimate reply, because every failure here happens to a real PI's mail and there is no + second copy. Four hardening items, all landed as part of #21 V3 — verify each is present before + you flip the flag: + - a malformed or unprocessable object is quarantined to `failed/` after + `MAX_S3_PROCESS_ATTEMPTS` (3) consecutive failures (`src/services/email_inbound.py:33-34`) + instead of being retried on every poll forever; + - an unknown MIME charset (`charset=unknown-8bit` and friends) falls back instead of raising + `LookupError` out of the decode (`_decode_part`, `:490-499`) — otherwise that PI's reply is + quarantined and lost rather than read; + - the LLM's `rating` is coerced to `int` and `bool` is rejected (`_coerce_rating`, `:563`), so a + string rating cannot raise `TypeError` past the range guard; + - the S3 listing is paginated with a bounded page count (`:193-195`, cap 20 pages = 1,000 + objects per poll), so a backlog cannot starve the tail of the bucket behind the first 50 keys. + + Two behaviours to expect once it is on, neither of which is a bug: a review or instruction is + committed **before** the SES confirmation is sent, so a send failure never rolls back work the + PI already did; and a failure *inside* the private-channel migration is terminal — the PI is + e-mailed and told to use the dashboard rather than the object being retried. See deploy note 21 + in `docs/plans/2026-09-02-close-issues-20-27.md` for the one case that still retries (a full + Postgres outage) and how to spot the orphan `priv-…` channels it can leave. + + Set `ENABLE_INBOUND_EMAIL=true` in the prod `.env` and recreate BOTH the worker (polling + proposal/reminder emails) and the app (the welcome email reads the same flag for its reply-vs-dashboard copy — recreating only the worker leaves new signups being told the dashboard is the only way in): diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/ci_run_2026-09-02.log b/docs/plans/2026-09-02-close-issues-20-27-evidence/ci_run_2026-09-02.log new file mode 100644 index 00000000..fb8baecb --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/ci_run_2026-09-02.log @@ -0,0 +1,275 @@ +==> alembic (single head, no duplicate revision ids) + single head: 0024 (head) +==> alembic round trip against a throwaway postgres:15 on 127.0.0.1:55433 + throwaway postgres ready +INFO [alembic.runtime.migration] Context impl PostgresqlImpl. +INFO [alembic.runtime.migration] Will assume transactional DDL. +INFO [alembic.runtime.migration] Running upgrade -> 0001, Initial schema +INFO [alembic.runtime.migration] Running upgrade 0001 -> 0002, Add llm_call_logs table +INFO [alembic.runtime.migration] Running upgrade 0002 -> 0003, Add thread_decisions table, thread_ts and flexible phase to agent_messages, channel to llm_call_logs +INFO [alembic.runtime.migration] Running upgrade 0003 -> 0004, Add agents registry and proposal_reviews tables +INFO [alembic.runtime.migration] Running upgrade 0004 -> 0005, Add private_profile_md and private_profile_seed columns +INFO [alembic.runtime.migration] Running upgrade 0005 -> 0006, Add delegate_slack_ids column to agents +INFO [alembic.runtime.migration] Running upgrade 0006 -> 0007, Add web delegate tables and proposal review audit column +INFO [alembic.runtime.migration] Running upgrade 0007 -> 0008, Add email notification tables and columns +INFO [alembic.runtime.migration] Running upgrade 0008 -> 0009, Add profile_revisions table +INFO [alembic.runtime.migration] Running upgrade 0009 -> 0010, Access gate + waitlist +INFO [alembic.runtime.migration] Running upgrade 0010 -> 0011, Channel visibility + private channel members +INFO [alembic.runtime.migration] Running upgrade 0011 -> 0012, Grantbot posted FOAs table (replaces data/grantbot_posted.json) +INFO [alembic.runtime.migration] Running upgrade 0012 -> 0013, Drop agent_registry.slack_app_token (Socket Mode never used) +INFO [alembic.runtime.migration] Running upgrade 0013 -> 0014, Add last_login_at to users +INFO [alembic.runtime.migration] Running upgrade 0014 -> 0015, Add proposal_votes (public no-login votes on graph proposals) +INFO [alembic.runtime.migration] Running upgrade 0015 -> 0016, Add notification categories: email_notification_preferences + category column +INFO [alembic.runtime.migration] Running upgrade 0016 -> 0017, Add app_settings (KV) + slack_app_provisions tables for self-service provisioning +INFO [alembic.runtime.migration] Running upgrade 0017 -> 0018, Add email hint column to access_allowlist (fallback for private ORCID emails) +INFO [alembic.runtime.migration] Running upgrade 0018 -> 0019, Add conversation-content columns to agent_messages (DB becomes primary store) +INFO [alembic.runtime.migration] Running upgrade 0019 -> 0020, Add pi_dm_messages table (durable PI<->bot direct messages) +INFO [alembic.runtime.migration] Running upgrade 0020 -> 0021, Index the DB inbox pollers' created_at cursor +INFO [alembic.runtime.migration] Running upgrade 0021 -> 0022, Add cohorts, cohort_memberships and cohort_audit_events +INFO [alembic.runtime.migration] Running upgrade 0022 -> 0023, Add synthesis-provenance columns to researcher_profiles +INFO [alembic.runtime.migration] Running upgrade 0023 -> 0024, Add role column to agents (per-role agent customization) +INFO [alembic.runtime.migration] Context impl PostgresqlImpl. +INFO [alembic.runtime.migration] Will assume transactional DDL. +INFO [alembic.runtime.migration] Running downgrade 0024 -> 0023, Add role column to agents (per-role agent customization) +INFO [alembic.runtime.migration] Running downgrade 0023 -> 0022, Add synthesis-provenance columns to researcher_profiles +INFO [alembic.runtime.migration] Running downgrade 0022 -> 0021, Add cohorts, cohort_memberships and cohort_audit_events +INFO [alembic.runtime.migration] Running downgrade 0021 -> 0020, Index the DB inbox pollers' created_at cursor +INFO [alembic.runtime.migration] Running downgrade 0020 -> 0019, Add pi_dm_messages table (durable PI<->bot direct messages) +INFO [alembic.runtime.migration] Running downgrade 0019 -> 0018, Add conversation-content columns to agent_messages (DB becomes primary store) +INFO [alembic.runtime.migration] Context impl PostgresqlImpl. +INFO [alembic.runtime.migration] Will assume transactional DDL. +INFO [alembic.runtime.migration] Running upgrade 0018 -> 0019, Add conversation-content columns to agent_messages (DB becomes primary store) +INFO [alembic.runtime.migration] Running upgrade 0019 -> 0020, Add pi_dm_messages table (durable PI<->bot direct messages) +INFO [alembic.runtime.migration] Running upgrade 0020 -> 0021, Index the DB inbox pollers' created_at cursor +INFO [alembic.runtime.migration] Running upgrade 0021 -> 0022, Add cohorts, cohort_memberships and cohort_audit_events +INFO [alembic.runtime.migration] Running upgrade 0022 -> 0023, Add synthesis-provenance columns to researcher_profiles +INFO [alembic.runtime.migration] Running upgrade 0023 -> 0024, Add role column to agents (per-role agent customization) + round trip clean (upgrade head -> downgrade 0018 -> upgrade head) + throwaway postgres destroyed +==> ruff (test-suite lint) +All checks passed! +==> ruff (src/ ratchet, ceiling 260) + 254 findings (ceiling 260) +==> pytest (full suite + branch coverage, fail-under=60%) +============================= test session starts ============================== +platform linux -- Python 3.12.3, pytest-9.1.1, pluggy-1.6.0 +rootdir: /home/a/scripps/coPI.science +configfile: pyproject.toml +plugins: Faker-40.31.0, syrupy-5.5.3, respx-0.23.1, anyio-4.14.2, cov-7.1.0, asyncio-1.4.0 +asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function +collected 2150 items + +tests/characterization/test_agent_turn_gm.py ............. [ 0%] +tests/characterization/test_auth_and_admin_routes.py ................... [ 1%] +....... [ 1%] +tests/characterization/test_profile_pipeline_gm.py ........... [ 2%] +tests/characterization/test_public_routes.py ..................... [ 3%] +tests/contract/test_grants_contract.py .......... [ 3%] +tests/contract/test_orcid_contract.py ............ [ 4%] +tests/contract/test_pubmed_contract.py ............ [ 4%] +tests/e2e/test_browser_flows.py ..sssssss [ 5%] +tests/integration/test_admin_users.py ............. [ 5%] +tests/integration/test_agent_page.py ................................... [ 7%] +........................................................ [ 10%] +tests/integration/test_cli.py ..................... [ 11%] +tests/integration/test_cohort_admin.py ................................. [ 12%] +..................... [ 13%] +tests/integration/test_cohort_engine_live.py ........................... [ 14%] +.................................................. [ 17%] +tests/integration/test_cohort_real_llm.py sssss [ 17%] +tests/integration/test_cohort_scenarios.py sssss [ 17%] +tests/integration/test_cohort_seed_apply.py ........... [ 18%] +tests/integration/test_conversation_feed.py ............................ [ 19%] +...... [ 19%] +tests/integration/test_db_contract.py ...................... [ 20%] +tests/integration/test_email_inbound_reply_paths.py ............ [ 21%] +tests/integration/test_factories_smoke.py ...... [ 21%] +tests/integration/test_full_run_live.py ssss [ 21%] +tests/integration/test_grantbot_live.py ssssssssssss [ 22%] +tests/integration/test_harness_smoke.py ... [ 22%] +tests/integration/test_health_route.py . [ 22%] +tests/integration/test_message_persistence.py ................... [ 23%] +tests/integration/test_onboarding_flow.py .............................. [ 24%] +..................................... [ 26%] +tests/integration/test_pi_inbox.py .... [ 26%] +tests/integration/test_profile_pipeline_live.py sssss [ 26%] +tests/integration/test_proposal_review.py ................. [ 27%] +tests/integration/test_public_graph.py ................................. [ 29%] + [ 29%] +tests/integration/test_role_live_flip.py . [ 29%] +tests/integration/test_seed_cohorts_script.py ........ [ 29%] +tests/integration/test_slack_client_live.py ssssssssssssssssssssssss [ 30%] +tests/integration/test_slack_cohort_live.py sssss [ 31%] +tests/integration/test_slack_lifecycle_live.py sssssssssss [ 31%] +tests/integration/test_slack_mirror_live.py sssssss [ 31%] +tests/integration/test_slack_pi_live.py ssss [ 32%] +tests/integration/test_slack_private_live.py sss [ 32%] +tests/integration/test_slack_provision_live.py sss [ 32%] +tests/integration/test_state_rebuild.py ...... [ 32%] +tests/integration/test_worker.py ............... [ 33%] +tests/live_api/test_grants_live.py sssssss [ 33%] +tests/live_api/test_marker.py ss [ 33%] +tests/live_api/test_orcid_live.py ssssss [ 34%] +tests/live_api/test_pubmed_live.py ssssssssss [ 34%] +tests/unit/test_admin_provisioning.py ... [ 34%] +tests/unit/test_agent_prompts.py .......... [ 35%] +tests/unit/test_authorship_emit_gate.py ................ [ 35%] +tests/unit/test_authorship_grounding_db.py . [ 35%] +tests/unit/test_authorship_rules.py .................................... [ 37%] +...................... [ 38%] +tests/unit/test_backfill_publications.py ..... [ 38%] +tests/unit/test_backup_checks.py ....................................... [ 40%] +........................................................................ [ 44%] +....... [ 44%] +tests/unit/test_cohort_isolation.py .................................... [ 46%] +........................................................................ [ 49%] +................. [ 50%] +tests/unit/test_cohort_seed.py ....................................... [ 51%] +tests/unit/test_config_secret_redaction.py .................. [ 52%] +tests/unit/test_delegates.py ............ [ 53%] +tests/unit/test_doi_validation.py ................. [ 54%] +tests/unit/test_email_inbound_hardening.py ............... [ 54%] +tests/unit/test_email_inbound_llm_pin.py . [ 54%] +tests/unit/test_email_inbound_security.py ......... [ 55%] +tests/unit/test_email_reply_solicitation.py ...... [ 55%] +tests/unit/test_email_templates.py ........ [ 55%] +tests/unit/test_fakes.py ....... [ 56%] +tests/unit/test_funding_rules.py ................................ [ 57%] +tests/unit/test_grantbot_lead_time.py ........ [ 58%] +tests/unit/test_hub_budget_scheduler.py ................................ [ 59%] +................ [ 60%] +tests/unit/test_ids.py ............ [ 60%] +tests/unit/test_invite_email_binding.py ...... [ 61%] +tests/unit/test_lab_directory_ordering.py ....... [ 61%] +tests/unit/test_llm_service.py ....... [ 61%] +tests/unit/test_login_redirect.py ....................... [ 62%] +tests/unit/test_memory_authorship_guard.py ... [ 63%] +tests/unit/test_message_log.py ......................................... [ 64%] +.. [ 65%] +tests/unit/test_migration_checks.py .................................... [ 66%] +........................................................................ [ 70%] +............................................... [ 72%] +tests/unit/test_model_tiering.py ... [ 72%] +tests/unit/test_open_redirect.py ........... [ 72%] +tests/unit/test_own_authored_papers.py ............ [ 73%] +tests/unit/test_post_types.py .................................... [ 75%] +tests/unit/test_privacy_scoping.py ...................... [ 76%] +tests/unit/test_private_channel_migration.py ......................... [ 77%] +tests/unit/test_profile_versioning.py ...... [ 77%] +tests/unit/test_prompt_safety.py ..... [ 77%] +tests/unit/test_rate_limit.py ....... [ 78%] +tests/unit/test_reachability.py ....................... [ 79%] +tests/unit/test_remediate_duplicates.py ................................ [ 80%] +........................................................................ [ 84%] +............................. [ 85%] +tests/unit/test_retrieve_tools_authors.py .......... [ 85%] +tests/unit/test_roles.py ................ [ 86%] +tests/unit/test_roster_sync.py ............... [ 87%] +tests/unit/test_service_bot_attribution.py ........................ [ 88%] +tests/unit/test_simulation_logic.py .................................... [ 90%] +................................... [ 91%] +tests/unit/test_slack_boundary.py .. [ 91%] +tests/unit/test_slack_client_contract.py ............................... [ 93%] +................................. [ 94%] +tests/unit/test_slack_provisioning.py ........... [ 95%] +tests/unit/test_slack_tokens.py ............................. [ 96%] +tests/unit/test_slack_web.py ............. [ 97%] +tests/unit/test_sweep_authorship_memories.py ........ [ 97%] +tests/unit/test_thread_guidance.py ........... [ 98%] +tests/unit/test_thread_not_found.py ......... [ 98%] +tests/unit/test_tool_gating.py .. [ 98%] +tests/unit/test_transport.py ................. [ 99%] +tests/unit/test_unsubscribe.py .... [ 99%] +tests/unit/test_validators.py ....... [100%] + +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.12.3-final-0 ________________ + +Name Stmts Miss Branch BrPart Cover Missing +---------------------------------------------------------------------------------- +src/__init__.py 0 0 0 0 100.00% +src/agent/__init__.py 0 0 0 0 100.00% +src/agent/agent.py 265 38 74 7 85.55% 313, 568, 570, 585, 617-643, 654->670, 674-679, 720-721, 727-728, 735-741, 745-765 +src/agent/authorship_rules.py 124 3 56 4 96.11% 145, 199->201, 266, 333 +src/agent/channels.py 33 12 4 0 56.76% 35-39, 44, 74-84 +src/agent/foa_cache.py 59 46 20 0 16.46% 26-31, 36-43, 52-78, 83-84, 89-101 +src/agent/funding_rules.py 122 9 66 14 87.77% 116, 120, 124, 161, 164, 166, 191->194, 204, 209->222, 217, 219, 230->234, 234->239, 239->244 +src/agent/grantbot.py 368 313 132 1 12.40% 54-76, 81-96, 107-131, 167-181, 212-237, 242-243, 260-267, 272-275, 286-346, 357-423, 439-459, 472-486, 496-701, 709-713, 718-720, 731-745, 761-786, 790 +src/agent/ids.py 41 0 4 0 100.00% +src/agent/main.py 132 116 34 1 10.24% 54-55, 67-298, 306 +src/agent/message_log.py 177 2 82 3 98.07% 312, 316, 381->384 +src/agent/pi_handler.py 204 105 66 12 42.59% 52-70, 91-93, 101, 144-172, 187-193, 201, 241-246, 253-260, 267-276, 283, 302-346, 360-388, 400-421, 428-431, 433->443, 435->443, 440->435 +src/agent/post_types.py 80 2 34 2 96.49% 178, 223 +src/agent/prompt_safety.py 5 0 0 0 100.00% +src/agent/roles.py 57 0 14 0 100.00% +src/agent/simulation.py 2219 740 1022 155 62.94% 88-91, 441, 444-445, 635, 735-743, 751-752, 760, 770, 774->786, 788, 815-816, 960, 982-1035, 1043-1059, 1108-1109, 1116-1121, 1134-1135, 1139, 1143-1168, 1192, 1195, 1197, 1204-1208, 1211->1189, 1215-1217, 1237-1267, 1287, 1293, 1305->1309, 1309->1285, 1319-1320, 1353-1358, 1362-1367, 1379-1386, 1410, 1435-1446, 1451-1471, 1499, 1513-1514, 1530->1555, 1531->1530, 1557-1562, 1596-1606, 1615->1632, 1629-1630, 1639-1644, 1654->exit, 1678, 1684, 1692, 1723, 1725-1749, 1757, 1778-1786, 1841, 1858-1862, 1868-1911, 1950, 1957, 1969, 1977->1967, 2014, 2045-2046, 2064-2065, 2071-2118, 2130-2131, 2148-2157, 2163-2166, 2185-2186, 2227-2228, 2233-2234, 2242-2247, 2249-2254, 2257-2258, 2278-2283, 2289-2307, 2311, 2331-2464, 2578->2575, 2632-2633, 2645->2649, 2668-2672, 2680, 2691, 2695, 2700, 2715-2719, 2736, 2774->2776, 2776->2778, 2781-2840, 2853, 2872-2874, 2915-2933, 2938-2939, 2947-2951, 2958-3003, 3013-3047, 3058, 3081->3080, 3083-3084, 3095, 3115-3117, 3125, 3126->3128, 3131-3132, 3137->exit, 3149-3260, 3356-3367, 3410-3411, 3488-3489, 3506-3509, 3511, 3516-3517, 3539-3574, 3584, 3600, 3610->exit, 3613-3614, 3632->3619, 3664, 3670-3674, 3686-3687, 3714-3716, 3725, 3751->3720, 3753->3720, 3771, 3781-3783, 3797, 3813-3815, 3818, 3842, 3853, 3878, 3932->3940, 4052-4056, 4066->4072, 4073-4075, 4098->4100, 4100->4104, 4104->4108, 4112-4113, 4120, 4138->4140, 4140->4115, 4158->4190, 4187-4188, 4208, 4216, 4220, 4225->4229, 4230, 4246->4326, 4275, 4281-4284, 4291, 4322-4323, 4326->4353, 4341->4339, 4343-4344, 4353->4396, 4390->4388, 4392-4393, 4397-4399, 4423-4426, 4434-4441, 4445-4446, 4452-4475, 4601-4604, 4633->4635, 4650-4651, 4675-4677, 4785->4787, 4794-4795, 4799->4797, 4861->4860, 4882->4865, 4973-4974, 4982-5154, 5225, 5237, 5239, 5372-5395, 5409->5412, 5425->5429, 5462-5464, 5473-5474, 5478->5483, 5481-5482, 5484-5490, 5496-5497 +src/agent/slack_client.py 426 108 120 15 73.44% 43-46, 437-456, 464, 494-495, 528, 546-550, 563, 576-581, 587, 600, 616-618, 631, 638-643, 648-649, 657-664, 668-675, 709, 723->731, 726-727, 737-742, 825-836, 840-844, 853-858, 896-899, 949, 968-985, 993-1004, 1036, 1076-1077, 1092, 1098 +src/agent/state.py 50 0 0 0 100.00% +src/agent/thread_guidance.py 15 0 4 0 100.00% +src/agent/tools.py 114 61 56 6 40.59% 116-148, 153-190, 195-200, 207, 238, 247, 258->260, 265-267 +src/agent/transport.py 53 2 0 0 96.23% 122, 125 +src/cli.py 174 1 36 1 99.05% 301 +src/config.py 231 2 20 1 98.80% 438, 455 +src/database.py 29 17 4 0 36.36% 16-17, 31-33, 38-44, 49-58 +src/dependencies.py 66 9 24 4 85.56% 30, 32, 53-55, 66-67, 103->111, 108-109 +src/main.py 71 14 10 2 77.78% 56-59, 79-95, 137-138 +src/models/__init__.py 15 0 0 0 100.00% +src/models/access.py 27 2 0 0 92.59% 34, 55 +src/models/agent_activity.py 125 8 0 0 93.60% 68, 134, 172, 206, 240, 301-302, 348 +src/models/agent_registry.py 42 2 0 0 95.24% 63, 111 +src/models/cohort.py 50 3 0 0 94.00% 61, 99, 154 +src/models/delegate.py 36 0 0 0 100.00% +src/models/email_notification.py 47 3 0 0 93.62% 67, 95, 138 +src/models/grantbot_posted.py 12 1 0 0 91.67% 28 +src/models/job.py 22 1 0 0 95.45% 44 +src/models/profile.py 42 1 6 0 97.92% 122 +src/models/profile_revision.py 21 0 0 0 100.00% +src/models/proposal_vote.py 24 1 0 0 95.83% 67 +src/models/provisioning.py 24 2 0 0 91.67% 35, 67 +src/models/publication.py 23 1 0 0 95.65% 41 +src/models/user.py 31 1 0 0 96.77% 68 +src/routers/__init__.py 0 0 0 0 100.00% +src/routers/admin.py 665 263 230 37 57.32% 120-127, 131, 133, 137, 144-147, 182-198, 221-232, 244-262, 284-309, 339, 358->360, 420, 425, 427, 429, 529-532, 533->536, 537, 559->561, 570->572, 579->581, 584, 588->590, 596, 613-620, 622, 639, 641, 646->648, 652, 657, 664-680, 710, 712, 714-715, 725-761, 793-842, 875, 879->883, 920-940, 951-961, 973-988, 1002-1020, 1032-1042, 1063, 1090->1122, 1115-1117, 1141-1143, 1158-1181, 1204-1225, 1236-1244, 1256-1290, 1301-1308, 1323-1328, 1346-1374, 1388-1395, 1677 +src/routers/agent_page.py 559 94 170 47 78.74% 78, 82, 89, 98-100, 146, 173-181, 300-303, 314-319, 476, 483, 569, 615, 642-650, 669, 672, 684-688, 698->712, 744, 825->837, 851, 894, 899, 972, 984, 1018-1024, 1049, 1052, 1055, 1076, 1101, 1127, 1138->1143, 1176, 1209, 1245, 1253-1254, 1329, 1336, 1361-1380, 1399, 1411, 1418, 1424->1428, 1431-1437, 1469, 1490-1491, 1503-1504, 1570->1574, 1599->1623, 1602-1614 +src/routers/auth.py 146 66 54 3 51.50% 122, 131, 168-300 +src/routers/invite.py 101 39 34 9 55.56% 59-116, 149-151, 159, 164, 180-184, 199-211, 232->254, 240->254, 242->254, 245-250 +src/routers/onboarding.py 137 2 38 2 97.71% 179-188, 222->227 +src/routers/profile.py 101 2 22 5 94.31% 127->141, 144->146, 146->148, 148->152, 157-158 +src/routers/public.py 367 72 124 22 75.56% 325, 355, 358-366, 372-375, 390-392, 394->387, 404-413, 416-422, 438-441, 458-459, 463-464, 474, 489, 522-524, 570-593, 683-690, 767, 855, 860, 1022, 1056->1066, 1069, 1085-1099, 1114 +src/routers/settings.py 81 0 22 0 100.00% +src/services/__init__.py 0 0 0 0 100.00% +src/services/admin_provisioning.py 102 54 22 2 46.77% 91-92, 100-101, 111-114, 121, 130-187, 201-233 +src/services/cohort_seed.py 93 2 30 1 97.56% 82-83 +src/services/cohorts.py 53 0 20 0 100.00% +src/services/conversation_feed.py 25 0 4 0 100.00% +src/services/email.py 111 24 14 2 77.60% 94-95, 109-110, 167-169, 380-381, 398-419 +src/services/email_inbound.py 344 101 108 20 69.25% 171, 179-184, 208-212, 215, 226, 238-239, 242-245, 253-254, 257-258, 265-269, 276-277, 291-297, 302-303, 311-312, 323, 339-353, 437, 440, 443, 513-514, 517-519, 545-546, 593-735, 826-827, 853-855 +src/services/email_notifications.py 428 205 132 15 47.50% 102, 107, 110-126, 137->141, 150, 208-209, 237, 249-250, 267-274, 293->297, 321-326, 369-376, 493-495, 502-525, 534-582, 594-595, 637-640, 653->656, 664-666, 671-686, 691-696, 706-727, 734-887, 897-918, 925-981, 1070->1075 +src/services/grants.py 87 9 30 3 82.91% 25->28, 58->33, 162-170, 188->191 +src/services/llm.py 209 95 52 6 50.57% 28-29, 42-43, 69-74, 90-91, 129-169, 247->249, 271-272, 285-287, 372-400, 411-412, 451-522, 526 +src/services/orcid.py 90 0 34 7 94.35% 41->40, 52->51, 54->52, 64->68, 93->91, 123->127, 133->128 +src/services/pi_inbox.py 35 1 6 1 95.12% 86 +src/services/private_channels.py 174 25 42 9 82.41% 157->160, 185-189, 194-197, 213, 302, 317, 428, 462, 479, 481, 529-533, 544-557 +src/services/profile_export.py 109 9 60 9 89.35% 38->44, 44->51, 84->109, 89->91, 91->93, 93->95, 104, 121-123, 145-147, 194, 210 +src/services/profile_pipeline.py 268 58 100 15 75.82% 73, 84, 89-91, 97-99, 121->120, 128-155, 157->165, 190, 216, 231->187, 251-252, 257, 264-282, 328-329, 489-497, 540->535, 545-548, 571-572, 576-577 +src/services/profile_versioning.py 21 4 2 1 78.26% 88-92, 129-138 +src/services/pubmed.py 258 79 118 10 65.69% 88, 144->143, 147->149, 179->190, 191->200, 202->201, 230-231, 288->283, 290-291, 307->297, 309-310, 320-339, 348-360, 365-401, 406-413, 431-435, 439, 463-482 +src/services/rate_limit.py 43 1 16 2 94.92% 64, 80->76 +src/services/slack_provisioning.py 43 11 14 1 71.93% 46-54, 66-74, 150 +src/services/slack_tokens.py 37 0 16 0 100.00% +src/services/slack_web.py 114 16 36 7 83.33% 94->121, 107->118, 110-111, 160, 173, 177, 187, 198-199, 204-209, 287, 292 +src/services/validators.py 13 0 4 0 100.00% +src/visibility.py 3 0 0 0 100.00% +src/worker/__init__.py 0 0 0 0 100.00% +src/worker/main.py 114 30 28 4 71.83% 31-32, 60, 142-159, 163-172, 179-181, 185 +---------------------------------------------------------------------------------- +TOTAL 10587 2888 3470 468 69.27% +Required test coverage of 60% reached. Total coverage: 69.27% +--------------------------- snapshot report summary ---------------------------- +20 snapshots passed. +================ 2030 passed, 120 skipped in 378.58s (0:06:18) ================= +==> reclaiming leaked testcontainers volumes + reclaimed leaked test volume f08af95b67c3 + done +==> CI passed. +EXIT=0 diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/conventions_dossier.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/conventions_dossier.md new file mode 100644 index 00000000..73b3b7e3 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/conventions_dossier.md @@ -0,0 +1,1507 @@ +# Conventions dossier — coPI.science @ copi-prod 18ba52c + +Collected read-only on 2026-09-02. Every claim cites `file:line` in `/home/a/scripps/coPI.science`. +Line numbers are from `cat -n` / `sed -n` at this commit. + +--- + +## A. Alembic + +### A.1 alembic.ini + env.py — how the URL is obtained, async vs sync + +- `alembic.ini:1-5`: `script_location = alembic`, `prepend_sys_path = .`, and a hard-coded fallback + `sqlalchemy.url = postgresql+asyncpg://copi:copi@localhost:5432/copi`. +- `alembic/env.py:20-23` overrides that URL from the environment: + ```python + db_url = os.environ.get("DATABASE_URL") + if db_url: + config.set_main_option("sqlalchemy.url", db_url) + ``` + `DATABASE_URL` is therefore the only knob tests/CI/prod use (`tests/conftest.py:65`, `scripts/ci.sh:229-231`). +- **Async.** `alembic/env.py:8` imports `async_engine_from_config`; `run_async_migrations()` + (`env.py:99-111`) builds an async engine with `poolclass=pool.NullPool` and runs + `await connection.run_sync(do_run_migrations)`; `run_migrations_online()` (`env.py:114-115`) + does `asyncio.run(run_async_migrations())`. +- `target_metadata = Base.metadata` (`env.py:29`), populated by `import src.models # noqa: F401` + (`env.py:14-15`). +- **One transaction for the whole chain.** `do_run_migrations` (`env.py:93-96`) calls + `context.configure(connection=connection, target_metadata=target_metadata)` with NO + `transaction_per_migration`; the header comment at `env.py:53-72` explains this is deliberate. +- **lock_timeout is a connect-time server setting**, not a `SET` statement: + `LOCK_TIMEOUT_MS = os.environ.get("ALEMBIC_LOCK_TIMEOUT_MS", "10000")` (`env.py:73`), applied as + `kwargs["connect_args"] = {"server_settings": {"lock_timeout": str(int(LOCK_TIMEOUT_MS))}}` + (`env.py:102-107`). The comment at `env.py:76-92` records the failure mode of doing it as a SQL + statement before `begin_transaction()` (whole chain silently rolls back, no `alembic_version`). + +### A.2 Naming convention for constraints/indexes + +**None.** `src/database.py:11-12` is: +```python +class Base(DeclarativeBase): + pass +``` +`grep -rn "naming_convention\|MetaData(" src alembic --include='*.py'` returns nothing (exit 1). +Constraint/index names are hand-written in each migration. Observed naming style: +- unique constraints: `uq_
_` — `uq_agent_messages_run_ts` (`0019:53`), + `uq_cohort_membership_cohort_agent` (`0022:78`) +- indexes: `ix__` — `ix_agent_messages_run_posted` (`0019:56`), + `ix_pi_dm_run_agent_posted` (`0020:53`), `ix_cohort_memberships_cohort_id` (`0022:82`) +- enums: `_enum` — `pi_dm_direction_enum` (`0020:39`) +- CHECK constraints: `pcm_exactly_one_of_agent_or_user` (asserted at + `tests/integration/test_db_contract.py:281`) + +### A.3 Engine construction (for reference to the app side) + +`src/database.py:15-22`: +```python +def _get_engine(): + settings = get_settings() + return create_async_engine( + settings.database_url, + echo=False, + pool_size=5, + max_overflow=10, + ) +``` +Lazy singletons `get_engine()` / `get_session_factory()` (`src/database.py:29-43`), the factory +is `async_sessionmaker(get_engine(), class_=AsyncSession, expire_on_commit=False)`. + +### A.4 Revision chain 0018 -> 0024 (single linear chain, no branches) + +| rev | down | file | date in header | +|---|---|---|---| +| 0018 | 0017 | `0018_allowlist_email_hint.py` (:15-16) | 2026-06-29 | +| 0019 | 0018 | `0019_agent_message_content.py` (:19-20) | 2026-07-20 | +| 0020 | 0019 | `0020_pi_dm_messages.py` (:19-20) | 2026-07-20 | +| 0021 | 0020 | `0021_inbox_cursor_created_at_indexes.py` (:20-21) | 2026-07-25 | +| 0022 | 0021 | `0022_add_cohorts.py` (:25-26) | 2026-07-30 | +| 0023 | 0022 | `0023_profile_synthesis_provenance.py` (:38-39) | 2026-07-31 | +| 0024 | 0023 | `0024_add_agent_role.py` (:21-22) | 2026-08-05 | + +Current head: **0024**. Revision ids are 4-digit zero-padded strings; the filename is +`NNNN_snake_description.py`. Rule from `0022:7-12`: "Revision ids are assigned at merge, never at branch." + +### A.5 Head-revision pins a new migration (0025) MUST update + +These hard-code "0024" and will fail or mislead if a 0025 lands without touching them: + +- `tests/integration/test_harness_smoke.py:7-15`: + ```python + async def test_container_is_migrated(engine): + async with engine.connect() as conn: + v = (await conn.execute(text("SELECT version_num FROM alembic_version"))).scalar_one() + # Head-revision pin: bump it deliberately with each new migration. ... + # 0019-0021 db-primary-conversations, 0022 cohorts, + # 0023 researcher_profiles synthesis provenance, 0024 agents.role column + assert v == "0024" + ``` +- `scripts/migrate/preflight.py:74` `DEFAULT_TARGET = "0024"`; `:206` + `REVISION_ORDER = ("0018", "0019", "0020", "0021", "0022", "0023", "0024")`; `:202-204` + `PlannedObject("0024", "column", "role", "agents"),` closes the `PLANNED_OBJECTS` tuple. +- `scripts/migrate/run_migration.sh:56` `TARGET="0024"`. +- `tests/unit/test_migration_checks.py:232` `assert pf.DEFAULT_TARGET == "0024"`; `:1129` and + `:1170` `assert args.target == "0024"`; `:838` the drift guard loops + `for revision in ("0019", "0020", "0021", "0022", "0023", "0024"):` and requires exactly one + file per revision and every created index/table/column/constraint/enum in that file to be + declared in `PLANNED_OBJECTS` (`:826-857`). Add 0025 to that tuple and to `PLANNED_OBJECTS`. +- `tests/unit/test_migration_checks.py:218-226` parametrize `revision_status(rev, "0023")` with + "0024" expected to BLOCK; `:231` `SUPPORTED_START_REVISIONS == ("0018","0019","0020","0021","0023")`. +- `tests/unit/test_cohort_isolation.py:1296-1310` `test_head_is_the_expected_revision` only asserts + `len(revs) >= 22` and `revs["0022"] == ["0022_add_cohorts.py"]` — a 0025 does NOT break it. +- `scripts/ci.sh:70` `MIGRATION_FLOOR="${MIGRATION_FLOOR:-0018}"` — the round trip downgrades to + 0018 and back, so a new migration's `downgrade()` IS exercised by the gate against an empty DB. + +### A.6 Migration file template (most recent three, pasted in full) + +**`alembic/versions/0022_add_cohorts.py`** (149 lines): +```python +"""Add cohorts, cohort_memberships and cohort_audit_events + +Revision ID: 0022 +Revises: 0021 +Create Date: 2026-07-30 00:00:00.000000 + +Renumbered from 0019 at merge time. The cohort branch was cut before main's +db-primary work, so its original "0019" collided with 0019_agent_message_content: +two revisions sharing an id resolve to whichever file sorts last, which silently +skips the other while stamping the DB as fully migrated. Revision ids are assigned +at merge, never at branch. See .notes/cohort-system-v2.md §4.2 / §14 and the +alembic guard in scripts/ci.sh. + +Downgrades are idempotent (if_exists) so a rollback cannot wedge on an object that +a partially-applied upgrade never created. See v2 §14.4. +""" + +from typing import Sequence, Union + +import sqlalchemy as sa +from sqlalchemy.dialects.postgresql import UUID + +from alembic import op + +revision: str = "0022" +down_revision: Union[str, None] = "0021" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + # A cohort is a named group of agents permitted to act on each other's + # activity during simulation. See .notes/cohort-system-v2.md. + op.create_table( + "cohorts", + sa.Column("id", UUID(as_uuid=True), primary_key=True), + sa.Column("name", sa.String(length=48), nullable=False, unique=True), + sa.Column("description", sa.Text(), nullable=True), + sa.Column( + "created_by", + UUID(as_uuid=True), + sa.ForeignKey("users.id", ondelete="SET NULL"), + nullable=True, + ), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + server_default=sa.func.now(), + nullable=False, + ), + ) + + # agent_id is the AgentRegistry slug (no FK — agent rows may not exist at + # membership-creation time; the app validates at add time). + op.create_table( + "cohort_memberships", + sa.Column("id", UUID(as_uuid=True), primary_key=True), + sa.Column( + "cohort_id", + UUID(as_uuid=True), + sa.ForeignKey("cohorts.id", ondelete="CASCADE"), + nullable=False, + ), + sa.Column("agent_id", sa.String(length=50), nullable=False), + sa.Column( + "added_by", + UUID(as_uuid=True), + sa.ForeignKey("users.id", ondelete="SET NULL"), + nullable=True, + ), + sa.Column( + "added_at", + sa.DateTime(timezone=True), + server_default=sa.func.now(), + nullable=False, + ), + sa.UniqueConstraint( + "cohort_id", "agent_id", name="uq_cohort_membership_cohort_agent" + ), + ) + op.create_index( + "ix_cohort_memberships_cohort_id", "cohort_memberships", ["cohort_id"] + ) + op.create_index( + "ix_cohort_memberships_agent_id", "cohort_memberships", ["agent_id"] + ) + + # Append-only audit trail. Deliberately denormalised: a cohort delete cascades + # its memberships away and a user delete nulls the actor FK, so the trail must + # not depend on either row surviving — hence cohort_name / actor_email columns + # and NO FK on cohort_id. `topology` snapshots the full cohort->members map + # plus the active gate settings at run start and on every change, so a + # completed simulation run stays attributable to the configuration that + # produced it (v2 §13.1). + op.create_table( + "cohort_audit_events", + sa.Column("id", UUID(as_uuid=True), primary_key=True), + sa.Column("cohort_id", UUID(as_uuid=True), nullable=True), + sa.Column("cohort_name", sa.String(length=48), nullable=False), + sa.Column("agent_id", sa.String(length=50), nullable=True), + sa.Column("action", sa.String(length=32), nullable=False), + sa.Column( + "actor_id", + UUID(as_uuid=True), + sa.ForeignKey("users.id", ondelete="SET NULL"), + nullable=True, + ), + sa.Column("actor_email", sa.String(length=255), nullable=True), + sa.Column("simulation_run_id", UUID(as_uuid=True), nullable=True), + sa.Column("topology", sa.JSON(), nullable=True), + sa.Column( + "created_at", + sa.DateTime(timezone=True), + server_default=sa.func.now(), + nullable=False, + ), + ) + op.create_index( + "ix_cohort_audit_events_cohort_id", "cohort_audit_events", ["cohort_id"] + ) + op.create_index( + "ix_cohort_audit_events_created_at", "cohort_audit_events", ["created_at"] + ) + + +def downgrade() -> None: + op.drop_index( + "ix_cohort_audit_events_created_at", + table_name="cohort_audit_events", + if_exists=True, + ) + op.drop_index( + "ix_cohort_audit_events_cohort_id", + table_name="cohort_audit_events", + if_exists=True, + ) + op.drop_table("cohort_audit_events", if_exists=True) + op.drop_index( + "ix_cohort_memberships_agent_id", + table_name="cohort_memberships", + if_exists=True, + ) + op.drop_index( + "ix_cohort_memberships_cohort_id", + table_name="cohort_memberships", + if_exists=True, + ) + op.drop_table("cohort_memberships", if_exists=True) + op.drop_table("cohorts", if_exists=True) +``` + +**`alembic/versions/0023_profile_synthesis_provenance.py`** (62 lines): +```python +"""Add synthesis-provenance columns to researcher_profiles + +Revision ID: 0023 +Revises: 0022 +Create Date: 2026-07-31 00:00:00.000000 + +Two defects in src/services/profile_pipeline.py were invisible because the +pipeline wrote down nothing about *how* a profile was produced: + + 1. Step 8 computed the validation result and step 9 stored on `if synthesized:` + alone, so a profile that failed _validate_profile twice was persisted as if + it had passed. `synthesis_validated` is the record of that decision. + 2. With PubMed unreachable, ORCID works never reach the synthesis prompt + (_build_synthesis_context is fed only pubs_for_synthesis, which is derived + solely from PubMed records), so the model invents a plausible profile from + ~150 characters of name/department context and zero Publication rows are + written. `evidence_pmid_count` / `evidence_pub_count` make that case + self-identifying and separate it from a genuinely publication-less + researcher (see ResearcherProfile.evidence_state). + +All three are nullable and are deliberately NOT backfilled. NULL means "unknown +— this row predates the columns". Backfilling evidence_pub_count from +count(publications) would look like a free win and would be a lie: stored +publications accumulate across runs and include records with no abstract and +non-research article types, none of which reached any prompt. Inventing +provenance is exactly the failure these columns exist to expose. + +Downgrades are idempotent (if_exists) so a rollback cannot wedge on a column a +partially-applied upgrade never created (see scripts/ci.sh and the 0022 note). +""" + +from typing import Sequence, Union + +import sqlalchemy as sa + +from alembic import op + +revision: str = "0023" +down_revision: Union[str, None] = "0022" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + "researcher_profiles", + sa.Column("synthesis_validated", sa.Boolean(), nullable=True), + ) + op.add_column( + "researcher_profiles", + sa.Column("evidence_pmid_count", sa.Integer(), nullable=True), + ) + op.add_column( + "researcher_profiles", + sa.Column("evidence_pub_count", sa.Integer(), nullable=True), + ) + + +def downgrade() -> None: + op.drop_column("researcher_profiles", "evidence_pub_count", if_exists=True) + op.drop_column("researcher_profiles", "evidence_pmid_count", if_exists=True) + op.drop_column("researcher_profiles", "synthesis_validated", if_exists=True) +``` + +**`alembic/versions/0024_add_agent_role.py`** (35 lines): +```python +"""Add role column to agents (per-role agent customization) + +Revision ID: 0024 +Revises: 0023 +Create Date: 2026-08-05 00:00:00.000000 + +`role` selects per-role prompt overrides (prompts/roles/{role}/) and a per-role +tool allow-list. Default 'pi_lab' == the pre-existing all-agents-identical +behaviour, so this column is a no-op until an agent is explicitly reassigned. +See docs/specs/2026-08-05-hub-bot-customization-design.md. + +Downgrade is idempotent (if_exists) per the branch convention (0022/0023). +""" + +from typing import Sequence, Union + +import sqlalchemy as sa + +from alembic import op + +revision: str = "0024" +down_revision: Union[str, None] = "0023" +branch_labels: Union[str, Sequence[str], None] = None +depends_on: Union[str, Sequence[str], None] = None + + +def upgrade() -> None: + op.add_column( + "agents", + sa.Column("role", sa.String(length=20), nullable=False, server_default="pi_lab"), + ) + + +def downgrade() -> None: + op.drop_column("agents", "role", if_exists=True) +``` + +Template summary: module docstring = one-line title, blank, `Revision ID:` / `Revises:` / +`Create Date: YYYY-MM-DD 00:00:00.000000`, blank, then a long **why** paragraph (0021-0024 all do +this; the docstring is where the rationale lives, not inline). Imports: `from typing import +Sequence, Union`, `import sqlalchemy as sa`, `from alembic import op` (postgres types via +`from sqlalchemy.dialects.postgresql import UUID` or `from sqlalchemy.dialects import postgresql`). +Typed module attrs `revision: str`, `down_revision: Union[str, None]`, `branch_labels`, `depends_on`. +`def upgrade() -> None:` / `def downgrade() -> None:`. Since 0022, every `op.drop_*` in +`downgrade()` carries `if_exists=True` — `tests/unit/test_cohort_isolation.py:1287-1294` pins this +for 0022 (`downgrade.count("if_exists=True") == len(drops)`); 0023 and 0024 follow by convention. +Note `scripts/ci.sh:62-69` records that 0019/0020/0021 *lack* the guards. + +### A.7 Data migrations — prior art for `op.execute` / row fixes + +Only three migrations touch rows; none since 0012. + +- **`alembic/versions/0010_access_gate_and_waitlist.py:42-50`** — add a NOT NULL column with a + server default, backfill existing rows, then drop the server default: + ```python + def upgrade() -> None: + # 1. Add access_status to users, default 'pending', backfill existing rows to 'allowed' + op.add_column( + "users", + sa.Column("access_status", sa.String(20), nullable=False, server_default="pending"), + ) + op.execute("UPDATE users SET access_status = 'allowed'") + # Drop the server default so new inserts rely on the model default + op.alter_column("users", "access_status", server_default=None) + ``` +- **`0010:89-103`** — seed rows with `sa.table(...)` + `op.bulk_insert(...)` from a module-level + `PILOT_ORCIDS` list. +- **`alembic/versions/0012_grantbot_posted_foas.py:37-53`** — parametrised INSERT through the bind: + ```python + if numbers: + bind = op.get_bind() + bind.execute( + sa.text( + "INSERT INTO grantbot_posted_foas (foa_number) " + "VALUES (:n) ON CONFLICT (foa_number) DO NOTHING" + ), + [{"n": n} for n in numbers if n], + ) + ``` +- All other `op.execute` uses are `DROP TYPE IF EXISTS ` in downgrades (`0001:269-274`, + `0003:66,103`); 0020 uses `sa.Enum(name=...).drop(op.get_bind(), checkfirst=True)` (`0020:66`). +- Recent precedent for a **one-shot data repair done OUTSIDE alembic**: commit `18ba52c` body says + the grantbot NULL-agent_id backlog "is repaired by a one-shot UPDATE at rollout (documented in + the start() comment)" — see `src/agent/simulation.py:570-575`. + +### A.8 Is `downgrade()` real in 0019-0024? + +Yes, all six are real inverses (none is `pass`): +- 0019 (`:73-85`): drops 3 indexes, the unique constraint, re-NOT-NULLs `agent_id`, drops 7 columns. No `if_exists`. +- 0020 (`:62-66`): drops 2 indexes, table, and the enum type with `checkfirst=True`. No `if_exists`. +- 0021 (`:37-39`): drops 2 indexes. No `if_exists`. +- 0022 (`:126-149`): drops 4 indexes + 3 tables, every call `if_exists=True`. +- 0023 (`:59-62`): drops 3 columns, `if_exists=True`. +- 0024 (`:34-35`): drops 1 column, `if_exists=True`. + +--- + +## B. Tests + +### B.1 `tests/conftest.py` — fixtures (351 lines) + +| fixture | scope | what it provides | lines | +|---|---|---|---| +| `_pg_container` | session | `None` if `TEST_DATABASE_URL` is set, else a `PostgresContainer("postgres:15", dbname="copi_test")` published on `127.0.0.1` only (`pg.ports["5432"] = ("127.0.0.1", None)`) | 28-45 | +| `pg_url` | session | asyncpg DSN — `TEST_DATABASE_URL` verbatim, else built from the container | 48-58 | +| `_migrated` | session | runs `/alembic upgrade head` as a subprocess with `DATABASE_URL=pg_url`, asserts rc==0; returns the URL. "NOT create_all — that omits migration-only indexes/constraints" (`:5-6`) | 61-74 | +| `engine` | session | `create_async_engine(_migrated, future=True, poolclass=NullPool)`; NullPool because pytest-asyncio uses a per-test loop | 77-85 | +| `db_session` | function (async) | `engine.connect()` + outer `conn.begin()`; `AsyncSession(bind=conn, expire_on_commit=False, join_transaction_mode="create_savepoint")` so route `commit()`s become savepoint releases; outer txn rolled back in `finally` | 88-103 | +| `client` | function (async) | `create_app()`; `app.dependency_overrides[get_db]` yields `db_session`; ALSO `monkeypatch.setattr("src.main.get_session_factory", lambda: badge_factory)` because `AgentBadgeMiddleware` bypasses `get_db`; `httpx.AsyncClient(transport=ASGITransport(app=app), base_url="http://testserver")` | 106-132 | +| `_text` | function | returns `sqlalchemy.text` | 135-137 | +| `pytest_collection_modifyitems` | hook | skips `live_slack` items unless `SLACK_TEST_WORKSPACE`, `SLACK_TEST_PI_USER_ID`, `SLACK_TEST_BOT_TOKEN_SU` are all set; skips `live_api` unless `LIVE_API_TESTS` | 144-166 | +| `slack_bot_tokens`, `slack_pi_user_id`, `slack_bot_tokens_all`, `slack_clients`, `slack_client_su`, `slack_list_all_channels`, `slack_probe_channel`, `api_budget` | session/function | live-tier only (real `AgentSlackClient` against a real workspace; `t-probe-*` channels archived on teardown; NCBI/ORCID rate budget) | 169-351 | + +**No settings-override fixture exists.** Tests that need settings monkeypatch `get_settings` or +module attributes directly (e.g. `tests/unit/test_roster_sync.py:23-25`, +`tests/unit/test_slack_web.py:112` `monkeypatch.setattr(slack_web, "_BACKOFF_BASE", 0)`). +**No fake-Slack fixture exists in conftest** — tests import from `tests/fakes.py` directly. +There are no per-directory `conftest.py` files (`ls tests/*/conftest.py` -> no matches). + +Exact fixture code for `db_session` and `client` (`tests/conftest.py:88-132`): +```python +@pytest_asyncio.fixture +async def db_session(engine): + """Function-scoped session whose writes (and route code's commits) roll back after the test.""" + async with engine.connect() as conn: + trans = await conn.begin() + session = AsyncSession( + bind=conn, + expire_on_commit=False, + join_transaction_mode="create_savepoint", # session.commit() -> savepoint release + ) + try: + yield session + finally: + await session.close() + if trans.is_active: + await trans.rollback() + + +@pytest_asyncio.fixture +async def client(db_session, engine, monkeypatch): + from src.database import get_db + from src.main import create_app + + badge_factory = async_sessionmaker(engine, expire_on_commit=False) + monkeypatch.setattr("src.main.get_session_factory", lambda: badge_factory) + + app = create_app() + + async def _override_get_db(): + yield db_session + + app.dependency_overrides[get_db] = _override_get_db + transport = ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://testserver") as c: + yield c + app.dependency_overrides.clear() +``` + +### B.2 `tests/fakes.py` (256 lines) — API surface + +- `FakeAnthropic(responses=None, *, default_text="OK")` (`:87-113`): `.messages.create(**kw)` + records to `.calls` and pops scripted responses (str | `_Message` | callable(kwargs)). Install + with `monkeypatch.setattr("src.services.llm.get_anthropic_client", lambda: fake)` (`:8`). + Helpers `text_response()`, `tool_use_response()`, `empty_response()` (`:57-75`). +- `FakeSlackClient(agent_id="agent1", bot_token="xoxb-fake")` (`:116-186`) — implements the + Transport protocol the engine uses: `connect()->True`, `is_connected`, `bot_user_id` (`U_`), + `post_message(channel, text, thread_ts=None)->{"ts","channel"}` (records to `.posted`, applies + `markdown_to_mrkdwn`), `send_dm(user_id, text)`, `poll_channel_messages(...)->[]`, + `get_thread_replies(...)->[]`, `create_channel(name)->{"id": f"C_{name}", ...}`, + `create_private_channel(name)->{"id": f"G_{name}", ...}`, `invite_to_channel(...)->True` + (records `.invites`), `list_channels(include_private=False)->{}`, `_resolve_channel_id(name)`. + Deterministic ts counter from `1_700_000_000` (`:130,143-145`). +- `RecordingSlackClient(responses=None, errors=None)` (`:189-222`) — stands in for the + `slack_sdk.WebClient` INSIDE `AgentSlackClient`; `__getattr__` records `(method_name, kwargs)` + into `.calls`; `errors[method]` is a list of exceptions popped one per call ("fail, then + succeed"); `.calls_to(method)`. +- `_SlackResponse(data)` (`:225-240`) with `.data/.headers/.status_code`, `get/__getitem__/__contains__`. +- `slack_error(code, *, retry_after=None)` (`:243-256`) — builds a `SlackApiError` whose + `response.headers["Retry-After"]` and `response.get("error")` are set the way + `_call_with_retry` reads them. + +### B.3 Directory layout and what distinguishes the tiers + +``` +tests/conftest.py tests/factories.py tests/fakes.py +tests/unit/ 57 files, pytestmark in 3 (no DB, no Docker; SimulationEngine(agents, slack_clients={})) +tests/integration/ 33 files, pytestmark in 32 (pytest.mark.integration; real Postgres via db_session/client/engine) +tests/characterization/ 4 files, pytestmark in 4 (pytest.mark.characterization; syrupy snapshots + client) +tests/contract/ 3 files, pytestmark in 3 (pytest.mark.contract; respx-mocked external HTTP) +tests/e2e/ 1 test file + helpers (httpx replays of browser flows; no marker) +tests/live_api/ 4 files (pytest.mark.live_api; skipped unless LIVE_API_TESTS=1) +``` +Markers are declared in `pyproject.toml:73-80`: +```toml +markers = [ + "integration: needs a real Postgres (testcontainers) + Docker", + "characterization: golden-master snapshot test", + "contract: respx-mocked external HTTP", + "real_llm: spends real Anthropic tokens; skipped unless ANTHROPIC_API_KEY is set", + "live_slack: hits a real Slack workspace; needs SLACK_TEST_WORKSPACE=1 plus bot tokens in the environment", + "live_api: calls a real third-party API (ORCID/NCBI/grants.gov); needs LIVE_API_TESTS=1", +] +``` +Marker placement is a module-level `pytestmark = pytest.mark.integration` line (e.g. +`tests/integration/test_agent_page.py:49`, `tests/integration/test_health_route.py:3`); live tiers +stack: `pytestmark = [pytest.mark.integration, pytest.mark.live_slack]`. Distinguisher in practice: +unit tests never take `db_session`/`client`/`engine`; integration tests do. Markers are +informational except `live_slack`/`live_api`, which are skip-gated by `conftest.py:148-166`. +Docstring at `tests/conftest.py:3-7`: "Integration/characterization/contract tests require a real +Postgres (the bugs worth pinning only reproduce on PG, not SQLite)." + +### B.4 Creating rows: `tests/factories.py` (async builders, not factory_boy) + +Docstring `tests/factories.py:3-8`: plain async helpers; each fills NOT NULL columns, uses a +process-wide `itertools.count` for unique columns, applies `**overrides` last, `session.add` + +`await session.flush()`, returns the instance. Available: `make_user`, `make_profile`, +`make_agent`, `make_simulation_run`, `make_agent_channel`, `make_agent_message`, +`make_thread_decision`, `make_private_channel_member`, `make_llm_call_log`; plus the +`SES_PASS_HEADER` constant (`:30-32`). + +**User** (`tests/factories.py:35-50`): +```python +async def make_user(session, **overrides) -> User: + n = next(_counter) + data = dict( + name=f"Researcher {n}", + orcid=f"0000-0000-0000-{n:04d}", + email=f"user{n}@example.edu", + institution="Test University", + is_admin=False, + onboarding_complete=True, + access_status="allowed", + ) + data.update(overrides) + obj = User(**data) + session.add(obj) + await session.flush() + return obj +``` +**AgentRegistry** (`:72-86`): defaults `agent_id=f"agent{n}"`, `bot_name=f"Agent{n}Bot"`, +`pi_name=f"Researcher {n}"`, `status="active"`, optional `user=` sets `user_id`. +**ThreadDecision** (`:137-154`): defaults `thread_id=f"{n}.000100"`, `channel=f"channel-{n}"`, +`agent_a="agent1"`, `agent_b="agent2"`, `outcome="proposal"`; creates a `SimulationRun` if none given. + +Concise in-test usage, User+Agent (`tests/integration/test_agent_page.py:192-197`): +```python +async def _agent_for(db, *, name, email, agent_id, bot_name, status="active"): + user = await factories.make_user(db, name=name, email=email) + agent = await factories.make_agent( + db, user=user, agent_id=agent_id, bot_name=bot_name, pi_name=name, status=status + ) + return user, agent +``` +ThreadDecision (`tests/characterization/test_public_routes.py:118-125`): +```python +async def test_proposal_vote_happy_path_returns_id(client, db_session): + d = await factories.make_thread_decision( + db_session, outcome="proposal", origin_visibility="public" + ) + body = {"decision_id": str(d.id), "vote": "up", "voter_token": "browser-tok-1"} + r = await client.post("/api/proposal-vote", json=body) + assert r.status_code == 200 + vote_id = r.json()["id"] +``` +Admin user: `return await factories.make_user(db_session, is_admin=True, email="admin@example.org")` +(`tests/integration/test_admin_users.py:59`). + +### B.5 How HTTP handlers are tested (httpx `AsyncClient` over ASGI + forged cookie) + +There is no login fixture. Each integration module defines a local `_auth(user_id)` that forges +the starlette `SessionMiddleware` cookie with `itsdangerous.TimestampSigner` +(`tests/integration/test_agent_page.py:52-56`; identical copies at `test_admin_users.py:36-39`, +`test_cohort_admin.py:24-27`, `tests/characterization/test_auth_and_admin_routes.py:23-31`; a +shared version exists at `tests/e2e/session.py:24-36` as `forge_session_cookie`/`auth_headers` +but the integration tests do not import it): +```python +def _auth(user_id) -> dict: + """Forge the signed session cookie SessionMiddleware would issue.""" + signer = TimestampSigner(get_settings().secret_key) + data = base64.b64encode(json.dumps({"user_id": str(user_id)}).encode()) + return {"Cookie": f"copi-session={signer.sign(data).decode()}"} +``` +Impersonation variant (`tests/integration/test_admin_users.py:42-52`) appends +`; copi-impersonate={impersonate_id}` to the Cookie header. + +Logged-in POST example (`tests/integration/test_agent_page.py:228-237`): +```python +async def _invite(client, world, email): + """Drive the real invite route; return the DelegateInvitation token.""" + r = await client.post( + f"/agent/{OWNER_AGENT}/delegates/invite", + data={"emails": email}, + headers=_auth(world.pi.id), + ) + assert r.status_code == 302, r.text + assert "delegate_error" not in r.headers["location"], r.headers["location"] + return r +``` +Another: `r = await client.post("/agent/request", headers=_auth(user.id))` (`:304`); form POST with +`data={"rating": "3", "comment": " solid "}` (`:606`); admin form POST +`client.post("/admin/impersonate", data={"orcid": f" {orcid} "}, headers=_auth(admin.id))` +(`test_admin_users.py:429`). Anonymous GET: `tests/integration/test_health_route.py:6-9` +(`r = await client.get("/api/health"); assert r.json() == {"status": "ok"}`). + +The auth dependency being exercised is `src/dependencies.py:36-39` +`async def get_current_user(request: Request, db: AsyncSession = Depends(get_db)) -> User`; it +reads `request.session.get("user_id")` (`:44`) and honours `copi-impersonate` for admins. + +### B.6 Constructing `SimulationEngine` in unit tests + +Constructor (`src/agent/simulation.py:240-255`): +```python +def __init__( + self, + agents: list[Agent], + slack_clients: dict, # agent_id -> AgentSlackClient + max_runtime_minutes: int = 60, + budget_cap: int = 0, + session_factory=None, + simulation_run_id: uuid.UUID | None = None, + reset_cursors: bool = False, + slack_enabled: bool = True, +): +``` +Minimal (`tests/unit/test_simulation_logic.py:165-167`): +```python +@pytest.fixture +def engine(self): + return SimulationEngine(agents=[], slack_clients={}) +``` +With agents and hermetic profiles (`tests/unit/test_authorship_emit_gate.py:17-32`): +```python +@pytest.fixture +def engine(tmp_path, monkeypatch): + # Deterministic empty profiles: no profile-parsed DOIs leak into either + # lab's "own" set regardless of what happens to be on disk in profiles/. + monkeypatch.setattr("src.agent.agent.PROFILES_DIR", tmp_path) + good = Agent(agent_id="good", bot_name="GoodBot", pi_name="Benjamin Good") + wu = Agent(agent_id="wu", bot_name="WuBot", pi_name="Chunlei Wu") + su = Agent(agent_id="su", bot_name="SuBot", pi_name="Andrew Su") + eng = SimulationEngine(agents=[good, wu, su], slack_clients={}) + eng._agent_publications = { + "wu": LabPublicationRecord(dois={DESIDERATA_DOI}, has_records=True), + "su": LabPublicationRecord(dois={DESIDERATA_DOI}, has_records=True), + } + return eng +``` +With a fake `session_factory` for DB-reading engine methods (`tests/unit/test_roster_sync.py:70-119`): +`_FakeDB` is an async context manager whose `execute()` returns `_FakeResult(rows)` with +`.all()`/`.scalar_one_or_none()`; `_factory_for(rows)` returns `lambda: _FakeDB(rows)`; +`_make_engine` passes `session_factory=_factory_for(active_rows)` and neuters side effects with +`engine._load_pi_mappings = AsyncMock()` and `engine._build_lab_directories = lambda: None`. +Slack client class swap: `monkeypatch.setattr("src.agent.slack_client.AgentSlackClient", _FakeSlackClient)` (`:122-123`). + +Integration tests that need the engine to read the rolled-back session use +`_FixtureSessionFactory(session)` — a callable whose `__aenter__` returns the fixture session and +whose `__aexit__` does NOT close it (`tests/integration/test_state_rebuild.py:56-75`, +`tests/integration/test_message_persistence.py:27-52`), passed as `session_factory=`. +Slack-off engines use `NullTransport(agent_id=...)` from `src/agent/transport.py:93,103`. + +### B.7 pytest / coverage config (`pyproject.toml`) + +```toml +[tool.pytest.ini_options] +asyncio_mode = "auto" # :72 (async tests need no decorator) +markers = [...] # :73-80 (see B.3) + +[tool.coverage.run] +branch = true # :83 +source = ["src"] # :84 +concurrency = ["thread", "greenlet"] # :98 — required or SQLAlchemy greenlet switches lose the tracer + +[tool.coverage.report] +show_missing = false # :101 +precision = 2 # :106 +``` +Comment `pyproject.toml:85-97`: without `concurrency`, coverage stops recording an async handler at +its first `await db.execute(...)`; measured admin.py 19% -> 41%. Dev deps `pyproject.toml:32-54` +include `pytest`, `pytest-asyncio`, `ruff`, `testcontainers[postgres]`, `respx`, `syrupy`, +`coverage[toml]`, `pytest-cov`, `factory-boy`, `mutmut>=2.4,<3`, `playwright`. + +### B.8 Snapshot tests + +Library: **syrupy** (`pyproject.toml:38`), via the `snapshot` fixture; assertions are +`assert value == snapshot` (`tests/characterization/test_agent_turn_gm.py:92,104,114,139,160,183,205,251,267`; +`tests/characterization/test_profile_pipeline_gm.py:213`). Snapshot files: +`tests/characterization/__snapshots__/test_agent_turn_gm.ambr` and `test_profile_pipeline_gm.ambr`. +Update mechanism is syrupy's `pytest --snapshot-update`, but the repo treats blind updates as +forbidden: `docs/plans/2026-08-10-org1-parity.md:16` "**Never run `pytest --snapshot-update`.**"; +`docs/specs/2026-08-06-role-topology-post-type-gating-design.md:474` "does **not** license a +blanket `pytest --snapshot-update`". The GM tests also assert the crux values explicitly "so a +careless --snapshot-update cannot silently" bless a regression +(`tests/characterization/test_profile_pipeline_gm.py:332,461`). (Note: that spec cites a "CLAUDE.md +prohibition" that is not present in the current CLAUDE.md.) + +### B.9 Tests that pin current behavior a fix would have to invert + +1. **`tests/integration/test_db_contract.py:268-281`** — DAT-1 pins that deleting a User who is a + PI member of a private channel RAISES (FK SET NULL drives `user_id` NULL while `agent_id` is + already NULL, violating the CHECK): + ```python + async def test_dat1_deleting_pi_member_user_violates_pcm_check(db_session): + ch = await factories.make_agent_channel(db_session, visibility="collab_private") + u = await factories.make_user(db_session) + await factories.make_private_channel_member( + db_session, channel=ch, agent_id=None, user_id=u.id, role="pi" + ) + with pytest.raises(IntegrityError) as ei: + async with db_session.begin_nested(): + await db_session.execute( + text("DELETE FROM users WHERE id = :id"), {"id": u.id} + ) + # Lock the SPECIFIC constraint that fires, so a future schema change that makes a + # different IntegrityError fire first can't silently re-point what "DAT-1" pins. + assert "pcm_exactly_one_of_agent_or_user" in str(ei.value) + ``` + A migration that changes `private_channel_members.user_id` to `ondelete="CASCADE"` (or a + route that deletes memberships first) must rewrite this test to assert the delete succeeds + and the membership row is gone. Header comment at `:262-266`. + +2. **`tests/unit/test_thread_not_found.py:134-144`** — pins that `_evict_dead_thread` also + forgets the thread was closed: + ```python + def test_evicts_from_all_agents(self, engine_with_agents): + engine, dead_ts, a, b = engine_with_agents + engine._evict_dead_thread(dead_ts) + + for ag in (a, b): + assert dead_ts not in ag.state.active_threads + assert not any(p.post_id == dead_ts for p in ag.state.interesting_posts) + assert not any(p.thread_id == dead_ts for p in ag.state.pending_proposals) + + assert f"proposal_thread:{dead_ts}" not in engine._poll_cursors + assert dead_ts not in engine._closed_thread_ids + ``` + Implementation it pins: `src/agent/simulation.py:1817-1818` + `self._poll_cursors.pop(f"proposal_thread:{thread_id}", None)` / + `self._closed_thread_ids.discard(thread_id)`. Note `_rebuild_agent_state` uses + `_closed_thread_ids` as the "already accounted for" marker (`simulation.py:4166-4177`), so a + fix that keeps evicted threads in `_closed_thread_ids` must flip line 144 to `in`. The fixture + (`:108-132`) seeds `engine._poll_cursors[f"proposal_thread:{dead_ts}"] = "1.0"` and + `engine._closed_thread_ids.add(dead_ts)`. + +3. **`tests/integration/test_worker.py`** — there is no test literally about a DB *connection* + error. The two DB-error-adjacent tests are: + - `:968-1000` `test_an_unknown_job_type_cannot_even_be_enqueued`: raw INSERT with + `type='bogus_type'` -> `with pytest.raises(DBAPIError) as exc:` ... `await db.rollback()` + ... `assert "job_type_enum" in str(exc.value)`, then a CONTROL insert with a legal type + commits. Pins that the enum is the rejecting object. + - `:485-523` `test_process_job_swallows_the_failure_so_the_next_job_still_runs` (T5.3): + `monkeypatch.setattr(worker_main, "run_profile_pipeline", crash_for_bad)`; `process_job` + must return normally; `assert (await wk.job_state(j_bad)).status == "dead"`; the next job + completes. Pins `src/worker/main.py:100-106` (`except Exception as exc: logger.error(...)`, + job marked dead after max attempts). A fix that makes `process_job` re-raise, or that + treats DB errors differently from pipeline errors, must adjust this. + The module docstring `:9-14` explains it uses a committing `async_sessionmaker(engine)` (not + `db_session`) because `claim_job`/`process_job` commit, and patches + `src.worker.main.run_profile_pipeline` (the import-time binding), not the service module (`:16-22`). + +4. Other static pins that constrain edits (not bugs, but must be kept green): + - `tests/unit/test_slack_client_contract.py` — `self._client.` may appear exactly once in + `src/agent/slack_client.py` (see docstring at `slack_client.py:297-302`); every Slack call + must go through `_api` -> `_call_with_retry`. + - `tests/unit/test_slack_boundary.py:15,44-58` — `slack_sdk` may be imported only in the two + `ALLOWED` modules (`src/agent/slack_client.py`, `src/services/slack_web.py`). + - `tests/unit/test_slack_web.py:183` `test_every_sync_entry_point_has_an_async_wrapper` — a + new sync function in `slack_web.py` needs a matching `_async` wrapper. + - `tests/unit/test_cohort_isolation.py:1312-1320` pins that `scripts/ci.sh` contains `alembic heads`. + +--- + +## C. Reusable in-repo patterns to copy + +### C.1 IntegrityError rollback + single retry, then 409 — `src/routers/agent_page.py:1000-1027` +```python + async def _write() -> None: + await record_pi_message( + db, + run_id=run_id, + channel_name=target_channel, + content=text, + sender_name=f"{current_user.name} (PI)", + thread_ts=thread_ts.strip() or None, + ) + await db.commit() + + # M1b guard: the canonical id can collide with another process (the sim) + # minting the same microsecond for this run, which hits the + # uq_agent_messages_run_ts constraint and would otherwise surface as a raw + # 500. Roll back and retry once — record_pi_message mints a fresh, monotonic + # id, so the retry gets a new ts. See PR #19 review M1. + try: + await _write() + except IntegrityError: + await db.rollback() + try: + await _write() + except IntegrityError: + await db.rollback() + raise HTTPException( + status_code=409, + detail="Message could not be saved due to a conflict, please retry", + ) +``` +(`from sqlalchemy.exc import IntegrityError` at `agent_page.py:14`.) + +### C.2 IntegrityError as lost-race -> rollback, fetch, update — `src/routers/public.py:1083-1102` +```python + try: + await db.commit() + except IntegrityError: + # Lost a race on the unique (decision, token) constraint — fetch & update. + await db.rollback() + vote_obj = ( + await db.execute( + select(ProposalVote).where( + ProposalVote.thread_decision_id == payload.decision_id, + ProposalVote.voter_token == token, + ) + ) + ).scalar_one() + vote_obj.vote = payload.vote + if details: + vote_obj.details = details + await db.commit() + + await db.refresh(vote_obj) + return {"id": str(vote_obj.id)} +``` +(Endpoint `submit_proposal_vote` starts at `public.py:1017`; import at `public.py:20`.) + +### C.3 `asyncio.to_thread` wrappers — `src/services/slack_web.py:267-300` +```python +async def list_channel_ids_async( + token: str, + *, + include_private: bool = True, + exclude_archived: bool = False, +) -> dict[str, str]: + """``list_channel_ids`` off the event loop.""" + return await asyncio.to_thread( + list_channel_ids, token, + include_private=include_private, exclude_archived=exclude_archived, + ) + + +async def lookup_user_by_email_async(token: str, email: str) -> str | None: + """``lookup_user_by_email`` off the event loop.""" + return await asyncio.to_thread(lookup_user_by_email, token, email) + + +async def post_message_async( + token: str, channel: str, text: str, *, thread_ts: str | None = None +) -> list[dict[str, Any]]: + """``post_message`` off the event loop.""" + return await asyncio.to_thread( + post_message, token, channel, text, thread_ts=thread_ts) +``` +Rule from the module docstring `slack_web.py:15-19`: async callers MUST use the `_async` wrappers +because the sync core `time.sleep`s between retries. `__all__` at `:61-73` lists both forms. + +### C.4 Bounded retry with Retry-After parse + cap — `src/services/slack_web.py:41-59, 86-122` +```python +_MAX_ATTEMPTS = 4 +_BACKOFF_BASE = 0.5 +_MAX_RETRY_AFTER = 30.0 +_TERMINAL = frozenset({ + "invalid_auth", "account_inactive", "token_revoked", "no_permission", + "user_not_found", "users_not_found", "channel_not_found", "not_in_channel", +}) + +def _call(client: WebClient, method: str, **kwargs: Any) -> Any: + last: Exception | None = None + for attempt in range(_MAX_ATTEMPTS): + try: + return getattr(client, method)(**kwargs) + except SlackApiError as exc: + code = _error_code(exc) + if code in _TERMINAL: + raise + last = exc + if attempt == _MAX_ATTEMPTS - 1: + break + delay = _BACKOFF_BASE * (2 ** attempt) + if code == "ratelimited": + retry_after = (getattr(exc.response, "headers", {}) or {}).get("Retry-After") + if retry_after is not None: + try: + asked = float(retry_after) + except (TypeError, ValueError): + asked = delay + if asked > _MAX_RETRY_AFTER: + logger.warning( + "[slack_web] %s asked for Retry-After=%.0fs; capping at %.0fs", + method, asked, _MAX_RETRY_AFTER, + ) + delay = min(asked, _MAX_RETRY_AFTER) + logger.warning("[slack_web] %s failed (%s); retrying in %.1fs", method, code, delay) + if delay > 0: + time.sleep(delay) + assert last is not None + raise last +``` +Unit-test pattern for it (`tests/unit/test_slack_web.py:191-208`): `monkeypatch.setattr(slack_web.time, "sleep", lambda d: slept.append(d))`, +`err.response.headers = {"Retry-After": "600"}`, `monkeypatch.setattr(slack_web, "_client", lambda _t: client)`, +`assert slept == [slack_web._MAX_RETRY_AFTER]`. + +### C.5 `_call_with_retry` — `src/agent/slack_client.py:310-342` (constants `:119` `MAX_RETRIES = 3`) +```python + def _call_with_retry(self, method, **kwargs) -> Any: + last_exc: SlackApiError | None = None + for attempt in range(MAX_RETRIES): + try: + return method(**kwargs) + except SlackApiError as exc: + if exc.response.get("error") == "ratelimited": + last_exc = exc + retry_after = int(exc.response.headers.get("Retry-After", 5)) + logger.warning( + "[%s] Rate limited, retrying in %ds (attempt %d/%d)", + self.agent_id, retry_after, attempt + 1, MAX_RETRIES, + ) + time.sleep(retry_after) + else: + raise + raise SlackApiError( + "Rate limit retries exhausted", + response=last_exc.response if last_exc else None, + ) +``` +Note the asymmetry with C.4: this one has **no cap** on `Retry-After`, no exponential backoff, +and `int()` (not `float()`) with no `ValueError` guard. Docstring `:317-322` explains `last_exc`. +Chokepoint `_api(self, method: str, **kwargs)` at `:294-308` raises `SlackNotConnected` when +`self._client is None`. Contract test for Retry-After: `tests/unit/test_slack_client_contract.py:172-182` +(`monkeypatch.setattr(time, "sleep", ...)`, `slack_error("ratelimited", retry_after=17)`, `assert slept == [17]`). + +### C.6 `_paginate` — `src/agent/slack_client.py:344-398` (constants `:127` `SLACK_PAGE_LIMIT = 200`, `:133` `MAX_PAGES = 200`) +```python + def _paginate( + self, + method: str, + key: str, + *, + limit: int = SLACK_PAGE_LIMIT, + **kwargs, + ) -> list[dict[str, Any]]: + items: list[dict[str, Any]] = [] + seen_cursors: set[str] = set() + cursor = "" + for page in range(MAX_PAGES): + call = dict(kwargs) + call["limit"] = limit + if cursor: + call["cursor"] = cursor + try: + result = self._api(method, **call) + except SlackApiError as exc: + if page == 0: + raise + raise SlackListingIncomplete( + method, items, + f"page {page + 1} failed: {exc.response.get('error') if exc.response else exc}", + ) from exc + items.extend(result.get(key) or []) + cursor = ((result.get("response_metadata") or {}).get("next_cursor") or "").strip() + if not cursor: + return items + if cursor in seen_cursors: + raise SlackListingIncomplete( + method, items, f"Slack repeated cursor {cursor!r} at page {page + 1}", + ) + seen_cursors.add(cursor) + raise SlackListingIncomplete( + method, items, f"still paginating after {MAX_PAGES} pages", + ) +``` +`SlackListingIncomplete(method, partial, reason)` is defined at `slack_client.py:71-84`. + +### C.7 Per-item `except Exception` poller pattern — `src/agent/simulation.py` + +Slack-side DM poller `_poll_pi_dms` (`:3005-3047`) — early-return guard, per-agent `continue` +when disconnected, per-message try/except around the DB write, cursor advance outside the try: +```python + for pi_slack_id, agent_ids in self._pi_slack_id_to_agent_ids.items(): + for agent_id in agent_ids: + client = self.slack_clients.get(agent_id) + if not client or not client.is_connected: + continue + + oldest = self._dm_poll_cursors.get(agent_id, default_cursor) + messages = client.poll_dm_messages(pi_slack_id, oldest=oldest) + + for msg in messages: + ts = msg.get("ts", "") + text = msg.get("text", "").strip() + if not text: + continue + logger.info("[%s] PI DM from %s: %s", agent_id, pi_slack_id, text[:80]) + try: + async with self.session_factory() as db: + await record_pi_dm( + db, run_id=self.simulation_run_id, agent_id=agent_id, + pi_user_id=pi_slack_id, direction="inbound", content=text, + sender_name="PI", slack_ts=ts or None, + ) + await db.commit() + except Exception as exc: + logger.error("[%s] Failed to record PI DM: %s", agent_id, exc) + if ts > oldest: + self._dm_poll_cursors[agent_id] = ts +``` +DB-side sibling `_poll_pi_dms_from_db` (`:3086-3140`) — one try/except around the SELECT that +`return`s on failure (`:3100-3117`), then per-row try/except around the handler (`:3128-3132`): +```python + for r in rows: + if r.created_at and r.created_at > self._pi_dm_cursor: + self._pi_dm_cursor = r.created_at + if r.ts and r.ts in self._pi_dm_seen: + continue # already processed (lookback re-scan) + if r.agent_id not in self.agents: + continue + if r.ts: + self._pi_dm_seen[r.ts] = r.created_at or EPOCH_UTC + try: + await self._pi_handler.handle_dm(r.agent_id, r.pi_user_id, r.content) + self.agents[r.agent_id].state.has_pi_directive = True + except Exception as exc: + logger.error("[%s] Failed to handle PI DM (DB): %s", r.agent_id, exc) +``` +The other siblings: `_poll_slack_for_pi_messages` (`:2685`), `_poll_inbound_from_db` (`:2842`), +`_poll_proposal_threads_for_pi` (`:3142`); `_seed_pi_dm_cursor` (`:3049-3084`) wraps its whole +DB read in `try/except Exception as exc: logger.warning("PI DM cursor seed failed: %s", exc)`. +Log style: `logger.error("[%s] ...: %s", agent_id, exc)` with %-formatting, never f-strings. + +### C.8 `_flush_persisted` re-queue-on-failure — `src/agent/simulation.py:3841-3847, 3888-3952` +```python + entries = self._pending_persist + self._pending_persist = [] + ... + try: + async with self.session_factory() as db: + for start in range(0, len(rows), chunk_size): + stmt = pg_insert(AgentMessage.__table__).values(rows[start:start + chunk_size]) + stmt = stmt.on_conflict_do_update( + constraint="uq_agent_messages_run_ts", + set_={...}, + where=or_( + AgentMessage.__table__.c.is_bot.is_(True), + stmt.excluded.is_bot.is_(False), + ), + ) + await db.execute(stmt) + ... + await db.commit() + except Exception as exc: + # Re-queue the failed batch instead of dropping it. The DB is now the + # source of truth for conversations, so a silently-dropped flush is + # unrecoverable — a restart rebuilds from the DB and these messages + # would be gone for good. New entries may have been enqueued while we + # were awaiting the (failed) commit; put the failed batch back in + # front to preserve chronological order for the next flush attempt. + self._pending_persist[0:0] = entries + logger.warning( + "Failed to flush %d messages, re-queued for retry: %s", + len(rows), exc, + ) +``` +Also note `:3883-3887` chunking to stay under `_PG_MAX_BIND_PARAMS` "because the except below +re-queues the whole batch on failure" (poison-pill avoidance). + +### C.9 `set_default_writer_id` — `src/agent/ids.py:114-127`; callers +```python +def set_default_writer_id(writer_id: int) -> None: + global _default + old = _default + new = TsMinter(writer_id) + with old._lock: + new._last_slot = old._last_slot + _default = new +``` +Writer slots `ids.py:47-50`: `WRITER_ENGINE = 0`, `WRITER_WEB = 1`, `WRITER_GRANTBOT = 2`, +`WRITER_ENGINE_AUX = 3`; `WRITER_SLOT_MODULUS = 100` (`:41`). Called once per process at entry: +- `src/main.py:110-113` inside `create_app()`: + ```python + # Claim the web process's canonical-id writer slot, so PI messages and DMs + # written here can never collide with ids minted by the engine or GrantBot + # processes (R1). See src/agent/ids.py. + set_default_writer_id(WRITER_WEB) + ``` +- `src/agent/main.py:51-54` inside the typer `main()` before `asyncio.run(...)`: + ```python + # Claim this process's canonical-id writer slot before anything mints. The + # engine's own minter owns WRITER_ENGINE; the module default is used here + # only for PI DM rows, so it takes the aux slot (R1). + set_default_writer_id(WRITER_ENGINE_AUX) + ``` + +### C.10 `_rebuild_agent_state` — signature and startup call +Signature `src/agent/simulation.py:4148-4154`: +```python + async def _rebuild_agent_state(self) -> None: + """Reconstruct per-agent state from the message log + DB. + + Runs after both the DB rebuild and the optional Slack reconcile, so it + behaves identically with Slack on or off. Reads only self.message_log, + thread_decisions, proposal_reviews and llm_call_logs — no Slack calls. + """ +``` +Startup order in `start()` (`simulation.py:556-579`): `_ensure_seeded_channels()` -> +`await _persist_seeded_channels()` -> `await _sync_private_channels_from_db()` -> +`await _load_pi_mappings()` -> `message_log.set_persist_callback(self._enqueue_persist)` -> +`await _rebuild_state_from_db()` -> `await _resolve_service_bot_uids()` -> +`await _rebuild_state_from_slack()` -> **`await self._rebuild_agent_state()` (`:578`)** -> +`await _seed_pi_dm_cursor()` -> `_rewind_cursors_for_private_channels()` -> ... -> +`await _recompute_allowed_sender_ids()` (`:594`) -> `refresh_lab_directories()` (`:597`). +Only production call site is `:578`; tests call it directly after `_rebuild_state_from_db()` +(`tests/integration/test_state_rebuild.py:132,152,...`; idempotency pinned by calling it twice `:196-201`). +Inside, the DB read is guarded `if self.session_factory: try: ... async with self.session_factory() as db:` (`:4158-4161`). + +### C.11 `Agent` construction in `_sync_roster_from_db` to_add branch — `src/agent/simulation.py:4638-4662` +```python + for aid in to_add: + r = desired[aid] + if self.slack_enabled: + token = r.slack_bot_token if is_valid_token(r.slack_bot_token) else env_token(aid) + if not is_valid_token(token): + logger.info( + "[roster] Agent %s is active but has no usable token yet — " + "skipping (will retry next sync once a token is set)", aid, + ) + continue + client = AgentSlackClient(agent_id=aid, bot_token=token) + if not client.connect(): + logger.warning("[roster] Slack connect failed for new agent %s — skipping", aid) + continue + else: + # Slack off: admit the agent with a no-op transport (never + # gate on a token/connection that doesn't apply in DB-only mode). + from src.agent.transport import NullTransport + client = NullTransport(agent_id=aid) + agent = Agent(agent_id=aid, bot_name=r.bot_name, pi_name=r.pi_name, role=r.role) + # In-place inserts (PIHandler shares these dicts by reference). + self.agents[aid] = agent + self.slack_clients[aid] = client + self._bot_name_to_id[agent.bot_name.lower()] = aid + logger.info("[roster] Added newly-active agent %s to live roster", aid) +``` +The roster SELECT (`:4538-4547`) reads `AgentRegistry.agent_id, bot_name, pi_name, +slack_bot_token, role` where `status == "active"`. Whole method is wrapped in +`try: ... except Exception as exc: logger.warning("[roster] roster sync failed: %s", exc)` +(`:4531, 4675-4677`) and throttled by `ROSTER_POLL_INTERVAL` (`:4526-4529`). Inner isolated +try/except for `_load_publication_records` at `:4557-4563`. After membership changes: +`self.message_log.set_bot_name_map(self._bot_name_to_id)`, `self._pi_slack_id_to_agent_ids.clear()`, +`await self._load_pi_mappings()`, `await self._recompute_allowed_sender_ids()` (`:4665-4674`). + +### C.12 `get_db` — `src/database.py:46-58` +```python +async def get_db() -> AsyncGenerator[AsyncSession, None]: + """FastAPI dependency for database sessions.""" + session_factory = get_session_factory() + async with session_factory() as session: + try: + yield session + await session.commit() + except Exception: + await session.rollback() + raise + finally: + await session.close() +``` +(Commits on successful handler exit; engine kwargs in A.3.) + +### C.13 `AgentBadgeMiddleware.dispatch` head — `src/main.py:25-43, 96-104` +```python +class AgentBadgeMiddleware(BaseHTTPMiddleware): + """Inject unreviewed proposal count into request.state for nav badge.""" + + async def dispatch(self, request: Request, call_next): + request.state.posthog_api_key = get_settings().posthog_api_key + request.state.agent_badge_count = 0 + user_id_str = request.session.get("user_id") if "session" in request.scope else None + if user_id_str: + try: + from src.models import ( + AgentDelegate, + AgentRegistry, + ProposalReview, + ThreadDecision, + User, + ) + session_factory = get_session_factory() + async with session_factory() as db: + uid = uuid.UUID(user_id_str) + ... + except Exception as exc: + # Deliberately swallowed: this middleware only computes a nav + # badge count, and no page should 500 because a count failed. + # But it is LOGGED — ... + logger.warning("Badge-count middleware failed, continuing: %s", exc) + return await call_next(request) +``` +It uses `get_session_factory()` directly (not `get_db`), which is why `tests/conftest.py:120-121` +monkeypatches `src.main.get_session_factory`. Registered first at `src/main.py:122` so it runs +inside `SessionMiddleware` (`:125-132`, cookie `copi-session`, 30-day max_age). + +### C.14 `/api/health` — `src/main.py:150-153` +```python + @application.get("/api/health") + async def health(): + """Health check endpoint.""" + return {"status": "ok"} +``` +Defined inline inside `create_app()`, not in a router. Test: `tests/integration/test_health_route.py:6-9`. + +### C.15 Defensive strip model — `strip_ungrounded_authorship_lines` usage, `src/agent/simulation.py:5340-5360` +```python + # Authorship hygiene (issue #29): a false authorship note written + # here is re-injected into every future prompt. Strip lines the + # publication records can't back before persisting. + own_db = self._agent_publications.get(agent.agent_id) + profile_dois = agent.own_publication_dois + own_record = LabPublicationRecord( + dois=(own_db.dois if own_db else set()) | profile_dois, + has_records=bool(own_db) or bool(profile_dois), + ) + response, stripped_lines = strip_ungrounded_authorship_lines( + response, + own_record, + self_names=lab_self_names( + agent.agent_id, agent.bot_name, agent.pi_name + ), + ) + for line in stripped_lines: + logger.warning( + "[%s] Memory update: stripped ungrounded authorship line: %s", + agent.agent_id, line[:160], + ) +``` +Shape to copy: pure function returns `(cleaned_text, removed_items)`; caller logs each removed +item at WARNING with a truncated preview, then proceeds with the cleaned text +(`agent.update_working_memory_file(response, ...)` at `:5362`). Defined in +`src/agent/authorship_rules.py:355`; imported at `simulation.py:18`. + +--- + +## D. Lint / format / running tests + +### D.1 ruff config — `pyproject.toml:63-69` +```toml +[tool.ruff] +line-length = 100 +target-version = "py311" + +[tool.ruff.lint] +select = ["E", "F", "I", "UP", "B"] +ignore = ["E501"] +``` +No formatter (`ruff format`/black) is configured or run by the gate. + +### D.2 The gate — `scripts/ci.sh` (301 lines), run by `.git/hooks/pre-push` (`exec .../scripts/ci.sh`) + +Steps in order (`ci.sh:7-20`): (1) alembic single head + no duplicate ids, offline (`:103-127`); +(2) alembic round trip `upgrade head -> downgrade $MIGRATION_FLOOR -> upgrade head` against a +throwaway postgres:15 on `127.0.0.1:55432` (`:129-240`, DSN +`postgresql+asyncpg://copi:copi@127.0.0.1:${MIGCHECK_PORT}/copi_migcheck` `:199`, floor `0018` `:70`; +skip with `CI_MIGRATION_DB=none`); (3) ruff on tests — **zero findings**; (4) ruff on src — ceiling; +(5) full pytest with coverage floor; then reclaim leaked testcontainers volumes (`:297-298`). + +Variables (`ci.sh:38-88`): +```bash +VENV_PY="${VENV_PY:-$REPO_ROOT/.venv-test/bin/python}" +COV_MIN="${COV_MIN:-60}" +SRC_LINT_MAX="${SRC_LINT_MAX:-260}" +MIGCHECK_PORT="${MIGCHECK_PORT:-55432}" +MIGRATION_FLOOR="${MIGRATION_FLOOR:-0018}" +LINT_TARGETS=( + tests/conftest.py tests/factories.py tests/fakes.py + tests/unit tests/integration tests/characterization tests/contract + tests/e2e + scripts/migrate + scripts/backup +) +``` +Test-suite lint, must be clean (`ci.sh:242-243`): +```bash +echo "==> ruff (test-suite lint)" +"$VENV_PY" -m ruff check "${LINT_TARGETS[@]}" +``` +The src ratchet, verbatim (`ci.sh:245-290`): +```bash +echo "==> ruff (src/ ratchet, ceiling ${SRC_LINT_MAX})" +set +e +src_lint_out="$("$VENV_PY" -m ruff check src --output-format=concise --quiet 2>&1)" +src_lint_rc=$? +set -e +if [ "$src_lint_rc" -gt 1 ]; then + echo "ERROR: ruff failed to run over src/ (exit ${src_lint_rc}):" >&2 + printf '%s\n' "$src_lint_out" >&2 + exit 1 +fi +if printf '%s' "$src_lint_out" | grep -q 'E902'; then + echo "ERROR: ruff could not read part of src/ (E902), so the finding count is not a" >&2 + ... + exit 1 +fi +src_findings="$(printf '%s' "$src_lint_out" | grep -c . || true)" +if [ "$src_findings" -gt "$SRC_LINT_MAX" ]; then + echo "ERROR: ruff findings in src/ rose to ${src_findings}; the ceiling is ${SRC_LINT_MAX}." >&2 + echo "Fix what you added. Do not raise SRC_LINT_MAX in scripts/ci.sh to make this pass." >&2 + printf '%s\n' "$src_lint_out" >&2 + exit 1 +fi +echo " ${src_findings} findings (ceiling ${SRC_LINT_MAX})" +``` +Latest measured: 254 findings vs ceiling 260 (commit `18ba52c` body). New src code should add +zero findings; net-negative is welcome ("LOWER THIS AS DEBT IS PAID; NEVER RAISE IT" `ci.sh:52`). + +Pytest step (`ci.sh:292-295`): +```bash +echo "==> pytest (full suite + branch coverage, fail-under=${COV_MIN}%)" +"$VENV_PY" -m pytest tests/ \ + --cov=src --cov-report=term-missing \ + --cov-fail-under="${COV_MIN}" +``` +Coverage floor **60%** (`ci.sh:46`); latest measured 69.2% (commit `18ba52c` body). Gate +prerequisites: `.venv-test/bin/python` must exist (`ci.sh:90-95`, create with +`uv venv .venv-test && uv pip install --python .venv-test/bin/python -e '.[dev]'`) and Docker must +be reachable (`ci.sh:97-101`). `.venv-test/bin/python -> /usr/bin/python3` (Python 3.12.3 on this host). + +### D.3 Running a single test file + +Host (what the gate uses): +```bash +.venv-test/bin/python -m pytest tests/unit/test_roster_sync.py -q +.venv-test/bin/python -m pytest tests/integration/test_agent_page.py -q # needs Docker for testcontainers +``` +Host, pointing integration tests at an existing Postgres instead of testcontainers: +`TEST_DATABASE_URL=postgresql+asyncpg://... .venv-test/bin/python -m pytest tests/integration/... -q` +(`tests/conftest.py:30-35, 51-53`). + +In-container (CLAUDE.md "Testing"; `TEST_DATABASE_URL` is REQUIRED because the app container has no +Docker socket; the named DB must pre-exist; never use `copi`): +```bash +docker compose exec -T -e TEST_DATABASE_URL=postgresql+asyncpg://copi:copi@postgres:5432/copi_a3 \ + app python -m pytest tests/unit/test_roster_sync.py -v +docker compose exec -T postgres createdb -U copi copi_xN # fresh scratch DB +``` +Same form is used throughout `.notes/cohort-thorough-test-plan.md` (`:163,386,905`). +Lint a single file the same way the gate does: `.venv-test/bin/python -m ruff check tests/unit/test_x.py`. + +### D.4 Rule: tests/ must be ruff-clean +`scripts/ci.sh:14` "ruff lint of the test suite. New test code is kept spotless — zero findings." +Enforced by `ci.sh:242-243` (plain `ruff check` on `LINT_TARGETS`, exit 1 on any finding). The same +zero-finding bar applies to `scripts/migrate` and `scripts/backup` (`ci.sh:80-87`). + +--- + +## E. Commit / PR conventions + +### E.1 Last 40 subjects (`git log --format='%h %s' -40`) +``` +18ba52c feat(cohort): grantbot service-bot membership + attribution; admin access visibility +c7c3427 fix(backup): install the OnFailure target unit +166242f Fix two fail-green critical findings in the verified-backup nightly (C1, C2) +6fe3c8e fix(cohort): add the four empty-institution Scripps PIs to scripps-investigators +2c0c021 fix(cohort): post-audit fix wave — robustness, drift reporting, tests, docs +ccf912c fix(graph): select /scripps-graph nodes from the cohort, not _SCRIPPS +7591a2a feat(cohort): idempotent seeding with a real audit trail +a6bc6e5 feat(cohort): seed planner — additive diff of manifest against DB +3c84371 feat(cohort): manifest of record for the three cohorts, plus its validator +9143e32 docs(cohort): implementation plan for seeding the three cohorts +965c066 docs(backup): stop overstating what the TOC precheck can see +cc7d7e6 Merge: verified nightly Postgres backups for both production stacks +9238f73 fix(backup): raise FREE_SPACE_FACTOR to 7 in the template too +37605e0 Fix seven Important findings from the final backup-system audit +75c639f test(backup): harness green — 12/12, with the TOC boundary measured +914a0e8 fix(backup): harness must never mutate a real backup +19a366e test(backup): failure-injection harness for the verified-backup system +7179744 fix(backup): validate BACKUP_ROOT path to prevent system directory escape +6e95183 feat(backup): systemd units, config template and installer +1a917e5 fix(backup): add name validation, containment check, explicit flag rejection +c14f08f test: bind the throwaway Postgres to loopback only +97395b5 ci: reclaim testcontainers' leaked anonymous volumes +01b9506 feat(backup): preflight guards, label-filtered sweep, retention and CLI +9d9ed55 fix(backup): add comprehensive guards against untested mutations +c72a8ea feat(backup): sidecars, status document, and SES reporting +3afbfa9 fix(backup): timeout handling, stack name deduplication, fast-fail guard +b2e5797 feat(backup): isolated verify-restore with OOM discrimination +87cce8a feat(backup): fix Task 6 tests — verify partial cleanup and session failures +96890b3 feat(backup): dump orchestration with container temp and atomic rename +728716b fix(backup): orphaned processes, zombies, and hung psql +33f5ad2 fix(backup): snapshot session requires docker exec -i for stdin attachment +e62f972 feat(backup): snapshot-consistent session and exact count capture +fd7bd92 feat(backup): command builders and injectable runner seam +e498841 docs(backup): fold host findings into the design +a01042f feat(backup): exact row-count parity comparison +2842829 feat(backup): dump naming and count-based retention selection +26876c4 feat(backup): config parsing scaffold and lint gate for backup tooling +30e6f61 docs(backup): verified Postgres backup design +5e79574 docs(cohort): design for seeding cabo/schultz/scripps cohorts +364bee3 fix(inbound): fail closed on null-email users; make the help email reply-able +``` +Style: conventional-commit prefix `type(scope): lowercase imperative summary`, types seen +`feat`, `fix`, `docs`, `test`, `ci`, `ops` (`a83060e ops(agent): ...`), scopes = subsystem +(`cohort`, `backup`, `agent`, `inbound`, `email`, `graph`, `tools`, `prompts`, `sweep`, `runbook`). +Em-dashes and semicolons in subjects are common; no trailing period. A minority use a bare +capitalized imperative ("Fix two fail-green critical findings..."). Merge commits are +`Merge pull request #N from SuLab/` or a hand-written `Merge: `. + +### E.2 Issue references +Yes, when an issue exists: subject suffix `(#29)` or `(#29 audit I5)` (e.g. `3bcd8d9 feat(agent): +strip ungrounded authorship lines from memory syntheses (#29)`, `12f7b46 fix(agent): identity-aware +memory strip — no laundering through third-person subjects (#29 audit I5)`), and PR bodies open +with `Closes #29`. Recent cohort/backup work has no issue numbers (bodies cite audit finding ids +like "C1, C2", "I3" instead). + +### E.3 Commit bodies +Long, explanatory prose in numbered/paragraph form; state root cause, what changed, measured +figures, and the gate result. Example tail from `18ba52c`: +``` +Tests: +44 (service-bot attribution incl. a start()-ordering pin, service-tag +thread rules, admin users page/badges/filters/gate, service-id cohort add, +seeding access status, impersonate-create). Full gate green: 2030 passed, +120 skipped, coverage 69.2% (floor 60), ruff src 254 (ceiling 260). + +Co-Authored-By: Claude Fable 5 +``` +Every recent commit ends with a `Co-Authored-By: Claude ... ` trailer. +No commit template (`git config commit.template` unset; no `.gitmessage`/`CONTRIBUTING.md`). + +### E.4 PR bodies (`gh pr view 32 --json body`, `gh pr view 36 --json body`; both base = `copi-prod`) +- **PR 32** (`SuLab/issue-29-authorship-grounding` -> `copi-prod`): opens `Closes #29 (targets + \`copi-prod\`, stacked on the \`email-fix\` line).` then a one-paragraph incident summary; + sections `## What this does (layered, deterministic-first)` (bulleted, bold lead-ins, file + paths in backticks), `## Verification` ("Full `./scripts/ci.sh` green: **1775 passed / 120 + skipped, 67.40% branch coverage, ruff 256/260**" + how it was audited), `## Rollout — read + \`docs/issue-29-remediation.md\` before deploying` (ordered ops steps), `## Known + conservative-direction follow-ups (non-blocking...)`. +- **PR 36** (`SuLab/deploy-followups` -> `copi-prod`): opens with stacking note ("Stacked on #31 + (`email-fix`): **merge #31 (and #32) first**"); sections `## What this adds` (each bullet + starts with the commit prefix in code, e.g. "**`fix(email)`: pin ...**"), `## What this does + NOT do` (explicitly "No migrations (alembic head stays `0024`)."), `## Testing` ("written + test-first ... Full `./scripts/ci.sh` gate green on this branch: **1681 passed / 120 skipped, + 20 snapshots**, migration round trip clean"), and ends with the footer + `🤖 Generated with [Claude Code](https://claude.com/claude-code)`. +Pattern to mirror: state target branch and stacking; What/What-not; Verification with exact +ci.sh numbers (passed/skipped/coverage/ruff count); Rollout steps if ops-affecting; footer. + +--- + +## F. File sizes (`wc -l`) + +| file | lines | +|---|---| +| `src/agent/simulation.py` | 5498 | +| `src/agent/slack_client.py` | 1098 | +| `src/agent/grantbot.py` | 790 | +| `src/services/email_inbound.py` | 855 | +| `src/services/email_notifications.py` | 1075 | +| `src/services/profile_pipeline.py` | 579 | +| `src/routers/agent_page.py` | 1623 | +| `src/routers/admin.py` | 1930 | +| `src/routers/public.py` | 1132 | +| `src/worker/main.py` | 185 | +| `src/main.py` | 158 | +| `src/database.py` | 58 | +| total | 14981 | + +Supporting sizes: `tests/conftest.py` 351, `tests/fakes.py` 256, `tests/factories.py` 190, +`scripts/ci.sh` 301, `alembic/env.py` ~118. diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/deploy_dossier.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/deploy_dossier.md new file mode 100644 index 00000000..136d1193 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/deploy_dossier.md @@ -0,0 +1,637 @@ +# Deploy dossier — coPI.science prod (org1), branch copi-prod @ 18ba52c + +Facts only. Every claim cites `file:line` in /home/a/scripps/coPI.science unless the path +is prefixed `MEMORY:` (operator memory notes under +`/home/a/.claude/projects/-home-a-scripps-coPI-science/memory/`, NOT repo content — treat as +unverified hearsay until re-measured on the host). Nothing was executed against Docker, SSH +or the network while compiling this. + +Alembic head in this checkout is **0024** — no file has `down_revision = "0024"` +(`alembic/versions/` listing; `0024_add_agent_role.py:21-22`). Whether prod is at 0024 must be +read from the DB (`select * from alembic_version`, `docs/production-migration.md:521`). + +--- + +## 1. Prod topology (`docker-compose.prod.yml` + `docker-compose.override.yml`) + +Rule: **production is `docker-compose.prod.yml` + `docker-compose.override.yml`; always pass +both `-f` flags** (`CLAUDE.md:28-29`). Shortcut: `export COMPOSE_FILE=docker-compose.prod.yml:docker-compose.override.yml` +(`CLAUDE.md:44`). Bare `docker compose` reads the **dev** file `docker-compose.yml` (no +`restart:`, `uvicorn --reload`, publishes 8001) (`CLAUDE.md:29-31`; `docker-compose.yml:20-22`). + +Compose project name is `copi-python` — container names follow `--1`: +`copi-python-app-1` (`CLAUDE.md:50`), `copi-python-postgres-1` +(`scripts/backup/backup.env.example:6`), `copi-python-worker-1` (`docs/inbound-email.md:93`), +`copi-python-certbot-1` (`scripts/ci.sh:170`). Project/path table: `docs/specs/2026-08-10-org1-parity-design.md:30`. + +| service | image / build | command | restart | volumes | healthcheck | depends_on | profile | ports | +|---|---|---|---|---|---|---|---|---| +| postgres | `postgres:15` (`prod.yml:3`) | image default | `unless-stopped` (`:4`) | `pgdata:/var/lib/postgresql/data` (`:10`) | `pg_isready -U ${POSTGRES_USER:-copi}` 10s/5s/5 retries/start 10s (`:11-16`) | — | — | **none published** (no `ports:`; confirmed `docs/production-migration.md:34-35`) | +| app | `build: context: .` (`:26-27`) | `uvicorn src.main:app --host 0.0.0.0 --port 8000` (`:29`) | `unless-stopped` (`:28`) | `./profiles:/app/profiles`, `./prompts:/app/prompts` (`:40-41`) | `urllib.request.urlopen("http://127.0.0.1:8000/api/health")` 30s/10s/3/start 15s (`:45-50`) | postgres healthy (`:42-44`) | — | `expose: 8000` only (`:30-31`) | +| worker | `build: .` (`:60-61`) | `python -m src.worker.main` (`:63`) | `unless-stopped` (`:62`) | `./profiles:/app/profiles` only (`:71-72`) — **no prompts mount** | none | postgres healthy (`:73-75`) | — | — | +| agent | `build: .` (`:85-86`) | `python -m src.agent.main` (`:87`) | **none → `no`** (`:84-110` has no `restart:`; design doc says "restart=no by design" `docs/specs/2026-08-18-postgres-backup-verification-design.md:555`) | `./profiles`, `./prompts`, `./data` (`:95-98`) | none | postgres healthy (`:99-101`) | `profiles: [agent]` (`:102-103`) | — | +| grantbot | `build: .` (`:113-114`) | `python -m src.agent.grantbot scheduler --run-hour 8 --max-per-channel 1` (`:116`) | `unless-stopped` (`:115`) | `./profiles`, `./prompts`, `./data` (`:124-127`) | none | postgres healthy (`:128-130`) | — | — | +| nginx | `nginx:1.27-alpine` (`:140`) | see §6 (`:171`) | `unless-stopped` (`:141`) | `./nginx/nginx.conf:/etc/nginx/templates/default.conf.template:ro`, `./certbot/conf:/etc/letsencrypt:ro`, `./certbot/www:/var/www/certbot:ro` (`:156-159`) | `wget --no-check-certificate https://localhost/` 30s/5s/3/start 10s (`:165-170`) | app healthy (`:162-164`) | — | `80:80`, `443:443` (`:142-144`) | +| certbot | `certbot/certbot:latest` (`:181`) | entrypoint loop `certbot renew --quiet; sleep 12h` (`:186`) | `unless-stopped` (`:182`) | `./certbot/conf:/etc/letsencrypt`, `./certbot/www:/var/www/certbot` rw (`:183-185`) | none | — | — | — | + +- Every app-family service: `env_file: .env` (`:32,64,88,117`) plus `environment:` overrides + `DATABASE_URL=postgresql+asyncpg://${POSTGRES_USER:-copi}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-copi}`, + `SECRET_KEY: ${SECRET_KEY:?...}`, `ENVIRONMENT: ${ENVIRONMENT:-production}` (`:33-38,65-70,89-94,118-123`). +- Compose-parse-time required vars (`:?`): `POSTGRES_PASSWORD` (`:7`), `SECRET_KEY` (`:37,69,93,122`), `DOMAIN` (`:161`). +- nginx networks: `default` + external `copi-edge` (`:153-155`); `copi-edge` is created out-of-band + with `docker network create copi-edge` and declared `external: true` (`:198-203`). nginx has + `extra_hosts: host.docker.internal:host-gateway` for the devel vhost (`:145-148`). +- Log driver: `prod.yml` sets `logging.driver: awslogs` on every service (`:17-23` etc.); + `docker-compose.override.yml:13-34` forces `json-file` for all seven services because the EC2 + role `copi-ec2-ses-role` lacks `logs:CreateLogStream` — without the override every container + dies at start with `AccessDeniedException` (`override.yml:1-4`; `CLAUDE.md:36-39`). +- Verify correct file set: `docker inspect copi-python-app-1 -f '{{.HostConfig.RestartPolicy.Name}}'` + → all six services must report `unless-stopped` (`CLAUDE.md:47-51`). (Six = everything but `agent`.) +- Only one volume: `pgdata` (`prod.yml:195-196`). Host volume name per design doc: `copi-python_pgdata` + (`docs/specs/2026-08-18-postgres-backup-verification-design.md:596`; `scripts/backup/failure_injection.sh:180`). +- `stop_grace_period: 30s` exists **only in the dev file** (`docker-compose.yml:53`); prod agent has + none, so `docker stop` without `-t 30` uses Docker's 10 s default. + +### Image contents (`Dockerfile`, `.dockerignore`) +- `python:3.11-slim`; `COPY pyproject.toml . ; COPY src/ src/ ; RUN pip install --no-cache-dir .` + (bakes `src/` into site-packages) then `COPY . .` (bakes the whole tree at `/app`) (`Dockerfile:1-17`). + `RUN mkdir -p profiles/public profiles/private prompts logs static` (`:20`). Default CMD uvicorn (`:24`). +- `.dockerignore` excludes `.git`, `certbot`, `logs`, `data`, `*.log`, caches, `.venv`, + `.provision_state.json`, **`.env` and `.env.*`** (`.dockerignore:1-17`). So `alembic/`, + `scripts/`, `prompts/`, `profiles/`, `templates/` ARE baked into the image; profiles/prompts/data + are then shadowed by bind mounts at runtime (`prod.yml:39-41,71-72,95-98,124-127`). +- Consequence: `import src` from `python scripts/X.py` resolves to the **site-packages copy** + unless `PYTHONPATH=/app` is set (`scripts/migrate/run_migration.sh:99-105`; + `docs/production-migration.md:341-345`). + +--- + +## 2. How code reaches prod + +- **No deploy script exists.** `scripts/` contains no deploy/pull/release script (`ls scripts/`); + no server-side CI — `./scripts/ci.sh` run by the local `pre-push` hook is "the whole gate" + (`CLAUDE.md:5-8`). +- Mechanism is manual: `git pull` on the host, then rebuild with compose. Documented sequence + (`docs/issue-29-remediation.md:14-48`): on the prod host (`ssh ubuntu@copi.science`, repo + `~/copi-python`) (`:14`) → "Merge/pull this branch; + `export COMPOSE_FILE=docker-compose.prod.yml:docker-compose.override.yml`" (`:16`) → stop agent + gracefully (`:17-25`) → `docker compose up -d --build app worker && docker compose --profile agent build agent` + (`:27-28`) → start agent (`:47-48`). +- Canonical rebuild commands (`CLAUDE.md:82-103`): + ```bash + C="-f docker-compose.prod.yml -f docker-compose.override.yml" + docker compose $C up -d --build app worker + docker compose $C --profile agent build agent + ``` + "prod bakes code into the image, so skipping [`build agent`] silently runs whatever source was + current at the last build" (`CLAUDE.md:98-100`, `:108-111`). +- Migration runbook's own deploy step: `docker compose up -d --build app worker` + (`docs/production-migration.md:442`) — **bare `docker compose`**, no `-f` flags, and the runbook + never mentions `COMPOSE_FILE` (grep of `docs/production-migration.md` and + `scripts/migrate/run_migration.sh` for `COMPOSE_FILE|docker-compose.prod|override.yml` → no matches). + `docs/specs/2026-08-10-org1-parity-design.md:418-420`: `run_migration.sh` "shells out to bare + `docker compose`, which resolves `docker-compose.yml` — the dev stack. + `COMPOSE_FILE=docker-compose.prod.yml` is not optional" (note: that line omits the override file; + `CLAUDE.md:44` includes it). +- grantbot is not in any documented rebuild command (`CLAUDE.md:96`, `issue-29-remediation.md:28`, + `production-migration.md:442` all say `app worker`). `docker compose up -d --build app worker` + rebuilds the shared image but does not recreate `grantbot`; the design-doc restore runbook is the + only place that stops/starts grantbot (`...backup-verification-design.md:547-548,573-574`). +- Prod host checkout path: `/home/ubuntu/copi-python` (`scripts/backup/copi-backup.service:3`; + `scripts/backup/copi-backup-failure@.service:3`; `scripts/set_cohort_active.py:24`; + `scripts/seed_cohorts.py:29`; `docs/specs/2026-08-18-postgres-backup-verification-design.md:547`; + `docs/specs/2026-08-10-org1-parity-design.md:30`). User: `ubuntu` (`docs/issue-29-remediation.md:14`). + Backup system runs "on the HOST as root under systemd" (`scripts/backup/copi_backup.py:6`). +- Docs also cite a `docker run` pattern for one-off scripts: + `-v /home/ubuntu/copi-python:/work -w /work copi-python-app python ...` (`scripts/seed_cohorts.py:29-30`) + — i.e. the built app image is tagged `copi-python-app`. +- **Second stack (blackbird)** — from `docs/specs/2026-08-10-org1-parity-design.md:28-38`: + path `/home/ubuntu/blackbird-copi-science`, compose project `copi-blackbird`, domain + `blackbird.copi.science` "proxied **by org1's nginx**", web service `blackbird-app` ("an + *uncommitted* host-local compose edit; the tracked `docker-compose.prod.yml` says `app` on every + branch"), star topology, `cohort_isolation_enabled=True`, alembic 0025 at the time of writing. + org1's `prod.yml:149-152` comments: `copi-edge` is "the shared external network the second + (blackbird) stack's app joins so nginx can proxy to blackbird-app:8000. See + ../blackbird-copi-science/SECOND_INSTANCE_SETUP.md" (that file is not in this repo). Backup config + lists both: `copi-python:copi-python-postgres-1:copi:copi` and + `copi-blackbird:copi-blackbird-postgres-1:copi:copi` (`scripts/backup/backup.env.example:6-7`). + blackbird's agent container is `blackbird-agent-run` (`...backup-verification-design.md:553`). + **Deploying org1 does not touch blackbird's containers, but recreating org1's nginx affects + blackbird's edge** (nginx.conf vhost, §6). +- MEMORY (not repo): `org1-prod-state-2026-08-14.md:19` — "a daily 07:00 cron runs + `claude -p --permission-mode auto` inside the prod repo (don't deploy across 07:00 blindly)". + Repo corroboration that a 07:00 UTC Claude cron exists on the host: + `docs/specs/2026-08-18-postgres-backup-verification-design.md:470-473` ("the 07:00 UTC + `daily_audit.md` Claude cron … runs for roughly 8 minutes"). +- README.md `Running locally` (`README.md:36-47`) and AGENT.md (`AGENT.md:141-146`) are **dev** + instructions (bare `docker compose`, `alembic upgrade head`); README:99 still references + `PILOT_LABS`, which `CLAUDE.md:121-122` says no longer exists. + +--- + +## 3. Schema migrations in prod today + +### Is anything automatic? +**No.** `grep -rn create_all src/` → nothing; `src/main.py` and `src/database.py` have no +lifespan/startup/alembic hooks (grep for `lifespan|startup|on_event|create_all|alembic|migrat` → +no matches); `src/worker/main.py:178-181` is `asyncio.run(run_worker())`; prod app command is bare +uvicorn (`prod.yml:29`). Confirmed in prose: "Nothing migrates automatically: the prod web command +is a bare `uvicorn` and there is no `create_all`" (`docs/specs/2026-08-10-org1-parity-design.md:422-423`). +The only automatic `alembic upgrade head` calls are test/CI-side (`tests/conftest.py:63-73`, +`scripts/ci.sh:229-231`). + +### How alembic finds the DB +- `alembic.ini:5` hardcodes `sqlalchemy.url = postgresql+asyncpg://copi:copi@localhost:5432/copi`. +- `alembic/env.py:20-23`: `db_url = os.environ.get("DATABASE_URL"); if db_url: config.set_main_option("sqlalchemy.url", db_url)`. + So **only `DATABASE_URL` overrides the localhost default**; app-side `settings.database_url` + (`src/config.py:111`, `src/database.py:16-18`) is NOT consulted by alembic. +- Inside prod containers `DATABASE_URL` is injected by compose (`prod.yml:34` etc.), so + `docker compose exec app python -m alembic ...` targets the right DB. On the **host**, the + bare hostname `postgres` resolved to a public IP (195.35.25.84) via a LAN search domain on the + dev machine (`docs/production-migration.md:33-39`; `run_migration.sh:133-140`). +- `alembic/env.py:78-81`: `do_run_migrations` calls `context.configure(connection=..., target_metadata=...)` + without `transaction_per_migration` → **the entire chain is one transaction** (`env.py:48-54`; + `production-migration.md:71-74`). +- `ALEMBIC_LOCK_TIMEOUT_MS` (default `10000`) applied as asyncpg `server_settings.lock_timeout` + at connect (`env.py:66,88-93`); `0` = wait forever (`production-migration.md:206`). It bounds lock + *wait*, not statement duration (`env.py:64-65`). +- Known silent-failure mode: executing SQL on the connection before `context.begin_transaction()` + makes every migration log "Running upgrade" then roll the whole chain back with **no** + `alembic_version` row (`env.py:69-77`; `production-migration.md:41-44`). Hence "read the revision + back" (`run_migration.sh:268-293`). + +### `scripts/migrate/run_migration.sh` — full flow +- Defaults: `TARGET="0024"` (`:56`; header comment `:3` still says "head 0023" — stale), + `DSN="${DATABASE_URL:-}"` (`:57`), `BACKUP_DIR="${MIGRATE_BACKUP_DIR:-backups}"` (`:58`), + `SVC="${MIGRATE_SERVICE:-app}"` (`:59`), `PG_SVC="${MIGRATE_PG_SERVICE:-postgres}"` (`:60`), + `LOCK_TIMEOUT_MS="${ALEMBIC_LOCK_TIMEOUT_MS:-10000}"` (`:61`). +- Flags: `--apply`, `--target `, `--database-url `, `--backup-dir `, + `--backup-verified-elsewhere ""`, `-h/--help`; `--skip-backup-check` is rejected loudly + (`:67-88`). **Default is rehearsal** — writes nothing (`:11-17`). +- Exit codes: `0` clear/applied, `1` BLOCKED, `2` rehearsal warnings, `3` operational, `64` usage (`:28-35`). +- `cd "$REPO_ROOT"` (`:52-53`); every compose call is **bare `docker compose`** (`:109,114,123,152,182,189,194-196,254`) + → `COMPOSE_FILE` must be exported for the prod file set (see §2). +- **Step 1** (`:107-128`): requires service `$SVC` running (`docker compose ps --status running --services`), + else exit 3 with hint `docker compose up -d --build app`; asserts + `docker compose exec -T -e PYTHONPATH=/app app python -c 'import src; print(src.__file__)'` == + `/app/src/__init__.py`, and `from src.models import Cohort` imports (`:123-127`). **Implication: + the running `app` container must already carry the new source (i.e. be rebuilt from the branch + that contains the migrations) before this script can run.** The runbook reconciles this with + "migrate before code" by noting old code keeps working on the new schema because new columns have + defaults (`production-migration.md:445-448`); MEMORY `org1-prod-access-and-local-copy.md:24` notes the + new web app "starts healthy against 0018". +- **Step 2** (`:142-149`): exit 64 if no DSN; prints DSN with password masked. + `run_py()` = `docker compose exec -T -e PYTHONPATH=/app -e DATABASE_URL="$DSN" "$SVC" python "$@"` (`:151-153`). +- **Step 3 backup** (`:160-204`), *before* preflight: if `--backup-verified-elsewhere` → WARN and pass + the reason to preflight; if rehearsal → prints would-dump and passes + `--backup-verified-elsewhere "rehearsal mode — no dump taken"`; if `--apply`: + `DBNAME` parsed from DSN (`:171`); file + `$BACKUP_DIR/${DBNAME}_pre${TARGET}_$(date +%Y%m%dT%H%M%S).dump` (`:172`); + `docker compose exec -T postgres pg_dump -U copi -Fc -f /tmp/copi_migrate_$$.dump "$DBNAME"` (`:181-182`); + verify TOC in-container `pg_restore -l` (`:189`); count TOC entries (`:194`); + `docker compose cp postgres:/tmp/... "$BACKUP_FILE"` (`:195`); rm temp; host size must be ≥1024 bytes (`:197-201`); + passes `--backup-path "$BACKUP_FILE"` to preflight (`:203`). **Hardcodes `-U copi`** (`:182`). +- **Step 4 preflight** (`:209-226`): snapshot path `SNAP="${MIGRATE_SNAPSHOT:-$BACKUP_DIR/preflight_snapshot.json}"` (`:211`); + `run_py scripts/migrate/preflight.py --target "$TARGET" --snapshot "$SNAP" "${EXTRA_PREFLIGHT[@]}"` (`:214-215`). + Exit 0 PASS, 2 WARN (continues), else BLOCKED exit 1 with remediate_duplicates hint (`:218-226`). + Rehearsal stops here: exit 2 if warnings, else 0 (`:228-245`). +- **Step 5** (`:251-266`): `docker compose exec -T -e PYTHONPATH=/app -e DATABASE_URL="$DSN" -e ALEMBIC_LOCK_TIMEOUT_MS="$LOCK_TIMEOUT_MS" app python -m alembic upgrade "$TARGET"`; + non-zero → exit 1 with hints (`LockNotAvailableError` → `docker stop -t 30 agent-run` and re-run). +- **Step 6** (`:276-293`): reads `select version_num from alembic_version` via SQLAlchemy inside the + container; must equal `$TARGET`, else "Treat this as a silent rollback. Do NOT deploy code." exit 1. +- **Step 7** (`:299-310`): `run_py scripts/migrate/postflight.py --target "$TARGET" --snapshot "$SNAP"`; + non-zero → exit 1, "Do NOT deploy application code." +- Footer prints steps 8–10 (backfill_slack_ts report/apply; deploy code + restart app+worker; start + agent-run last) (`:312-326`). +- What it deliberately does NOT do: stop/start containers, run `backfill_slack_ts.py`, resolve + duplicates (`:37-47`). + +### Preflight (`scripts/migrate/preflight.py`) assumptions +- `DEFAULT_TARGET = "0024"` (`:74`); `SUPPORTED_START_REVISIONS = ("0018", "0019", "0020", "0021", "0023")` (`:89`) + — note **0022 is not a supported start, and 0024 is not a supported start** (tooling is specific to + the 0018→0024 chain; `REVISION_ORDER` ends at `"0024"` `:206`). Docs list only 0018/0019/0020/0021 + (`production-migration.md:7`); code comment explains 0023 (`preflight.py:75-77`). +- 13 checks (`production-migration.md:281-295`): 1 supported start rev; 2 single head/no dup ids; + 3 the 0019 stamp is the *content* 0019 (three historical 0019s: `production-migration.md:297-323`); + 4 no duplicate `(simulation_run_id, message_ts)`; 5 planned objects don't pre-exist; 6 rows blocking + downgrade (`agent_id IS NULL`); 7 blocking sessions (idle-in-transaction always BLOCKs; active xact + older than `--max-xact-age-s` 5.0 BLOCKs, `preflight.py:471,1883-1888`); 8 env.py harness commits; + 9 sizing/lock-window estimate; 10 disk headroom; 11 legacy-row inventory (WARN); 12 recent + non-trivial backup (default max age 24h `:138`, min 1024 bytes `:143`, searched in + `("backups", "data/backups", "/backups", "/var/backups/copi")` `:148`, globs `:149`; WARN in + rehearsal); 13 row-count snapshot written. +- Exit: `0` ok, `1` BLOCKED, `2` warnings (`:70-72`). CLI flags `:1826-1889`. +- Postflight (`scripts/migrate/postflight.py`) 13 checks (`production-migration.md:373-387`); + `EXPECTED_TABLES = ("pi_dm_messages","cohorts","cohort_memberships","cohort_audit_events")` (`:132`); + hardcoded expected columns/indexes/constraints/enums for the 0019–0024 chain (`:97-180`). + "Postflight must be 0 FAIL before you deploy code" (`production-migration.md:393`). + +### Runbook order (`docs/production-migration.md`) +Five hard rules `:21-44`: backup+verify; **migrate DB BEFORE deploying code**; `alembic downgrade` +is not a rollback; never let DSN default / always inside container; alembic output is not evidence. +Steps: §2 read-only measurement Q1–Q4 (`:88-135`), §6a rehearse +`export DATABASE_URL=postgresql+asyncpg://copi:copi@postgres:5432/copi; ./scripts/migrate/run_migration.sh` (`:263-266`), +§6b `./scripts/migrate/run_migration.sh --apply` (`:335-337`), §8 step 8 backfill_slack_ts (`:422-424`), +step 9 `docker compose up -d --build app worker` (`:441-442`), step 10 +`docker compose --profile agent run -d --name agent-run agent python -m src.agent.main --budget 0` (`:459-460`). +Lock-timeout hit → `docker stop -t 30 agent-run` then re-run (`:190-198`); +`ALEMBIC_LOCK_TIMEOUT_MS=30000 ./scripts/migrate/run_migration.sh --apply` to raise (`:202-204`). +Duplicates: `remediate_duplicates.py` dry-run / `--apply`, strategies, exit codes (`:226-255`). +Quick reference table (`:544-558`). Untested: >2.5 M rows, restore drill did not start the app, +managed Postgres, replication (`:586-597`). + +--- + +## 4. Backups + +### Nightly verified backup (host-level, `scripts/backup/`) +- Runs on the host as root under systemd; talks to DBs only via `docker exec`; every dump is + restored into a throwaway `--network none`, memory-capped `postgres:15` container and per-table + row counts compared against the dump's own snapshot (`copi_backup.py:2-11`; design §4.2/§4.4 + `...design.md:134-196,224-309`). +- Install: `sudo scripts/backup/install.sh` → `/usr/local/bin/copi-backup`, `/etc/copi-backup/backup.env` + (0600, from `backup.env.example` if absent), `/var/backups/copi` (0700), five units into + `/etc/systemd/system/`, `systemctl daemon-reload`; does NOT enable timers (`install.sh:16-40`). + Enable: `systemctl enable --now copi-backup.timer copi-backup-report.timer` (`install.sh:40`). +- Units: `copi-backup.service` `ExecStart=/usr/local/bin/copi-backup run`, `TimeoutStartSec=3600`, + `Nice=10`, `IOSchedulingClass=idle`, `ProtectSystem=strict`, `ReadWritePaths=/var/backups /run`, + `OnFailure=copi-backup-failure@%n.service` (`copi-backup.service:10-20`); + `copi-backup.timer` `OnCalendar=*-*-* 01:00:00 America/Los_Angeles`, `Persistent=true` (`copi-backup.timer:5-6`); + weekly `copi-backup-report.timer` `Mon *-*-* 08:00:00 America/Los_Angeles` (`copi-backup-report.timer:5`) + → `copi-backup report` (`copi-backup-report.service:7`); failure unit runs `copi-backup report` + (`copi-backup-failure@.service:12`). +- Config keys (`backup.env.example:6-38`): `STACKS` (`stack:container:db:user` per line — org1 = + `copi-python:copi-python-postgres-1:copi:copi`), `BACKUP_ROOT=/var/backups/copi`, + `RETENTION_COUNT=5`, `RETENTION_UNVERIFIED=2`, `VERIFY_IMAGE=postgres:15`, `VERIFY_MEM=768m`, + `VERIFY_TIMEOUT_SEC=1800`, `FREE_SPACE_FACTOR=7`, `REGRESSION_TOLERANCE_PCT=20`, `OFFSITE_CMD=""`, + `AWS_REGION`, `SES_SENDER_EMAIL`, `MAIL_TO`. +- Where dumps land / naming: `/var/backups/copi//__YYYYMMDDTHHMMSSZ.dump` + + `.json` sidecar; failed verify → `.dump.unverified` (`copi_backup.py:192-216`; + `_cmd_run_inner` `:1321-1344`; design `:73-76`). For org1: `/var/backups/copi/copi-python/copi-python_copi_.dump`. + `status.json` at `/var/backups/copi/status.json` (`:1226`; design `:76`). +- Retention: count-based — keep 5 newest verified + 2 newest unverified per stack; **never prune a + stack with zero verified dumps**; sidecar goes with dump (`copi_backup.py:234-256`; design `:316-352`). + Pruner regex is anchored so `run_migration.sh` dumps (`copi_pre0024_...dump`, no `T`/`Z`) are + unreachable (`copi_backup.py:194-201`). +- Flow: flock `/run/copi-backup.lock` — **exits 0 silently if already held** (`:1083,1141-1144`); + sweep leftovers (label-filtered, never `docker volume prune`) (`:1102-1129`; design `:118-127`); + free-space guard `free >= FREE_SPACE_FACTOR × last dump` (`:1292-1315`); per stack: REPEATABLE READ + snapshot session → `pg_dump -Fc --snapshot` into container `/tmp` → `pg_restore -l` in container → + `docker cp` to `.partial` → fsync → atomic rename (`:686-720`; design `:134-196`); verify + (`:735-800`); regression check vs previous sidecar (`:1264-1283`); write status; mail on failure; + prune last (`:1355-1380`). +- CLI (`copi_backup.py:1485-1530`): `copi-backup run [--no-prune]`, `copi-backup report`, + `copi-backup prune [--dry-run]`, `--config` default `/etc/copi-backup/backup.env`. + **`run` has no dry mode** — `run --dry-run` is rejected (`:1489-1506`). +- **Manual pre-deploy dump via the nightly system**: `sudo /usr/local/bin/copi-backup run --no-prune` + (dumps + verifies both stacks, ~800 MB, ~4 min per `...design.md:645-647`; if the timer's run + holds the lock it exits 0 having done nothing `:1141-1144`). Under systemd: + `systemctl start copi-backup.service` is not stated in docs; the design doc records "two full runs + under systemd (`Result=success`)" (`...design.md:657-658`). +- Health check of the nightly (`...design.md:659-663`): + ```bash + systemctl status copi-backup.service + journalctl -u copi-backup.service --since yesterday + sudo jq '{last_run_utc,last_success_utc,ok}' /var/backups/copi/status.json + ``` +- Host has **no postgres client tools**; read a TOC with the image: + `docker run --rm --network none -v :/d.bin:ro postgres:15 pg_restore -l /d.bin` + (`scripts/backup/failure_injection.sh:33-38`; `copi_backup.py:753-764`; design `:227-229`). + TOC check only sees the first ~755 KB; full restore is the real proof (design `:233-253`). +- Limitations: same volume as pgdata; 24h RPO; no PITR ("The pre-deploy dumps in + `run_migration.sh` partially cover this case"); verify ≠ app correctness; unencrypted at rest + (`...design.md:594-617`). `OFFSITE_CMD` never exercised (`:653-654`). + +### Pre-deploy dump alternatives +- `run_migration.sh --apply` Step 3 (see §3): `backups/_pre_.dump`, TOC-verified in + container, copied out, size-checked (`run_migration.sh:170-203`). `backups/` is gitignored + (`.gitignore:91`; `production-migration.md:554`). +- Design-doc manual snapshot (restore runbook step 2): + `docker exec copi-python-postgres-1 pg_dump -Fc -U copi -d copi > /var/backups/copi/pre-restore-$(date -u +%Y%m%dT%H%M%SZ).dump` + (`...design.md:559-560`). (Streams via stdout to a host file — fine for writing; only + `pg_restore -l` over a pipe fails, `run_migration.sh:174-180`.) + +### Restore commands +- Runbook §9 (`docs/production-migration.md:523-538`): + ```bash + docker stop -t 30 agent-run || true + docker compose stop app worker + docker compose cp backups/copi_pre0023_.dump postgres:/tmp/restore.dump + docker compose exec -T postgres psql -U copi -d postgres -c 'ALTER DATABASE copi RENAME TO copi_failed_migration' + docker compose exec -T postgres psql -U copi -d postgres -c 'CREATE DATABASE copi' + docker compose exec -T postgres pg_restore -U copi -d copi --exit-on-error /tmp/restore.dump + docker compose exec -T postgres psql -U copi -d copi -c 'select * from alembic_version' + ``` + "Rename rather than drop"; "`--exit-on-error` is not optional" (`:536-538`). +- Design doc §11 (`...design.md:544-575`): `cd /home/ubuntu/copi-python`; stop `app worker grantbot` + with both `-f` flags; `docker stop agent-run` explicitly (one-off container, not in the compose + invocation) (`:546-556`); pre-restore dump (`:558-560`); + `psql -U copi -d postgres -c "DROP DATABASE copi WITH (FORCE);" -c "CREATE DATABASE copi OWNER copi;"` (`:562-564`); + `docker exec -i copi-python-postgres-1 pg_restore --no-owner --no-privileges --exit-on-error -U copi -d copi < "$DUMP"` (`:566-568`); + `\dt`; `docker compose ... start app worker grantbot` (`:570-574`). Warning: a dump older than + the running image's schema needs migrations re-applied before starting writers (`:577-583`). + "never pass `--profile agent`" when bringing things back (`:554-556`). + +--- + +## 5. agent-run lifecycle + +- One-off container named `agent-run` (`CLAUDE.md:55`). Start variants (`CLAUDE.md:58-70`): + ```bash + C="-f docker-compose.prod.yml -f docker-compose.override.yml" + docker compose $C --profile agent run -d --name agent-run agent python -m src.agent.main --budget 0 + docker compose $C --profile agent run -d --name agent-run agent python -m src.agent.main --budget 50 + docker compose $C --profile agent run -d --name agent-run agent python -m src.agent.main --fresh --budget 0 + docker compose $C --profile agent run -d --name agent-run agent python -m src.agent.main --max-runtime 60 --budget 0 + ``` +- Restart procedure, verbatim (`CLAUDE.md:79-104`): + ```bash + C="-f docker-compose.prod.yml -f docker-compose.override.yml" + # 1. Save logs + docker logs agent-run > logs/run_$(date +%s).log 2>&1 + ls -t logs/run_*.log | tail -n +11 | xargs rm -f + # 2. Stop GRACEFULLY + docker stop -t 30 agent-run + docker rm agent-run + # 3. Rebuild app + worker + docker compose $C up -d --build app worker + # 4. Rebuild the agent image too + docker compose $C --profile agent build agent + # 5. Start the new run + docker compose $C --profile agent run -d --name agent-run agent python -m src.agent.main --budget 0 + ``` +- Graceful-stop requirement: "`docker rm -f` sends SIGKILL, which skips the shutdown flush and + permanently loses the in-flight turn's messages (the DB, not Slack, is the durable store). + `docker stop` sends SIGTERM; -t 30 leaves room for an in-flight LLM call to finish" + (`CLAUDE.md:88-91`; `production-migration.md:193-198`). Code: SIGTERM/SIGINT handler sets a stop + flag (`src/agent/main.py:238-250`); final flush `await sim_engine.stop()` runs in `finally` on + every exit path (`:271-279`). No `stop_grace_period` in prod (§1) → `-t 30` must be explicit. +- In-flight state: buffered messages/LLM logs are flushed to Postgres on stop (`main.py:272-275`); + a SIGKILL loses them. On resume the sim fetches Slack history per bot; expect ~10 min of + `[] Rate limited, retrying in 10s (attempt 1/3)` before `=== Turn 1 ===`; repeated + `attempt 1/3` is progress (`CLAUDE.md:73-77`; log string `src/agent/slack_client.py:333`). +- `--budget N>0` is deprecated: cumulative cap rebuilt from `llm_call_logs` benches agents + permanently; use `--budget 0` (`src/agent/main.py:254-261`). +- Roster changes need no restart — `_sync_roster_from_db` every `ROSTER_POLL_INTERVAL = 30.0` s + (`src/agent/simulation.py:149,4513-4520`; `CLAUDE.md:113-116`). **Code** changes do need a + restart + `build agent` (`CLAUDE.md:108-111`; `production-migration.md:467-468`). +- Start agent-run **last**, after app+worker are on new code (`production-migration.md:457-465`). +- Never `--remove-orphans` (`CLAUDE.md:106`). +- **grantbot**: `restart: unless-stopped` (`prod.yml:115`); scheduler is a `while True: … time.sleep(check_interval)` + loop with `asyncio.run(run_grantbot(...))` once per day when `now.hour >= run_hour` (08 UTC) + (`src/agent/grantbot.py:748-786`); **no signal handler** in that module (grep `signal|SIGTERM` → + none) → `docker stop` kills it at Docker's default 10 s. State file `data/grantbot_last_run.txt` + (`grantbot.py:704`) on the `./data` bind mount (`prod.yml:127`) survives recreation; + `_mark_run_complete()` runs only after a successful daily run (`:772-777`), and "If the container + starts after the scheduled hour, it runs immediately to catch up" (`:758-759`) — recreating + grantbot after 08:00 UTC before its daily run has completed triggers an immediate run. No doc + imposes a graceful-stop constraint on grantbot; the design-doc restore runbook simply + `stop`s/`start`s it with app and worker (`...design.md:547-548,573-574`). +- **worker**: inbound-email rate limiter and quarantine counters are in-memory and reset on restart + "by design" (`docs/inbound-email.md:106-107`). + +--- + +## 6. nginx + +- Config generation: `./nginx/nginx.conf` is mounted read-only at + `/etc/nginx/templates/default.conf.template` (`prod.yml:157`); the stock nginx image's + `/docker-entrypoint.d/20-envsubst-on-templates.sh` substitutes `${DOMAIN}` and writes + `/etc/nginx/conf.d/default.conf` (`nginx/nginx.conf:4-8`). Because the service overrides + `command:`, the entrypoint script is invoked explicitly: + `command: "/bin/sh -c '/docker-entrypoint.d/20-envsubst-on-templates.sh && (while :; do sleep 6h & wait $${!}; nginx -s reload; done &) && nginx -g \"daemon off;\"'"` + (`prod.yml:171`). `DOMAIN` comes from `.env` (`prod.yml:160-161`, `:?` required). + **`${DOMAIN}` is substituted only in the first two server blocks** (`nginx.conf:55,75,78-79`); the + devel and blackbird vhosts are literal hostnames (`:180,199,257,272`). +- Reload without downtime: the mechanism the container itself uses is `nginx -s reload` + (`prod.yml:171`), fired every 6 h. A manual reload is + `docker exec copi-python-nginx-1 nginx -s reload` (container name by compose convention; not + written in any repo doc — MEMORY `nginx-stale-upstream-ip-after-app-recreate.md:20-22` records it + and suggests `nginx -t` first). Note: `nginx -s reload` re-reads the already-rendered + `/etc/nginx/conf.d/default.conf`; **changing `nginx/nginx.conf` or `DOMAIN` requires recreating the + container** so envsubst runs again (`prod.yml:171` runs the template step only at start). +- Upstreams: `upstream app { server app:8000; }` (`nginx.conf:38-40`) — a **static** upstream; + `upstream devel_app { server host.docker.internal:8858; }` (`:45-47`); + `upstream blackbird_app { server blackbird-app:8000; }` (`:249-251`). The `resolver 8.8.8.8 8.8.4.4` + directives (`:94,214`) are for OCSP stapling in the server blocks. +- Three vhosts, each an HTTP→HTTPS redirect + ACME block and an HTTPS block: + 1. `${DOMAIN}` — `:52-66` (80) and `:71-172` (443): proxies `/` to `http://app` with rate limits + (`limit_req zone=req_general burst=40`), tighter `req_graph` for + `^/(cabo-graph|scripps-graph|schultz-alumni-pilot|schultz-group-alumni)$` (`:111-125`), + PostHog proxies `/ingest/static/`, `/ingest/` (`:153-164`), `/_next/static/` cache (`:167-171`). + Certs `/etc/letsencrypt/live/${DOMAIN}/{fullchain,privkey}.pem` (`:78-79`). + 2. `devel.copi.science` — `:177-189` / `:195-243`: proxies to `devel_app` (host:8858 via + `host.docker.internal`); uses the `copi.science` SAN cert (`:201-202`). + 3. `blackbird.copi.science` — `:254-265` / `:268-307`: proxies to `blackbird_app` over `copi-edge`; + certs `/etc/letsencrypt/live/blackbird.copi.science/` (`:274-275`). +- Rate-limit zones/http-context directives at top (`:29-35`); `limit_req_status 429`. +- Certs live on the host at `./certbot/conf` (→ `/etc/letsencrypt`) and ACME webroot `./certbot/www` + (`prod.yml:158-159,184-185`); `certbot/` is gitignored (`.gitignore:59`) and absent from this + checkout (`ls certbot/` → no such directory) — host-only state. Renewal loop: `certbot renew --quiet` + every 12 h (`prod.yml:186`). `copi-python-certbot-1`'s volume is anonymous (`scripts/ci.sh:170-171`). +- nginx `depends_on: app: condition: service_healthy` (`prod.yml:162-164`): nginx cannot start + until app is healthy; if app is absent nginx crash-loops on `host not found in upstream "app:8000"` + (`CLAUDE.md:33-34`). +- MEMORY (`nginx-stale-upstream-ip-after-app-recreate.md:8-29`): recreating `app` gives it a new + bridge IP; the static upstream keeps the old IP → site-wide 502 until `nginx -s reload` (up to 6 h + self-heal via `prod.yml:171`). Fingerprint: nginx healthcheck `unhealthy` while app `healthy`. + Observed 2026-08-18, ~65 min of 502s. Repo-side mechanism confirmed at `nginx.conf:38-40` and + `prod.yml:171`; the incident itself is memory, not repo. + +--- + +## 7. Post-deploy health / verification commands + +- App health route: `GET /api/health` → `{"status": "ok"}` (`src/main.py:150-153`); used by the + compose healthcheck (`prod.yml:46`). Through nginx: `https:///api/health`. +- Compose/Docker state: `docker compose $C ps` (healthchecks: postgres `prod.yml:11-16`, app + `:45-50`, nginx `:165-170`); restart policy check + `docker inspect copi-python-app-1 -f '{{.HostConfig.RestartPolicy.Name}}'` → `unless-stopped` + for all six (`CLAUDE.md:47-51`). +- Migration state: `docker compose exec -T postgres psql -U copi -d copi -c 'select * from alembic_version'` + (`production-migration.md:521`); `docker compose exec app alembic heads` must print one line + (`README.md:45`); `alembic current` (`README.md:47`). Read-only Q1–Q4 SQL (`production-migration.md:94-124`); + disk `docker compose exec -T postgres df -h /var/lib/postgresql/data` (`:132`). +- `run_migration.sh` Step 1 self-test that the container runs current code: + `docker compose exec -T -e PYTHONPATH=/app app python -c 'import src; print(src.__file__)'` → + `/app/src/__init__.py` (`run_migration.sh:114-117`). +- Admin pages (all under `/admin`, `src/main.py:146`): `/admin/users`, `/admin/users/{id}`, + `/admin/jobs`, `/admin/activity`, `/admin/activity/{run_id}`, `/admin/activity/{run_id}/llm-calls`, + `/admin/discussions`, `/admin/agents`, `/admin/agents/{id}`, `/admin/access-requests`, + `/admin/waitlist`, `/admin/cohorts`, `/admin/cohorts/topology`, `/admin/cohorts/{id}` + (`src/routers/admin.py:76,174,235,277,326,402,507,786,862,1151,1316,1475,1550,1717`). + Agent provisioning UI at `/admin/agents → Provision → Approve & Activate` (`CLAUDE.md:144-150`). +- Agent log greps (`docs/issue-29-remediation.md:51-65`): + ```bash + docker logs agent-run 2>&1 | grep -iE "Rejected (draft|reply to thread)|Suppressed post to|stripped ungrounded" + docker logs agent-run 2>&1 | grep -i "publication-record load failed" # MUST be empty + ``` + plus per-agent grouping `grep -oE "^\[[a-z]+\]" | sort | uniq -c`. Resume progress: `=== Turn 1 ===` + and `Rate limited, retrying` lines (`CLAUDE.md:73-77`). +- Worker/inbound email: `docker logs -f copi-python-worker-1` for `Email review created` + (`docs/inbound-email.md:91-94`); `python scripts/setup_inbound_email.py --check` (admin AWS creds) + (`inbound-email.md:69-71`). +- Backup system: `systemctl status copi-backup.service`, `journalctl -u copi-backup.service --since yesterday`, + `sudo jq '{last_run_utc,last_success_utc,ok}' /var/backups/copi/status.json` (`...design.md:659-663`); + `systemctl --failed` (`:404`). +- Roster sync confirmation: INFO line fires on change (`src/agent/simulation.py:381`). + +--- + +## 8. Rollback + +### Schema (documented) +- "`alembic downgrade` is not a rollback. It either destroys data silently or refuses to run. Your + rollback is a restore from the dump" (`production-migration.md:27-29`, §9 `:472-541`). + Measured: with no `agent_id IS NULL` rows `alembic downgrade 0018` exits 0, keeps row count, + and drops `content`, `pi_dm_messages`, `cohorts` (`:478-491`); with any PI row it fails + `NotNullViolationError` on `ALTER TABLE agent_messages ALTER COLUMN agent_id SET NOT NULL` and + rolls back cleanly (`:493-507`). Door-closing analysis: after new code runs, + `_rebuild_state_from_slack` writes `agent_id=NULL` rows and downgrade past 0019 stops being + possible (`docs/specs/2026-08-10-org1-parity-design.md:428-435`). +- If postflight fails: do not deploy code; nothing half-applied (one transaction); restore per §4 + (`production-migration.md:511-541`). + +### `downgrade()` per revision (all six have a body) +| rev | downgrade | guarded? | notes | +|---|---|---|---| +| 0019 | drops 3 indexes, `uq_agent_messages_run_ts`, sets `agent_id` NOT NULL, drops 7 columns (`0019_agent_message_content.py:73-85`) | **no** `if_exists` | fails if any `agent_id IS NULL`; otherwise silently destroys message bodies | +| 0020 | drops 2 indexes, `pi_dm_messages`, enum `pi_dm_direction_enum` with `checkfirst=True` (`0020_pi_dm_messages.py:62-66`) | enum only | drops all PI DMs | +| 0021 | drops `ix_pi_dm_run_direction_created`, `ix_agent_messages_run_created` (`0021_...py:37-39`) | no | index-only, non-destructive | +| 0022 | drops 4 indexes + 3 tables, all `if_exists=True` (`0022_add_cohorts.py:126-149`) | yes | drops all cohorts/memberships/audit | +| 0023 | drops 3 columns `if_exists=True` (`0023_...py:59-62`) | yes | loses provenance values | +| 0024 | `op.drop_column("agents", "role", if_exists=True)` (`0024_add_agent_role.py:34-35`) | yes | loses role assignments | + +CI proves the chain round-trips upgrade→downgrade→upgrade on a throwaway DB +(`scripts/ci.sh:229-231`; `docs/specs/2026-08-10-org1-parity-design.md:439-441`). + +### Code / image rollback +**Not covered by any document.** No doc describes checking out a previous commit and rebuilding, image +tagging, or keeping the previous image. Images are built from the working tree each time +(`prod.yml:26-27`, `Dockerfile:17`) and are untagged beyond compose's default `copi-python-app` +(`scripts/seed_cohorts.py:30`). The only stated directional safety is "old code keeps working against +the new schema" (`production-migration.md:445-448`) with one known gap (private-channel close marker +not mirrored, `:450-455`). + +### nginx / certs rollback +Not documented. Config is a tracked file (`nginx/nginx.conf`) rendered at container start (§6). + +--- + +## 9. Environment variables (names only) and where they live + +- **Location**: `.env` in the repo root on the host (`/home/ubuntu/copi-python/.env`), consumed via + `env_file: .env` (`prod.yml:32,64,88,117`) and by compose interpolation for `${POSTGRES_PASSWORD}`, + `${SECRET_KEY}`, `${DOMAIN}`, `${ENVIRONMENT}`, `${AWS_REGION}`, `${POSTGRES_USER}`, `${POSTGRES_DB}` + (`prod.yml:6-8,23,34-38,161`). `.env` is gitignored (`.gitignore:20`) and dockerignored + (`.dockerignore:16-17`). pydantic also reads `.env` directly (`src/config.py:99`, + `SettingsConfigDict(env_file=".env", extra="ignore")`). `docs/inbound-email.md:84-90`: changing + `.env` requires `up -d` to recreate — `docker restart` re-runs the OLD environment. +- **Backup system env**: `/etc/copi-backup/backup.env` (host, root 0600) (`install.sh:18-21`; + design `:71`), deliberately duplicating SES settings so an app `.env` change cannot break backups + (`...design.md:105-107`). +- **Compose-level names**: `POSTGRES_USER` (default `copi`), `POSTGRES_PASSWORD` (required), + `POSTGRES_DB` (default `copi`), `SECRET_KEY` (required), `ENVIRONMENT` (default `production`), + `DOMAIN` (required by nginx), `AWS_REGION` (default `us-east-2`, awslogs only) (`prod.yml:6-8,34-38,161,23`). + `COMPOSE_FILE` in the operator's shell (`CLAUDE.md:44`); MEMORY `org1-prod-access-and-local-copy.md:23` + says local dev sets `COMPOSE_FILE=docker-compose.yml` in `.env` — verify what prod's `.env` says + before relying on `-f` flags vs `COMPOSE_FILE` precedence. +- **`.env.example` names** (`.env.example:2-51`): `ORCID_CLIENT_ID`, `ORCID_CLIENT_SECRET`, + `ORCID_REDIRECT_URI`, `DATABASE_URL`, `ANTHROPIC_API_KEY`, `NCBI_API_KEY`, `ENVIRONMENT`, + `SECRET_KEY`, `BASE_URL`, `ALLOW_HTTP_SESSIONS`, `SLACK_BOT_TOKEN_` / `SLACK_APP_TOKEN_` + pairs. (Note: `.env.example` lacks `POSTGRES_PASSWORD` and `DOMAIN`, which `prod.yml` requires.) +- **`src/config.py` Settings fields (name = default)** — env var is the upper-cased name: + `environment="development"` (`:103`); `orcid_client_id=""`, `orcid_client_secret=""`, + `orcid_redirect_uri="http://localhost:8000/auth/callback"` (`:106-108`); + `database_url="postgresql+asyncpg://copi:copi@localhost:5432/copi"` (`:111`); + `anthropic_api_key=""` (`:114`); `ncbi_api_key=""`, `ncbi_contact_email=""` (`:117-120`); + `secret_key=INSECURE_SECRET_KEY` (`:123`; guard raises unless `environment` ∈ dev set, `:428-450`); + `base_url="http://localhost:8000"` (`:124`); `allow_http_sessions=False` (`:128`); + `slack_config_token=""`, `slack_config_refresh_token=""` (`:134-135`; rotated pair persisted in + `app_settings` KV, `:130-133`, `CLAUDE.md:152-153`); `slack_enabled: bool|None=None` (None = + auto-detect from tokens; `false` forces DB-only) (`:137-141`); `aws_region="us-east-2"`, + `ses_sender_email="noreply@copi.science"`, `ses_reply_domain="reply.copi.science"`, + `ses_inbound_s3_bucket="copi-inbound-email"`, `ses_inbound_s3_prefix="inbound/"` (`:144-148`); + `outbound_email_allowlist=""` (empty = everyone) (`:150`); `enable_inbound_email=False` (`:152`); + `audit_recipients="asu@…,malanjary@…,ahuebschen@…"` (`:155`); `notification_check_interval=300`, + `inbound_poll_interval=60` (`:158-159`); ~125 `slack_bot_token_=""` incl. + `slack_bot_token_grantbot` (`:162-290`; DB column `AgentRegistry.slack_bot_token` is authoritative, + `.env` is fallback, `CLAUDE.md:169-170`); `posthog_api_key=""` (`:293`); + `llm_profile_model="claude-opus-5"`, `llm_agent_model="claude-sonnet-5"`, + `llm_agent_model_opus="claude-opus-5"`, `llm_agent_model_sonnet="claude-sonnet-5"` (`:296,317-319`); + `worker_poll_interval=5` (`:322`); `active_thread_threshold=3`, `unreviewed_proposal_block_count=2`, + `max_thread_messages=12`, `interesting_posts_cap=20`, `turn_delay_seconds=0.0`, `daily_post_cap=5`, + `max_abstracts_other_per_thread=10`, `max_full_text_per_thread=2` (`:325-335`); + `cohort_isolation_enabled=False`, `cohort_default_policy="open"` (`:342,353`); + `max_consecutive_reactive_turns=3` (`:360`); `llm_rate_window_seconds=600`, + `llm_calls_per_load_per_window=8` (`:382-383`); `enable_private_refinement=True` (`:391`). +- `ALEMBIC_LOCK_TIMEOUT_MS` (default 10000) read by `alembic/env.py:66`; `MIGRATE_BACKUP_DIR`, + `MIGRATE_SERVICE`, `MIGRATE_PG_SERVICE`, `MIGRATE_SNAPSHOT`, `DATABASE_URL` read by + `run_migration.sh:57-61,211`. `PYTHONPATH=/app` must be passed for any `scripts/*.py` in-container + (`production-migration.md:557-558`). +- `SLACK_CONFIG_TOKEN`/`SLACK_CONFIG_REFRESH_TOKEN` + public `base_url` needed for admin-UI + provisioning (`CLAUDE.md:152-153`). `ENABLE_INBOUND_EMAIL` unset on prod as of 2026-08-11 + (`docs/inbound-email.md:41-42`). + +--- + +## 10. Documented gotchas + +1. **2026-08-06 restart-policy incident**: stack had been recreated from the dev compose file + (`restart: no`); host freeze rebooted the box; app/worker/postgres/grantbot stayed dead; nginx + crash-looped on `host not found in upstream "app:8000"` (`CLAUDE.md:29-34`; + `docker-compose.override.yml:6-9` — the override had been added 2026-05-26 but never committed and + was deleted from the tree). +2. **awslogs / AccessDeniedException** without the override file (`CLAUDE.md:36-39`; `override.yml:1-4`). +3. **`--remove-orphans` deletes prod nginx/certbot** (`CLAUDE.md:106`). +4. **Prod bakes agent code**: `build agent` required, restart alone runs stale code (`CLAUDE.md:98-100,108-111`). +5. **Bind mounts** `./profiles`, `./prompts`, `./data` shadow image copies for app/agent/grantbot; + worker mounts only `./profiles` (`prod.yml:39-41,71-72,95-98,124-127`). Prompt edits are live + without rebuild for mounted services but need a process restart to reload + (`production-migration.md:467-468`). +6. **Slack rate limiting on resume**: ~10 min of `Rate limited, retrying in 10s (attempt 1/3)` + before Turn 1 (`CLAUDE.md:73-77`). +7. **SIGKILL loses the in-flight turn** — use `docker stop -t 30 agent-run` (`CLAUDE.md:88-93`; + `production-migration.md:193-198`); prod agent has no `stop_grace_period` (§1). +8. **Bare `docker compose` in the migration tooling and runbook** (§2/§3): `run_migration.sh` + and `production-migration.md:442,460` rely on `COMPOSE_FILE` being exported; the org1-parity spec + calls it "not optional" (`org1-parity-design.md:418-420`). +9. **run_migration.sh Step 1 requires the running `app` container to carry the new source** + (`run_migration.sh:109-128`) — the image must be rebuilt from the deploy commit before migrating, + even though the runbook's rule is "migrate before deploying code" (`production-migration.md:24-26`). +10. **DSN default trap**: `alembic.ini:5` localhost default; only `DATABASE_URL` overrides + (`env.py:21-23`); `postgres` hostname resolved to a public IP from the dev host + (`production-migration.md:30-40`). Postgres is not published to the host (`:34-35`). +11. **Alembic output is not evidence** — read `alembic_version` back (`production-migration.md:41-44`; + `env.py:69-77`; `run_migration.sh:268-293`). +12. **`PYTHONPATH=/app`** for every in-container script or `import src` hits stale site-packages + (`run_migration.sh:99-105`; `production-migration.md:341-345`). +13. **Lock timeout** 10 s default; idle-in-transaction sessions block the migration and everything + behind it; stop writers and re-run (`production-migration.md:176-206`). +14. **`docker system prune` / `docker volume prune` on the host destroy unreferenced production + volumes** (`copi_pgdata`, `copi-prod_pgdata`, `copi-python_grantbot_data`, + `collab-platform_mongodb_data`) (`...design.md:124-127`; `scripts/ci.sh:165-171`; + `production-migration.md:137-140`). MEMORY `org1-prod-state-2026-08-14.md:19`: prune also deletes + stopped `agent-run` containers and their logs — save logs first. +15. **`docker restart` re-runs the OLD environment**; `.env` changes need `up -d` (`inbound-email.md:88-90`). +16. **07:00 UTC daily Claude cron** on the host (`...design.md:470-473`); backup timer at 01:00 + America/Los_Angeles (`copi-backup.timer:5`) — avoid deploy windows overlapping either. +17. **Host RAM is tight**: 3.7 GB, ~1.2 GB swapped, prior OOM kills (`...design.md:624-626`). +18. **Legacy rows after 0019 have `content=''`, `posted_at=0`**; step 8 `backfill_slack_ts.py` + (report then `--apply`) needs valid bot tokens; exit 2 = UNVERIFIED, not done + (`production-migration.md:397-437`). +19. **`copi-backup run` silently exits 0 if the flock is held** (`copi_backup.py:1141-1144`) — a manual + run overlapping the timer does nothing; check `status.json`/journal. +20. **grantbot catch-up run** fires immediately if the container starts after 08:00 UTC on a day it + has not run (`grantbot.py:758-759,769`). +21. **`--budget N`** deprecated; use `--budget 0` (`src/agent/main.py:254-261`). +22. **`.env.example` is incomplete for prod** — lacks `POSTGRES_PASSWORD`, `DOMAIN` that + `prod.yml` requires (`prod.yml:7,161`). +23. **`README.md` is stale/dev-only** (`PILOT_LABS` at `README.md:99` vs `CLAUDE.md:121-122`). +24. `SECRET_KEY` guard: with `ENVIRONMENT` defaulting to `production` in prod compose (`prod.yml:38`), + a missing/default `SECRET_KEY` makes the app refuse to start (`src/config.py:428-450`). +25. `run_migration.sh` hardcodes `pg_dump -U copi` (`:182`) and the runbook hardcodes `-U copi -d copi` + throughout; `MIGRATE_PG_SERVICE`/`MIGRATE_SERVICE` cover service names only (`:59-60`). +26. `pre-push` hook is the CI gate; MEMORY `org1-prod-state-2026-08-14.md:17`: the hook's throwaway + Postgres defaults to port 55432 (`scripts/ci.sh:60`) and can collide with a long-lived container — + `MIGCHECK_PORT= git push`. diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/README.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/README.md new file mode 100644 index 00000000..50f91553 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/README.md @@ -0,0 +1,11 @@ +# Audit of GitHub issues #20–#27 (author: ahueb) vs copi-prod @ 18ba52c — 2026-09-02 + +Files: +- issues/issue_NN.md — full issue text as fetched from GitHub (no comments exist on any of them) +- findings/issue_NN.md — first-wave verification report (one agent per issue; every sub-claim, evidence, snippets run) +- findings/issue_NN_redteam.md — second-wave adversarial report attacking every first-wave verdict +- ci/ci_run2.log — full ./scripts/ci.sh output (MIGCHECK_PORT=55433): CI passed, 2030 passed / 120 skipped, 69.27 % + +Repo facts: branch copi-prod, HEAD 18ba52c, clean tree, == origin/copi-prod. main is stale at b7edcbc (the original +audit baseline). copi-prod contains merged PRs #30, #31, #32, #36 and 52 later commits (backup tooling, cohort seeding). +No commit since the issues' re-verification tip b1d54da references any of #20–#27 or any COR-/PR item. diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_20.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_20.md new file mode 100644 index 00000000..a67c5848 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_20.md @@ -0,0 +1,218 @@ +# Issue #20 — Agent engine: turn & thread state-machine correctness — verification against `copi-prod` @ 18ba52c + +Tree: /home/a/scripps/coPI.science, branch `copi-prod`, HEAD 18ba52c, clean. All line numbers below are CURRENT (they differ from the issue's `b1d54da` numbers by roughly +40..+110 in `simulation.py`). Snippets were run with `.venv-test/bin/python` against the real modules; outputs are quoted verbatim. + +## 1. Summary table + +| id | claim (one line) | verdict | key evidence (current file:line) | conf | +|---|---|---|---|---| +| COR-1a | `_post_message -> bool`; 4 call sites gate on it; ThreadNotFound path returns False before any LogEntry | FIXED (commit 6af8207) | simulation.py:3280-3286 signature; 1494-1498, 2370-2371, 2390-2394, 2432-2433 guards; 3356-3366 ThreadNotFound `return False` | high | +| COR-1b | swallowed non-`thread_not_found` `SlackApiError` → local id minted, row persisted, returns **True** | STILL PRESENT | slack_client.py:704-712 `_post_one` returns None; :815-816 `post_message` returns None; simulation.py:3400 `_mirrored_messages(None)`→`[]`; 3423 `enumerate(mirrored or [None])`; 3447 `return True`. Ran: `_mirrored_messages(None,"hello",None) -> []`, loop iterates `[None]` | high | +| COR-1c | `_evict_dead_thread` never purges the message log and `discard`s `_closed_thread_ids` (un-closes) | STILL PRESENT — and pinned by a test | simulation.py:1788-1825; :1818 `self._closed_thread_ids.discard(thread_id)`; tests/unit/test_thread_not_found.py:144 asserts `dead_ts not in engine._closed_thread_ids` | high | +| COR-1d | `_check_private_channel_outcome` runs outside the `posted` guard | STILL PRESENT | simulation.py:2455-2462 — gated on `message_text` + channel visibility only, after both `if not posted` branches | high | +| COR-3 | public ✅ finalizes against the first prior other-agent `:memo:` in reversed history, no recency check | STILL PRESENT | simulation.py:1527-1531; private sibling 1687-1700 skips only the handover post | high | +| COR-4 | public path matches raw ✅ only; private accepts ✅ and `:white_check_mark:`; pause accepts both forms | STILL PRESENT | :1527 `if "✅" in latest_reply`; :1563 `"⏸️" ... or ":pause_button:"`; :1674 `"✅" not in ... and ":white_check_mark:" not in ...` | high | +| COR-7 | in-process `_prior_threads` append has no dedup and no `thread_id`; DB rebuild is guarded + tested | STILL PRESENT (in-process); rebuild FIXED (02f5749) | :1583-1589 append dict {channel,outcome,summary}; :4166-4185 rebuild guard on `_closed_thread_ids`; tests/integration/test_state_rebuild.py:208-231 | high | +| COR-6 | `last_seen_cursor = time.time()` vs `posted_at <= since` filters; `latest_timestamp` exists; Phase 4 comment concedes | STILL PRESENT (issue detail about rebuild is stale) | :1033 `time.time()`; message_log.py:199/311/339/435 `<= since`; :445-455 `latest_timestamp` property; simulation.py:1302 + 1311-1315 comment; rebuild :4400-4403 computes `max(e.posted_at...)` inline, does NOT call `latest_timestamp` | high | +| COR-2 | Phase-3 activation builds `ThreadState` without `message_count_offset`; reopen paths set it; funding threads open-to-all | STILL PRESENT | :1218-1225 (tag), :1264-1271 (reply) no offset; :1348 recompute; :1359-1365 close "timeout" at `>= max_thread_messages` (=12, config.py:327); offsets set at :2989/:3000/:5131/:5142; message_log.py:363-364 funding→None | high | +| COR-5 | `_check_pi_proposal_review` matches thread_id only, any sender, flips all agents, persists nothing; Slack site outside PI loop; web thread_ts free-form; any PI clears another lab's block | STILL PRESENT | :2941-2954; Slack call :2797 precedes `for pi_agent_id in pi_agent_ids` :2800; DB path :2913; agent_page.py:952 `thread_ts: str = Form("")`, :991-998; pi_inbox.py:52-101 allows any non-private/unknown channel; dashboard agent_page.py:251-268 + email_notifications.py:171-173,763-764 read `ProposalReview`; rebuild :4297 `(td.id, aid) in reviewed_set` | high | +| COR-13 | rebuild keys `(thread_decision_id, agent_id)`, tick keys `(agent_id, thread_id)`; `ProposalRef` lacks `thread_decision_id`; `_db_reopened_thread_ids` never seeded; reopen persists a synthetic PI row per restart; Slack `_reopen_thread` writes nothing durable; rebuild re-closes | STILL PRESENT | :4264/:4297 vs :5007/:5054; state.py:50-58; :337 init, :5075 check, :5145 add (only 3 refs); :5106-5117 mint+append; :5064-5066 "independent of the reviewed flag"; :2956-3003 no DB write; :4160-4165,4186 | high | +| COR-10(1) | `_poll_pi_dms` calls `poll_dm_messages` unguarded; `_call_with_retry` catches only `SlackApiError`; poller outside turn try → transient socket error kills the sim | STILL PRESENT | :3028 unguarded; slack_client.py:310-341 `except SlackApiError` only; :505-550 `poll_channel_messages` catches `SlackListingIncomplete`/`SlackApiError` only; main loop :683 vs per-turn try :742-745; main.py:272-273 logs and falls to `finally`. slack_sdk 3.43.0 `base_client.py:483-515` re-raises the raw `err` | high | +| COR-10(2) | no DM-poll throttle — rated low | N/A (agree: low) | :3005-3046 per-agent token; slack_client.py:_dm_channels cache :825-827; idle backoff :629-640 | high | +| COR-10(3) | `_poll_inbound_from_db` advances cursor + appends before the unguarded handler; lookback dedups on the log entry → crash + permanent loss | STILL PRESENT (also lost across restart) | :2877-2878 cursor, :2879 dedup, :2893 append, :2898 `await self._handle_pi_inbound_entry` unguarded; try/except :2855-2874 covers query only; `_seed_pi_inbox_cursor` :3764-3785 seeds to max(created_at) on restart | high | +| COR-11 | `_flush_llm_logs` clears buffer before write; except only logs; under-counts the sliding window after restart | STILL PRESENT | :4452-4453 `batch = buf[:]; buf.clear()` before `try`; :4474-4475 warning only; contrast `_flush_persisted` re-queue :3946-3950; `call_times` rebuilt from `llm_call_logs` :4352-4393 | high | +| COR-9a | visibility mislabel fixed: `_post_message` resolves from channel and persists | FIXED (commit d311170) | :3419 `visibility = self._resolve_channel_visibility(channel)`; :3441 stamped on LogEntry | high | +| COR-9b | reply channel is `action_data.get("channel","general")`, never reconciled with the target post; private→public path | STILL PRESENT (nuanced: persisted leak on Slack-off; collateral eviction on Slack-on) | :2264 channel from LLM; :2366-2368 `is_private_channel` from that channel; :2390-2393 posts to `channel` with `thread_ts=target_post_id`; :2286-2294 `is_private_reply` reads `target_entry.channel` but only to unblock; no cross-check in `_post_message` :3280-3448 / `_slack_parent_ts` :3463-3479; acknowledged in tests/integration/test_full_run_live.py:41-44 | high | +| COR-9c | memory-synthesis strip downgraded to hygiene | CHANGED / agree (no live trigger) | :5262-5396 prompt never asks for ``; :5349 `strip_ungrounded_authorship_lines` (b6c2de4); no tag strip on that path | high | +| COR-8 | `<@Uxxx>` undetected; regex `@(\w+[Bb]ot)\b` case-blind above `[Bb]`; literal PI check; `bot_uid_to_agent` only for attribution; web UI synthesizes literal | STILL PRESENT (plus 2 more regex copies the issue missed) | message_log.py:407; funding_rules.py:154; **simulation.py:2523 and :2558** (same pattern); PI literal :2831-2832; `_bot_uid_map` :4004-4019 used at :4036/:4067/:4123; agent_page.py:975-976; `grep -rn '<@' src/` → 0 hits. Ran regex: `<@U12345>`→None, `@subot`→su, `@SUBOT`→None, `@SuBot`→su | high | +| E6(1) | no mid-turn rate check; Phase 4 fans out in one `gather` with per-retry booking | STILL PRESENT | `_within_rate_limit` called only at :891/:893 (`_turn_eligible`); `record_api_call()` at :1099,:1145,:1414,:2217,:5323 + `on_retry=agent.record_api_call` :1428; `asyncio.gather(*tasks, return_exceptions=True)` :1330-1334 | high | +| E6(2) | roster re-add builds a fresh `Agent` → pending_proposals/threads/cursors/call_times/throttled lost; no rebuild follows | STILL PRESENT (also resets `last_seen_cursor` to 0.0 → full rescan) | :4658 `agent = Agent(agent_id=aid, ...)`; agent.py:87 `self.state = AgentState()`; after-add only `set_bot_name_map`/`_load_pi_mappings`/`_recompute_allowed_sender_ids` :4665-4675; tests/unit/test_roster_sync.py:127 asserts presence only | high | +| E6(3) | `total_api_calls` recomputed from live roster, non-monotonic | STILL PRESENT (cosmetic, as labelled) | :3939; main.py:293; comment :186-191 | high | +| E7a | daily-cap gate returns before PI-priority/private/funding bypasses are computed | STILL PRESENT | :2043-2046 return; `has_pi_priority` :2061; bypasses :2070-2091 | high | +| E7b | `has_pi_directive` cleared unconditionally at `_run_turn` scope | STILL PRESENT (wording: "throttled" turns never reach `_run_turn`; "capped/skipped/blocked" is the real path) | :1030 (origin 713aa20); consumed only at :1007 as a Phase-5 trigger; never read by any prompt builder (no consumers outside simulation.py/state.py); Phase 5 bails at :2044 (cap), :2063 (random skip — `has_pi_priority` is PostRef-based), :2107 (blocked) | high | +| E7c | `thread.pi_context` set at four sites, never cleared in-process; rebuild omits it; re-injected as authoritative | STILL PRESENT | set :2826,:2931,:2988,:5130; `grep "pi_context = None"` → 0; rebuild ThreadState :4165-4171 omits; agent.py:461-467 injects | high | +| E7d | `interesting_posts` swap/restore both precede the `try` | STILL PRESENT | swap :2134-2135; restore :2215; `try` :2218; `_run_turn` is inside loop try :742 so a raise leaves state narrowed | high | + +## 2. Per-item detail + +### PR E1 + +**COR-1a (FIXED, 6af8207 "fix(sched): suppress a post that strips to nothing, and tell the caller").** +``` +3280 async def _post_message(self, agent_id, channel, text, thread_ts=None) -> bool: +3356 except ThreadNotFound: +3361 if thread_ts: +3362 self._evict_dead_thread(thread_ts) +3366 return False +``` +Callers: Phase 4 `posted = await self._post_message(...)` / `if not posted:` (1494-1498), Phase 5 private (2370-2371), Phase 5 threaded reply (2390-2394), Phase 5 top-level (2432-2433). All four skip `message_count += 1`, backoff resets and the `interesting_posts`→`active_threads` move when `posted` is False. + +**COR-1b (STILL PRESENT).** `_post_one` (slack_client.py:704-712): +``` + except SlackApiError as exc: + err = exc.response.get("error") + if err == "thread_not_found" and thread_ts and may_raise_thread_not_found: + raise ThreadNotFound(...) + if err in ("channel_not_found", "not_in_channel") and self._is_private_channel(channel_id): + raise BotNotInvitedToPrivateChannel(...) + logger.error("[%s] Failed to post to #%s: %s", ...) + return None +``` +`post_message` (815-816): `if not posted: return None`. Back in `_post_message`: `result = client.post_message(...)` → None; `mirrored = self._mirrored_messages(result, text, slack_parent)` (3400) → `[]`; `for index, message in enumerate(mirrored or [None]):` (3423) → one iteration with `message=None` → `ts = slack_ts or self.mint_ts()` → `LogEntry(... slack_ts=None ...)` appended → `return True` (3447). Ran: +``` +mirrored for result=None -> [] | loop iterates over: [None] +``` +So a connected client whose `chat.postMessage` failed with e.g. `msg_too_long`, `is_archived`, `not_in_channel` (public), `invalid_auth` is indistinguishable from the Slack-off mock path: turn counted, `_check_thread_outcome` runs (can close the thread / mint a ProposalRef / DM the PI) for a message that is not on Slack. No test asserts either behaviour (`tests/unit/test_slack_web.py`, `test_transport.py` cover other aspects of `_post_message`). + +**COR-1c (STILL PRESENT).** `_evict_dead_thread` (1788-1825) pops active_threads / interesting_posts / pending_proposals per agent, pops the proposal-thread poll cursor, then `self._closed_thread_ids.discard(thread_id)` (1818). No `message_log` purge. `tests/unit/test_thread_not_found.py:131` adds the ts to `_closed_thread_ids` in the fixture and `:144` asserts it is gone afterwards — the test pins the un-close. Low severity as the issue says (no phantom entry any more), but a fix that stops discarding will break that test. + +**COR-1d (STILL PRESENT).** 2455-2462: +``` + if ( + message_text + and self._channel_visibility.get(channel) == VISIBILITY_COLLAB_PRIVATE + ): + await self._check_private_channel_outcome(agent, channel, message_text) +``` +sits after both `posted` if/else blocks and does not read `posted`. A suppressed private post carrying ✅ can still call `_finalize_private_proposal` (writes a `ThreadDecision`, blocks both bots, DMs the PI). + +**COR-3 (STILL PRESENT).** 1527-1531: +``` + if "✅" in latest_reply: + history = self.message_log.get_thread_history(thread.thread_id) + for entry in reversed(history): + if entry.sender_agent_id == thread.other_agent_id and ":memo:" in entry.content: +``` +First match wins — nothing requires that memo to be newer than this agent's last message or to be the latest other-agent message. The private analogue (1687-1700) also takes the most-recent other-member memo but adds the handover skip ("without this a casual ✅ could finalize the un-revised proposal"). `tests/integration/test_proposal_review.py:237` drives only the happy path (`✅` immediately after the memo). + +**COR-4 (STILL PRESENT).** Three marker checks, two dual-form: +- 1527 `if "✅" in latest_reply:` (public, single form) +- 1563 `if "⏸️" in latest_reply or ":pause_button:" in latest_reply:` (pause, dual) +- 1674 `if "✅" not in message_text and ":white_check_mark:" not in message_text:` (private, dual — introduced by af94bf8) +Mitigation check: the prompts instruct the unicode form (agent.py:48-56, prompts/phase4-thread-reply.md:126, prompts/agent-system.md:178), and both checks run on the LLM's raw output (pre-Slack), so exposure is the model writing the shortcode on its own. Consequence remains "missed finalization → timeout close", medium. + +**COR-7 (STILL PRESENT in-process).** 1583-1589: +``` + pair_key = tuple(sorted([agent.agent_id, thread.other_agent_id])) + self._prior_threads.setdefault(pair_key, []).append({ + "channel": thread.channel, "outcome": outcome, + "summary": (summary_text or "")[:400] or None, + }) +``` +No `thread_id`, no dedup. The `pending_proposals` sibling at 1601-1612 filters by `p.thread_id`. The rebuild path (4166-4185, commit 02f5749) is guarded by `_closed_thread_ids` and tested (`test_a_second_rebuild_does_not_duplicate_prior_thread_context`, test_state_rebuild.py:208-231). The rebuild comment explicitly says a thread with several decision rows "still contributes each of them on the first pass", so duplicates after propose→reopen→re-propose reappear on restart as the issue states. + +### PR E2 + +**COR-6 (STILL PRESENT; one stale detail).** `_run_turn` ends with `agent.state.last_seen_cursor = time.time()` (1033, present since 497ec82). Filters: message_log.py:199, 311, 339, 435 all `if entry.posted_at <= since: continue`. `latest_timestamp` exists (445-455) backed by `_max_posted_at` (baa5583). Phase 4 uses the cursor at 1302 and its comment (1311-1315) reads "The cursor advances unconditionally each turn, so has_new can't be relied on for retry". External writers stamp `posted_at` from their own minter: `record_pi_message` pi_inbox.py:120-134 `posted_at=float(ts)` with `ts = mint_local_ts()`; grantbot.py:174-179 same. message_log.py:222-223 flags "a writer's clock can run behind — see PI_INBOX_LOOKBACK_S". Loss window as described: a row committed with `posted_at < cursor` but ingested by `_poll_inbound_from_db` after the turn ends is filtered forever; PI rows get direct side effects via `_handle_pi_inbound_entry` (mitigation), bot rows (GrantBot) do not. Partial mitigation not in the issue: `_rewind_cursors_for_private_channels` (1922-1987) rewinds member-bot cursors for private channels only. +*Issue text stale:* the rebuild (4400-4403) does `latest_ts = max(e.posted_at for e in self.message_log._entries)` inline — it does not call `latest_timestamp`. Same value, different mechanism. + +**COR-2 (STILL PRESENT).** Phase 3 tag path (1218-1225) and reply path (1264-1271) construct `ThreadState(thread_id, channel, other_agent_id, message_count=self.message_log.get_thread_message_count(thread_id), has_pending_reply=True, foa_number=...)` — no `message_count_offset`. `_reply_to_thread`: `thread.message_count = len(history_entries) - thread.message_count_offset` (1348); `if thread.message_count >= settings.max_thread_messages: ... await self._close_thread(agent, thread, "timeout"); return` (1359-1365). `max_thread_messages: int = 12` (config.py:327). Reopen paths set `message_count_offset=existing_count` at 2989, 3000, 5131, 5142. `get_thread_allowed_agents` returns None for funding roots (message_log.py:363-364) so the Phase-3 `allowed` guard passes. Net effect confirmed: ThreadDecision(timeout) + PI DM + two memory updates, zero replies. No test references `message_count_offset`. + +### PR E3 + +**COR-5 (STILL PRESENT).** 2941-2954: +``` + def _check_pi_proposal_review(self, entry: LogEntry) -> None: + thread_ts = entry.thread_ts + if not thread_ts: return + for agent in self.agents.values(): + for proposal in agent.state.pending_proposals: + if proposal.thread_id == thread_ts and not proposal.reviewed: + proposal.reviewed = True +``` +Callers: Slack channel poller 2797 (before the `for pi_agent_id in pi_agent_ids` loop at 2800 — any human), proposal-thread poller 3250 (PI-gated by `user_id not in pi_user_ids` at 3232, but not owner-gated), DB path 2913. Web writer `post_agent_message` (agent_page.py:948-1032): `thread_ts: str = Form("")` passed through as `thread_ts.strip() or None`; `pi_may_post_to_channel` (pi_inbox.py:52-101) returns True for any channel whose visibility is not `collab_private`, and True for unknown names. Nothing writes a `ProposalReview`; dashboard (agent_page.py:251-268), email (email_notifications.py:171-173, 763-764) and the rebuild (4297) all read `ProposalReview`, so the in-memory flip is invisible to them and is undone on restart. No test references `_check_pi_proposal_review`. + +**COR-13 (STILL PRESENT).** Rebuild: `reviewed_set = {(r.thread_decision_id, r.agent_id) ...}` (4264), `is_reviewed = (td.id, aid) in reviewed_set` (4297), keyed on the latest ThreadDecision per `(aid, thread_id)` (4268-4283). Tick: `reviewed_set = {(r.agent_id, r.thread_id) for r in rows}` (5007), `(agent.agent_id, proposal.thread_id) in reviewed_set` (5054). `ProposalRef` (state.py:50-58) has `thread_id, channel, other_agent_id, summary_text, proposed_at, reviewed` — no decision id. `_db_reopened_thread_ids`: `set()` at 337, checked 5075, added 5145 — never loaded from DB. Reopen loop is "independent of the reviewed flag" (5064-5066), mints `minted = self.mint_ts()` and appends a `LogEntry(sender_name="PI (via web)", content=guidance, ...)` (5106-5117) which the persist callback writes to `agent_messages`, then builds both ThreadStates with a fresh `message_count_offset` (5125-5143). So every restart with a rating-0 review and a still-pending proposal re-appends one PI-guidance row and re-grants a reply budget. Slack `_reopen_thread` (2956-3003) mutates memory only. Rebuild adds every `ThreadDecision.thread_id` to `closed_thread_ids`/`_closed_thread_ids` (4160-4165, 4186). + +### PR E4 + +**COR-10(1) (STILL PRESENT).** 3024-3028: +``` + oldest = self._dm_poll_cursors.get(agent_id, default_cursor) + messages = client.poll_dm_messages(pi_slack_id, oldest=oldest) +``` +The only try/except in the method (3034-3043) wraps `record_pi_dm`. `poll_dm_messages` (slack_client.py:846-857) → `open_dm_channel` (`except SlackApiError` only, 826-833) → `poll_channel_messages` (`except SlackListingIncomplete` / `except SlackApiError` only, 539-550). `_call_with_retry` (310-341) `except SlackApiError`. slack_sdk 3.43.0 `web/base_client.py:483-515`: non-HTTP failures hit `except Exception as err:` and end in `raise err` / `raise last_error` — the raw `URLError`/`socket.timeout`/`ssl.SSLError` propagates. The client is built as `WebClient(token=self.bot_token)` (slack_client.py:437) with default retry handlers, so the SDK retries a connection error once and then re-raises it unchanged; nothing converts it to `SlackApiError`. Main loop (`_run_main_loop`): `await self._poll_pi_dms()` at 683; the per-turn `try: did_work = await self._run_turn(agent) except Exception:` is at 742-745. `start()` has no try around `_run_main_loop`; main.py:272-273 `except Exception: logger.exception("Simulation engine raised an exception")` then `finally` flushes and marks the run "stopped". Sibling pollers use per-item `except Exception` (2839-2840, 3203-3208). `tests/unit/test_hub_budget_scheduler.py:57-70` stubs `_poll_pi_dms` out of the loop tests, so no test exercises the raise. + +**COR-10(2).** Agree with the low re-rating: per-agent client, `_dm_channels` cache (slack_client.py:825-827), tick cadence floored by `_idle_backoff` (629-640). No change needed beyond noting it. + +**COR-10(3) (STILL PRESENT, worse across restart).** 2876-2898: +``` + for r in rows: + if r.created_at and r.created_at > self._pi_inbox_cursor: + self._pi_inbox_cursor = r.created_at # cursor first + if not r.message_ts or self.message_log.get_entry(r.message_ts): + continue # dedup on log + entry = LogEntry(...) + self.message_log.append(entry) # log second + if r.is_bot: ... + else: + await self._handle_pi_inbound_entry(entry) # handler last, unguarded +``` +The `try/except` (2855-2874) covers only the SELECT. A raise inside `_handle_pi_inbound_entry` (e.g. `_reopen_thread`'s `_hydrate_thread_from_db`, `handle_channel_tag`) propagates to the main loop and ends the run; the entry is already in the log so the in-process lookback re-scan skips it, and on restart `_seed_pi_inbox_cursor` (3764-3785) seeds `_pi_inbox_cursor = max(created_at)` so the row is never re-polled. No test references `_handle_pi_inbound_entry`. + +**COR-11 (STILL PRESENT).** 4448-4475: +``` + batch = self._llm_log_buffer[:] + self._llm_log_buffer.clear() + try: + ... db.add(record) ... await db.commit() + except Exception as exc: + logger.warning("Failed to flush LLM call logs: %s", exc) +``` +Contrast `_flush_persisted` (3946-3950) which re-queues. The sliding-window rebuild reads `LlmCallLog.created_at >= cutoff` (4364-4375) into `call_times`, so dropped rows under-count the window after a restart exactly as the issue says. Only test reference is the no-op stub (test_hub_budget_scheduler.py:69). + +### PR E5 + +**COR-9a (FIXED, d311170).** 3419 `visibility = self._resolve_channel_visibility(channel)`; 3441 `visibility=visibility` on the LogEntry; `_resolve_channel_visibility` (1989-1997) maps from `_channel_visibility`, default public. + +**COR-9b (STILL PRESENT, with a Slack-on nuance).** `channel = action_data.get("channel", "general").lstrip("#")` (2264). `is_private_channel` is derived from that `channel` (2366-2368), so a reply targeting a collab_private post while declaring `"general"` takes the threaded-reply branch and calls `self._post_message(agent.agent_id, channel, message_text, thread_ts=target_post_id)` (2390-2393). The blocked-agent bypass (2286-2294) looks up `target_entry.channel` but only to set `is_private_reply=True`; it never overwrites `channel`. `_post_message` never compares `channel` to the root's channel (`_slack_parent_ts` 3463-3479 only maps canonical→Slack ts). +- Slack-off / DB-only: entry persisted with `channel="general"`, `visibility=public` (3419), `thread_ts=` → feeds the public memory-synthesis segment (5284-5288 filters by `e.visibility == visibility`). Leak as described. +- Slack-on: `chat.postMessage(channel=, thread_ts=)` either errors `thread_not_found` or silently drops `thread_ts` (slack_client.py:718-735); either way `_post_one` raises `ThreadNotFound`, `_post_message` calls `_evict_dead_thread(target_post_id)` on a *live* private post (removing it from every agent's interesting_posts/pending_proposals) and returns False. In the silent-drop case the text is briefly top-level in #general before `chat_delete`. So on Slack-on the failure mode is collateral eviction plus a transient public post, not a persisted row. +- Acknowledged in the codebase: tests/integration/test_full_run_live.py:41-44 documents that Phase 5 "posts there without checking it against the target post's channel" and guards it externally in the live test. + +**COR-9c (CHANGED — hygiene, agree).** `_update_agent_memory` (5262-5396): the user prompt asks for working-memory text, no `` tags; `strip_ungrounded_authorship_lines` at 5349 (b6c2de4). There is no `` strip on this path; no live trigger. + +**COR-8 (STILL PRESENT).** Four copies of the pattern, not two: message_log.py:407 `re.search(r"@(\w+[Bb]ot)\b", content)`, funding_rules.py:154 `_TAG_RE = re.compile(r"@(\w+[Bb]ot)\b")`, simulation.py:2523 (`_strip_disallowed_tags`) and :2558 (tag enumeration). PI literal check at 2831-2832 `if bot_name and f"@{bot_name.lower()}" in msg.get("text", "").lower():`. Ran (`scratchpad/20/cor8.py`, bot map `{"subot": "su"}`): +``` +raw regex r'@(\w+[Bb]ot)\b': MessageLog._extract_tagged_agent: PI literal (lower): funding_rules._TAG_RE: + '<@U12345>' -> None -> None -> False -> None + '@subot' -> subot -> su -> True -> subot + '@SUBOT' -> None -> None -> True -> None + '@SuBot' -> SuBot -> su -> True -> SuBot + '@SUBot' -> SUBot -> su -> True -> SUBot + 'hey <@U12345> look' -> None -> None -> False -> None +``` +Precisely: the regex is case-insensitive over `\w+` and `[Bb]`, case-sensitive on the trailing `ot`; the literal check is fully case-insensitive but only matches the `@Name` form. `grep -rn '<@' src/` → no output. `_bot_uid_map` (4004-4019) is consulted only at 4036/4067/4123 (Slack reconcile attribution). Web UI synthesizes the literal: agent_page.py:975-976 `text = f"@{agent.bot_name} {text}"`. Test coverage: tests/unit/test_message_log.py:99-133 covers the service-bot exemption only; no case or `<@U…>` assertions. + +### PR E6 + +**E6(1) (STILL PRESENT).** `_within_rate_limit` (492-515) is called only from `_turn_eligible` (891 side-effect call, 893 gate). Booking sites: Phase 2 1099 and 1145, Phase 4 1414 plus `on_retry=agent.record_api_call` (1428), Phase 5 2217, memory 5323 — none consult the limiter. Phase 4 `tasks = [self._reply_to_thread(agent, thread) for thread in threads_to_reply]; await asyncio.gather(*tasks, return_exceptions=True)` (1330-1334). + +**E6(2) (STILL PRESENT; one extra consequence).** 4658 `agent = Agent(agent_id=aid, bot_name=r.bot_name, pi_name=r.pi_name, role=r.role)`; `Agent.__init__` sets `self.api_call_count = 0` and `self.state = AgentState()` (agent.py:85-87). Post-add steps (4665-4675) are `set_bot_name_map`, `_pi_slack_id_to_agent_ids.clear()` + `_load_pi_mappings()`, `_recompute_allowed_sender_ids()` — no `_rebuild_agent_state`. Lost on an inactive→active flip: `pending_proposals` (block evaporates), `active_threads`, `interesting_posts`, `call_times`/`throttled` (limiter reset), `api_call_count` (legacy cap reset), and `last_seen_cursor` → 0.0, so the re-added agent's first Phase 2 rescans every top-level post since epoch (not in the issue). Engine-level `_closed_thread_ids`/`_prior_threads` survive; working memory is on disk. `tests/unit/test_roster_sync.py:127 test_adds_newly_active_agent` asserts membership only. + +**E6(3) (STILL PRESENT, cosmetic).** 3939 `run.total_api_calls = sum(a.api_call_count for a in self.agents.values())`; main.py:293 same over the boot-time `agents` list; comment 186-191 labels the counters cosmetic. + +### PR E7 + +**E7a (STILL PRESENT).** 2043-2046 `if today_posts >= settings.daily_post_cap: ... return` precedes `has_pi_priority` (2061) and the funding/private/pi_priority bypasses (2070-2091). `_count_today_posts` (2005-2023) already excludes collab_private posts, so the private case is partially mitigated; PI-priority and funding replies are not. + +**E7b (STILL PRESENT; wording nuance).** 1030 `agent.state.has_pi_directive = False` (713aa20) is unconditional at the end of `_run_turn`. The flag's only consumer is 1007 `has_pi = agent.state.has_pi_directive` → `has_new_work` (1020) → whether `_phase5_new_post` is called. No prompt builder reads it (grep: no references outside simulation.py/state.py; the DM *content* reaches prompts through PIHandler's profile/instruction writes, not this flag). Phase 5 can still return without an LLM call at 2044 (daily cap), 2063 (random skip — `has_pi_priority` is `PostRef.pi_priority`, not the directive flag), 2107 (blocked, nothing available); the flag is then cleared. A *throttled* agent is not selected (`_turn_eligible`) so `_run_turn` never runs and the flag is preserved — the issue's "throttled" wording is inaccurate; "capped, randomly skipped or blocked" is the real path. + +**E7c (STILL PRESENT).** `thread.pi_context = entry.content` at 2826 (Slack active thread), 2931 (DB active thread), `pi_context=pi_entry.content` 2988 (`_reopen_thread`), `pi_context=guidance` 5130 (web reopen). `grep -n "pi_context = None\|pi_context=None" src/agent/simulation.py` → none. Rebuild ThreadState (4165-4171) has no `pi_context`. agent.py:461-467 injects "**Your PI has posted in this thread.** Their message is authoritative" on every Phase 4 build while set. + +**E7d (STILL PRESENT).** 2134-2135 `original_posts = agent.state.interesting_posts; agent.state.interesting_posts = available_posts`; 2215 `agent.state.interesting_posts = original_posts`; `try:` at 2218. Between them: `get_agent_top_level_posts` (2138), `format_foa_for_prompt`/`summarize_funding_thread` (2147-2156), `_get_prior_threads_for_agent` (2189-2191), `build_phase5_prompt` (2203-2212). `_phase5_new_post` is awaited from `_run_turn` (1023) inside the loop's `try` (742), so a raise is logged and the agent keeps the narrowed list. + +## 3. Counts + +26 sub-claims: **22 still present** (COR-1b, 1c, 1d, 3, 4, 7, 6, 2, 5, 13, 10(1), 10(3), 11, 9b, 8, E6-1, E6-2, E6-3, E7a-d), **2 fixed** (COR-1a, COR-9a), **0 partial** at sub-claim level (COR-1 as a whole = PARTIALLY FIXED, matching the issue), **1 changed** (COR-9c, hygiene only), **1 N/A** (COR-10(2), agreed low), **0 not reproducible**. + +Contradictions with the issue text: none at verdict level — every "fixed in stack" item is fixed, every "still present" item is present. Stale/inaccurate details: (i) COR-6 says the rebuild "already uses" `latest_timestamp` — it computes `max(posted_at)` inline; (ii) E7b's "throttled" turn does not reach `_run_turn`, so the flag survives it; the real consumers are the daily cap, the random skip and the blocked-return; (iii) COR-8 lists two regex copies — there are four (add simulation.py:2523 and :2558); (iv) COR-9b on Slack-on produces collateral eviction of a live private thread plus a transient orphan in #general, not a persisted public row (the persisted leak is the Slack-off/DB-only path); (v) all line numbers have drifted (~+40..+110 in simulation.py). + +Surprises: `tests/unit/test_thread_not_found.py:144` pins the COR-1c un-close (`assert dead_ts not in engine._closed_thread_ids`) — fixing COR-1c must update that test. E6(2) additionally resets `last_seen_cursor` to 0.0 (full rescan). slack_sdk 3.43.0 confirmed to re-raise raw transport errors (`base_client.py:483-515`), so COR-10(1) is not hypothetical. + +## 4. What I could not verify and why + +- Live Slack behaviour for a `chat.postMessage` with a `thread_ts` from another channel (COR-9b Slack-on branch): reasoned from `_post_one`'s two handled cases (`thread_not_found` error vs silent `thread_ts` drop, slack_client.py:704-735); no network calls were made. +- Whether the LLM ever emits `:white_check_mark:` in practice (COR-4 exposure): prompts ask for ✅; no run logs were inspected. +- Whether `_reopen_thread`/`handle_channel_tag` actually raise in production (COR-10(3) trigger): verified the control flow, not a live failure. +- Integration/contract tests were not executed (Docker required per PREAMBLE); test coverage claims rest on grep + reading the assertions. diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_20_redteam.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_20_redteam.md new file mode 100644 index 00000000..009737e5 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_20_redteam.md @@ -0,0 +1,121 @@ +# Issue #20 — red-team pass over findings/issue_20.md (copi-prod @ 18ba52c) + +Second, adversarial reviewer. Every row below was re-derived from the current tree with my own grep/sed/python; the first +agent's quotes were not used as a starting point. Line numbers are CURRENT (18ba52c). Written incrementally. + +## 1. Summary table + +| id | first-agent verdict | red-team result | one-line reason | evidence | +|---|---|---|---|---| +| COR-1a | FIXED (6af8207) | UPHELD | Only 4 `_post_message(` call sites exist in `src/` (grep), all four gate on the bool; ThreadNotFound path returns False before the LogEntry; pre-fix signature was `-> None` | simulation.py:1494-1498, 2370-2371, 2390-2394, 2432-2433; 3357-3367 (`except ThreadNotFound` → `return False`), LogEntry at 3416; `git show 6af8207^:src/agent/simulation.py` :3065-3071 `-> None`. Test test_simulation_logic.py:974-978 would fail pre-fix (posted None, entry appended) but covers only the empty-strip path; no test pins ThreadNotFound→False or the call-site gating | +| COR-1b | STILL PRESENT | UPHELD | `_post_one` → None on any other SlackApiError; wrapper → None; engine mints local id and returns True. `connect()` dropping the client (slack_client.py:447-457) only covers auth failure at connect time | slack_client.py:704-712, 815-816; simulation.py:3400, 3423, 3447. Ran: `_mirrored_messages(None,"hello",None) -> []` | +| COR-1c | STILL PRESENT | UPHELD | discard at 1818, no log purge; test pins it | simulation.py:1788-1825; tests/unit/test_thread_not_found.py:131,144. Ran the file: 9 passed | +| COR-1d | STILL PRESENT | QUALIFIED | Position confirmed (outside both `posted` branches) but for a flat private post no `return False` path can carry a ✅ today — latent, as the issue says; first agent implies live reachability | simulation.py:2455-2462; False sources 3300-3312 (empty strip), 3323-3329 (authorship, already passed on the same draft at 2319), 3357-3367 (needs thread_ts) | +| COR-3 | STILL PRESENT | UPHELD | first prior other-agent `:memo:` in reversed history, no recency check | simulation.py:1527-1531 vs 1687-1700 | +| COR-4 | STILL PRESENT | UPHELD | ✅ single-form at 1527; dual-form at 1563 and 1674 | simulation.py:1527, 1563, 1674 | +| COR-7 | STILL PRESENT | UPHELD | append dict has no thread_id/dedup; rebuild guarded by `_closed_thread_ids` | simulation.py:1583-1589; 4175-4185 | +| COR-6 | STILL PRESENT | UPHELD | wall-clock cursor vs `posted_at <= since`; rebuild uses inline `max(...)` not `latest_timestamp` (first agent right) | simulation.py:1033, 1071; message_log.py:199/311/339/435, 445-454; simulation.py:4401-4403 | +| COR-2 | STILL PRESENT | QUALIFIED | Code true (no offset on either Phase-3 path; close at ≥12) but reachability is narrower than "one @-mention closes a thread": Phase 3 skips `_closed_thread_ids`, and every participant's Phase 4 closes a ≥12 thread anyway, so the live window is the gap between the 12th message and the next participant turn (or a thread no roster agent tracks) | simulation.py:1196/1242 closed-skip; 1218-1225, 1259-1266 no offset; 1348, 1359-1365; `_close_thread` 1581 adds to closed set. Ran: `ThreadState(...).message_count_offset == 0` | +| COR-5 | STILL PRESENT | UPHELD | thread_id-only, any sender, all agents, nothing persisted; Slack site precedes PI loop; proposal poller gated on *any* PI; web writer free-form thread_ts + any public channel; `get_agent_with_access` checks ownership of the posting agent, not of the thread | simulation.py:2941-2954; 2797 vs 2800; 3232, 3250; 2913; agent_page.py:952, 990-998; pi_inbox.py:52-101; dependencies.py:114-140. `grep -n "ProposalReview(" src/agent/` → none | +| COR-13 | STILL PRESENT | UPHELD | rebuild `(td.id, aid)` vs tick `(agent_id, thread_id)`; reopened-set never seeded; synthetic PI row appended → persisted via callback | simulation.py:4264, 4297 vs 5007, 5054; state.py:50-58; 337/5075/5145; 5106-5117; persist callback 568 | +| COR-10(1) | STILL PRESENT | UPHELD | poll at 683 outside per-turn try 742-745; `start()` has no try; only `SlackApiError` is caught anywhere on the path; slack_sdk 3.43.0 re-raises transport errors (read installed file). Reachability condition: Slack-on AND `AgentRegistry.slack_user_id` set on an active agent | simulation.py:683, 742-745, 616; main.py:272-273; slack_client.py:310-341, 539-550, 826-833; .venv-test slack_sdk/web/base_client.py:484-515 `except Exception as err ... raise err` / `raise last_error` | +| COR-10(2) | N/A (low) | UPHELD | per-agent client, `_dm_channels` cache, idle backoff | slack_client.py:825-832; simulation.py:623-636 | +| COR-10(3) | STILL PRESENT | QUALIFIED | Ordering confirmed (cursor → dedup → append → unguarded handler). But the first agent's raise example `_hydrate_thread_from_db` is fully guarded; the real unguarded raise is `handle_channel_tag → _send_dm → client.send_dm` (Slack transport error). And the *row* is not lost (persisted by the web app, reloaded by the rebuild); the *side effects* are | simulation.py:2877-2898; 3800-3815 (hydrate guarded); pi_handler.py:343-344 → `_send_dm` 394-403 (`client.send_dm` at 403 is outside the `try` at 410); rebuild 3718-3736; cursor seed 3783-3785; 4401-4403 | +| COR-11 | STILL PRESENT | UPHELD | buffer cleared before write, except only logs; window rebuild reads `llm_call_logs` | simulation.py:4452-4453, 4474-4475; 4360-4380 | +| COR-9a | FIXED (d311170) | QUALIFIED | `_post_message` does stamp from channel (fixed, test would fail pre-fix). But two other engine writers still omit `visibility` and take the dataclass default `"public"`: the proposal-thread poller (which explicitly polls collab_private channels) and the web-reopen synthetic PI row. Human rows only → no live reader leaks today | simulation.py:3419, 3441; 3223 (no visibility; private channels explicitly routed at 3190); 5108-5117 (no visibility); message_log.py:30 default. Ran: `LogEntry(...).visibility == 'public'` | +| COR-9b | STILL PRESENT | QUALIFIED | Slack-off trace confirmed. Slack-on has three sub-branches, not one: root with `slack_ts` → ThreadNotFound/evict (first agent's case, Slack behaviour unverifiable offline); root WITHOUT `slack_ts` → `can_mirror=False` → the same persisted public row as Slack-off; root windowed out → as case 1 | simulation.py:2264, 2366-2368, 2390-2393, 3344-3352, 3463-3479, 3419, 5284-5288; slack_client.py:704-735 | +| COR-9c | CHANGED/hygiene | UPHELD | no `` request in the synthesis prompt; authorship strip present | simulation.py:5349 | +| COR-8 | STILL PRESENT (+2 regex copies) | UPHELD | four copies confirmed; case behaviour reproduced | message_log.py:407; funding_rules.py:154; simulation.py:2523, 2558. Ran: `<@U12345>`→None, `@SUBOT`→None, `@SuBot`/`@SUBot`/`@Subot`→su | +| E6(1) | STILL PRESENT | UPHELD | limiter consulted only in `_turn_eligible`; six booking sites incl. `on_retry`; one `gather` | simulation.py:891, 893; 1099, 1145, 1414, 1428, 2217, 5323; 1330-1334 | +| E6(2) | STILL PRESENT (+cursor reset) | UPHELD | fresh `Agent` → `AgentState()`; `_rebuild_agent_state` only called at startup; Phase 2 with `since=0.0` has no cap | simulation.py:4658, 578 (only call site besides the def); agent.py:87; 1070-1075. Ran: `AgentState().last_seen_cursor == 0.0` | +| E6(3) | STILL PRESENT (cosmetic) | UPHELD | sum over live roster | simulation.py:3939; main.py:293 | +| E7a | STILL PRESENT | UPHELD | cap return precedes `has_pi_priority` and the bypasses | simulation.py:2044-2046; 2061; 2070-2091 | +| E7b | STILL PRESENT (wording) | UPHELD | flag cleared unconditionally at end of `_run_turn`; read only at 1007; throttled agents never reach `_run_turn` | simulation.py:1007, 1030; 885-896 `_turn_eligible`; refs grep (no prompt reader) | +| E7c | STILL PRESENT | UPHELD (mis-cite) | never cleared; rebuild omits it; injected as authoritative. Rebuild ThreadState is at 4237-4243, not 4165-4171 | `grep pi_context` → set at 2826, 2931, 2988, 5130, 5247; agent.py:461-467; simulation.py:4237-4243 | +| E7d | STILL PRESENT | UPHELD | swap/restore before `try` | simulation.py:2134-2135, 2215, 2218 | +| NEW-1 | test pins the un-close | UPHELD | read + ran | tests/unit/test_thread_not_found.py:131, 144; 9 passed | +| NEW-2 | re-add resets `last_seen_cursor` → full rescan | UPHELD | see E6(2) | agent.py:87; state.py:70; simulation.py:1070-1075 | +| NEW-3 | `_seed_pi_inbox_cursor` seeds past a lost row | UPHELD (wording) | seeds to `max(created_at)`; but the row is re-loaded by the rebuild — only its side effects are lost | simulation.py:3764-3785, 3718-3736 | +| NEW-4 | four regex copies | UPHELD | grep | see COR-8 | +| NEW-5 | E7b "throttled" wording | UPHELD | see E7b | simulation.py:885-896 | +| NEW-6 | COR-9b Slack-on = collateral eviction only | QUALIFIED | one of three Slack-on sub-branches | see COR-9b | +| NEW-7 | COR-10(3) raise example `_hydrate_thread_from_db` | OVERTURNED (example only) | guarded by try/except → return | simulation.py:3801-3815 | + +## 2. Detail — QUALIFIED / OVERTURNED rows + +### COR-1a (FIXED) — upheld, with a coverage note +Every `_post_message(` call in `src/` (grep, four hits + the def) reads the return value and skips `message_count += 1`, the backoff resets and the `interesting_posts`→`active_threads` move when it is False (1494-1512, 2370-2386, 2390-2431, 2432-2440). The pre-fix tree (`git show 6af8207^:src/agent/simulation.py`, :3065-3071) had `-> None`. The cited regression test (tests/unit/test_simulation_logic.py:974-978) asserts `posted is False` and `_entries == []` — pre-fix it would get `None` and one appended entry, so it fails pre-fix. It exercises only the empty-strip branch; no unit test pins `ThreadNotFound → False` through `_post_message` (test_thread_not_found.py tests the client and `_evict_dead_thread` separately) nor the four call-site guards. Not an overturn — the fix is complete by reading — but the "definition of done" test for this PR does not yet exist. + +One adjacent path checked and cleared: `BotNotInvitedToPrivateChannel` (slack_client.py:708-709) is not caught by `_post_message`; it propagates to Phase 5's `except Exception` (2464) / Phase 4's `gather(return_exceptions=True)` (1332). It is raised before any LogEntry, so it cannot mint a phantom. + +### COR-1d — QUALIFIED (latent, not live) +``` +2455 if ( +2456 message_text +2457 and self._channel_visibility.get(channel) == VISIBILITY_COLLAB_PRIVATE +2458 ): +2459 await self._check_private_channel_outcome(agent, channel, message_text) +``` +sits after both `if not posted` blocks and never reads `posted` — position confirmed. But for the private branch (flat post, `thread_ts=None`) `_post_message` returns False only from: (a) text empty after `` strip (3300-3312) — the strip removes only tags, so a draft that still carries `✅` cannot hit it; (b) the authorship chokepoint (3323-3329) — Phase 5 already ran the identical gate on the same draft at 2319 and returned, and the second pass runs on the tag-stripped text (fewer matches, never more); (c) `ThreadNotFound` (3357-3367) — requires `thread_ts`. `BotNotInvitedToPrivateChannel` raises instead of returning, which skips 2455-2459 entirely. `_finalize_private_proposal` is also idempotent per channel (1728-1751 DB existence check + `_finalized_private_channels`). So the issue's own label ("latent re-instance") is the right one; the first agent's "A suppressed private post carrying ✅ can still call `_finalize_private_proposal`" describes a path with no live trigger on this tree. + +### COR-2 — QUALIFIED (reachability narrower than filed) +Code as claimed: both Phase-3 constructors (1218-1225 tag, 1259-1266 reply) omit `message_count_offset`; `_reply_to_thread` recomputes at 1348 and closes at 1359-1365 when `>= max_thread_messages` (config.py:327 = 12); reopen paths set it (2989, 3000, 5131, 5142). Ran: `ThreadState(...).message_count_offset == 0`. `_phase3_activate_threads` and `_phase4_reply_threads` run in the same `_run_turn` (relative lines 37-41 of the 955-1000 window), so activation → close happens within one turn. + +What the issue and the first agent do not weigh: both Phase-3 paths skip `thread_id in self._closed_thread_ids` (1196, 1242); `_close_thread` adds to that set (1581); every roster participant's own Phase 4 closes the thread the moment its recomputed count reaches 12 (1359-1365); and the rebuild re-closes every ThreadDecision thread (4165, 4185). So an *open* ≥12-message thread exists only in three situations: (a) the race window between the 12th message (the one that tags C) and the next participant's Phase 4 — in which the thread would have been closed at the same count regardless; (b) a thread no roster agent tracks (≥12 messages from PI/humans/GrantBot/non-roster bots); (c) a thread `_evict_dead_thread` un-closed (COR-1c, 1818). "Multi-party funding threads cross 12 messages quickly" does not by itself make the thread reachable, because the participants close it at 12. + +Marginal harm in case (a) is real but different from "a thread closed by one @-mention": a `ThreadDecision(timeout, agent_a=C, agent_b=B)` for a pair that never conversed, a DM to C's PI, memory events, and a `_prior_threads[(B,C)]` "you already tried this" entry that pollutes both agents' Phase-5 dedup context (1583-1589) — plus C's silence toward the tag. Also note the proposed fix (offset on activation) grants C a fresh 12-reply budget on a thread the other participants have just capped, which is a design decision the PR should make explicitly, not a pure bug fix. + +### COR-10(3) — QUALIFIED (trigger example wrong; "message lost" overstated) +Ordering confirmed verbatim: +``` +2877 if r.created_at and r.created_at > self._pi_inbox_cursor: +2878 self._pi_inbox_cursor = r.created_at +2879 if not r.message_ts or self.message_log.get_entry(r.message_ts): +2880 continue +... +2893 self.message_log.append(entry) +... +2898 await self._handle_pi_inbound_entry(entry) +``` +with the only `try` (2855-2874) around the SELECT. Two corrections: +1. The first agent names `_reopen_thread`'s `_hydrate_thread_from_db` as an example raise. It cannot raise: 3800-3815 wraps its query in `try/except Exception → logger.warning; return`. `_check_pi_proposal_review` (2941-2954) and `_reopen_thread` (2956-3003) are pure in-memory after that. The one unguarded raise on the handler path is `handle_channel_tag` (pi_handler.py:300-346) → `_send_dm` (394-403): `result = client.send_dm(pi_slack_id, text)` at 403 is *outside* the `try` at 410; `send_dm` → `open_dm_channel` / `post_message` catch only `SlackApiError`, so a transport error (same class as COR-10(1)) propagates to the main loop. Trigger: a PI web message that @-tags a bot while Slack is on and the DM call fails at the socket level. Real, but narrower than "any raise in the handler". +2. "The PI message is silently lost" is too strong. The row was written by the web app and survives; the rebuild re-loads it (3718-3736) and it appears in every thread history. What is lost are the *triggers*: the in-memory review clear, the reopen, `pi_context`, `has_pi_directive`, and `handle_channel_tag`'s PI-priority PostRef. The tag is doubly lost because the rebuild sets `last_seen_cursor = max(posted_at)` (4401-4403), so Phase 3 never scans it either, and `_seed_pi_inbox_cursor` (3783-3785) keeps the poller from re-processing it — the first agent's NEW-3 claim holds with that wording. + +### COR-9a (FIXED) — QUALIFIED: `_post_message` fixed, two other writers still default to public +`_post_message` stamps `visibility = self._resolve_channel_visibility(channel)` (3419) onto the LogEntry (3441); the cited integration test (tests/integration/test_cohort_engine_live.py:1196-1232) asserts three distinct persisted values and would fail pre-fix (all `public`). Could not run it (needs Docker); reasoned from the assertions. + +Hunting for the same mislabel elsewhere: `LogEntry.visibility` defaults to `"public"` (message_log.py:30; ran: `LogEntry(...).visibility == 'public'`). Of the ten `LogEntry(` constructors in simulation.py, eight pass `visibility=`; two do not: +- **3223** in `_poll_proposal_threads_for_pi`. This poller explicitly supports collab_private proposal threads (3186-3193 routes a member bot via `_client_for_channel`), and `_finalize_private_proposal` puts the private channel into `pending_proposals` (1761-1768), so a PI's in-thread Slack reply to a memo inside a private refinement channel is appended and persisted with `visibility='public'`. +- **5108-5117** in `_sync_proposal_reviews_from_db`: the synthetic "PI (via web)" guidance row takes the proposal's channel, which is the private channel when the pending proposal came from `_finalize_private_proposal` and the review carries `refined_in_channel is None`. +Impact today is bounded because both are human rows: `_entry_allowed` returns True for `not entry.is_bot` before it reads `visibility` (message_log.py:73-76); the G2 memory filter keys on `sender_agent_id == agent.agent_id` (5284-5288); `conversation_feed.gate_clause` admits `is_bot False` (conversation_feed.py:68). So no reader leaks on them — but the "visibility mislabel" defect class is not closed, and any future reader that filters `agent_messages.visibility` will misclassify these rows. Recommend the E5 PR stamp both from `_resolve_channel_visibility(channel)`. + +### COR-9b — QUALIFIED: Slack-on has three sub-branches, and one of them persists the public row +Slack-off trace (agree): `channel = action_data.get("channel", "general")` (2264) → `is_private_channel` derived from it (2366-2368) → threaded branch `_post_message(agent, "general", text, thread_ts=)` (2390-2393) → MOCK → LogEntry `channel="general"`, `visibility=public` (3419), `thread_ts=` → included in the public memory segment (5284-5288). The blocked-agent bypass (2288-2301) reads `target_entry.channel` only to set `is_private_reply`. `_post_message` never compares `channel` to the root's channel; `_slack_parent_ts` (3463-3479) only maps canonical→Slack ts. + +Slack-on, by the state of the private root in the log: +1. Root has `slack_ts` → `chat.postMessage(channel=, thread_ts=)`. `_post_one` turns either a `thread_not_found` error or a silent `thread_ts` drop into `ThreadNotFound` (704-735) → `_evict_dead_thread()` + `return False`. This is the first agent's case. Slack's actual response to a cross-channel `thread_ts` cannot be verified offline. +2. Root has **no** `slack_ts` (persisted DB-only: posted before that bot was connected, or its own `chat.postMessage` failed and COR-1b minted a local id, or the channel never existed on Slack) → `_slack_parent_ts` → None → `can_mirror = False` (3344) → the warning branch (3347-3352) skips Slack and falls through to the *same* persisted public row as Slack-off. +3. Root windowed out of the log → `_slack_parent_ts` returns the canonical id → behaves as case 1. +So "the persisted leak is the Slack-off/DB-only path" (first agent, and the counts line item iv) is too narrow: it is the DB-only *root* path, which also occurs with Slack on. + +### NEW-7 — OVERTURNED (example): `_hydrate_thread_from_db` as a raise source +See COR-10(3) item 1. The function is guarded end-to-end (3800-3815). The verdict on COR-10(3) is unaffected because `_send_dm` supplies a real unguarded raise. + +## 3. First agent's mis-cited lines / symbols +- E7c: "rebuild ThreadState (4165-4171) has no `pi_context`" — 4164-4174 is the ThreadDecision loop's comment block; the rebuild `ThreadState(` is at **4237-4243** (and does omit `pi_context` and `message_count_offset`). +- COR-2: reply-path constructor cited as ":1264-1271" — it is **1259-1266**. +- COR-10(3): `_hydrate_thread_from_db` given as a raise example — guarded (3800-3815). Real unguarded raise: pi_handler.py:403 via `handle_channel_tag`. +- COR-1d: "A suppressed private post carrying ✅ can still call `_finalize_private_proposal`" — no `return False` path exists for a flat post that still carries ✅ (see detail). +- COR-9b counts-line item (iv): "the persisted leak is the Slack-off/DB-only path" — also the Slack-on path when the private root lacks `slack_ts`. +- COR-9a: "FIXED" is correct for `_post_message` but the report does not mention the two remaining unstamped writers (3223, 5108). +- COR-10(2): `_dm_channels` cache cited at "825-827" — the cache is declared at 283 and read/written at 825-832 (trivial). +All other cited lines checked (COR-1a/1b/1c, 3, 4, 5, 6, 7, 8, 11, 13, E6, E7a/b/d) point at the quoted code on 18ba52c. + +## 4. Counts +26 first-agent table rows + 7 NEW claims = 33 items. +- UPHELD: 27 (COR-1a, 1b, 1c, 3, 4, 7, 6, 5, 13, 10(1), 10(2), 11, 9c, 8, E6(1), E6(2), E6(3), E7a, E7b, E7c, E7d, NEW-1..5 — NEW-3 with a wording note) +- QUALIFIED: 5 table rows (COR-1d, COR-2, COR-10(3), COR-9a, COR-9b) + NEW-6 = 6 +- OVERTURNED: 0 verdicts; 1 supporting example (NEW-7, `_hydrate_thread_from_db`) +- UNVERIFIABLE: 0 rows; one sub-point (Slack's response to a cross-channel `thread_ts`, inside COR-9b) cannot be settled offline. + +No verdict flips. Every "fixed in stack" item is fixed at the cited symbol; every "still present" item is present. The material corrections are: COR-9a's mislabel class survives at two other writers; COR-9b's persisted leak also occurs Slack-on; COR-2's live window is much narrower than the issue implies; COR-10(3)'s real trigger is the PI-tag DM, and the row itself is not lost; COR-1d is latent, not live. diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_21.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_21.md new file mode 100644 index 00000000..fc73342b --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_21.md @@ -0,0 +1,259 @@ +# Issue #21 verification — Worker & background jobs (V11, V2, V3, V4) + +Verified against `copi-prod` @ `18ba52c` (clean tree), 2026-09-02. All line numbers below are CURRENT. +Method: symbol grep + reading the actual function bodies, `git log -S` for the "fixed in #31" rows, and python +snippets against the real `src.services.email_inbound` module for the two mechanically testable COR-19 rows. +Nothing was executed against a database, Docker, Slack or AWS. + +## 1. Summary table + +| id | claim (one line) | verdict | key evidence | conf | +|---|---|---|---|---| +| V11-a | `ids.py` defines four slots; module default is `TsMinter(WRITER_WEB)` | ACCURATE / STILL PRESENT | `src/agent/ids.py:47-50`, `:111` | high | +| V11-b | `set_default_writer_id` called by web, engine, grantbot only | ACCURATE (lines drifted) | `src/main.py:113`, `src/agent/main.py:54`, `src/agent/grantbot.py:731,764` | high | +| V11-c | `src/worker/main.py` never imports `src.agent.ids` / never claims a slot | STILL PRESENT | `src/worker/main.py:6-18` (imports), whole file has no `ids` reference | high | +| V11-d | worker mints via `record_pi_message` and `migrate_public_thread_to_private` | STILL PRESENT (one stale ref) | `email_inbound.py:674`→`pi_inbox.py:120`; `email_inbound.py:638`→`private_channels.py:269` | high | +| V11-e | both paths gated on `enable_inbound_email` (default False) | ACCURATE | `src/config.py:152`, `src/worker/main.py:162` | high | +| V11-f | runbook step 4 mentions no writer slot | STILL PRESENT | `docs/inbound-email.md:84-90`; `grep -i writer` → no hits | high | +| V11-g | backfill scripts don't mint | ACCURATE | `grep mint_local_ts\|TsMinter scripts/` → no hits | high | +| V11-h | spec says "Three processes"; counts now wrong (4 slots / 5 with worker / 6 with slot 99) | STILL PRESENT | `specs/local-db-conversations.md:66-67`; `scripts/migrate/remediate_duplicates.py:131` | high | +| COR-17 | worker except block commits without rollback; DB error strands job in `processing` | STILL PRESENT (test *characterizes* the bug) | `src/worker/main.py:100-111`; `tests/integration/test_worker.py:760-827` | high | +| COR-18a | `started_at` written, never read | STILL PRESENT | only writer `worker/main.py:49`; no reader anywhere in `src/` or `scripts/` | high | +| COR-18b | no reaper for orphaned `processing` rows | STILL PRESENT | only other `"processing"` ref is display-only `src/routers/admin.py:126` | high | +| COR-18c | failed jobs re-queue with no backoff ("~15 s for 3 attempts") | STILL PRESENT — issue understates it: retry is **immediate** (no sleep) | `worker/main.py:127-137` sleeps only when `claim_job` returns None | high | +| COR-18d | `completed_at` set on failure | STILL PRESENT | `worker/main.py:110` | high | +| COR-18e | enum `'failed'` never written | STILL PRESENT | `src/models/job.py:23`; worker writes only processing/completed/dead/pending; also documented in `tests/unit/test_reachability.py:39-44` | high | +| COR-18f | onboarding self-heals only when `job is None` → wedged job = permanent spinner, no retry | STILL PRESENT | `src/routers/onboarding.py:79`; `templates/onboarding/profile_review.html:28` (spinner for none/pending/processing), `:47-55` (retry only under `'failed'`) | high | +| COR-19.1 | poison objects retried forever | FIXED (8f96f86, PR #31) | `email_inbound.py:33`, `:186-209`; test `tests/unit/test_email_inbound_hardening.py:259-292` | high | +| COR-19.2 | `MAX_REPLIES_PER_TOKEN_PER_HOUR` never enforced | FIXED (8f96f86, PR #31) | `email_inbound.py:52-65`, `:241-245`; tests `hardening.py:195-211` | high | +| COR-19.3 | `charset=unknown-8bit` → `LookupError` (codec lookup precedes `errors="replace"`) | STILL PRESENT — reproduced on the real helper | `email_inbound.py:386-389`; snippet output below | high | +| COR-19.4 | string rating → `TypeError`; `classify_reply` returns raw `json.loads` | STILL PRESENT — reproduced | `email_inbound.py:320-322`, `:516` | high | +| COR-19.5 | `list_objects_v2(MaxKeys=50)` unpaginated | STILL PRESENT | `email_inbound.py:165`; no `ContinuationToken`/`IsTruncated` anywhere in file | high | +| COR-19.6 | side effects (SES confirm, Slack post, migration) happen before `db.commit()` | STILL PRESENT | side effects `:335`, `:359`, `:605`, `:638`, `:655`, `:706`; commit is in the poller at `:179`, S3 delete `:182` | high | +| COR-32 | `_handle_instruction` returning False still marks notification responded + S3 delete; PI not told on token/channel failures | STILL PRESENT | caller `:338-353` (`mark_notification_responded` at `:348` unconditional); silent `return False` at `:682,:699,:704,:719`; emailing paths `:605-612`, `:655-662` | high | +| V4-1 | zero `rollback` in `email_notifications.py`; per-item excepts fall through to sweep commit | STILL PRESENT | `grep -c rollback` = 0; excepts `:208-214`, `:721-725`, `:912-916`; commits `:216`, `:726`, `:917` | high | +| V4-2 | phantom-sent: row created `status="sent"` + flushed before SES send, never cleared on failure | STILL PRESENT | `:331-340` vs send `:481-485`/fail `:493-495`; `_send_new_proposal_email` `:995-1004` vs `:1066`; bails `:240-250`, `:968-976` | high | +| V4-3 | `"expired"` never written | STILL PRESENT | `grep -c expired src/services/email_notifications.py` = 0; only `src/models/email_notification.py:42` comment | high | +| V4-4a | downgrade ladder dead: one increment, unreachable past 1, `MISSED_THRESHOLD=3` unreachable | STILL PRESENT | increment `:295`; bail `:247-250`; resets `:517`, `:523`, `:594`, `src/routers/settings.py:133`; dead code `:505-529` | high | +| V4-4b | delegate reply marks only the delegate's rows; PI's `sent` row is immortal | STILL PRESENT | `email_inbound.py:334,348` and `src/routers/agent_page.py:511,714` pass the *replier's* id; filter `email_notifications.py:608` | high | +| T-1 | `enable_inbound_email` defaults False | ACCURATE | `src/config.py:152` | high | +| T-2 | …and is unset in prod | NOT VERIFIABLE here (no prod access allowed). Local dev `.env` has no `ENABLE_INBOUND_EMAIL` key; `.env.example` doesn't list it either | — | low | +| T-3 | `proposal_review` sweep gates on `User` columns, deliberate per model comment | ACCURATE | `email_notifications.py:192-199`; `src/models/email_notification.py:99-105` | high | +| T-4 | all three sweeps run every 300 s | ACCURATE | `src/config.py:158`; `worker/main.py:141-157` | high | +| T-5 | sweeps "still write — `get_or_create_pref` inserts on a composite-PK table each cycle" | PARTIALLY ACCURATE — inserts only when the row is missing; SELECT-only afterwards. Other writes per cycle do exist | `email_notifications.py:56-77`, `:230-233`, `:267` | high | +| T-6 | V11 / V3-remainder / COR-32 all arm at runbook step 4 | ACCURATE | `docs/inbound-email.md:84-90` | high | +| DoD | "takes `worker/main.py` from 0 % coverage" | STALE — `tests/integration/test_worker.py` (15 tests, commit d732804, 2026-07-30) already exercises `claim_job`/`process_job`/`run_worker` | `tests/integration/test_worker.py:1-29` | high | + +## 2. Per-item detail + +### PR V11 — writer slot + +**ids.py** (`src/agent/ids.py:47-50`): +``` +WRITER_ENGINE = 0 # SimulationEngine._ts_minter (agent_messages) +WRITER_WEB = 1 # web app process (PI messages + DMs) +WRITER_GRANTBOT = 2 # grantbot process (funding posts) +WRITER_ENGINE_AUX = 3 # module default inside the engine process (PI DMs) +``` +`:111 _default = TsMinter(WRITER_WEB)`. Four slots defined. Claimers: `src/main.py:113 set_default_writer_id(WRITER_WEB)`, +`src/agent/main.py:54 set_default_writer_id(WRITER_ENGINE_AUX)`, `src/agent/grantbot.py:731` and `:764` (`WRITER_GRANTBOT`). +Issue cited grantbot `:724/757` — drifted by 7 lines, same calls. + +**Worker**: `src/worker/main.py` imports (lines 6-18) are asyncio/logging/signal/sys/uuid/datetime/sqlalchemy/`src.config`/ +`src.models`/`src.services.profile_pipeline`. No `src.agent.ids`. The worker therefore mints in residue class 1 — the same as +the web app. + +**Mint paths reached from the worker** (both inside `_handle_instruction`, only when `enable_inbound_email`): +- `email_inbound.py:674 await record_pi_message(...)` → `src/services/pi_inbox.py:120 ts = mint_local_ts()`. + (Issue also cites `pi_inbox.py:151`; that is `record_pi_dm`, which nothing in `email_inbound.py` calls — stale/over-inclusive + reference, does not change the verdict.) +- `email_inbound.py:638 await migrate_public_thread_to_private(...)` → `src/services/private_channels.py:269 ts = slack_ts or mint_local_ts()` + (inside `_add_handover_message`, used by both online and offline migration paths). + +**Gate**: `src/config.py:152 enable_inbound_email: bool = False`; `src/worker/main.py:162 if settings.enable_inbound_email and ...`. + +**Runbook**: `docs/inbound-email.md:84-90` step 4 = flip flag + recreate app/worker. `grep -n -i "writer\|slot\|ids.py"` on the doc → no hits. + +**Spec count**: `specs/local-db-conversations.md:66-67`: "Three processes mint into the same run — the engine, the web app and +GrantBot". Slots defined: 4. Processes that would mint once inbound is on: engine (0 and 3), web (1), grantbot (2), worker +(unclaimed → 1). `scripts/migrate/remediate_duplicates.py:131 REMEDIATION_WRITER_SLOT = 99`. Issue's "wrong three ways over" is +fair. + +**Mitigation the issue missed**: `remediate_duplicates.py:189-215` introspects `ids.py` for every `WRITER_*` int and raises if +`REMEDIATION_WRITER_SLOT` is ever claimed, and `tests/unit/test_remediate_duplicates.py:726-727` asserts 99 is free — so adding +`WRITER_WORKER = 4` is safe against that script. No test asserts the worker claims a slot. + +**Backfill scripts**: `grep -rl "mint_local_ts\|TsMinter" scripts/` → nothing. Claim accurate. + +### PR V2 — COR-17 (no rollback) + +`src/worker/main.py:100-111` (unchanged since 1342cc0; the issue's line numbers still match exactly): +``` + except Exception as exc: + logger.error("Job %s failed: %s", job.id, exc, exc_info=True) + job.last_error = str(exc)[:2000] + + if job.attempts >= job.max_attempts: + job.status = "dead" + ... + else: + job.status = "pending" # Will be retried + + job.completed_at = datetime.now(timezone.utc) + await db.commit() +``` +No `rollback` anywhere in the file (`git log -S rollback -- src/worker/main.py` → empty). `run_profile_pipeline` only +`flush()`es (`src/services/profile_pipeline.py:234, 293, 466, 497`; no `commit`). A flush-time DB error leaves the session in +needs-rollback state; the failure-path `commit()` raises `PendingRollbackError`, escapes `process_job`, is caught by +`run_worker`'s outer handler at `:170-172`, and the row stays `processing` (committed by `claim_job` `:48-51`) with +`last_error=None`. + +**Test found**: `tests/integration/test_worker.py:760-827 test_a_database_error_in_the_pipeline_orphans_the_job_in_processing` +reproduces exactly this and **asserts the buggy behaviour** (`status == "processing"`, `last_error is None`, +`PendingRollbackError` escapes), with a message saying to flip the assertion when fixed. It is a characterization test, not a +fail-against-pre-fix test, so the issue's Definition-of-done test is still owed. + +### PR V2 — COR-18 (reaper / backoff / status) + +- `started_at`: `grep -rn started_at src/ scripts/` → written `worker/main.py:49`, declared `models/job.py:37`, and an unrelated + `agent_activity.py:43`. Never read. STILL PRESENT. +- Reaper: `grep "processing" src/ scripts/` → only `src/routers/admin.py:126` (counts active jobs for display). None. STILL PRESENT. +- Backoff: `worker/main.py:127-137` — `asyncio.sleep(worker_poll_interval)` is in the `else:` branch (no job). After + `process_job` returns the loop immediately re-enters `claim_job`, so a job re-queued as `pending` is re-claimed on the very + next iteration with **no delay**. The issue's "~15 s" is an over-estimate; all 3 attempts burn as fast as the pipeline can + fail. `tests/integration/test_worker.py:381-419` (T5.2) asserts the 3-attempts-then-`dead` behaviour but not timing. +- `completed_at` on failure: `:110`. STILL PRESENT. +- `'failed'`: `src/models/job.py:23` enum has it; the worker writes only `processing` (`:48`), `completed` (`:95`), `dead` + (`:105`), `pending` (`:108`). `tests/unit/test_reachability.py:39-44` records the same fact as a known false negative + ("the retry button is unreachable at runtime"). +- Onboarding: `src/routers/onboarding.py:79 if job is None and profile is None and current_user.access_status == "allowed":`. + Template `templates/onboarding/profile_review.html:28 {% if job_status == 'pending' or job_status == 'processing' or job_status == 'none' %}` + → spinner; `:47 {% elif job_status == 'failed' %}` → the only "Try Again" form (`:53-55`, posts `/onboarding/retry`, + `onboarding.py:318-337`). Since `'failed'` is never written, a `processing`-wedged job (COR-17) or a `dead` job leaves the PI + on the spinner with no self-service retry. STILL PRESENT. + **Manual mitigation only**: an admin can enqueue a fresh `generate_profile` job (`src/routers/admin.py:1108-1114`), which + becomes the "latest job" the onboarding page reads (`onboarding.py:63-68` orders by `enqueued_at desc`). + +### PR V3 — COR-19 table + +1. **Poison retried forever — FIXED.** `email_inbound.py:33 MAX_S3_PROCESS_ATTEMPTS = 3`; `:186-209` copies to `failed/` and + deletes after the 3rd consecutive failure. Introduced in `8f96f86 fix(email): harden inbound reply processing…` (PR #31). + Tests: `tests/unit/test_email_inbound_hardening.py:259` (`test_poison_email_is_quarantined_after_repeated_failures`) and + `:279` (transient failure not quarantined). +2. **Rate limit — FIXED.** `_reply_rate_ok` `:52-65`, enforced at `:241-245`. Same commit. Tests `hardening.py:195-211`. +3. **Charset — STILL PRESENT.** `email_inbound.py:386-389`: + ``` + def _decode_part(part): + charset = part.get_content_charset() or "utf-8" + payload = part.get_payload(decode=True) or b"" + return payload.decode(charset, errors="replace") + ``` + Ran against the real module (`.venv-test/bin/python`, message with `charset="unknown-8bit"`): + `_decode_part -> LookupError: unknown encoding: unknown-8bit`; `_extract_reply_body -> LookupError` (same). Expression + sweep: `unknown-8bit`, `x-unknown`, `UNKNOWN`, `default`, `x-user-defined` all raise `LookupError`; `utf-8`, `iso-8859-1`, + `windows-1252`, `gb2312`, `ks_c_5601-1987` decode fine. The exception propagates out of `process_inbound_email` → poller + `except` `:186` → counted → quarantined on the 3rd poll (3 × 60 s). Net effect as the issue says: legitimate reply silently + lost to `failed/`. No test mentions `unknown-8bit`/`LookupError`. +4. **String rating — STILL PRESENT.** `:320-322 rating = classification.get("rating") … if not rating or rating < 1 or rating > 4:` + with `classify_reply` returning `json.loads(response_text)` unmodified at `:516`. Snippet on the exact guard expression: + `rating='3' -> TypeError: '<' not supported between instances of 'str' and 'int'`; `'abc'` same; `3`, `None`, `0`, `5` + behave; `2.5` and `True` **pass** the guard (minor extra). Same quarantine fate as row 3. All tests feed integer ratings + (`tests/integration/test_email_inbound_reply_paths.py:105,126,184,215,238`); none feed a string. +5. **Pagination — STILL PRESENT.** `:165 response = s3.list_objects_v2(Bucket=bucket, Prefix=prefix, MaxKeys=50)`; no + `ContinuationToken`/`IsTruncated` in the file. Each cycle processes-and-deletes whatever 50 it sees, so backlog drains + 50/min; starvation is bounded by the quarantine as the issue says. +6. **Side effects before commit — STILL PRESENT.** In `process_inbound_email` (no commit of its own): `_send_review_confirmation` + `:335` (SES), `_send_help_email` `:359` (SES), and inside `_handle_instruction`: inactive-agent email `:605`, migration + `:638`, private-origin email `:655`, Slack post `:706`. The only commit is the poller's `await db.commit()` at `:179`, then + `s3.delete_object` at `:182`. A commit failure keeps the S3 object, re-runs everything next poll, and re-sends. Issue's + cited lines (`:301/:658/:590`) map to `:335/:706/:638`. + +### PR V3 — COR-32 remainder + +Caller `email_inbound.py:338-353`: +``` + if category == "instruction": + instruction = classification.get("instruction", body) + reopened = await _handle_instruction(...) + await record_engagement(user.id, db) + await mark_notification_responded(user.id, td.id, "instruction", db) + if reopened: + await _send_instruction_confirmation(user, notification, td, db) + return +``` +`mark_notification_responded` is unconditional; the poller then commits and deletes the object (`:179-182`). Inside +`_handle_instruction`, `return False` **with** a PI email: inactive agent `:605-612`, already-private origin `:655-662`. +`return False` **without** any email: no simulation run `:681-682`, no bot token `:697-699`, channel not found `:702-704`, +blanket `except` `:717-719`. The already-acted-on guard `:623-628` also returns False silently (arguably correct). A resend +of the same notification hits `:256-258 if notification.status != "sent": … return`. STILL PRESENT. No test exercises the +silent-False paths (reply_paths tests cover the confirmation copy only). + +### PR V4 — email-notification transactional safety + +1. **No rollback**: `grep -c rollback src/services/email_notifications.py` → `0`. Per-item excepts: + `:208-214` (proposal_review, per user), `:721-725` (status_overview, per user), `:912-916` (new_proposal, per proposal/agent); + each logs and continues to the sweep-level `await db.commit()` at `:216`, `:726`, `:917`. A flush error in one item poisons + the session; later items raise `PendingRollbackError` inside the try (logged), and the final commit raises out to + `worker/main.py:158-159`. Rows for earlier users whose SES send already returned success are discarded; next cycle they + have no `sent` row and get re-emailed with a fresh token. Mechanism confirmed by reading; not executed (needs DB). +2. **Phantom-sent**: `send_proposal_notification` `:331-340` builds `EmailNotification(status="sent")`, `db.add`, `await + db.flush()`; the SES call is `:481-485`, failure returns False at `:493-495` leaving the row. `_process_user_notifications` + then skips `last_notification_sent_at`/`consecutive_missed` (`:293-295`) but the row remains; next cycle + `:240-250` finds an outstanding `sent` row and bails forever. Same shape in `_send_new_proposal_email` `:995-1004` vs + `_send_html_email` `:1066`; dedup `:968-976` checks existence regardless of status → permanently lost. + (Positive note: the allowlist check `:320-326` runs *before* row creation, so suppressed recipients don't get phantom rows.) +3. **`"expired"`**: zero occurrences in `email_notifications.py`; in `src/` only the model comment + `src/models/email_notification.py:42`. STILL PRESENT. +4. **Ladder dead**: sole increment `:295` (only after a successful send). Outstanding-row bail `:247-250` precedes it and is + only cleared by `mark_notification_responded`, which is always paired with `record_engagement` (`:594` resets to 0) in every + caller (`email_inbound.py:333-334, 347-348`; `agent_page.py:510-511, 713-714`). Other resets: `:517`, `:523`, + `src/routers/settings.py:133`. So the counter oscillates 0→1→0; `:502 if tracker.consecutive_missed < MISSED_THRESHOLD: return` + always returns; `:505-529` unreachable. STILL PRESENT. + **Delegate reply**: `mark_notification_responded(user_id, …)` filters `EmailNotification.user_id == user_id` (`:606-611`); + both email (`email_inbound.py:334/:348`, `user` = sender matched to the notification's `user_id`) and web + (`agent_page.py:511/:714`, `current_user.id`) pass the *replier's* id. If a delegate answers, the PI's own `sent` row for + that proposal stays `sent`; the PI can't clear it by reviewing (proposal already reviewed → `_handle_review` `:544` and + `_get_unreviewed_proposals_for_user` `:169-177` treat it as done), so the PI is blocked at `:240-250` indefinitely. STILL + PRESENT. `tests/integration/test_proposal_review.py:673-731` covers the PI-reviews-own-row case only. + +### Triage claims + +- `enable_inbound_email` default False — `src/config.py:152`. Accurate. Prod `.env` not inspectable from here. +- `proposal_review` gated on `User.email_notification_frequency != "off"`, `User.email_notifications_paused_by_system.is_(False)`, + `User.email.isnot(None)` — `email_notifications.py:192-199`; rationale `src/models/email_notification.py:99-105`. Accurate. +- 300 s cadence — `src/config.py:158 notification_check_interval: int = 300`; loop `worker/main.py:141`. Accurate. +- "`get_or_create_pref` inserts … each cycle" — `:60-77` SELECTs first and only INSERTs+flushes when no row exists. After the + first pass every existing user is read-only. The sweeps do still write each cycle in other ways: tracker creation for new + users `:230-233`, `last_notification_sent_at` bump for allowlist-suppressed recipients `:267`. Partially accurate. +- Runbook step 4 arms V11/V3/COR-32 — `docs/inbound-email.md:84-90`. Accurate; no prerequisite list there. + +### Definition of done / coverage + +`tests/integration/test_worker.py` exists (commit `d732804 2026-07-30 "Full-system T5: the worker, 15 passed — and a silent +job-loss path"`), drives `claim_job`, `process_job`, `execute_generate_profile`, `execute_monthly_refresh`, `run_worker` +against a real Postgres. The "from 0 % coverage" wording is stale on `copi-prod`. It does not, however, provide a +fail-against-pre-fix test for COR-17/COR-18 — the DB-error test asserts the *current* broken behaviour by design. + +### Where the issue text is stale/wrong (separate from the defects) + +- All `email_inbound.py` / `email_notifications.py` line refs drifted (see mapping in rows above); `worker/main.py` refs still exact. +- `pi_inbox.py:151` (`record_pi_dm`) is not on the worker's path; only `:120` is. +- `grantbot.py:724/757` → `:731/:764`. +- "all 3 attempts burn in ~15 s" — actually immediate; there is no inter-attempt sleep at all. +- "worker/main.py 0 % coverage" — a 15-test integration suite exists. +- "`get_or_create_pref` inserts … each cycle" — inserts only on first sight of a user. + +## 3. Counts + +Sub-claims assessed: 33. **26 still present / accurate**, **2 fixed** (COR-19.1, COR-19.2), **1 partially accurate** (T-5), +**1 stale** (DoD 0 % coverage), **0 changed-mechanism**, **1 not verifiable** (T-2 prod env), plus 2 minor stale references +(pi_inbox.py:151, "~15 s") that do not alter any verdict. + +## 4. What I could not verify and why + +- Whether `ENABLE_INBOUND_EMAIL` is unset on prod: prohibited from ssh/containers. Only the local dev `.env` was checked + (key absent), which says nothing about prod. +- V4-1's "duplicate email next cycle" end-to-end consequence and COR-17's `PendingRollbackError` path were confirmed by code + reading and by the existing T5 characterization test's assertions, not by executing against a database (no DB permitted). +- Prod user `email_notification_frequency` values (whether the `proposal_review` sweep is actually sending today): needs prod DB. +- Whether SES/Slack side effects actually duplicate on a commit failure in practice: mechanism confirmed, not exercised. diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_21_redteam.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_21_redteam.md new file mode 100644 index 00000000..7e9788b6 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_21_redteam.md @@ -0,0 +1,140 @@ +# Issue #21 — red-team pass (second reviewer) + +Tree: `copi-prod` @ 18ba52c, clean. Everything below re-derived by my own grep/sed/python against the current tree; +`git show 8f96f86^:src/services/email_inbound.py` used to read the pre-fix code. No DB/Docker/network. + +## 1. Table + +| id | first-agent verdict | red-team result | one-line reason | evidence | +|---|---|---|---|---| +| V11-a | ACCURATE | UPHELD | four slots + `_default = TsMinter(WRITER_WEB)` | `src/agent/ids.py:47-50`, `:111` | +| V11-b | ACCURATE (lines drifted) | UPHELD | callers exactly `src/main.py:113`, `src/agent/main.py:54`, `src/agent/grantbot.py:731,764` | grep `set_default_writer_id` | +| V11-c | STILL PRESENT | UPHELD | `grep -c ids src/worker/main.py` = 0 in imports; no `set_default_writer_id` in file | `src/worker/main.py:6-18` | +| V11-d | STILL PRESENT | UPHELD | `email_inbound.py:674`→`pi_inbox.py:120`; `:638`→`private_channels.py:269`; `record_pi_dm` (`pi_inbox.py:139/151`) is called only from `pi_handler.py`, `simulation.py`, `agent_page.py` — not worker | grep `record_pi_dm` | +| V11-e | ACCURATE | UPHELD | `src/config.py:152 enable_inbound_email: bool = False`; gate `src/worker/main.py:162` | sed | +| V11-f | STILL PRESENT | UPHELD | `docs/inbound-email.md:84-90` step 4; `grep -i "writer\|slot"` on the doc → no hits | grep | +| V11-g | ACCURATE | UPHELD | `grep -rln "mint_local_ts\|TsMinter" scripts/` → nothing | grep | +| V11-h | STILL PRESENT | UPHELD | `specs/local-db-conversations.md:66-67` "Three processes"; `remediate_duplicates.py:131 = 99` | sed | +| COR-17 | STILL PRESENT (characterization test) | UPHELD | `worker/main.py:100-111` no rollback; `tests/integration/test_worker.py:760-827` asserts `status == "processing"` and `last_error is None` — i.e. asserts the BUG; flipping it is still owed | sed | +| COR-18a | STILL PRESENT | UPHELD | only writer `worker/main.py:49`; all other `started_at` hits are `SimulationRun.started_at` | grep | +| COR-18b | STILL PRESENT | UPHELD | `"processing"` in src/ only at `worker/main.py:48` and display `admin.py:126` | grep | +| COR-18c | STILL PRESENT, retry immediate | UPHELD | `worker/main.py:127-137`: `asyncio.sleep` only in the `else:` (no-job) branch; after `process_job` returns the loop re-enters `claim_job` with no delay; `claim_job` orders by `enqueued_at` so the re-`pending` job is first | sed 127-137 | +| COR-18d | STILL PRESENT | UPHELD | `worker/main.py:110` | sed | +| COR-18e | STILL PRESENT | UPHELD | enum `models/job.py:23`; worker writes processing/completed/dead/pending only; `tests/unit/test_reachability.py:39-44` documents it | grep | +| COR-18f | STILL PRESENT | QUALIFIED | verdict stands, but a `dead` job does NOT show the spinner — it renders no branch at all (blank body); the spinner is for `processing`-wedged (COR-17) only. Admin mitigation mis-cited (see §3) | template `profile_review.html:28,47,60` | +| COR-19.1 | FIXED (8f96f86) | UPHELD | pre-fix file has no `copy_object`/`_S3_FAILURE_COUNTS`/`MAX_S3_PROCESS_ATTEMPTS`; the test's `fake.copied == [("inbound/poison","failed/poison")]` would fail (pre-fix: `[]`; the `monkeypatch.setattr(inbound, "_S3_FAILURE_COUNTS")` would already AttributeError). Quarantine-path exceptions are caught (`:207-208`), counter not popped → re-attempted next poll. Caveats in §2 | `email_inbound.py:33,48,186-208`; `git show 8f96f86^:…` | +| COR-19.2 | FIXED (8f96f86) | UPHELD | pre-fix has the constant (`:26`) and no `_reply_rate_ok`; test imports the symbol → ImportError pre-fix; enforced at `:241` before the notification lookup. Note: a rate-limited reply returns normally → poller commits + deletes the S3 object (dropped, not deferred) | `email_inbound.py:52-65,241-245` | +| COR-19.3 | STILL PRESENT | UPHELD (re-run) | `_decode_part -> LookupError unknown encoding: unknown-8bit`; `_extract_reply_body` same | python run below | +| COR-19.4 | STILL PRESENT | UPHELD (re-run) | `'3' -> TypeError`; `2.5`/`True` accepted (extra) | python run below | +| COR-19.5 | STILL PRESENT | UPHELD | `:165 MaxKeys=50`, no `ContinuationToken`/`IsTruncated` | grep | +| COR-19.6 | STILL PRESENT | UPHELD | side effects `:335,:352,:359,:605,:638,:655,:706`; sole commit `:179`, delete `:182` | grep | +| COR-32 | STILL PRESENT | QUALIFIED | cites exact (`:612,:628,:662,:682,:699,:704,:719`); caller `:348` unconditional. Wording: the silent-consume path is reachable on the DEFAULT config too — `enable_private_refinement: bool = True` (`config.py:391`) routes into `migrate_public_thread_to_private` inside the same `try`, so a migration failure hits the blanket `except` `:717-719` → False → notification retired, no email. Not legacy-only | awk over 596-722 | +| V4-1 | STILL PRESENT | UPHELD | `grep -c rollback` = 0; excepts/commits at cited lines | grep/sed | +| V4-2 | STILL PRESENT | QUALIFIED (worse) | for `new_proposal` the phantom row is created on EVERY allowlist-suppressed recipient, not just SES failure: row+flush `:995-1004` precedes `_send_html_email` whose allowlist check (`:634-636`) returns False; dedup `:968-976` then skips forever. First agent's "positive note" holds only for `proposal_review` (`:319-326` before row) | sed | +| V4-3 | STILL PRESENT | UPHELD | 0 hits in service; only `models/email_notification.py:42` comment | grep | +| V4-4a | STILL PRESENT | UPHELD | increment `:295`; bail `:247-250` → `_check_engagement_and_downgrade` `:502` returns at `<3`; resets `:517,:523,:594`, `settings.py:133`; only `status="responded"` writer is `:614` (`mark_notification_responded`), always paired with `record_engagement` | grep | +| V4-4b | STILL PRESENT | UPHELD (+ sharpened) | filter `:606-611` on replier id; web `review_proposal` raises 400 "Already reviewed" (`agent_page.py:490-497`) BEFORE `mark_notification_responded` (`:511`) → PI cannot clear its row via web once a delegate reviewed; an email reply would clear it (`_handle_review` returns early but caller still marks responded `:334`) but inbound is off | sed | +| T-1 | ACCURATE | UPHELD | `config.py:152` | | +| T-2 | NOT VERIFIABLE | UPHELD | repo-side evidence all consistent with "unset": `.env.example` has no inbound key (51 lines, 0 hits), local `.env` 0 hits, `docker-compose.prod.yml` passes only `env_file: .env` + `ENVIRONMENT`, `docs/inbound-email.md:41` and `scripts/setup_inbound_email.py:16` both state it is unset in prod | grep | +| T-3 | ACCURATE | UPHELD | `email_notifications.py:192-199`; `get_or_create_pref` callers are only `status_overview` (`:714`), `new_proposal` (`:963`), settings router — never `proposal_review` | grep callers | +| T-4 | ACCURATE | UPHELD | `config.py:158 = 300`; `worker/main.py:141` | | +| T-5 | PARTIALLY ACCURATE | UPHELD | `:60-77` SELECT then INSERT only if None; composite PK `(user_id, category)` confirmed `models/email_notification.py:109-115`. Per-cycle write inventory in §2 | sed | +| T-6 | ACCURATE | UPHELD | `docs/inbound-email.md:84-90` | | +| DoD | STALE | UPHELD | `test_worker.py` 15 tests, d732804 (2026-07-30), NOT an ancestor of `main@b7edcbc` — so the issue was right when filed and is stale now | `git merge-base` | + +## 2. Detail — QUALIFIED rows and NEW claims + +### COR-19.1 FIXED — upheld, with three caveats the first agent did not state +Pre-fix (`git show 8f96f86^:src/services/email_inbound.py`): only `list_objects_v2` (:93) and the success-path `delete_object` (:110); no +copy, no counter. Current `:186-208`: counter++ on any exception; at >= `MAX_S3_PROCESS_ATTEMPTS` (3) `copy_object` → `failed/` then `delete_object`, then pop. Test `test_poison_email_is_quarantined_after_repeated_failures` (`hardening.py:259-276`) would fail +pre-fix on the `fake.copied` assertion (and earlier on the missing module attribute). Caveats: +1. If `copy_object`/`delete_object` themselves throw, the inner `except Exception` (`:207-208`) only logs; the counter is NOT popped, so the + next poll re-attempts quarantine. If the failure is permanent (IAM lacks `s3:PutObject` on `failed/*`) the object is retried every poll + forever — pre-fix behaviour returns. `scripts/setup_inbound_email.py:168-171` does grant PutObject on `failed/*`, so this is a runbook + prerequisite, not a code bug. +2. The counter cannot distinguish poison from transient: a DB outage of 3 × `inbound_poll_interval` (3 × 60 s) quarantines every object in + the bucket, i.e. legitimate replies are lost to `failed/` (recoverable only by hand). Same mechanism the first agent noted for COR-19.3. +3. `_S3_FAILURE_COUNTS` is in-memory (documented at `:47`); a worker restart resets the count. + +### COR-19.2 FIXED — upheld +`_reply_rate_ok` (`:52-65`) is a sliding window; called at `:241` after auth-results and auto-submitted gates, before the notification +lookup. Tests `hardening.py:195-211` exercise the window directly. Pre-fix code has the constant but no function → the test module's +`from ... import _reply_rate_ok` fails at import. Nuance: a rate-limited reply is `return`ed normally, so the poller commits and deletes +the S3 object — the 11th reply in an hour is dropped, not deferred. + +### COR-18c NEW claim (retry immediate) — reproduced by reading +``` +127 while not _shutdown: +130 job = await claim_job(db) +132 if job: +134 await process_job(...) +135 else: +137 await asyncio.sleep(settings.worker_poll_interval) +``` +No sleep after `process_job`. `process_job` re-sets `status="pending"` (`:108`) and commits; the next iteration's `claim_job` (ordered by +`enqueued_at`) picks it straight back up. Three attempts run back-to-back; the issue's "~15 s" is an overstatement of the delay. + +### T-5 NEW claim (`get_or_create_pref` SELECTs first) — confirmed; per-300 s-cycle write inventory +- `check_and_send_notifications` (proposal_review): only users with `frequency != 'off'` AND not paused AND email set. Per user: + `EmailEngagementTracker` INSERT on first sight (`:230-233`); if due and no outstanding `sent` row and unreviewed proposals exist: + allowlist-suppressed → `tracker.last_notification_sent_at` UPDATE (`:267`); else `EmailNotification` INSERT (`:331-340`) + tracker UPDATE + (`:294-295`). If quieting was done by setting `User.email_notification_frequency='off'` this sweep writes nothing; if it was done via + pref rows only, it is unaffected (T-3). +- `check_and_send_status_overviews`: ALL users with email. `get_or_create_pref` INSERT on first sight only; disabled pref → no writes. + Enabled+due → `pref.last_sent_at` UPDATE on success. +- `check_and_send_new_proposal_emails`: proposals in last 7 days × 2 agents × recipients. `get_or_create_pref` INSERT on first sight; + disabled → skip; enabled + no existing row → `EmailNotification` INSERT (`:995-1004`) regardless of send outcome (see V4-2). +So with prefs disabled, steady-state per-cycle writes are: tracker inserts for newly eligible proposal_review users, pref inserts for +newly created users, and nothing else. The issue's "inserts on a composite-PK table each cycle" is wrong as stated. + +### COR-18f QUALIFIED +`templates/onboarding/profile_review.html`: `:28 if pending/processing/none` → spinner; `:47 elif 'failed'` → retry form; `:60 elif profile` +→ profile. A `dead` job with no profile matches none → blank body (no spinner, no retry). A `processing`-wedged job (COR-17) → spinner. +Verdict (no self-service retry) stands; the first agent's "dead job leaves the PI on the spinner" is wrong. + +### COR-32 QUALIFIED (wording/reachability) +Exact `return False` sites: `:612` (inactive, emailed), `:628` (already acted on), `:662` (private origin, emailed), `:682` (no run), +`:699` (no token), `:704` (no channel), `:719` (blanket except). `config.py:391 enable_private_refinement: bool = True` means the default +path is `migrate_public_thread_to_private` (`:638`) inside the same `try`; any failure there lands in `:717-719`. The issue's "can't resolve +a bot token/channel" framing (legacy path) understates reach: the default path silently consumes the instruction too. + +### V4-2 QUALIFIED (worse for new_proposal) +`_send_new_proposal_email` `:995-1004` creates `status="sent"` + `flush()`, then `_send_html_email` (`:624`) checks +`is_allowed_recipient` at `:634-636` and returns False. `outbound_email_allowlist` defaults `""` (allow all, `config.py:150`, +`email.py:91-93`), so this only bites when the allowlist is set — but when it is, every suppressed recipient gets a permanent phantom row +that the dedup (`:968-976`) honours even after the allowlist is widened. + +### V4-4b sharpened +`agent_page.py:490-497` raises `HTTPException(400, "Already reviewed")` before `:511 mark_notification_responded`. So after a delegate +reviews, the PI has no web path to retire their `sent` row. The email path would (caller marks responded unconditionally `:334`) but +inbound is off. + +### Reproductions (run with `.venv-test/bin/python`) +``` +_decode_part -> LookupError unknown encoding: unknown-8bit +_extract_reply_body -> LookupError unknown encoding: unknown-8bit +'3' -> TypeError '<' not supported between instances of 'str' and 'int' +'abc' -> TypeError '<' not supported between instances of 'str' and 'int' +3 -> accepted None -> unparseable 0 -> unparseable 5 -> unparseable 2.5 -> accepted True -> accepted +``` + +### Reachability +`enable_inbound_email` default False (`config.py:152`); the only consumer is `worker/main.py:162`; `poll_inbound_emails` / +`process_inbound_email` have no other callers in src/ or scripts/. `.env.example` (51 lines) has no inbound key; local `.env` none; +`docker-compose.prod.yml` sets only `ENVIRONMENT` explicitly. Later commits on the three files since b1d54da: `364bee3` (fail closed on +null-email users; help email reply-able), `f94d2a8` (merge #31), `0e2ed84` (model bump) — none touch the mechanisms above. + +## 3. Mis-cites / wording errors in the first-agent report +- COR-18f mitigation "admin can enqueue a fresh generate_profile job (`src/routers/admin.py:1108-1114`)": `:1109` is inside the admin + *impersonation* flow that creates a brand-new user; the re-enqueue-for-existing-user sites are `:1217` and `:1283` (approve/allow + routes) and fire only when the user has no profile yet. There is no admin "re-run profile" action for a wedged job. +- COR-18f "a `dead` job leaves the PI on the spinner": renders no branch (blank), see §2. +- COR-19.1 report cites quarantine at `:186-209`; the block is `:186-208` (trivial). +- Everything else I checked (ids.py, grantbot, pi_inbox, private_channels, email_inbound `:335/:352/:359/:605/:638/:655/:706`, + email_notifications `:208-214/:721-725/:912-916/:216/:726/:917/:331-340/:481-495/:502/:517/:523/:594/:606-611`, settings `:133`, + agent_page `:511/:714`, template `:28/:47`) points at the quoted code. + +## 4. Counts +33 rows: **29 upheld**, **0 overturned**, **4 qualified** (COR-18f, COR-32, V4-2, V4-4b — all verdicts stand; evidence/wording/severity +adjusted), **0 unverifiable** beyond T-2 (already marked so by the first agent; upheld as unverifiable). diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_22.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_22.md new file mode 100644 index 00000000..f3cdfa74 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_22.md @@ -0,0 +1,263 @@ +# Issue #22 verification — Profile pipeline & write integrity + +Verified against `copi-prod` @ `18ba52c` (clean tree), 2026-09-02. All line numbers below are CURRENT (HEAD), located by symbol. Snippets were executed with `.venv-test/bin/python` against the real modules (no network, no DB, no Docker). + +Drift note: between the issue's reference tree (`b1d54da`) and HEAD, the only one of the named files that changed is `src/routers/admin.py` (commit `18ba52c`), plus `src/agent/simulation.py`. Every other line number the issue cites is still exact. The two admin.py references drifted: `admin.py:92` → `:105-108`, `admin.py:105` → `:120`; `_load_publication_records` moved from `simulation.py:4565` → `:4679`. + +## 1. Summary table + +| id | claim | verdict | key evidence | conf | +|---|---|---|---|---| +| V1-15a | `fetch_orcid_works`: `"external-ids": null` → `AttributeError`, parse is outside the try | STILL PRESENT | `src/services/orcid.py:103-109` (try wraps fetch only), `:127`; ran → `AttributeError: 'NoneType' object has no attribute 'get'` | high | +| V1-15b | same for `"title": null` (`:115`) | STILL PRESENT | `orcid.py:115`; ran → `AttributeError` (also for `title.title: null`) | high | +| V1-15c | same chain in `fetch_orcid_grants` (`:92`) | STILL PRESENT | `orcid.py:92`; ran → `AttributeError` for `title: null` and `title.title: null` | high | +| V1-15d | same chain in `fetch_orcid_profile` (`:29-39`) | PARTIALLY (issue overstates) | `orcid.py:29-31` — `name: null` IS guarded (`if name_block else ""`); but `given-names: null`, `emails: null`, `researcher-urls: null`, `organization: null`, `display-index: null` all raise (`:30,:35-39,:58,:61,:68`) | high | +| V1-15e | `int(pub_date["year"]["value"])` on null / non-numeric year (`:124`) | STILL PRESENT | `orcid.py:122-124`; ran → `TypeError` for `value: null`, `ValueError` for `"n.d."` | high | +| V1-15f | step-3 catch zeroes the works list; `lost_evidence` gate partly contains; first-run victims stored ungrounded | STILL PRESENT (accurate; two nuances) | `profile_pipeline.py:107-112` sets `orcid_works=[]`, `works_lookup_failed=True`; gate `:391-392` protects only rows with stored `evidence_pub_count>0` (post-0023) and `synthesis_validated is not False` | high | +| V1-15g | no test pins null tolerance | STILL PRESENT | `tests/contract/test_orcid_contract.py:108-114` uses fully populated `external-ids`; no `None` container anywhere | high | +| V1-16a | no `(user_id, pmid)` unique constraint | STILL PRESENT | `src/models/publication.py:13-41` no `__table_args__`; `alembic/versions/0001_initial.py:121-122` non-unique indexes only; no later migration touches `publications` | high | +| V1-16b | `pmids` built without dedup | STILL PRESENT | `profile_pipeline.py:115`, `:147` (DOI path appends without check); replicated: `['111','111','111']` | high | +| V1-16c | `existing_pubs` never updated inside the insert loop | STILL PRESENT | `profile_pipeline.py:182` built once; loop `:212-229` adds to `new_publications`, never to `existing_pubs` | high | +| V1-16d | `scalar_one_or_none()` → `MultipleResultsFound` swallowed at `logger.debug` | STILL PRESENT | `profile_pipeline.py:272-282` (`except Exception` → `logger.debug`) | high | +| V1-16e | duplicated citation lines in export | STILL PRESENT | `profile_export.py:78-106` — sort/slice, no pmid/doi dedup | high | +| V1-16f | inflated admin counts | STILL PRESENT (line drifted) | `admin.py:105-108` `func.count(Publication.id)` grouped by user, no `DISTINCT pmid` | high | +| V1-16g | `_load_publication_records` full join, duplicates multiply rows, set absorbs | STILL PRESENT (line drifted) | `simulation.py:4679-4709` — plain join, `record.dois.add(...)` | high | +| V1-pm1 | `pubmed.py:207-208` `.text` truncates title at first child | STILL PRESENT | ran `_parse_pubmed_xml` → title `'Role of '`; `"".join(itertext())` → `'Role of TP53 in cancer'` | high | +| V1-pm2 | abstract has the identical defect (`:213-218`) | STILL PRESENT | `pubmed.py:214` `abstract_el.text or ""`; ran → `'BACKGROUND: We studied Plain '` | high | +| V1-pm3 | no `itertext()` anywhere in `src/` | STILL PRESENT (with a missed helper) | grep: none; BUT `pubmed.py:404-413 _extract_text` is a recursive text collector already used for PMC methods | high | +| V1-pm4 | propagates to `fetch_abstract`/`fetch_full_text` | STILL PRESENT | `pubmed.py:441-453` returns `rec.get("title")` from the same parser | high | +| V1-val1 | `_validate_profile` does `research_summary.split()` on `None` → `AttributeError`; call `:317` not in try | STILL PRESENT | `profile_pipeline.py:561-562`, `:317`; ran → `AttributeError`; `llm._extract_json` (`llm.py:120-167`) is raw `json.loads`, no schema | high | +| V1-val2 | techniques check uses `len()` not `isinstance`; `"PCR"` passes; `:409` assigns str to ARRAY | STILL PRESENT | `:569-570`; ran `techniques="PCR"` → `True`; `:409` `profile.techniques = synthesized.get("techniques", [])` | high | +| V1-val3 | word gate `<100 or >350` disagrees with log + retry prompt ("150-250") | STILL PRESENT | `:563` vs `:565`, `:324`, `:430`; ran: 120 and 300 words both return `True` | high | +| V6-22-gate | pipeline gates overwrite on `validated`/`lost_evidence` ("fixed in stack") | FIXED (confirmed) | `profile_pipeline.py:383-406`; commit `d311170` | high | +| V6-22-flag | `synthesis_validated` persisted (`:414`) | FIXED (confirmed) | `profile_pipeline.py:414`; commit `d311170` | high | +| V6-22a | no web save route resets `synthesis_validated` | STILL PRESENT | grep: only writer is `profile_pipeline.py:414`; consequence chain `:389` | high | +| V6-22b | first-ever run stores unvalidated / zero-evidence unconditionally | STILL PRESENT (deliberate) | `:384-385` requires `profile_version>0`, so first run always takes the `else` at `:407-418` | high | +| V6-22c | `raw_abstracts_hash` written on the discard path | STILL PRESENT | `profile_pipeline.py:372` (before the `if synthesized:` gate) | high | +| V6-pend | `pending_profile` read-but-never-written; sole reader `admin.py` unreachable | STILL PRESENT (line drifted 105→120) | writer grep: none; `admin.py:120`; `templates/profile/view.html:42-48` comment confirms; 4 spec files still describe the flow | high | +| V6-form1 | `Form("")` blanking `agent_page.py:1233-1262` | STILL PRESENT | `agent_page.py:1233-1238` defaults, `:1256-1262` unconditional assigns | high | +| V6-form2 | `Form("")` blanking `profile.py:113-166` | STILL PRESENT | `profile.py:113-118`, `:160-166` | high | +| V6-form3 | `Form("")` blanking `onboarding.py:111-154` | STILL PRESENT | `onboarding.py:111-116`, `:148-154` | high | +| V6-form4 | `profile.py` guards user fields (`if name:`) but not profile fields | STILL PRESENT (and worse) | `profile.py:144-149` vs `:160-165`; `if institution is not None` / `if department is not None` are dead guards (Form("") is never None) | high | +| V6-24a | no `os.replace`/tempfile/`flock` in `src/` | STILL PRESENT | grep over `src/` for `os.replace|tempfile|flock|NamedTemporaryFile|mkstemp`: 0 hits | high | +| V6-24b | truncate-then-write at `profile_export.py:118/:142`, `agent_page.py:1131`, `agent.py:711/:738` | STILL PRESENT (lines exact) | all five `write_text` sites confirmed at those lines | high | +| V6-24c | public writers consistently DB→disk | STILL PRESENT (accurate) | `profile.py:168→185`, `onboarding.py:156→172`, `agent_page.py:1264→1276` (commit then export) | high | +| V6-24d | private save `agent_page.py:1129-1140` disk-first, `if profile:` guard, no `encoding=` | STILL PRESENT (lines exact) | `agent_page.py:1131 profile_path.write_text(content)`, `:1138 if profile:` | high | +| V6-24e | pipeline writes disk `:482` under flush-only txn; commit in `worker/main.py:97` | STILL PRESENT (lines exact) | `profile_pipeline.py:466 flush`, `:482 export`, `:497 flush`; `worker/main.py:97 commit` | high | +| V6-24f | `create_revision` content-dedup ("fixed in stack") | FIXED (confirmed) | `profile_versioning.py:84-92`; commit `da405cb`; pinned by `tests/integration/test_cli.py:731` | high | +| V6-23 | seed written to DB, never exported; agent reads disk only; `export_private_profile` no-ops on empty `private_profile_md` | STILL PRESENT | `profile_pipeline.py:458-464`; `agent.py:119-126`; `profile_export.py:136-137`; no script exports seeds | high | +| C1-a | `profile_version = (x or 0)+1` at 4 sites | STILL PRESENT (lines exact) | `profile_pipeline.py:417`, `profile.py:166`, `onboarding.py:154`, `agent_page.py:1262` | high | +| C1-b | `delegate_slack_ids` whole-column reassign at 3 sites after an awaited Slack lookup | STILL PRESENT (lines exact) | `agent_page.py:1423-1426`, `agent_page.py:1609-1612`, `invite.py:241-244` | high | +| C1-c | `with_for_update` appears once (job claim); no SQL-side increment / `array_append` | STILL PRESENT | `worker/main.py:42` only; grep `array_append|array_remove`: 0 | high | +| DoD | migration test for unique constraint vs pre-existing duplicates | N/A (nothing to test yet) | no constraint, no migration | high | + +## 2. Per-item detail + +### PR V1 — COR-15 (ORCID null-safety) + +Current code (`src/services/orcid.py`): + +``` +102 async with httpx.AsyncClient(timeout=30) as client: +103 try: +104 resp = await client.get(url, headers=headers) +105 resp.raise_for_status() +106 data = resp.json() +107 except Exception as exc: +108 logger.warning("Failed to fetch ORCID works for %s: %s", orcid_id, exc) +109 return [] +... +115 "title": summary.get("title", {}).get("title", {}).get("value", ""), +... +122 pub_date = summary.get("publication-date", {}) +123 if pub_date and pub_date.get("year"): +124 work["year"] = int(pub_date["year"]["value"]) +... +127 ext_ids = summary.get("external-ids", {}).get("external-id", []) +128 for eid in ext_ids: +129 id_type = eid.get("external-id-type", "").lower() +``` + +Ran the real functions with `httpx.AsyncClient` patched to return canned payloads: + +``` +works external-ids=None -> RAISED AttributeError: 'NoneType' object has no attribute 'get' +works title=None -> RAISED AttributeError: 'NoneType' object has no attribute 'get' +works title.title=None -> RAISED AttributeError +works year.value=None -> RAISED TypeError: int() argument must be ... not 'NoneType' +works year.value='n.d.' -> RAISED ValueError: invalid literal for int() with base 10: 'n.d.' +works publication-date.year=None -> OK (this one IS guarded by :123) +works external-id-type=None -> RAISED AttributeError: 'NoneType' object has no attribute 'lower' (not in the issue) +works fully populated -> OK +grants title=None -> RAISED AttributeError +grants title.title=None -> RAISED AttributeError +profile name=None -> OK (guarded, :30-31 — issue's ":29-39" overstates) +profile given-names=None -> RAISED AttributeError +profile emails=None -> RAISED AttributeError +profile researcher-urls=None -> RAISED AttributeError +profile organization=None -> RAISED AttributeError +profile display-index=None -> RAISED TypeError (:58 int(); not in the issue) +``` + +Try scope: the `try` at `:103-109` wraps only the HTTP fetch/JSON decode; the parse loop `:111-137` is outside it, so any of the above escapes `fetch_orcid_works`. The last commit to `orcid.py` is `1c34878` (display-index sort) — no null-hardening since. + +Wrapper in the caller (`profile_pipeline.py:107-112`): +``` +107 try: +108 orcid_works = await fetch_orcid_works(orcid_id) +109 except Exception as exc: +110 logger.warning("Step 3 failed: %s", exc) +111 orcid_works = [] +112 works_lookup_failed = True +``` +So the issue's "step-3 catch zeroes the whole works list" holds. Two nuances the issue did not state: +- `works_lookup_failed=True` makes `evidence_pmid_count=None` (`:380`), so the stored first-run profile carries `evidence_state == "evidence_lost"` — it is flagged, not silent. (`test_profile_pipeline_orcid_works_failure_is_not_reported_as_no_works`, `tests/characterization/test_profile_pipeline_gm.py:728`, pins this for a *raising* stub, which is exactly the AttributeError path.) +- The `lost_evidence` containment (`:391`) requires the STORED row to have `evidence_pub_count > 0`. Pre-0023 rows have `NULL` there (`models/profile.py:44-46`: "legacy rows are NOT backfilled"), so a legacy grounded profile is NOT protected from a refresh that hits this crash — it gets overwritten by a validated, zero-evidence synthesis. The issue's "damage is now partly contained" is therefore narrower than stated. + +Tests: `tests/contract/test_orcid_contract.py:104-125` — one work, fully populated `external-ids`, `publication-date`, `title`. No test feeds a null container. `tests/live_api/test_orcid_live.py` inspects live payload shape (needs network; not run). + +### PR V1 — COR-16 (publication dedup) + +Constraint: `src/models/publication.py:13-41` has no `__table_args__`, `pmid` is `String(20), nullable=True` (`:22`). Migrations: grep of `alembic/versions/*.py` for `publication|pmid|unique` shows only `0001_initial.py:121-122` (`create_index` `ix_publications_user_id`, `ix_publications_pmid`, both non-unique); migrations 0002–0024 never touch `publications`. + +Pipeline (`profile_pipeline.py`): +``` +115 pmids = [w["pmid"] for w in orcid_works if w.get("pmid")] +... +146 w["pmid"] = resolved_pmid +147 pmids.append(resolved_pmid) +... +182 existing_pubs = {p.pmid: p for p in existing_result.scalars().all() if p.pmid} +... +187 for rec in pubmed_records: +... +212 if pmid in existing_pubs: +213 pub = existing_pubs[pmid] +... +217 else: +218 pub = Publication( +... +228 db.add(pub) +229 new_publications.append(pub) +``` +Replicated the list/loop logic with two ORCID listings sharing PMID 111 plus a DOI-only listing resolving to 111: `pmids == ['111','111','111']`, and the loop (fresh user) would `db.add` three rows because `existing_pubs` is never updated. Note the DOI-only path dedups DOIs (`:127-134`) but not against PMIDs already in `pmids`. + +Whether `pubmed_records` actually contains a duplicate for a *same-batch* duplicate id depends on NCBI efetch echoing duplicates — not verifiable offline. Two deterministic paths do not depend on that: (a) the same PMID landing in two different 100-id batches (`pubmed.py:109-110`), (b) a re-run where the same run's flush already inserted the row is fine, but two *concurrent* runs for one user (monthly refresh + manual `/profile/refresh`) both see empty `existing_pubs`. + +Downstream, all as claimed: +- `profile_pipeline.py:272-282`: `scalar_one_or_none()` on `(user_id, pmid)` inside `try/except Exception` → `logger.debug(...)`; `MultipleResultsFound` silently drops `methods_text`. +- `profile_export.py:78-106`: filters on `p.title`, sorts by year, slices 20, emits one line per row; no pmid/doi dedup. +- `admin.py:105-108`: `select(Publication.user_id, func.count(Publication.id)).group_by(Publication.user_id)`. +- `simulation.py:4679-4709`: `select(AgentRegistry.agent_id, Publication.doi).join(...)`, accumulates into `record.dois` (a set) — dedup-immune in result, linear in row count. + +Other writers of `publications` (relevant to the "IntegrityError discipline" note): `scripts/backfill_publications.py:86` (skips PMIDs already in DB, `:59-74`, but does NOT dedup its own input list — a PMID listed twice in the JSON for one agent is added twice, `:68-97`), `scripts/generate_sparsedata_user.py:576` (dev seed). + +Tests: `test_profile_pipeline_rerun_increments_version_and_updates_pubs` (`test_profile_pipeline_gm.py:337`) pins `pub_count_after_two_runs == 2` — i.e. cross-run dedup via `existing_pubs` works. No test feeds duplicate PMIDs within one run. `tests/unit/test_backfill_publications.py` has no duplicate-input case. + +### PR V1 — PubMed `.text` truncation + +``` +207 title_el = article.find(".//ArticleTitle") +208 record["title"] = (title_el.text or "") if title_el is not None else "" +... +212 for abstract_el in article.findall(".//AbstractText"): +213 label = abstract_el.get("Label") +214 text = abstract_el.text or "" +``` +Ran `_parse_pubmed_xml` on `Role of TP53 in cancer` and a labelled abstract with ``/``: +``` +title = 'Role of ' +abstract= 'BACKGROUND: We studied Plain ' +''.join(itertext()) = 'Role of TP53 in cancer' +``` +No `itertext` in `src/` (grep). Missed by the issue: `pubmed.py:404-413 _extract_text(element)` is a recursive text+tail collector already in the module (used for PMC ``), so the fix is one call away. `fetch_abstract` (`:441-453`) and `fetch_full_text` (`:463-486`) return `rec["title"]`/`rec["abstract"]` from this parser, so the truncated title reaches the agent tools as claimed; DOI (`:190-204`) is unaffected. + +Tests: `tests/contract/test_pubmed_contract.py:30,66-67` use plain-text `A Great Paper` and assert the full string — passes with either `.text` or `itertext`; nothing pins markup handling. + +### PR V1 — `_validate_profile` + +``` +561 research_summary = profile.get("research_summary", "") +562 word_count = len(research_summary.split()) +563 if word_count < 100 or word_count > 350: +564 logger.warning( +565 "Research summary word count %d outside 150-250 range", word_count +... +569 techniques = profile.get("techniques", []) +570 if len(techniques) < 3: +``` +Ran: +``` +research_summary=None -> RAISED AttributeError 'NoneType' object has no attribute 'split' +research_summary=123 -> RAISED AttributeError +techniques='PCR' -> True +techniques=None -> RAISED TypeError object of type 'NoneType' has no len() +disease_areas='cancer'-> True +word_count 120 -> True (message/prompt say 150-250) +word_count 300 -> True +``` +Call site `profile_pipeline.py:317 validated = _validate_profile(synthesized)` is not in a try; the retry call `:327` is inside a try that only logs. `synthesize_profile` → `_extract_json` (`llm.py:120-167`) is bare `json.loads` with no schema/type coercion, so `null` and string-typed arrays pass straight through. An escaping exception reaches `worker/main.py:99-111` → retry → `dead` (pinned generically by `tests/integration/test_worker.py:381`). `:409 profile.techniques = synthesized.get("techniques", [])` would hand a `str` to `ARRAY(String)`. Retry prompt `:324` and progress text `:430` both say "150-250". + +### PR V6 — COR-22 + +FIXED as claimed (commit `d311170`): `profile_pipeline.py:383-406` gate (`stored_is_worth_keeping`, `lost_evidence`), `:414 profile.synthesis_validated = validated`. Pinned by `test_profile_pipeline_gm.py:472,532,759`. + +Residuals, all STILL PRESENT: +- (a) grep `synthesis_validated` across `src/`: the only assignment is `profile_pipeline.py:414`. None of `profile.py:160-166`, `onboarding.py:148-154`, `agent_page.py:1256-1262` touch it. Consequence per `:389`: a PI-edited draft that was stored with `False` stays `False`, so `stored_is_worth_keeping` is `False` and the next refresh overwrites it even with a synthesis that itself failed validation. +- (b) `:384-385` `(profile.profile_version or 0) > 0` — a first run always falls to `:407-418` and stores whatever came back, including `validated=False` / `evidence_pub_count=0`. Documented as deliberate `:342-366`. +- (c) `:372 profile.raw_abstracts_hash = abstracts_hash` executes before the `if synthesized:` gate — written on the discard path. + +`pending_profile`: model columns `models/profile.py:67-70`; sole reader `admin.py:120 elif profile.pending_profile:`; no writer anywhere in `src/` or `scripts/`. `templates/profile/view.html:42-48` carries a comment saying exactly this (banner removed because nothing writes it). Specs still describing the flow: `specs/profile-ingestion.md:186`, `specs/data-model.md:53-60`, `specs/auth-and-user-management.md:144` (plus `specs/tech-stack.md:35` mentions the column) — the issue's "three specs" undercounts by one if tech-stack is included. + +`Form("")` blanking — three routes, all unconditional: +``` +profile.py:113-118 research_summary/techniques/.../keywords: str = Form("") +profile.py:160-166 profile.research_summary = research_summary ... profile.profile_version = (…)+1 +onboarding.py:111-116 / :148-154 same shape +agent_page.py:1233-1238 / :1256-1262 same shape +``` +`profile.py:144-149`: `if name:` is a real guard; `if institution is not None:` and `if department is not None:` are dead (a `Form("")` default is never `None`), so `institution or None` blanks those too — the issue's point (user vs profile asymmetry) is right but the asymmetry is narrower than "user fields are guarded". Mitigation of severity: all three templates (`templates/profile/edit.html`, `templates/onboarding/profile_review.html`, `templates/agent/public_profile.html`) render every one of the six fields inside the single form, so a normal browser POST always carries all six; blanking requires a partial/crafted POST or a template regression. Test `tests/integration/test_onboarding_flow.py:837` posts all fields and asserts the version bump — it exercises, but does not pin, the unconditional overwrite. + +### PR V6 — COR-24 + +grep `os\.replace|tempfile|flock|NamedTemporaryFile|mkstemp` over `src/`: zero hits. All file writers are `Path.write_text` (truncate-then-write): `profile_export.py:118`, `:142`; `agent_page.py:1131`; `agent/agent.py:711`, `:738` (plus `grantbot.py:720`, `foa_cache.py:29`, out of scope). + +Ordering per writer: +- `profile.py`: `:168 commit` → `:185 export_profile_to_markdown` → `:192 create_revision` → `:200 commit`. DB→disk. +- `onboarding.py save_profile`: `:156 commit` → `:172 export` → `:179 create_revision` → `:188 commit`. DB→disk. +- `agent_page.py save_public_profile`: `:1264 commit` → `:1276 export` → `:1281 create_revision` → `:1290 commit`. DB→disk. +- `agent_page.py save_private_profile` (`:1117-1150`): `:1131 profile_path.write_text(content)` (no `encoding=`) BEFORE `:1133-1140` DB lookup; `:1138 if profile:` — with no `ResearcherProfile` row the write is disk-only and no error is raised; then `create_revision` regardless. Inversion + guard + encoding all as claimed. +- Pipeline: `profile_pipeline.py:466 flush` → `:482 export` → `:489 create_revision` → `:497 flush`; the transaction is committed by the caller at `worker/main.py:97` (after `job.status = "completed"`), and rolled back implicitly on exception (`:99-111`). Disk ahead of DB on any failure between `:482` and the commit. + +`create_revision` dedup: FIXED (commit `da405cb`), `profile_versioning.py:84-92` compares against `latest_revision` and returns it unchanged on byte-identical content; pinned by `tests/integration/test_cli.py:731,751`. `tests/unit/test_profile_versioning.py` predates it (model-only tests). + +### PR V6 — COR-23 + +`profile_pipeline.py:458-464` writes `profile.private_profile_seed` only; nothing in `src/` or `scripts/` exports a seed (grep `private_profile_seed`: pipeline, model, onboarding GET `:211` and POST `:269`, tests). `export_private_profile` (`profile_export.py:126-147`) returns `None` when `private_profile_md` is falsy (`:136-137`), so it could not export a seed even if called. Agent reads disk only: `agent.py:119-126` with default `"No private instructions yet."`. Mitigation the issue does not mention: `onboarding.py:211` shows `private_profile_md or private_profile_seed` to the PI, and the POST (`:268-285`) copies it into `private_profile_md` and exports — so the seed reaches disk once the PI completes onboarding step 4. Admin-seeded labs whose PI never logs in run without it, as the issue says. + +### PR C1 — RMW races + +All seven sites at the issue's exact lines: +``` +profile_pipeline.py:417 profile.profile_version = (profile.profile_version or 0) + 1 (row loaded :286-289; awaits at :310, :323, :461 between) +profile.py:166 profile.profile_version = (profile.profile_version or 0) + 1 +onboarding.py:154 profile.profile_version = (profile.profile_version or 0) + 1 +agent_page.py:1262 profile.profile_version = (profile.profile_version or 0) + 1 +agent_page.py:1423-1426 current_ids = list(agent.delegate_slack_ids or []) … agent.delegate_slack_ids = current_ids (after `await lookup_user_by_email_async` :1417) +agent_page.py:1609-1612 same, remove path (after await :1608) +invite.py:241-244 same, add path (after await :240) +``` +`with_for_update`: one hit, `worker/main.py:42` (job claim). `array_append|array_remove`: none. No `version_id_col` on `ResearcherProfile` (`models/profile.py`). No tests exercise concurrent saves or delegate accepts. + +## 3. Counts + +39 sub-claims assessed: **33 still present**, **3 fixed** (COR-22 gate, COR-22 `synthesis_validated` persist, `create_revision` dedup — all confirmed with commits), **1 partially** (COR-15 `fetch_orcid_profile`: `name: null` is guarded, five sibling containers are not), **0 changed**, **0 not reproducible**, **1 N/A** (unique-constraint migration test — no constraint exists to test), plus 1 "accurate observation" row (public writers are DB→disk) folded into still-present. + +Issue text inaccuracies (defect unaffected): `admin.py:92`→`:105-108`, `admin.py:105`→`:120`, `simulation.py:4565`→`:4679` (drift from `18ba52c`); `fetch_orcid_profile :29-39` overstates (name block guarded); "three specs" is four if `tech-stack.md` counts; `profile.py` user-field guards are mostly dead (`is not None` on `Form("")`), which strengthens rather than weakens the point; the issue omits that `pubmed._extract_text` already exists as an itertext-equivalent, and that two extra ORCID null sites raise (`external-id-type: null` → `.lower()`, `display-index: null` → `int()`). + +## 4. What I could not verify and why + +- Whether NCBI efetch returns two `` elements when the same PMID appears twice in one `id=` batch (no network). The cross-batch and concurrent-run duplicate paths do not depend on this. +- Whether ORCID's public API actually emits `"external-ids": null` / `"title": null` for real records (no network). The code's own `publication-date` guard at `orcid.py:123` and the issue's assertion are the only evidence; the parsers demonstrably raise if it does. +- Live behaviour of the `with_for_update`-free RMW under real concurrency (needs a database; not run per constraints). The code shape is unambiguous. +- `tests/live_api/test_orcid_live.py` shape checks (network-gated). diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_22_redteam.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_22_redteam.md new file mode 100644 index 00000000..2c4b247a --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_22_redteam.md @@ -0,0 +1,136 @@ +# Issue #22 — red-team pass (second reviewer) + +Tree: `copi-prod` @ 18ba52c, clean. All verdicts re-derived by my own grep/sed and by running the real modules with +`.venv-test/bin/python` (ORCID parsers driven through a fake `httpx.AsyncClient`; PubMed parser on inline XML; validator on dicts). +`git show da405cb^:…` used to read pre-fix `create_revision`. No DB/Docker/network. + +## 1. Table + +| id | first-agent verdict | red-team result | one-line reason | evidence | +|---|---|---|---|---| +| V1-15a | STILL PRESENT | UPHELD (re-run) | `works external-ids=None -> AttributeError 'NoneType' object has no attribute 'get'`; parse loop `orcid.py:111-137` is outside the try `:103-109` | run below | +| V1-15b | STILL PRESENT | UPHELD (re-run) | `title=None` and `title.title=None` both raise | run | +| V1-15c | STILL PRESENT | UPHELD (re-run) | grants `title=None` / `title.title=None` raise; `:92` | run | +| V1-15d | PARTIALLY | UPHELD (re-run) | `name=None -> OK` (guarded `:30-31`); `given-names/emails/researcher-urls=None` raise | run | +| V1-15e | STILL PRESENT | UPHELD (re-run) | `year.value=None -> TypeError`, `'n.d.' -> ValueError`; `publication-date=None -> OK` (guard `:123`) | run | +| V1-15f | STILL PRESENT | UPHELD | catch `profile_pipeline.py:107-112`; NEW sub-claim (legacy rows unprotected) confirmed: gate `:390 (profile.evidence_pub_count or 0) > 0`; `alembic/versions/0023…:21-22` "deliberately NOT backfilled"; `models/profile.py:44-46` | sed | +| V1-15g | STILL PRESENT | UPHELD | only `None` in `tests/contract/test_orcid_contract.py` is `:46 "end-date": None` (a leaf, not a container) | grep | +| V1-16a | STILL PRESENT | UPHELD | `models/publication.py` no `__table_args__`, `pmid` nullable `:22`; `0001_initial.py:121-122` non-unique; `0023` mentions `publications` only in its docstring `:23-24` | grep | +| V1-16b | STILL PRESENT | UPHELD | `profile_pipeline.py:115`, `:147` | sed | +| V1-16c | STILL PRESENT | UPHELD | `:182` built once; `:212-213` lookup; `:228-229` add without updating dict | sed | +| V1-16d | STILL PRESENT | UPHELD | `:278 scalar_one_or_none()` inside `except Exception` → `logger.debug` `:281-282` | sed | +| V1-16e | STILL PRESENT | UPHELD | `profile_export.py:79-84` sort+`[:20]`, per-row citation, no pmid/doi set | sed | +| V1-16f | STILL PRESENT | UPHELD | `admin.py:107 func.count(Publication.id)` | grep | +| V1-16g | STILL PRESENT | UPHELD (anchor only) | `simulation.py:4679 _load_publication_records`; body not re-read | grep | +| V1-pm1 | STILL PRESENT | UPHELD (re-run) | `title = 'Role of '` | run | +| V1-pm2 | STILL PRESENT | UPHELD (re-run) | `abstract = 'BACKGROUND: We studied Plain '` | run | +| V1-pm3 | STILL PRESENT (+ missed helper) | QUALIFIED | `_extract_text` (`pubmed.py:404-413`) exists and is unused for title/abstract, but it is NOT itertext-equivalent: it strips and space-joins, so `H2O` → `'H 2 O'` vs itertext `'H2O'`. Using it as the fix would corrupt formulas/gene symbols with sub/superscripts | run | +| V1-pm4 | STILL PRESENT | UPHELD | `fetch_abstract` returns `rec.get("title", "")` from the same parser (`pubmed.py:444-445`) | sed | +| V1-val1 | STILL PRESENT | UPHELD (re-run) | `summary=None -> AttributeError`; call `:317` bare (no try) | run/sed | +| V1-val2 | STILL PRESENT | UPHELD (re-run) | `techniques='PCR' -> True`; `:409 profile.techniques = synthesized.get(...)` | run | +| V1-val3 | STILL PRESENT | UPHELD (re-run) | 120 and 300 words → True; message `:565` says 150-250 | run | +| V6-22-gate | FIXED | QUALIFIED (residual sites) | pipeline gate `:383-406` confirmed (d311170) and tests 472/532/759 would fail pre-fix (532 asserts stored summary kept AND `profile_version == 1`); BUT four scripts re-synthesize and overwrite `research_summary` + bump `profile_version` with no `_validate_profile`, no gate: `scripts/resynth_from_current_pubs.py:76-82`, `regen_profile_from_cv.py:112-118`, `vet_publications.py:177-183`, `regen_profiles_from_web.py:107-113` | grep | +| V6-22-flag | FIXED | QUALIFIED (residual sites) | `:414` is the ONLY writer of `synthesis_validated`; the same four scripts write a new synthesis without touching `synthesis_validated` / `evidence_pmid_count` / `evidence_pub_count`, so after a script run the provenance columns describe the previous synthesis (stale, not just unset) | grep | +| V6-22a | STILL PRESENT | UPHELD | writers of `synthesis_validated`: `profile_pipeline.py:414` only | grep | +| V6-22b | STILL PRESENT | UPHELD | `:384-385 (profile.profile_version or 0) > 0` | sed | +| V6-22c | STILL PRESENT | UPHELD | `:372` precedes `if synthesized:` `:383` | sed | +| V6-pend | STILL PRESENT | UPHELD | no `pending_profile =` writer in src/ or scripts/; reader `admin.py:120` | grep | +| V6-form1 | STILL PRESENT | UPHELD | `agent_page.py:1233-1238` six `Form("")`; `:1262` | grep | +| V6-form2 | STILL PRESENT | UPHELD | `profile.py:113-118`, `:160-166` | sed | +| V6-form3 | STILL PRESENT | UPHELD | `onboarding.py:111-116` six `Form("")`; `:154` | grep | +| V6-form4 | STILL PRESENT (and worse) | UPHELD | `profile.py:144-149`: `if name:` real; `if institution is not None` / `if department is not None` always true under `Form("")` → `institution or None` blanks user fields too | sed | +| V6-24a | STILL PRESENT | UPHELD | 0 hits for `os.replace|tempfile|flock|NamedTemporaryFile|mkstemp` in src/ | grep | +| V6-24b | STILL PRESENT | UPHELD | `write_text` at `profile_export.py:118,:142`, `agent_page.py:1131`, `agent.py:711,:738` (+ grantbot/foa_cache out of scope) | grep | +| V6-24c | STILL PRESENT (accurate) | UPHELD | `profile.py:168→185→192→200`; `onboarding.py:156→172→179→188`; `agent_page.py:1264→1274→1281→1289` | grep | +| V6-24d | STILL PRESENT | UPHELD | `agent_page.py:1131 write_text(content)` (no encoding) → `:1138 if profile:` → `:1140 commit` → `:1144 create_revision` | grep | +| V6-24e | STILL PRESENT | UPHELD | `profile_pipeline.py:466 flush → :482 export → :489 create_revision → :497 flush`; commit `worker/main.py:97` | grep | +| V6-24f | FIXED | QUALIFIED (test does not pin it) | fix confirmed `profile_versioning.py:84-92` (da405cb; pre-fix had no `latest_revision` compare); sole `ProfileRevision(` constructor is `:94` so no bypass. BUT the cited test (`test_cli.py:731`) drives `backfill-profile-revisions`, and `cli.py:273-282` performs its OWN `latest_revision`/content compare before calling `create_revision` — the test passes even if `create_revision`'s dedup is removed. `tests/unit/test_profile_versioning.py` has no dedup case (0 hits for unchanged/identical/previous) | sed | +| V6-23 | STILL PRESENT | UPHELD | `profile_pipeline.py:458-462` seed only; `agent.py:119-126` disk-only; `profile_export.py:136-137` returns None on empty md; NEW mitigation confirmed `onboarding.py:211 private_profile_md or private_profile_seed`, POST `:269` clears seed, `:285` exports | sed | +| C1-a | STILL PRESENT | UPHELD | 4 sites exact: `profile_pipeline.py:417`, `onboarding.py:154`, `agent_page.py:1262`, `profile.py:166` | grep | +| C1-b | STILL PRESENT | UPHELD | `agent_page.py:1423-1426` (await `:1416`), `:1609-1612` (await `:1608`), `invite.py:241-244` (await `:239`); `:347` is a read | sed | +| C1-c | STILL PRESENT | UPHELD | `with_for_update` only `worker/main.py:42`; `array_append|array_remove|version_id_col` 0 hits | grep | +| DoD | N/A | UPHELD | no constraint exists | | + +## 2. Detail — QUALIFIED rows and NEW claims + +### V6-22-gate / V6-22-flag — FIXED in the pipeline, residual in four scripts +`profile_pipeline.py:383-418` gates the overwrite (`stored_is_worth_keeping`, `lost_evidence`) and writes `synthesis_validated` / +`evidence_*` alongside the synthesized fields. The three cited characterization tests would fail on the pre-d311170 code: `:472` +asserts `synthesis_validated is False` (column did not exist), `:532` asserts the stored `_VALID_PROFILE` summary survives and +`profile_version == 1` (pre-fix overwrote and bumped), `:759` asserts `evidence_pub_count == 2 and profile_version == 1`. +Residual the first agent missed: `grep -ln synthesize_profile scripts/` → `resynth_from_current_pubs.py`, `regen_profile_from_cv.py`, +`vet_publications.py`, `regen_profiles_from_web.py`. Each does `profile.research_summary = synthesized.get(...)` and +`profile.profile_version = (profile.profile_version or 0) + 1` with zero hits for `_validate_profile`, `synthesis_validated`, +`evidence_pub_count`, `evidence_pmid_count`. Consequences: (1) an unvalidated script synthesis is stored unconditionally (the COR-22 +defect, one layer out); (2) the row's provenance columns now describe a synthesis that is no longer stored — and `stored_is_worth_keeping` +on the next pipeline run reads that stale `synthesis_validated`. The first agent's residual (a) (web routes don't reset the flag) is the +same class; the scripts are worse because they write a *new synthesis*. + +### V6-24f — fix real, cited test does not isolate it +`profile_versioning.py:84-92` compares `previous.content == content` and returns `previous`. Pre-fix (`git show da405cb^`) has no such +compare. But `src/cli.py:273-282`: +``` +previous = await latest_revision(...) +if previous is not None and previous.content == content: + ... "Unchanged {profile_type} profile for {agent_id}" ... +await create_revision(...) +``` +so `test_backfill_run_twice_does_not_duplicate_any_revision` (`test_cli.py:731`) is satisfied by the CLI-side check alone. A regression +that drops the compare inside `create_revision` (which every web/pipeline caller relies on) would not be caught. Verdict FIXED stands; +"pinned by test_cli.py:731" does not. + +### V1-pm3 — `_extract_text` is not an itertext drop-in +``` +title = 'Role of ' (current .text) +itertext title = 'Role of TP53 in cancer' +_extract_text title = 'Role of TP53 in cancer' +itertext abstract = 'We studied TP53 in H2O.' +_extract_text abs = 'We studied TP53 in H 2 O.' +``` +`_extract_text` (`pubmed.py:404-413`) does `" ".join(p.strip() for p in parts if p.strip())` — fine for `` paragraphs, wrong for +inline markup. The fix should be `"".join(el.itertext())`, not a call to the existing helper. + +### NEW claims re-verified +- **lost_evidence protects only `evidence_pub_count>0` rows** — UPHELD. `0023_profile_synthesis_provenance.py:21-24`: "All three are + nullable and are deliberately NOT backfilled … count(publications) would look like a free win and would be a lie". Gate at `:390` + uses `(profile.evidence_pub_count or 0) > 0`, so every pre-0023 grounded profile is unprotected against a zero-evidence refresh. +- **Templates always post all six fields** — UPHELD with precision: in all three templates the six inputs sit inside a profile-exists + block (`profile/edit.html:50-121 {% if profile %}`, `agent/public_profile.html:34-109 {% if profile %}`, + `onboarding/profile_review.html:60-185 {% elif profile %}`); no per-field conditional. So whenever there is a row to blank, a browser + POST carries all six; blanking needs a crafted/partial POST. +- **`onboarding.py:211` surfaces the seed** — UPHELD (`content = profile.private_profile_md or profile.private_profile_seed or ""`; + POST `:268-285` copies into `private_profile_md`, nulls the seed, exports). +- **`profile.py` `is not None` guards dead** — UPHELD (`:108-119` all `Form("")`; `:146-149`). +- **`backfill_publications.py` no input dedup** — UPHELD: `wanted` (`:68`) and `missing` (`:69-74`) keep duplicates; `records` is keyed + by pmid but the insert loop iterates `missing` (`:77-97`) → two `db.add` for a twice-listed PMID. +- **Extra ORCID null sites** — `external-id-type=None -> AttributeError … 'lower'` reproduced; `display-index` not re-run. + +### Reproductions (all `.venv-test/bin/python`, real modules) +``` +works fully populated OK +works external-ids=None RAISED AttributeError: 'NoneType' object has no attribute 'get' +works title=None RAISED AttributeError +works title.title=None RAISED AttributeError +works year.value=None RAISED TypeError: int() argument must be ... not 'NoneType' +works year.value='n.d.' RAISED ValueError: invalid literal for int() with base 10: 'n.d.' +works publication-date=None OK +works external-id-type=None RAISED AttributeError: 'NoneType' object has no attribute 'lower' +grants title=None / title.title=None RAISED AttributeError +profile name=None OK (name falls back to the ORCID id) +profile given-names/emails/researcher-urls=None RAISED AttributeError +summary=None -> AttributeError; techniques='PCR' -> True; techniques=None -> TypeError; 120 words -> True; 300 words -> True +``` + +## 3. Mis-cites / wording in the first-agent report +- V1-16d cited `:272-282` as the block; the `scalar_one_or_none()` is at `:278`, `except`/`debug` at `:280-282` — block range fine. +- V6-24f "pinned by `tests/integration/test_cli.py:731,751`": the CLI does its own compare (`cli.py:273-282`); not a pin of + `create_revision`. +- V1-pm3 "recursive text collector … the fix is one call away": `_extract_text` inserts spaces at element boundaries (shown above). +- V6-22 residual list omits the four `scripts/` synthesizers. +- C1-b await anchors: add path `:1416` (report says `:1417`), invite `:239` (report says `:240`) — off by one, same statements. +- All other cites checked (orcid.py, pubmed.py, profile_pipeline.py, profile.py, onboarding.py, agent_page.py, invite.py, + profile_export.py, admin.py:107/120, models/publication.py, 0001/0023 migrations, agent.py:119-126) point at the quoted code. + +## 4. Counts +40 rows: **36 upheld**, **0 overturned**, **4 qualified** (V6-22-gate, V6-22-flag — residual script writers; V6-24f — cited test does +not isolate the fix; V1-pm3 — helper is not itertext-equivalent), **0 unverifiable** (V1-16g upheld on anchor only, body not re-read). diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_23.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_23.md new file mode 100644 index 00000000..a7bd0263 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_23.md @@ -0,0 +1,328 @@ +# Issue #23 verification — External clients: Slack SDK, GrantBot, FOA regexes, HTTP robustness + +Verified against `/home/a/scripps/coPI.science` @ `copi-prod` HEAD `18ba52c` (clean tree), 2026-09-02. +Method: symbol-located code, mechanical reproduction with `.venv-test/bin/python` against the real modules, +`git blame`/`git log -S` for provenance, and a run of the DB-free unit tests for the touched modules. + +Fix commits cited below are all confirmed ancestors of HEAD (`git merge-base --is-ancestor`): 9dbc9e0, d311170, +02143de, fa143a6, ae123447, a1f9f92f, 18ba52c. The issue's b1d54da is also an ancestor. + +## 1. Summary table + +| id | claim (one line) | verdict | key evidence | conf | +|---|---|---|---|---| +| V7a | `_call_with_retry` raised `UnboundLocalError` on retry exhaustion — "fixed in stack" | FIXED (9dbc9e0) | `src/agent/slack_client.py:324-342` (`last_exc`); pinned `tests/unit/test_slack_client_contract.py:146-169`; passes | high | +| V7b | `poll_channel_messages` unpaginated — "fixed in stack" | FIXED (d311170) | `slack_client.py:344-398` `_paginate`, `:505-550`, `MAX_PAGES=200` `:133`; incomplete → `[]` `:539-545`; sim cursor per-message `simulation.py:2777` | high | +| V7c | `list_channels` single page — "fixed in stack" | FIXED (d311170) | `slack_client.py:1006-1059`; re-raises `SlackListingIncomplete` `:1043-1053`; tests `:441-579` | high | +| V7d | `resolve_user_name` reads top-level `display_name` (dead branch) | STILL PRESENT | `slack_client.py:662`, unchanged since 1812fa9 (2026-03-20); sibling consumer `agent_page.py:1378` reads `real_name`/`name` only; no unit test; live test passes via fallback | med (shape not network-verified) | +| V7e | `int(Retry-After)` unguarded: HTTP-date → `ValueError` escapes; unclamped | STILL PRESENT — reproduced | `slack_client.py:331,336`; my probe: HTTP-date → `ValueError` escapes `_call_with_retry`; `99999999` → `time.sleep(99999999)` x3; tests feed only `str(int)` (`tests/fakes.py:243-254`) | high | +| C26a | Selection-failure fallback posts unvetted FOAs | STILL PRESENT | `grantbot.py:344-346` returns `list(opportunities.keys())[:max_select]`; downstream caps volume only (`:590-600`) | high | +| C26b | Dict element in LLM JSON → `TypeError: unhashable` at `num in all_opps`, uncaught | STILL PRESENT — reproduced | `grantbot.py:538`; no type-check at `:341-343`; `run_grantbot` `:476-486` has only `finally: dispose` | high | +| C26b' | …so `_mark_run_complete` is skipped and the scheduler re-fires every `check_interval` | STILL PRESENT | `grantbot.py:771-780`: `_mark_run_complete()` only after success; `except Exception` logs → `_should_run_today()` stays True; compose runs `scheduler` (`docker-compose.prod.yml:116`) | high | +| C26b'' | "…and `main()` eventually dies" | NOT REPRODUCIBLE | scheduler's `except Exception` (`:779-780`) swallows and loops; only the one-shot `main` (`:723-745`) has no try, and prod does not run it. Also each re-fire re-queries the LLM, so a later attempt can succeed — it is a retry-until-parse loop, not a hard crash loop | high | +| C26c | GrantBot falls back to SuBot's token → posts authored as SuBot | STILL PRESENT (log level raised in 18ba52c; no code fix) | `grantbot.py:615-625`; engine explicitly refuses to map su's uid to grantbot (`simulation.py:3967-3971`, `_bot_uid_map` `:4004-4010`) | high | +| C26d | `_load_researcher_profiles`/`_extract_list_section`/`_build_search_queries` are 84 dead lines | STILL PRESENT | `grantbot.py:49-131` (83 code lines; 84 with trailing blank); only refs are helper→helper `:68-71`; no callers in src/scripts/tests; `PROFILES_DIR` `:44` is dead with them | high | +| C26e | "Already fixed earlier: two-phase `_claim_foa`/`_release_foa`" | FIXED (confirmed) | `grantbot.py:246-275`; used `:658,:673,:681,:691` | high | +| C27a | 6-row regex divergence table | STILL PRESENT — all 6 rows reproduce exactly | run output below; `foa_cache.py:19-21`, `funding_rules.py:95` | high | +| C27b | Neither regex is `IGNORECASE` | STILL PRESENT | `flags & IGNORECASE == False` for both; all lowercase variants fail both | high | +| C27c | `foa_cache.py:18` docstring falsified on 2 of its 3 examples; `extract_foa_number` → `None` for PA/PAR/PAS | STILL PRESENT | `extract_foa_number("...PAR-24-293...")` → `None`; `DE-FOA-0003456` → `None`; `RFA-AI-27-019` → ok. Callers `simulation.py:1119,1217,1258` | high | +| C28a | Apostrophe classes ASCII-only; U+2019 form passes the announcement filter | STILL PRESENT — reproduced | `funding_rules.py:23` class is `['']` = U+0027 twice (codepoints verified); `:25` `'?`; `"I'll spin up…"`→True, `"I’ll spin up…"`→False | high | +| C28b | Ack-only detector false-rejects substantive short replies | STILL PRESENT — reproduced | `is_acknowledgment_only_funding_reply("Agreed, we can send the plasmids and the mice next week.")` → `True` | high | +| C28b' | `funding_reject_count` resets only on successful post; one-strike mode after 2 rejections | STILL PRESENT | reset only at `simulation.py:1506`; increment `:1457`; back-off `:1463-1466`. Issue's `:1465` drifted → `:1506`; "three sites" re-arm count is stale (≥10 sites) | high | +| C28c | `_TAG_RE` lacks `IGNORECASE` (`@grantbot` ok, `@GRANTBOT`/`@SuBOT` miss) | STILL PRESENT — reproduced | `funding_rules.py:154`; results below | high | +| C28d | Cross-issue: mirrors issue #20 E5's `_extract_tagged_agent` fix | N/A (cross-ref) — but note: `message_log.py:407` is ALSO still case-sensitive; the two are "in sync" only in that neither is fixed. Same literal also at `simulation.py:2523,2558` | high | +| C29a | No retry/backoff in `orcid.py`/`pubmed.py`/`grants.py` | STILL PRESENT | `grep retry\|backoff\|tenacity` → nothing; pyproject has no tenacity; every call is `httpx` + `raise_for_status()` | high | +| C29b | `raise_for_status()` before the pacing sleep | STILL PRESENT | `pubmed.py:93-95`; `tests/contract/test_pubmed_contract.py:5` documents "sleeps per *successful* call" | high | +| C29c | `Semaphore(8)` never sized to key/no-key; `api_key` check does not feed back | STILL PRESENT (impact qualified) | `pubmed.py:73`, `:87-88`; all pipeline callers are sequential `await`s — the only real concurrency is Phase 4 `asyncio.gather` (`simulation.py:1328-1332`); semaphore is process-local across app/worker/agent | high | +| C29d | PR #32 edited `_ncbi_get` (tool/email) without touching rate math | confirmed (context) | `pubmed.py:76-90` | high | +| C30 | Budget debited before the awaited fetch; blanket `except` → no refund | STILL PRESENT | `tools.py:126-129`, `:135-138`, `:146-148`; no `-=` anywhere in src/. Sharper: a *non-raising* miss (`_execute_retrieve_abstract` `:205-207` returns `result["error"]`) also consumes budget | high | +| V10a | Invite Slack sync imports fixed (`token_for_agent_row` + `lookup_user_by_email_async`), logs instead of `pass` | FIXED (02143de; async wrapper via fa143a6) | `src/routers/invite.py:232-252`; `slack_tokens.py:46` | high | +| V10b | Residual nit: a `None` token skips the block with no log line | STILL PRESENT (nit) | `invite.py:237-238` `if bot_token:` with no else | high | + +Counts: **18 still present, 5 fixed, 0 partial, 0 changed, 1 not reproducible, 2 N/A/context.** + +Tests run (host, no DB): `tests/unit/test_slack_client_contract.py tests/unit/test_funding_rules.py +tests/unit/test_grantbot_lead_time.py tests/unit/test_slack_tokens.py tests/unit/test_service_bot_attribution.py +tests/unit/test_slack_web.py` → **170 passed in 8.16s**. None of these asserts the fixed behaviour for any +STILL PRESENT item above (details per item). + +--- + +## 2. Per-item detail + +### V7 — Slack client (`src/agent/slack_client.py`) + +**V7a UnboundLocalError — FIXED.** Current code: + +``` +324 last_exc: SlackApiError | None = None +325 for attempt in range(MAX_RETRIES): +326 try: +327 return method(**kwargs) +328 except SlackApiError as exc: +329 if exc.response.get("error") == "ratelimited": +330 last_exc = exc +331 retry_after = int(exc.response.headers.get("Retry-After", 5)) +... +336 time.sleep(retry_after) +337 else: +338 raise +339 raise SlackApiError( +340 "Rate limit retries exhausted", +341 response=last_exc.response if last_exc else None, +342 ) +``` +`git log -S"last_exc"` → 9dbc9e0 ("Slack T2: client wire contract…"). Regression test +`test_retries_are_bounded_and_raise_a_SlackApiError` at `tests/unit/test_slack_client_contract.py:146-169` — the +issue's line range is exact. Passes. + +**V7b / V7c pagination — FIXED.** `_paginate` (`:344-398`) walks `response_metadata.next_cursor`, bounded by +`MAX_PAGES = 200` (`:133`), detects repeated cursors, raises `SlackListingIncomplete` with the partial. `git log -S"def _paginate"` +→ d311170. `poll_channel_messages` (`:505-550`) returns `[]` on `SlackListingIncomplete` (`:539-545`); the sim advances +`_poll_cursors[ch_id] = ts` per processed message (`simulation.py:2776-2777`), so the issue's "closed at both ends" holds. +`list_channels` (`:1006-1059`) caches the partial then re-raises (`:1043-1053`). Tests: `:409` (every cursor read goes +through `_paginate`), `:441-579` (list_channels), `:599-659` (poll). All pass. Issue line refs `:344-398`, `:505-545`, +`:1006-1059` are still exact — this file region has not drifted since b1d54da. + +**V7d `resolve_user_name` — STILL PRESENT.** +``` +655 def resolve_user_name(self, user_id: str) -> str: +... +660 info = self._api("users_info", user=user_id) +661 user = info.get("user", {}) +662 return user.get("display_name") or user.get("real_name") or user_id +``` +`git blame` → 1812fa9 (2026-03-20), never touched. Slack's `users.info` puts `display_name` under `user.profile` +(top-level has `name`, `real_name`); I could not call Slack to re-confirm (constraint), but the codebase's other +consumer of the same object — `src/routers/agent_page.py:1378` `info.get("real_name") or info.get("name") or sid` — +reads exactly the top-level keys that exist and does not attempt `display_name`. Callers are live paths +(`simulation.py:2781` channel poll of human posts, `:3222` thread PI replies). No unit test covers +`resolve_user_name`; the live test `tests/integration/test_slack_client_live.py:47-51` only asserts the result is not the raw +id, which the `real_name` fallback satisfies — it would not catch the dead branch. + +**V7e Retry-After — STILL PRESENT; reproduced.** Probe (`AgentSlackClient.__new__`, `time.sleep` mocked, fake +`SlackResponse` with `error=ratelimited`): +``` +HTTP-date -> ESCAPES as ValueError : invalid literal for int() with base 10: 'Wed, 21 Oct 2015 07:28:00 GMT' +huge int -> sleep calls: [99999999, 99999999, 99999999] +negative -> sleep calls: [-5, -5, -5] (real time.sleep(-5) raises ValueError too) +``` +So a non-integer header raises `ValueError` out of the `except SlackApiError` block; `post_message`'s +`except SlackApiError` never sees it — the same type-substitution class V7a fixed. No clamp. Tests only ever build +`{"Retry-After": str(retry_after)}` with `retry_after: int | None` (`tests/fakes.py:243-256`); +`test_retry_after_header_is_honoured` (`:172-182`) feeds `17`. Mitigation that exists *elsewhere*: the sibling +`src/services/slack_web.py:_call` (`:104-120`) already does `float()` in `try/except (TypeError, ValueError)` and caps at +`_MAX_RETRY_AFTER = 30.0` (`:49`), with tests `tests/unit/test_slack_web.py:191-218`. That is the pattern to port; the +`slack_client.py` path (the agent engine) does not have it. + +### V8 — GrantBot (`src/agent/grantbot.py`), FOA regexes, funding detection + +**COR-26a fallback — STILL PRESENT.** +``` +341 selected = json.loads(cleaned) +342 logger.info("Selected %d of %d opportunities", len(selected), len(opportunities)) +343 return selected[:max_select] +344 except Exception as exc: +345 logger.warning("Selection failed: %s — falling back to all", exc) +346 return list(opportunities.keys())[:max_select] +``` +Blame a1f9f92f (2026-03-28). Downstream `:590-600` caps by `max_per_channel` (prod: 1) and `max_posts` (10) — a +volume cap, not a relevance filter, so up to 10 arbitrary FOAs (first 30 keys in Grants.gov order, one per LLM-chosen +channel) get drafted and posted. Extra observation: a non-list JSON (`{}`, `123`, `null`) fails at `selected[:max_select]` +*inside* the try and lands in the same fallback; a bare JSON string (`"PAR-24-293"`) passes the slice and is returned +as a `str`, which `:538` then iterates character by character → 0 selected, silently. + +**COR-26b TypeError — STILL PRESENT; reproduced.** +``` +537 selected_nums = await _select_opportunities(all_opps) +538 selected_opps = {num: all_opps[num] for num in selected_nums if num in all_opps} +``` +``` +selected=['PAR-24-293', {'number': 'RFA-AI-27-019'}] -> TypeError: unhashable type: 'dict' +selected=['PAR-24-293', ['RFA-AI-27-019']] -> TypeError: unhashable type: 'list' +selected=['PAR-24-293', 42, None] -> OK ['PAR-24-293'] +``` +No `isinstance` check on the parsed list anywhere in `_select_opportunities` (`:341-343`); no try around `:537-538` in +`_run_grantbot_with_session`; `run_grantbot` (`:476-486`) is `try/finally: engine.dispose()` only. + +**COR-26b' re-fire — STILL PRESENT; COR-26b'' "main() dies" — NOT REPRODUCIBLE.** +``` +767 while True: +769 if _should_run_today() and now.hour >= run_hour: +771 try: +772 results = asyncio.run(run_grantbot(...)) +777 _mark_run_complete() +779 except Exception as exc: +780 logger.error("Daily run failed: %s", exc, exc_info=True) +786 time.sleep(check_interval) +``` +`_mark_run_complete()` runs only on success, so a failed run leaves `_should_run_today()` True and the scheduler +retries every `check_interval` (default 900 s; `docker-compose.prod.yml:116` passes no override). But the scheduler +does **not** die — `except Exception` swallows it. The one-shot `main()` (`:723-745`) has no try and would exit non-zero, +but prod runs `scheduler`. Also: each re-fire re-queries Grants.gov and the LLM, so a later attempt can parse cleanly; +the failure mode is an unbounded retry-until-parse loop (cost + log noise), not a permanent crash loop. + +**COR-26c SuBot fallback — STILL PRESENT (log raised to WARNING in 18ba52c).** +``` +615 candidate = getattr(settings, "slack_bot_token_grantbot", "") +616 if not candidate or candidate.startswith("xoxb-placeholder"): +617 candidate = settings.slack_bot_token_su +622 logger.warning( +623 "No grantbot Slack token — using SuBot's token as fallback; " +624 "these posts will be attributed to su, not grantbot", +``` +Blame ae123447 (2026-08-04) for the lines, 18ba52c for the WARNING. The engine side now deliberately does *not* map +su's uid to grantbot (`simulation.py:3967-3971`; `_bot_uid_map` `:4004-4010` "roster clients first"), so fallback posts +attribute to `su`, as the issue says. `tests/integration/test_grantbot_live.py:270-283` (`_SettingsWithFakeToken`) +returns the same fake for both token names — no test distinguishes the fallback. + +**COR-26d dead helpers — STILL PRESENT.** `_load_researcher_profiles` `:49-76`, `_extract_list_section` `:79-96`, +`_build_search_queries` `:99-131`. `grep -rn` over src/ scripts/ tests/ docs/ specs/: the only references are the +in-file helper→helper calls at `:68-71` and a docs inventory line (`docs/superpowers/plans/2026-08-12-branch2-inventories/branch2-inventory-funding.md:23`, +"no external importers"). 83 code lines (`:49-131`); the issue's `:49-132` = 84 counts one blank. `PROFILES_DIR` (`:44`) +has no other user either. + +**COR-26e** two-phase claim/release — present at `:246-275`, used at `:658/:673/:681/:691`. Matches the issue's +"already fixed". + +**COR-27 regex table — STILL PRESENT; all 6 rows reproduce.** Run against the real modules: +``` +FOA_PATTERN = \b((?:RFA|PAR|PA|NOT|OTA|RFI|DE-FOA)-[A-Z]{2,4}-\d{2,4}-\d{2,5})\b | IGNORECASE: False +_FOA_NUMBER_RE = \b(PA[RS]?-\d{2}-\d{3,4}|RFA-[A-Z]{2,3}-\d{2}-\d{3,4})\b | IGNORECASE: False + +input foa_cache.FOA_PATTERN funding_rules._FOA_NUMBER_RE extract_foa_number(sentence) +PAR-24-293 False True None +par-24-293 False False None +PA-24-293 False True None +pa-24-293 False False None +PAS-24-293 False True None +pas-24-293 False False None +NOT-OD-24-001 True False 'NOT-OD-24-001' +not-od-24-001 False False None +DE-FOA-0003456 False False None +de-foa-0003456 False False None +RFA-AI-27-019 True True 'RFA-AI-27-019' +rfa-ai-27-019 False False None +``` +Blame: `foa_cache.py:20` → 12dbefe (2026-04-03); `funding_rules.py:95` → 03af17e (2026-04-13); neither touched since. +Docstring `foa_cache.py:18` lists `RFA-AI-27-019, PAR-24-293, DE-FOA-0003456`; 2 of 3 fail its own regex. +Consequence path: `simulation.py:1119,1217,1258` call `extract_foa_number` on GrantBot roots; for every PA/PAR/PAS +post `thread.foa_number` stays `None` and `format_foa_for_prompt` never resolves. Tests: no unit test imports +`FOA_PATTERN`/`extract_foa_number`; `tests/unit/test_funding_rules.py` only ever uses `PAR-25-297`, which +`_FOA_NUMBER_RE` matches — so the suite is green on the divergent pair. Additional gap the issue does not mention: +GrantBot searches `BIOMEDICAL_AGENCIES = ["HHS-NIH11", "NSF"]` (`grants.py:14`); NSF numbers (e.g. `25-543`) match +*neither* regex. (`data/foa_cache/` does not exist in this checkout, so I could not measure the real posted distribution.) + +**COR-28a apostrophes — STILL PRESENT; reproduced.** `funding_rules.py:23` raw pattern +`"\\bi['']?ll (start|…)\\b"`; the class contains codepoints `['0x27', '0x27']` — ASCII twice, no U+2019. `:25` is `i'?m`. +``` +"I'll spin up a dedicated thread." announcement_only=True +'I’ll spin up a dedicated thread.' announcement_only=False +"I'm going to start a new thread." announcement_only=True +'I’m going to start a new thread.' announcement_only=False +``` +`TestAnnouncementOnly` (`tests/unit/test_funding_rules.py:34-70`) uses ASCII apostrophes only. + +**COR-28b ack detector — STILL PRESENT; reproduced.** +``` +'Agreed, we can send the plasmids and the mice next week.' ack_only=True (REJECTED) +'Agreed, … next week — aim 2 fits.' ack_only=False +``` +`_SUBSTANTIVE_MARKERS_RE` (`:43-49`) has `mouse model` but not `mice`/`plasmid`/`send`. Reset semantics: +`funding_reject_count` is incremented at `simulation.py:1457`, triggers `has_pending_reply = False` at `:1463-1466` +when `>= 2`, and is reset **only** at `:1506` inside the successful-post branch (the issue's `:1465` has drifted to +`:1506`). `has_pending_reply` is re-armed at ≥10 sites (`:1223, :1264, :1315, :2827, :2932, :2987, :2999, :4242, +:5129, :5141`), not "three" — the count in the issue is stale but the mechanism (one-strike mode after two rejections +until a post succeeds) is exactly as described. + +**COR-28c `_TAG_RE` — STILL PRESENT; reproduced.** `funding_rules.py:154` `@(\w+[Bb]ot)\b`, no IGNORECASE: +`@grantbot`→`grantbot`, `@GRANTBOT`→None, `@SuBOT`→None, `@SuBot`→ok, `@Subot`→ok. +Cross-issue note: `src/agent/message_log.py:407` `_extract_tagged_agent` uses the identical case-sensitive literal, as do +`simulation.py:2523` and `:2558` and the live test `test_grantbot_live.py:579`. Whatever issue #20 E5 proposed has not +landed at HEAD; there are four production copies of this regex, none IGNORECASE. + +### V9 — HTTP robustness (`src/services/orcid.py`, `pubmed.py`, `grants.py`) + tool charging (`src/agent/tools.py`) + +**COR-29a — STILL PRESENT.** `grep -rn "retry\|backoff\|tenacity"` across the three files → no matches (exit 1); +`pyproject.toml` has `httpx>=0.27.0` and no tenacity. Every request is `client.get/post(...)` + `resp.raise_for_status()`: +`orcid.py:18-19, :82-83, :104-105`; `grants.py:40-41, :94-95, :130-131`; `pubmed.py:93-94`. Callers only ever +`except Exception: log; continue` (`pubmed.py:114, :140, :290, :309, :337, :358`; `orcid.py:85, :107`; `grants.py:212`). + +**COR-29b — STILL PRESENT.** +``` +91 async with _request_semaphore: +92 async with httpx.AsyncClient(timeout=60, follow_redirects=True) as client: +93 resp = await client.get(url, params=params) +94 resp.raise_for_status() +95 await asyncio.sleep(0.12) # ~8 req/s +96 return resp +``` +Blame 5972103 (2026-03-20). On a 429/5xx the sleep is skipped and the caller's `except` moves straight to the next +request. `tests/contract/test_pubmed_contract.py:5` says "_ncbi_get sleeps ~0.12s per *successful* call" — the test +suite documents the defect rather than asserting against it. + +**COR-29c Semaphore sizing — STILL PRESENT, impact qualified.** `pubmed.py:73` `asyncio.Semaphore(8)`; `:87-88` +`if settings.ncbi_api_key: params["api_key"] = …` — nothing sizes the semaphore. In practice: `profile_pipeline.py` +(`:81-268`), `cli.py:48`, `auth.py:189`, `scripts/generate_sparsedata_user.py` all `await` sequentially; the only +concurrency that can reach `_ncbi_get` is Phase 4 `asyncio.gather` over an agent's threads (`simulation.py:1328-1332`) +→ `execute_tool` → `fetch_abstract`. So "tens of req/s" is the worst case under parallel tool calls, not the steady +state. Not mentioned by the issue: the semaphore is process-local — app, worker and agent containers each get their +own 8, and NCBI's limit is per key/IP. PR #32's `tool`/`email` edit (`:76-90`) confirmed; rate math untouched. + +**COR-30 — STILL PRESENT.** +``` +126 if thread_state.abstracts_other >= settings.max_abstracts_other_per_thread: +127 return "Rate limit: …" +128 thread_state.abstracts_other += 1 +129 return await _execute_retrieve_abstract(tool_input["pmid_or_doi"]) +... +137 thread_state.full_text += 1 +138 return await _execute_retrieve_full_text(tool_input["pmid_or_doi"]) +... +146 except Exception as exc: +147 logger.error("Tool execution failed: %s(%s) — %s", tool_name, tool_input, exc) +148 return f"Error executing {tool_name}: {exc}" +``` +`grep -rn "abstracts_other -=\|full_text -="` → nothing. Sharper than filed: `fetch_abstract` returns +`{"error": …}` without raising for "could not resolve DOI"/"no record" (`pubmed.py:434, :439`) and +`_execute_retrieve_abstract` (`tools.py:205-207`) returns that string — budget is consumed on those non-exception +misses too. Also the comment at `:122-123` ("We don't enforce limits on own-lab lookups") does not match the code, +which increments for every `retrieve_abstract` regardless of whose paper it is. No test in `tests/unit/` touches +`abstracts_other`. + +### V10 — invite import (`src/routers/invite.py`) + +**FIXED.** `git log -S"token_for_agent_row" -- src/routers/invite.py` → 02143de replaced +`from src.routers.agent_page import _get_bot_token` + raw `WebClient` + bare `except: pass` with +`token_for_agent_row(agent)` + `lookup_user_by_email` + `logger.warning`; fa143a6 switched to +`lookup_user_by_email_async`. Current `:232-252` matches the issue's description and line range exactly. +`token_for_agent_row` is at `src/services/slack_tokens.py:46`. **Residual nit confirmed:** `:237-238` +`bot_token = token_for_agent_row(agent); if bot_token:` — a `None` token (no DB token, no `.env` token) skips the +whole lookup silently; the only log is on exception. + +--- + +## 3. Counts + +**18 still present, 5 fixed, 0 partially fixed, 0 changed, 1 not reproducible** (+2 cross-reference/context rows). + +Where the issue text is stale or wrong (separate from the defects): +- "…and `main()` eventually dies" (COR-26b): the prod `scheduler` catches `Exception` and keeps looping; nothing dies. +- `simulation.py:1465` → now `:1506`; "`has_pending_reply` re-arms at three sites" → ≥10 sites. +- `_load_researcher_profiles…:49-132` "84 dead lines" → 83 code lines `:49-131` (84 with the blank). +- All other line references (`slack_client.py`, `grantbot.py`, `funding_rules.py`, `pubmed.py`, `tools.py`, `invite.py`) + are still exact at HEAD — none of these regions moved after b1d54da. +- The "cross-issue #20 E5" mirror: nothing case-insensitive has landed in `_extract_tagged_agent` either. + +## 4. What I could not verify and why + +- **Slack `users.info` response shape** (V7d): constraint forbids network calls. Verdict rests on the code being + unchanged since 2026-03-20, on the sibling consumer `agent_page.py:1378` reading only `real_name`/`name`, and on + the documented API shape (`user.profile.display_name`). Confidence medium. +- **Real posted-FOA number distribution** (COR-27 impact): `data/foa_cache/` and `data/grantbot_last_run.txt` are not + present in this checkout (prod-only volume), so I could not count how many real posts hit the PA/PAR gap. +- **Live NCBI rate behaviour** (COR-29): not exercised; the concurrency analysis is static. +- **Contract/integration tests** (`tests/contract/test_pubmed_contract.py`, `tests/integration/test_grantbot_live.py`) + were read, not run, per the preamble. + +Scratch scripts used: `/tmp/claude-1000/-home-a-scripps-coPI-science/c8a1ec5c-25b0-4282-b6cd-c2567cafee26/scratchpad/23/{cor27,cor28,cor26,v7_retry,realfoas}.py`. diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_23_redteam.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_23_redteam.md new file mode 100644 index 00000000..1f1ed559 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_23_redteam.md @@ -0,0 +1,120 @@ +# Issue #23 — red-team pass on the first agent's report + +Tree: `/home/a/scripps/coPI.science` @ `copi-prod` 18ba52c, clean. Re-derived every row by fresh grep/read; every +mechanical claim re-run with `.venv-test/bin/python` (scripts in `scratchpad/rt23/probe.py`, `probe2.py`; the second +probe hard-blocks `socket.connect`). Pre-fix code read via `git show ^:` only. + +## 1. Table + +| id | first-agent verdict | red-team | one-line reason | evidence | +|---|---|---|---|---| +| V7a | FIXED (9dbc9e0) | UPHELD | pre-fix `raise SlackApiError(..., response=exc.response)` after the loop (`git show 9dbc9e0^` :167) → UnboundLocalError; test :146-169 would fail pre-fix on both asserts | `slack_client.py:324-342`; test body read | +| V7b | FIXED (d311170) | UPHELD | pre-fix was a single `conversations_history(limit=limit)` call (d311170^ :216-217); no unpaginated poller survives — `poll_dm_messages` delegates to `poll_channel_messages`; `get_thread_replies`/`get_all_thread_replies`/`get_full_channel_history` all go through `_paginate`; sim cursor advances per message at `simulation.py:2777` (bot) and `:2837` (human) | `slack_client.py:534-535,571-572,602-603,633-634,856`; tests :599-660 | +| V7c | FIXED (d311170) | UPHELD (residual in scripts only) | pre-fix single `conversations_list(types, limit=200)` (d311170^ :619); `grantbot._ensure_channel_membership` now uses paginated `slack_web.list_channel_ids`; only unpaginated listing left is `scripts/wipe_slack.py:231` (`limit=200`, no cursor) — a host script, not a prod process | `slack_client.py:1039-1040`; `slack_web.py:125-177`; `grantbot.py:440-443` | +| V7d | STILL PRESENT (med) | UPHELD (UNVERIFIABLE on API shape) | code unchanged; no offline artefact in repo or slack_sdk shows `users.info` shape; only other consumer `agent_page.py:1378` reads `real_name`/`name` | `slack_client.py:662` | +| V7e | STILL PRESENT | UPHELD + sharpened | reproduced: HTTP-date → `ValueError` escapes; **also `"2.5"` (a legal delta-seconds header is integer, but proxies emit floats) escapes**; `99999999` and `-5` accepted; slack_web's `_call` has the cap/float but `slack_client` does not | probe output §2 | +| C26a | STILL PRESENT | UPHELD | `:344-346` fallback; `{}`/`123`/`null`/`garbage` all → full key list (reproduced) | probe2 | +| C26b | STILL PRESENT | UPHELD | dict/list element → `TypeError: unhashable` at `:538` (reproduced); no isinstance check | probe2 | +| C26b' | STILL PRESENT | UPHELD | `_mark_run_complete()` only after success `:777`; `except Exception` logs; `_should_run_today()` stays True; re-fires every `check_interval` (900 s default; compose passes none) while `now.hour >= 8` UTC | `grantbot.py:767-786`; `docker-compose.prod.yml:116` | +| C26b'' | NOT REPRODUCIBLE | UPHELD | prod runs `scheduler` (compose :116); `except Exception` swallows `TypeError`; only `BaseException` (KeyboardInterrupt/SystemExit/CancelledError out of `asyncio.run`) escapes, and compose `restart: unless-stopped` (:115) would restart even that. The issue is wrong for the prod path; the one-shot `main` (:724-745) would exit non-zero but is not what runs. Refinement: re-fire is bounded to 08:00–23:59 UTC (≤64 attempts/day), and each attempt re-calls the LLM so it can succeed | `grantbot.py:749-786` | +| C26c | STILL PRESENT | UPHELD | 18ba52c only changed INFO→WARNING (diff confirms); engine `_resolve_service_bot_uids` explicitly refuses su-token fallback and `_bot_uid_map` resolves roster first | `grantbot.py:615-625`; `simulation.py:3967-3971,4004-4010` | +| C26d | STILL PRESENT | UPHELD | only refs are in-file `:55,68-71`; `PROFILES_DIR` at `:44` is grantbot-local (other modules define their own) | grep over src/scripts/tests | +| C26e | FIXED (confirmed) | UPHELD | `_claim_foa` ON CONFLICT DO NOTHING + commit, `_release_foa` delete + commit | `grantbot.py:246-275` | +| C27a | STILL PRESENT | UPHELD | all 6 rows reproduce exactly | probe §2 | +| C27b | STILL PRESENT | UPHELD | both `flags & IGNORECASE` False | probe | +| C27c | STILL PRESENT | UPHELD | `extract_foa_number` → None for PAR/PA/PAS/DE-FOA; docstring `:18` lists 2 failing examples | `foa_cache.py:18-21,81-84` | +| C27-new (NSF) | new claim | QUALIFIED | mechanically true that neither regex admits any prefix other than RFA/PAR/PA/NOT/OTA/RFI/DE-FOA — so no NSF number can match — but the actual Grants.gov `number` format for NSF is not verifiable offline (no fixture in repo) | probe: `NSF 25-543`, `25-543`, `PD 24-7275` all False | +| C28a | STILL PRESENT | UPHELD | class codepoints `['0x27','0x27']`; U+2019 forms → False | probe | +| C28b | STILL PRESENT | UPHELD | `"Agreed, we can send the plasmids and the mice next week."` → True | probe | +| C28b' | STILL PRESENT | UPHELD | reset only `:1506`; increment `:1457`; back-off `:1463-1466`; `has_pending_reply=True` at 9 literal sites + 1 variable (`:4242`) = the first agent's "≥10" | grep | +| C28c | STILL PRESENT | UPHELD | `@GRANTBOT`/`@SuBOT` → None | probe | +| C28d | N/A + note | UPHELD | `message_log.py:407` still case-sensitive (18ba52c added a service-bot skip, not IGNORECASE); `simulation.py:2523,2558` same literal | grep | +| C29a | STILL PRESENT | UPHELD | no retry/backoff/tenacity in the three files | grep | +| C29b | STILL PRESENT | UPHELD | `raise_for_status()` `:94` before `sleep(0.12)` `:95` | `pubmed.py:91-96` | +| C29c | STILL PRESENT (qualified) | QUALIFIED | "all callers sequential except Phase 4 gather" is true for the agent/worker (`src/worker/main.py` has no gather/create_task) but omits the **app** process: `auth.py:189` `fetch_orcid_profile` and other request handlers run concurrently per HTTP request, so concurrent logins/onboardings are a second concurrency source through the same process-local `Semaphore(8)` | `auth.py:189`; `pubmed.py:73` | +| C29d | context | UPHELD | `_ncbi_get` :76-90 sets tool/email only | — | +| C30 | STILL PRESENT | UPHELD | debit `:128`/`:137` before await; `except` `:146-148`; `_execute_retrieve_abstract` `:205-207` returns `result["error"]` for the non-raising `{"error":…}` at `pubmed.py:434,439` — new claim reproduced by read | `tools.py:126-148,203-207` | +| V10a | FIXED (02143de, fa143a6) | UPHELD (no regression test) | pre-fix imported `_get_bot_token` from agent_page + raw `WebClient` + bare `except` (02143de^ :234-245); current uses `token_for_agent_row` + `lookup_user_by_email_async` + `logger.warning`. **No test exercises invite.py's sync**: `lookup_user_by_email_async` appears only in `tests/unit/test_slack_web.py`; `tests/integration/test_agent_page.py:819` covers the agent-page delegate link, not the invite path — the issue's "definition of done" is unmet for V10 | `invite.py:232-252` | +| V10b | STILL PRESENT (nit) | UPHELD | `if bot_token:` with no else | `invite.py:237-238` | + +New claims by the first agent, each re-verified: +- `slack_web._call` float()+try+30 s cap — **confirmed** (`slack_web.py:49,104-116`). +- `fetch_abstract` non-raising `{"error":…}` charged to budget — **confirmed** (`pubmed.py:434,439`; `tools.py:128,205-207`). +- NCBI callers sequential except Phase 4 — **qualified** (see C29c). +- NSF numbers match neither regex — **qualified** (see C27-new). +- 18ba52c raised log level only; engine refuses su→grantbot mapping — **confirmed**. +- bare JSON string iterates char-by-char → 0 selected silently — **reproduced**: `'"PAR-24-293"'` → `sel='PAR-24-293'`, chosen `[]`, log says "Selected 10 of 2 opportunities". +- `tools.py:122-123` comment contradicts code — **confirmed** (increments for every `retrieve_abstract`). +- "≥10 re-arm sites" — **confirmed** (9 literal + 1 variable). + +## 2. Detail for QUALIFIED rows and sharpened items + +**V7e (sharpened).** Real `_call_with_retry` with `time.sleep` stubbed: +``` +'Wed, 21 Oct 2015 07:28:00 GMT' -> ESCAPES as ValueError: invalid literal for int() with base 10: 'Wed, 21 Oct 2015 07:28:00 GMT' +'99999999' -> SlackApiError (callers catch) sleeps=[99999999, 99999999, 99999999] +'-5' -> SlackApiError (callers catch) sleeps=[-5, -5, -5] +'2.5' -> ESCAPES as ValueError: invalid literal for int() with base 10: '2.5' +``` +The `-5` case: the real `time.sleep(-5)` raises `ValueError` too (the probe's stub hides it), so a negative header is a +third escape route. `tests/unit/test_slack_client_contract.py:110-113` autouse-stubs `time.sleep`, so the suite can never +observe any of these. + +**C26b'' (NOT REPRODUCIBLE, upheld with precision).** `grantbot.py:767-786`: +``` + while True: + now = datetime.now(UTC) + if _should_run_today() and now.hour >= run_hour: + try: + results = asyncio.run(run_grantbot(...)) + _mark_run_complete() + except Exception as exc: + logger.error("Daily run failed: %s", exc, exc_info=True) + ... + time.sleep(check_interval) +``` +- The `TypeError` from `:538` is an `Exception` → caught → loop continues. Nothing kills the process. +- `_mark_run_complete()` is skipped, so `_should_run_today()` stays True → re-fire every 900 s, but only while + `now.hour >= 8` UTC (the loop is idle 00:00–07:59). So the worst case is ~64 Grants.gov+LLM rounds/day, not unbounded. +- Every re-fire re-queries the LLM (`_select_opportunities` :322), so a well-formed answer on a later attempt ends the + loop — retry-until-parse, as the first agent said. +- Paths that *would* end the process: `BaseException` subclasses (`KeyboardInterrupt`, `SystemExit`, + `asyncio.CancelledError` propagating out of `asyncio.run`) — none is produced by the JSON defect; and compose `restart: + unless-stopped` (`docker-compose.prod.yml:115`) restarts the container anyway. +- The only code path where `main()` "dies" is the typer one-shot `main` (`:724-745`, no try) — not the prod command. +Verdict: the **issue text is wrong** for the prod configuration; the first agent is right. + +**C29c (QUALIFIED).** The first agent's reachability statement ("all pipeline callers are sequential awaits — the only +real concurrency is Phase 4 gather") ignores that the `app` process serves requests concurrently. `src/routers/auth.py:189` +awaits `fetch_orcid_profile` inside the OAuth callback; onboarding/profile routes reach `pubmed._ncbi_get` likewise. N +concurrent users = N concurrent NCBI/ORCID calls through the same `Semaphore(8)`. Verdict unchanged (still present), but +"worst case only under parallel tool calls" understates reachability. + +**C27-new NSF (QUALIFIED).** Both regexes are prefix-anchored (`RFA|PAR|PA|NOT|OTA|RFI|DE-FOA` and `PA[RS]?|RFA`), so +*whatever* format Grants.gov returns for NSF, it cannot match unless it starts with one of those. That part is certain. +Whether GrantBot actually posts NSF numbers depends on the LLM selection and the `number` field's shape, which no fixture +in the repo shows — the first agent presented it as fact; it is an inference. + +**V10a (QUALIFIED on test coverage).** The fix is real, but no test would fail against the pre-fix `invite.py`: +`grep -rn lookup_user_by_email_async tests/` → `tests/unit/test_slack_web.py` only; `delegate_slack_ids` assertions live in +`tests/integration/test_agent_page.py:835,853` (agent-page link flow). The issue's DoD ("each PR ships a test that fails +against the pre-fix code") is not met for V10 even though the issue marks it done. + +**V7c residual.** `scripts/wipe_slack.py:231` `client.conversations_list(types="public_channel", limit=200)["channels"]` +is a single page with no cursor — a destructive script that would silently skip channels past 200. `:168` likewise +single-page history. Out of the issue's scope (agent process), noted for completeness. `scripts/slack_test_teardown.py:48-59` +does paginate. + +## 3. Mis-cites by the first agent + +None material. Checked: `slack_client.py:324-342, 344-398, 505-550, 662, 1006-1059`; `grantbot.py:341-346, 476-486, +537-538, 615-625, 771-780`; `funding_rules.py:23, 25, 95, 154`; `foa_cache.py:18-21`; `pubmed.py:73, 87-88, 91-96, 434, 439`; +`tools.py:126-148, 205-207`; `invite.py:232-252`; `simulation.py:1457, 1463-1466, 1506, 2777, 3967-3971, 4004-4010`; +`tests/fakes.py:243-254`; `test_slack_client_contract.py:146-169, 172-182, 409, 441-579, 599-659` — all point at the quoted +code. Minor: the report says `_bot_uid_map` at `:4004-4010`; the def is at `:4004` and the "roster first" text is in its +docstring `:4005-4010` — fine. The "84 dead lines" arithmetic (83 code + 1 blank) is correct. + +## 4. Counts + +**28 rows: Upheld 24 · Overturned 0 · Qualified 3 (C29c reachability, C27-new NSF inference, V10a test coverage) · Unverifiable 1 +(V7d API shape — rests on the code being unchanged, not on an observed response).** diff --git a/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_24.md b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_24.md new file mode 100644 index 00000000..5e79c875 --- /dev/null +++ b/docs/plans/2026-09-02-close-issues-20-27-evidence/findings/issue_24.md @@ -0,0 +1,273 @@ +# Issue #24 verification — "Web request-path robustness: concurrent-insert 500s and a 5-minute event-loop freeze" + +Verified against `/home/a/scripps/coPI.science` @ `copi-prod` HEAD `18ba52c` (clean tree), 2026-09-02. +Method: symbol lookup + quoted code, AST inspection of the handlers, git blame/ancestry against the audit +baseline `b7edcbc`, and three executed python snippets (event-loop starvation demo with a `to_thread` control, +worst-case sleep count, autoflush check). No repo edits, no docker, no network. + +## 1. Summary table + +| id | claim (one line) | verdict | key evidence | conf | +|---|---|---|---|---| +| P1 | `app` runs a single uvicorn worker (Dockerfile:24, compose:29, no `--workers`) | STILL PRESENT (premise holds) | `Dockerfile:24` CMD uvicorn w/o `--workers`; `docker-compose.prod.yml:29` same; no `WEB_CONCURRENCY`/`--workers` anywhere in compose files, Dockerfile, `.env.example`, `src/main.py` | high | +| P2 | nginx `proxy_read_timeout 120s` | STILL PRESENT | `nginx/nginx.conf:145` (in `location /` of the `${DOMAIN}` server, line 128); also :124, :237, :302 | high | +| P3 | Neither endpoint touched by the PR stack / unchanged since baseline | CONFIRMED (no change) | every blame commit on the four code blocks is an ancestor of `b7edcbc` (b581c047, c10b3ddd, 38be9df0, 3d0717a9, aa38b04b, 5e796c68, a0f4b155); `git log -S to_thread` over admin.py/admin_provisioning.py/slack_provisioning.py = empty | high | +| P4 | Cited line numbers | STALE (symbols correct) | see §2.0 — `public.py:485-503` is now 479-541, `agent_page.py:487-513` is now 454-515, `admin.py:945-963/972-996` is now 964-989/991-1024, vote pattern `1038-1057` is now 1083-1099, etc. `slack_provisioning.py:124-152` and `:44-54` are unchanged | high | +| V5-1 | `waitlist_submit` does SELECT-then-INSERT with no `IntegrityError` catch | STILL PRESENT | `src/routers/public.py:516-519` SELECT, `:526` add, `:534` commit; AST: 0 try-blocks in the function; `IntegrityError` is imported (`:20`) but used only at `:1085` (vote) | high | +| V5-2 | `waitlist_signups.email` is UNIQUE | CONFIRMED | `src/models/access.py:43` `unique=True`; `alembic/versions/0010_access_gate_and_waitlist.py:76` `unique=True` | high | +| V5-3 | `review_proposal` guard is SELECT + `HTTPException(400)` then unguarded `db.add`/`commit` | STILL PRESENT | `src/routers/agent_page.py:487-494` SELECT+400, `:506` add, `:513` commit; AST: 0 try-blocks; `IntegrityError` imported `:14`, used only in `post_agent_message` (`:1018,:1022`) | high | +| V5-4 | `uq_proposal_reviews_decision_agent` exists | CONFIRMED (DB-side only) | `alembic/versions/0004_...py:84-88` on `(thread_decision_id, agent_id)`; the ORM model `ProposalReview` (`src/models/agent_registry.py:66-99`) declares NO `__table_args__` for it — constraint lives only in the migration; `tests/integration/test_db_contract.py:300-310` pins it | high | +| V5-5 | Per-IP limiter (10/3600 s) doesn't stop a double-click | CONFIRMED | `public.py:39` `SlidingWindowRateLimiter(max_events=10, window_seconds=3600)`; nginx `req_general` is 20r/s burst 40 (`nginx.conf:29,130`); no JS double-submit guard on the form (`templates/landing.html:620-634`) or the review form (`templates/agent/dashboard.html`) | high | +| V5-6 | Vote endpoint has the correct pattern | CONFIRMED | `public.py:1083-1099` try commit / except `IntegrityError` → rollback → re-select `scalar_one()` → update → commit | high | +| V5-7 | PI web-message writer: rollback + one retry then 409 | CONFIRMED | `agent_page.py:1015-1027` | high | +| V5-8 | The race surfaces as a 500 | CONFIRMED | no `exception_handler` anywhere in `src/`; `get_db` (`src/database.py:54-56`) rolls back and re-raises → Starlette 500 | high | +| V5-9 | DoD: a concurrent-insert test | STILL MISSING | only sequential duplicate tests exist (`tests/integration/test_agent_page.py:602-633`, `tests/integration/test_proposal_review.py:475-523`); no `asyncio.gather` against `/waitlist` or `/review` anywhere in tests/ | high | +| C2-1 | `admin_provision_slack` / `_callback` are `async def` awaiting `start_`/`complete_provisioning` | CONFIRMED | `src/routers/admin.py:964-989` (`:982 await start_provisioning`), `:991-1024` (`:1015 await complete_provisioning`) | high | +| C2-2 | `start_provisioning` calls sync `httpx.post` + `time.sleep` on the loop | STILL PRESENT (demonstrated) | `admin_provisioning.py:132-139,147,153` → `slack_provisioning.py:124-146`; **measured: 0 heartbeat ticks in a 2.07 s mocked rate-limit; `to_thread` control: 40 ticks** | high | +| C2-3 | Up to 5 retries × 60 s default ⇒ ~5 min freeze | CONFIRMED, and UNDERSTATED | measured 5 sleeps × 60 s = 300 s, plus up to 5 × 20 s httpx timeout = 400 s; `Retry-After` is **uncapped** (header 900 ⇒ 4500 s). `slack_web._call` caps at 30 s (fa143a6) but that cap is not applied here | high | +| C2-4 | `lookup_team_id` sync on the async path | STILL PRESENT | `slack_provisioning.py:44-54` (httpx, timeout 10) called from `admin_provisioning.py:184` | high | +| C2-5 | `exchange_code` sync on the async path | STILL PRESENT | `slack_provisioning.py:155-180` (httpx, timeout 15) called from `admin_provisioning.py:212` | high | +| C2-6 | (not in issue) `rotate_config_token` sync on the async path | NEW — MISSED BY ISSUE | `slack_provisioning.py:57-74` (httpx, timeout 15) called from `admin_provisioning.py:99` inside `_config_token` | high | +| C2-7 | `get_db` session + pooled connection held across the blocking call | STILL PRESENT | `_config_token` runs SELECTs (`:85-86,94`) → autobegin checks out a connection; `_create` at `:147` runs before `commit` at `:176`; `complete_provisioning` SELECTs `:193-205` precede `exchange_code :212`, commit `:231`; `get_db` yields the session for the whole handler (`database.py:47-58`); pool 5+10 | high | +| C2-8 | `asyncio.to_thread` precedents exist | CONFIRMED | `slack_web.py:274-300`, `grantbot.py:631`, `agent_page.py:319`; none in admin/provisioning; fa143a6 touched no provisioning file | high | +| C2-9 | nginx 120 s turns the freeze into a 504 for the admin | CONFIRMED (conditional) | needs ≥2 rate-limited rounds at the 60 s default; a single 60 s wait returns before the 120 s read timeout | high | +| C2-10 | DoD: a test that the handler does not block the loop | STILL MISSING | precedent/template exists at `tests/unit/test_slack_web.py:161-180` (thread-identity assertion) but covers only `slack_web`; `tests/unit/test_slack_provisioning.py:117` monkeypatches `time.sleep` to a no-op so it cannot observe the block; `test_admin_provisioning.py` covers only `_config_token` | high | + +## 2. Per-item detail + +### 2.0 Line-number drift (P4) +Issue cites (`b1d54da`) → current (`18ba52c`): +- `waitlist_submit` public.py:485-503 → **479-541** +- `review_proposal` agent_page.py:487-513 → **454-515** +- vote endpoint public.py:1038-1057 → **1017-1102** (the try/except is 1083-1099) +- PI web-message writer agent_page.py:1013-1027 → **949-1029** (the try/except is 1015-1027) +- `admin_provision_slack` admin.py:945-963 → **964-989**; callback :972-996 → **991-1024** +- `start_provisioning`/`complete_provisioning` admin_provisioning.py:124-186 → **124-187 / 190-233** +- `create_app` retry loop slack_provisioning.py:124-152 → **124-152 (unchanged)**; `lookup_team_id` :44-54 **(unchanged)**; `exchange_code` :154-172 → **155-180** +- `slack_web.py:274-299` → 274-300; `grantbot.py:624` → **631**; `agent_page.py:319` → 319 +All symbol names are correct. + +### 2.1 Premise (P1, P2, P3) +`Dockerfile:24`: +``` +CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] +``` +`docker-compose.prod.yml:29`: +``` + command: ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] +``` +`grep -rn "WEB_CONCURRENCY\|--workers\|workers=" docker-compose*.yml Dockerfile .env.example src/main.py` → no hits. Single process, single event loop. + +`nginx/nginx.conf:128-145` (`location /` under `server_name ${DOMAIN}`): `proxy_connect_timeout 60s; proxy_send_timeout 120s; proxy_read_timeout 120s;`. The admin provisioning routes (`/admin/agents/...`) match this block. + +Blame ancestry: every commit that last touched the four code blocks predates the audit baseline: +``` +b581c047 2026-04-15 waitlist (landing page) in b7edcbc: yes +c10b3ddd 2026-07-16 SEC-17 waitlist truncate/throttle in b7edcbc: yes +38be9df0 2026-03-26 review_proposal in b7edcbc: yes +3d0717a9 / aa38b04b review_proposal (email/delegate) in b7edcbc: yes +5e796c68 2026-07-16 SEC-10 start_provisioning in b7edcbc: yes +a0f4b155 2026-06-26 create_app retry loop in b7edcbc: yes +``` +`git log --oneline b7edcbc..HEAD -- ` shows 25 commits, none of which changed these blocks (blame above). `git log -S to_thread -- src/services/admin_provisioning.py src/services/slack_provisioning.py src/routers/admin.py` → empty. + +### 2.2 V5-1 `waitlist_submit` — STILL PRESENT +`src/routers/public.py:479-541` (excerpt, current lines): +``` +488 if not _waitlist_limiter.allow(client_ip(request)): +489 raise HTTPException(status_code=429, detail="too many requests") +... +516 result = await db.execute( +517 select(WaitlistSignup).where(WaitlistSignup.email == email_clean) +518 ) +519 existing = result.scalar_one_or_none() +520 +521 if existing: +522 existing.name = name_clean or existing.name +... +525 else: +526 db.add( +527 WaitlistSignup( +528 email=email_clean, +... +533 ) +534 await db.commit() +``` +AST check (`scratchpad/24/ast_check.py`, run against the real module): +``` +src/routers/public.py::waitlist_submit lines 479-541 + try-blocks: 0 except-types: [] + db.add at: [526] commit at: [534] +``` +`IntegrityError` is imported at `public.py:20` and used exactly once, at `:1085` inside `submit_proposal_vote`. Two concurrent first-time posts of the same email both see `existing is None`, both `add`, and the loser's `commit` raises `UniqueViolation` → unhandled → 500 (no `exception_handler` in `src/`; `get_db` rolls back and re-raises, `database.py:54-56`). + +Mitigations looked for and not found: no client-side disable on the form (`templates/landing.html:620-634` is a plain `
` with a `
__key`; confirm + the exact name in the traceback) with no `try/except` to catch it (that guard is Task + 26.10, landing after this one). The `client` fixture's `httpx.ASGITransport` is constructed + with its default `raise_app_exceptions=True` (`tests/conftest.py:106-132` does not override + it), so the exception propagates out of `await client.post(...)` itself rather than coming + back as a 500 response — pytest reports this test as `ERROR`, not `FAILED`. + +- [ ] **Step 3: Implement** + `src/routers/agent_page.py:393-414`, before: +```python +async def derive_agent_identity( + db: AsyncSession, full_name: str +) -> tuple[str, str]: + """Return ``(agent_id, bot_name)`` for a PI's display name. + + Both values are derived here, together, because they must agree: the + collision prefix used to be applied to agent_id at one line and bot_name + rebuilt from the bare last name four lines later, so Peng Wu got + ``pwu`` / ``WuBot`` — colliding with Chunlei Wu's bot while the ids differed. + CLAUDE.md documents ``pwu`` / ``PWuBot``. + """ + last_name = full_name.split()[-1] + stem = "".join(c for c in last_name.lower() if c.isalpha()) + display = last_name + + collision = await db.execute( + select(AgentRegistry).where(AgentRegistry.agent_id == stem) + ) + if collision.scalar_one_or_none(): + initial = full_name[0] + return f"{initial.lower()}{stem}", f"{initial.upper()}{display}Bot" + return stem, f"{display}Bot" +``` + after: +```python +async def derive_agent_identity( + db: AsyncSession, full_name: str +) -> tuple[str, str]: + """Return ``(agent_id, bot_name)`` for a PI's display name. + + Both values are derived here, together, because they must agree: the + collision prefix used to be applied to agent_id at one line and bot_name + rebuilt from the bare last name four lines later, so Peng Wu got + ``pwu`` / ``WuBot`` — colliding with Chunlei Wu's bot while the ids differed. + CLAUDE.md documents ``pwu`` / ``PWuBot``. + + A THIRD same-initial namesake collides on the prefixed candidate too + (issue #26 C1): fall back to a numeric suffix appended to the prefixed + candidate (``pwu2``, ``pwu3``, ...), matching + ``scripts/backfill_agents.py``'s ``_resolve_agent_id``/``_bot_name_for``. + """ + last_name = full_name.split()[-1] + stem = "".join(c for c in last_name.lower() if c.isalpha()) + display = last_name + + collision = await db.execute( + select(AgentRegistry).where(AgentRegistry.agent_id == stem) + ) + if not collision.scalar_one_or_none(): + return stem, f"{display}Bot" + + initial = full_name[0] + prefixed = f"{initial.lower()}{stem}" + collision = await db.execute( + select(AgentRegistry).where(AgentRegistry.agent_id == prefixed) + ) + if not collision.scalar_one_or_none(): + return prefixed, f"{initial.upper()}{display}Bot" + + for i in range(2, 20): + candidate = f"{prefixed}{i}" + collision = await db.execute( + select(AgentRegistry).where(AgentRegistry.agent_id == candidate) + ) + if not collision.scalar_one_or_none(): + return candidate, f"{initial.upper()}{display}{i}Bot" + raise HTTPException( + status_code=409, + detail="Could not derive a unique agent identity, please contact support", + ) +``` + +- [ ] **Step 4: Run it, expect PASS** + Same command as Step 2. + +- [ ] **Step 5: Run the neighbours** + `docker compose -f docker-compose.yml exec -T -e TEST_DATABASE_URL=postgresql+asyncpg://copi:copi@postgres:5432/copi_a26 app python -m pytest tests/integration/test_agent_page.py -k "signup" -v` + Must stay green with no inversion: `test_signup_creates_a_pending_agent_row`, + `test_signup_prefixes_the_first_initial_only_on_a_last_name_collision` (two Wus, unaffected — + the third-collision branch is never reached), `test_signup_collision_also_disambiguates_the_bot_name`, + `test_signup_needs_a_completed_profile`. + +- [ ] **Step 6: Commit** +``` +git add src/routers/agent_page.py tests/integration/test_agent_page.py +git commit -m "fix(agent-page): numeric-suffix fallback on a third same-initial signup collision (#26 DOC-C C1)" +``` + +--- + +### Task 26.10: IntegrityError guard on request_agent's commit -> 409 [closes: #26 C1(b) / DOC-C] + +Depends on Task 26.9 landing first (same function region); this task only wraps the existing +`db.add(agent); await db.commit()` in `request_agent`, it does not touch `derive_agent_identity`. + +**Files:** +- Modify: `src/routers/agent_page.py` — `request_agent` (currently :418-444, the + `db.add(agent); await db.commit()` pair at :442-443) +- Test: `tests/integration/test_agent_page.py` + +**Interfaces:** +- Consumes: `IntegrityError` already imported at `agent_page.py:14`. Mirrors the rollback-then-409 + shape at `src/routers/agent_page.py:1018-1027` (a different endpoint's guard — that one retries + once before giving up; this one cannot usefully retry because `derive_agent_identity` would + recompute the exact same candidate from the exact same DB state it just lost the race against, + so it fails fast to 409 instead). + +- [ ] **Step 1: Write the failing test** (add to `tests/integration/test_agent_page.py`; add + `from unittest.mock import AsyncMock` to the file's imports) +```python +async def test_signup_returns_409_on_a_lost_identity_race(client, db_session, monkeypatch): + """Two concurrent signups can both pass derive_agent_identity's SELECT + before either commits (TOCTOU on the agent_id unique constraint). Force + the race by making derive_agent_identity return an id that's already + taken, and assert the handler converts the resulting IntegrityError into + a 409, not a raw 500 (issue #26 C1: no try/except existed on this commit + at all).""" + monkeypatch.setattr( + "src.routers.agent_page.derive_agent_identity", + AsyncMock(return_value=("racer", "RacerBot")), + ) + other_user = await factories.make_user( + db_session, name="Already There", email="already@example.org" + ) + await factories.make_agent( + db_session, user=other_user, agent_id="racer", bot_name="RacerBot", + pi_name="Already There", status="pending", + ) + + user = await factories.make_user(db_session, name="Race Newcomer", email="race@example.org") + await factories.make_profile(db_session, user=user) + await db_session.flush() + + r = await client.post("/agent/request", headers=_auth(user.id)) + assert r.status_code == 409 + assert (await _agent_of(db_session, user)) is None +``` + +- [ ] **Step 2: Run it, expect FAIL** + `docker compose -f docker-compose.yml exec -T -e TEST_DATABASE_URL=postgresql+asyncpg://copi:copi@postgres:5432/copi_a26 app python -m pytest tests/integration/test_agent_page.py::test_signup_returns_409_on_a_lost_identity_race -v` + Expected: an **ERROR**, not a clean assertion failure — `sqlalchemy.exc.IntegrityError` + propagates out of `await client.post(...)` (same `raise_app_exceptions=True` transport + default as Task 26.9's Step 2) because `request_agent` has no `try/except` around its + `db.add`/`db.commit()` yet. + +- [ ] **Step 3: Implement** + `src/routers/agent_page.py:433-444`, before: +```python + agent = AgentRegistry( + agent_id=agent_id, + user_id=current_user.id, + bot_name=bot_name, + pi_name=current_user.name, + status="pending", + ) + db.add(agent) + await db.commit() + + return RedirectResponse(url="/agent", status_code=302) +``` + after: +```python + agent = AgentRegistry( + agent_id=agent_id, + user_id=current_user.id, + bot_name=bot_name, + pi_name=current_user.name, + status="pending", + ) + db.add(agent) + try: + await db.commit() + except IntegrityError as exc: + # Lost a race on the agent_id unique constraint — another request + # committed the same derived identity between our SELECT and our + # commit. Retrying would recompute the identical candidate from the + # same now-stale read, so fail fast instead of looping. See #26 C1. + await db.rollback() + raise HTTPException( + status_code=409, + detail="An agent request for this identity is already in progress, please retry", + ) from exc + + return RedirectResponse(url="/agent", status_code=302) +``` + (`except IntegrityError as exc:` / `... from exc` — a bare `except IntegrityError:` + + `raise HTTPException(...)` here is ruff **B904**; reconciliation item 15's lint ceiling has no + headroom to spare, see 25.2/25.6/26.10's shared note.) + +- [ ] **Step 4: Run it, expect PASS** + Same command as Step 2. + +- [ ] **Step 5: Run the neighbours** + `docker compose -f docker-compose.yml exec -T -e TEST_DATABASE_URL=postgresql+asyncpg://copi:copi@postgres:5432/copi_a26 app python -m pytest tests/integration/test_agent_page.py -k "signup" -v` + All signup tests (including Task 26.9's) must stay green — none of them hit the + `IntegrityError` branch, since `derive_agent_identity` is unpatched in those tests and + genuinely returns a free id. + +- [ ] **Step 6: Commit** +``` +git add src/routers/agent_page.py tests/integration/test_agent_page.py +git commit -m "fix(agent-page): convert a lost agent_id race into a 409, not a 500 (#26 DOC-C C1)" +``` + +--- + +### Task 26.11: scripts/backfill_agents.py — numeric-branch parity + docstring fix [closes: #26 C2 / DOC-C] + +**Files:** +- Modify: `scripts/backfill_agents.py` — `_resolve_agent_id` (currently :47-70) and + `_bot_name_for` (currently :72-77) +- Modify: `scripts/generate_sparsedata_user.py` — `_resolve_agent_id` (currently :165-183): the identical bare-stem numeric branch (`candidate = f"{base}{i}"` → `f"{prefixed}{i}"`), same unit-test shape (Decision D22) +- Test: `tests/unit/test_backfill_agents.py` (new) + +**Interfaces:** +- Consumes: none (script has no runtime dependency on Task 26.9's route change; this task only + makes the two independent implementations of the same collision policy agree, per the issue's + finding that `_resolve_agent_id`'s docstring falsely claimed parity with `agent_page.py`). + +**Scope limit (minor, pre-existing, not fixed here):** "parity" in this task's title is about the +numeric-suffix branch only. `_bot_name_for` still capitalizes `last_alpha` (letters-only) while +`derive_agent_identity` (agent_page.py) uses the raw last name, so the two still diverge for a +name like `"van Dyke"`. Out of scope for this task — it is not part of the C2 finding being +closed here, and fixing it would touch `derive_agent_identity` (owned by Task 26.9/26.8's region, +not this script). + +- [ ] **Step 1: Write the failing test** +```python +"""scripts/backfill_agents.py's collision/bot-name logic must match the web +path (agent_page.derive_agent_identity, fixed in issue #26 Task 26.9): the +numeric branch extends the PREFIXED candidate ('pwu2'), not the bare stem +('wu2') — before the fix these diverged, and _bot_name_for('wu2', ...) even +produced 'WWuBot' (agent_id[0] of 'wu2' is 'w'), silently colliding with any +bare-stem 'w...' bot via SimulationEngine._bot_name_to_id (issue #26 C2, +red-team sharpening). +""" + +from types import SimpleNamespace + +from scripts.backfill_agents import _bot_name_for, _resolve_agent_id + + +class _FakeAgentRegistryDb: + """Serves AgentRegistry.agent_id == lookups from a fixed set.""" + + def __init__(self, taken): + self._taken = set(taken) + + async def execute(self, stmt): + candidate = stmt.whereclause.right.value + hit = candidate if candidate in self._taken else None + return SimpleNamespace(scalar_one_or_none=lambda: hit) + + +async def test_resolve_agent_id_bare_stem_when_free(): + assert await _resolve_agent_id("Chunlei Wu", _FakeAgentRegistryDb(taken=[])) == "wu" + + +async def test_resolve_agent_id_prefixes_on_first_collision(): + assert await _resolve_agent_id("Peng Wu", _FakeAgentRegistryDb(taken=["wu"])) == "pwu" + + +async def test_resolve_agent_id_numeric_suffix_extends_the_prefixed_candidate(): + db = _FakeAgentRegistryDb(taken=["wu", "pwu"]) + assert await _resolve_agent_id("Pei Wu", db) == "pwu2" + + +def test_bot_name_for_bare_stem(): + assert _bot_name_for("wu", "Chunlei Wu") == "WuBot" + + +def test_bot_name_for_prefixed(): + assert _bot_name_for("pwu", "Peng Wu") == "PWuBot" + + +def test_bot_name_for_numeric_suffix_matches_the_web_path(): + assert _bot_name_for("pwu2", "Pei Wu") == "PWu2Bot" +``` + +- [ ] **Step 2: Run it, expect FAIL** + `.venv-test/bin/python -m pytest tests/unit/test_backfill_agents.py -q -p no:cacheprovider` + Expected failures: `test_resolve_agent_id_numeric_suffix_extends_the_prefixed_candidate` gets + `"wu2"` instead of `"pwu2"` (current loop is `candidate = f"{base}{i}"`); the bare-stem and + prefixed tests already pass (unaffected by the bug); `test_bot_name_for_numeric_suffix_matches_the_web_path` + gets `"PWuBot"` (missing the "2" entirely — `agent_id[0]` of `"pwu2"` is `"p"`, and the + current implementation has no digit handling at all). + +- [ ] **Step 3: Implement** + `scripts/backfill_agents.py:47-70`, before: +```python +async def _resolve_agent_id(name: str, db: AsyncSession) -> str: + """Same collision logic as scripts/generate_sparsedata_user.py + agent_page.py. + + Order: bare last name → first-initial prefix → numeric suffix. + """ + base = _slugify_last_name(name) + candidate = base + coll = await db.execute(select(AgentRegistry).where(AgentRegistry.agent_id == candidate)) + if coll.scalar_one_or_none() is None: + return candidate + initial = name.strip()[0].lower() if name.strip() else "x" + candidate = f"{initial}{base}" + coll = await db.execute(select(AgentRegistry).where(AgentRegistry.agent_id == candidate)) + if coll.scalar_one_or_none() is None: + return candidate + for i in range(2, 20): + candidate = f"{base}{i}" + coll = await db.execute( + select(AgentRegistry).where(AgentRegistry.agent_id == candidate) + ) + if coll.scalar_one_or_none() is None: + return candidate + raise RuntimeError(f"Could not find unique agent_id for {name!r}") +``` + after: +```python +async def _resolve_agent_id(name: str, db: AsyncSession) -> str: + """Same collision logic as scripts/generate_sparsedata_user.py + agent_page.py. + + Order: bare last name -> first-initial prefix -> numeric suffix appended + to the PREFIXED candidate (agent_page.derive_agent_identity, issue #26 + C1/C2 — the numeric branch used to restart from the bare stem, diverging + from the web path on a third same-initial collision). + """ + base = _slugify_last_name(name) + candidate = base + coll = await db.execute(select(AgentRegistry).where(AgentRegistry.agent_id == candidate)) + if coll.scalar_one_or_none() is None: + return candidate + initial = name.strip()[0].lower() if name.strip() else "x" + prefixed = f"{initial}{base}" + coll = await db.execute(select(AgentRegistry).where(AgentRegistry.agent_id == prefixed)) + if coll.scalar_one_or_none() is None: + return prefixed + for i in range(2, 20): + candidate = f"{prefixed}{i}" + coll = await db.execute( + select(AgentRegistry).where(AgentRegistry.agent_id == candidate) + ) + if coll.scalar_one_or_none() is None: + return candidate + raise RuntimeError(f"Could not find unique agent_id for {name!r}") +``` + `scripts/backfill_agents.py:72-77`, before: +```python +def _bot_name_for(agent_id: str, name: str) -> str: + last = name.strip().split()[-1] + last_alpha = "".join(c for c in last if c.isalpha()) + if agent_id.lower() == last_alpha.lower(): + return f"{last_alpha.capitalize()}Bot" + return f"{agent_id[0].upper()}{last_alpha.capitalize()}Bot" +``` + after: +```python +def _bot_name_for(agent_id: str, name: str) -> str: + last = name.strip().split()[-1] + last_alpha = "".join(c for c in last if c.isalpha()) + # Strip a numeric suffix (issue #26 C2): agent_id[0] of "pwu2" is "p", but + # the "2" belongs before "Bot" (PWu2Bot), matching derive_agent_identity. + stem = "".join(c for c in agent_id if not c.isdigit()) + suffix = agent_id[len(stem):] + if stem.lower() == last_alpha.lower(): + return f"{last_alpha.capitalize()}{suffix}Bot" + return f"{agent_id[0].upper()}{last_alpha.capitalize()}{suffix}Bot" +``` + +- [ ] **Step 4: Run it, expect PASS** + Same command as Step 2. + +- [ ] **Step 5: Run the neighbours** + No existing test file covers `scripts/backfill_agents.py` (`find tests -iname + "*backfill_agents*"` -> none before this task). Sanity-check the script still imports + cleanly: `.venv-test/bin/python -c "import scripts.backfill_agents"`. + +- [ ] **Step 6: Commit** +``` +git add scripts/backfill_agents.py tests/unit/test_backfill_agents.py +git commit -m "fix(scripts): match backfill_agents' numeric-collision fallback to the web path (#26 DOC-C C2)" +``` + +--- + +### Task 26.12: Expose extract_json publicly from llm.py [closes: #26 C4 / DOC-C] + +**Files:** +- Modify: `src/services/llm.py` — `_extract_json` (currently :122-166) +- Modify: `scripts/generate_sparsedata_user.py` — import line (:57) +- Test: `tests/unit/test_llm_service.py` + +**Interfaces:** +- Produces: `extract_json(text: str) -> dict[str, Any]` (public name; `_extract_json` kept as a + module-level alias so `llm.py`'s own internal call sites at :68 and :308 and any other + in-repo caller of the private name keep working unchanged — `scripts/vet_publications.py:34,93` + is a second existing importer of `_extract_json` this task doesn't otherwise touch; the alias + is exactly why it does not need to be migrated in this same commit). + +- [ ] **Step 1: Write the failing test** (add to `tests/unit/test_llm_service.py`) +```python +def test_extract_json_is_public(): + """issue #26 C4: scripts/generate_sparsedata_user.py imported the private + _extract_json because no public alternative existed. extract_json must be + the same function as _extract_json (an alias, not a copy), so the two + names can never drift apart.""" + assert llm.extract_json is llm._extract_json + assert llm.extract_json('{"a": 1}') == {"a": 1} +``` + +- [ ] **Step 2: Run it, expect FAIL** + `.venv-test/bin/python -m pytest tests/unit/test_llm_service.py::test_extract_json_is_public -q -p no:cacheprovider` + Expected failure: `AttributeError: module 'src.services.llm' has no attribute 'extract_json'`. + +- [ ] **Step 3: Implement** + `src/services/llm.py:122`, before: +```python +def _extract_json(text: str) -> dict[str, Any]: + """Extract JSON object from LLM response text.""" +``` + after (rename the function, alias the old name right below it — every internal call site + that spells `_extract_json(...)` keeps working via the alias): +```python +def extract_json(text: str) -> dict[str, Any]: + """Extract JSON object from LLM response text.""" +``` + ...(body unchanged)... + then, immediately after the function's closing line (currently :169, the + `raise ValueError(...)` line and the blank line after it): +```python +_extract_json = extract_json # back-compat alias (#26 C4) — same function, not a copy. +``` + `scripts/generate_sparsedata_user.py:57`, before/after: +```diff +-from src.services.llm import _extract_json, get_anthropic_client ++from src.services.llm import extract_json, get_anthropic_client +``` + and its use at `:500` (currently `_extract_json(...)`), before/after: +```diff +- result = _extract_json(response_text) ++ result = extract_json(response_text) +``` + (confirm the exact call-site text with `grep -n "_extract_json" scripts/generate_sparsedata_user.py` + before editing — line numbers may have drifted from the :500 cited in the issue.) + +- [ ] **Step 4: Run it, expect PASS** + Same command as Step 2. + +- [ ] **Step 5: Run the neighbours** + `.venv-test/bin/python -m pytest tests/unit/test_llm_service.py -q -p no:cacheprovider` + (covers `synthesize_profile`/`make_decision`, both internal callers of the now-aliased + `_extract_json`). Also confirm the script still imports: + `.venv-test/bin/python -c "import scripts.generate_sparsedata_user"`. + `src/agent/simulation.py:5467`'s independent copy of this function is untouched (out of + scope — the issue only asks for a public helper in `llm.py` and its use in + `generate_sparsedata_user.py`; consolidating `simulation.py`'s copy is a separate, larger + change against a file part #20 owns — flagged in "Open decisions" below). + +- [ ] **Step 6: Commit** +``` +git add src/services/llm.py scripts/generate_sparsedata_user.py tests/unit/test_llm_service.py +git commit -m "refactor(llm): expose extract_json publicly, keep _extract_json as an alias (#26 DOC-C C4)" +``` + +--- + +### Task 26.13: build_cabo_sankey.py — clarify the default-date comment [closes: #26 C6 / DOC-C] + +One-line comment change, no behaviour change (`--start` has taken a real argparse value since +`06c7ba5`, pre-baseline — see findings/issue_26.md C6). The ask is purely to stop the comment +reading like an unfinished TODO ("parameterize the date") when `--start` already does that. + +**Files:** +- Modify: `scripts/build_cabo_sankey.py` — comment above `DEFAULT_START` (currently :34-35) +- Test: `tests/unit/test_cabo_sankey_comment.py` (new) + +**Interfaces:** none. + +- [ ] **Step 1: Write the failing test** +```python +"""build_cabo_sankey.py's DEFAULT_START comment must not read like an open +TODO — --start has taken a real argparse value since before issue #26 was +filed (git log -S"add_argument.*--start" -- scripts/build_cabo_sankey.py). +""" + +from pathlib import Path + +SCRIPT = ( + Path(__file__).resolve().parents[2] / "scripts" / "build_cabo_sankey.py" +).read_text() + + +def test_default_start_comment_mentions_the_start_flag(): + line = next( + line for line in SCRIPT.splitlines() if 'DEFAULT_START = "2026-05-01"' in line + ) + idx = SCRIPT.splitlines().index(line) + comment = SCRIPT.splitlines()[idx - 1] + assert "--start" in comment +``` + +- [ ] **Step 2: Run it, expect FAIL** + `.venv-test/bin/python -m pytest tests/unit/test_cabo_sankey_comment.py -q -p no:cacheprovider` + Expected failure: the current comment ("Defaults preserve the original Cabo behavior.") + does not mention `--start`. + +- [ ] **Step 3: Implement** + `scripts/build_cabo_sankey.py:34-35`, before: +```python +# Defaults preserve the original Cabo behavior. +DEFAULT_START = "2026-05-01" +``` + after: +```python +# Default reproduces the original 40-PI Cabo run window; pass --start for any +# other window (see the Schultz example in the module docstring above). This +# is not a hardcoded date awaiting parameterization — --start already does that. +DEFAULT_START = "2026-05-01" +``` + +- [ ] **Step 4: Run it, expect PASS** + Same command as Step 2. + +- [ ] **Step 5: Run the neighbours** + No existing test covers this script. `.venv-test/bin/python -m ruff check scripts/build_cabo_sankey.py` + must stay clean (comment-only change). + +- [ ] **Step 6: Commit** +``` +git add scripts/build_cabo_sankey.py tests/unit/test_cabo_sankey_comment.py +git commit -m "docs(scripts): clarify build_cabo_sankey's default-date comment (#26 DOC-C C6)" +``` + +--- + +## Coverage matrix + +(every row from `findings/issue_26.md`'s summary table, id -> task or reason not planned) + +| id | task | +|---|---| +| A1 | deliberately not planned — by-design behaviour (`_restored_slack_ts`/`_slack_parent_ts` correctly refuse to infer a Slack ts), not a defect. The doc gap it exposes is A3/A4. | +| A2 | deliberately not planned — issue confirms `docs/production-migration.md` §8/preflight-11/`run_migration.sh` are already correct as stated. | +| A3 | 26.7 | +| A4 | resolved as a byproduct of 26.7 (the CLAUDE.md/README paragraph is a second, restart-runbook entry point into the repair regardless of alembic revision) — NOT independently planned. `docs/production-migration.md`'s own "supported starting points" scoping is widened by Part M (M.1 + M.3). | +| A5 | deliberately not planned — owned by part #21 (writer-slot-count fix); `specs/local-db-conversations.md:66-67` is explicitly out of scope for this part. | +| A6 | 26.3 | +| A6-extra (README:87-88 "mounts source", compose `-f` flags) | 26.3 | +| A7 | 26.4 | +| A8 | 26.4 | +| A9 | 26.4 | +| A10 | 26.5 | +| A11 | 26.6 | +| A12 | 26.1 | +| A13 | 26.2 (documented in the spec; prompt files intentionally untouched — no prompt changes in this PR, per binding decision D33) | +| A14 | deliberately not planned — prompt files stay untouched per binding decision D33. Correction (audit #26 Minor 9): 26.2's spec edit says nothing about `prompts/daily_audit.md` or `audit_recipient_list`; the out-of-band consumer is documented by the pre-existing comment at `src/config.py:153-154`. | +| B1 (red-team residual: token rotation not adopted) | 26.8 | +| B2 | 26.8 | +| C1 (numeric fallback) | 26.9 | +| C1 (IntegrityError guard) | 26.10 | +| C2 | 26.11 | +| C3 | deliberately not planned — already FIXED (`02143de`), confirmed by both verification passes. | +| C4 | 26.12 | +| C5 | deliberately not planned — already FIXED (`c46918a`'s `.gitignore` rule); the issue's own history claim was wrong, no code/doc action follows from that correction. | +| C6 | 26.13 | + +## Files this part modifies + +``` +templates/base.html +specs/email-proposal-review.md (doc-only row correction; no prompts/ file touched — D33) +README.md +AGENT.md +specs/tech-stack.md +CLAUDE.md +src/agent/simulation.py (ONLY _sync_roster_from_db, :4513-4677) +src/routers/agent_page.py (ONLY derive_agent_identity :393-414 and + request_agent :418-444) +scripts/backfill_agents.py +src/services/llm.py +scripts/generate_sparsedata_user.py (import line + one call site; and `_resolve_agent_id` numeric fallback per D22, Task 26.11) +scripts/build_cabo_sankey.py (comment only) + +tests/unit/test_base_html_posthog.py (new) +tests/unit/test_prompt_hygiene.py (new) +tests/unit/test_readme_agent_runbook.py (new) +tests/unit/test_agent_md_facts.py (new) +tests/unit/test_lab_count_docs.py (new) +tests/unit/test_tech_stack_spec.py (new) +tests/unit/test_slack_ts_repair_documented.py (new) +tests/unit/test_roster_sync.py +tests/integration/test_agent_page.py +tests/unit/test_backfill_agents.py (new) +tests/unit/test_llm_service.py +tests/unit/test_cabo_sankey_comment.py (new) +``` + +No migration. No compose/nginx/Dockerfile changes (README's compose-command edits are prose +only, not actual compose file edits). + +## Open decisions (for the user, not implementation details) + +1. **`scripts/generate_sparsedata_user.py`'s own `_resolve_agent_id`** (currently :165-183) has the identical bare-stem numeric-branch bug — **RESOLVED (Decision D22): fixed in Task 26.11** alongside `backfill_agents.py`, with the same unit-test shape. + +2. **Three independent JSON-extraction implementations exist**: `src/services/llm.py` + (`extract_json`/`_extract_json` after Task 26.12), `src/agent/simulation.py:5467`, and + (structurally) nothing else calls a fourth copy per the findings. Task 26.12 only exposes + `llm.py`'s version publicly per the issue's exact ask; it does not consolidate + `simulation.py`'s copy (that file is part #20's). Worth a follow-up to point + `simulation.py:5467` at the now-public `extract_json` instead of maintaining a duplicate? + +3. **AGENT.md's Decisions Log entry at :76** (Task 26.4) is corrected in place with an inline + "(Corrected 2026-09: ...)" note rather than left untouched with a new entry appended below + it. The repo has no established convention for amending a historical Decisions Log entry + vs. appending a superseding one — confirm the in-place correction is acceptable, or prefer + appending a new dated entry instead. + +4. **`docs/production-migration.md`'s "Supported starting points: 0018/0019/0020/0021"** (A4) — **RESOLVED by Part M**: M.1 makes 0024 a supported start and M.3 rewrites the title and the scope line; Task 26.7 adds the restart-runbook pointer in CLAUDE.md/README. + + +--- + +# Part 27 — Issue #27 + +# Part #27 — Deploy, CI & coverage gate (I3, I2, I4, I5, I1-residual) + +Branch: `close-issues-20-27` (shared across all parts). Base: `copi-prod` @ 18ba52c. +Sole owner of `nginx/nginx.conf`, `Dockerfile`, `.dockerignore`, `docker-compose*.yml`, +`pyproject.toml`, `scripts/ci.sh` (per the shared file-ownership table — no other part +touches these). Consumes nothing from other parts. Owns no alembic revision id (0025-0028 +are reserved for other parts; nothing here is a schema change). + +Line anchors below are current as of `copi-prod` @ 18ba52c, re-verified against the tree on +2026-09-02 (not copied from the issue's stale numbers — see +`scratchpad/findings/issue_27.md` / `issue_27_redteam.md`, both re-verified same day, 24 +items STILL PRESENT, 0 fixed). **Relocate by content, not line number** — every task in this +part edits one of six small, unowned-elsewhere files, and earlier tasks in this same part +shift later line numbers within them. + +**Task order deliberately deviates from the issue's own priority list** ("I3-hole → I2 → I4 → +I5 → I1-residual"): `Task 27.1` (the `.dockerignore` hole) is still first, and `I2` still +follows it, but `I3`'s Dockerfile-hardening sub-items (layer order, multi-stage) are moved +*after* `I4` (Tasks 27.6-27.8), because the layer-order fix installs from `I4`'s lockfile — +implementing it before the lockfile exists would mean writing it twice. `I5` and the `I1` +residual close the part, matching the issue's order. + +Two test-only Python packages this part relies on are confirmed present in `.venv-test`: +PyYAML 6.0.3 (`import yaml`) and `tomllib` (stdlib, Python 3.12.3). Neither `pathspec`, +`mypy`, `pyright`, `uv` (inside `.venv-test/bin` — the host has a separate `uv` at +`~/.local/bin/uv`, but PLAN_DRAFT's check is specifically `.venv-test/bin`), nor `pip-tools` +is installed; `.venv-test/bin/pip` does not exist either (`ImportError: No module named pip` +via `python -m pip`). Decisions this forces, stated once here so later tasks don't repeat the +reasoning: +- **I4 lockfile tool: pip-tools**, not `uv`, per PLAN_DRAFT's own tie-breaker ("unless `uv` + exists in `.venv-test/bin`" — it does not). +- **I1 type checker: mypy**, not pyright — pure-Python pip package, no Node/npm runtime + needed, and it pairs with the existing pip-tools/ruff toolchain already in `.venv-test`. + Flagged as an open decision below in case the user prefers pyright. +- Package **version numbers quoted below come from `importlib.metadata` inside + `.venv-test`**, not `pip list` (no `pip` executable there): `anthropic==0.117.0`, + `fastapi==0.139.2`, `SQLAlchemy==2.0.51`, `slack_sdk==3.43.0`. + +--- + +### Task 27.1: Close the `.dockerignore` secret-and-bloat hole [closes: #27 I3-b, I3-c, I3-d, I3-e, I3-f, I3-g] + +**First commit of the whole branch** — this is the single most severe finding in the issue +(`findings/issue_27.md` headline: `COPY . .` bakes a 668 MB `backups/` tree including a +plaintext production `.env` — 146 keys, 125 `SLACK_BOT_TOKEN_*`, `POSTGRES_PASSWORD`, +`DATABASE_URL`, `SLACK_CONFIG_TOKEN`/`_REFRESH_TOKEN` — into every app/worker/agent/grantbot +image; the redteam pass confirms this is not latent: a local `copi-blackbird-agent:latest` +image built 2026-08-21 already contains a 6.3 KB `.env` and four prod-era pg dumps, because +the `blackbird` tree has never had a `.dockerignore` at all). + +**Files:** +- Modify: `.dockerignore` (currently 17 lines, full text below) +- Modify: `CLAUDE.md` — the in-container pytest note (`:10-18`) and the `=== Turn 1 ===` example + (`:76`) — see the binding note below (`COORD_A.md` reconciliation item 6) +- Test: `tests/unit/test_dockerignore.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: nothing another task needs (this task's only follower in this part, 27.6, reads + `requirements.lock`/`pyproject.toml`, not this file). + +**Binding note (`COORD_A.md` reconciliation item 6):** this task removes `tests/` from the built image +(it is nowhere in the "Verified NOT to touch" list below, and the new `.dockerignore` entries include +`tests`). `CLAUDE.md`'s "Testing" section (`:10-18`) documents an in-container pytest recipe +(`docker compose exec ... app python -m pytest tests/ -v`) that silently breaks against a prod-built +`app` container once this lands — it would report `file or directory not found: tests/`. That recipe +still works today because it targets the **dev** compose file's `app` service, which bind-mounts the +whole repo at `/app` (`docker-compose.yml:25,37,56,70`) and therefore supplies `tests/` regardless of +what `.dockerignore` excludes from the image. Step 3 below adds a note to `CLAUDE.md` making that +explicit, so the documented command doesn't quietly stop matching reality once this task merges. +Separately (same reconciliation item, unrelated to `.dockerignore`): `CLAUDE.md`'s "Running the Agent +Simulation" section quotes the turn-boundary log line as `=== Turn 1 ===`; the real format, printed by +`src/agent/simulation.py:745` (`logger.info("=== Turn %d: %s ===", turn_count + 1, agent.agent_id)`), +includes the agent id — `=== Turn 1: wu ===`. Fixed in the same commit since it's the same file/section +family and otherwise has no other owning task. + +Current `.dockerignore` (verbatim): +``` +.git +.gitignore +certbot +logs +data +*.log +__pycache__ +**/__pycache__ +*.pyc +.venv +venv +.pytest_cache +.provision_state.json + +# Secrets — never in an image layer (compose injects these at runtime via env_file) +.env +.env.* +``` +Verified NOT to touch: `profiles/`, `prompts/`, `data/` (already excluded — bind-mounted at +runtime, `docker-compose.prod.yml:40-41,71-72,95-98,124-127`), `static/`, `templates/`, +`alembic/`, `alembic.ini`, `scripts/` (used in-container per `CLAUDE.md` — e.g. +`docker compose exec app python scripts/backfill_agent_tokens.py`). `docs/specs/*.md` is +referenced only in code *comments* (`git grep` across `src/` confirms zero `open()`/`Path()` +reads of `docs/`), so it is safe to exclude. + +- [ ] **Step 1: Write the failing test** +```python +"""Static check that .dockerignore actually excludes the paths issue #27 I3 +flagged as baked into every image layer, and does not exclude paths the +running app reads at runtime. + +Docker (moby/patternmatcher) matches each pattern per path *component*, Go +filepath.Match semantics (`*`/`?` never cross `/`); a `**/` prefix means "at +any depth". This reimplements just enough of that to check the patterns in +.dockerignore — mirrors the reference Go program in +scratchpad/findings/issue_27_redteam.md §2 (not shipped in this repo). +""" + +import fnmatch +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _patterns() -> list[str]: + text = (REPO_ROOT / ".dockerignore").read_text() + return [ + line.strip() + for line in text.splitlines() + if line.strip() and not line.strip().startswith("#") + ] + + +def _pattern_matches(pattern: str, path: str) -> bool: + any_depth = pattern.startswith("**/") + pat_parts = pattern[3:].split("/") if any_depth else pattern.split("/") + path_parts = path.split("/") + width = len(pat_parts) + starts = range(len(path_parts)) if any_depth else [0] + for start in starts: + window = path_parts[start : start + width] + if len(window) != width: + continue + if all(fnmatch.fnmatchcase(p, pp) for p, pp in zip(window, pat_parts, strict=False)): + return True + return False + + +def _is_excluded(path: str) -> bool: + return any(_pattern_matches(p, path) for p in _patterns()) + + +# The I3 baked-secret/bloat paths this task closes (findings/issue_27.md I3-b..g, +# redteam NEW-1/NEW-2). +MUST_EXCLUDE = [ + "backups/prod-sync-20260810/env.prod", + "backups/prod-sync-20260810/copi_prod_20260810.dump", + ".venv-test/bin/python", + "tests/unit/test_x.py", + ".notes/cohort-system-v2.md", + "mutants/1/src/foo.py", + "build/lib/foo.py", + ".hypothesis/examples/x", + ".coverage", + ".mutmut-cache", + ".playwright-mcp/page.png", + "copi.egg-info/PKG-INFO", + ".superpowers/state.json", + "docs/specs/2026-08-05-hub-bot-customization-design.md", + ".env.local", + "backups/x/.env", # nested dotfile — belt-and-suspenders via **/.env* +] + +# Paths the running app/worker/agent/grantbot reads from the tree at runtime — +# must stay reachable in the image (deploy_dossier.md §1 "Image contents"). +# NOTE: data/ and logs/ are deliberately NOT in this list — both are excluded +# from the image (pre-existing .dockerignore lines) and bind-mounted at runtime +# (docker-compose.prod.yml:95-98,124-127). The image never needs their contents, +# so "excluded from the image" is correct behaviour for them, not a bug. +MUST_NOT_EXCLUDE = [ + "src/main.py", + "prompts/profile-synthesis.md", + "profiles/public/x.md", + "static/app.css", + "templates/base.html", + "alembic/env.py", + "alembic.ini", + "scripts/build_cabo_sankey.py", + "requirements.lock", + "pyproject.toml", +] + + +def test_dockerignore_excludes_the_i3_baked_paths(): + for path in MUST_EXCLUDE: + assert _is_excluded(path), f"{path} should be excluded by .dockerignore but is not" + + +def test_dockerignore_does_not_exclude_runtime_paths(): + for path in MUST_NOT_EXCLUDE: + assert not _is_excluded(path), f"{path} is excluded by .dockerignore but is read at runtime" +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerignore.py -q -p no:cacheprovider +``` +Expected: `test_dockerignore_excludes_the_i3_baked_paths` fails on the first entry — +`AssertionError: backups/prod-sync-20260810/env.prod should be excluded by .dockerignore but is not` +(current file has no `backups` pattern, and the root-anchored `.env`/`.env.*` patterns don't +match a nested, undotted `env.prod`). `test_dockerignore_does_not_exclude_runtime_paths` already +passes against the current tree (verified) — `data/agent_roster.json` is deliberately not in +`MUST_NOT_EXCLUDE` (see the note above it), so this task does not need to, and must not, touch the +pre-existing `data` line to make this test pass. + +- [ ] **Step 3: Implement** +``` +# .dockerignore — append after the existing "Secrets" block + +# Bloat + secrets that COPY . . would otherwise bake into every image layer +# (#27 I3). backups/ is the sharp one: it holds full production pg_dumps AND, +# as of 2026-08-10, a plaintext prod .env (125 SLACK_BOT_TOKEN_*, +# POSTGRES_PASSWORD, SLACK_CONFIG_TOKEN/_REFRESH_TOKEN) that the .env/.env.* +# patterns above do not catch — it has no leading dot and sits two +# directories down. See scratchpad findings/issue_27.md I3-b. +backups +.venv-test +.venv* +tests +.notes +mutants +build +.hypothesis +.ruff_cache +.coverage +.mutmut-cache +.playwright-mcp +copi.egg-info +.superpowers +**/.env* +docs/specs +``` + +`CLAUDE.md` — after the existing pytest code block (currently `:15-18`), before the "The named +database must already exist" paragraph (currently `:20`): +```markdown +**Note (2026-09, #27 I3):** the command above works against the **dev** compose file, whose +`.:/app` bind mount supplies `tests/`. `tests/` is excluded from the built image by +`.dockerignore`, so the same command against a prod-built container reports +"file or directory not found: tests/". Run the suite on the host (`./scripts/ci.sh`) or with the +dev compose file. +``` +`CLAUDE.md` — the turn-boundary example (currently `:76`): +```diff +-`=== Turn 1 ===`. Repeated `attempt 1/3` (never `2/3`) means each call 429s once then ++`=== Turn 1: ===`. Repeated `attempt 1/3` (never `2/3`) means each call 429s once then +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerignore.py -q -p no:cacheprovider +``` +(Verified: both tests pass — `MUST_EXCLUDE` paths are now all covered by the appended patterns, +`ruff check --select E,F,I,UP,B --ignore E501` on the test file is clean with `zip(..., strict=False)`.) + +- [ ] **Step 5: Run the neighbours** +No existing test references `.dockerignore` or `Dockerfile` (`git grep -l dockerignore -- +tests/` → no matches, confirmed in both the first-pass and red-team findings), so there is +nothing to invert. `CLAUDE.md` has no test coverage (it is documentation) — nothing to run. + +- [ ] **Step 6: Commit** +``` +git add .dockerignore CLAUDE.md tests/unit/test_dockerignore.py +git commit -m "fix(deploy): close the .dockerignore hole that baked backups/ (incl. a plaintext prod .env) into every image (#27 I3)" +``` + +**Deploy note:** +1. **Cherry-pick this exact commit onto the `blackbird` branch** (it exists locally and at + `origin/blackbird`) — that tree has *no* `.dockerignore` at all + (`findings/issue_27_redteam.md` NEW-1), so today it bakes `.git`, `.env`, and multiple + prod-era pg dumps into every blackbird image. If a clean cherry-pick fails because the + branches have diverged, `git checkout blackbird -- .dockerignore`-style copy of this exact + file content is an acceptable substitute. +2. **Rotate every credential in `backups/prod-sync-20260810/env.prod`** if there is any chance + an image was pushed to a registry or copied off this host — a local `copi-blackbird-agent` + image built 2026-08-21 already has a 6.3 KB `.env` baked into a layer on this machine, + proof the leak is not merely theoretical for that tree. +3. Rebuild all four images once so the fixed `.dockerignore` actually takes effect (a + `.dockerignore` change invalidates the `COPY . .` layer's cache, but do this explicitly to + also purge any already-built stale layer from the local Docker cache): + ```bash + C="-f docker-compose.prod.yml -f docker-compose.override.yml" + docker compose $C build --no-cache app worker grantbot + docker compose $C --profile agent build --no-cache agent + ``` + +--- + +### Task 27.2: `/api/health` probes the database, returns 503 when it can't [closes: #27 I2-a] + +**Files:** +- Modify: `src/main.py` — imports (`:6`, `:8`); `health` route inside `create_app` (currently + `:150-153`) +- Test: `tests/unit/test_health_route_unit.py` (create — named `..._unit.py`, not `test_health_ + route.py`, so it doesn't share a basename with the existing `tests/integration/test_health_ + route.py`; both dirs have `__init__.py` so a collision was never a real import hazard, this is + purely to remove any doubt for a future reader grepping by filename) + +**Interfaces:** +- Consumes: `get_session_factory` (`src/database.py:36-44`, unchanged signature — this task + reads it the same way `AgentBadgeMiddleware.dispatch` already does at `src/main.py:36-37`, + which is exactly why `tests/conftest.py:120-121`'s `monkeypatch.setattr("src.main.get_session_factory", ...)` pattern applies here too). +- Produces: no new symbol; only the response shape of `GET /api/health` on DB failure changes + (200 → 503). + +**Cross-part note (binding — `COORD_A.md` reconciliation item 16):** Task 25.4 makes +`AgentBadgeMiddleware.dispatch` short-circuit `/api/health` before it can run any badge-count +query. This task then makes the `health` route ITSELF open a DB session for its own `SELECT 1` +probe. 25.4's test asserts "no *badge* query ran", not "no session was opened" — it uses a +recording session fake and checks which tables were touched, specifically so this task's session +open doesn't make it start failing. After implementing Step 3, re-run +`tests/unit/test_agent_badge_middleware.py` in full and confirm it is still green (both tests — +`/static/` and `/api/health` — must still report zero badge-table queries). + +Current route (`src/main.py:150-153`): +```python + @application.get("/api/health") + async def health(): + """Health check endpoint.""" + return {"status": "ok"} +``` +Current imports (`src/main.py:6`, `:8`): +```python +from fastapi import FastAPI, Request +... +from sqlalchemy import func, select +``` + +- [ ] **Step 1: Write the failing test** +```python +"""Unit-tier coverage for the DB probe on /api/health (#27 I2). No real +Postgres — get_session_factory is monkeypatched on src.main, mirroring the +badge_factory override tests/conftest.py's `client` fixture already does +(tests/conftest.py:120-121: monkeypatch.setattr("src.main.get_session_factory", ...)).""" + +import httpx +from httpx import ASGITransport + +from src.main import create_app + + +class _FakeSession: + """Async-context-manager stand-in for AsyncSession — enough surface for + the health route's `async with session_factory() as db: await db.execute(...)`.""" + + def __init__(self, *, fail: bool): + self._fail = fail + + async def __aenter__(self): + return self + + async def __aexit__(self, exc_type, exc, tb): + return False + + async def execute(self, *args, **kwargs): + if self._fail: + raise ConnectionRefusedError("db unreachable") + + +async def _get_health(monkeypatch, *, fail: bool): + monkeypatch.setattr("src.main.get_session_factory", lambda: (lambda: _FakeSession(fail=fail))) + app = create_app() + transport = ASGITransport(app=app) + async with httpx.AsyncClient(transport=transport, base_url="http://testserver") as client: + return await client.get("/api/health") + + +async def test_health_ok_when_db_probe_succeeds(monkeypatch): + r = await _get_health(monkeypatch, fail=False) + assert r.status_code == 200 + assert r.json() == {"status": "ok"} + + +async def test_health_503_when_db_probe_fails(monkeypatch): + r = await _get_health(monkeypatch, fail=True) + assert r.status_code == 503 +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_health_route_unit.py -q -p no:cacheprovider +``` +Expected failure (the `fail=True` case): `AssertionError: assert 200 == 503` — the current +route takes no dependency and always returns `{"status": "ok"}`. + +- [ ] **Step 3: Implement** +```python +# src/main.py — imports, before/after + +# BEFORE (:6): +from fastapi import FastAPI, Request +# AFTER: +from fastapi import FastAPI, HTTPException, Request + +# BEFORE (:8): +from sqlalchemy import func, select +# AFTER: +from sqlalchemy import func, select, text +``` +```python +# src/main.py — health route (:150-153), before/after + +# BEFORE: + @application.get("/api/health") + async def health(): + """Health check endpoint.""" + return {"status": "ok"} + +# AFTER: + @application.get("/api/health") + async def health(): + """Health check endpoint. Probes the DB so a broken schema or a + downed Postgres is never reported healthy (#27 I2 — on 2026-07-30 + this route returned 200 while every ORM read of agent_messages + raised UndefinedColumnError, and nginx's depends_on: service_healthy + let traffic through regardless).""" + try: + session_factory = get_session_factory() + async with session_factory() as db: + await db.execute(text("SELECT 1")) + except Exception as exc: + logger.warning("Health check DB probe failed: %s", exc) + raise HTTPException(status_code=503, detail="database unavailable") from exc + return {"status": "ok"} +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_health_route_unit.py -q -p no:cacheprovider +``` + +- [ ] **Step 5: Run the neighbours** +``` +.venv-test/bin/python -m pytest tests/integration/test_health_route.py -q -p no:cacheprovider # needs Docker/TEST_DATABASE_URL +.venv-test/bin/python -m pytest tests/unit/test_reachability.py -q -p no:cacheprovider +.venv-test/bin/python -m pytest tests/unit/test_agent_badge_middleware.py -q -p no:cacheprovider +``` +`tests/integration/test_health_route.py::test_health_ok` uses the `client` fixture, whose +`get_session_factory` override already points at a real (rolled-back) session — stays 200, +unaffected. `tests/unit/test_reachability.py`'s `ROUTE_ALLOWLIST[("GET", "/api/health")]` entry +and the `>50 routes` walker check the route table, not its body — path/method/dependant shape +(zero `Depends()`) is unchanged, so both stay green. No existing test pins the old +unconditional-200 behavior. `tests/unit/test_agent_badge_middleware.py` (Task 25.4) is the +mandatory cross-part check per the note above — it must still report zero badge-table queries +for `/api/health` now that the route opens its own session. + +- [ ] **Step 6: Commit** +``` +git add src/main.py tests/unit/test_health_route_unit.py +git commit -m "fix(deploy): /api/health probes the DB and returns 503 when it can't connect (#27 I2)" +``` + +**Deploy note:** after this deploys, the compose healthcheck (`docker-compose.prod.yml:45-50`) +and nginx's `depends_on: app: condition: service_healthy` (`:162-164`) now genuinely gate on +DB reachability. If `app` shows unhealthy for longer than before during a migration window +(Task 27.3), that is this fix working as intended, not a regression — do not raise +`start_period` reflexively. + +--- + +### Task 27.3: A `migrate` one-shot service gates `app`/`worker`/`grantbot` on `alembic upgrade head` [closes: #27 I2-d] + +**Files:** +- Modify: `docker-compose.prod.yml` — add `migrate` service; add `depends_on.migrate` to + `app` (`:42-44`), `worker` (`:73-75`), `grantbot` (`:128-130`) +- Modify: `docker-compose.override.yml` — add a `migrate: logging: driver: json-file` entry + (**required**, see below — without it this task causes a full outage on the first + `docker compose up` after deploy) +- Test: `tests/unit/test_deploy_compose.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: the `migrate` service block, extended (mem/cpus) by Task 27.13. + +`migrate` shares `app`'s `build: context: .` — it is `docker compose`'s second built image from the +same Dockerfile (tagged separately, e.g. `copi-python-migrate`), fully layer-cached against `app`'s +build since both share every layer up to the final `CMD`. Worth knowing before the first +`docker compose $C up -d --build app worker` after this lands: the operator will see TWO images +build (or one build plus a fast cache-hit "image already built" for the second), not just `app`'s — +expected, not a sign anything is wrong. + +**Binding note (`COORD_A.md` reconciliation item 13):** `docker-compose.override.yml` forces the +`json-file` log driver on every service in `docker-compose.prod.yml` because the EC2 instance role +(`copi-ec2-ses-role`) lacks `logs:CreateLogStream` — a service present in the prod file but missing +from the override dies at start with `AccessDeniedException`. This task's new `migrate` service sets +`logging.driver: awslogs` (matching every other service in `docker-compose.prod.yml` before the +override is applied), so it MUST get its own override entry in the same commit. Skipping this is not a +cosmetic gap: because this task also adds `depends_on: migrate: condition: service_completed_ +successfully` to `app`/`worker`/`grantbot`, a `migrate` that dies at start makes all three refuse to +start too ("dependency failed to start") — i.e. a missing override entry here is a **full outage** on +the first `docker compose up` after this deploys, not a logging inconvenience. + +- [ ] **Step 1: Write the failing test** +```python +"""Static assertions over docker-compose.prod.yml and docker-compose.override.yml. +No compose CLI, no Docker daemon — parsed as plain YAML; ${VAR:-default} +interpolation syntax is just a string to PyYAML, so this needs no substitution.""" + +from pathlib import Path + +import yaml + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _prod_compose() -> dict: + return yaml.safe_load((REPO_ROOT / "docker-compose.prod.yml").read_text()) + + +def _override_compose() -> dict: + return yaml.safe_load((REPO_ROOT / "docker-compose.override.yml").read_text()) + + +def test_migrate_service_runs_alembic_upgrade_head_and_does_not_restart(): + svc = _prod_compose()["services"]["migrate"] + assert svc["command"] == ["python", "-m", "alembic", "upgrade", "head"] + assert svc["restart"] == "no" + assert svc["depends_on"]["postgres"]["condition"] == "service_healthy" + + +def test_app_worker_grantbot_wait_for_migrate_to_complete(): + services = _prod_compose()["services"] + for name in ("app", "worker", "grantbot"): + dep = services[name]["depends_on"] + assert dep["migrate"]["condition"] == "service_completed_successfully", name + assert dep["postgres"]["condition"] == "service_healthy", name + + +def test_every_prod_service_including_migrate_has_the_json_file_log_override(): + # CLAUDE.md "Compose file set": docker-compose.override.yml forces json-file + # logging because the EC2 role lacks logs:CreateLogStream; a service missing + # from it dies at start with AccessDeniedException. A new prod-only service + # (like this task's `migrate`) is invisible to that protection unless it is + # added to the override in the SAME commit. + prod = set(_prod_compose()["services"]) + override = set(_override_compose()["services"]) + assert prod <= override, f"no json-file logging override for {sorted(prod - override)}" +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_deploy_compose.py -q -p no:cacheprovider +``` +Expected: `KeyError: 'migrate'` on the first two tests (no such service exists yet), and +`test_every_prod_service_including_migrate_has_the_json_file_log_override` FAILS once the Step 3 +`docker-compose.prod.yml` edit lands ahead of the override edit, with +`AssertionError: no json-file logging override for ['migrate']` — do the `docker-compose.prod.yml` edit +and observe this red state before adding the override entry, to prove the test actually catches the +outage this task would otherwise ship. + +- [ ] **Step 3: Implement** +```yaml +# docker-compose.prod.yml — new service, inserted after `postgres` (:23) and +# before `app` (:25) + + migrate: + build: + context: . + restart: "no" + command: ["python", "-m", "alembic", "upgrade", "head"] + env_file: .env + environment: + DATABASE_URL: postgresql+asyncpg://${POSTGRES_USER:-copi}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB:-copi} + SECRET_KEY: ${SECRET_KEY:?Set a strong random SECRET_KEY in .env} + ENVIRONMENT: ${ENVIRONMENT:-production} + # Same knob run_migration.sh already reads (alembic/env.py:66) — a + # routine `alembic upgrade head` here should never need to wait long, + # but give it the same override surface as the gated runbook. + ALEMBIC_LOCK_TIMEOUT_MS: ${ALEMBIC_LOCK_TIMEOUT_MS:-10000} + depends_on: + postgres: + condition: service_healthy + logging: + driver: awslogs + options: + awslogs-group: /copi/migrate + tag: migrate + awslogs-create-group: "true" + awslogs-region: ${AWS_REGION:-us-east-2} +``` +```yaml +# docker-compose.prod.yml — app depends_on (:42-44), before/after + +# BEFORE: + depends_on: + postgres: + condition: service_healthy +# AFTER: + depends_on: + postgres: + condition: service_healthy + migrate: + condition: service_completed_successfully +``` +Apply the identical `depends_on` diff to `worker` (`:73-75`) and `grantbot` (`:128-130`). +`agent` (`:99-101`) is deliberately left unchanged — it is started later via +`docker compose --profile agent run`, well after `migrate` has already completed as part of +`up -d --build app worker`, and PLAN_DRAFT's own scope for this fix names only +app/worker/grantbot. + +```yaml +# docker-compose.override.yml — append (REQUIRED, see the binding note above). +# Every other entry in this file follows the identical two-line shape. + migrate: + logging: + driver: json-file +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_deploy_compose.py -q -p no:cacheprovider +``` + +- [ ] **Step 5: Run the neighbours** +No existing test parses `docker-compose.prod.yml` or `docker-compose.override.yml` (first such tests +in the repo). Nothing to invert. Task 27.13 extends this same file/test module later — re-run +`tests/unit/test_deploy_compose.py` in full at that point; the `<=` shape of +`test_every_prod_service_including_migrate_has_the_json_file_log_override` means it stays correct even +if 27.13 (or a later part) adds more services to either file, as long as every prod service still has +an override entry. + +- [ ] **Step 6: Commit** +``` +git add docker-compose.prod.yml docker-compose.override.yml tests/unit/test_deploy_compose.py +git commit -m "feat(deploy): gate app/worker/grantbot on a migrate one-shot running alembic upgrade head (#27 I2)" +``` + +**Deploy note:** this changes what `docker compose $C up -d --build app worker` does — it now +also builds and runs `migrate` to completion first. **`scripts/migrate/run_migration.sh` is +unchanged and still required for any migration that needs its preflight/backup/postflight +gate** (large tables, data backfills, anything not safe to apply unattended); the existing +runbook order in `docs/production-migration.md` already puts "migrate DB" before "deploy +code", so by the time this `migrate` service's own `alembic upgrade head` runs, a properly +rehearsed deploy has already applied the risky migration by hand and this step is a no-op +(alembic upgrading to a revision it's already at does nothing). Three additional operational notes: +(a) `docker compose $C run --rm --no-deps app ...` (used elsewhere in the runbook, e.g. Part R.6/M.2) +is unaffected — `--no-deps` skips the `depends_on` gate entirely; (b) `docker compose $C --profile +agent run agent ...` is unaffected because `agent`'s `depends_on` is untouched by this task; (c) on a +host reboot, the Docker daemon's `unless-stopped` restart policy restarts `app`/`worker`/`grantbot` +directly and does **not** re-consult `depends_on`, so an already-exited `migrate` container from a +prior run cannot deadlock the reboot path. **Open decision:** this makes +"routine" migrations fully automatic on every `docker compose up`, which is a real change in +operational risk posture for any migration nobody explicitly rehearsed first — flagged below, +not resolved unilaterally here. + +--- + +### Task 27.4: Hash-pinned lockfile, upper caps on `fastapi`/`sqlalchemy`/`anthropic`/`slack-sdk`, move `plotly` out of the runtime install [closes: #27 I4-a (partial, scoped to the 4 named packages), I4-c, I4-d, I4-e] + +**Files:** +- Create: `requirements.lock` (generated by pip-compile, not hand-written) +- Modify: `pyproject.toml` — `dependencies` (currently `:11-29`); add a `scripts` key to the + EXISTING `[project.optional-dependencies]` table (`:31`, already has a `dev` key at `:32-54`) +- Test: `tests/unit/test_dependencies_lock.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: `requirements.lock`, consumed by Task 27.6's Dockerfile layer-order fix. + +**`pyproject.toml` already has exactly one `[project.optional-dependencies]` table (`:31-80`, +containing `dev`).** TOML forbids declaring the same table twice — `tomllib` raises +`TOMLDecodeError: Cannot declare ('project', 'optional-dependencies') twice` if this task adds a +second `[project.optional-dependencies]` header, which breaks `pip install .`, the Dockerfile build, +and this task's own test. The new `scripts` key goes INSIDE the existing table, as a sibling of `dev` +(order inside a table doesn't matter to TOML; putting `scripts` first here is just for readability). + +Current `dependencies` (`pyproject.toml:11-29`, exact): +```toml +dependencies = [ + "fastapi>=0.111.0", + "uvicorn[standard]>=0.29.0", + "jinja2>=3.1.4", + "python-multipart>=0.0.9", + "authlib>=1.3.0", + "httpx>=0.27.0", + "sqlalchemy[asyncio]>=2.0.30", + "asyncpg>=0.29.0", + "alembic>=1.13.0", + "anthropic>=0.26.0", + "slack-sdk>=3.27.0", + "pydantic-settings>=2.2.0", + "itsdangerous>=2.2.0", + "boto3>=1.34.0", + "typer>=0.12.0", + "rich>=13.7.0", + "plotly>=5.20.0", +] +``` +Installed versions today in `.venv-test` (via `importlib.metadata`, since `pip` is not +installed there): `fastapi==0.139.2`, `sqlalchemy==2.0.51`, `anthropic==0.117.0`, +`slack_sdk==3.43.0`. `anthropic` and `fastapi` are both pre-1.0 (any minor can break per +semver's own 0.x rule); `slack-sdk`/`sqlalchemy` are capped at their current major only. All four +caps are satisfiable against what's actually installed (`fastapi 0.139.2<1.0.0` ✓, `sqlalchemy +2.0.51<3.0.0` ✓, `anthropic 0.117.0<1.0.0` ✓, `slack_sdk 3.43.0<4.0.0` ✓). +`plotly`'s only importer anywhere in the tree is `scripts/build_cabo_sankey.py:29,128` +(`git grep -c plotly -- src/ templates/` → 0) — it ships in all four +`build: context: .` images (`docker-compose.prod.yml:26,60,85,113`) for a script nobody runs +from inside a container in the normal path. + +- [ ] **Step 1: Write the failing test** +```python +"""Static checks for #27 I4: upper caps on version-sensitive deps, plotly out +of the runtime install, and a lockfile that exists and names every runtime +dependency. Does not attempt to re-validate pip-compile's own hash pinning — +that's pip's job at install time (--require-hashes, wired in Task 27.6).""" + +import re +import tomllib +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _pyproject() -> dict: + return tomllib.loads((REPO_ROOT / "pyproject.toml").read_text()) + + +def _dep_names(deps: list[str]) -> set[str]: + return {re.split(r"[><=\[!~]", d, maxsplit=1)[0].strip() for d in deps} + + +def test_version_sensitive_deps_have_upper_caps(): + deps = { + re.split(r"[><=\[!~]", d, maxsplit=1)[0].strip(): d + for d in _pyproject()["project"]["dependencies"] + } + for name in ("fastapi", "sqlalchemy", "anthropic", "slack-sdk"): + assert "<" in deps[name], f"{name} has no upper cap: {deps[name]!r}" + + +def test_plotly_is_a_scripts_extra_not_a_runtime_dependency(): + proj = _pyproject()["project"] + assert "plotly" not in _dep_names(proj["dependencies"]) + assert any(d.startswith("plotly") for d in proj["optional-dependencies"]["scripts"]) + + +def test_lockfile_exists_and_covers_every_runtime_dependency(): + lock_path = REPO_ROOT / "requirements.lock" + assert lock_path.exists(), "requirements.lock is missing — run pip-compile" + lock_text = lock_path.read_text() + for name in _dep_names(_pyproject()["project"]["dependencies"]): + # pip-compile may emit either the '-' or '_' spelling of a PEP 503 + # name (e.g. slack-sdk vs slack_sdk); accept both rather than + # pinning to whichever one happened to come out of a given run. + escaped = re.escape(name).replace("\\-", "[-_]") + pattern = rf"(?im)^{escaped}==" + assert re.search(pattern, lock_text), f"{name} not pinned in requirements.lock" +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_dependencies_lock.py -q -p no:cacheprovider +``` +Expected: `test_version_sensitive_deps_have_upper_caps` fails first — +`AssertionError: fastapi has no upper cap: 'fastapi>=0.111.0'`. + +- [ ] **Step 3: Implement** +```toml +# pyproject.toml — dependencies (:11-29), before/after + +# BEFORE: +dependencies = [ + "fastapi>=0.111.0", + "uvicorn[standard]>=0.29.0", + "jinja2>=3.1.4", + "python-multipart>=0.0.9", + "authlib>=1.3.0", + "httpx>=0.27.0", + "sqlalchemy[asyncio]>=2.0.30", + "asyncpg>=0.29.0", + "alembic>=1.13.0", + "anthropic>=0.26.0", + "slack-sdk>=3.27.0", + "pydantic-settings>=2.2.0", + "itsdangerous>=2.2.0", + "boto3>=1.34.0", + "typer>=0.12.0", + "rich>=13.7.0", + "plotly>=5.20.0", +] + +# AFTER: +dependencies = [ + "fastapi>=0.111.0,<1.0.0", + "uvicorn[standard]>=0.29.0", + "jinja2>=3.1.4", + "python-multipart>=0.0.9", + "authlib>=1.3.0", + "httpx>=0.27.0", + "sqlalchemy[asyncio]>=2.0.30,<3.0.0", + "asyncpg>=0.29.0", + "alembic>=1.13.0", + "anthropic>=0.26.0,<1.0.0", + "slack-sdk>=3.27.0,<4.0.0", + "pydantic-settings>=2.2.0", + "itsdangerous>=2.2.0", + "boto3>=1.34.0", + "typer>=0.12.0", + "rich>=13.7.0", +] +``` +```toml +# pyproject.toml — [project.optional-dependencies] (:31), before/after. +# Do NOT add a second [project.optional-dependencies] header — one already +# exists here with a `dev` key. Add `scripts` INSIDE it, above `dev`: + +# BEFORE: +[project.optional-dependencies] +dev = [ + "pytest>=8.2.0", + ... + +# AFTER: +[project.optional-dependencies] +scripts = [ + # Sole importer: scripts/build_cabo_sankey.py (#27 I4). Not needed by + # app/worker/agent/grantbot — moved out of the runtime install so it + # stops shipping in all four images for a script nobody runs from + # inside a container in the normal path. + "plotly>=5.20.0", +] +dev = [ + "pytest>=8.2.0", + ... +``` +(The rest of `dev`'s existing 11 entries, `:33-53`, and the table's closing `]` at `:54`, are +untouched — Task 27.14 adds `mypy` to `dev` later, in the same table.) + +Generate the lockfile (this is the one step in this task that must be run with real tooling — +hand-writing hashes would be wrong; `--no-header` is required so the committed file's contents +match what Task 27.5's staleness check regenerates and compares against — a header records the +exact `--output-file` path it was invoked with, which differs between this one-time generation and +27.5's throwaway temp-file regeneration on every `ci.sh` run): +```bash +uv pip install --python .venv-test/bin/python pip-tools # one-time, pip-tools isn't installed yet +.venv-test/bin/python -m piptools compile --generate-hashes --no-header \ + -o requirements.lock pyproject.toml +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_dependencies_lock.py -q -p no:cacheprovider +``` +(Verified DB-free: applied this exact `pyproject.toml` diff to a scratch copy — `tomllib.loads` +parses cleanly, `scripts == ["plotly>=5.20.0"]`, `dev` keeps its other 11 entries, `plotly` is absent +from `dependencies`; ran the corrected test module against the scratch file plus a synthetic +`requirements.lock` containing `slack_sdk==3.43.0` (underscore spelling) — all 3 tests pass, proving +the dash/underscore-tolerant regex in `test_lockfile_exists_and_covers_every_runtime_dependency` +works. `ruff check --select E,F,I,UP,B --ignore E501` on the test file: clean, 0 findings — in +particular `re.split(..., maxsplit=1)` avoids ruff's **B034**, which fires on the positional form.) + +- [ ] **Step 5: Run the neighbours** +No existing test reads `pyproject.toml`'s dependency list. Nothing to invert. + +- [ ] **Step 6: Commit** +``` +git add pyproject.toml requirements.lock tests/unit/test_dependencies_lock.py +git commit -m "fix(deploy): hash-pinned lockfile, cap fastapi/sqlalchemy/anthropic/slack-sdk, move plotly out of the runtime install (#27 I4)" +``` + +**Deploy note:** +1. If `scripts/build_cabo_sankey.py` is ever run again, `plotly` is no longer preinstalled: + `docker compose exec app pip install 'plotly>=5.20.0'` inside the running container first + (its own docstring already documents the `docker cp` + `exec` workflow this fits into). +2. Rebuild once `Task 27.6` also lands (the Dockerfile doesn't consume `requirements.lock` + until then) — no separate rebuild needed for this commit alone. + +--- + +### Task 27.5: `ci.sh` fails the gate when `requirements.lock` is stale [closes: #27 I4-e (staleness enforcement)] + +**Files:** +- Modify: `scripts/ci.sh` — header "Steps:" list (`:7-20`) and "Overridable env" comment + (`:25-28`); new step inserted after the ruff src ratchet (ends `:290`) and before the pytest + step (`:292`); prerequisite check near `:90-95` +- Test: `tests/unit/test_dependencies_lock.py` (modify) + +**Interfaces:** +- Consumes: `requirements.lock` (Task 27.4). +- Produces: nothing another task needs. + +**`pip-compile`'s own header defeats a raw file diff.** `pip-compile` writes a header recording the +exact command it was invoked with, including the `--output-file` path. The committed lock was +generated with `--output-file requirements.lock` (Task 27.4); this step regenerates into a scratch +temp file to compare against it, so a plain `diff -q "$LOCK_TMP" requirements.lock` compares two files +whose headers can NEVER match — even when every pin is byte-identical — and the gate would fail on +every single run, for everyone (reproduced: two lockfiles differing only in the recorded +`--output-file` path fail a raw `diff -q` but are identical once comment lines are stripped). Fix: +compare pins only (`grep -v '^#'` on both sides before diffing), and pass `--no-header` when +(re)generating so the committed file and every future regeneration agree regardless. + +- [ ] **Step 1: Write the failing test** +Mirrors the existing precedent for pinning `ci.sh` content by substring — +`tests/unit/test_cohort_isolation.py::TestMigrationHygiene::test_ci_script_gates_on_alembic_before_running_tests` +already does exactly this for the alembic checks. +```python +# tests/unit/test_dependencies_lock.py — add to the bottom of the file + +def _ci_sh() -> str: + return (REPO_ROOT / "scripts" / "ci.sh").read_text() + + +def test_ci_sh_fails_the_gate_when_the_lockfile_is_stale(): + text = _ci_sh() + assert "piptools compile" in text + assert "requirements.lock is stale" in text + assert text.index("piptools compile") < text.index("-m pytest") +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_dependencies_lock.py::test_ci_sh_fails_the_gate_when_the_lockfile_is_stale -q -p no:cacheprovider +``` +Expected: `AssertionError: assert 'piptools compile' in text`. + +- [ ] **Step 3: Implement** +```bash +# scripts/ci.sh — prerequisite check, added right after the existing venv-python +# check (:90-95), before the Docker-reachability check (:97-101) + +if ! "$VENV_PY" -c 'import piptools' >/dev/null 2>&1; then + echo "ERROR: pip-tools not installed in ${VENV_PY}'s environment." >&2 + echo "Install it with: uv pip install --python $VENV_PY pip-tools" >&2 + exit 1 +fi +``` +```bash +# scripts/ci.sh — new step, inserted after the existing ruff-src-ratchet block +# (ends at :290 with `echo " ${src_findings} findings (ceiling ${SRC_LINT_MAX})"`) +# and before the pytest step (:292) + +echo "==> lockfile freshness (requirements.lock matches pyproject.toml)" +# Set LOCKCHECK=none to skip (offline, or a different interpreter than the lock +# was cut with — pip-compile's resolution is Python-version/platform-sensitive). +if [ "${LOCKCHECK:-}" = "none" ]; then + echo " lockfile check skipped (LOCKCHECK=none)" +else + # A stale lock looks pinned but has silently drifted from what pyproject.toml + # actually asks for. Regenerate into a scratch file and diff — pip-compile is + # deterministic given the same pyproject.toml + resolver state. --no-header + # is required on BOTH sides of the diff: pip-compile's header records the + # --output-file path it was invoked with, so a raw diff between the + # committed file and a scratch temp file can never match even when every + # pin is identical (verified) — compare pins only, belt-and-suspenders with + # --no-header on the regeneration too. + LOCK_TMP="$(mktemp)" + if ! "$VENV_PY" -m piptools compile --generate-hashes --no-header \ + --output-file "$LOCK_TMP" pyproject.toml >/dev/null 2>&1; then + echo "ERROR: pip-compile failed to resolve pyproject.toml — see above." >&2 + rm -f "$LOCK_TMP" + exit 1 + fi + if ! diff -q <(grep -v '^#' requirements.lock) <(grep -v '^#' "$LOCK_TMP") >/dev/null; then + echo "ERROR: requirements.lock is stale — pyproject.toml changed without regenerating it." >&2 + echo "Run: $VENV_PY -m piptools compile --generate-hashes --no-header -o requirements.lock pyproject.toml" >&2 + diff <(grep -v '^#' requirements.lock) <(grep -v '^#' "$LOCK_TMP") >&2 || true + rm -f "$LOCK_TMP" + exit 1 + fi + rm -f "$LOCK_TMP" + echo " requirements.lock is current" +fi +``` +``` +# scripts/ci.sh — header "Steps:" list (:7-20), before/after + +# BEFORE (last two entries): +# 4. ruff lint of src/ against a CEILING (SRC_LINT_MAX) rather than zero. src/ +# carries pre-existing style debt, so this is a ratchet: it blocks NEW debt +# without demanding the old debt be paid first. +# 5. Full pytest run — unit + integration + characterization + contract — with +# branch coverage over src/, failing under COV_MIN (a ratchet floor: raise it as +# coverage grows, never lower it). + +# AFTER: +# 4. ruff lint of src/ against a CEILING (SRC_LINT_MAX) rather than zero. src/ +# carries pre-existing style debt, so this is a ratchet: it blocks NEW debt +# without demanding the old debt be paid first. +# 5. requirements.lock freshness: regenerate with pip-compile and diff against the +# committed lock (pins only — pip-compile's own header would never match +# otherwise), so a pyproject.toml edit can't silently drift from what +# actually gets installed (#27 I4). Set LOCKCHECK=none to skip. +# 6. Full pytest run — unit + integration + characterization + contract — with +# branch coverage over src/, failing under COV_MIN (a ratchet floor: raise it as +# coverage grows, never lower it). +``` +``` +# scripts/ci.sh — "Overridable env" comment (:25-28), before/after + +# BEFORE: +# Overridable env: VENV_PY (python interpreter), COV_MIN (coverage floor %), +# SRC_LINT_MAX (src/ lint ceiling), CI_MIGRATION_DB (round-trip DSN, or `none` to +# skip the round trip), MIGCHECK_PORT (host port for the throwaway Postgres), +# MIGRATION_FLOOR (the revision the round trip downgrades to). + +# AFTER: +# Overridable env: VENV_PY (python interpreter), COV_MIN (coverage floor %), +# SRC_LINT_MAX (src/ lint ceiling), CI_MIGRATION_DB (round-trip DSN, or `none` to +# skip the round trip), MIGCHECK_PORT (host port for the throwaway Postgres), +# MIGRATION_FLOOR (the revision the round trip downgrades to), LOCKCHECK (set to +# `none` to skip the requirements.lock freshness check — offline, or a different +# interpreter than the lock was cut with). +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_dependencies_lock.py -q -p no:cacheprovider +``` +(Verified DB-free, the diff mechanism only: two synthetic lockfiles differing solely in a +`pip-compile`-style header's `--output-file` path — `diff -q` on the raw files fails, `diff -q` on +`grep -v '^#'`-filtered copies of both succeeds. pip-tools itself is not installed in `.venv-test` and +installing it needs network per the task's own constraints, so the full `ci.sh` step was not executed +end-to-end; the mechanism it depends on was verified in isolation.) + +- [ ] **Step 5: Run the neighbours** +``` +.venv-test/bin/python -m pytest tests/unit/test_cohort_isolation.py::TestMigrationHygiene::test_ci_script_gates_on_alembic_before_running_tests -q -p no:cacheprovider +``` +That test only asserts `"alembic heads"` and `"uniq -d"` both precede `-m pytest`'s position in +the file — both are far earlier (`:109-127`), and inserting content between the ruff step and +`pytest` only pushes `-m pytest`'s index further down, so the ordering assertion still holds. + +- [ ] **Step 6: Commit** +``` +git add scripts/ci.sh tests/unit/test_dependencies_lock.py +git commit -m "ci: fail the gate when requirements.lock drifts from pyproject.toml (#27 I4)" +``` + +**Deploy note:** none — `ci.sh` runs only on developer machines via the `pre-push` hook and is +never invoked on the prod host. + +--- + +### Task 27.6: Dockerfile installs deps from `requirements.lock` before `src/` is copied [closes: #27 I3-j] + +**Files:** +- Modify: `Dockerfile` — dependency-install block (currently `:11-14`) +- Test: `tests/unit/test_dockerfile_build.py` (create) + +**Interfaces:** +- Consumes: `requirements.lock` (Task 27.4). +- Produces: the deps-first Dockerfile shape Tasks 27.7 and 27.8 build on. + +Current (`Dockerfile:11-14`): +```dockerfile +# Install Python dependencies +COPY pyproject.toml . +COPY src/ src/ +RUN pip install --no-cache-dir . +``` + +- [ ] **Step 1: Write the failing test** +```python +"""Static Dockerfile structure checks for #27 I3 (layer order, multi-stage, +non-root — Tasks 27.6/27.7/27.8). No `docker build` here: these assert the +*text* is ordered/shaped correctly. Real image-build verification is a manual +step noted in each task's Deploy note (mirrors nginx's `nginx -t` check in +Task 27.12).""" + +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _dockerfile() -> str: + return (REPO_ROOT / "Dockerfile").read_text() + + +def test_dependencies_install_from_the_lockfile_before_source_is_copied(): + text = _dockerfile() + assert "requirements.lock" in text, "Dockerfile must install from the hash-pinned lockfile" + assert "--require-hashes" in text + lock_copy = text.index("requirements.lock") + pip_install_lock = text.index("--require-hashes") + src_copy = text.index("COPY src/ src/") + assert lock_copy < pip_install_lock < src_copy, ( + "deps must install from requirements.lock BEFORE src/ is copied, so a " + "source-only change doesn't bust the dependency layer" + ) +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py -q -p no:cacheprovider +``` +Expected: `ValueError: substring not found` — `text.index("requirements.lock")` raises, since +the current Dockerfile has no reference to it. + +- [ ] **Step 3: Implement** +```dockerfile +# Dockerfile — dependency-install block (:11-14), before/after + +# BEFORE: +# Install Python dependencies +COPY pyproject.toml . +COPY src/ src/ +RUN pip install --no-cache-dir . + +# AFTER: +# Install Python dependencies from the hash-pinned lockfile first — this layer +# only invalidates when requirements.lock changes, not on every src/ edit (#27 I4). +COPY pyproject.toml requirements.lock ./ +RUN pip install --no-cache-dir --require-hashes -r requirements.lock +COPY src/ src/ +RUN pip install --no-cache-dir --no-deps . +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py -q -p no:cacheprovider +``` + +- [ ] **Step 5: Run the neighbours** +No existing test references `Dockerfile` (`git grep -l Dockerfile -- tests/` → no matches). +Nothing to invert. + +- [ ] **Step 6: Commit** +``` +git add Dockerfile tests/unit/test_dockerfile_build.py +git commit -m "fix(deploy): install from requirements.lock before copying src/, so source edits don't bust the deps layer (#27 I3, I4)" +``` + +**Deploy note:** manual verification (not part of `ci.sh`, needs a Docker daemon): +```bash +docker build -t copi-layer-check . +touch src/main.py && docker build -t copi-layer-check . 2>&1 | grep -A1 "pip install --no-cache-dir --require-hashes" +``` +The second build's lockfile-install line should show `CACHED`. Requires Task 27.4's +`requirements.lock` to already be committed. + +--- + +### Task 27.7: Multi-stage build — builder (gcc/libpq-dev) → runtime (libpq5 only) [closes: #27 I3-i] + +**Files:** +- Modify: `Dockerfile` (whole file, now shaped by Task 27.6) +- Test: `tests/unit/test_dockerfile_build.py` (modify) + +**Interfaces:** +- Consumes: Task 27.6's deps-first layer order. +- Produces: the `builder`/`runtime` stage split Task 27.8 adds `USER` to. + +- [ ] **Step 1: Write the failing test** +```python +def test_two_stage_build_with_a_slim_runtime(): + text = _dockerfile() + assert text.count("FROM python:3.11-slim") == 2, "expected a builder stage and a runtime stage" + assert "AS builder" in text + assert "--from=builder" in text + runtime_section = text[text.rindex("FROM python:3.11-slim"):] + assert "gcc" not in runtime_section + assert "libpq-dev" not in runtime_section + assert "libpq5" in runtime_section +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py -q -p no:cacheprovider +``` +Expected: `AssertionError: assert 1 == 2` — the Dockerfile is still single-stage. + +- [ ] **Step 3: Implement** +```dockerfile +# Dockerfile — full file, before/after (state after Task 27.6) + +# BEFORE: +FROM python:3.11-slim + +WORKDIR /app + +# Install build dependencies +RUN apt-get update && apt-get install -y --no-install-recommends \ + gcc \ + libpq-dev \ + && rm -rf /var/lib/apt/lists/* + +# Install Python dependencies from the hash-pinned lockfile first — this layer +# only invalidates when requirements.lock changes, not on every src/ edit (#27 I4). +COPY pyproject.toml requirements.lock ./ +RUN pip install --no-cache-dir --require-hashes -r requirements.lock +COPY src/ src/ +RUN pip install --no-cache-dir --no-deps . + +# Copy source +COPY . . + +# Create directories for profiles and prompts +RUN mkdir -p profiles/public profiles/private prompts logs static + +EXPOSE 8000 + +CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] + +# AFTER: +FROM python:3.11-slim AS builder + +WORKDIR /app + +# Build-time only: gcc/libpq-dev compile any dependency that ships as an sdist +# for this platform/Python combination. Not present in the runtime image +# below (#27 I3 — asyncpg itself needs none of this, it bundles its own wire +# protocol implementation rather than linking libpq). +RUN apt-get update && apt-get install -y --no-install-recommends \ + gcc \ + libpq-dev \ + && rm -rf /var/lib/apt/lists/* + +COPY pyproject.toml requirements.lock ./ +RUN pip install --no-cache-dir --require-hashes -r requirements.lock +COPY src/ src/ +RUN pip install --no-cache-dir --no-deps . + +FROM python:3.11-slim AS runtime + +WORKDIR /app + +# libpq5 only: the runtime client library a compiled wheel may dlopen. No +# compiler, no -dev headers, no build toolchain of any kind in this stage +# (#27 I3). Deliberately avoids naming the builder-stage packages here — the +# structural test in tests/unit/test_dockerfile_build.py asserts their names +# are absent from this section. +RUN apt-get update && apt-get install -y --no-install-recommends \ + libpq5 \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages +COPY --from=builder /usr/local/bin /usr/local/bin +COPY . . + +RUN mkdir -p profiles/public profiles/private prompts logs static + +EXPOSE 8000 + +CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py -q -p no:cacheprovider +``` + +- [ ] **Step 5: Run the neighbours** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py::test_dependencies_install_from_the_lockfile_before_source_is_copied -q -p no:cacheprovider +``` +Still passes — `text.index(...)` finds the FIRST occurrence of each marker, which is inside +the (still deps-first-ordered) `builder` stage. + +- [ ] **Step 6: Commit** +``` +git add Dockerfile tests/unit/test_dockerfile_build.py +git commit -m "fix(deploy): multi-stage build — gcc/libpq-dev stay in the builder, runtime ships libpq5 only (#27 I3)" +``` + +**Deploy note:** manual verification: +```bash +docker build -t copi-multistage . +docker run --rm copi-multistage python -c "import asyncpg, sqlalchemy, fastapi" +docker history copi-multistage --no-trunc | grep -i gcc # expect: no output +``` +Rebuild all four images before deploying: +```bash +C="-f docker-compose.prod.yml -f docker-compose.override.yml" +docker compose $C build --no-cache app worker grantbot +docker compose $C --profile agent build --no-cache agent +``` + +--- + +### Task 27.8: Runtime stage drops to a non-root fixed UID (10001) [closes: #27 I3-h] + +**Files:** +- Modify: `Dockerfile` (runtime stage, from Task 27.7) +- Test: `tests/unit/test_dockerfile_build.py` (modify) + +**Interfaces:** +- Consumes: Task 27.7's `runtime` stage. +- Produces: nothing another task needs. + +- [ ] **Step 1: Write the failing test** +```python +def test_runtime_stage_drops_to_a_non_root_fixed_uid(): + text = _dockerfile() + assert "USER 10001" in text + assert "useradd" in text and "10001" in text + user_idx = text.rindex("USER 10001") + copy_idx = text.rindex("COPY . .") + assert copy_idx < user_idx, "USER must be set after the app tree is copied in" +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py -q -p no:cacheprovider +``` +Expected: `ValueError: substring not found` — `text.rindex("USER 10001")` raises, no `USER` +directive exists yet. + +- [ ] **Step 3: Implement** +```dockerfile +# Dockerfile — runtime stage tail, before/after (state after Task 27.7) + +# BEFORE: +COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages +COPY --from=builder /usr/local/bin /usr/local/bin +COPY . . + +RUN mkdir -p profiles/public profiles/private prompts logs static + +EXPOSE 8000 + +CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] + +# AFTER: +COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages +COPY --from=builder /usr/local/bin /usr/local/bin +COPY . . + +# Fixed UID so it matches whatever the prod host chowns the bind-mounted +# profiles/data trees to (see this task's Deploy note) — a plain chown +# target on the host, not a real host account. +RUN useradd --uid 10001 --no-create-home --shell /usr/sbin/nologin copi \ + && mkdir -p profiles/public profiles/private prompts logs static \ + && chown -R 10001:10001 /app + +USER 10001 + +EXPOSE 8000 + +CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"] +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py -q -p no:cacheprovider +``` + +- [ ] **Step 5: Run the neighbours** +``` +.venv-test/bin/python -m pytest tests/unit/test_dockerfile_build.py -q -p no:cacheprovider +``` +Run the whole file — Task 27.6's and 27.7's tests must stay green (neither asserts anything +about `USER`, so this is unaffected). + +- [ ] **Step 6: Commit** +``` +git add Dockerfile tests/unit/test_dockerfile_build.py +git commit -m "fix(deploy): run app/worker/agent/grantbot as a fixed non-root UID (#27 I3)" +``` + +**Deploy note:** the bind-mounted `profiles/` and `data/` trees on the prod host are +root-owned today (verified locally by the coordinator). **Before recreating any container from +this image**, on the prod host: +```bash +sudo mkdir -p /home/ubuntu/copi-python/data +sudo chown -R 10001:10001 /home/ubuntu/copi-python/profiles /home/ubuntu/copi-python/data +# Do NOT chown prompts/ — it is git-tracked (13 files) and read-only at runtime; chowning it makes +# the next `git pull` fail with "unable to unlink old 'prompts/…'" (Dockerfile-review ruling; R.5 / R.12 note 15). +``` +Do this before `docker compose $C up -d --build app worker` — otherwise UID 10001 hits +`PermissionError` the instant it reads or writes a profile. This also covers `agent` and +`grantbot` (same host directories). Apply the identical chown before rebuilding blackbird's +stack at `/home/ubuntu/blackbird-copi-science` once this change reaches that branch. + +**Dev-workflow note (this task's `needs-fix` finding):** `docker-compose.yml` (the DEV file) +builds from this same `Dockerfile` and bind-mounts the checkout at `.:/app` on all four +services (`:25,37,56,70`). Once the image's default user is UID 10001, a dev container +started with a checkout owned by the developer's own UID (typically 1000/1001, not 10001) +cannot write into the bind-mounted tree — `alembic upgrade`, log files, and anything else the +app writes under `/app` at runtime will hit `PermissionError`. This is a real, immediate +regression to the documented dev workflow (`docker compose up`, no prod compose files +involved), not just a prod hardening step. Do not silently rely on "it happens to work"; +pick one before merging: +- Add `user: root` under each service in `docker-compose.yml` (dev only — the prod `USER + 10001` in the image is untouched, since `docker-compose.yml` is a separate compose file and + its `user:` key overrides the image default only for containers started from that file), or +- `chown -R 10001:10001` the checkout on the host before running `docker compose up` (fragile — + breaks again after every `git clean`/fresh clone). +This plan does not pick one unilaterally — it is a workflow choice, not an implementation +detail (added to "Open decisions" below). `docker-compose.yml` is already Part 27's file per +the shared file-ownership table, so whichever option the user picks is a one-line follow-up in +this same task, not a new one. + +--- + +### Task 27.9: Fix the stale "Next.js" comments and remove the dead `/_next/static/` cache block [closes: #27 I5-a, I5-d] + +**Files:** +- Modify: `nginx/nginx.conf` — comments at `:14`, `:37`, `:69`, `:127`, `:166`; dead block + `:166-171` +- Test: `tests/unit/test_nginx_config.py` (create) + +**Interfaces:** +- Consumes: nothing. +- Produces: `_server_blocks`/`_https_block` test helpers, consumed by Tasks 27.10-27.12. + +- [ ] **Step 1: Write the failing test** +```python +"""Static text/structure checks over nginx/nginx.conf — no nginx binary +needed for these. The full `nginx -t` syntax check is a manual verification +step, noted in Task 27.12's Deploy note once all nginx tasks have landed.""" + +from pathlib import Path + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _nginx_conf() -> str: + return (REPO_ROOT / "nginx" / "nginx.conf").read_text() + + +def _server_blocks(text: str) -> list[str]: + """Split nginx.conf into individual `server { ... }` blocks by brace depth. + Used by Tasks 27.10-27.12 to check one vhost at a time.""" + blocks = [] + i = 0 + while True: + i = text.find("server {", i) + if i == -1: + break + depth, j = 0, i + while True: + if text[j] == "{": + depth += 1 + elif text[j] == "}": + depth -= 1 + if depth == 0: + break + j += 1 + blocks.append(text[i : j + 1]) + i = j + 1 + return blocks + + +def _https_block(text: str, server_name: str) -> str: + for block in _server_blocks(text): + if f"server_name {server_name};" in block and "listen 443 ssl" in block: + return block + raise AssertionError(f"no HTTPS server block for {server_name!r}") + + +def test_no_stale_nextjs_comments(): + text = _nginx_conf() + assert "Next.js" not in text, "nginx.conf still describes the backend as Next.js" + + +def test_dead_next_static_cache_block_is_removed(): + text = _nginx_conf() + assert "/_next/static/" not in text + assert "proxy_cache_valid" not in text, "proxy_cache_valid with no proxy_cache_path is inert" +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_nginx_config.py -q -p no:cacheprovider +``` +Expected: `AssertionError: nginx.conf still describes the backend as Next.js` (5 hits today: +`:14,37,69,127,166`). + +- [ ] **Step 3: Implement** +``` +# nginx/nginx.conf — 5 comment fixes, exact before/after + +# :14 BEFORE: # - Reverse proxy to Next.js app on port 3000 +# AFTER: # - Reverse proxy to the FastAPI app (uvicorn) on port 8000 + +# :37 BEFORE: # Upstream definition for the Next.js app +# AFTER: # Upstream definition for the FastAPI app + +# :69 BEFORE: # HTTPS server — reverse proxy to Next.js app +# AFTER: # HTTPS server — reverse proxy to the FastAPI app + +# :127 BEFORE: # Reverse proxy to Next.js app +# AFTER: # Reverse proxy to the FastAPI app +``` +``` +# nginx/nginx.conf — dead block removed entirely (:166-171) + +# BEFORE (immediately before the closing '}' of the ${DOMAIN} HTTPS server): + # Cache Next.js static assets (hashed filenames, safe to cache forever) + location /_next/static/ { + proxy_pass http://app; + proxy_cache_valid 200 365d; + add_header Cache-Control "public, max-age=31536000, immutable"; + } +} + +# AFTER (block deleted, closing brace of the server left in place): +} +``` + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_nginx_config.py -q -p no:cacheprovider +``` + +- [ ] **Step 5: Run the neighbours** +No existing test parses `nginx/nginx.conf` (first such test). Nothing to invert. Tasks +27.10-27.12 extend this same file/test module. + +- [ ] **Step 6: Commit** +``` +git add nginx/nginx.conf tests/unit/test_nginx_config.py +git commit -m "fix(deploy): correct stale Next.js comments, remove the dead /_next/static/ cache block (#27 I5)" +``` + +**Deploy note:** `docker compose $C up -d nginx` to pick this up — the bind-mounted template +is only re-rendered by envsubst at container start, not by `nginx -s reload`. Recreating +`nginx` alone (without also recreating `app`) does not trigger the documented +stale-upstream-IP 502 (`nginx-stale-upstream-ip-after-app-recreate.md`) — that only happens +when `app` gets a new bridge IP. + +--- + +### Task 27.10: Rate-limit the devel/blackbird vhosts; restore blackbird's TLS hardening [closes: #27 I5-g] + +**Files:** +- Modify: `nginx/nginx.conf` — devel HTTPS server (content-anchored: `server_name + devel.copi.science;` + `listen 443 ssl`), blackbird HTTPS server (`server_name + blackbird.copi.science;` + `listen 443 ssl`) +- Test: `tests/unit/test_nginx_config.py` (modify) + +**Interfaces:** +- Consumes: `_https_block` (Task 27.9). +- Produces: nothing another task needs. + +- [ ] **Step 1: Write the failing test** +```python +def test_devel_and_blackbird_https_vhosts_are_rate_limited(): + text = _nginx_conf() + for name in ("devel.copi.science", "blackbird.copi.science"): + block = _https_block(text, name) + assert "limit_conn conn_perip" in block, f"{name} has no connection cap" + assert "limit_req zone=req_general" in block, f"{name} has no request-rate cap" + + +def test_blackbird_https_vhost_has_the_same_tls_hardening_as_the_others(): + block = _https_block(_nginx_conf(), "blackbird.copi.science") + for directive in ("ssl_ciphers", "ssl_stapling on", "ssl_stapling_verify on", "resolver "): + assert directive in block, f"blackbird vhost is missing {directive!r}" +``` + +- [ ] **Step 2: Run it, expect FAIL** +``` +.venv-test/bin/python -m pytest tests/unit/test_nginx_config.py -q -p no:cacheprovider +``` +Expected: `AssertionError: devel.copi.science has no connection cap`. + +- [ ] **Step 3: Implement** +``` +# nginx/nginx.conf — devel HTTPS server, before/after + +# BEFORE: + client_max_body_size 10m; + + location / { + proxy_pass http://devel_app; + proxy_http_version 1.1; + +# AFTER: + client_max_body_size 10m; + + # Per-IP connection cap for the whole vhost (#27 I5 — SEC-15 zones existed + # but were only ever wired into the ${DOMAIN} server). + limit_conn conn_perip 30; + + location / { + # Broad per-IP request cap; burst absorbs normal page loads. + limit_req zone=req_general burst=40 nodelay; + + proxy_pass http://devel_app; + proxy_http_version 1.1; +``` +``` +# nginx/nginx.conf — blackbird HTTPS server, before/after + +# BEFORE: + ssl_protocols TLSv1.2 TLSv1.3; + ssl_prefer_server_ciphers off; + ssl_session_timeout 1d; + ssl_session_cache shared:SSL:10m; + ssl_session_tickets off; + + add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always; + ... + client_max_body_size 10m; + + location / { + proxy_pass http://blackbird_app; + proxy_http_version 1.1; + +# AFTER: + ssl_protocols TLSv1.2 TLSv1.3; + ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384; + ssl_prefer_server_ciphers off; + ssl_session_timeout 1d; + ssl_session_cache shared:SSL:10m; + ssl_session_tickets off; + + ssl_stapling on; + ssl_stapling_verify on; + resolver 8.8.8.8 8.8.4.4 valid=300s; + resolver_timeout 5s; + + add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always; + ... + client_max_body_size 10m; + + limit_conn conn_perip 30; + + location / { + limit_req zone=req_general burst=40 nodelay; + + proxy_pass http://blackbird_app; + proxy_http_version 1.1; +``` +(`ssl_ciphers`/`ssl_stapling*`/`resolver*` lines are copied verbatim from the `${DOMAIN}` +server, `nginx.conf:83,91-95`, which the primary and devel vhosts already carry — blackbird +was the only one missing them.) + +- [ ] **Step 4: Run it, expect PASS** +``` +.venv-test/bin/python -m pytest tests/unit/test_nginx_config.py -q -p no:cacheprovider +``` + +- [ ] **Step 5: Run the neighbours** +``` +.venv-test/bin/python -m pytest tests/unit/test_nginx_config.py::test_no_stale_nextjs_comments tests/unit/test_nginx_config.py::test_dead_next_static_cache_block_is_removed -q -p no:cacheprovider +``` + +- [ ] **Step 6: Commit** +``` +git add nginx/nginx.conf tests/unit/test_nginx_config.py +git commit -m "fix(deploy): rate-limit devel/blackbird vhosts, restore blackbird's TLS hardening (#27 I5)" +``` + +**Deploy note:** `docker compose $C up -d nginx`. No cert/DNS change needed — blackbird's +existing cert already supports OCSP stapling (same CA chain as the primary domain). + +--- + +### Task 27.11: Add `Content-Security-Policy-Report-Only` to all three vhosts (decision: Report-Only, not enforcing) [closes: #27 I5-c] + +**Files:** +- Modify: `nginx/nginx.conf` — 3 security-header blocks (immediately after each vhost's + `Referrer-Policy` `add_header` line) +- Test: `tests/unit/test_nginx_config.py` (modify) + +**Interfaces:** +- Consumes: `_https_block` (Task 27.9). +- Produces: nothing another task needs. + +**Decision baked into this task** (per PLAN_DRAFT guidance): `templates/base.html` loads +Tailwind from a CDN (`