From 32fc1b4c9f794381f6e8317017ddb3afdebcd5f4 Mon Sep 17 00:00:00 2001 From: Static Date: Tue, 11 Aug 2026 11:04:34 -0400 Subject: [PATCH 01/65] docs: design HumWatch 2.1 fleet hub --- ...026-08-11-humwatch-2.1-fleet-hub-design.md | 542 ++++++++++++++++++ 1 file changed, 542 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md new file mode 100644 index 0000000..3651c0f --- /dev/null +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -0,0 +1,542 @@ +# HumWatch 2.1.0 Fleet Hub Design + +**Status:** Approved by Jeff on 2026-08-11 +**Target release:** 2.1.0 +**Scope:** Replace the fragmented multi-machine browser flow with one movable fleet hub, one owner login, and one-time agent enrollment. + +## Outcome + +HumWatch 2.1.0 has one normal entry point: + +```text +http://:9100 +``` + +The owner signs in with a username and password, then selects any enrolled PC inside the same dashboard. Ordinary LAN access works without Tailscale. Tailscale access works when available. HTTPS is optional. + +Tokens are not part of normal browser use. A short-lived pairing code is used only when a PC or future integration joins the hub. Each enrolled participant receives its own cryptographic identity and can be revoked without affecting the rest of the fleet. + +The hub role is movable. Every HumWatch installation is capable of becoming the hub through the web application. + +## Why 2.0.x regressed + +The 2.0 security model protected the network surface, but it split the user experience into separate browser origins and manual deployment work: + +* The browser bearer token lives in `sessionStorage`, which is scoped to one origin and browser session. +* The Machines page opens another PC's dashboard in a new tab instead of switching the active machine inside one application. +* Each installer creates a random token, while multi-machine discovery expects every machine to share the same token. +* Browser access to peers depends on CORS origin lists. +* Peer discovery requires certificate trust that the default per-machine self-signed certificates do not provide across LAN and Tailscale addresses. +* Machine lists are stored in each browser rather than owned by the fleet. + +A live deployment check found multiple healthy 2.0.2 agents reachable through Tailscale, while discovery still returned no peers because their certificates and credentials did not form a usable fleet trust model. + +The fix is architectural. More token documentation or certificate setup would preserve the regression. + +## Product decisions + +1. One installation package supports unconfigured, agent, and hub states. +2. One hub owns the dashboard, owner authentication, fleet registry, and central history. +3. The browser communicates only with the hub during normal use. +4. Agents collect data, keep a local rolling buffer, and report to the hub. +5. One owner account is supported in 2.1.0. +6. HTTP over a trusted private LAN is the default. +7. Tailscale is detected and supported, but never required. +8. HTTPS is optional. +9. Enrollment codes are short-lived and single use. +10. Every agent has an independent device identity. +11. The hub can move to another enrolled PC. +12. A printable recovery key supports recovery after hub loss. +13. The fleet API is a stable boundary for a future SlipShell integration. + +## System states + +### Unconfigured + +First run offers two actions: + +* **Create Fleet** makes this PC the hub. +* **Join Fleet** keeps this PC as an agent and shows its address plus a pairing code. + +An upgraded 2.0.x installation enters a migration version of this screen after its existing data is secured. + +### Hub + +The hub performs five jobs: + +* Serves the fleet dashboard. +* Authenticates the owner and manages browser sessions. +* Owns the machine registry and fleet settings. +* Accepts signed reports and history backfill from agents. +* Stores central fleet history. + +The hub also runs the collector locally. Its hardware appears as an ordinary machine in the fleet. + +### Agent + +An agent performs four jobs: + +* Collects local hardware data. +* Stores a rolling local history buffer. +* Sends live reports and ordered backfill to the hub. +* Serves a small local setup and recovery page. + +An agent does not serve the full fleet dashboard during normal operation. + +## Component boundaries + +The implementation must separate these responsibilities even if they continue to run in one Python process: + +### Collector + +Owns sensor discovery, sampling, normalization, and the local measurement stream. It does not know about browser sessions or fleet presentation. + +### Local history buffer + +Persists samples before hub acknowledgement. It owns sequence allocation, retention, and backfill reads. The default retention remains seven days unless a later product decision changes it. + +### Agent transport + +Packages measurements into signed, ordered batches. It owns reconnect behavior, backoff, acknowledgement handling, and hub announcements. + +### Fleet registry + +Owns fleet identity, machine identity, display names, enrollment status, revocation state, hub addresses, and hub generation. + +### Hub ingestion + +Verifies agent identity, rejects replayed batches, stores accepted samples, and advances each machine's acknowledgement watermark. + +### Owner authentication + +Owns password verification, browser sessions, recovery, login throttling, logout, and session revocation. It does not authenticate agents. + +### Fleet web API + +Exposes the stable `/api/v2` browser and integration contract. It queries the fleet registry and central history through service boundaries rather than reading their tables directly. + +### Local setup surface + +Owns first run, pairing code display, migration choices, and recovery actions. It remains small and literal. + +## Identity model + +### Fleet identity + +Creating a fleet generates: + +* A random fleet UUID. +* A fleet signing key pair. +* A recovery encryption key pair. +* Hub generation `1`. +* A printable recovery key that encodes the recovery private key and is shown once to the owner. + +Agents retain the fleet public signing key and recovery public key. Only the active hub holds the fleet private signing key in usable form. HumWatch does not retain the recovery private key. + +### Machine identity + +Each installation generates a stable random machine UUID, an Ed25519 signing key pair, and an X25519 encryption key pair. The private keys stay on that PC in protected service-owned files. Enrollment registers the machine UUID and both public keys with the hub. + +Agent report envelopes include: + +* Fleet UUID. +* Machine UUID. +* Monotonic sequence range. +* Batch identifier. +* Capture timestamps. +* Payload digest. +* Agent signature. + +The hub verifies the signature, fleet membership, revocation state, sequence range, and batch identifier before accepting data. + +Signatures authenticate data and hub movement. They do not hide telemetry from someone who can observe an ordinary HTTP LAN connection. HTTPS or Tailscale supplies confidentiality when required. + +### Owner identity + +2.1.0 supports one owner account with a username and password. The password is stored with Python's `scrypt`, a random salt, and versioned work parameters. Parameters are calibrated for supported hardware and stored with the hash so they can be upgraded after a successful login. + +No recoverable password is stored. + +### Browser sessions + +Successful login creates a random server-side session. The browser receives an `HttpOnly`, `SameSite=Strict` cookie. Sessions default to 30 days and can be revoked individually or all at once. + +The cookie gains the `Secure` attribute when HTTPS is enabled. State-changing browser requests require CSRF protection. Login responses use generic failure text and rate limiting by account plus source address. + +Hub transfer and dead hub recovery revoke every browser session. + +### Integration identity + +Future integrations use their own revocable, read-only identity. They never receive the owner's password, browser cookie, agent private key, or fleet signing key. + +## Agent enrollment + +1. An unconfigured agent generates a random single-use pairing code. +2. The setup page or installer on that PC shows the agent address and pairing code locally. +3. The code expires after ten minutes and after one successful use. +4. The owner opens **Add PC** on the hub and enters the address plus code. +5. The hub confirms the agent's machine UUID and public key. +6. The agent confirms the fleet UUID, fleet public key, hub addresses, and signed enrollment record. +7. The hub adds the machine to the fleet registry. +8. The agent sends its current state and available buffered history. +9. The hub marks enrollment complete only after a signed report is stored. + +Pairing attempts are rate limited. The agent must show whether a code is active, expired, or already used without disclosing fleet secrets. + +The pairing code is never returned by a network API. The unconfigured agent exposes its enrollment endpoint on selected private listeners, while code display remains local to the target PC. + +Plain HTTP enrollment assumes a trusted LAN. The setup screen must state that limitation. Tailscale or optional HTTPS protects the exchange on networks that are not trusted. + +Removing a machine revokes its public key and rejects later reports. Re-enrollment creates a new enrollment record without changing other machines. + +## Network behavior + +### Default access + +The hub presents ordinary LAN and Tailscale addresses as IP plus port. The primary display format is: + +```text +http://192.168.x.x:9100 +http://100.x.x.x:9100 +``` + +The first run screen lists reachable private addresses and lets the owner choose which listeners to enable. + +### Listener policy + +Fresh installs enable localhost plus selected private LAN listeners. A detected Tailscale address is offered as another private listener. HumWatch must not silently expose the hub through a public address. + +Custom listeners remain available for advanced setups and carry a clear warning when they are not private, loopback, or Tailscale ranges. + +### HTTP and HTTPS + +HTTP is supported and is the default for trusted LAN use. The UI must not imply that HTTP encrypts credentials or telemetry. + +HTTPS is an optional setting. Existing operator-provided certificate support remains available. Tailscale certificate or Serve integration can be documented as an optional path, but it is not part of the required setup. + +### Discovery + +Discovery helps find candidates. It does not establish trust. + +HumWatch can use LAN discovery and Tailscale peer information to suggest addresses, but a machine joins only through its active pairing code. Browser CORS is not part of discovery because the browser never calls agents directly. + +Enrolled agents also use discovery to recover from a hub address change. A candidate hub exposes a signed announcement containing the fleet UUID, hub machine UUID, addresses, and generation. Agents verify that announcement with the fleet public key before changing destinations. The new hub actively sends the same announcement to discovered enrolled agents. Manual hub address entry remains available on the local recovery page. + +## Data flow + +### Live reporting + +1. The collector emits normalized measurements. +2. The local buffer commits the sample and allocates its sequence number. +3. Agent transport sends a signed batch to the active hub. +4. Hub ingestion verifies and stores the batch in one transaction. +5. The hub returns the highest contiguous stored sequence. +6. The agent advances its acknowledgement watermark. +7. The dashboard receives the update through the hub event stream. + +### Offline buffering and backfill + +If the hub is unavailable, collection and local storage continue. Reconnect uses exponential backoff with jitter. After reconnect, the agent uploads missing ranges oldest first while continuing to send current data. + +Batch identifiers and sequence ranges make retries idempotent. The hub may accept duplicated transport attempts, but it must store each sample once. + +If the offline interval exceeds local retention, the agent reports the exact missing range. The hub displays a data gap rather than inventing continuity. + +### Central history + +The hub stores measurements under stable machine UUIDs. Hostname, IP, and display name changes do not create a new machine. + +Central history keeps the existing seven-day default in 2.1.0. Changing product retention is outside this redesign. + +The existing per-machine database is imported into the local buffer during migration. After enrollment, that history uploads through the same ordered backfill path as new data. + +## Dashboard behavior + +### Login + +The hub address opens a normal username and password form. There is no shared fleet token field. + +### Fleet overview + +The landing page shows every enrolled machine with: + +* Display name and current hostname. +* Online, offline, enrolling, revoked, or migration status. +* Last report time. +* CPU load and temperature when available. +* GPU temperature when available. +* Memory use. +* Battery state when available. +* Data gap or backfill status. + +### Machine selection + +A persistent machine selector changes the active machine inside the existing dashboard shell. CPU, GPU, memory, disk, network, battery, process, and history pages read the selected machine UUID from application state. + +Selecting a PC does not navigate the browser to that PC and does not open another tab. + +### Settings + +Settings gains these sections: + +* Owner and sessions. +* Fleet and enrolled machines. +* Add PC. +* Move Hub. +* Recovery key status. +* Network listeners. +* Optional HTTPS. +* Integrations. +* Legacy compatibility. + +## Hub transfer + +### Planned transfer + +The owner chooses **Settings > Fleet > Move Hub** and selects an enrolled target. + +Preflight verifies: + +* Target is online and not revoked. +* Target runs a compatible HumWatch version. +* Target has enough free storage. +* Target can listen on at least one selected private address. +* The current database can produce a consistent snapshot. + +Transfer then: + +1. Pauses fleet configuration changes. +2. Creates a consistent database and fleet state snapshot. +3. Encrypts the transfer to the target machine's X25519 public key with an ephemeral key and authenticated encryption. +4. Transfers owner hash, fleet registry, history, settings, integration records, and the fleet signing key. +5. Excludes browser sessions. +6. Starts the target in pending hub mode. +7. Increments the hub generation and signs the new hub announcement. +8. Sends the announcement to online agents. +9. Confirms that the target accepts signed reports and serves the owner login. +10. Demotes the old hub to agent state. + +If validation fails before demotion, the old hub remains authoritative and the target discards pending hub state. + +### Split brain prevention + +Every hub announcement contains the fleet UUID, hub machine UUID, addresses, and monotonically increasing generation. It is signed by the fleet key. + +Agents accept only a valid announcement with a higher generation than the one stored locally. An old hub that returns later cannot reclaim the fleet with stale state. + +### Address change + +Moving the hub normally changes its IP. The completion screen shows and opens the new IP plus port. The owner signs in again because browser sessions are revoked and the origin changed. + +Stable virtual addresses supplied by external network products are compatible but not required. + +## Dead hub recovery + +### Recovery bundle + +The hub periodically creates a compact recovery bundle containing: + +* Fleet UUID and current generation. +* Fleet signing key. +* Machine registry and public keys. +* Listener and fleet settings. +* Integration registration records. +* Schema and bundle version. +* Creation timestamp. + +The recovery bundle does not contain browser sessions or full central history. + +Each bundle is encrypted to the recovery public key with an ephemeral X25519 key, key derivation, and authenticated encryption. The printable recovery key holds the corresponding private key. It is shown once and is never stored by HumWatch. The encrypted bundle is replicated to enrolled agents. + +### Recovery flow + +1. Choose **Recover Fleet** on another installation. +2. Enter the printable recovery key. +3. Select the newest valid recovery bundle found locally. +4. Set a new owner username and password. +5. Promote the PC to hub with a higher generation. +6. Revoke browser sessions and integration credentials. +7. Accept reconnecting agents and rebuild central history from their retained buffers. + +The UI states how much history can be recovered before promotion. History older than all agent buffers is not promised. + +## SlipShell compatibility boundary + +SlipShell monitoring is not implemented in HumWatch 2.1.0, but the contract required for it is. + +The hub provides a versioned `/api/v2` fleet API with stable machine UUIDs. The future SlipShell flow is: + +1. The owner creates a short-lived SlipShell pairing code in HumWatch. +2. SlipShell pairs with the hub and receives a revocable read-only integration identity. +3. SlipShell associates a saved SSH server with a HumWatch machine UUID. +4. SlipShell reads current metrics, history, and availability from the hub. +5. SlipShell can subscribe to a fleet event stream while active. + +SlipShell does not run another monitoring service, read HumWatch's database, or reuse the owner login. + +The API separates public fleet resources from the private hub-to-agent protocol: + +* `/api/v2/session/*` for browser login and sessions. +* `/api/v2/fleet/*` for fleet metadata and administration. +* `/api/v2/machines/*` for machine data and history. +* `/api/v2/integrations/*` for pairing and revocation. +* `/api/internal/v1/*` for agent enrollment, reports, backfill, and hub announcements. + +## `/api/v1` compatibility + +The public v1 health endpoint remains available for version and capability checks. + +Fresh 2.1.0 installs do not create a legacy bearer token. Protected v1 endpoints are disabled unless legacy compatibility is explicitly enabled. + +Upgraded installations can keep their protected v1 endpoints temporarily with the existing bearer token. The migration wizard defaults this option off, labels it as legacy, and provides a removal action. The fleet dashboard and `/api/v2` never depend on it. + +## Upgrade from 2.0.x + +### Safety before choices + +Before changing roles, the upgrade must: + +1. Stop the service cleanly. +2. Locate the active database, including legacy Windows paths. +3. Copy the database plus live WAL state through SQLite's supported backup mechanism. +4. Validate that the backup opens and contains the expected schema and machine record. +5. Archive config, token paths, certificate paths, and CORS origins. +6. Log every migration decision and source path. + +When the migration screen runs in a browser that has a 2.0.x manual peer list in `localStorage`, it reads that list as enrollment suggestions. It clears the old browser list only after the owner confirms migration. + +Issue #14, where Windows upgrades silently orphan the existing database, is a release blocker for 2.1.0. A migration that cannot prove which database it preserved must stop with a visible error. + +### Create Fleet migration + +Choosing **Create Fleet**: + +* Creates the fleet and owner account. +* Imports existing history under this installation's stable machine UUID. +* Makes this PC the hub and first agent. +* Shows the recovery key. +* Disables the browser token gate. +* Archives legacy peer, CORS, token, and certificate configuration unless the owner explicitly enables temporary v1 compatibility. + +### Join Fleet migration + +Choosing **Join Fleet**: + +* Preserves existing history in the local buffer. +* Shows an enrollment code and reachable addresses. +* Joins after the hub proves enrollment. +* Uploads preserved history through ordered backfill. +* Archives legacy peer, CORS, token, and certificate configuration unless the owner explicitly enables temporary v1 compatibility. + +### Rollback + +Legacy files are not deleted automatically. The owner can remove the migration archive after the fleet is verified. Rollback instructions must state that data collected after role conversion may require export before returning to 2.0.x. + +### Duplicate service detection + +Installers check for multiple HumWatch services, conflicting service definitions, and port ownership. When an obsolete development or user service conflicts with the installed service, setup identifies both paths and offers to disable the obsolete service after confirmation. + +The installer does not stop an unrelated process merely because it uses port 9100. + +## Failure behavior + +### Hub unavailable + +Agents continue collecting, buffer locally, and reconnect with bounded backoff. The dashboard is unavailable until the hub returns or recovery promotes another PC. + +### Agent unavailable + +The hub retains its last known state and shows the last report time. It does not label stale data as live. + +### Enrollment interrupted + +The hub does not show an agent as enrolled until it stores the first signed report. A partially registered machine can be retried or removed without changing the fleet identity. + +### Backfill interrupted + +The hub acknowledgement watermark resumes the same range. Duplicate batches remain idempotent. + +### Transfer interrupted + +The old hub stays authoritative until the new hub passes validation. If demotion completed, signed generation rules prevent the old hub from returning as leader. + +### Recovery key unavailable + +Dead hub recovery is not possible. The owner must create a new fleet and re-enroll each PC. This limitation is stated when the recovery key is shown. + +### Version mismatch + +The hub can reject an agent whose internal protocol is incompatible while still showing its address and version. Upgrade instructions must name which side needs updating. + +## Security boundary + +HumWatch 2.1.0 protects against unauthorized application access with owner authentication, independent machine identities, signed data, replay controls, revocation, throttling, and restricted default listeners. + +HTTP on ordinary LAN does not provide confidentiality. Anyone able to observe that network path may read credentials and telemetry. HumWatch must say this plainly. Tailscale or optional HTTPS is the path for untrusted networks. + +HumWatch does not expose the hub to the public internet by default. Public exposure, reverse proxies, and cloud relays are outside 2.1.0. + +## Verification and acceptance + +### Automated coverage + +* Fresh create-fleet flow. +* Fresh join-fleet flow. +* Owner password hashing and migration. +* Login persistence, logout, expiry, throttling, and session revocation. +* CSRF protection for state changes. +* Pairing code expiry, one-use behavior, and rate limiting. +* Machine key registration, report signatures, replay rejection, and revocation. +* Ordered ingestion, duplicate delivery, gaps, and acknowledgement watermarks. +* Offline buffering and reconnect backfill. +* Stable machine identity across hostname and address changes. +* Planned hub transfer success and rollback. +* Stale hub generation rejection. +* Recovery bundle encryption, validation, and promotion. +* Integration scope enforcement. +* Protected `/api/v1` disabled on fresh installs. +* Windows and Linux 2.0.x database migration. +* Duplicate service detection without unrelated process termination. +* HTTP listeners and optional HTTPS listeners. + +### Real system acceptance + +Before release, verify with at least one Windows agent and one Linux agent: + +1. Create a hub on one platform. +2. Enroll the other platform over ordinary LAN. +3. Add or reconnect one agent through Tailscale when available. +4. Use one browser login to switch among all machines. +5. Disconnect the hub, collect data, reconnect it, and verify ordered backfill. +6. Move the hub to the other platform and verify agent reconnection. +7. Recover from an encrypted bundle in a controlled destructive test. +8. Upgrade a real 2.0.x database with history and prove sample continuity. +9. Confirm no manual shared token, CORS, or certificate setup is required. + +### Release acceptance statement + +HumWatch 2.1.0 is accepted when a person can install it on several Windows and Linux PCs, choose one hub, sign in once, enroll each other PC with one temporary code, switch among them inside one dashboard, survive hub downtime, and move or recover the hub without manually distributing credentials. + +## Explicitly outside 2.1.0 + +* Multiple human accounts or roles. +* Cloud relay or public internet publishing. +* Continuous hub replication. +* Automatic high availability or leader election. +* SlipShell user interface work. +* Mobile push notifications. +* Required Tailscale setup. +* Required HTTPS setup. +* Long-term compatibility for the shared bearer token model. + +## Implementation constraints + +* Preserve existing collector behavior unless the hub protocol requires a narrow boundary change. +* Keep the collector usable without the fleet web layer. +* Do not let browser code call agents directly. +* Do not make SQLite table layouts part of `/api/v2`. +* Keep the internal agent protocol versioned separately from the public fleet API. +* Use transactional, resumable migrations with explicit logs. +* Never delete a legacy database, token, certificate, or recovery archive automatically. +* Keep secrets out of URLs, logs, process arguments, and browser storage. +* Keep destructive setup and recovery copy literal and plain. From df36c1d8d28aec103787fa84096bc1f3fd853fab Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 09:24:10 -0400 Subject: [PATCH 02/65] docs: plan HumWatch 2.1 fleet hub implementation --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 1473 +++++++++++++++++ 1 file changed, 1473 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md new file mode 100644 index 0000000..c78d71e --- /dev/null +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -0,0 +1,1473 @@ +# HumWatch 2.1.0 Fleet Hub Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace HumWatch's fragmented per-PC dashboards, shared browser token, mandatory TLS, and CORS setup with one movable fleet hub, one owner login, one-time PC enrollment, buffered agent reporting, optional HTTPS, and a stable read-only integration boundary for SlipShell. + +**Architecture:** One Python package runs in `unconfigured`, `hub`, `agent`, or temporary `legacy` mode. Every installation keeps a stable machine identity and local sample buffer. The active hub owns owner sessions, the fleet registry, central history, `/api/v2`, and the dashboard. Agents authenticate signed reports with independent device keys. Hub transfer uses signed generations, while dead hub recovery uses encrypted bundles copied to agents. + +**Tech Stack:** Python 3.10+, FastAPI, Uvicorn, SQLite through aiosqlite, Pydantic 2, httpx, `cryptography` Ed25519/X25519/AES-GCM, vanilla JavaScript, Chart.js, pytest, PowerShell, systemd, Tailscale as an optional transport. + +## Global Constraints + +* The approved design is `docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md`. +* The normal hub address is `http://:9100`. +* Ordinary private LAN works without Tailscale. +* Tailscale works when installed, but is never required. +* HTTPS is optional and operator supplied. +* Fresh installs never require a shared bearer token, CORS list, or certificate. +* Browser authentication uses one owner username and password plus server-side sessions. +* Pairing codes are ten-minute, single-use values shown only on the target PC. +* Each machine owns separate Ed25519 signing and X25519 encryption key pairs. +* Each integration owns an independent revocable read-only credential. +* Every machine remains capable of becoming the hub. +* The public fleet API is `/api/v2`. The agent protocol is `/api/internal/v1`. +* Fresh installs disable protected `/api/v1` routes unless legacy compatibility is explicitly enabled. +* Central and local history retain seven days by default. +* Never delete legacy databases, tokens, certificates, or migration archives automatically. +* No task may project beyond roughly 800 lines of production code or 25 touched files. Split before implementation if either estimate is crossed. +* One independent completion review is required for every task. Apply fixes, then run a fresh review against the new HEAD until it returns zero new findings. +* UI and endpoint tasks require running surface evidence plus a deliberate-failure canary before PR creation. +* Before every push run `~/.local/bin/gitleaks git --log-opts="--all --not --remotes"` and stop on findings. +* Before every PR, write `/review-receipt.json` for code tasks with the current HEAD, zero new findings, and the evidence path. + +## Delivery shape and dependency order + +This design is too large for one safe branch. Each numbered task below is one reviewable PR and one bounded implementation session. Merge tasks in order because later interfaces depend on earlier schemas. + +| PR | Deliverable | Production estimate | Maximum file count | +|---:|---|---:|---:| +| 1 | Safe 2.0 database preservation and migration log | 180 lines | 6 | +| 2 | Runtime roles, schemas, and machine or fleet identities | 760 lines | 14 | +| 3 | Owner account, browser sessions, and create-fleet API | 760 lines | 14 | +| 4 | Single-use agent enrollment and signed internal requests | 760 lines | 14 | +| 5 | Local outbox, hub ingestion, acknowledgement, and backfill | 780 lines | 14 | +| 6 | Fleet query API and hub event stream | 650 lines | 12 | +| 7 | First-run setup and owner session UI | 650 lines | 14 | +| 8 | Fleet overview and in-app machine selection | 740 lines | 19 | +| 9 | Fleet administration, listener settings, and optional HTTPS | 680 lines | 16 | +| 10 | Planned hub transfer and signed generation handling | 760 lines | 13 | +| 11 | Encrypted recovery bundles and dead hub promotion | 740 lines | 12 | +| 12 | Read-only integration pairing and SlipShell contract | 680 lines | 16 | +| 13 | 2.0 migration wizard and legacy compatibility retirement | 780 lines | 17 | +| 14 | Windows and Linux installer transition plus duplicate-service detection | 720 lines | 16 | +| 15 | End-to-end acceptance, release docs, and 2.1.0 version | 500 lines | 18 | + +The total program is expected to span 15 PRs. That is deliberate. Combining transfer, recovery, migration, UI, and installers would violate the batch ceiling and make review evidence meaningless. + +## Spec coverage map + +| Approved design area | Implemented by | +|---|---| +| Unconfigured, hub, agent, and legacy runtime states | Tasks 2, 3, 13 | +| Collector boundary and seven-day local history buffer | Tasks 2, 5 | +| Agent transport, signed reporting, offline buffering, and ordered backfill | Tasks 4, 5 | +| Fleet registry, central history, current data, and event stream | Tasks 2, 5, 6 | +| Owner authentication, 30-day browser sessions, CSRF, and recovery password reset | Tasks 3, 11 | +| One-time PC enrollment and independent revocation | Tasks 4, 8 | +| Ordinary LAN, optional Tailscale, explicit listeners, and optional HTTPS | Tasks 9, 13 | +| First-run setup, one dashboard, fleet overview, and in-app machine selection | Tasks 7, 8 | +| Planned hub transfer, higher generation enforcement, and split-brain prevention | Task 10 | +| Encrypted recovery replicas and dead hub promotion | Task 11 | +| Read-only SlipShell integration boundary | Task 12 | +| Protected v1 compatibility and explicit 2.0 upgrade choices | Task 13 | +| Database preservation, rollback archive, and issue #14 | Tasks 1, 13 | +| Duplicate service detection and installer transition | Task 14 | +| Real Windows and Linux acceptance plus release evidence | Task 15 | + +## Shared names and wire rules + +These names are fixed for every task. Change them only by updating this plan and every consumer in the same planning commit. + +```python +class RuntimeRole(str, Enum): + UNCONFIGURED = "unconfigured" + HUB = "hub" + AGENT = "agent" + LEGACY = "legacy" + +class MachineStatus(str, Enum): + ENROLLING = "enrolling" + ONLINE = "online" + OFFLINE = "offline" + REVOKED = "revoked" + MIGRATING = "migrating" + +class MetricValue(BaseModel): + category: str + metric_name: str + value: float | int | str | None + unit: str | None = None + +class ProcessValue(BaseModel): + pid: int + name: str + cpu_percent: float + memory_mb: float + +class GapRange(BaseModel): + sequence_start: int + sequence_end: int + reason: Literal["local-retention", "source-corruption"] + +class ReportBatch(BaseModel): + sequence: int + batch_id: UUID + captured_at: datetime + metrics: list[MetricValue] + processes: list[ProcessValue] + +class SignedReportUpload(BaseModel): + fleet_id: UUID + machine_id: UUID + upload_id: UUID + sequence_start: int + sequence_end: int + batches: list[ReportBatch] + gaps: list[GapRange] = Field(default_factory=list) + payload_digest: str + signature: str + +class IngestAck(BaseModel): + highest_contiguous_sequence: int + +class EnrollmentCode(BaseModel): + value: str + expires_at: datetime + +class HubClaim(BaseModel): + enrollment_id: UUID + fleet_id: UUID + hub_machine_id: UUID + hub_generation: int + hub_addresses: list[str] + fleet_signing_public_key: str + recovery_public_key: str + +class AgentClaim(BaseModel): + enrollment_id: UUID + claim_nonce: str + machine_id: UUID + hostname: str + signing_public_key: str + encryption_public_key: str + addresses: list[str] + agent_version: str + +class SignedEnrollmentRecord(BaseModel): + enrollment_id: UUID + agent_claim_digest: str + hub_claim: HubClaim + signature: str + +class EnrolledMachine(BaseModel): + machine_id: UUID + status: MachineStatus + +class HubAnnouncement(BaseModel): + fleet_id: UUID + hub_machine_id: UUID + generation: int + addresses: list[str] + issued_at: datetime + signature: str +``` + +Internal signed requests use these headers: + +```text +X-HumWatch-Machine: +X-HumWatch-Fleet: +X-HumWatch-Timestamp: +X-HumWatch-Nonce: +X-HumWatch-Signature: +``` + +`SignedHeaders` is the typed mapping containing those five headers. The signed bytes are `METHOD + "\n" + PATH + "\n" + FLEET_ID + "\n" + MACHINE_ID + "\n" + SHA256(BODY) + "\n" + TIMESTAMP + "\n" + NONCE`. The hub rejects timestamps beyond 120 seconds, reused nonces, unknown machines, revoked machines, relabeled fleet or machine headers, and invalid signatures. + +Browser mutations send the CSRF token in `X-HumWatch-CSRF`. Session cookies are named `humwatch_session`. + +--- + +### Task 1: Preserve the real 2.0 database before any fleet migration + +**Purpose:** Close issue #14 and make data preservation fail visibly before 2.1 adds more migration paths. + +**Files:** +* Modify: `agent/config.py` +* Modify: `agent/migrations.py` +* Modify: `agent/main.py` +* Modify: `installer/service-setup.ps1` +* Modify: `tests/test_database_migration.py` +* Modify: `tests/test_windows_service_security.py` + +**Interfaces:** +* Produces: `HumWatchConfig.legacy_db_path: Optional[str]` +* Produces: `HumWatchConfig.resolved_legacy_db_path -> Optional[Path]` +* Produces: `MigrationResult(source: Path | None, target: Path, status: str, detail: str, archive_manifest: Path | None)` +* Produces: `migrate_legacy_database(config: HumWatchConfig) -> MigrationResult` + +- [ ] **Step 1: Pin the Windows installer failure with a test** + +Add a test that gives the runtime an absolute destination plus an explicit live SQLite source. Put `machine_info` and `metrics` in the database, enable WAL, commit one more metric through the WAL connection, and leave the connection open while migration runs: + +```python +def test_installer_absolute_destination_migrates_explicit_legacy_source(tmp_path): + legacy = tmp_path / "legacy" / "humwatch.db" + target = tmp_path / "runtime" / "humwatch.db" + legacy.parent.mkdir() + source = sqlite3.connect(legacy) + source.execute("PRAGMA journal_mode=WAL") + source.execute("CREATE TABLE machine_info (machine_id TEXT PRIMARY KEY)") + source.execute("CREATE TABLE metrics (timestamp TEXT, metric_name TEXT, value REAL)") + source.execute("INSERT INTO machine_info VALUES ('machine-a')") + source.execute("INSERT INTO metrics VALUES ('2026-08-13T12:00:00Z', 'cpu', 42)") + source.commit() + source.execute("INSERT INTO metrics VALUES ('2026-08-13T12:01:00Z', 'cpu', 43)") + source.commit() + config = HumWatchConfig( + db_path=str(target), + legacy_db_path=str(legacy), + data_root=str(tmp_path / "runtime"), + ) + + result = migrate_legacy_database(config) + + assert result.status == "migrated" + migrated = sqlite3.connect(target) + assert migrated.execute("PRAGMA quick_check").fetchone() == ("ok",) + assert migrated.execute("SELECT machine_id FROM machine_info").fetchone() == ("machine-a",) + assert migrated.execute("SELECT COUNT(*) FROM metrics").fetchone() == (2,) + assert result.archive_manifest.is_file() + assert legacy.is_file() +``` + +- [ ] **Step 2: Run the focused tests and confirm the old absolute-path guard fails** + +Run: `.venv/bin/python -m pytest tests/test_database_migration.py -q` + +Expected: FAIL because `legacy_db_path` and `MigrationResult` do not exist. + +- [ ] **Step 3: Implement explicit source selection and verifiable preservation** + +Add `legacy_db_path` to config and use it before the old inferred path: + +```python +@dataclass(frozen=True) +class MigrationResult: + source: Path | None + target: Path + status: str + detail: str + archive_manifest: Path | None + +def legacy_database_path(config: HumWatchConfig) -> Optional[Path]: + if config.resolved_legacy_db_path is not None: + return config.resolved_legacy_db_path + if Path(config.db_path).is_absolute(): + return None + return PROJECT_ROOT / Path(config.db_path) +``` + +Open the source read-only, use `sqlite3.Connection.backup()` into a staging file so committed WAL pages are included, then run `PRAGMA quick_check`, verify the required 2.0 tables, verify at least one `machine_info` row, and compare source and staged row counts. Only then rename the staging database atomically to the target. + +Write `resolved_data_root / "migration-archive" / / "manifest.json"` with source and target paths, SHA256 digests, verified table row counts, status, and time. Return and log `migrated`, `source_missing`, `target_exists`, `same_path`, or `failed`. Never unlink or rewrite the source database, its WAL or shared-memory sidecars, or an existing archive. Startup must raise `MigrationSafetyError` when an explicit source exists but backup or proof fails. + +- [ ] **Step 4: Make the installer pass the old path explicitly** + +Set `HUMWATCH_LEGACY_DB` to `C:\HumWatch\humwatch.db` only when that file exists. Preserve existing upgrade behavior when it does not. + +- [ ] **Step 5: Verify the focused migration and installer contracts** + +Run: `.venv/bin/python -m pytest tests/test_database_migration.py tests/test_windows_service_security.py -q` + +Expected: PASS with explicit assertions for every decline status, WAL continuity, manifest proof, source preservation, and no silent branch. + +- [ ] **Step 6: Run the full task gate and commit** + +Run: `.venv/bin/python -m pytest -q` + +Commit: + +```bash +git add agent/config.py agent/migrations.py agent/main.py installer/service-setup.ps1 tests/test_database_migration.py tests/test_windows_service_security.py +git commit -m "fix: preserve Windows history during upgrades" +``` + +Use issue #14 as the PR close target. + +--- + +### Task 2: Add runtime roles, fleet schema, and protected identities + +**Purpose:** Establish durable types and storage without changing the current dashboard behavior. + +**Files:** +* Create: `agent/fleet/__init__.py` +* Create: `agent/fleet/models.py` +* Create: `agent/fleet/schema.py` +* Create: `agent/fleet/identity.py` +* Create: `agent/fleet/state.py` +* Modify: `agent/config.py` +* Modify: `agent/database.py` +* Modify: `agent/main.py` +* Modify: `config.json` +* Modify: `tests/conftest.py` +* Create: `tests/test_fleet_schema.py` +* Create: `tests/test_fleet_identity.py` +* Create: `tests/test_runtime_roles.py` +* Modify: `tests/test_security_config.py` + +**Interfaces:** +* Produces: `RuntimeRole`, `MachineStatus`, `MetricValue`, `ProcessValue`, `ReportBatch`, `HubAnnouncement` +* Produces: `MachineIdentity.load_or_create(root: Path) -> MachineIdentity` +* Produces: `FleetIdentity.create(root: Path) -> tuple[FleetIdentity, str]` +* Produces: `FleetStateRepository.get_runtime_state() -> RuntimeState` +* Produces: `async FleetStateRepository.set_role(role: RuntimeRole) -> None` +* Produces: `transaction(db: aiosqlite.Connection) -> AsyncContextManager[None]` +* Produces: `detect_runtime_role(config: HumWatchConfig) -> RuntimeRole` + +- [ ] **Step 1: Write schema and identity failure tests** + +Cover stable UUIDs across restart, mode defaulting, key file permissions, signature verification, X25519 round trip, and seven-day default retention: + +```python +async def test_new_database_starts_unconfigured(db): + state = await FleetStateRepository(db).get_runtime_state() + assert state.role is RuntimeRole.UNCONFIGURED + +def test_machine_identity_survives_restart(tmp_path): + first = MachineIdentity.load_or_create(tmp_path) + second = MachineIdentity.load_or_create(tmp_path) + assert first.machine_id == second.machine_id + assert first.sign(b"sample") == second.sign(b"sample") +``` + +- [ ] **Step 2: Run the new test files and confirm missing modules fail** + +Run: `.venv/bin/python -m pytest tests/test_fleet_schema.py tests/test_fleet_identity.py tests/test_runtime_roles.py -q` + +Expected: collection FAIL for missing `agent.fleet` modules. + +- [ ] **Step 3: Add the v2 tables** + +`agent/fleet/schema.py` owns idempotent SQL for: + +```sql +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, legacy_api_enabled, updated_at) +fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) +owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) +owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) +fleet_settings(id, listeners_json, https_cert_path, https_key_path, allow_public_listener, updated_at) +fleet_metrics(machine_id, sequence, timestamp, category, metric_name, value, unit) +fleet_processes(machine_id, sequence, timestamp, pid, name, cpu_percent, memory_mb) +report_outbox(sequence, batch_id, captured_at, payload_json, acknowledged_at) +report_receipts(machine_id, upload_id, payload_digest, sequence_start, sequence_end, highest_contiguous_sequence, accepted_at) +data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) +used_nonces(machine_id, nonce_hash, expires_at) +enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) +integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) +recovery_bundles(generation, created_at, encrypted_payload) +recovery_deliveries(generation, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) +``` + +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, and `(generation, machine_id)` for recovery delivery. Check constraints require ordered sequence ranges, nonnegative attempts, and known state values. + +It also owns the transaction helper used by every fleet repository: + +```python +@asynccontextmanager +async def transaction(db: aiosqlite.Connection): + await db.execute("BEGIN IMMEDIATE") + try: + yield + except BaseException: + await db.rollback() + raise + else: + await db.commit() +``` + +- [ ] **Step 4: Implement machine and fleet identity storage** + +Use raw private key bytes stored below `resolved_data_root / "identity"`. Create files atomically with mode `0600` on POSIX. Windows ACL enforcement remains the installer task. + +```python +@dataclass(frozen=True) +class MachineIdentity: + machine_id: UUID + signing_private_key: Ed25519PrivateKey + encryption_private_key: X25519PrivateKey + + def sign(self, payload: bytes) -> bytes: + return self.signing_private_key.sign(payload) +``` + +`FleetIdentity.create()` returns the stored fleet signing identity plus a printable grouped base64url recovery private key with a checksum. Persist only the recovery public key. + +- [ ] **Step 5: Detect the role before applying any network security gate** + +Move role detection ahead of the current unconditional `validate_security_config()` calls in `agent/main.py`. Existing databases with an auth token start as `legacy`. Fresh databases start as `unconfigured`. Unconfigured mode binds HTTP only to `127.0.0.1` and does not require a token or certificate. Legacy mode keeps the existing token, TLS, and CORS validation exactly. Hub and agent modes use loopback or explicitly stored private listeners until Task 9 adds the settings surface. No role may fall back to `0.0.0.0`. + +`hub` starts collector, retention, and hub services. `agent` starts collector, retention, and agent transport. Until those services exist, role branches log the selected mode and start only the services already implemented for that role. Add startup tests proving a fresh config with no token and no TLS reaches the unconfigured app, while an unsafe legacy config still fails closed. + +Set both `HumWatchConfig.retention_days` and the checked-in `config.json` value to `7` here so later outbox and hub retention tests share the approved default. + +- [ ] **Step 6: Verify schema idempotency and the full suite** + +Run: + +```bash +.venv/bin/python -m pytest tests/test_fleet_schema.py tests/test_fleet_identity.py tests/test_runtime_roles.py tests/test_security_config.py -q +.venv/bin/python -m pytest -q +``` + +- [ ] **Step 7: Commit the foundation** + +```bash +git add agent/fleet agent/config.py agent/database.py agent/main.py config.json tests/conftest.py tests/test_fleet_schema.py tests/test_fleet_identity.py tests/test_runtime_roles.py tests/test_security_config.py +git commit -m "feat: add fleet runtime foundation" +``` + +--- + +### Task 3: Create the owner account and server-side browser sessions + +**Purpose:** Replace the normal bearer-token prompt for hub mode and create a fleet from local setup. + +**Files:** +* Create: `agent/security/passwords.py` +* Create: `agent/security/sessions.py` +* Create: `agent/fleet/setup.py` +* Create: `agent/routes/v2_session.py` +* Create: `agent/routes/v2_setup.py` +* Modify: `agent/security/limits.py` +* Modify: `agent/main.py` +* Modify: `tests/conftest.py` +* Create: `tests/test_owner_passwords.py` +* Create: `tests/test_owner_sessions.py` +* Create: `tests/test_create_fleet.py` +* Create: `tests/test_v2_csrf.py` +* Create: `tests/test_login_throttle.py` +* Modify: `tests/test_api_auth.py` + +**Interfaces:** +* Consumes: `FleetIdentity.create`, `MachineIdentity.load_or_create`, `FleetStateRepository` +* Produces: `hash_password(password: str) -> PasswordRecord` +* Produces: `verify_password(password: str, record: PasswordRecord) -> bool` +* Produces: `require_owner_session(request: Request) -> OwnerSession` +* Produces: `require_csrf(request: Request, session: OwnerSession) -> None` +* Produces: `POST /api/v2/setup/create-fleet` +* Produces: `POST /api/v2/session/login`, `GET /me`, `POST /logout`, `GET /sessions`, `DELETE /sessions/{session_hash}` + +- [ ] **Step 1: Write failing password, session, and setup tests** + +Test `hashlib.scrypt`, parameter persistence, rehash after successful login from an older parameter version, generic login failures, a 30-day expiry, `HttpOnly`, `SameSite=Strict`, conditional `Secure`, CSRF rejection, logout, individual revocation, all-session revocation, and loopback-only fleet creation. + +```python +def test_login_sets_server_side_session_cookie(hub_client, owner_password): + response = hub_client.post("/api/v2/session/login", json={ + "username": "static", + "password": owner_password, + }) + assert response.status_code == 204 + assert "HttpOnly" in response.headers["set-cookie"] + assert "SameSite=strict" in response.headers["set-cookie"] +``` + +- [ ] **Step 2: Run the focused tests and confirm missing endpoints fail** + +Run: `.venv/bin/python -m pytest tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py -q` + +- [ ] **Step 3: Implement versioned scrypt and opaque sessions** + +Use a random 32-byte session token and store only `SHA256(token)`. Return the raw token only in the cookie. Generate a separate CSRF token, store only its digest, and return the raw value from `/api/v2/session/me`. + +```python +PASSWORD_VERSION = 1 +SCRYPT_N_CHOICES = (1 << 14, 1 << 15, 1 << 16) +SCRYPT_R = 8 +SCRYPT_P = 1 +SESSION_TTL = timedelta(days=30) +``` + +On first fleet creation, benchmark the allowed `N` choices with an injected clock and choose the strongest value that stays at or below 200 ms and 64 MiB. Store `password_version`, `N`, `r`, and `p` with the hash. Do not calibrate on every startup. On a successful login, rehash inside a transaction when the stored version or parameters are below the current policy. Tests use an injected deterministic calibrator and never depend on workstation speed. + +- [ ] **Step 4: Add two-dimensional login throttling** + +Keep the existing generic v1 request limiter. Add a login-specific limiter with three counters: 5 attempts per username and source pair per 60 seconds, 15 attempts per username across all sources per 5 minutes, and 30 attempts per source across all usernames per 5 minutes. Normalize usernames before keying, hash stored counter keys, return the same public failure for unknown users and bad passwords, and add tests for one username across multiple addresses plus many usernames from one address. + +- [ ] **Step 5: Implement create-fleet as one transaction** + +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. Create fleet identity, owner record, self machine row, and runtime role `hub` in one database transaction. Return the printable recovery key exactly once. + +- [ ] **Step 6: Mount `/api/v2` without weakening v1** + +Session login and setup routes are public but rate limited. All fleet routes added later depend on `require_owner_session`. Keep existing v1 behavior unchanged for `legacy` mode. + +- [ ] **Step 7: Verify auth behavior and commit** + +Run: + +```bash +.venv/bin/python -m pytest tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py tests/test_login_throttle.py tests/test_api_auth.py tests/test_request_limits.py -q +.venv/bin/python -m pytest -q +``` + +Commit: + +```bash +git add agent/security/passwords.py agent/security/sessions.py agent/fleet/setup.py agent/routes/v2_session.py agent/routes/v2_setup.py agent/security/limits.py agent/main.py tests/conftest.py tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py tests/test_login_throttle.py tests/test_api_auth.py +git commit -m "feat: add owner sessions and fleet creation" +``` + +--- + +### Task 4: Enroll agents with single-use codes and signed requests + +**Purpose:** Add a PC without distributing a shared token. + +**Files:** +* Create: `agent/fleet/request_signing.py` +* Create: `agent/fleet/enrollment.py` +* Create: `agent/fleet/hub_client.py` +* Create: `agent/routes/internal_enrollment.py` +* Create: `agent/routes/v2_machines.py` +* Modify: `agent/fleet/models.py` +* Modify: `agent/fleet/state.py` +* Modify: `agent/security/limits.py` +* Modify: `agent/main.py` +* Create: `tests/test_internal_request_signing.py` +* Create: `tests/test_agent_enrollment.py` +* Create: `tests/test_enrollment_limits.py` +* Modify: `tests/test_request_limits.py` +* Modify: `tests/test_security_contract.py` + +**Interfaces:** +* Produces: `EnrollmentService.create_code() -> EnrollmentCode` +* Produces: `EnrollmentService.claim(code: str, hub: HubClaim) -> AgentClaim` +* Produces: `EnrollmentService.finalize(enrollment_id: UUID, nonce: str, record: SignedEnrollmentRecord) -> RuntimeState` +* Produces: `EnrollmentRepository.store_claimed_attempt(record: SignedEnrollmentRecord, address: str, claim_nonce: str) -> None` +* Produces: `sign_request(identity, fleet_id, machine_id, method, path, body, now, nonce) -> SignedHeaders` +* Produces: `verify_internal_request(request, machine_record) -> None` +* Produces: hub `POST /api/v2/fleet/machines/enroll` +* Produces: local-only `POST /api/v2/setup/pairing-code` +* Produces: agent `POST /api/internal/v1/enrollment/claim` and `/finalize` + +Use this one canonical implementation boundary: + +```python +def canonical_request_bytes( + fleet_id: UUID, + machine_id: UUID, + method: str, + path: str, + body: bytes, + timestamp: str, + nonce: str, +) -> bytes: + digest = hashlib.sha256(body).hexdigest() + return f"{method.upper()}\n{path}\n{fleet_id}\n{machine_id}\n{digest}\n{timestamp}\n{nonce}".encode("utf-8") + +async def enroll_machine(address: str, code: str) -> EnrolledMachine: + hub_claim = local_hub_claim() + claim = await hub_client.claim(address, code, hub_claim) + record = sign_enrollment_record(hub_claim, claim) + await repository.store_claimed_attempt(record, address, claim.claim_nonce) + await hub_client.finalize(address, claim.enrollment_id, claim.claim_nonce, record) + return await repository.mark_finalized(claim.enrollment_id) +``` + +- [ ] **Step 1: Write signing and enrollment protocol tests** + +Cover body tampering, clock skew, nonce replay, expired code, reused code, private-address validation, interrupted finalization, a hub crash after its durable claim record, idempotent finalize retry, first-report pending status, and fleet or machine header relabeling with an otherwise valid signature. + +```python +def test_pairing_code_expires_after_ten_minutes(clock, enrollment): + issued = enrollment.create_code() + clock.advance(minutes=10, seconds=1) + with pytest.raises(EnrollmentExpired): + enrollment.claim(issued.value, sample_hub_claim()) +``` + +- [ ] **Step 2: Run focused tests and observe the missing protocol** + +Run: `.venv/bin/python -m pytest tests/test_internal_request_signing.py tests/test_agent_enrollment.py tests/test_enrollment_limits.py -q` + +- [ ] **Step 3: Implement canonical request signing** + +Keep canonicalization in one module. Reject duplicate signature headers and decode base64url strictly. Store nonce digests until their 120-second replay window expires. + +- [ ] **Step 4: Implement local-only code display and network claim endpoints** + +`create_code()` is callable only from the local setup surface. Reject a request to `/api/v2/setup/pairing-code` unless the actual peer address is loopback, including when forwarding headers claim loopback. No HTTP response from `/api/internal/v1` returns the code. Claim binds the `enrollment_id`, hub claim digest, code, and agent claim. A retry with the same enrollment ID and identical hub claim returns the same `AgentClaim`. Any changed claim under that ID fails. + +- [ ] **Step 5: Implement hub-orchestrated enrollment** + +The hub validates `http` or `https` private addresses and calls claim. Before finalize, it durably inserts the hub claim JSON, signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. + +- [ ] **Step 6: Verify enrollment and commit** + +Run the three new files, `tests/test_request_limits.py`, `tests/test_security_contract.py`, then the full suite. + +Commit message: `feat: enroll agents with one-time codes` + +--- + +### Task 5: Buffer reports locally and ingest them at the hub + +**Purpose:** Make agent telemetry reliable across hub downtime. + +**Files:** +* Create: `agent/fleet/outbox.py` +* Create: `agent/fleet/ingestion.py` +* Create: `agent/fleet/transport.py` +* Create: `agent/routes/internal_reports.py` +* Modify: `agent/fleet/models.py` +* Modify: `agent/collector.py` +* Modify: `agent/services/retention.py` +* Modify: `agent/main.py` +* Create: `tests/test_report_outbox.py` +* Create: `tests/test_hub_ingestion.py` +* Create: `tests/test_agent_backfill.py` +* Create: `tests/test_collection_to_outbox.py` +* Modify: `tests/test_request_limits.py` +* Modify: `tests/test_database_migration.py` + +**Interfaces:** +* Produces: `OutboxRepository.append(captured_at, metrics, processes) -> ReportBatch` +* Produces: `OutboxRepository.pending(after: int, limit: int) -> list[ReportBatch]` +* Produces: `OutboxRepository.acknowledge(sequence: int) -> None` +* Produces: `HubIngestion.ingest(upload: SignedReportUpload) -> IngestAck` +* Produces: `AgentTransport.run() -> None` +* Produces: `POST /api/internal/v1/reports` + +The ingestion transaction has this shape: + +```python +async def ingest(self, upload: SignedReportUpload) -> IngestAck: + await self.machines.require_active(upload.machine_id) + async with transaction(self.database): + await self.receipts.require_new_or_identical(upload) + await self.samples.insert_upload_if_absent(upload) + await self.gaps.insert_if_absent(upload.machine_id, upload.gaps) + highest = await self.samples.highest_contiguous(upload.machine_id) + await self.machines.mark_reported(upload.machine_id, highest, upload.batches[-1].captured_at) + await self.receipts.record(upload) + await self.events.publish_machine_update(upload.machine_id, upload.sequence_end) + return IngestAck(highest_contiguous_sequence=highest) +``` + +- [ ] **Step 1: Write outbox and idempotency tests** + +```python +async def test_duplicate_upload_is_stored_once(hub_ingestion, enrolled_machine, upload): + first = await hub_ingestion.ingest(upload) + second = await hub_ingestion.ingest(upload) + assert first.highest_contiguous_sequence == upload.sequence_end + assert second.highest_contiguous_sequence == upload.sequence_end + assert await count_metrics(upload.machine_id, upload.sequence_start) == len(upload.batches[0].metrics) +``` + +Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. + +- [ ] **Step 2: Run focused tests and confirm missing outbox behavior** + +Run: `.venv/bin/python -m pytest tests/test_report_outbox.py tests/test_hub_ingestion.py tests/test_agent_backfill.py tests/test_collection_to_outbox.py -q` + +- [ ] **Step 3: Add one collector persistence boundary** + +Refactor the end of each collection tick to build one `ReportBatch`. In `legacy`, keep the existing `_write_metrics` and `_write_processes`. In `agent` or `hub`, append the same normalized data to `report_outbox`. A hub immediately ingests its own batch through `HubIngestion`, then acknowledges its own outbox row only after that transaction commits. + +- [ ] **Step 4: Build and verify the durable upload envelope** + +Build canonical compact JSON with sorted keys for `{batches, gaps}` and set `payload_digest` to its SHA256. Sign canonical compact JSON containing `fleet_id`, `machine_id`, `upload_id`, `sequence_start`, `sequence_end`, and `payload_digest` with the machine key. The outer HTTP request is also signed through Task 4, binding the exact body, fleet, and machine headers. + +The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. Insert metrics, processes, gap ranges, the machine watermark, and `report_receipts` in one transaction. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. + +- [ ] **Step 5: Implement transport and retention** + +Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. + +- [ ] **Step 6: Verify collection, reporting, and full regression** + +Run the new tests plus existing current, history, SSE, retention, and full suite tests. + +Commit message: `feat: buffer and ingest fleet telemetry` + +--- + +### Task 6: Expose machine-scoped fleet data through `/api/v2` + +**Purpose:** Give the hub dashboard and SlipShell seam one stable read model. + +**Files:** +* Create: `agent/fleet/queries.py` +* Create: `agent/fleet/events.py` +* Create: `agent/routes/v2_fleet.py` +* Create: `agent/routes/v2_history.py` +* Create: `agent/routes/v2_events.py` +* Modify: `agent/routes/v2_machines.py` +* Modify: `agent/main.py` +* Create: `tests/test_v2_fleet_api.py` +* Create: `tests/test_v2_history_api.py` +* Create: `tests/test_v2_events.py` +* Modify: `tests/test_request_limits.py` +* Modify: `tests/test_frontend_sse.py` + +**Interfaces:** +* Produces: `GET /api/v2/fleet` +* Produces: `GET /api/v2/machines` +* Produces: `GET /api/v2/machines/{machine_id}` +* Produces: `GET /api/v2/machines/{machine_id}/current` +* Produces: `GET /api/v2/machines/{machine_id}/history` +* Produces: `GET /api/v2/machines/{machine_id}/processes` +* Produces: `GET /api/v2/machines/{machine_id}/availability` +* Produces: `GET /api/v2/events` +* Produces: `MachineAvailabilityResponse(status, last_report_at, stale_after_seconds, data_gaps, backfill)` + +Route handlers stay thin and require the owner session at the router boundary: + +```python +@router.get("/machines/{machine_id}/current", response_model=MachineCurrentResponse) +async def current_machine( + machine_id: UUID, + _session: OwnerSession = Depends(require_owner_session), +): + return await get_fleet_queries().current(machine_id) +``` + +- [ ] **Step 1: Write machine isolation and stale-data tests** + +Assert stable UUID routing, no cross-machine data, explicit `data_gap` ranges, last report timestamps, stale threshold, backfill watermark and pending count, revoked machine visibility to the owner, history bounds, and SSE capacity limits. + +- [ ] **Step 2: Run focused tests and confirm v2 reads are absent** + +Run: `.venv/bin/python -m pytest tests/test_v2_fleet_api.py tests/test_v2_history_api.py tests/test_v2_events.py -q` + +- [ ] **Step 3: Implement query services over v2 tables only** + +Do not expose table rows directly. Return typed response models. Current values come from the newest sequence per machine. Offline status is computed from the last report time and configured collection interval. Availability returns `status`, nullable `last_report_at`, `stale_after_seconds`, ordered persisted gap ranges, and a typed backfill object with acknowledged sequence, newest local sequence when known, and pending count. + +- [ ] **Step 4: Implement a hub-owned event stream** + +Emit `machine_metrics`, `machine_status`, `backfill_progress`, `machine_revoked`, and heartbeat events. Every event includes `machine_id`. Use owner session cookie authentication. + +- [ ] **Step 5: Verify endpoint surface with a running test server** + +Start a temporary hub, log in, fetch two machine IDs, prove their current and history responses differ, then capture the command output under `artifacts/task-6-api-surface.txt`. + +Run a deliberate canary with a revoked session and require `401` before accepting the evidence. + +- [ ] **Step 6: Run full tests and commit** + +Commit message: `feat: expose fleet data through api v2` + +--- + +### Task 7: Replace the token gate with owner login and first-run setup + +**Purpose:** Make Create Fleet, Join Fleet, owner login, and logout usable before changing the monitoring pages. + +**Files:** +* Create: `static/js/pages/login.js` +* Create: `static/js/pages/setup.js` +* Create: `scripts/update-static-digests.py` +* Create: `tests/test_frontend_session_auth.py` +* Create: `tests/test_frontend_setup.py` +* Modify: `static/index.html` +* Modify: `static/js/auth.js` +* Modify: `static/js/api.js` +* Modify: `static/js/app.js` +* Modify: `static/css/components.css` +* Modify: `static/css/layout.css` +* Modify: `static/vendor/SHA256SUMS` +* Modify: `tests/test_frontend_auth.py` +* Modify: `tests/test_static_asset_policy.py` + +**Interfaces:** +* Consumes: Task 3 session and create-fleet endpoints +* Consumes: Task 4 local pairing-code endpoint +* Produces: `HumWatch.auth.loadSession() -> Promise` +* Produces: `HumWatch.auth.csrfHeaders(method) -> Headers` +* Produces: setup routes `#/setup`, `#/login`, and `#/recovery` + +- [ ] **Step 1: Write browser harness tests for setup and session login** + +Assert no `sessionStorage` token, credentials included on fetch, CSRF header on mutations, loopback Create Fleet, local pairing-code display for Join Fleet, normal logout, and generic login failures. + +```javascript +await HumWatch.auth.loadSession(); +assert.equal(fetchCalls[0].url, "/api/v2/session/me"); +assert.equal(fetchCalls[0].options.credentials, "same-origin"); +assert.equal(sessionStorage.getItem("humwatch_auth_token"), null); +``` + +- [ ] **Step 2: Run frontend tests and confirm the token flow fails them** + +Run: `.venv/bin/python -m pytest tests/test_frontend_auth.py tests/test_frontend_session_auth.py tests/test_frontend_setup.py -q` + +Expected: FAIL while bearer headers, token storage, and the token prompt remain. + +- [ ] **Step 3: Convert the browser client to owner sessions** + +Remove bearer token storage and remote fetch helpers. Set `credentials: "same-origin"`. Load `/api/v2/session/me` before protected navigation. Send the returned CSRF token only on state-changing requests. + +- [ ] **Step 4: Implement setup and login pages** + +Create Fleet submits username and password and displays the recovery key once. Join Fleet requests a code from the local endpoint and displays the selected private addresses. Login accepts username and password. Recovery is a routed shell until Task 11 adds its backend. + +- [ ] **Step 5: Add the deterministic digest updater** + +Create `scripts/update-static-digests.py`. It reads the existing manifest, refuses missing files, duplicate paths, paths outside `static`, or malformed rows, recomputes only the listed local assets, sorts output by path, and replaces the manifest atomically. Add a test that copies the static tree, changes one CSS file, runs the updater against the copy, and proves only that file's digest changes. + +Run: `.venv/bin/python scripts/update-static-digests.py` + +- [ ] **Step 6: Update pinned assets and produce surface evidence** + +Capture unconfigured setup, recovery key display, login failure, successful login, and logout at desktop and mobile widths in default, light, and terminal themes under `artifacts/task-7-auth-ui/`. + +Canary: return `200` from login without a session cookie in the browser fixture and require the dashboard to remain locked. + +- [ ] **Step 7: Verify and commit** + +Run focused browser tests, `.venv/bin/python -m pytest tests/test_static_asset_policy.py -q`, `.venv/bin/python scripts/verify-static-assets.py`, `node tests/browser_globals_smoke.js`, and the full pytest suite. + +Commit message: `feat: add fleet setup and owner login` + +--- + +### Task 8: Add fleet overview and in-app machine selection + +**Purpose:** Deliver one dashboard for every enrolled PC with no new tabs or cross-origin browser calls. + +**Files:** +* Create: `static/js/fleet-state.js` +* Create: `tests/test_frontend_fleet_state.py` +* Create: `tests/test_frontend_machine_switching.py` +* Modify: `static/index.html` +* Modify: `static/js/api.js` +* Modify: `static/js/app.js` +* Modify: `static/js/sse.js` +* Modify: `static/js/pages/machines.js` +* Modify: `static/js/pages/overview.js` +* Modify: `static/js/pages/cpu.js` +* Modify: `static/js/pages/gpu.js` +* Modify: `static/js/pages/memory.js` +* Modify: `static/js/pages/disk.js` +* Modify: `static/js/pages/network.js` +* Modify: `static/js/pages/battery.js` +* Modify: `static/js/pages/processes.js` +* Modify: `static/css/components.css` +* Modify: `static/css/layout.css` +* Modify: `static/vendor/SHA256SUMS` + +**Interfaces:** +* Consumes: Task 6 `/api/v2` responses and events +* Produces: `HumWatch.fleet.selectMachine(machineId)` +* Produces: `HumWatch.fleet.getSelectedMachineId()` +* Produces: `HumWatch.api.machinePath(suffix)` + +- [ ] **Step 1: Write browser harness tests for switching** + +Assert stable selected machine, same-origin URLs only, no `window.open`, per-machine current and history paths, process routing, selected-machine SSE filtering, and page refresh after selection. + +```javascript +HumWatch.fleet.selectMachine("machine-b"); +await HumWatch.pages.cpu.init(container); +assert.equal(fetchCalls[0], "/api/v2/machines/machine-b/current"); +``` + +- [ ] **Step 2: Run focused tests and confirm the current new-tab behavior fails** + +Run: `.venv/bin/python -m pytest tests/test_frontend_fleet_state.py tests/test_frontend_machine_switching.py tests/test_frontend_sse.py -q` + +- [ ] **Step 3: Add persistent machine selection** + +Put the selector at the top of the desktop sidebar and mobile header. Store only the selected machine UUID in `localStorage`. Every data page obtains its URL through `machinePath()`. + +- [ ] **Step 4: Replace Machines with the fleet overview** + +Render enrolled machine cards from the hub. Clicking a card calls `selectMachine()` and navigates within the current origin. Show status, last report, current highlights, gaps, and backfill progress. + +- [ ] **Step 5: Route live events and every data page by machine UUID** + +The hub event stream remains one connection. Dispatch each event only to handlers for its `machine_id`. Refresh selected-machine availability after a selection changes. + +- [ ] **Step 6: Update pinned assets and produce surface evidence** + +Run `.venv/bin/python scripts/update-static-digests.py` after the final JavaScript and CSS edits, then run the verifier before capturing evidence. + +Capture fleet overview and two selected machines at desktop and mobile widths in default, light, and terminal themes under `artifacts/task-8-fleet-ui/`. + +Canary: break the selected-machine API path in the browser fixture, confirm the switching test fails, restore it, then rerun. + +- [ ] **Step 7: Verify and commit** + +Run focused browser tests, static asset verification, browser smoke, and the full pytest suite. + +Commit message: `feat: unify fleet access in one dashboard` + +--- + +### Task 9: Add fleet administration, listeners, and optional HTTPS + +**Purpose:** Make enrollment, removal, sessions, network access, and TLS manageable from one settings surface. + +**Files:** +* Create: `agent/fleet/listeners.py` +* Create: `agent/routes/v2_settings.py` +* Create: `static/js/pages/fleet-settings.js` +* Create: `tests/test_listener_policy.py` +* Create: `tests/test_v2_fleet_admin.py` +* Create: `tests/test_frontend_fleet_settings.py` +* Modify: `agent/config.py` +* Modify: `agent/security/tls.py` +* Modify: `agent/main.py` +* Modify: `static/index.html` +* Modify: `static/js/api.js` +* Modify: `static/js/pages/settings.js` +* Modify: `static/css/components.css` +* Modify: `static/css/layout.css` +* Modify: `static/vendor/SHA256SUMS` +* Modify: `tests/test_security_config.py` + +**Interfaces:** +* Produces: `discover_private_listeners() -> list[ListenerCandidate]` +* Produces: `GET/PUT /api/v2/fleet/settings/listeners` +* Produces: `GET/PUT /api/v2/fleet/settings/https` +* Produces: `DELETE /api/v2/fleet/machines/{machine_id}` +* Produces: `ListenerSettingsResponse(saved, active, restart_required)` + +Listener selection is an explicit typed configuration: + +```python +class ListenerCandidate(BaseModel): + address: IPvAnyAddress + kind: Literal["loopback", "lan", "tailscale", "public"] + enabled: bool + warning_required: bool + +def bind_listener_sockets(listeners: list[ListenerCandidate], port: int) -> list[socket.socket]: + enabled = [item for item in listeners if item.enabled] + if not enabled: + raise SecurityConfigurationError("at least one listener is required") + return [bind_tcp_socket(str(item.address), port) for item in enabled] +``` + +Pass every returned socket to one `uvicorn.Server.run(sockets=sockets)` call so application startup, the collector, and database initialization run once. + +- [ ] **Step 1: Write listener and settings tests** + +Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public-address warnings, optional cert pairs, missing half-pairs, CSRF, self-removal rejection, independent revocation, persisted listeners across restart, and a saved change that does not affect active sockets before restart. + +- [ ] **Step 2: Change the security rule from mandatory TLS to explicit listeners** + +`validate_security_config()` must allow HTTP for selected private listeners without requiring a legacy bearer token. Public listeners require an explicit `allow_public_listener` acknowledgement and still show a warning. Store the exact list, HTTPS paths, and acknowledgement in the Task 2 `fleet_settings` row. Startup loads that row and binds exactly those addresses. Never silently widen to `0.0.0.0` after a saved specific-listener configuration. + +`PUT` validates and saves settings but does not mutate live sockets. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. + +- [ ] **Step 3: Implement Settings sections** + +Add Owner and Sessions, Fleet and Enrolled PCs, Add PC, Network Listeners, Optional HTTPS, Integrations status, and Legacy Compatibility. Destructive actions use plain confirmation copy. + +- [ ] **Step 4: Produce running surface evidence and canary** + +Capture settings on default, light, and terminal themes. Exercise add PC, revoke PC, session revoke, LAN listener selection, detected Tailscale selection, and optional HTTPS validation. + +Canary: select a public test address without acknowledgement and confirm the API and UI block saving. + +- [ ] **Step 5: Verify and commit** + +Run focused tests, static asset verification, browser smoke, full pytest, then commit as `feat: add fleet and network administration`. + +--- + +### Task 10: Move the hub safely to another enrolled PC + +**Purpose:** Transfer hub ownership without creating two authoritative hubs. + +**Files:** +* Create: `agent/fleet/announcements.py` +* Create: `agent/fleet/transfer.py` +* Create: `agent/routes/internal_transfer.py` +* Create: `agent/routes/v2_transfer.py` +* Modify: `agent/fleet/models.py` +* Modify: `agent/fleet/state.py` +* Modify: `agent/fleet/transport.py` +* Modify: `agent/main.py` +* Modify: `static/js/pages/fleet-settings.js` +* Create: `tests/test_hub_announcements.py` +* Create: `tests/test_hub_transfer.py` +* Create: `tests/test_transfer_rollback.py` +* Create: `tests/test_frontend_hub_transfer.py` + +**Interfaces:** +* Produces: `HubAnnouncement.sign(fleet_identity) -> HubAnnouncement` +* Produces: `verify_announcement(announcement, fleet_public_key, current_generation) -> bool` +* Produces: `HubTransferService.preflight(target_id) -> TransferPreflight` +* Produces: `HubTransferService.transfer(target_id) -> TransferResult` +* Produces: `POST /api/v2/fleet/hub-transfer/preflight` and `/execute` + +Keep the authority switch behind one method: + +```python +async def transfer(self, target_id: UUID) -> TransferResult: + preflight = await self.preflight(target_id) + preflight.require_ready() + staged = await self.stage_encrypted_snapshot(preflight.target) + announcement = await self.prepare_new_generation(preflight.target) + await self.target_client.stage_pending_hub(staged, announcement) + try: + await self.target_client.commit_unannounced_hub(announcement) + await self.target_client.require_ready(announcement) + except BaseException: + await self.target_client.abort_pending_hub(announcement) + raise + async with transaction(self.database): + await self.sessions.revoke_all() + await self.state.retire_local_hub(announcement, remove_signing_key=True) + await self.target_client.publish_hub(announcement) + return TransferResult(announcement=announcement) +``` + +- [ ] **Step 1: Write generation, rollback, and interruption tests** + +Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. + +- [ ] **Step 2: Implement signed announcements** + +Canonicalize announcement JSON with sorted keys and compact separators before signing. Store the accepted generation atomically with new hub addresses. + +- [ ] **Step 3: Implement encrypted transfer staging** + +Create a SQLite backup through the backup API. Package fleet config, owner hash, registry, history, integrations, and fleet signing key. Exclude sessions. Encrypt to the target X25519 public key with HKDF-SHA256 and AES-256-GCM. + +- [ ] **Step 4: Implement the staged transfer and explicit abort path** + +The target imports into staging, persists the signed proposed generation, and starts in unannounced hub mode on its configured private listener. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. + +If staging, activation, or readiness fails, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active. Only after readiness succeeds does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. The target then publishes the already signed announcement. A restarted old hub cannot mint a later generation. A crash after local retirement is handled by Task 11 and must never restore the removed key. + +- [ ] **Step 5: Add Move Hub UI and evidence** + +Show preflight results, transfer progress, exact new IP and port, and a link to open it. Capture successful transfer and forced rollback. Canary by failing the target validation probe and confirming the old hub remains active. + +- [ ] **Step 6: Verify and commit** + +Run focused transfer tests, full pytest, static assets, and the task surface. Commit as `feat: move fleet hub between enrolled PCs`. + +--- + +### Task 11: Recover a dead hub from encrypted bundles + +**Purpose:** Promote another PC without continuous replication. + +**Files:** +* Create: `agent/fleet/recovery.py` +* Create: `agent/routes/internal_recovery.py` +* Create: `agent/routes/v2_recovery.py` +* Create: `static/js/pages/recovery.js` +* Modify: `agent/fleet/models.py` +* Modify: `agent/fleet/transport.py` +* Modify: `agent/main.py` +* Modify: `static/index.html` +* Create: `tests/test_recovery_bundle.py` +* Create: `tests/test_dead_hub_recovery.py` +* Create: `tests/test_recovery_history_limits.py` +* Create: `tests/test_frontend_recovery.py` + +**Interfaces:** +* Produces: `RecoveryService.create_bundle() -> EncryptedRecoveryBundle` +* Produces: `RecoveryService.store_replica(bundle) -> None` +* Produces: `RecoveryService.inspect(recovery_key: str) -> RecoveryPreview` +* Produces: `RecoveryService.promote(recovery_key: str, owner: NewOwner) -> HubAnnouncement` +* Produces: `RecoveryService.run_replication_loop(interval: timedelta = timedelta(hours=6)) -> None` + +Recovery decryption must authenticate before parsing: + +```python +def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> RecoveryPayload: + private_key = decode_checked_recovery_key(recovery_key) + shared = private_key.exchange(bundle.ephemeral_public_key) + key = HKDF(algorithm=hashes.SHA256(), length=32, salt=bundle.salt, info=b"humwatch-recovery-v1").derive(shared) + plaintext = AESGCM(key).decrypt(bundle.nonce, bundle.ciphertext, bundle.associated_data()) + return RecoveryPayload.model_validate_json(plaintext) +``` + +- [ ] **Step 1: Write cryptographic and recovery tests** + +Cover wrong keys, tampering, newest-valid selection, generation increment, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. + +- [ ] **Step 2: Implement bundle creation and replication** + +Serialize fleet identity, machine registry, settings, integrations, schema version, generation, and timestamp. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. Copy the ciphertext to enrolled agents through signed internal requests. + +Start `run_replication_loop()` from the hub lifespan. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. Creating a bundle upserts one `recovery_deliveries` row per active agent in the same transaction. On conflict at `(generation, machine_id)`, an unchanged bundle digest preserves its delivery state. A changed digest atomically replaces the digest and resets state to `pending`, attempt count to zero, next attempt to now, and acknowledgement to null. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. A signed agent acknowledgement marks that machine and generation acknowledged only when its digest matches the current row. Startup resumes every pending or due row. Agents atomically replace only an older valid replica or a different authenticated bundle for the same generation, retain the newest valid bundle across restart, and return the accepted generation plus bundle digest so the hub cannot acknowledge the wrong ciphertext. + +- [ ] **Step 3: Implement local recovery preview and promotion** + +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Promotion creates a new owner password, revokes integrations, raises generation, switches to hub mode, and publishes a signed announcement. + +- [ ] **Step 4: Add the Recover Fleet page and evidence** + +Keep recovery copy literal. Capture valid preview, invalid key, recovery confirmation, and post-recovery login. Canary by flipping one ciphertext byte and proving preview refuses it. + +- [ ] **Step 5: Verify and commit** + +Run focused recovery tests, full pytest, static assets, and browser smoke. Commit as `feat: recover fleets after hub loss`. + +--- + +### Task 12: Add scoped integration pairing for future SlipShell monitoring + +**Purpose:** Ship the HumWatch half of the integration without adding SlipShell UI work. + +**Files:** +* Create: `agent/fleet/integrations.py` +* Create: `agent/routes/v2_integrations.py` +* Create: `agent/security/integration_auth.py` +* Create: `tests/test_integration_pairing.py` +* Create: `tests/test_integration_scopes.py` +* Create: `tests/test_slipshell_contract.py` +* Create: `tests/fixtures/slipshell/contract.json` +* Modify: `agent/fleet/models.py` +* Modify: `agent/main.py` +* Modify: `agent/routes/v2_fleet.py` +* Modify: `agent/routes/v2_history.py` +* Modify: `agent/routes/v2_events.py` +* Modify: `agent/routes/v2_machines.py` +* Modify: `static/js/pages/fleet-settings.js` +* Modify: `static/js/api.js` +* Modify: `static/vendor/SHA256SUMS` + +**Interfaces:** +* Produces: `POST /api/v2/integrations/pairing-code` +* Produces: `POST /api/v2/integrations/exchange` +* Produces: `GET /api/v2/integrations` +* Produces: `DELETE /api/v2/integrations/{integration_id}` +* Produces read-only scopes: `fleet:read`, `machines:read`, `metrics:read`, `history:read`, `events:read` + +Public read routes accept either an owner session or the named scope: + +```python +def owner_or_scope(required_scope: IntegrationScope): + async def dependency(request: Request) -> RequestPrincipal: + session = await optional_owner_session(request) + if session is not None: + return RequestPrincipal.owner(session) + integration = await verify_integration_request(request) + integration.require_scope(required_scope) + return RequestPrincipal.integration(integration) + return dependency +``` + +- [ ] **Step 1: Write pairing and scope tests** + +Assert ten-minute one-use pairing, independent credential revocation, no owner or agent credential reuse, read-only enforcement, stable machine UUIDs, and access to current, history, availability, and events. Exercise every existing Task 6 route through its real router with an owner session, its required integration scope, a missing scope, and a revoked integration. + +- [ ] **Step 2: Implement integration request authentication** + +Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. + +- [ ] **Step 3: Publish a machine-readable contract fixture** + +Add one versioned `tests/fixtures/slipshell/contract.json` containing fleet discovery, machine current state, history, availability, and event frames. `tests/test_slipshell_contract.py` validates every fixture against the live Pydantic response models and asserts UUID, timestamp, units, stale threshold, gap range, and backfill field names. This is the stable HumWatch half of the future SlipShell integration. + +- [ ] **Step 4: Add integration administration UI and evidence** + +Show paired clients, scopes, creation time, and revoke. Generate a short-lived pairing code without exposing stored credentials. Canary by attempting a write with an integration identity and require `403`. + +- [ ] **Step 5: Verify and commit** + +Run focused integration tests, v2 API tests, full pytest, static assets, and browser smoke. Commit as `feat: add read-only monitoring integrations`. + +--- + +### Task 13: Migrate 2.0 installations through an explicit wizard + +**Purpose:** Convert existing nodes without silently keeping the broken security and peer model. + +**Files:** +* Create: `agent/fleet/upgrade.py` +* Create: `agent/routes/v2_migration.py` +* Create: `static/js/pages/migration.js` +* Create: `tests/test_fleet_upgrade.py` +* Create: `tests/test_legacy_api_compatibility.py` +* Create: `tests/test_frontend_migration.py` +* Create: `tests/fixtures/v2_0_database.py` +* Modify: `agent/fleet/state.py` +* Modify: `agent/migrations.py` +* Modify: `agent/main.py` +* Modify: `agent/routes/health.py` +* Modify: `static/index.html` +* Modify: `static/js/app.js` +* Modify: `static/js/auth.js` +* Modify: `static/js/pages/setup.js` +* Modify: `static/vendor/SHA256SUMS` +* Modify: `tests/test_database_migration.py` + +**Interfaces:** +* Produces: `UpgradeService.inspect() -> UpgradePreview` +* Produces: `UpgradeService.create_fleet(owner: NewOwner) -> UpgradeResult` +* Produces: `UpgradeService.prepare_join() -> PairingDisplay` +* Produces: `UpgradeService.enable_legacy_api(enabled: bool) -> None` + +Inspection returns proof, not a boolean: + +```python +class UpgradePreview(BaseModel): + source_database: str + source_sha256: str + metric_rows: int + process_rows: int + oldest_sample: datetime | None + newest_sample: datetime | None + archive_path: str + safe_to_continue: bool + blocking_errors: list[str] +``` + +- [ ] **Step 1: Build a real 2.0 fixture and continuity tests** + +The fixture contains machine information, metrics, process history, token paths, certificate paths, and CORS origins. Assert row counts and sample values before and after Create Fleet and Join Fleet conversions. + +- [ ] **Step 2: Implement safety inspection and archive** + +Use SQLite backup, validate schema and machine rows, copy config and security path metadata, and record a manifest with source paths and SHA256 digests. Stop migration with a visible error when proof fails. + +- [ ] **Step 3: Implement Create Fleet and Join Fleet conversions** + +Create Fleet imports history under the stable self machine UUID. Join Fleet converts history to ordered outbox batches and uploads through Task 5. Neither path deletes original runtime files. + +- [ ] **Step 4: Gate protected v1 endpoints by mode** + +Fresh installations have only public v1 health. Upgraded installations default protected v1 off, can explicitly enable it with the archived bearer token, and can remove compatibility permanently. + +- [ ] **Step 5: Implement the migration wizard and peer suggestions** + +Read `humwatch_machines` from browser local storage as untrusted enrollment suggestions. Clear it only after owner confirmation. Present Create Fleet and Join Fleet with data counts and exact archive location. + +- [ ] **Step 6: Produce migration evidence and canary** + +Run one Create Fleet and one Join Fleet conversion against copied 2.0 fixtures. Canary by corrupting the backup and proving the source remains untouched and the wizard blocks completion. + +- [ ] **Step 7: Verify and commit** + +Run focused migration tests, all v1 and v2 auth tests, full pytest, static assets, and browser smoke. Commit as `feat: migrate v2 nodes into fleets`. + +--- + +### Task 14: Transition Windows and Linux installers to the fleet model + +**Purpose:** Make fresh setup boring and detect duplicate HumWatch services without harming unrelated processes. + +**Files:** +* Modify: `installer/service-setup.ps1` +* Modify: `installer/HumWatch.iss` +* Modify: `installer/linux/humwatch.service` +* Modify: `scripts/install-service.ps1` +* Modify: `scripts/provision-security.ps1` +* Modify: `scripts/provision-security.sh` +* Modify: `scripts/build-deb.sh` +* Modify: `run.bat` +* Modify: `run-no-admin.bat` +* Modify: `run.sh` +* Modify: `tests/test_windows_service_security.py` +* Modify: `tests/test_linux_service_security.py` +* Modify: `tests/test_security_provisioning.py` +* Modify: `tests/test_installation_docs.py` +* Create: `tests/test_duplicate_service_detection.py` +* Modify: `.github/workflows/build-windows-installer.yml` + +**Interfaces:** +* Produces installer state under the existing protected runtime root +* Produces `Find-HumWatchServiceConflicts -Port 9100` and `Disable-ObsoleteHumWatchService` +* Produces `find_humwatch_service_conflicts 9100` for Linux package post-install + +The Windows detector returns records and never mutates during discovery: + +```powershell +function Find-HumWatchServiceConflicts { + param([int]$Port = 9100) + $services = Get-CimInstance Win32_Service | + Where-Object { $_.Name -like '*HumWatch*' } | + ForEach-Object { + [pscustomobject]@{ + Kind = 'service' + Name = $_.Name + ProcessId = $_.ProcessId + State = $_.State + PathName = $_.PathName + KnownHumWatchPath = Test-KnownHumWatchPath $_.PathName + } + } + $owners = Get-NetTCPConnection -State Listen -LocalPort $Port -ErrorAction SilentlyContinue | + ForEach-Object { + $process = Get-Process -Id $_.OwningProcess -ErrorAction SilentlyContinue + [pscustomobject]@{ + Kind = 'port-owner' + Port = $Port + ProcessId = $_.OwningProcess + ProcessName = $process.ProcessName + PathName = $process.Path + KnownHumWatchPath = Test-KnownHumWatchPath $process.Path + } + } + @($services) + @($owners) +} +``` + +The Linux detector parses `systemctl --user`, system `systemctl`, and `ss -H -ltnp "sport = :$port"` into the same service and port-owner concepts. Treat missing process metadata as unknown, never as HumWatch. + +- [ ] **Step 1: Write installer behavior tests first** + +Fresh install must create identity directories, no bearer token, no certificate, and `unconfigured` mode. Upgrade must preserve the archive. Duplicate detection must identify service name, executable path, user or system scope, and port conflict. + +- [ ] **Step 2: Remove mandatory security provisioning from fresh setup** + +Keep protected directories and ACLs. Stop generating token and self-signed certificate files by default. Preserve explicit operator TLS files during upgrades. + +- [ ] **Step 3: Add duplicate service detection with confirmation** + +Discovery never mutates. Offer to disable only a service whose executable or working directory resolves below a known HumWatch installation root. Port ownership alone is never enough. Report unrelated and unknown port owners with PID, executable when readable, and scope without terminating or disabling them. Tests cover a renamed HumWatch service on 9100, a stale known service on another port, and an unrelated process on 9100 on both platforms. + +- [ ] **Step 4: Update CI installation contracts** + +Exercise fresh Windows portable, Windows service, Linux package, upgrade preservation, duplicate user plus system service, and unrelated port owner cases. + +- [ ] **Step 5: Run packaging evidence and canary** + +Build the deb and Windows installer workflow fixtures. Canary with a fake unrelated process on 9100 and prove setup reports it without stopping it. + +- [ ] **Step 6: Verify and commit** + +Run installer-focused tests, package builds, full pytest, workflow YAML parsing, and script syntax checks. Commit as `feat: install fleet-ready HumWatch nodes`. + +--- + +### Task 15: Prove the complete 2.1.0 flow and prepare the release + +**Purpose:** Validate the product on real Windows and Linux machines before changing the version. + +**Files:** +* Create: `scripts/verify-fleet-e2e.py` +* Create: `tests/test_fleet_e2e_harness.py` +* Create: `docs/2.1-UPGRADE.md` +* Create: `docs/2.1-ACCEPTANCE.md` +* Modify: `README.md` +* Create: `docs/START-HERE.md` +* Modify: `HumWatch-Spec.md` +* Modify: `agent/__init__.py` +* Modify: `tests/test_installation_docs.py` +* Modify: `tests/test_security_contract.py` +* Modify: `tests/test_supply_chain_policy.py` +* Modify: `.github/workflows/build-windows-installer.yml` +* Modify: `scripts/build-release.ps1` +* Modify: `scripts/build-linux-portable.sh` +* Modify: `scripts/verify-static-assets.py` +* Modify: `static/vendor/SHA256SUMS` + +**Interfaces:** +* Produces: `python scripts/verify-fleet-e2e.py --hub URL --owner USER` +* Produces: release version `2.1.0` + +The verifier emits fixed sanitized checkpoints: + +```python +CHECKPOINTS = ( + "login", + "enrollment", + "live-report", + "machine-switch", + "offline-buffer", + "ordered-backfill", + "hub-transfer", + "recovery-preview", + "integration-read", + "revocation", +) +``` + +- [ ] **Step 1: Write the deterministic E2E harness test** + +The harness accepts secrets through hidden prompts or protected files, never command arguments. It prints sanitized checkpoints for login, enrollment, live report, machine switch, offline buffer, ordered backfill, transfer, recovery preview, integration read, and revocation. + +- [ ] **Step 2: Run the complete automated gate** + +```bash +.venv/bin/python -m pytest -q +.venv/bin/python scripts/verify-static-assets.py +node tests/browser_globals_smoke.js +python -m compileall -q agent scripts +bash -n run.sh setup.sh scripts/provision-security.sh scripts/build-deb.sh scripts/build-linux-portable.sh +``` + +- [ ] **Step 3: Prove the failure canaries** + +Corrupt one static digest and require static verification failure. Tamper with one signed report and require ingestion rejection. Corrupt one recovery bundle and require preview failure. Restore all fixtures and rerun the green gate. + +- [ ] **Step 4: Run real Windows and Linux acceptance** + +Use at least one Windows PC and one Linux PC. Record: + +1. Hub creation over ordinary LAN. +2. Agent enrollment using one temporary code. +3. Optional Tailscale reconnection. +4. One browser login and in-app machine switching. +5. Hub outage, continued collection, and ordered backfill. +6. Planned hub transfer to the other platform. +7. Controlled dead hub recovery. +8. Real 2.0 database upgrade with sample continuity. +9. Independent machine and integration revocation. +10. HTTP default plus optional HTTPS. + +Save logs and screenshots below `artifacts/2.1.0-acceptance/`. Redact addresses only if the artifact will be public. Never include secrets. + +- [ ] **Step 5: Rewrite setup and security documentation** + +Make IP plus port, owner login, Create Fleet, Join Fleet, recovery key, optional Tailscale, and optional HTTPS the primary flow. Move shared bearer token, CORS, certificate trust, and peer browser calls into a clearly labeled 2.0 compatibility section. + +- [ ] **Step 6: Change the version only after acceptance passes** + +Set `agent.__version__` to `2.1.0`, rebuild release artifacts, and rerun the complete gate. + +- [ ] **Step 7: Complete the review receipt and release PR** + +Run the independent review loop against current HEAD until zero new findings. Write the receipt with the real acceptance evidence directory. Run gitleaks. Push and open a ready-for-review PR. Do not create the `v2.1.0` tag until CI, Windows installer proof, Linux package proof, and the receipt all pass on the exact release commit. + +Commit message: `release: prepare HumWatch 2.1.0` + +## Per-task completion gate + +Every task ends with this sequence after its implementation commit: + +1. Run all focused tests named in the task. +2. Run the full pytest suite. +3. Run static asset and browser checks when browser files changed. +4. Run package or script checks when installer files changed. +5. Produce running surface evidence for every touched UI or endpoint. +6. Prove the relevant checker with one deliberate failure, then restore and rerun. +7. Run one independent review against current HEAD. +8. Fix every actionable finding and commit. +9. Repeat review against the new HEAD until zero new findings. +10. Write `/review-receipt.json` with current HEAD and evidence path. +11. Run `~/.local/bin/gitleaks git --log-opts="--all --not --remotes"`. +12. Push and open a ready-for-review PR. + +Do not carry a receipt across a fix commit. Do not treat a merged dependency PR as proof that the next task passes. Each task begins from freshly fetched and verified `origin/main` after its dependencies merge. + +## New-session reentry + +The implementation session should begin with: + +```bash +cd /projects/HumWatch +git fetch origin +git status --short +git worktree list --porcelain +``` + +Then inspect current open PRs and issue #14. The planning branch is `codex/humwatch-2.1-fleet-hub-design` at or after commit `32fc1b4`. Do not implement on the existing `docs/start-here-guide` checkout. Create a fresh worktree and branch for Task 1 from the latest merged `origin/main` after the design and plan PR is merged. + +Open PR #16 changes the old token gate. Recheck it before Task 7. If it is still open and Task 7 replaces that gate, close it as superseded instead of carrying dead CSS into 2.1.0. Open PR #15 documents the 2.0 setup path. Recheck it before Task 15 and rewrite any merged content that still makes mandatory TLS or shared tokens the normal path. From 6a2c28e1242bda9b14c537006a2274d3ce0c414d Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 09:25:12 -0400 Subject: [PATCH 03/65] chore: document gitleaks false positives --- .gitleaksignore | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .gitleaksignore diff --git a/.gitleaksignore b/.gitleaksignore new file mode 100644 index 0000000..4de41a4 --- /dev/null +++ b/.gitleaksignore @@ -0,0 +1,2 @@ +df36c1d8d28aec103787fa84096bc1f3fd853fab:docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md:generic-api-key:399 +df36c1d8d28aec103787fa84096bc1f3fd853fab:docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md:generic-api-key:400 From 581b0e557543eec5dec976ce17da89130a5ec8f4 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 11:01:54 -0400 Subject: [PATCH 04/65] docs: close fleet hub design P1 gaps --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 39 +++++++++++-------- ...026-08-11-humwatch-2.1-fleet-hub-design.md | 11 +++--- 2 files changed, 29 insertions(+), 21 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index c78d71e..6bc4d24 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -354,7 +354,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -367,11 +367,14 @@ data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) used_nonces(machine_id, nonce_hash, expires_at) enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) -recovery_bundles(generation, created_at, encrypted_payload) -recovery_deliveries(generation, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) +hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) +hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) +hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) +recovery_bundles(generation, bundle_revision, created_at, encrypted_payload) +recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` -Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, and `(generation, machine_id)` for recovery delivery. Check constraints require ordered sequence ranges, nonnegative attempts, and known state values. +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. It also owns the transaction helper used by every fleet repository: @@ -407,7 +410,7 @@ class MachineIdentity: - [ ] **Step 5: Detect the role before applying any network security gate** -Move role detection ahead of the current unconditional `validate_security_config()` calls in `agent/main.py`. Existing databases with an auth token start as `legacy`. Fresh databases start as `unconfigured`. Unconfigured mode binds HTTP only to `127.0.0.1` and does not require a token or certificate. Legacy mode keeps the existing token, TLS, and CORS validation exactly. Hub and agent modes use loopback or explicitly stored private listeners until Task 9 adds the settings surface. No role may fall back to `0.0.0.0`. +Move role detection ahead of the current unconditional `validate_security_config()` calls in `agent/main.py`. Existing databases with an auth token start as `legacy`. Fresh databases start as `unconfigured`. Unconfigured mode starts its setup surface on `127.0.0.1` and does not require a token or certificate. Task 4 adds a loopback-selected, restricted private enrollment listener before its hub-orchestrated claim runs. That listener may expose only `/api/internal/v1/enrollment/claim` and `/finalize`, never the setup page, pairing-code endpoint, dashboard, or any v1 route. Legacy mode keeps the existing token, TLS, and CORS validation exactly. Hub and agent modes use loopback or explicitly stored private listeners until Task 9 adds the owner-managed settings surface. No role may fall back to `0.0.0.0`. `hub` starts collector, retention, and hub services. `agent` starts collector, retention, and agent transport. Until those services exist, role branches log the selected mode and start only the services already implemented for that role. Add startup tests proving a fresh config with no token and no TLS reaches the unconfigured app, while an unsafe legacy config still fails closed. @@ -552,6 +555,7 @@ git commit -m "feat: add owner sessions and fleet creation" * Produces: `verify_internal_request(request, machine_record) -> None` * Produces: hub `POST /api/v2/fleet/machines/enroll` * Produces: local-only `POST /api/v2/setup/pairing-code` +* Produces: local-only `GET/PUT /api/v2/setup/enrollment-listeners` * Produces: agent `POST /api/internal/v1/enrollment/claim` and `/finalize` Use this one canonical implementation boundary: @@ -580,7 +584,7 @@ async def enroll_machine(address: str, code: str) -> EnrolledMachine: - [ ] **Step 1: Write signing and enrollment protocol tests** -Cover body tampering, clock skew, nonce replay, expired code, reused code, private-address validation, interrupted finalization, a hub crash after its durable claim record, idempotent finalize retry, first-report pending status, and fleet or machine header relabeling with an otherwise valid signature. +Cover body tampering, clock skew, nonce replay, expired code, reused code, loopback-only listener selection, selected-private-listener binding, rejection of every non-enrollment route on that listener, private-address validation, interrupted finalization, a hub crash after its durable claim record, idempotent finalize retry, first-report pending status, and fleet or machine header relabeling with an otherwise valid signature. ```python def test_pairing_code_expires_after_ten_minutes(clock, enrollment): @@ -598,9 +602,10 @@ Run: `.venv/bin/python -m pytest tests/test_internal_request_signing.py tests/te Keep canonicalization in one module. Reject duplicate signature headers and decode base64url strictly. Store nonce digests until their 120-second replay window expires. -- [ ] **Step 4: Implement local-only code display and network claim endpoints** -`create_code()` is callable only from the local setup surface. Reject a request to `/api/v2/setup/pairing-code` unless the actual peer address is loopback, including when forwarding headers claim loopback. No HTTP response from `/api/internal/v1` returns the code. Claim binds the `enrollment_id`, hub claim digest, code, and agent claim. A retry with the same enrollment ID and identical hub claim returns the same `AgentClaim`. Any changed claim under that ID fails. +- [ ] **Step 4: Implement local-only listener selection, code display, and network claim endpoints** + +The local setup surface first reads private LAN and Tailscale candidates, then writes the exact selected bootstrap listeners through `PUT /api/v2/setup/enrollment-listeners`. Reject that route unless the actual peer address is loopback, including when forwarding headers claim loopback. Rebind only those selected private addresses before the hub can invoke claim, and reject every path there except `/api/internal/v1/enrollment/claim` and `/finalize`. `create_code()` is likewise callable only from the local setup surface. No HTTP response from `/api/internal/v1` returns the code. Claim binds the `enrollment_id`, hub claim digest, code, and agent claim. A retry with the same enrollment ID and identical hub claim returns the same `AgentClaim`. Any changed claim under that ID fails. Task 9 later replaces this bootstrap-only selection with the persisted owner-managed listener settings. - [ ] **Step 5: Implement hub-orchestrated enrollment** @@ -1024,16 +1029,18 @@ async def transfer(self, target_id: UUID) -> TransferResult: except BaseException: await self.target_client.abort_pending_hub(announcement) raise + await self.state.enter_draining_transfer(announcement) + publication = await self.target_client.activate_publication_obligation(announcement) + await self.target_client.require_publication_persisted(publication) async with transaction(self.database): await self.sessions.revoke_all() await self.state.retire_local_hub(announcement, remove_signing_key=True) - await self.target_client.publish_hub(announcement) return TransferResult(announcement=announcement) ``` - [ ] **Step 1: Write generation, rollback, and interruption tests** -Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. +Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. - [ ] **Step 2: Implement signed announcements** @@ -1045,9 +1052,9 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, - [ ] **Step 4: Implement the staged transfer and explicit abort path** -The target imports into staging, persists the signed proposed generation, and starts in unannounced hub mode on its configured private listener. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -If staging, activation, or readiness fails, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active. Only after readiness succeeds does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. The target then publishes the already signed announcement. A restarted old hub cannot mint a later generation. A crash after local retirement is handled by Task 11 and must never restore the removed key. +If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. - [ ] **Step 5: Add Move Hub UI and evidence** @@ -1079,7 +1086,7 @@ Run focused transfer tests, full pytest, static assets, and the task surface. Co **Interfaces:** * Produces: `RecoveryService.create_bundle() -> EncryptedRecoveryBundle` -* Produces: `RecoveryService.store_replica(bundle) -> None` +* Produces: `RecoveryService.store_replica(bundle) -> ReplicaAcceptance` * Produces: `RecoveryService.inspect(recovery_key: str) -> RecoveryPreview` * Produces: `RecoveryService.promote(recovery_key: str, owner: NewOwner) -> HubAnnouncement` * Produces: `RecoveryService.run_replication_loop(interval: timedelta = timedelta(hours=6)) -> None` @@ -1097,13 +1104,13 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, newest-valid selection, generation increment, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, newest-valid selection by `(generation, bundle_revision)`, generation increment, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. - [ ] **Step 2: Implement bundle creation and replication** -Serialize fleet identity, machine registry, settings, integrations, schema version, generation, and timestamp. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. Copy the ciphertext to enrolled agents through signed internal requests. +Serialize fleet identity, machine registry, settings, integrations, schema version, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind both generation and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. Copy the ciphertext to enrolled agents through signed internal requests. -Start `run_replication_loop()` from the hub lifespan. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. Creating a bundle upserts one `recovery_deliveries` row per active agent in the same transaction. On conflict at `(generation, machine_id)`, an unchanged bundle digest preserves its delivery state. A changed digest atomically replaces the digest and resets state to `pending`, attempt count to zero, next attempt to now, and acknowledgement to null. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. A signed agent acknowledgement marks that machine and generation acknowledged only when its digest matches the current row. Startup resumes every pending or due row. Agents atomically replace only an older valid replica or a different authenticated bundle for the same generation, retain the newest valid bundle across restart, and return the accepted generation plus bundle digest so the hub cannot acknowledge the wrong ciphertext. +Start `run_replication_loop()` from the hub lifespan. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple has a higher generation or the same generation with a higher revision. They reject an older or equal revision with a different digest, retain the newest valid bundle across restart, and return the accepted generation, revision, and digest so the hub cannot acknowledge stale ciphertext. - [ ] **Step 3: Implement local recovery preview and promotion** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 3651c0f..199b9e8 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -312,11 +312,11 @@ Transfer then: 5. Excludes browser sessions. 6. Starts the target in pending hub mode. 7. Increments the hub generation and signs the new hub announcement. -8. Sends the announcement to online agents. -9. Confirms that the target accepts signed reports and serves the owner login. -10. Demotes the old hub to agent state. +8. Confirms that the target accepts signed reports and serves the owner login. +9. Makes the old hub durably enter a draining state, then has the target persist and activate a retryable announcement-delivery obligation. +10. Demotes the old hub to agent state only after the target acknowledges that durable publication obligation. -If validation fails before demotion, the old hub remains authoritative and the target discards pending hub state. +If validation fails before draining, the old hub remains authoritative and the target discards pending hub state. Once draining starts, the old hub cannot resume normal hub service until it either completes the target-owned publication path or sends a signed abort. A lost activation response is reconciled by the announcement digest and never permits a blind abort of the target. A target restart resumes the persisted delivery obligation, so a crash after old-hub retirement cannot leave agents permanently pointed at the retired hub. ### Split brain prevention @@ -342,6 +342,7 @@ The hub periodically creates a compact recovery bundle containing: * Listener and fleet settings. * Integration registration records. * Schema and bundle version. +* A monotonically increasing bundle revision within the current hub generation. * Creation timestamp. The recovery bundle does not contain browser sessions or full central history. @@ -352,7 +353,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 1. Choose **Recover Fleet** on another installation. 2. Enter the printable recovery key. -3. Select the newest valid recovery bundle found locally. +3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Promote the PC to hub with a higher generation. 6. Revoke browser sessions and integration credentials. From 7f4d54aae686acbddaaf6b8b5ac4db1751efc495 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 13:33:23 -0400 Subject: [PATCH 05/65] docs: close the second-round fleet hub P1 gaps Create Fleet now stores and binds the first private hub listener, the recovery key survives a lost setup response behind an acknowledge-to-delete pending record, hub transfer freezes report ingestion before the snapshot so no acknowledged sample can miss the target, and dead-hub promotion picks a generation above both the bundle and the locally accepted announcement. Co-Authored-By: Claude Fable 5 --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 23 ++++++++++++------- ...026-08-11-humwatch-2.1-fleet-hub-design.md | 11 +++++---- 2 files changed, 21 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 6bc4d24..5b55dae 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -410,7 +410,7 @@ class MachineIdentity: - [ ] **Step 5: Detect the role before applying any network security gate** -Move role detection ahead of the current unconditional `validate_security_config()` calls in `agent/main.py`. Existing databases with an auth token start as `legacy`. Fresh databases start as `unconfigured`. Unconfigured mode starts its setup surface on `127.0.0.1` and does not require a token or certificate. Task 4 adds a loopback-selected, restricted private enrollment listener before its hub-orchestrated claim runs. That listener may expose only `/api/internal/v1/enrollment/claim` and `/finalize`, never the setup page, pairing-code endpoint, dashboard, or any v1 route. Legacy mode keeps the existing token, TLS, and CORS validation exactly. Hub and agent modes use loopback or explicitly stored private listeners until Task 9 adds the owner-managed settings surface. No role may fall back to `0.0.0.0`. +Move role detection ahead of the current unconditional `validate_security_config()` calls in `agent/main.py`. Existing databases with an auth token start as `legacy`. Fresh databases start as `unconfigured`. Unconfigured mode starts its setup surface on `127.0.0.1` and does not require a token or certificate. Task 4 adds a loopback-selected, restricted private enrollment listener before its hub-orchestrated claim runs. That listener may expose only `/api/internal/v1/enrollment/claim` and `/finalize`, never the setup page, pairing-code endpoint, dashboard, or any v1 route. Legacy mode keeps the existing token, TLS, and CORS validation exactly. Hub and agent modes use loopback or explicitly stored private listeners until Task 9 adds the owner-managed settings surface, and Task 3's Create Fleet stores and binds the first private hub listener inside its transaction, so a new hub is reachable for enrollment claims and agent reports from the moment the fleet exists rather than waiting for Task 9. No role may fall back to `0.0.0.0`. `hub` starts collector, retention, and hub services. `agent` starts collector, retention, and agent transport. Until those services exist, role branches log the selected mode and start only the services already implemented for that role. Add startup tests proving a fresh config with no token and no TLS reaches the unconfigured app, while an unsafe legacy config still fails closed. @@ -461,11 +461,12 @@ git commit -m "feat: add fleet runtime foundation" * Produces: `require_owner_session(request: Request) -> OwnerSession` * Produces: `require_csrf(request: Request, session: OwnerSession) -> None` * Produces: `POST /api/v2/setup/create-fleet` +* Produces: `GET /api/v2/setup/recovery-key`, `POST /api/v2/setup/recovery-key/ack` * Produces: `POST /api/v2/session/login`, `GET /me`, `POST /logout`, `GET /sessions`, `DELETE /sessions/{session_hash}` - [ ] **Step 1: Write failing password, session, and setup tests** -Test `hashlib.scrypt`, parameter persistence, rehash after successful login from an older parameter version, generic login failures, a 30-day expiry, `HttpOnly`, `SameSite=Strict`, conditional `Secure`, CSRF rejection, logout, individual revocation, all-session revocation, and loopback-only fleet creation. +Test `hashlib.scrypt`, parameter persistence, rehash after successful login from an older parameter version, generic login failures, a 30-day expiry, `HttpOnly`, `SameSite=Strict`, conditional `Secure`, CSRF rejection, logout, individual revocation, all-session revocation, and loopback-only fleet creation. Also test that create-fleet stores and binds the requested private hub listener, rejects creation with no private address, that the recovery key remains retrievable through the pending-setup record after a simulated lost response, and that acknowledgement deletes it permanently. ```python def test_login_sets_server_side_session_cookie(hub_client, owner_password): @@ -502,7 +503,9 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. Create fleet identity, owner record, self machine row, and runtime role `hub` in one database transaction. Return the printable recovery key exactly once. +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub` in one database transaction, then bind the stored listener before the endpoint reports success, so enrollment claims and agent reports work without a restart. + +Do not return the printable recovery key as the transaction's only copy. Persist it in a protected pending-setup record inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. - [ ] **Step 6: Mount `/api/v2` without weakening v1** @@ -820,7 +823,7 @@ Remove bearer token storage and remote fetch helpers. Set `credentials: "same-or - [ ] **Step 4: Implement setup and login pages** -Create Fleet submits username and password and displays the recovery key once. Join Fleet requests a code from the local endpoint and displays the selected private addresses. Login accepts username and password. Recovery is a routed shell until Task 11 adds its backend. +Create Fleet submits username, password, and the hub listener selection pre-filled from the detected private addresses, then displays the recovery key until the owner confirms it is saved, which sends the acknowledgement that deletes the pending record. If the create response is lost, the setup surface offers the retrieval path from Task 3 instead of a dead end. Join Fleet requests a code from the local endpoint and displays the selected private addresses. Login accepts username and password. Recovery is a routed shell until Task 11 adds its backend. - [ ] **Step 5: Add the deterministic digest updater** @@ -1020,6 +1023,7 @@ Keep the authority switch behind one method: async def transfer(self, target_id: UUID) -> TransferResult: preflight = await self.preflight(target_id) preflight.require_ready() + await self.state.enter_ingestion_freeze() staged = await self.stage_encrypted_snapshot(preflight.target) announcement = await self.prepare_new_generation(preflight.target) await self.target_client.stage_pending_hub(staged, announcement) @@ -1028,6 +1032,7 @@ async def transfer(self, target_id: UUID) -> TransferResult: await self.target_client.require_ready(announcement) except BaseException: await self.target_client.abort_pending_hub(announcement) + await self.state.lift_ingestion_freeze() raise await self.state.enter_draining_transfer(announcement) publication = await self.target_client.activate_publication_obligation(announcement) @@ -1040,7 +1045,7 @@ async def transfer(self, target_id: UUID) -> TransferResult: - [ ] **Step 1: Write generation, rollback, and interruption tests** -Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. +Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Cover freeze lift after a signed abort and old-hub restart while frozen. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. - [ ] **Step 2: Implement signed announcements** @@ -1054,7 +1059,9 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. +The old hub durably enters an ingestion freeze before the snapshot is taken. The freeze rejects new report batches, so no acknowledgement watermark can advance past the snapshot and every sample the old hub ever acknowledged is present on the target. Agents treat the rejection like any hub outage: batches stay in the local outbox and delivery resumes against whichever hub wins. Any failure before `enter_draining_transfer()` commits, including a snapshot or staging failure, lifts the freeze when the old hub resumes normal service, and an old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. + +If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. - [ ] **Step 5: Add Move Hub UI and evidence** @@ -1104,7 +1111,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, newest-valid selection by `(generation, bundle_revision)`, generation increment, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. - [ ] **Step 2: Implement bundle creation and replication** @@ -1114,7 +1121,7 @@ Start `run_replication_loop()` from the hub lifespan. Create and distribute a bu - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Promotion creates a new owner password, revokes integrations, raises generation, switches to hub mode, and publishes a signed announcement. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Promotion creates a new owner password, revokes integrations, switches to hub mode, and publishes a signed announcement at `max(bundle_generation, locally_accepted_generation) + 1`. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 199b9e8..aee12e2 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -129,9 +129,10 @@ Creating a fleet generates: * A fleet signing key pair. * A recovery encryption key pair. * Hub generation `1`. -* A printable recovery key that encodes the recovery private key and is shown once to the owner. +* A stored private hub listener, selected from the machine's detected private addresses and bound before setup reports success, so enrollment and agent reports are reachable immediately. +* A printable recovery key that encodes the recovery private key. It is shown to the owner until they confirm it is saved. That acknowledgement deletes the protected pending record holding it, and a setup response lost in transit is recovered by logging in at the hub PC and retrieving the key before acknowledging. -Agents retain the fleet public signing key and recovery public key. Only the active hub holds the fleet private signing key in usable form. HumWatch does not retain the recovery private key. +Agents retain the fleet public signing key and recovery public key. Only the active hub holds the fleet private signing key in usable form. After the owner's acknowledgement, HumWatch does not retain the recovery private key. ### Machine identity @@ -305,7 +306,7 @@ Preflight verifies: Transfer then: -1. Pauses fleet configuration changes. +1. Pauses fleet configuration changes and report ingestion, so no acknowledgement can advance past the snapshot. Agents buffer rejected batches in their local outboxes, and a failed transfer lifts the pause. 2. Creates a consistent database and fleet state snapshot. 3. Encrypts the transfer to the target machine's X25519 public key with an ephemeral key and authenticated encryption. 4. Transfers owner hash, fleet registry, history, settings, integration records, and the fleet signing key. @@ -347,7 +348,7 @@ The hub periodically creates a compact recovery bundle containing: The recovery bundle does not contain browser sessions or full central history. -Each bundle is encrypted to the recovery public key with an ephemeral X25519 key, key derivation, and authenticated encryption. The printable recovery key holds the corresponding private key. It is shown once and is never stored by HumWatch. The encrypted bundle is replicated to enrolled agents. +Each bundle is encrypted to the recovery public key with an ephemeral X25519 key, key derivation, and authenticated encryption. The printable recovery key holds the corresponding private key. It is shown until the owner confirms it is saved and is never retained after that acknowledgement. The encrypted bundle is replicated to enrolled agents. ### Recovery flow @@ -355,7 +356,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 2. Enter the printable recovery key. 3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. -5. Promote the PC to hub with a higher generation. +5. Promote the PC to hub at a generation above both the bundle's generation and the generation this machine last accepted from an announcement. 6. Revoke browser sessions and integration credentials. 7. Accept reconnecting agents and rebuild central history from their retained buffers. From 7b6f0eb5a1defdf6d0204c44a653e868d1fa2e9a Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 13:39:31 -0400 Subject: [PATCH 06/65] docs: close the third-round fleet hub P1 gaps The transaction helper now serializes spans with a per-connection asyncio lock, the pending recovery key gets a real table in the Task 2 schema with Task 3 owning its lifecycle, and recovery bundles carry a fleet-key signature so an agent holding only the recovery public key cannot forge a replica. Co-Authored-By: Claude Fable 5 --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 34 ++++++++++++------- ...026-08-11-humwatch-2.1-fleet-hub-design.md | 2 +- 2 files changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 5b55dae..41df413 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -370,7 +370,8 @@ integration_clients(integration_id, name, public_key, scopes_json, created_at, r hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) -recovery_bundles(generation, bundle_revision, created_at, encrypted_payload) +pending_recovery_key(id, printable_key, created_at) +recovery_bundles(generation, bundle_revision, created_at, encrypted_payload, fleet_signature) recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` @@ -378,19 +379,27 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id It also owns the transaction helper used by every fleet repository: +The application shares one `aiosqlite` connection, and aiosqlite serializes statements but not transaction spans, so two coroutines could interleave `BEGIN IMMEDIATE` calls. The helper therefore holds one per-connection `asyncio.Lock` across the whole transaction: + ```python +_transaction_locks: WeakKeyDictionary = WeakKeyDictionary() + @asynccontextmanager async def transaction(db: aiosqlite.Connection): - await db.execute("BEGIN IMMEDIATE") - try: - yield - except BaseException: - await db.rollback() - raise - else: - await db.commit() + lock = _transaction_locks.setdefault(db, asyncio.Lock()) + async with lock: + await db.execute("BEGIN IMMEDIATE") + try: + yield + except BaseException: + await db.rollback() + raise + else: + await db.commit() ``` +Add a test that runs overlapping transactions from concurrent coroutines on one connection and proves they serialize with no `cannot start a transaction within a transaction` error and no lost write. + - [ ] **Step 4: Implement machine and fleet identity storage** Use raw private key bytes stored below `resolved_data_root / "identity"`. Create files atomically with mode `0600` on POSIX. Windows ACL enforcement remains the installer task. @@ -444,6 +453,7 @@ git commit -m "feat: add fleet runtime foundation" * Create: `agent/fleet/setup.py` * Create: `agent/routes/v2_session.py` * Create: `agent/routes/v2_setup.py` +* Modify: `agent/fleet/schema.py` * Modify: `agent/security/limits.py` * Modify: `agent/main.py` * Modify: `tests/conftest.py` @@ -505,7 +515,7 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub` in one database transaction, then bind the stored listener before the endpoint reports success, so enrollment claims and agent reports work without a restart. -Do not return the printable recovery key as the transaction's only copy. Persist it in a protected pending-setup record inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. +Do not return the printable recovery key as the transaction's only copy. Persist it in the `pending_recovery_key` row (Task 2 schema, service-owned database, same protection class as the owner password hash) inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. - [ ] **Step 6: Mount `/api/v2` without weakening v1** @@ -1111,11 +1121,11 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. - [ ] **Step 2: Implement bundle creation and replication** -Serialize fleet identity, machine registry, settings, integrations, schema version, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind both generation and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. Copy the ciphertext to enrolled agents through signed internal requests. +Serialize fleet identity, machine registry, settings, integrations, schema version, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind both generation and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the generation, revision, ciphertext digest, and associated data. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. Start `run_replication_loop()` from the hub lifespan. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple has a higher generation or the same generation with a higher revision. They reject an older or equal revision with a different digest, retain the newest valid bundle across restart, and return the accepted generation, revision, and digest so the hub cannot acknowledge stale ciphertext. diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index aee12e2..9fb71b5 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -348,7 +348,7 @@ The hub periodically creates a compact recovery bundle containing: The recovery bundle does not contain browser sessions or full central history. -Each bundle is encrypted to the recovery public key with an ephemeral X25519 key, key derivation, and authenticated encryption. The printable recovery key holds the corresponding private key. It is shown until the owner confirms it is saved and is never retained after that acknowledgement. The encrypted bundle is replicated to enrolled agents. +Each bundle is encrypted to the recovery public key with an ephemeral X25519 key, key derivation, and authenticated encryption, and is signed by the fleet key so no agent can forge a replica with the widely held recovery public key. Agents verify the signature before storing a replica, and recovery verifies it before decrypting. The printable recovery key holds the corresponding private key. It is shown until the owner confirms it is saved and is never retained after that acknowledgement. The encrypted bundle is replicated to enrolled agents. ### Recovery flow From 3b2513a21a4b6759bea90ac75bb42967b1ef898d Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 14:07:56 -0400 Subject: [PATCH 07/65] docs: widen the transfer freeze and its cleanup path The pre-snapshot freeze now blocks fleet mutations as well as report ingestion, so the staged snapshot cannot go stale, and every fallible step after entering the freeze sits inside the cleanup path that lifts it, so a failed snapshot no longer strands the old hub rejecting work. Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 41df413..fe18012 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1033,16 +1033,18 @@ Keep the authority switch behind one method: async def transfer(self, target_id: UUID) -> TransferResult: preflight = await self.preflight(target_id) preflight.require_ready() - await self.state.enter_ingestion_freeze() - staged = await self.stage_encrypted_snapshot(preflight.target) - announcement = await self.prepare_new_generation(preflight.target) - await self.target_client.stage_pending_hub(staged, announcement) + await self.state.enter_transfer_freeze() + announcement = None try: + staged = await self.stage_encrypted_snapshot(preflight.target) + announcement = await self.prepare_new_generation(preflight.target) + await self.target_client.stage_pending_hub(staged, announcement) await self.target_client.commit_unannounced_hub(announcement) await self.target_client.require_ready(announcement) except BaseException: - await self.target_client.abort_pending_hub(announcement) - await self.state.lift_ingestion_freeze() + if announcement is not None: + await self.target_client.abort_pending_hub(announcement) + await self.state.lift_transfer_freeze() raise await self.state.enter_draining_transfer(announcement) publication = await self.target_client.activate_publication_obligation(announcement) @@ -1055,7 +1057,7 @@ async def transfer(self, target_id: UUID) -> TransferResult: - [ ] **Step 1: Write generation, rollback, and interruption tests** -Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Cover freeze lift after a signed abort and old-hub restart while frozen. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. +Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. - [ ] **Step 2: Implement signed announcements** @@ -1069,7 +1071,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -The old hub durably enters an ingestion freeze before the snapshot is taken. The freeze rejects new report batches, so no acknowledgement watermark can advance past the snapshot and every sample the old hub ever acknowledged is present on the target. Agents treat the rejection like any hub outage: batches stay in the local outbox and delivery resumes against whichever hub wins. Any failure before `enter_draining_transfer()` commits, including a snapshot or staging failure, lifts the freeze when the old hub resumes normal service, and an old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. +The old hub durably enters a transfer freeze before the snapshot is taken. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()`, including snapshot creation and generation preparation, runs inside the cleanup path that lifts the freeze on failure, so a failed transfer never leaves the old hub rejecting work until someone restarts it. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. From 0dfd00b18e5e4586d06a4cdad5d3da1d4fb719dd Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 14:16:31 -0400 Subject: [PATCH 08/65] docs: close the fifth-round fleet hub race findings Ingestion checks machine-active and accepting-reports inside the serialized transaction so revocation and the transfer freeze cannot race a stored acknowledgement, the failure path enqueues a durable signed abort and lifts the freeze locally before any network work, and transfer explicitly preserves both machines' local report outboxes. Co-Authored-By: Claude Fable 5 --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 23 +++++++++++-------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index fe18012..b4b59bb 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -660,12 +660,13 @@ Commit message: `feat: enroll agents with one-time codes` * Produces: `AgentTransport.run() -> None` * Produces: `POST /api/internal/v1/reports` -The ingestion transaction has this shape: +The ingestion transaction has this shape. Both admission predicates run inside the serialized transaction, not before it: enrollment, revocation, and the Task 10 transfer-freeze all commit through the same per-connection transaction lock, so checking them outside the transaction would let a batch pass the check, lose the lock race, and be stored and acknowledged after a revocation or freeze committed. `require_accepting_reports()` reads `runtime_state` and rejects while the hub is frozen or draining; in this task nothing sets those states yet, and Task 10 relies on the predicate being in place. ```python async def ingest(self, upload: SignedReportUpload) -> IngestAck: - await self.machines.require_active(upload.machine_id) async with transaction(self.database): + await self.machines.require_active(upload.machine_id) + await self.state.require_accepting_reports() await self.receipts.require_new_or_identical(upload) await self.samples.insert_upload_if_absent(upload) await self.gaps.insert_if_absent(upload.machine_id, upload.gaps) @@ -687,7 +688,7 @@ async def test_duplicate_upload_is_stored_once(hub_ingestion, enrolled_machine, assert await count_metrics(upload.machine_id, upload.sequence_start) == len(upload.batches[0].metrics) ``` -Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. +Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. - [ ] **Step 2: Run focused tests and confirm missing outbox behavior** @@ -1025,6 +1026,7 @@ Run focused tests, static asset verification, browser smoke, full pytest, then c * Produces: `verify_announcement(announcement, fleet_public_key, current_generation) -> bool` * Produces: `HubTransferService.preflight(target_id) -> TransferPreflight` * Produces: `HubTransferService.transfer(target_id) -> TransferResult` +* Produces: `FleetStateRepository.enqueue_signed_abort(announcement) -> None` * Produces: `POST /api/v2/fleet/hub-transfer/preflight` and `/execute` Keep the authority switch behind one method: @@ -1042,9 +1044,10 @@ async def transfer(self, target_id: UUID) -> TransferResult: await self.target_client.commit_unannounced_hub(announcement) await self.target_client.require_ready(announcement) except BaseException: - if announcement is not None: - await self.target_client.abort_pending_hub(announcement) - await self.state.lift_transfer_freeze() + async with transaction(self.database): + if announcement is not None: + await self.state.enqueue_signed_abort(announcement) + await self.state.lift_transfer_freeze() raise await self.state.enter_draining_transfer(announcement) publication = await self.target_client.activate_publication_obligation(announcement) @@ -1057,7 +1060,7 @@ async def transfer(self, target_id: UUID) -> TransferResult: - [ ] **Step 1: Write generation, rollback, and interruption tests** -Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. +Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the target is unreachable so abort delivery itself fails (the local lift must still commit and the abort must retry from its durable queue), freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. Add a target continuity test: the target collects samples into its own outbox during staging and readiness, activation preserves that outbox and the target's identity, and those samples appear in the new hub's history afterward, alongside the demoted old hub delivering the outbox rows it kept out of the snapshot. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. - [ ] **Step 2: Implement signed announcements** @@ -1065,13 +1068,13 @@ Canonicalize announcement JSON with sorted keys and compact separators before si - [ ] **Step 3: Implement encrypted transfer staging** -Create a SQLite backup through the backup API. Package fleet config, owner hash, registry, history, integrations, and fleet signing key. Exclude sessions. Encrypt to the target X25519 public key with HKDF-SHA256 and AES-256-GCM. +Create a SQLite backup through the backup API. Package fleet config, owner hash, registry, history, integrations, and fleet signing key. Exclude sessions and the old hub's machine-local `report_outbox`. Encrypt to the target X25519 public key with HKDF-SHA256 and AES-256-GCM. - [ ] **Step 4: Implement the staged transfer and explicit abort path** -The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. The import applies only hub-authority tables. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -The old hub durably enters a transfer freeze before the snapshot is taken. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()`, including snapshot creation and generation preparation, runs inside the cleanup path that lifts the freeze on failure, so a failed transfer never leaves the old hub rejecting work until someone restarts it. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. +The old hub durably enters a transfer freeze before the snapshot is taken. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()`, including snapshot creation and generation preparation, runs inside the cleanup path that lifts the freeze on failure, so a failed transfer never leaves the old hub rejecting work until someone restarts it. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. From 080bd5c9b87203d79bc15c53a40abf6f5f4e8986 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 14:22:56 -0400 Subject: [PATCH 09/65] docs: close the sixth-round fleet hub findings The transfer freeze revalidates the target machine in its own transaction, create-fleet binds and holds the socket before committing the hub role, migration routes require the legacy bearer token or a loopback peer, and hub retention ages out the fleet history tables. Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index b4b59bb..575c2f0 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -513,7 +513,7 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub` in one database transaction, then bind the stored listener before the endpoint reports success, so enrollment claims and agent reports work without a restart. +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. Cover the bind-failure path with a test proving the role stays `unconfigured`. Do not return the printable recovery key as the transaction's only copy. Persist it in the `pending_recovery_key` row (Task 2 schema, service-owned database, same protection class as the owner password hash) inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. @@ -706,7 +706,7 @@ The reports route verifies content length, the outer request signature, the uplo - [ ] **Step 5: Implement transport and retention** -Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. +Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Retention on a hub also ages out central fleet history: the existing service deletes only the legacy `metrics` and `process_snapshots` tables, so extend it to purge `fleet_metrics` and `fleet_processes` rows older than `retention_days` in one transaction, with tests covering both fleet tables. Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. - [ ] **Step 6: Verify collection, reporting, and full regression** @@ -1035,7 +1035,7 @@ Keep the authority switch behind one method: async def transfer(self, target_id: UUID) -> TransferResult: preflight = await self.preflight(target_id) preflight.require_ready() - await self.state.enter_transfer_freeze() + await self.state.enter_transfer_freeze(require_active_target=preflight.target) announcement = None try: staged = await self.stage_encrypted_snapshot(preflight.target) @@ -1074,7 +1074,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. The import applies only hub-authority tables. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -The old hub durably enters a transfer freeze before the snapshot is taken. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()`, including snapshot creation and generation preparation, runs inside the cleanup path that lifts the freeze on failure, so a failed transfer never leaves the old hub rejecting work until someone restarts it. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. +The old hub durably enters a transfer freeze before the snapshot is taken. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()`, including snapshot creation and generation preparation, runs inside the cleanup path that lifts the freeze on failure, so a failed transfer never leaves the old hub rejecting work until someone restarts it. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. @@ -1269,6 +1269,8 @@ Use SQLite backup, validate schema and machine rows, copy config and security pa Create Fleet imports history under the stable self machine UUID. Join Fleet converts history to ordered outbox batches and uploads through Task 5. Neither path deletes original runtime files. +The migration router is not open: an upgraded legacy installation is reachable on its existing LAN listener, and these routes replace runtime authority and credentials before any owner account exists. Inspection and every conversion mutation require the installation's existing legacy bearer token, exactly as the protected v1 routers do, or an actual loopback peer. Add unauthorized-route tests proving a tokenless LAN request cannot inspect or convert. + - [ ] **Step 4: Gate protected v1 endpoints by mode** Fresh installations have only public v1 health. Upgraded installations default protected v1 off, can explicitly enable it with the archived bearer token, and can remove compatibility permanently. From 66271af93d41d29f7f179fe78a3a52c467158290 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 16:29:40 -0400 Subject: [PATCH 10/65] docs: close the seventh-round fleet hub findings Recovery promotion selects, binds, and announces a listener owned by the recovering PC instead of inheriting the dead hub's addresses, and the draining transition moves inside the transfer cleanup scope so a failed journal commit can no longer strand the freeze. Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 10 +++++----- .../specs/2026-08-11-humwatch-2.1-fleet-hub-design.md | 7 ++++--- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 575c2f0..da60d86 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1043,13 +1043,13 @@ async def transfer(self, target_id: UUID) -> TransferResult: await self.target_client.stage_pending_hub(staged, announcement) await self.target_client.commit_unannounced_hub(announcement) await self.target_client.require_ready(announcement) + await self.state.enter_draining_transfer(announcement) except BaseException: async with transaction(self.database): if announcement is not None: await self.state.enqueue_signed_abort(announcement) await self.state.lift_transfer_freeze() raise - await self.state.enter_draining_transfer(announcement) publication = await self.target_client.activate_publication_obligation(announcement) await self.target_client.require_publication_persisted(publication) async with transaction(self.database): @@ -1074,7 +1074,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. The import applies only hub-authority tables. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -The old hub durably enters a transfer freeze before the snapshot is taken. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()`, including snapshot creation and generation preparation, runs inside the cleanup path that lifts the freeze on failure, so a failed transfer never leaves the old hub rejecting work until someone restarts it. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. +The old hub durably enters a transfer freeze before the snapshot is taken. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. @@ -1110,7 +1110,7 @@ Run focused transfer tests, full pytest, static assets, and the task surface. Co * Produces: `RecoveryService.create_bundle() -> EncryptedRecoveryBundle` * Produces: `RecoveryService.store_replica(bundle) -> ReplicaAcceptance` * Produces: `RecoveryService.inspect(recovery_key: str) -> RecoveryPreview` -* Produces: `RecoveryService.promote(recovery_key: str, owner: NewOwner) -> HubAnnouncement` +* Produces: `RecoveryService.promote(recovery_key: str, owner: NewOwner, listener: ListenerSelection) -> HubAnnouncement` * Produces: `RecoveryService.run_replication_loop(interval: timedelta = timedelta(hours=6)) -> None` Recovery decryption must authenticate before parsing: @@ -1126,7 +1126,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. - [ ] **Step 2: Implement bundle creation and replication** @@ -1136,7 +1136,7 @@ Start `run_replication_loop()` from the hub lifespan. Create and distribute a bu - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Promotion creates a new owner password, revokes integrations, switches to hub mode, and publishes a signed announcement at `max(bundle_generation, locally_accepted_generation) + 1`. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, switches to hub mode, and publishes a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 9fb71b5..12f91ea 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -356,9 +356,10 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 2. Enter the printable recovery key. 3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. -5. Promote the PC to hub at a generation above both the bundle's generation and the generation this machine last accepted from an announcement. -6. Revoke browser sessions and integration credentials. -7. Accept reconnecting agents and rebuild central history from their retained buffers. +5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. +6. Promote the PC to hub at a generation above both the bundle's generation and the generation this machine last accepted from an announcement, announcing the newly bound listener. +7. Revoke browser sessions and integration credentials. +8. Accept reconnecting agents and rebuild central history from their retained buffers. The UI states how much history can be recovered before promotion. History older than all agent buffers is not promised. From b3b094a3aa50d67ff46ad2eb2c8f5e8b1b09ad7f Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 16:37:25 -0400 Subject: [PATCH 11/65] docs: close the eighth-round fleet hub findings Claims persist the complete agent claim and create the enrolling machine row atomically, transfer rejects while a claimed enrollment is pending, recovery rewinds acknowledgement watermarks through an explicit replay handshake and reuses the durable announcement obligation, and fleet identity files stage until the creation transaction commits. Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index da60d86..4a3a60a 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -513,7 +513,7 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. Cover the bind-failure path with a test proving the role stays `unconfigured`. +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory and moves them into place only after the database transaction commits, deleting the staged files on rollback, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. Cover the bind-failure path with a test proving the role stays `unconfigured`, and a database commit failure after key creation proving the identity directory stays clean and a retry succeeds. Do not return the printable recovery key as the transaction's only copy. Persist it in the `pending_recovery_key` row (Task 2 schema, service-owned database, same protection class as the owner password hash) inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. @@ -622,7 +622,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w - [ ] **Step 5: Implement hub-orchestrated enrollment** -The hub validates `http` or `https` private addresses and calls claim. Before finalize, it durably inserts the hub claim JSON, signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. +The hub validates `http` or `https` private addresses and calls claim. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction creates the `fleet_machines` row with status `enrolling` from that agent claim. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. - [ ] **Step 6: Verify enrollment and commit** @@ -1074,7 +1074,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. The import applies only hub-authority tables. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -The old hub durably enters a transfer freeze before the snapshot is taken. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. +The old hub durably enters a transfer freeze before the snapshot is taken. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The same transaction rejects the transfer while any enrollment attempt is `claimed`: a claimed attempt's retry nonce is encrypted to the old hub's machine identity, which never transfers, so the new hub could not finish it and the enrollment would strand. Claims resolve or expire within the ten-minute pairing window, and expired or failed attempts do not block. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. @@ -1126,7 +1126,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. - [ ] **Step 2: Implement bundle creation and replication** @@ -1136,7 +1136,9 @@ Start `run_replication_loop()` from the hub lifespan. Create and distribute a bu - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, switches to hub mode, and publishes a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. + +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** From dd3eabbe792372a727d3bbb816f26926c305b389 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 16:46:47 -0400 Subject: [PATCH 12/65] docs: close the ninth-round fleet hub findings Migration conversions take a listener selection with the same bind-before-commit rule as fresh setup, promotion stages the decrypted fleet key through a journaled finalization instead of straddling the role commit, and gap-only uploads validate and acknowledge without assuming a batch exists. Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 4a3a60a..6685c41 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -671,7 +671,7 @@ async def ingest(self, upload: SignedReportUpload) -> IngestAck: await self.samples.insert_upload_if_absent(upload) await self.gaps.insert_if_absent(upload.machine_id, upload.gaps) highest = await self.samples.highest_contiguous(upload.machine_id) - await self.machines.mark_reported(upload.machine_id, highest, upload.batches[-1].captured_at) + await self.machines.mark_reported(upload.machine_id, highest, upload.latest_activity_at()) await self.receipts.record(upload) await self.events.publish_machine_update(upload.machine_id, upload.sequence_end) return IngestAck(highest_contiguous_sequence=highest) @@ -702,7 +702,7 @@ Refactor the end of each collection tick to build one `ReportBatch`. In `legacy` Build canonical compact JSON with sorted keys for `{batches, gaps}` and set `payload_digest` to its SHA256. Sign canonical compact JSON containing `fleet_id`, `machine_id`, `upload_id`, `sequence_start`, `sequence_end`, and `payload_digest` with the machine key. The outer HTTP request is also signed through Task 4, binding the exact body, fleet, and machine headers. -The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. Insert metrics, processes, gap ranges, the machine watermark, and `report_receipts` in one transaction. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. +The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. An upload must contain at least one batch or one gap range, and a gap-only upload (retention removed the unacknowledged history, leaving nothing else to send) commits and acknowledges normally: `latest_activity_at()` falls back to the gap's reported time, so the agent can advance past the missing range. Test a gap-only upload end to end. Insert metrics, processes, gap ranges, the machine watermark, and `report_receipts` in one transaction. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. - [ ] **Step 5: Implement transport and retention** @@ -1136,7 +1136,7 @@ Start `run_replication_loop()` from the hub lifespan. Create and distribute a bu - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction record a journal reference to that staged key, and move it into the identity location as finalization. A rollback deletes the staged file, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired at startup from the journal reference, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. @@ -1240,8 +1240,8 @@ Run focused integration tests, v2 API tests, full pytest, static assets, and bro **Interfaces:** * Produces: `UpgradeService.inspect() -> UpgradePreview` -* Produces: `UpgradeService.create_fleet(owner: NewOwner) -> UpgradeResult` -* Produces: `UpgradeService.prepare_join() -> PairingDisplay` +* Produces: `UpgradeService.create_fleet(owner: NewOwner, listener: ListenerSelection) -> UpgradeResult` +* Produces: `UpgradeService.prepare_join(listener: ListenerSelection) -> PairingDisplay` * Produces: `UpgradeService.enable_legacy_api(enabled: bool) -> None` Inspection returns proof, not a boolean: @@ -1271,6 +1271,8 @@ Use SQLite backup, validate schema and machine rows, copy config and security pa Create Fleet imports history under the stable self machine UUID. Join Fleet converts history to ordered outbox batches and uploads through Task 5. Neither path deletes original runtime files. +Both conversions take a listener selection and follow the same bind-before-commit rule as fresh setup: the legacy listener may be loopback-only, a forbidden wildcard, or archived with the legacy security settings, so Create Fleet reserves and persists a private hub listener (Task 3's rules) and Join Fleet selects the restricted private enrollment listener Task 4 requires, in each case binding before the role transition and failing visibly without changing role when binding fails. The wizard offers the machine's detected private addresses for the choice. + The migration router is not open: an upgraded legacy installation is reachable on its existing LAN listener, and these routes replace runtime authority and credentials before any owner account exists. Inspection and every conversion mutation require the installation's existing legacy bearer token, exactly as the protected v1 routers do, or an actual loopback peer. Add unauthorized-route tests proving a tokenless LAN request cannot inspect or convert. - [ ] **Step 4: Gate protected v1 endpoints by mode** From f4f227b85613f12aa832d48beb949e8e247284c7 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 22:29:40 -0400 Subject: [PATCH 13/65] docs: close the tenth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 23 +++++++++++-------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 6685c41..f2d5f19 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -362,6 +362,7 @@ fleet_settings(id, listeners_json, https_cert_path, https_key_path, allow_public fleet_metrics(machine_id, sequence, timestamp, category, metric_name, value, unit) fleet_processes(machine_id, sequence, timestamp, pid, name, cpu_percent, memory_mb) report_outbox(sequence, batch_id, captured_at, payload_json, acknowledged_at) +report_batches(machine_id, sequence, captured_at, accepted_at) report_receipts(machine_id, upload_id, payload_digest, sequence_start, sequence_end, highest_contiguous_sequence, accepted_at) data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) used_nonces(machine_id, nonce_hash, expires_at) @@ -371,11 +372,14 @@ hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_ hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) pending_recovery_key(id, printable_key, created_at) +staged_key_journal(journal_id, purpose, staged_path, created_at, finalized_at) recovery_bundles(generation, bundle_revision, created_at, encrypted_payload, fleet_signature) recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` -Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. + +`report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3) and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. It also owns the transaction helper used by every fleet repository: @@ -513,7 +517,7 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory and moves them into place only after the database transaction commits, deleting the staged files on rollback, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. Cover the bind-failure path with a test proving the role stays `unconfigured`, and a database commit failure after key creation proving the identity directory stays clean and a retry succeeds. +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, and a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key. Do not return the printable recovery key as the transaction's only copy. Persist it in the `pending_recovery_key` row (Task 2 schema, service-owned database, same protection class as the owner password hash) inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. @@ -669,8 +673,9 @@ async def ingest(self, upload: SignedReportUpload) -> IngestAck: await self.state.require_accepting_reports() await self.receipts.require_new_or_identical(upload) await self.samples.insert_upload_if_absent(upload) + await self.batches.insert_presence_if_absent(upload) await self.gaps.insert_if_absent(upload.machine_id, upload.gaps) - highest = await self.samples.highest_contiguous(upload.machine_id) + highest = await self.batches.highest_contiguous(upload.machine_id) await self.machines.mark_reported(upload.machine_id, highest, upload.latest_activity_at()) await self.receipts.record(upload) await self.events.publish_machine_update(upload.machine_id, upload.sequence_end) @@ -688,7 +693,7 @@ async def test_duplicate_upload_is_stored_once(hub_ingestion, enrolled_machine, assert await count_metrics(upload.machine_id, upload.sequence_start) == len(upload.batches[0].metrics) ``` -Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. +Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. - [ ] **Step 2: Run focused tests and confirm missing outbox behavior** @@ -702,11 +707,11 @@ Refactor the end of each collection tick to build one `ReportBatch`. In `legacy` Build canonical compact JSON with sorted keys for `{batches, gaps}` and set `payload_digest` to its SHA256. Sign canonical compact JSON containing `fleet_id`, `machine_id`, `upload_id`, `sequence_start`, `sequence_end`, and `payload_digest` with the machine key. The outer HTTP request is also signed through Task 4, binding the exact body, fleet, and machine headers. -The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. An upload must contain at least one batch or one gap range, and a gap-only upload (retention removed the unacknowledged history, leaving nothing else to send) commits and acknowledges normally: `latest_activity_at()` falls back to the gap's reported time, so the agent can advance past the missing range. Test a gap-only upload end to end. Insert metrics, processes, gap ranges, the machine watermark, and `report_receipts` in one transaction. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. +The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. An upload must contain at least one batch or one gap range, and a gap-only upload (retention removed the unacknowledged history, leaving nothing else to send) commits and acknowledges normally: `latest_activity_at()` falls back to the gap's reported time, so the agent can advance past the missing range. Test a gap-only upload end to end. Insert metrics, processes, one `report_batches` presence row per accepted batch sequence, gap ranges, the machine watermark, and `report_receipts` in one transaction. Compute `highest_contiguous_sequence` by extending the persisted machine watermark through `report_batches` presence rows and disclosed gap ranges, never by scanning sample rows, so an empty batch and a retention-purged prefix both count as covered. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. - [ ] **Step 5: Implement transport and retention** -Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Retention on a hub also ages out central fleet history: the existing service deletes only the legacy `metrics` and `process_snapshots` tables, so extend it to purge `fleet_metrics` and `fleet_processes` rows older than `retention_days` in one transaction, with tests covering both fleet tables. Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. +Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Retention on a hub also ages out central fleet history: the existing service deletes only the legacy `metrics` and `process_snapshots` tables, so extend it to purge `fleet_metrics`, `fleet_processes`, and `report_batches` presence rows at or below each machine's persisted watermark older than `retention_days` in one transaction, with tests covering all three fleet tables (presence rows above the watermark are kept, since contiguity extends from the watermark through them). Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. - [ ] **Step 6: Verify collection, reporting, and full regression** @@ -1126,7 +1131,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. - [ ] **Step 2: Implement bundle creation and replication** @@ -1136,9 +1141,9 @@ Start `run_replication_loop()` from the hub lifespan. Create and distribute a bu - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction record a journal reference to that staged key, and move it into the identity location as finalization. A rollback deletes the staged file, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired at startup from the journal reference, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** From 09027b9c09ad7ba7e290d1752764fc63cda21405 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 22:34:59 -0400 Subject: [PATCH 14/65] docs: close the eleventh-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 21 +++++++++++-------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index f2d5f19..f8963fe 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -379,7 +379,7 @@ recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, stat Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. -`report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3) and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. +`report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. It also owns the transaction helper used by every fleet repository: @@ -1029,23 +1029,24 @@ Run focused tests, static asset verification, browser smoke, full pytest, then c **Interfaces:** * Produces: `HubAnnouncement.sign(fleet_identity) -> HubAnnouncement` * Produces: `verify_announcement(announcement, fleet_public_key, current_generation) -> bool` -* Produces: `HubTransferService.preflight(target_id) -> TransferPreflight` -* Produces: `HubTransferService.transfer(target_id) -> TransferResult` +* Produces: `HubTransferService.preflight(target_id) -> TransferPreflight` (includes the target's detected private listener candidates) +* Produces: `HubTransferService.transfer(target_id, listener: ListenerSelection) -> TransferResult` * Produces: `FleetStateRepository.enqueue_signed_abort(announcement) -> None` * Produces: `POST /api/v2/fleet/hub-transfer/preflight` and `/execute` Keep the authority switch behind one method: ```python -async def transfer(self, target_id: UUID) -> TransferResult: +async def transfer(self, target_id: UUID, listener: ListenerSelection) -> TransferResult: preflight = await self.preflight(target_id) preflight.require_ready() + preflight.require_listener_candidate(listener) await self.state.enter_transfer_freeze(require_active_target=preflight.target) announcement = None try: staged = await self.stage_encrypted_snapshot(preflight.target) - announcement = await self.prepare_new_generation(preflight.target) - await self.target_client.stage_pending_hub(staged, announcement) + announcement = await self.prepare_new_generation(preflight.target, listener) + await self.target_client.stage_pending_hub(staged, announcement, listener) await self.target_client.commit_unannounced_hub(announcement) await self.target_client.require_ready(announcement) await self.state.enter_draining_transfer(announcement) @@ -1065,7 +1066,7 @@ async def transfer(self, target_id: UUID) -> TransferResult: - [ ] **Step 1: Write generation, rollback, and interruption tests** -Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the target is unreachable so abort delivery itself fails (the local lift must still commit and the abort must retry from its durable queue), freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. Add a target continuity test: the target collects samples into its own outbox during staging and readiness, activation preserves that outbox and the target's identity, and those samples appear in the new hub's history afterward, alongside the demoted old hub delivering the outbox rows it kept out of the snapshot. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. +Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, a target bind failure on the selected listener failing staging with the target still an agent and the old hub aborting with the freeze lifted, the activated target listening on the owner's selection with the announcement advertising it and never the old hub's imported addresses or certificate paths, an aborted target holding no staged fleet key file, a crash between the target's activation commit and key finalization repaired at startup into a hub that can sign, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the target is unreachable so abort delivery itself fails (the local lift must still commit and the abort must retry from its durable queue), freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. Add a target continuity test: the target collects samples into its own outbox during staging and readiness, activation preserves that outbox and the target's identity, and those samples appear in the new hub's history afterward, alongside the demoted old hub delivering the outbox rows it kept out of the snapshot. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. - [ ] **Step 2: Implement signed announcements** @@ -1077,7 +1078,9 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, - [ ] **Step 4: Implement the staged transfer and explicit abort path** -The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode on its configured private listener. The import applies only hub-authority tables. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target binds and holds that socket before `commit_unannounced_hub`, exactly like Create Fleet and recovery: a bind failure fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. + +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. A signed abort deletes the staged key file along with the staged authority, so an aborted target resumes agent mode holding no authoritative key material, and a crash between the activation commit and the move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. The old hub durably enters a transfer freeze before the snapshot is taken. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The same transaction rejects the transfer while any enrollment attempt is `claimed`: a claimed attempt's retry nonce is encrypted to the old hub's machine identity, which never transfers, so the new hub could not finish it and the enrollment would strand. Claims resolve or expire within the ten-minute pairing window, and expired or failed attempts do not block. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. @@ -1085,7 +1088,7 @@ If staging, activation, or readiness fails before `enter_draining_transfer()` co - [ ] **Step 5: Add Move Hub UI and evidence** -Show preflight results, transfer progress, exact new IP and port, and a link to open it. Capture successful transfer and forced rollback. Canary by failing the target validation probe and confirming the old hub remains active. +Show preflight results including the target's detected private listener candidates, require the owner to pick the new hub's listener from them before execute, then show transfer progress, exact new IP and port, and a link to open it. Capture successful transfer and forced rollback. Canary by failing the target validation probe and confirming the old hub remains active. - [ ] **Step 6: Verify and commit** From 108c5d3307db9c2ade2370937a90b6840858581e Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 22:47:22 -0400 Subject: [PATCH 15/65] docs: close the twelfth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 9 +++++---- .../specs/2026-08-11-humwatch-2.1-fleet-hub-design.md | 2 +- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index f8963fe..3c663bf 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -379,7 +379,7 @@ recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, stat Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. -`report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. +`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. It also owns the transaction helper used by every fleet repository: @@ -693,7 +693,7 @@ async def test_duplicate_upload_is_stored_once(hub_ingestion, enrolled_machine, assert await count_metrics(upload.machine_id, upload.sequence_start) == len(upload.batches[0].metrics) ``` -Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. +Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Test sequence allocation after total purge: acknowledge every outbox row, delete them all through the retention path, restart the repository, and prove the next append allocates the prior maximum plus one from the persisted `sqlite_sequence` high-water mark rather than reusing an acknowledged sequence. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. - [ ] **Step 2: Run focused tests and confirm missing outbox behavior** @@ -1120,6 +1120,7 @@ Run focused transfer tests, full pytest, static assets, and the task surface. Co * Produces: `RecoveryService.inspect(recovery_key: str) -> RecoveryPreview` * Produces: `RecoveryService.promote(recovery_key: str, owner: NewOwner, listener: ListenerSelection) -> HubAnnouncement` * Produces: `RecoveryService.run_replication_loop(interval: timedelta = timedelta(hours=6)) -> None` +* Produces: `GET /api/internal/v1/announcement` (every fleet member returns its accepted signed hub announcement for the recovery generation probe) Recovery decryption must authenticate before parsing: @@ -1134,7 +1135,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. - [ ] **Step 2: Implement bundle creation and replication** @@ -1144,7 +1145,7 @@ Start `run_replication_loop()` from the hub lifespan. Create and distribute a bu - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation) + 1`. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 12f91ea..867d34f 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -357,7 +357,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. -6. Promote the PC to hub at a generation above both the bundle's generation and the generation this machine last accepted from an announcement, announcing the newly bound listener. +6. Promote the PC to hub at a generation above the bundle's generation, the generation this machine last accepted from an announcement, and the highest fleet-signature-verified generation any reachable fleet member reports accepting, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. 7. Revoke browser sessions and integration credentials. 8. Accept reconnecting agents and rebuild central history from their retained buffers. From e93155415dd55b876f008107702e2990615c76be Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 22:53:28 -0400 Subject: [PATCH 16/65] docs: close the thirteenth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 13 ++++++++++--- .../2026-08-11-humwatch-2.1-fleet-hub-design.md | 2 ++ 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 3c663bf..58e0427 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -455,6 +455,7 @@ git commit -m "feat: add fleet runtime foundation" * Create: `agent/security/passwords.py` * Create: `agent/security/sessions.py` * Create: `agent/fleet/setup.py` +* Create: `agent/fleet/listeners.py` * Create: `agent/routes/v2_session.py` * Create: `agent/routes/v2_setup.py` * Modify: `agent/fleet/schema.py` @@ -474,6 +475,7 @@ git commit -m "feat: add fleet runtime foundation" * Produces: `verify_password(password: str, record: PasswordRecord) -> bool` * Produces: `require_owner_session(request: Request) -> OwnerSession` * Produces: `require_csrf(request: Request, session: OwnerSession) -> None` +* Produces: `ListenerManager.attach(held_socket) -> None` and `ListenerManager.detach(listener) -> None` * Produces: `POST /api/v2/setup/create-fleet` * Produces: `GET /api/v2/setup/recovery-key`, `POST /api/v2/setup/recovery-key/ack` * Produces: `POST /api/v2/session/login`, `GET /me`, `POST /logout`, `GET /sessions`, `DELETE /sessions/{session_hash}` @@ -517,7 +519,9 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, and a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key. +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). + +Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key, an in-process attach test proving a request on the newly advertised listener succeeds with no process restart, and a kill between commit and attach proving the restarted process serves the stored listener. Do not return the printable recovery key as the transaction's only copy. Persist it in the `pending_recovery_key` row (Task 2 schema, service-owned database, same protection class as the owner password hash) inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. @@ -1078,7 +1082,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, - [ ] **Step 4: Implement the staged transfer and explicit abort path** -The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target binds and holds that socket before `commit_unannounced_hub`, exactly like Create Fleet and recovery: a bind failure fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. +The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target binds and holds that socket before `commit_unannounced_hub`, then attaches it through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. A signed abort deletes the staged key file along with the staged authority, so an aborted target resumes agent mode holding no authoritative key material, and a crash between the activation commit and the move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. @@ -1121,6 +1125,7 @@ Run focused transfer tests, full pytest, static assets, and the task surface. Co * Produces: `RecoveryService.promote(recovery_key: str, owner: NewOwner, listener: ListenerSelection) -> HubAnnouncement` * Produces: `RecoveryService.run_replication_loop(interval: timedelta = timedelta(hours=6)) -> None` * Produces: `GET /api/internal/v1/announcement` (every fleet member returns its accepted signed hub announcement for the recovery generation probe) +* Produces: `POST /api/v2/recovery/adopt-hub` (loopback-only manual hub adoption on any fleet member) Recovery decryption must authenticate before parsing: @@ -1145,10 +1150,12 @@ Start `run_replication_loop()` from the hub lifespan. Create and distribute a bu - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, and the replayed history acknowledges. + - [ ] **Step 4: Add the Recover Fleet page and evidence** Keep recovery copy literal. Capture valid preview, invalid key, recovery confirmation, and post-recovery login. Canary by flipping one ciphertext byte and proving preview refuses it. diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 867d34f..331ac87 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -325,6 +325,8 @@ Every hub announcement contains the fleet UUID, hub machine UUID, addresses, and Agents accept only a valid announcement with a higher generation than the one stored locally. An old hub that returns later cannot reclaim the fleet with stale state. +One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. + ### Address change Moving the hub normally changes its IP. The completion screen shows and opens the new IP plus port. The owner signs in again because browser sessions are revoked and the origin changed. From d34914dfdb8bfe4f8c836398bd4aba73b6d9c3b1 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:04:58 -0400 Subject: [PATCH 17/65] docs: harden the fleet hub plan against the audited crash and durability gaps Twenty findings from an internal adversarial audit: durable homes for transfer freeze state, signed aborts, and pending hub authority, allocator preservation across transfer import, agent-side listeners and hub key trust after enrollment, session revocation on every authority change, recovery rewind for empty outboxes and the promoting machine itself, degraded startup for unbindable listeners, retention owners for every unowned table, and idempotency rules for re-enrollment, second claims, and integration exchange. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 97 +++++++++++++------ ...026-08-11-humwatch-2.1-fleet-hub-design.md | 3 +- 2 files changed, 70 insertions(+), 30 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 58e0427..94bfbcd 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -85,6 +85,7 @@ class RuntimeRole(str, Enum): HUB = "hub" AGENT = "agent" LEGACY = "legacy" + PENDING_HUB = "pending_hub" # transfer target holds staged or unannounced authority, Task 10 class MachineStatus(str, Enum): ENROLLING = "enrolling" @@ -142,6 +143,7 @@ class HubClaim(BaseModel): hub_generation: int hub_addresses: list[str] fleet_signing_public_key: str + hub_signing_public_key: str recovery_public_key: str class AgentClaim(BaseModel): @@ -169,6 +171,7 @@ class HubAnnouncement(BaseModel): hub_machine_id: UUID generation: int addresses: list[str] + hub_signing_public_key: str issued_at: datetime signature: str ``` @@ -185,6 +188,8 @@ X-HumWatch-Signature: `SignedHeaders` is the typed mapping containing those five headers. The signed bytes are `METHOD + "\n" + PATH + "\n" + FLEET_ID + "\n" + MACHINE_ID + "\n" + SHA256(BODY) + "\n" + TIMESTAMP + "\n" + NONCE`. The hub rejects timestamps beyond 120 seconds, reused nonces, unknown machines, revoked machines, relabeled fleet or machine headers, and invalid signatures. +Agents authenticate the hub's side of this envelope with `hub_signing_public_key`: enrollment finalize persists the key carried on `HubClaim`, and every later fleet-signed `HubAnnouncement` the agent accepts (Task 10 transfer, Task 11 recovery) carries the current hub's `hub_signing_public_key` too, so the agent's authorized hub key rotates automatically the moment a higher-generation announcement passes fleet-key verification, with no separate distribution step. Inbound hub-originated signed requests to an agent (announcement delivery, recovery replica delivery, both Task 10 and 11) are verified against that persisted key rather than trusting the claimed machine ID alone. + Browser mutations send the CSRF token in `X-HumWatch-CSRF`. Session cookies are named `humwatch_session`. --- @@ -354,7 +359,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -365,10 +370,12 @@ report_outbox(sequence, batch_id, captured_at, payload_json, acknowledged_at) report_batches(machine_id, sequence, captured_at, accepted_at) report_receipts(machine_id, upload_id, payload_digest, sequence_start, sequence_end, highest_contiguous_sequence, accepted_at) data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) -used_nonces(machine_id, nonce_hash, expires_at) +used_nonces(principal_type, principal_id, nonce_hash, expires_at) enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) +hub_transfer_aborts(transfer_id, announcement_generation, announcement_digest, abort_json, state, attempt_count, next_attempt_at, acknowledged_at) +pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staged_snapshot_path, staged_key_path, state, created_at, activated_at) hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) pending_recovery_key(id, printable_key, created_at) @@ -377,10 +384,14 @@ recovery_bundles(generation, bundle_revision, created_at, encrypted_payload, fle recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` -Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. + +`used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves nothing but the internal activation, abort, and status routes, never the dashboard or agent report ingestion. + It also owns the transaction helper used by every fleet repository: The application shares one `aiosqlite` connection, and aiosqlite serializes statements but not transaction spans, so two coroutines could interleave `BEGIN IMMEDIATE` calls. The helper therefore holds one per-connection `asyncio.Lock` across the whole transaction: @@ -578,6 +589,7 @@ git commit -m "feat: add owner sessions and fleet creation" * Produces: local-only `POST /api/v2/setup/pairing-code` * Produces: local-only `GET/PUT /api/v2/setup/enrollment-listeners` * Produces: agent `POST /api/internal/v1/enrollment/claim` and `/finalize` +* Produces: the agent's persisted post-enrollment listener set (`fleet_settings.listeners_json`, Task 2 schema) Use this one canonical implementation boundary: @@ -605,7 +617,7 @@ async def enroll_machine(address: str, code: str) -> EnrolledMachine: - [ ] **Step 1: Write signing and enrollment protocol tests** -Cover body tampering, clock skew, nonce replay, expired code, reused code, loopback-only listener selection, selected-private-listener binding, rejection of every non-enrollment route on that listener, private-address validation, interrupted finalization, a hub crash after its durable claim record, idempotent finalize retry, first-report pending status, and fleet or machine header relabeling with an otherwise valid signature. +Cover body tampering, clock skew, nonce replay, expired code, reused code, loopback-only listener selection, selected-private-listener binding, rejection of every non-enrollment route on that listener, private-address validation, interrupted finalization, a hub crash after its durable claim record, idempotent finalize retry, first-report pending status, fleet or machine header relabeling with an otherwise valid signature, a second claim attempt against an unconsumed code superseding the earlier unfinalized claim for that code, and revoking a previously enrolled machine then re-enrolling it under the same stable machine identity. ```python def test_pairing_code_expires_after_ten_minutes(clock, enrollment): @@ -630,7 +642,9 @@ The local setup surface first reads private LAN and Tailscale candidates, then w - [ ] **Step 5: Implement hub-orchestrated enrollment** -The hub validates `http` or `https` private addresses and calls claim. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction creates the `fleet_machines` row with status `enrolling` from that agent claim. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. +The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its keys, addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. + +Finalize also persists the two pieces of trust the agent needs for everything after enrollment. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. - [ ] **Step 6: Verify enrollment and commit** @@ -668,7 +682,7 @@ Commit message: `feat: enroll agents with one-time codes` * Produces: `AgentTransport.run() -> None` * Produces: `POST /api/internal/v1/reports` -The ingestion transaction has this shape. Both admission predicates run inside the serialized transaction, not before it: enrollment, revocation, and the Task 10 transfer-freeze all commit through the same per-connection transaction lock, so checking them outside the transaction would let a batch pass the check, lose the lock race, and be stored and acknowledged after a revocation or freeze committed. `require_accepting_reports()` reads `runtime_state` and rejects while the hub is frozen or draining; in this task nothing sets those states yet, and Task 10 relies on the predicate being in place. +The ingestion transaction has this shape. Both admission predicates run inside the serialized transaction, not before it: enrollment, revocation, and the Task 10 transfer-freeze all commit through the same per-connection transaction lock, so checking them outside the transaction would let a batch pass the check, lose the lock race, and be stored and acknowledged after a revocation or freeze committed. `require_accepting_reports()` reads `runtime_state.transfer_state` and rejects unless it is `accepting`. In this task nothing ever sets it to `frozen` or `draining`, and Task 10 relies on the predicate reading that exact column being in place before its freeze and drain transactions exist. ```python async def ingest(self, upload: SignedReportUpload) -> IngestAck: @@ -697,7 +711,7 @@ async def test_duplicate_upload_is_stored_once(hub_ingestion, enrolled_machine, assert await count_metrics(upload.machine_id, upload.sequence_start) == len(upload.batches[0].metrics) ``` -Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Test sequence allocation after total purge: acknowledge every outbox row, delete them all through the retention path, restart the repository, and prove the next append allocates the prior maximum plus one from the persisted `sqlite_sequence` high-water mark rather than reusing an acknowledged sequence. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. +Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Test sequence allocation after total purge: acknowledge every outbox row, delete them all through the retention path, restart the repository, and prove the next append allocates the prior maximum plus one from the persisted `sqlite_sequence` high-water mark rather than reusing an acknowledged sequence. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state.transfer_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. Also cover retention purging expired `used_nonces`, expired `owner_sessions`, aged `report_receipts`, fully-covered `data_gaps`, and all but the newest-plus-one `recovery_bundles` rows. - [ ] **Step 2: Run focused tests and confirm missing outbox behavior** @@ -717,6 +731,8 @@ The reports route verifies content length, the outer request signature, the uplo Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Retention on a hub also ages out central fleet history: the existing service deletes only the legacy `metrics` and `process_snapshots` tables, so extend it to purge `fleet_metrics`, `fleet_processes`, and `report_batches` presence rows at or below each machine's persisted watermark older than `retention_days` in one transaction, with tests covering all three fleet tables (presence rows above the watermark are kept, since contiguity extends from the watermark through them). Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. +The same retention transaction also owns purging every other table nothing else claims an owner for: `used_nonces` rows past `expires_at`, `owner_sessions` rows past `expires_at`, `report_receipts` rows older than `retention_days`, `data_gaps` rows fully covered by history already purged above, and `recovery_bundles` rows below the newest `(generation, bundle_revision)` while keeping one prior bundle so an in-flight recovery preview against the second-newest revision does not fail mid-purge. Add tests for each: an expired nonce and an expired session are both gone after the pass, an old receipt is purged, a fully-covered gap range is dropped while a partially-covered one survives, and exactly one prior recovery bundle remains alongside the newest. + - [ ] **Step 6: Verify collection, reporting, and full regression** Run the new tests plus existing current, history, SSE, retention, and full suite tests. @@ -987,17 +1003,19 @@ Pass every returned socket to one `uvicorn.Server.run(sockets=sockets)` call so - [ ] **Step 1: Write listener and settings tests** -Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public-address warnings, optional cert pairs, missing half-pairs, CSRF, self-removal rejection, independent revocation, persisted listeners across restart, and a saved change that does not affect active sockets before restart. +Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public-address warnings, optional cert pairs, missing half-pairs, CSRF, self-removal rejection, independent revocation, persisted listeners across restart, a saved change that does not affect active sockets before restart, a `PUT` that rejects a requested address failing its test bind, and startup with one previously saved non-loopback listener that can no longer bind. - [ ] **Step 2: Change the security rule from mandatory TLS to explicit listeners** `validate_security_config()` must allow HTTP for selected private listeners without requiring a legacy bearer token. Public listeners require an explicit `allow_public_listener` acknowledgement and still show a warning. Store the exact list, HTTPS paths, and acknowledgement in the Task 2 `fleet_settings` row. Startup loads that row and binds exactly those addresses. Never silently widen to `0.0.0.0` after a saved specific-listener configuration. -`PUT` validates and saves settings but does not mutate live sockets. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. +`PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds every requested address (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. + +A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. - [ ] **Step 3: Implement Settings sections** -Add Owner and Sessions, Fleet and Enrolled PCs, Add PC, Network Listeners, Optional HTTPS, Integrations status, and Legacy Compatibility. Destructive actions use plain confirmation copy. +Add Owner and Sessions, Fleet and Enrolled PCs, Add PC, Network Listeners, Optional HTTPS, Integrations status, and Legacy Compatibility. Destructive actions use plain confirmation copy. Network Listeners shows any saved listener startup could not bind as degraded, distinct from an active one, with the plain remediation of fixing or removing it. - [ ] **Step 4: Produce running surface evidence and canary** @@ -1035,8 +1053,11 @@ Run focused tests, static asset verification, browser smoke, full pytest, then c * Produces: `verify_announcement(announcement, fleet_public_key, current_generation) -> bool` * Produces: `HubTransferService.preflight(target_id) -> TransferPreflight` (includes the target's detected private listener candidates) * Produces: `HubTransferService.transfer(target_id, listener: ListenerSelection) -> TransferResult` -* Produces: `FleetStateRepository.enqueue_signed_abort(announcement) -> None` +* Produces: `FleetStateRepository.enqueue_signed_abort(announcement) -> None` (durable row in `hub_transfer_aborts`, Task 2 schema, resumed from startup like `recovery_deliveries`) * Produces: `POST /api/v2/fleet/hub-transfer/preflight` and `/execute` +* Produces: `GET /api/v2/fleet/hub-transfer/status` +* Produces: `HubTransferService.reconcile_draining_transfers() -> None` +* Produces: internal `POST /api/internal/v1/transfer/stage`, `/commit`, `/activate`, `/abort`, and `GET /status` on the target, backed by `pending_hub_authority` Keep the authority switch behind one method: @@ -1045,6 +1066,7 @@ async def transfer(self, target_id: UUID, listener: ListenerSelection) -> Transf preflight = await self.preflight(target_id) preflight.require_ready() preflight.require_listener_candidate(listener) + preflight.require_no_pending_recovery_key() await self.state.enter_transfer_freeze(require_active_target=preflight.target) announcement = None try: @@ -1060,39 +1082,52 @@ async def transfer(self, target_id: UUID, listener: ListenerSelection) -> Transf await self.state.enqueue_signed_abort(announcement) await self.state.lift_transfer_freeze() raise - publication = await self.target_client.activate_publication_obligation(announcement) - await self.target_client.require_publication_persisted(publication) - async with transaction(self.database): - await self.sessions.revoke_all() - await self.state.retire_local_hub(announcement, remove_signing_key=True) - return TransferResult(announcement=announcement) + return TransferResult(announcement=announcement, status="draining") + + +async def reconcile_draining_transfers(self) -> None: + for row in await self.state.due_draining_transfers(): + try: + publication = await self.target_client.activate_publication_obligation(row.announcement) + await self.target_client.require_publication_persisted(publication) + except TransferRetryable: + continue + async with transaction(self.database): + await self.sessions.revoke_all() + await self.state.retire_local_hub(row.announcement, remove_signing_key=True) ``` +`transfer()` returns as soon as draining commits, with `status="draining"`, instead of calling activation inline. Activation sat outside the original `try`/`except` in an earlier draft of this method: a network error there would leave the hub durably draining with no in-process code left to finish it, recoverable only by an eventual restart. `reconcile_draining_transfers()` closes that gap as a background task started from the hub lifespan (the same pattern Task 11's replication loop uses), polling `hub_transfers` rows in state `draining` and retrying activation until `require_publication_persisted()` succeeds, at which point it performs the same session revocation and local retirement the inline call used to. The `/execute` route and the transfer UI observe progress through `GET /api/v2/fleet/hub-transfer/status` rather than blocking on the HTTP request that started the transfer. + - [ ] **Step 1: Write generation, rollback, and interruption tests** Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, a target bind failure on the selected listener failing staging with the target still an agent and the old hub aborting with the freeze lifted, the activated target listening on the owner's selection with the announcement advertising it and never the old hub's imported addresses or certificate paths, an aborted target holding no staged fleet key file, a crash between the target's activation commit and key finalization repaired at startup into a hub that can sign, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the target is unreachable so abort delivery itself fails (the local lift must still commit and the abort must retry from its durable queue), freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. Add a target continuity test: the target collects samples into its own outbox during staging and readiness, activation preserves that outbox and the target's identity, and those samples appear in the new hub's history afterward, alongside the demoted old hub delivering the outbox rows it kept out of the snapshot. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. +Add a preflight test that an unacknowledged `pending_recovery_key` row blocks transfer with an explicit error and changes nothing, and a follow-up transfer succeeding once the key is acknowledged. Add abort-scoping tests: a signed abort for a digest the target has already activated is rejected and reported so the old hub drops it from `hub_transfer_aborts` without retrying forever, and a stale abort queued for a failed transfer #1 does not touch a target that went on to activate a successful transfer #2 to the same machine. Add a `pending_hub_authority` restart test: kill the target between `commit_unannounced_hub` and activation, restart it, and prove it comes back in `RuntimeRole.PENDING_HUB` serving only the internal activation, abort, and status routes, never the dashboard. Add a background reconciler test: raise inside `activate_publication_obligation` after draining commits, prove `transfer()` itself returns `status="draining"` rather than raising, then run `reconcile_draining_transfers()` and prove it completes the retirement from the persisted `hub_transfers` row with no further caller involvement. Add a test that recovery bundle creation and replication (Task 11) are rejected for the whole frozen and draining span, matching every other fleet mutation. + - [ ] **Step 2: Implement signed announcements** Canonicalize announcement JSON with sorted keys and compact separators before signing. Store the accepted generation atomically with new hub addresses. - [ ] **Step 3: Implement encrypted transfer staging** -Create a SQLite backup through the backup API. Package fleet config, owner hash, registry, history, integrations, and fleet signing key. Exclude sessions and the old hub's machine-local `report_outbox`. Encrypt to the target X25519 public key with HKDF-SHA256 and AES-256-GCM. +Create a SQLite backup through the backup API. Package fleet config, owner hash, registry, history, integrations, and fleet signing key. Exclude sessions, the old hub's machine-local `report_outbox`, and its `sqlite_sequence` allocator row for that table: `report_outbox.sequence` is `INTEGER PRIMARY KEY AUTOINCREMENT` (Task 2), and importing the old hub's `sqlite_sequence` entry over the target's own would rewind the target's outbox allocator below its already-acknowledged watermark, silently making every later local sample look like an old already-acknowledged sequence and stopping that machine's own history permanently. Encrypt to the target X25519 public key with HKDF-SHA256 and AES-256-GCM. - [ ] **Step 4: Implement the staged transfer and explicit abort path** -The target imports into staging, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and starts in unannounced hub mode. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target binds and holds that socket before `commit_unannounced_hub`, then attaches it through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. +The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target binds and holds that socket before `commit_unannounced_hub`, then attaches it through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. + +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. -The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. A signed abort deletes the staged key file along with the staged authority, so an aborted target resumes agent mode holding no authoritative key material, and a crash between the activation commit and the move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. It never replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +Preflight also rejects the transfer outright while an unacknowledged `pending_recovery_key` row exists, with an explicit "acknowledge the recovery key before moving the hub" error and no state change: transferring the fleet signing key to the target while the owner has never confirmed the printable recovery key is saved would silently destroy the only other recovery path this fleet has, so the plan picks the branch of refusing the transfer rather than the branch of carrying the pending key across it. -The old hub durably enters a transfer freeze before the snapshot is taken. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The same transaction rejects the transfer while any enrollment attempt is `claimed`: a claimed attempt's retry nonce is encrypted to the old hub's machine identity, which never transfers, so the new hub could not finish it and the enrollment would strand. Claims resolve or expire within the ten-minute pairing window, and expired or failed attempts do not block. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, and integration changes. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path durably enqueues the signed abort and lifts the freeze in one local transaction, and a delivery loop retries the abort until the target acknowledges it. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. +The old hub durably enters a transfer freeze before the snapshot is taken, and that freeze transaction is the write to `runtime_state.transfer_state` (`frozen`) and `runtime_state.transfer_id` (naming the new `hub_transfers` row) that the shared-names schema section describes: everything from here through draining reads that one column. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The same transaction rejects the transfer while any enrollment attempt is `claimed`: a claimed attempt's retry nonce is encrypted to the old hub's machine identity, which never transfers, so the new hub could not finish it and the enrollment would strand. Claims resolve or expire within the ten-minute pairing window, and expired or failed attempts do not block. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, integration changes, and recovery bundle creation or replication (Task 11), all reading the same `transfer_state` column. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Rejecting recovery bundle work means no bundle naming the pre-transfer generation and address set can be created or handed to an agent mid-transfer, which would otherwise let a promotion race the transfer to a stale outcome. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path inserts the row into `hub_transfer_aborts` and lifts the freeze (writing `transfer_state` back to `accepting`) in one local transaction, and a delivery loop retries the abort from that durable row, on the same 1, 2, 4, 8, 16, then 30 minute backoff Task 11's delivery loops use, until the target acknowledges it or reports the digest already resolved, either of which stamps `acknowledged_at` and stops the retry. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal (`runtime_state.transfer_id` into `hub_transfers`) to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. -If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining and reconciles the target's signed obligation status, retrying the same activation request until it obtains the persisted receipt. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. +If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, `enter_draining_transfer()` writes `runtime_state.transfer_state` to `draining` in the same transaction that records the announcement on the `hub_transfers` row, so the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining, and `reconcile_draining_transfers()` (the background task described above) is what retries the same activation request against the persisted `hub_transfers` row until it obtains the persisted receipt, rather than a caller blocking inline on the original `/execute` request. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. - [ ] **Step 5: Add Move Hub UI and evidence** -Show preflight results including the target's detected private listener candidates, require the owner to pick the new hub's listener from them before execute, then show transfer progress, exact new IP and port, and a link to open it. Capture successful transfer and forced rollback. Canary by failing the target validation probe and confirming the old hub remains active. +Show preflight results including the target's detected private listener candidates and the explicit pending-recovery-key block when it applies, require the owner to pick the new hub's listener from them before execute, then poll `GET /api/v2/fleet/hub-transfer/status` for progress (draining, publication pending, complete) rather than blocking on the execute request, and show the exact new IP and port with a link to open it once complete. Capture successful transfer and forced rollback. Canary by failing the target validation probe and confirming the old hub remains active. - [ ] **Step 6: Verify and commit** @@ -1140,19 +1175,23 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence, integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, missing history disclosure, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, and replica replacement on agents. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, and the replication loop declining to run and exiting cleanly once role leaves `RuntimeRole.HUB` or `transfer_state` leaves `accepting`. - [ ] **Step 2: Implement bundle creation and replication** Serialize fleet identity, machine registry, settings, integrations, schema version, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind both generation and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the generation, revision, ciphertext digest, and associated data. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. -Start `run_replication_loop()` from the hub lifespan. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple has a higher generation or the same generation with a higher revision. They reject an older or equal revision with a different digest, retain the newest valid bundle across restart, and return the accepted generation, revision, and digest so the hub cannot acknowledge stale ciphertext. +Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. The loop exits cleanly the moment role changes away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting), rather than continuing to run against a database it no longer owns. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple has a higher generation or the same generation with a higher revision. They reject an older or equal revision with a different digest, retain the newest valid bundle across restart, and return the accepted generation, revision, and digest so the hub cannot acknowledge stale ciphertext. - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. + +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. + +The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. +The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, and the replayed history acknowledges. @@ -1211,11 +1250,11 @@ def owner_or_scope(required_scope: IntegrationScope): - [ ] **Step 1: Write pairing and scope tests** -Assert ten-minute one-use pairing, independent credential revocation, no owner or agent credential reuse, read-only enforcement, stable machine UUIDs, and access to current, history, availability, and events. Exercise every existing Task 6 route through its real router with an owner session, its required integration scope, a missing scope, and a revoked integration. +Assert ten-minute one-use pairing, independent credential revocation, no owner or agent credential reuse, read-only enforcement, stable machine UUIDs, and access to current, history, availability, and events. Exercise every existing Task 6 route through its real router with an owner session, its required integration scope, a missing scope, and a revoked integration. Assert exchange idempotency: an identical retry of `POST /api/v2/integrations/exchange` with the same pairing code and the same client public key returns the same integration record rather than minting a second one, and a retry against a code already consumed by a different client public key returns `409`. - [ ] **Step 2: Implement integration request authentication** -Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. +Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`: a retry presenting the same client public key against the same still-valid or already-consumed code returns the existing integration record unchanged, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. - [ ] **Step 3: Publish a machine-readable contract fixture** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 331ac87..168b79c 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -303,6 +303,7 @@ Preflight verifies: * Target has enough free storage. * Target can listen on at least one selected private address. * The current database can produce a consistent snapshot. +* No unacknowledged recovery key is pending. An owner who has not yet confirmed the printable recovery key is saved must acknowledge it first, since transferring the fleet signing key without that acknowledgement would destroy the fleet's only other recovery path. Transfer then: @@ -363,7 +364,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 7. Revoke browser sessions and integration credentials. 8. Accept reconnecting agents and rebuild central history from their retained buffers. -The UI states how much history can be recovered before promotion. History older than all agent buffers is not promised. +The UI states how much history can be recovered before promotion. History older than all agent buffers is not promised. The UI also states the recovered bundle's creation time and that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion, not merely reconnected. A machine reporting to the recovered hub without a registry entry gets a rejection distinguishable from an ordinary authentication failure, and its own local recovery page surfaces that it needs re-enrollment. ## SlipShell compatibility boundary From 13885b85fc5baeb6ced56728fa5444c0dde1f90f Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:13:43 -0400 Subject: [PATCH 18/65] docs: close the fourteenth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../2026-08-13-humwatch-2.1-fleet-hub.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 94bfbcd..0d5c7d7 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -359,7 +359,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, hub_announcement_json, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -373,6 +373,7 @@ data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) used_nonces(principal_type, principal_id, nonce_hash, expires_at) enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) +integration_pairing_codes(pairing_code_id, code_hash, created_at, expires_at, consumed_at, integration_id, client_public_key) hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) hub_transfer_aborts(transfer_id, announcement_generation, announcement_digest, abort_json, state, attempt_count, next_attempt_at, acknowledged_at) pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staged_snapshot_path, staged_key_path, state, created_at, activated_at) @@ -384,13 +385,13 @@ recovery_bundles(generation, bundle_revision, created_at, encrypted_payload, fle recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` -Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. -`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves nothing but the internal activation, abort, and status routes, never the dashboard or agent report ingestion. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves nothing but the internal activation, abort, and status routes, never the dashboard or agent report ingestion. It also owns the transaction helper used by every fleet repository: @@ -530,7 +531,7 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the generation-1 fleet-signed `HubAnnouncement` built from that listener (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key, an in-process attach test proving a request on the newly advertised listener succeeds with no process restart, and a kill between commit and attach proving the restarted process serves the stored listener. @@ -644,7 +645,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its keys, addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the two pieces of trust the agent needs for everything after enrollment. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. +Finalize also persists the two pieces of trust the agent needs for everything after enrollment. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json`, the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. - [ ] **Step 6: Verify enrollment and commit** @@ -719,7 +720,7 @@ Run: `.venv/bin/python -m pytest tests/test_report_outbox.py tests/test_hub_inge - [ ] **Step 3: Add one collector persistence boundary** -Refactor the end of each collection tick to build one `ReportBatch`. In `legacy`, keep the existing `_write_metrics` and `_write_processes`. In `agent` or `hub`, append the same normalized data to `report_outbox`. A hub immediately ingests its own batch through `HubIngestion`, then acknowledges its own outbox row only after that transaction commits. +Refactor the end of each collection tick to build one `ReportBatch`. In `legacy`, keep the existing `_write_metrics` and `_write_processes`. In `agent` or `hub`, append the same normalized data to `report_outbox`. A hub immediately ingests its own batch through `HubIngestion`, then acknowledges its own outbox row only after that transaction commits. The self-ingest loop does not start with the newest batch: whenever the process enters `hub` role (normal startup, transfer activation in Task 10, recovery promotion in Task 11), it first drains every unacknowledged `report_outbox` row oldest first through the same `HubIngestion` path before live batches, because a machine that buffered samples while it could not reach a hub (a transfer target collecting through the frozen span, or any hub restarted mid-outage) holds rows that no live tick will ever revisit and that agent transport, stopped in `hub` role, will never send. Test that rows appended in `agent` role appear in central history after the machine becomes a hub with no process restart between. - [ ] **Step 4: Build and verify the durable upload envelope** @@ -1117,7 +1118,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target binds and holds that socket before `commit_unannounced_hub`, then attaches it through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. -The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch. The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. Preflight also rejects the transfer outright while an unacknowledged `pending_recovery_key` row exists, with an explicit "acknowledge the recovery key before moving the hub" error and no state change: transferring the fleet signing key to the target while the owner has never confirmed the printable recovery key is saved would silently destroy the only other recovery path this fleet has, so the plan picks the branch of refusing the transfer rather than the branch of carrying the pending key across it. @@ -1175,7 +1176,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, and the replication loop declining to run and exiting cleanly once role leaves `RuntimeRole.HUB` or `transfer_state` leaves `accepting`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, and the replication loop declining to run and exiting cleanly once role leaves `RuntimeRole.HUB` or `transfer_state` leaves `accepting`. - [ ] **Step 2: Implement bundle creation and replication** @@ -1185,7 +1186,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. @@ -1254,7 +1255,7 @@ Assert ten-minute one-use pairing, independent credential revocation, no owner o - [ ] **Step 2: Implement integration request authentication** -Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`: a retry presenting the same client public key against the same still-valid or already-consumed code returns the existing integration record unchanged, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. +Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema), and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. - [ ] **Step 3: Publish a machine-readable contract fixture** From 158b9be864591db176403763131c9cdcfacfa60c Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:20:21 -0400 Subject: [PATCH 19/65] docs: close the fifteenth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 0d5c7d7..ef9271b 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -188,7 +188,7 @@ X-HumWatch-Signature: `SignedHeaders` is the typed mapping containing those five headers. The signed bytes are `METHOD + "\n" + PATH + "\n" + FLEET_ID + "\n" + MACHINE_ID + "\n" + SHA256(BODY) + "\n" + TIMESTAMP + "\n" + NONCE`. The hub rejects timestamps beyond 120 seconds, reused nonces, unknown machines, revoked machines, relabeled fleet or machine headers, and invalid signatures. -Agents authenticate the hub's side of this envelope with `hub_signing_public_key`: enrollment finalize persists the key carried on `HubClaim`, and every later fleet-signed `HubAnnouncement` the agent accepts (Task 10 transfer, Task 11 recovery) carries the current hub's `hub_signing_public_key` too, so the agent's authorized hub key rotates automatically the moment a higher-generation announcement passes fleet-key verification, with no separate distribution step. Inbound hub-originated signed requests to an agent (announcement delivery, recovery replica delivery, both Task 10 and 11) are verified against that persisted key rather than trusting the claimed machine ID alone. +Agents authenticate the hub's side of this envelope with `hub_signing_public_key`: enrollment finalize persists the key carried on `HubClaim`, and every later fleet-signed `HubAnnouncement` the agent accepts (Task 10 transfer, Task 11 recovery) carries the current hub's `hub_signing_public_key` too, so the agent's authorized hub key rotates automatically the moment a higher-generation announcement passes fleet-key verification, with no separate distribution step. Inbound hub-originated signed requests to an agent (recovery replica delivery and every other push) are verified against that persisted key rather than trusting the claimed machine ID alone. Announcement delivery is the one deliberate exception, because it is the rotation mechanism itself: during a transfer or recovery the new hub signs with its own machine key, which no agent has persisted yet, and requiring the old key would make redirection impossible. The announcement payload is fleet-signed and self-authenticating, so the delivery route verifies the fleet signature and the generation rule on the payload alone, and acceptance atomically rotates the persisted `hub_signing_public_key` to the announcement's, after which every other inbound push from the new hub verifies normally. A forged delivery envelope around a legitimate announcement gains nothing: the payload either passes fleet-key verification or is discarded. Browser mutations send the CSRF token in `X-HumWatch-CSRF`. Session cookies are named `humwatch_session`. @@ -391,7 +391,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. -`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves nothing but the internal activation, abort, and status routes, never the dashboard or agent report ingestion. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. It also owns the transaction helper used by every fleet repository: @@ -1010,7 +1010,7 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `validate_security_config()` must allow HTTP for selected private listeners without requiring a legacy bearer token. Public listeners require an explicit `allow_public_listener` acknowledgement and still show a warning. Store the exact list, HTTPS paths, and acknowledgement in the Task 2 `fleet_settings` row. Startup loads that row and binds exactly those addresses. Never silently widen to `0.0.0.0` after a saved specific-listener configuration. -`PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds every requested address (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. +`PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. @@ -1104,7 +1104,7 @@ async def reconcile_draining_transfers(self) -> None: Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, a target bind failure on the selected listener failing staging with the target still an agent and the old hub aborting with the freeze lifted, the activated target listening on the owner's selection with the announcement advertising it and never the old hub's imported addresses or certificate paths, an aborted target holding no staged fleet key file, a crash between the target's activation commit and key finalization repaired at startup into a hub that can sign, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the target is unreachable so abort delivery itself fails (the local lift must still commit and the abort must retry from its durable queue), freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. Add a target continuity test: the target collects samples into its own outbox during staging and readiness, activation preserves that outbox and the target's identity, and those samples appear in the new hub's history afterward, alongside the demoted old hub delivering the outbox rows it kept out of the snapshot. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. -Add a preflight test that an unacknowledged `pending_recovery_key` row blocks transfer with an explicit error and changes nothing, and a follow-up transfer succeeding once the key is acknowledged. Add abort-scoping tests: a signed abort for a digest the target has already activated is rejected and reported so the old hub drops it from `hub_transfer_aborts` without retrying forever, and a stale abort queued for a failed transfer #1 does not touch a target that went on to activate a successful transfer #2 to the same machine. Add a `pending_hub_authority` restart test: kill the target between `commit_unannounced_hub` and activation, restart it, and prove it comes back in `RuntimeRole.PENDING_HUB` serving only the internal activation, abort, and status routes, never the dashboard. Add a background reconciler test: raise inside `activate_publication_obligation` after draining commits, prove `transfer()` itself returns `status="draining"` rather than raising, then run `reconcile_draining_transfers()` and prove it completes the retirement from the persisted `hub_transfers` row with no further caller involvement. Add a test that recovery bundle creation and replication (Task 11) are rejected for the whole frozen and draining span, matching every other fleet mutation. +Add a preflight test that an unacknowledged `pending_recovery_key` row blocks transfer with an explicit error and changes nothing, and a follow-up transfer succeeding once the key is acknowledged. Add abort-scoping tests: a signed abort for a digest the target has already activated is rejected and reported so the old hub drops it from `hub_transfer_aborts` without retrying forever, and a stale abort queued for a failed transfer #1 does not touch a target that went on to activate a successful transfer #2 to the same machine. Add a `pending_hub_authority` restart test: kill the target between `commit_unannounced_hub` and activation, restart it, and prove it comes back in `RuntimeRole.PENDING_HUB` serving the internal activation, abort, and status routes plus the readiness surface (login page, generic `401` login, proposed-generation report ingestion), and nothing else: no authenticated dashboard data route and no fleet mutation route. Add a background reconciler test: raise inside `activate_publication_obligation` after draining commits, prove `transfer()` itself returns `status="draining"` rather than raising, then run `reconcile_draining_transfers()` and prove it completes the retirement from the persisted `hub_transfers` row with no further caller involvement. Add a test that recovery bundle creation and replication (Task 11) are rejected for the whole frozen and draining span, matching every other fleet mutation. - [ ] **Step 2: Implement signed announcements** @@ -1116,7 +1116,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, - [ ] **Step 4: Implement the staged transfer and explicit abort path** -The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target binds and holds that socket before `commit_unannounced_hub`, then attaches it through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. +The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target secures that socket before `commit_unannounced_hub`: when the selection names an address the target's `ListenerManager` already serves (the normal case, since Task 4 keeps the enrolled agent's listener bound for announcement and recovery routes), staging reuses that existing bound socket and atomically widens its route set at commit instead of binding the same address a second time, which would fail address-in-use on every ordinary transfer. Only a selection naming a genuinely new address is freshly bound and held, then attached through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure on such a new address fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. @@ -1176,13 +1176,13 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, and the replication loop declining to run and exiting cleanly once role leaves `RuntimeRole.HUB` or `transfer_state` leaves `accepting`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. - [ ] **Step 2: Implement bundle creation and replication** Serialize fleet identity, machine registry, settings, integrations, schema version, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind both generation and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the generation, revision, ciphertext digest, and associated data. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. -Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. The loop exits cleanly the moment role changes away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting), rather than continuing to run against a database it no longer owns. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple has a higher generation or the same generation with a higher revision. They reject an older or equal revision with a different digest, retain the newest valid bundle across restart, and return the accepted generation, revision, and digest so the hub cannot acknowledge stale ciphertext. +Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. A non-`accepting` `transfer_state` pauses the loop rather than ending it: the tick skips its work and the loop keeps polling, so when a failed transfer's cleanup lifts the freeze back to `accepting` the very next tick resumes periodic bundles and queued delivery rows with no process restart. Only a role change away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting) exits the loop for good, since that database is no longer its to replicate. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple has a higher generation or the same generation with a higher revision. They reject an older or equal revision with a different digest, retain the newest valid bundle across restart, and return the accepted generation, revision, and digest so the hub cannot acknowledge stale ciphertext. - [ ] **Step 3: Implement local recovery preview and promotion** From 9f8c724c18e70acaa85d0c67fbd2a4214ce848c1 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:27:38 -0400 Subject: [PATCH 20/65] docs: close the sixteenth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .gitleaksignore | 2 -- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 12 ++++++------ .../2026-08-11-humwatch-2.1-fleet-hub-design.md | 2 +- 3 files changed, 7 insertions(+), 9 deletions(-) delete mode 100644 .gitleaksignore diff --git a/.gitleaksignore b/.gitleaksignore deleted file mode 100644 index 4de41a4..0000000 --- a/.gitleaksignore +++ /dev/null @@ -1,2 +0,0 @@ -df36c1d8d28aec103787fa84096bc1f3fd853fab:docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md:generic-api-key:399 -df36c1d8d28aec103787fa84096bc1f3fd853fab:docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md:generic-api-key:400 diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index ef9271b..fffe3db 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -359,7 +359,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, hub_announcement_json, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -391,7 +391,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. -`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. It also owns the transaction helper used by every fleet repository: @@ -424,11 +424,11 @@ Use raw private key bytes stored below `resolved_data_root / "identity"`. Create @dataclass(frozen=True) class MachineIdentity: machine_id: UUID - signing_private_key: Ed25519PrivateKey - encryption_private_key: X25519PrivateKey + signing: Ed25519PrivateKey + encryption: X25519PrivateKey def sign(self, payload: bytes) -> bytes: - return self.signing_private_key.sign(payload) + return self.signing.sign(payload) ``` `FleetIdentity.create()` returns the stored fleet signing identity plus a printable grouped base64url recovery private key with a checksum. Persist only the recovery public key. @@ -1194,7 +1194,7 @@ The promoting machine's own history needs the identical treatment, because it ne The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, and the replayed history acknowledges. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the watermark handshake after adoption carries the agent's `generation_floor`, and a hub that learns of a floor at or above its own generation mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 168b79c..940da05 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -326,7 +326,7 @@ Every hub announcement contains the fleet UUID, hub machine UUID, addresses, and Agents accept only a valid announcement with a higher generation than the one stored locally. An old hub that returns later cannot reclaim the fleet with stale state. -One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. +One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. Adoption never lowers the machine's monotonic generation floor, so the dead lineage's higher-generation announcement cannot be replayed to redirect the adopted machine, and the recovered hub learns of the floor and re-announces itself above it, converging the fleet past the dead lineage. ### Address change From 24149c1d0e09074fb077718b73e38c4199d9ed40 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:34:31 -0400 Subject: [PATCH 21/65] docs: close the seventeenth-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index fffe3db..92f63af 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -359,7 +359,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -391,7 +391,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. -`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. It also owns the transaction helper used by every fleet repository: @@ -645,7 +645,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its keys, addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the two pieces of trust the agent needs for everything after enrollment. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json`, the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json`, the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. - [ ] **Step 6: Verify enrollment and commit** From c87c1d50a567d05ccbc7b099a303f11ee3fab947 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:40:34 -0400 Subject: [PATCH 22/65] docs: close the eighteenth-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 92f63af..e660f3b 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -376,7 +376,7 @@ integration_clients(integration_id, name, public_key, scopes_json, created_at, r integration_pairing_codes(pairing_code_id, code_hash, created_at, expires_at, consumed_at, integration_id, client_public_key) hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) hub_transfer_aborts(transfer_id, announcement_generation, announcement_digest, abort_json, state, attempt_count, next_attempt_at, acknowledged_at) -pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staged_snapshot_path, staged_key_path, state, created_at, activated_at) +pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staged_snapshot_path, staged_key_path, prior_settings_json, state, created_at, activated_at) hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) pending_recovery_key(id, printable_key, created_at) @@ -1104,7 +1104,7 @@ async def reconcile_draining_transfers(self) -> None: Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, a target bind failure on the selected listener failing staging with the target still an agent and the old hub aborting with the freeze lifted, the activated target listening on the owner's selection with the announcement advertising it and never the old hub's imported addresses or certificate paths, an aborted target holding no staged fleet key file, a crash between the target's activation commit and key finalization repaired at startup into a hub that can sign, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the target is unreachable so abort delivery itself fails (the local lift must still commit and the abort must retry from its durable queue), freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. Add a target continuity test: the target collects samples into its own outbox during staging and readiness, activation preserves that outbox and the target's identity, and those samples appear in the new hub's history afterward, alongside the demoted old hub delivering the outbox rows it kept out of the snapshot. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. -Add a preflight test that an unacknowledged `pending_recovery_key` row blocks transfer with an explicit error and changes nothing, and a follow-up transfer succeeding once the key is acknowledged. Add abort-scoping tests: a signed abort for a digest the target has already activated is rejected and reported so the old hub drops it from `hub_transfer_aborts` without retrying forever, and a stale abort queued for a failed transfer #1 does not touch a target that went on to activate a successful transfer #2 to the same machine. Add a `pending_hub_authority` restart test: kill the target between `commit_unannounced_hub` and activation, restart it, and prove it comes back in `RuntimeRole.PENDING_HUB` serving the internal activation, abort, and status routes plus the readiness surface (login page, generic `401` login, proposed-generation report ingestion), and nothing else: no authenticated dashboard data route and no fleet mutation route. Add a background reconciler test: raise inside `activate_publication_obligation` after draining commits, prove `transfer()` itself returns `status="draining"` rather than raising, then run `reconcile_draining_transfers()` and prove it completes the retirement from the persisted `hub_transfers` row with no further caller involvement. Add a test that recovery bundle creation and replication (Task 11) are rejected for the whole frozen and draining span, matching every other fleet mutation. +Add a preflight test that an unacknowledged `pending_recovery_key` row blocks transfer with an explicit error and changes nothing, and a follow-up transfer succeeding once the key is acknowledged. Add abort-scoping tests: a signed abort for a digest the target has already activated is rejected and reported so the old hub drops it from `hub_transfer_aborts` without retrying forever, and a stale abort queued for a failed transfer #1 does not touch a target that went on to activate a successful transfer #2 to the same machine. Add abort listener-restore tests: after an abort of a transfer that selected a genuinely new address, the target serves announcement and recovery pushes on its original enrolled address (in the running process and again after a restart) with the transfer socket detached, and after an abort of a transfer that reused the enrolled socket, that socket answers only the post-enrollment allowlist with every readiness route refused. Add a `pending_hub_authority` restart test: kill the target between `commit_unannounced_hub` and activation, restart it, and prove it comes back in `RuntimeRole.PENDING_HUB` serving the internal activation, abort, and status routes plus the readiness surface (login page, generic `401` login, proposed-generation report ingestion), and nothing else: no authenticated dashboard data route and no fleet mutation route. Add a background reconciler test: raise inside `activate_publication_obligation` after draining commits, prove `transfer()` itself returns `status="draining"` rather than raising, then run `reconcile_draining_transfers()` and prove it completes the retirement from the persisted `hub_transfers` row with no further caller involvement. Add a test that recovery bundle creation and replication (Task 11) are rejected for the whole frozen and draining span, matching every other fleet mutation. - [ ] **Step 2: Implement signed announcements** @@ -1118,7 +1118,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target secures that socket before `commit_unannounced_hub`: when the selection names an address the target's `ListenerManager` already serves (the normal case, since Task 4 keeps the enrolled agent's listener bound for announcement and recovery routes), staging reuses that existing bound socket and atomically widens its route set at commit instead of binding the same address a second time, which would fail address-in-use on every ordinary transfer. Only a selection naming a genuinely new address is freshly bound and held, then attached through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure on such a new address fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. -The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. Preflight also rejects the transfer outright while an unacknowledged `pending_recovery_key` row exists, with an explicit "acknowledge the recovery key before moving the hub" error and no state change: transferring the fleet signing key to the target while the owner has never confirmed the printable recovery key is saved would silently destroy the only other recovery path this fleet has, so the plan picks the branch of refusing the transfer rather than the branch of carrying the pending key across it. From 1a21189511aeb20fb8c83a7cc50ef2472bfe9f3c Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:44:47 -0400 Subject: [PATCH 23/65] docs: close the nineteenth-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index e660f3b..b4b183c 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -553,7 +553,7 @@ Run: Commit: ```bash -git add agent/security/passwords.py agent/security/sessions.py agent/fleet/setup.py agent/routes/v2_session.py agent/routes/v2_setup.py agent/security/limits.py agent/main.py tests/conftest.py tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py tests/test_login_throttle.py tests/test_api_auth.py +git add agent/security/passwords.py agent/security/sessions.py agent/fleet/setup.py agent/fleet/listeners.py agent/fleet/schema.py agent/routes/v2_session.py agent/routes/v2_setup.py agent/security/limits.py agent/main.py tests/conftest.py tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py tests/test_login_throttle.py tests/test_api_auth.py git commit -m "feat: add owner sessions and fleet creation" ``` From 7909fe0520faf5a4c50be00efaf928a551528246 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:49:48 -0400 Subject: [PATCH 24/65] docs: close the twentieth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index b4b183c..8f86222 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -531,7 +531,7 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the generation-1 fleet-signed `HubAnnouncement` built from that listener (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the generation-1 fleet-signed `HubAnnouncement` built from that listener (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Every attached server task is configured with lifespan disabled: the process's original server already ran the app's startup and owns the shared database connection, the collectors, and the retention and delivery loops, and a second server invocation running the lifespan again would reopen or replace that connection and start duplicate copies of every background task the moment the fleet is created. Attached servers only serve requests, and the one lifecycle they participate in is the original server's shutdown, which detaches them. Test it by asserting exactly one collector tick source and one database connection before and after an attach. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key, an in-process attach test proving a request on the newly advertised listener succeeds with no process restart, and a kill between commit and attach proving the restarted process serves the stored listener. @@ -683,14 +683,16 @@ Commit message: `feat: enroll agents with one-time codes` * Produces: `AgentTransport.run() -> None` * Produces: `POST /api/internal/v1/reports` -The ingestion transaction has this shape. Both admission predicates run inside the serialized transaction, not before it: enrollment, revocation, and the Task 10 transfer-freeze all commit through the same per-connection transaction lock, so checking them outside the transaction would let a batch pass the check, lose the lock race, and be stored and acknowledged after a revocation or freeze committed. `require_accepting_reports()` reads `runtime_state.transfer_state` and rejects unless it is `accepting`. In this task nothing ever sets it to `frozen` or `draining`, and Task 10 relies on the predicate reading that exact column being in place before its freeze and drain transactions exist. +The ingestion transaction has this shape. Both admission predicates run inside the serialized transaction, not before it: enrollment, revocation, and the Task 10 transfer-freeze all commit through the same per-connection transaction lock, so checking them outside the transaction would let a batch pass the check, lose the lock race, and be stored and acknowledged after a revocation or freeze committed. `require_accepting_reports()` reads `runtime_state.transfer_state` and rejects unless it is `accepting`. In this task nothing ever sets it to `frozen` or `draining`, and Task 10 relies on the predicate reading that exact column being in place before its freeze and drain transactions exist. `stored_ack_or_reject_conflict` is the identical-retry exit: a matching upload ID and digest returns the persisted acknowledgement from the receipt row before any recomputation, so a retry of upload A after a later upload B advanced the watermark answers with A's original acknowledgement, never B's, and skips the event publish since nothing changed (a same-ID different-digest upload raises the `409` here instead). ```python async def ingest(self, upload: SignedReportUpload) -> IngestAck: async with transaction(self.database): await self.machines.require_active(upload.machine_id) await self.state.require_accepting_reports() - await self.receipts.require_new_or_identical(upload) + stored = await self.receipts.stored_ack_or_reject_conflict(upload) + if stored is not None: + return stored await self.samples.insert_upload_if_absent(upload) await self.batches.insert_presence_if_absent(upload) await self.gaps.insert_if_absent(upload.machine_id, upload.gaps) From 9aa2c90f0346167b65e3621290b8ce18b16f0cd3 Mon Sep 17 00:00:00 2001 From: Static Date: Thu, 13 Aug 2026 23:57:40 -0400 Subject: [PATCH 25/65] docs: close the twenty-first-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 8f86222..0fc5631 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -389,7 +389,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. -`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row by completing the move into the identity location, and finalization stamps `finalized_at`. `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. +`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. Test both windows for all three writers (creation, transfer activation, promotion). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. `runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. @@ -645,7 +645,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its keys, addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json`, the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json`, the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. - [ ] **Step 6: Verify enrollment and commit** @@ -1188,7 +1188,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. From 548b72b0deacc6bffedf72f4d27ba065ebb7676f Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:02:56 -0400 Subject: [PATCH 26/65] docs: close the twenty-second-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 ++---- .../specs/2026-08-11-humwatch-2.1-fleet-hub-design.md | 2 +- 2 files changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 0fc5631..f3768e5 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1178,7 +1178,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. - [ ] **Step 2: Implement bundle creation and replication** @@ -1188,14 +1188,12 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. Preview probes every address in the imported registry, keeps the highest verified generation, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, locally_accepted_generation, highest_probed_generation) + 1`, where the third term is the highest fleet-signature-verified generation returned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response also reports the machine's `generation_floor` beside the announcement: the floor is not separately signed, but an inflated floor can only push the promotion generation higher, which costs nothing and protects the reporter, while the fleet-signed announcement remains the authenticated component. Preview probes every address in the imported registry, keeps the highest value among verified announcement generations and reported floors, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The second term matters after a hub transfer: the recovering machine may have accepted generation `G+1` through the announcement obligation while its newest bundle is still from `G`, and promoting to `G+1` would be rejected by every agent, including itself, as an equal generation. - The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the watermark handshake after adoption carries the agent's `generation_floor`, and a hub that learns of a floor at or above its own generation mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 940da05..b022f9f 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -360,7 +360,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. -6. Promote the PC to hub at a generation above the bundle's generation, the generation this machine last accepted from an announcement, and the highest fleet-signature-verified generation any reachable fleet member reports accepting, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. +6. Promote the PC to hub at a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its floor, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. 7. Revoke browser sessions and integration credentials. 8. Accept reconnecting agents and rebuild central history from their retained buffers. From f77e2ebfbb6517fd2351e3045dbc9d4019b536ae Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:06:49 -0400 Subject: [PATCH 27/65] docs: close the twenty-third-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- .../specs/2026-08-11-humwatch-2.1-fleet-hub-design.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index f3768e5..72f4ba8 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1178,7 +1178,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe response whose floor signature fails registry-key verification contributing its announcement but not its floor (an attacker-supplied huge floor never reaches the generation formula), a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. - [ ] **Step 2: Implement bundle creation and replication** @@ -1188,7 +1188,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response also reports the machine's `generation_floor` beside the announcement: the floor is not separately signed, but an inflated floor can only push the promotion generation higher, which costs nothing and protects the reporter, while the fleet-signed announcement remains the authenticated component. Preview probes every address in the imported registry, keeps the highest value among verified announcement generations and reported floors, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response also reports the machine's `generation_floor` beside the announcement, and the floor arrives authenticated: the responding machine signs the tuple of its machine ID, its floor, and the prober's request nonce with its machine signing key, and the prober verifies that signature against the machine's public key in the imported registry before using the floor. An unauthenticated floor would let any LAN host that can answer a registered address pair a replayed fleet-signed announcement with an arbitrary number, and a floor near the integer ceiling would poison the monotonic generation counter for every later transfer and recovery, so an unverifiable floor is discarded while the fleet-signed announcement in the same response remains usable on its own signature. Preview probes every address in the imported registry, keeps the highest value among verified announcement generations and signature-verified floors, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index b022f9f..817116e 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -360,7 +360,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. -6. Promote the PC to hub at a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its floor, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. +6. Promote the PC to hub at a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its machine-signed floor, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. 7. Revoke browser sessions and integration credentials. 8. Accept reconnecting agents and rebuild central history from their retained buffers. From ee7b8a85f8cde40146b6b05574990b0d86dfa4a3 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:12:03 -0400 Subject: [PATCH 28/65] docs: close the twenty-fourth-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 72f4ba8..4a52398 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1190,11 +1190,11 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response also reports the machine's `generation_floor` beside the announcement, and the floor arrives authenticated: the responding machine signs the tuple of its machine ID, its floor, and the prober's request nonce with its machine signing key, and the prober verifies that signature against the machine's public key in the imported registry before using the floor. An unauthenticated floor would let any LAN host that can answer a registered address pair a replayed fleet-signed announcement with an arbitrary number, and a floor near the integer ceiling would poison the monotonic generation counter for every later transfer and recovery, so an unverifiable floor is discarded while the fleet-signed announcement in the same response remains usable on its own signature. Preview probes every address in the imported registry, keeps the highest value among verified announcement generations and signature-verified floors, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `GET /api/internal/v1/reports/watermark` returning the hub's generation and stored watermark for the calling machine. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation and `generation_floor` (the envelope's body digest authenticates both, and a tampering test proves a modified floor fails verification), returning the hub's generation and stored watermark for that machine. The floor in that signed body is what drives the convergence rule: a hub that reads a verified floor at or above its own generation mints its fresh self-announcement at floor plus one, so the mechanism is a defined, authenticated field rather than an unspecified header. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the watermark handshake after adoption carries the agent's `generation_floor`, and a hub that learns of a floor at or above its own generation mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads a verified floor at or above its own generation mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** From eee3e62956af1ff19391f2b932e5329f29a0139b Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:17:38 -0400 Subject: [PATCH 29/65] docs: close the twenty-fifth-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 12 ++++++------ .../2026-08-11-humwatch-2.1-fleet-hub-design.md | 2 +- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 4a52398..d0ac561 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -359,7 +359,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, floor_announcement_json, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -391,7 +391,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. Test both windows for all three writers (creation, transfer activation, promotion). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. -`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. It also owns the transaction helper used by every fleet repository: @@ -1178,7 +1178,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe response whose floor signature fails registry-key verification contributing its announcement but not its floor (an attacker-supplied huge floor never reaches the generation formula), a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge floor never reaches the generation formula), preview failing loudly when a verified generation sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. - [ ] **Step 2: Implement bundle creation and replication** @@ -1188,13 +1188,13 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response also reports the machine's `generation_floor` beside the announcement, and the floor arrives authenticated: the responding machine signs the tuple of its machine ID, its floor, and the prober's request nonce with its machine signing key, and the prober verifies that signature against the machine's public key in the imported registry before using the floor. An unauthenticated floor would let any LAN host that can answer a registered address pair a replayed fleet-signed announcement with an arbitrary number, and a floor near the integer ceiling would poison the monotonic generation counter for every later transfer and recovery, so an unverifiable floor is discarded while the fleet-signed announcement in the same response remains usable on its own signature. Preview probes every address in the imported registry, keeps the highest value among verified announcement generations and signature-verified floors, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation and `generation_floor` (the envelope's body digest authenticates both, and a tampering test proves a modified floor fails verification), returning the hub's generation and stored watermark for that machine. The floor in that signed body is what drives the convergence rule: a hub that reads a verified floor at or above its own generation mints its fresh self-announcement at floor plus one, so the mechanism is a defined, authenticated field rather than an unspecified header. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and only then, when that proven floor is at or above its own generation, mints its fresh self-announcement at floor plus one, rejecting any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads a verified floor at or above its own generation mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum with its evidence announcement retained in `floor_announcement_json` while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads a verified floor at or above its own generation mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 817116e..704ac7c 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -360,7 +360,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. -6. Promote the PC to hub at a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its machine-signed floor, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. +6. Promote the PC to hub at a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its fleet-signed floor evidence announcement, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. 7. Revoke browser sessions and integration credentials. 8. Accept reconnecting agents and rebuild central history from their retained buffers. From 5be43930ccd6f2f38c2ef3e095291be6aebe4ad8 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:28:13 -0400 Subject: [PATCH 30/65] docs: close the twenty-sixth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index d0ac561..b572eff 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -645,7 +645,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its keys, addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json`, the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. - [ ] **Step 6: Verify enrollment and commit** @@ -1126,7 +1126,7 @@ Preflight also rejects the transfer outright while an unacknowledged `pending_re The old hub durably enters a transfer freeze before the snapshot is taken, and that freeze transaction is the write to `runtime_state.transfer_state` (`frozen`) and `runtime_state.transfer_id` (naming the new `hub_transfers` row) that the shared-names schema section describes: everything from here through draining reads that one column. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The same transaction rejects the transfer while any enrollment attempt is `claimed`: a claimed attempt's retry nonce is encrypted to the old hub's machine identity, which never transfers, so the new hub could not finish it and the enrollment would strand. Claims resolve or expire within the ten-minute pairing window, and expired or failed attempts do not block. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, integration changes, and recovery bundle creation or replication (Task 11), all reading the same `transfer_state` column. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Rejecting recovery bundle work means no bundle naming the pre-transfer generation and address set can be created or handed to an agent mid-transfer, which would otherwise let a promotion race the transfer to a stale outcome. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path inserts the row into `hub_transfer_aborts` and lifts the freeze (writing `transfer_state` back to `accepting`) in one local transaction, and a delivery loop retries the abort from that durable row, on the same 1, 2, 4, 8, 16, then 30 minute backoff Task 11's delivery loops use, until the target acknowledges it or reports the digest already resolved, either of which stamps `acknowledged_at` and stops the retry. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal (`runtime_state.transfer_id` into `hub_transfers`) to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. -If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, `enter_draining_transfer()` writes `runtime_state.transfer_state` to `draining` in the same transaction that records the announcement on the `hub_transfers` row, so the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining, and `reconcile_draining_transfers()` (the background task described above) is what retries the same activation request against the persisted `hub_transfers` row until it obtains the persisted receipt, rather than a caller blocking inline on the original `/execute` request. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. +If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, `enter_draining_transfer()` writes `runtime_state.transfer_state` to `draining` in the same transaction that records the announcement on the `hub_transfers` row, so the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest, and the authorization to drive it survives the authority flip: the target's internal transfer routes authenticate the staging hub's requests against the old hub's machine key recorded with the `pending_hub_authority` row, scoped to that announcement digest, and that digest-scoped authorization remains answerable after activation until the old hub acknowledges the receipt. A retried `activate_publication_obligation` for an already-activated digest therefore returns the stored persistence receipt even though the target is already `HUB` and fleet-wide trust has rotated to the new key, so a lost first activation response can never leave the old hub draining forever against a target that no longer recognizes it. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining, and `reconcile_draining_transfers()` (the background task described above) is what retries the same activation request against the persisted `hub_transfers` row until it obtains the persisted receipt, rather than a caller blocking inline on the original `/execute` request. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. - [ ] **Step 5: Add Move Hub UI and evidence** From 56f5930a5b0efabdf0d4e5e58134cbfcdfd1c18f Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:32:18 -0400 Subject: [PATCH 31/65] docs: close the twenty-seventh-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index b572eff..a7533bd 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1120,7 +1120,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target secures that socket before `commit_unannounced_hub`: when the selection names an address the target's `ListenerManager` already serves (the normal case, since Task 4 keeps the enrolled agent's listener bound for announcement and recovery routes), staging reuses that existing bound socket and atomically widens its route set at commit instead of binding the same address a second time, which would fail address-in-use on every ordinary transfer. Only a selection naming a genuinely new address is freshly bound and held, then attached through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure on such a new address fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. -The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, installs the transfer announcement as the target's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` advance together, so the target's probe endpoint and any later enrollment serve the new authority rather than the old hub's), and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. Preflight also rejects the transfer outright while an unacknowledged `pending_recovery_key` row exists, with an explicit "acknowledge the recovery key before moving the hub" error and no state change: transferring the fleet signing key to the target while the owner has never confirmed the printable recovery key is saved would silently destroy the only other recovery path this fleet has, so the plan picks the branch of refusing the transfer rather than the branch of carrying the pending key across it. @@ -1188,7 +1188,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and only then, when that proven floor is at or above its own generation, mints its fresh self-announcement at floor plus one, rejecting any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. From d76b828655e6c5634f2e2b1af2e7c3aa3b2c709a Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:38:33 -0400 Subject: [PATCH 32/65] docs: close the twenty-eighth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index a7533bd..73c049f 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -331,6 +331,7 @@ Use issue #14 as the PR close target. * Produces: `async FleetStateRepository.set_role(role: RuntimeRole) -> None` * Produces: `transaction(db: aiosqlite.Connection) -> AsyncContextManager[None]` * Produces: `detect_runtime_role(config: HumWatchConfig) -> RuntimeRole` +* Produces: `RoleServices.reconcile() -> None` (starts and stops role services on every persisted role change, no restart required) - [ ] **Step 1: Write schema and identity failure tests** @@ -437,7 +438,7 @@ class MachineIdentity: Move role detection ahead of the current unconditional `validate_security_config()` calls in `agent/main.py`. Existing databases with an auth token start as `legacy`. Fresh databases start as `unconfigured`. Unconfigured mode starts its setup surface on `127.0.0.1` and does not require a token or certificate. Task 4 adds a loopback-selected, restricted private enrollment listener before its hub-orchestrated claim runs. That listener may expose only `/api/internal/v1/enrollment/claim` and `/finalize`, never the setup page, pairing-code endpoint, dashboard, or any v1 route. Legacy mode keeps the existing token, TLS, and CORS validation exactly. Hub and agent modes use loopback or explicitly stored private listeners until Task 9 adds the owner-managed settings surface, and Task 3's Create Fleet stores and binds the first private hub listener inside its transaction, so a new hub is reachable for enrollment claims and agent reports from the moment the fleet exists rather than waiting for Task 9. No role may fall back to `0.0.0.0`. -`hub` starts collector, retention, and hub services. `agent` starts collector, retention, and agent transport. Until those services exist, role branches log the selected mode and start only the services already implemented for that role. Add startup tests proving a fresh config with no token and no TLS reaches the unconfigured app, while an unsafe legacy config still fails closed. +`hub` starts collector, retention, and hub services. `agent` starts collector, retention, and agent transport. Role services are not a startup-only branch: the lifespan owns a `RoleServices` supervisor whose `reconcile()` starts and stops each role's services against the currently persisted role, and `FleetStateRepository.set_role()` triggers it after every in-process role change, because Create Fleet, enrollment finalize, transfer activation, abort back to agent, promotion, and Rejoin all change the role without a process restart. A fresh process that started `unconfigured` therefore begins collecting locally the moment Create Fleet commits, and begins collecting and reporting the moment finalize commits, with a no-restart test for each flow (Tasks 3 and 4). Until those services exist, role branches log the selected mode and start only the services already implemented for that role. Add startup tests proving a fresh config with no token and no TLS reaches the unconfigured app, while an unsafe legacy config still fails closed. Set both `HumWatchConfig.retention_days` and the checked-in `config.json` value to `7` here so later outbox and hub retention tests share the approved default. @@ -643,9 +644,11 @@ The local setup surface first reads private LAN and Tailscale candidates, then w - [ ] **Step 5: Implement hub-orchestrated enrollment** -The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its keys, addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. +The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity: the agent claim naming a registered machine ID must be signed by the previously registered machine signing key (the Rejoin flow holds it, since machine identity survives revocation), and the registered keys are then kept or replaced only under that proven signature. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. + +The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-detects its current private addresses, persists an adopted change to its `fleet_settings.listeners_json` row, and pushes it through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. - [ ] **Step 6: Verify enrollment and commit** From ed4e883465a9b8ecc015e2bce3759417e5f445a1 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:44:33 -0400 Subject: [PATCH 33/65] docs: close the twenty-ninth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 73c049f..52a4a32 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -648,7 +648,7 @@ The hub validates `http` or `https` private addresses and calls claim. Claim is Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. -The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-detects its current private addresses, persists an adopted change to its `fleet_settings.listeners_json` row, and pushes it through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. +The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. - [ ] **Step 6: Verify enrollment and commit** @@ -1123,7 +1123,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target secures that socket before `commit_unannounced_hub`: when the selection names an address the target's `ListenerManager` already serves (the normal case, since Task 4 keeps the enrolled agent's listener bound for announcement and recovery routes), staging reuses that existing bound socket and atomically widens its route set at commit instead of binding the same address a second time, which would fail address-in-use on every ordinary transfer. Only a selection naming a genuinely new address is freshly bound and held, then attached through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure on such a new address fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. -The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, installs the transfer announcement as the target's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` advance together, so the target's probe endpoint and any later enrollment serve the new authority rather than the old hub's), and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report from an enrolled test machine under the committed proposed generation. The target returns a signed readiness receipt containing the generation and each check digest. +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, installs the transfer announcement as the target's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` advance together, so the target's probe endpoint and any later enrollment serve the new authority rather than the old hub's), and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report under the committed proposed generation, but that report check is non-committing by design: the old hub constructs a dedicated probe payload rather than handing over a real outbox batch, the target runs the full ingestion path (signature, admission, inserts, contiguity) inside a transaction it rolls back deliberately, and the result leaves the target as a check digest, never as an `IngestAck` any agent outbox could consume. Nothing about the probe persists, so a draining commit failure that aborts the still-staged target discards no acknowledged sample and no machine sequence was ever consumed, with a test asserting the staged database is byte-identical before and after the readiness probe. The target returns a signed readiness receipt containing the generation and each check digest. Preflight also rejects the transfer outright while an unacknowledged `pending_recovery_key` row exists, with an explicit "acknowledge the recovery key before moving the hub" error and no state change: transferring the fleet signing key to the target while the owner has never confirmed the printable recovery key is saved would silently destroy the only other recovery path this fleet has, so the plan picks the branch of refusing the transfer rather than the branch of carrying the pending key across it. @@ -1193,11 +1193,11 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and only then, when that proven floor is at or above its own generation, mints its fresh self-announcement at floor plus one, rejecting any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and mints its fresh self-announcement at floor plus one only when that evidence is a strictly higher generation than its own, or an equal generation whose announcement digest names a different hub lineage. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum with its evidence announcement retained in `floor_announcement_json` while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads a verified floor at or above its own generation mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum with its evidence announcement retained in `floor_announcement_json` while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a generation above its own, or an equal one from a different lineage, never its own current announcement echoed back) mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. - [ ] **Step 4: Add the Recover Fleet page and evidence** From a5bbeb988efb59cd3fa7acf577d30438bea29897 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:49:52 -0400 Subject: [PATCH 34/65] docs: close the thirtieth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 52a4a32..0be6cda 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -377,7 +377,7 @@ integration_clients(integration_id, name, public_key, scopes_json, created_at, r integration_pairing_codes(pairing_code_id, code_hash, created_at, expires_at, consumed_at, integration_id, client_public_key) hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) hub_transfer_aborts(transfer_id, announcement_generation, announcement_digest, abort_json, state, attempt_count, next_attempt_at, acknowledged_at) -pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staged_snapshot_path, staged_key_path, prior_settings_json, state, created_at, activated_at) +pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staging_hub_machine_id, staging_hub_public_key, staged_snapshot_path, staged_key_path, prior_settings_json, state, created_at, activated_at) hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) pending_recovery_key(id, printable_key, created_at) @@ -1017,6 +1017,8 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds a listener set differing from the addresses advertised in its own `runtime_state.hub_announcement_json` mints a higher-generation fleet-signed announcement of itself carrying the new set, installs it as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables) that the Task 10 delivery loop retries until every agent acknowledges, the same push path transfers use. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement. + A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. - [ ] **Step 3: Implement Settings sections** @@ -1129,7 +1131,7 @@ Preflight also rejects the transfer outright while an unacknowledged `pending_re The old hub durably enters a transfer freeze before the snapshot is taken, and that freeze transaction is the write to `runtime_state.transfer_state` (`frozen`) and `runtime_state.transfer_id` (naming the new `hub_transfers` row) that the shared-names schema section describes: everything from here through draining reads that one column. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The same transaction rejects the transfer while any enrollment attempt is `claimed`: a claimed attempt's retry nonce is encrypted to the old hub's machine identity, which never transfers, so the new hub could not finish it and the enrollment would strand. Claims resolve or expire within the ten-minute pairing window, and expired or failed attempts do not block. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, integration changes, and recovery bundle creation or replication (Task 11), all reading the same `transfer_state` column. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Rejecting recovery bundle work means no bundle naming the pre-transfer generation and address set can be created or handed to an agent mid-transfer, which would otherwise let a promotion race the transfer to a stale outcome. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path inserts the row into `hub_transfer_aborts` and lifts the freeze (writing `transfer_state` back to `accepting`) in one local transaction, and a delivery loop retries the abort from that durable row, on the same 1, 2, 4, 8, 16, then 30 minute backoff Task 11's delivery loops use, until the target acknowledges it or reports the digest already resolved, either of which stamps `acknowledged_at` and stops the retry. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal (`runtime_state.transfer_id` into `hub_transfers`) to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. -If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, `enter_draining_transfer()` writes `runtime_state.transfer_state` to `draining` in the same transaction that records the announcement on the `hub_transfers` row, so the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest, and the authorization to drive it survives the authority flip: the target's internal transfer routes authenticate the staging hub's requests against the old hub's machine key recorded with the `pending_hub_authority` row, scoped to that announcement digest, and that digest-scoped authorization remains answerable after activation until the old hub acknowledges the receipt. A retried `activate_publication_obligation` for an already-activated digest therefore returns the stored persistence receipt even though the target is already `HUB` and fleet-wide trust has rotated to the new key, so a lost first activation response can never leave the old hub draining forever against a target that no longer recognizes it. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining, and `reconcile_draining_transfers()` (the background task described above) is what retries the same activation request against the persisted `hub_transfers` row until it obtains the persisted receipt, rather than a caller blocking inline on the original `/execute` request. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. +If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, `enter_draining_transfer()` writes `runtime_state.transfer_state` to `draining` in the same transaction that records the announcement on the `hub_transfers` row, so the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest, and the authorization to drive it survives the authority flip: the target's internal transfer routes authenticate the staging hub's requests against `pending_hub_authority.staging_hub_machine_id` and `staging_hub_public_key`, persisted in the row itself so a target restarted after activation can still reconstruct exactly this authorization, scoped to that announcement digest, and that digest-scoped authorization remains answerable after activation until the old hub acknowledges the receipt. A retried `activate_publication_obligation` for an already-activated digest therefore returns the stored persistence receipt even though the target is already `HUB` and fleet-wide trust has rotated to the new key, so a lost first activation response can never leave the old hub draining forever against a target that no longer recognizes it. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining, and `reconcile_draining_transfers()` (the background task described above) is what retries the same activation request against the persisted `hub_transfers` row until it obtains the persisted receipt, rather than a caller blocking inline on the original `/execute` request. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. - [ ] **Step 5: Add Move Hub UI and evidence** @@ -1330,7 +1332,7 @@ Use SQLite backup, validate schema and machine rows, copy config and security pa Create Fleet imports history under the stable self machine UUID. Join Fleet converts history to ordered outbox batches and uploads through Task 5. Neither path deletes original runtime files. -Both conversions take a listener selection and follow the same bind-before-commit rule as fresh setup: the legacy listener may be loopback-only, a forbidden wildcard, or archived with the legacy security settings, so Create Fleet reserves and persists a private hub listener (Task 3's rules) and Join Fleet selects the restricted private enrollment listener Task 4 requires, in each case binding before the role transition and failing visibly without changing role when binding fails. The wizard offers the machine's detected private addresses for the choice. +Both conversions take a listener selection and follow the same bind-before-commit rule as fresh setup: the legacy listener may be loopback-only, a forbidden wildcard, or archived with the legacy security settings, so Create Fleet reserves and persists a private hub listener (Task 3's rules) and Join Fleet selects the restricted private enrollment listener Task 4 requires, in each case securing the socket before the role transition and failing visibly without changing role when that fails. Securing does not always mean a fresh bind, because the migration wizard itself is being served by the live legacy socket on port 9100 and a typical 2.0 `agent/main.py` bound it as a wildcard: when the selected address and port are already covered by the process's own active socket, wildcard included, the conversion adopts that socket through the `ListenerManager` handoff instead of attempting a second bind that would fail address-in-use, and the role commit narrows the adopted wildcard to exactly the selected private addresses and the new role's route set. Only a selection the process does not already own is freshly bound. Test both paths: conversion from a wizard served on the legacy wildcard socket succeeds without an address-in-use error, and the post-commit listener serves only the selected addresses and routes. The wizard offers the machine's detected private addresses for the choice. The migration router is not open: an upgraded legacy installation is reachable on its existing LAN listener, and these routes replace runtime authority and credentials before any owner account exists. Inspection and every conversion mutation require the installation's existing legacy bearer token, exactly as the protected v1 routers do, or an actual loopback peer. Add unauthorized-route tests proving a tokenless LAN request cannot inspect or convert. From be9e82e256bf825e67c2c9323ce522f2d90e186d Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 00:56:15 -0400 Subject: [PATCH 35/65] docs: close the thirty-first-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 0be6cda..652bb7e 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -368,7 +368,7 @@ fleet_settings(id, listeners_json, https_cert_path, https_key_path, allow_public fleet_metrics(machine_id, sequence, timestamp, category, metric_name, value, unit) fleet_processes(machine_id, sequence, timestamp, pid, name, cpu_percent, memory_mb) report_outbox(sequence, batch_id, captured_at, payload_json, acknowledged_at) -report_batches(machine_id, sequence, captured_at, accepted_at) +report_batches(machine_id, sequence, batch_id, batch_digest, captured_at, accepted_at) report_receipts(machine_id, upload_id, payload_digest, sequence_start, sequence_end, highest_contiguous_sequence, accepted_at) data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) used_nonces(principal_type, principal_id, nonce_hash, expires_at) @@ -390,7 +390,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. -`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. Test both windows for all three writers (creation, transfer activation, promotion). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. +`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. Test both windows for all three writers (creation, transfer activation, promotion). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. `runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. @@ -648,7 +648,7 @@ The hub validates `http` or `https` private addresses and calls claim. Claim is Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. -The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. Task 9 later lets the owner edit this persisted listener set through the same row. +The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. - [ ] **Step 6: Verify enrollment and commit** @@ -1260,7 +1260,7 @@ Assert ten-minute one-use pairing, independent credential revocation, no owner o - [ ] **Step 2: Implement integration request authentication** -Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema), and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. +Authentication at connection time is not enough for the streaming route: `owner_or_scope(...)` runs once while the response is constructed, so a revoked integration or a revoked owner session holding an open `/api/v2/events` stream would keep receiving telemetry indefinitely. The events service therefore revalidates the stream's principal before every event and heartbeat write, and revocation acts eagerly as well: deleting an integration or revoking a session publishes a cancellation that closes any open stream bound to that principal. Test both directions for both principal kinds: an open integration stream dies at revocation, and an open owner stream dies at session revocation, each before the next heartbeat lands. Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema), and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. - [ ] **Step 3: Publish a machine-readable contract fixture** @@ -1332,7 +1332,7 @@ Use SQLite backup, validate schema and machine rows, copy config and security pa Create Fleet imports history under the stable self machine UUID. Join Fleet converts history to ordered outbox batches and uploads through Task 5. Neither path deletes original runtime files. -Both conversions take a listener selection and follow the same bind-before-commit rule as fresh setup: the legacy listener may be loopback-only, a forbidden wildcard, or archived with the legacy security settings, so Create Fleet reserves and persists a private hub listener (Task 3's rules) and Join Fleet selects the restricted private enrollment listener Task 4 requires, in each case securing the socket before the role transition and failing visibly without changing role when that fails. Securing does not always mean a fresh bind, because the migration wizard itself is being served by the live legacy socket on port 9100 and a typical 2.0 `agent/main.py` bound it as a wildcard: when the selected address and port are already covered by the process's own active socket, wildcard included, the conversion adopts that socket through the `ListenerManager` handoff instead of attempting a second bind that would fail address-in-use, and the role commit narrows the adopted wildcard to exactly the selected private addresses and the new role's route set. Only a selection the process does not already own is freshly bound. Test both paths: conversion from a wizard served on the legacy wildcard socket succeeds without an address-in-use error, and the post-commit listener serves only the selected addresses and routes. The wizard offers the machine's detected private addresses for the choice. +Both conversions take a listener selection and follow the same bind-before-commit rule as fresh setup: the legacy listener may be loopback-only, a forbidden wildcard, or archived with the legacy security settings, so Create Fleet reserves and persists a private hub listener (Task 3's rules) and Join Fleet selects the restricted private enrollment listener Task 4 requires, in each case securing the socket before the role transition and failing visibly without changing role when that fails. Securing does not always mean a fresh bind, because the migration wizard itself is being served by the live legacy socket on port 9100 and a typical 2.0 `agent/main.py` bound it as a wildcard: when the selected address and port are already covered by the process's own active specific-address socket, the conversion adopts that socket through the `ListenerManager` handoff instead of attempting a second bind that would fail address-in-use, and the role commit narrows its route set. A wildcard socket cannot be adopted, because a bound socket's kernel local address is immutable and `0.0.0.0` would leave the converted role reachable on every interface against the exact-listener rule, so the wildcard case is a coordinated replacement inside the conversion: stop accepting on the legacy wildcard listener (the wizard's own already-accepted connection survives socket closure and completes normally), bind the selected private addresses on the freed port, and only then commit the role. A bind failure in that window rebinds the wildcard and fails the conversion visibly with the role unchanged, so the wizard stays reachable for a retry. Only a selection the process does not cover at all is freshly bound with no replacement dance. Test all three paths: conversion from a wizard served on the legacy wildcard replaces it with no address-in-use error and the post-commit listener answers only on the selected addresses (a request to a non-selected interface fails), a bind failure mid-replacement restores the wildcard wizard with the role unchanged, and a specific-address adoption narrows routes without rebinding. The wizard offers the machine's detected private addresses for the choice. The migration router is not open: an upgraded legacy installation is reachable on its existing LAN listener, and these routes replace runtime authority and credentials before any owner account exists. Inspection and every conversion mutation require the installation's existing legacy bearer token, exactly as the protected v1 routers do, or an actual loopback peer. Add unauthorized-route tests proving a tokenless LAN request cannot inspect or convert. From 2e99dbb298102178aed1cc7bfbe3d68eb6d54a5f Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 01:02:42 -0400 Subject: [PATCH 36/65] docs: close the thirty-second-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 652bb7e..7a267e2 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -646,7 +646,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity: the agent claim naming a registered machine ID must be signed by the previously registered machine signing key (the Rejoin flow holds it, since machine identity survives revocation), and the registered keys are then kept or replaced only under that proven signature. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 10), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. @@ -965,7 +965,9 @@ Commit message: `feat: unify fleet access in one dashboard` **Purpose:** Make enrollment, removal, sessions, network access, and TLS manageable from one settings surface. **Files:** -* Create: `agent/fleet/listeners.py` +* Modify: `agent/fleet/listeners.py` +* Create: `agent/fleet/announcements.py` +* Create: `agent/routes/internal_announcements.py` * Create: `agent/routes/v2_settings.py` * Create: `static/js/pages/fleet-settings.js` * Create: `tests/test_listener_policy.py` @@ -1017,7 +1019,7 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. -A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds a listener set differing from the addresses advertised in its own `runtime_state.hub_announcement_json` mints a higher-generation fleet-signed announcement of itself carrying the new set, installs it as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables) that the Task 10 delivery loop retries until every agent acknowledges, the same push path transfers use. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds a listener set differing from the addresses advertised in its own `runtime_state.hub_announcement_json` mints a higher-generation fleet-signed announcement of itself carrying the new set, installs it as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule, persists the announcement columns atomically, and redirects transport. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement. A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. @@ -1042,7 +1044,7 @@ Run focused tests, static asset verification, browser smoke, full pytest, then c **Purpose:** Transfer hub ownership without creating two authoritative hubs. **Files:** -* Create: `agent/fleet/announcements.py` +* Modify: `agent/fleet/announcements.py` * Create: `agent/fleet/transfer.py` * Create: `agent/routes/internal_transfer.py` * Create: `agent/routes/v2_transfer.py` From 3749f76b26aa20af33316792656ae0474a3c38a5 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 01:10:12 -0400 Subject: [PATCH 37/65] docs: close the thirty-third-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 7a267e2..342e442 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -155,6 +155,7 @@ class AgentClaim(BaseModel): encryption_public_key: str addresses: list[str] agent_version: str + prior_identity_proof: str | None = None class SignedEnrollmentRecord(BaseModel): enrollment_id: UUID @@ -392,7 +393,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. Test both windows for all three writers (creation, transfer activation, promotion). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. -`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes plus the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes, the loopback recovery page, and the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. The loopback recovery page stays reachable because the old hub can die between `commit_unannounced_hub` and its activation request, leaving no machine able to authorize activation or abort: its Cancel Pending Authority action runs the abort cleanup locally for an authority whose state was never `activated` (staged files deleted, prior settings restored, role back to `AGENT`), refuses to touch an activated one, and frees the machine to run ordinary dead-hub recovery from its replicated bundle, which a two-machine fleet's sole survivor depends on. Task 10's tests cover the stranded-target cancel followed by a successful promotion. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. It also owns the transaction helper used by every fleet repository: @@ -644,7 +645,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w - [ ] **Step 5: Implement hub-orchestrated enrollment** -The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity: the agent claim naming a registered machine ID must be signed by the previously registered machine signing key (the Rejoin flow holds it, since machine identity survives revocation), and the registered keys are then kept or replaced only under that proven signature. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. +The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity, carried in the wire model itself: `AgentClaim.prior_identity_proof` is an Ed25519 signature by the previously registered machine signing key over the claim's canonical compact JSON with the proof field omitted (the same sorted-key canonicalization every other signed object here uses), present exactly when the claim names an already-registered machine ID. The Rejoin flow holds that key, since machine identity survives revocation, and the hub verifies the proof against the registered public key before keeping or replacing anything. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. From 91c25a754eaf457525a59ca81986f028466267f0 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 01:15:41 -0400 Subject: [PATCH 38/65] docs: close the thirty-fourth-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 342e442..e0c0872 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1186,7 +1186,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge floor never reaches the generation formula), preview failing loudly when a verified generation sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge floor never reaches the generation formula), preview failing loudly when a verified generation sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, that same machine completing the full path (rejection, Rejoin under its proven identity, finalize-triggered watermark rewind with gap disclosure, retained history acknowledged on the recovered hub), zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. - [ ] **Step 2: Implement bundle creation and replication** @@ -1198,7 +1198,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and mints its fresh self-announcement at floor plus one only when that evidence is a strictly higher generation than its own, or an equal generation whose announcement digest names a different hub lineage. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes, and when the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and mints its fresh self-announcement at floor plus one only when that evidence is a strictly higher generation than its own, or an equal generation whose announcement digest names a different hub lineage. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. From 06ad23e03a274cbc3840be88af95fcf1f43c8f45 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 01:20:12 -0400 Subject: [PATCH 39/65] docs: close the thirty-fifth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index e0c0872..bc3bd12 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -277,7 +277,7 @@ def legacy_database_path(config: HumWatchConfig) -> Optional[Path]: Open the source read-only, use `sqlite3.Connection.backup()` into a staging file so committed WAL pages are included, then run `PRAGMA quick_check`, verify the required 2.0 tables, verify at least one `machine_info` row, and compare source and staged row counts. Only then rename the staging database atomically to the target. -Write `resolved_data_root / "migration-archive" / / "manifest.json"` with source and target paths, SHA256 digests, verified table row counts, status, and time. Return and log `migrated`, `source_missing`, `target_exists`, `same_path`, or `failed`. Never unlink or rewrite the source database, its WAL or shared-memory sidecars, or an existing archive. Startup must raise `MigrationSafetyError` when an explicit source exists but backup or proof fails. +Write `resolved_data_root / "migration-archive" / / "manifest.json"` with source and target paths, SHA256 digests, verified table row counts, status, and time. Return and log `migrated`, `source_missing`, `target_exists`, `same_path`, or `failed`. `target_exists` is not automatically safe to continue on, because the absolute-path bug this task exists for can have already created a newer empty or partial database at the target while the explicit legacy source still holds the missing history: a successful migration records itself in the target archive manifest, and `target_exists` proceeds only when a manifest proves this target came from a prior successful migration of this source. A pre-existing target with no such proof while an explicit source exists raises `MigrationSafetyError` with both paths, their row counts, and the manual merge guidance in the message, stopping visibly instead of quietly adopting the empty database and leaving issue #14 alive. Test the bug-shaped case: an empty target, a populated explicit source, and no manifest must refuse to start. Never unlink or rewrite the source database, its WAL or shared-memory sidecars, or an existing archive. Startup must raise `MigrationSafetyError` when an explicit source exists but backup or proof fails. - [ ] **Step 4: Make the installer pass the old path explicitly** @@ -381,7 +381,7 @@ hub_transfer_aborts(transfer_id, announcement_generation, announcement_digest, a pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staging_hub_machine_id, staging_hub_public_key, staged_snapshot_path, staged_key_path, prior_settings_json, state, created_at, activated_at) hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) -pending_recovery_key(id, printable_key, created_at) +pending_recovery_key(id, key_path, created_at) staged_key_journal(journal_id, purpose, staged_path, created_at, finalized_at) recovery_bundles(generation, bundle_revision, created_at, encrypted_payload, fleet_signature) recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) @@ -537,7 +537,7 @@ Validate that role is `unconfigured`, request source is loopback, username is no Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key, an in-process attach test proving a request on the newly advertised listener succeeds with no process restart, and a kill between commit and attach proving the restarted process serves the stored listener. -Do not return the printable recovery key as the transaction's only copy. Persist it in the `pending_recovery_key` row (Task 2 schema, service-owned database, same protection class as the owner password hash) inside the same transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) returns the printable key for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes it, which is the only deletion path. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. +Do not return the printable recovery key as the transaction's only copy, and do not put the plaintext in the database at all: a SQLite `DELETE` removes a row logically while its bytes survive in pages, the freelist, and WAL files, and from there ride into backups and transfer snapshots, which would break the promise that the key is unrecoverable after acknowledgement. The pending key lives in a service-owned mode `0600` file under `resolved_data_root / "identity"`, written with the same atomic discipline as the identity keys, and the `pending_recovery_key` row (Task 2 schema) persists only its path and metadata inside the create-fleet transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) reads and returns the printable key from that file for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` overwrites the file's bytes best-effort, unlinks it, and deletes the row, which is the only deletion path. The identity directory is already excluded from database backups and transfer snapshots, and a test scans the raw database, WAL, and staged transfer payload bytes to prove the printable key never appears in any of them and the file is gone after acknowledgement. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. - [ ] **Step 6: Mount `/api/v2` without weakening v1** From 80ff666f436fea673c739902b08d7e921e6d83a5 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 01:24:52 -0400 Subject: [PATCH 40/65] docs: close the thirty-sixth-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- .../superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index bc3bd12..58e6935 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -373,7 +373,8 @@ report_batches(machine_id, sequence, batch_id, batch_digest, captured_at, accept report_receipts(machine_id, upload_id, payload_digest, sequence_start, sequence_end, highest_contiguous_sequence, accepted_at) data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) used_nonces(principal_type, principal_id, nonce_hash, expires_at) -enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) +enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, agent_claim_json, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) +agent_enrollment_state(enrollment_id, agent_claim_json, record_digest, finalize_result_json, state, created_at) integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) integration_pairing_codes(pairing_code_id, code_hash, created_at, expires_at, consumed_at, integration_id, client_public_key) hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) @@ -387,9 +388,9 @@ recovery_bundles(generation, bundle_revision, created_at, encrypted_payload, fle recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` -Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. -`used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. +`enrollment_attempts.agent_claim_json` holds the complete claim Task 4 promises to reconstruct retries from, not merely its digest, and `agent_enrollment_state` is the agent-side mirror: the exact claim bytes the agent sent, the record digest, and the stored finalize result that the digest-scoped `/finalize` retry serves, so either process restarting between claim and acknowledged finalize resumes from durable rows even when mutable inputs like hostname or detected addresses have since changed. `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. Test both windows for all three writers (creation, transfer activation, promotion). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. From 0a3eb652977fc7cbcfce7a5c21573103597ba217 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 01:42:43 -0400 Subject: [PATCH 41/65] docs: close the thirty-seventh-round fleet hub findings Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 58e6935..6d6e7c4 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -392,7 +392,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `enrollment_attempts.agent_claim_json` holds the complete claim Task 4 promises to reconstruct retries from, not merely its digest, and `agent_enrollment_state` is the agent-side mirror: the exact claim bytes the agent sent, the record digest, and the stored finalize result that the digest-scoped `/finalize` retry serves, so either process restarting between claim and acknowledged finalize resumes from durable rows even when mutable inputs like hostname or detected addresses have since changed. `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. -`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. Test both windows for all three writers (creation, transfer activation, promotion). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. +`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. The journal also cannot cover the window before it exists: a transfer import or recovery promotion decrypts the fleet key to its staging path before the transaction that records any reference commits, and a crash there would otherwise leave an untracked, directly usable plaintext key on a machine that restarts as a plain agent. Every staged key therefore lives only under the dedicated `resolved_data_root / "staging"` directory, and startup scrubs that directory: any file not referenced by an uncommitted-repair candidate (an unfinalized `staged_key_journal` row or a live `pending_hub_authority` row) is deleted before any role service starts, so the pre-commit crash window ends with the plaintext destroyed rather than orphaned. Test both journal windows for all three writers (creation, transfer activation, promotion), plus the pre-commit window for both stagers (kill after the key file is written but before the commit, restart, and prove the scrub removed it). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. `runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes, the loopback recovery page, and the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. The loopback recovery page stays reachable because the old hub can die between `commit_unannounced_hub` and its activation request, leaving no machine able to authorize activation or abort: its Cancel Pending Authority action runs the abort cleanup locally for an authority whose state was never `activated` (staged files deleted, prior settings restored, role back to `AGENT`), refuses to touch an activated one, and frees the machine to run ordinary dead-hub recovery from its replicated bundle, which a two-machine fleet's sole survivor depends on. Task 10's tests cover the stranded-target cancel followed by a successful promotion. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. @@ -803,7 +803,7 @@ Do not expose table rows directly. Return typed response models. Current values - [ ] **Step 4: Implement a hub-owned event stream** -Emit `machine_metrics`, `machine_status`, `backfill_progress`, `machine_revoked`, and heartbeat events. Every event includes `machine_id`. Use owner session cookie authentication. +Emit `machine_metrics`, `machine_status`, `backfill_progress`, `machine_revoked`, and heartbeat events. Every event includes `machine_id`. Use owner session cookie authentication, and not only at connection time: the stream revalidates its owner session before every event and heartbeat write, and session revocation publishes a cancellation that closes any open stream bound to that session, because a dependency that runs once while the response is constructed would let a revoked session keep receiving telemetry until it chose to disconnect. Test that an open stream dies at session revocation before the next heartbeat, in this task's PR, since every task merges as a separately safe HEAD. Task 12 later extends the same revalidation to integration principals. - [ ] **Step 5: Verify endpoint surface with a running test server** @@ -1264,7 +1264,7 @@ Assert ten-minute one-use pairing, independent credential revocation, no owner o - [ ] **Step 2: Implement integration request authentication** -Authentication at connection time is not enough for the streaming route: `owner_or_scope(...)` runs once while the response is constructed, so a revoked integration or a revoked owner session holding an open `/api/v2/events` stream would keep receiving telemetry indefinitely. The events service therefore revalidates the stream's principal before every event and heartbeat write, and revocation acts eagerly as well: deleting an integration or revoking a session publishes a cancellation that closes any open stream bound to that principal. Test both directions for both principal kinds: an open integration stream dies at revocation, and an open owner stream dies at session revocation, each before the next heartbeat lands. Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema), and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. +Task 6 already revalidates the owner session on every event and heartbeat write and cancels open streams at session revocation. This task extends exactly that mechanism to the second principal kind: the events service revalidates an integration principal the same way, and deleting an integration publishes the same eager cancellation. Test that an open integration stream dies at revocation before the next heartbeat lands. Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema), and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. - [ ] **Step 3: Publish a machine-readable contract fixture** From a8ea3627f771aaead3a0c4cf402f22ae69df5564 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 01:48:28 -0400 Subject: [PATCH 42/65] docs: close the thirty-eighth-round fleet hub finding Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_017A8pnsob3w5giPzcRNNbGk --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 6d6e7c4..71fbd07 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -109,7 +109,8 @@ class ProcessValue(BaseModel): class GapRange(BaseModel): sequence_start: int sequence_end: int - reason: Literal["local-retention", "source-corruption"] + reason: Literal["local-retention", "source-corruption", "recovery"] + reported_at: datetime class ReportBatch(BaseModel): sequence: int @@ -733,7 +734,7 @@ Refactor the end of each collection tick to build one `ReportBatch`. In `legacy` Build canonical compact JSON with sorted keys for `{batches, gaps}` and set `payload_digest` to its SHA256. Sign canonical compact JSON containing `fleet_id`, `machine_id`, `upload_id`, `sequence_start`, `sequence_end`, and `payload_digest` with the machine key. The outer HTTP request is also signed through Task 4, binding the exact body, fleet, and machine headers. -The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. An upload must contain at least one batch or one gap range, and a gap-only upload (retention removed the unacknowledged history, leaving nothing else to send) commits and acknowledges normally: `latest_activity_at()` falls back to the gap's reported time, so the agent can advance past the missing range. Test a gap-only upload end to end. Insert metrics, processes, one `report_batches` presence row per accepted batch sequence, gap ranges, the machine watermark, and `report_receipts` in one transaction. Compute `highest_contiguous_sequence` by extending the persisted machine watermark through `report_batches` presence rows and disclosed gap ranges, never by scanning sample rows, so an empty batch and a retention-purged prefix both count as covered. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. +The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. An upload must contain at least one batch or one gap range, and a gap-only upload (retention removed the unacknowledged history, leaving nothing else to send) commits and acknowledges normally: `latest_activity_at()` falls back to `GapRange.reported_at`, the signed timestamp the agent stamps when it persists the disclosure (also what ingestion writes into `data_gaps.reported_at`), so the agent can advance past the missing range and the machine's activity value stays defined with no server-side guessing. Test a gap-only upload end to end. Insert metrics, processes, one `report_batches` presence row per accepted batch sequence, gap ranges, the machine watermark, and `report_receipts` in one transaction. Compute `highest_contiguous_sequence` by extending the persisted machine watermark through `report_batches` presence rows and disclosed gap ranges, never by scanning sample rows, so an empty batch and a retention-purged prefix both count as covered. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. - [ ] **Step 5: Implement transport and retention** From fa171899ee42afdfd6fa0daaa9b25dd8396a41c1 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 10:49:38 -0400 Subject: [PATCH 43/65] docs: close the thirty-ninth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 71fbd07..073e5b6 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -362,7 +362,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, floor_announcement_json, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, floor_announcement_json, hub_signing_public_key, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -649,7 +649,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity, carried in the wire model itself: `AgentClaim.prior_identity_proof` is an Ed25519 signature by the previously registered machine signing key over the claim's canonical compact JSON with the proof field omitted (the same sorted-key canonicalization every other signed object here uses), present exactly when the claim names an already-registered machine ID. The Rejoin flow holds that key, since machine identity survives revocation, and the hub verifies the proof against the registered public key before keeping or replacing anything. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. From d87081e32099ab5887e4173d45783c5222023036 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 10:54:06 -0400 Subject: [PATCH 44/65] docs: close the fortieth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 073e5b6..effe3f1 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -276,9 +276,9 @@ def legacy_database_path(config: HumWatchConfig) -> Optional[Path]: return PROJECT_ROOT / Path(config.db_path) ``` -Open the source read-only, use `sqlite3.Connection.backup()` into a staging file so committed WAL pages are included, then run `PRAGMA quick_check`, verify the required 2.0 tables, verify at least one `machine_info` row, and compare source and staged row counts. Only then rename the staging database atomically to the target. +Open the source read-only, use `sqlite3.Connection.backup()` into a staging file so committed WAL pages are included, then run `PRAGMA quick_check`, verify the required 2.0 tables, verify at least one `machine_info` row, and compare source and staged row counts. Only then rename the staging database atomically to the target, and the proof protocol brackets that rename so no crash window can strand a completed copy without evidence: before the rename, write and fsync the manifest with status `pending`, carrying the staged file's digest and verified row counts, and after the rename rewrite it to its final status. -Write `resolved_data_root / "migration-archive" / / "manifest.json"` with source and target paths, SHA256 digests, verified table row counts, status, and time. Return and log `migrated`, `source_missing`, `target_exists`, `same_path`, or `failed`. `target_exists` is not automatically safe to continue on, because the absolute-path bug this task exists for can have already created a newer empty or partial database at the target while the explicit legacy source still holds the missing history: a successful migration records itself in the target archive manifest, and `target_exists` proceeds only when a manifest proves this target came from a prior successful migration of this source. A pre-existing target with no such proof while an explicit source exists raises `MigrationSafetyError` with both paths, their row counts, and the manual merge guidance in the message, stopping visibly instead of quietly adopting the empty database and leaving issue #14 alive. Test the bug-shaped case: an empty target, a populated explicit source, and no manifest must refuse to start. Never unlink or rewrite the source database, its WAL or shared-memory sidecars, or an existing archive. Startup must raise `MigrationSafetyError` when an explicit source exists but backup or proof fails. +The manifest lives at `resolved_data_root / "migration-archive" / / "manifest.json"` with source and target paths, SHA256 digests, verified table row counts, status, and time. Return and log `migrated`, `source_missing`, `target_exists`, `same_path`, or `failed`. `target_exists` is not automatically safe to continue on, because the absolute-path bug this task exists for can have already created a newer empty or partial database at the target while the explicit legacy source still holds the missing history: a successful migration records itself in the target archive manifest, and `target_exists` proceeds only when a manifest proves this target came from a prior successful migration of this source. A `pending` manifest is proof too, once verified against the disk: a crash between the rename and the manifest finalize leaves exactly that pair behind, so `target_exists` with a `pending` manifest for this source recomputes the target's digest and row counts, and a match finalizes the manifest and continues while a mismatch is treated as no proof at all. A pre-existing target with no such proof while an explicit source exists raises `MigrationSafetyError` with both paths, their row counts, and the manual merge guidance in the message, stopping visibly instead of quietly adopting the empty database and leaving issue #14 alive. Test the bug-shaped case: an empty target, a populated explicit source, and no manifest must refuse to start. Test the interruption window too: kill the migration after the rename but before the manifest finalize, and the next startup must verify the target against the `pending` manifest, finalize it, and start normally instead of raising. Never unlink or rewrite the source database, its WAL or shared-memory sidecars, or an existing archive. Startup must raise `MigrationSafetyError` when an explicit source exists but backup or proof fails. - [ ] **Step 4: Make the installer pass the old path explicitly** From 5186832192e5e6585df8c9186e3767f5d6171cae Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:00:31 -0400 Subject: [PATCH 45/65] docs: close the forty-first-round fleet hub findings Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index effe3f1..0a053cb 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -375,7 +375,7 @@ report_receipts(machine_id, upload_id, payload_digest, sequence_start, sequence_ data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) used_nonces(principal_type, principal_id, nonce_hash, expires_at) enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, agent_claim_json, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) -agent_enrollment_state(enrollment_id, agent_claim_json, record_digest, finalize_result_json, state, created_at) +agent_enrollment_state(enrollment_id, agent_claim_json, hub_claim_digest, record_digest, finalize_result_json, state, created_at) integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) integration_pairing_codes(pairing_code_id, code_hash, created_at, expires_at, consumed_at, integration_id, client_public_key) hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) @@ -391,7 +391,7 @@ recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, stat Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. -`enrollment_attempts.agent_claim_json` holds the complete claim Task 4 promises to reconstruct retries from, not merely its digest, and `agent_enrollment_state` is the agent-side mirror: the exact claim bytes the agent sent, the record digest, and the stored finalize result that the digest-scoped `/finalize` retry serves, so either process restarting between claim and acknowledged finalize resumes from durable rows even when mutable inputs like hostname or detected addresses have since changed. `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. +`enrollment_attempts.agent_claim_json` holds the complete claim Task 4 promises to reconstruct retries from, not merely its digest, and `agent_enrollment_state` is the agent-side mirror: the exact claim bytes the agent sent, the canonical digest of the hub claim it accepted at claim time, the record digest, and the stored finalize result that the digest-scoped `/finalize` retry serves, so either process restarting between claim and acknowledged finalize resumes from durable rows even when mutable inputs like hostname or detected addresses have since changed. The stored `hub_claim_digest` is what keeps finalization bound to the hub claim the pairing code authenticated across that restart: the signed record finalize presents must embed a hub claim whose canonical digest matches the stored one, a record carrying any other hub claim under the same enrollment ID is rejected as changed bytes, and a restart test proves it (claim, restart the agent, finalize with a substituted hub claim, expect rejection, then finalize with the original and expect success). `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. The journal also cannot cover the window before it exists: a transfer import or recovery promotion decrypts the fleet key to its staging path before the transaction that records any reference commits, and a crash there would otherwise leave an untracked, directly usable plaintext key on a machine that restarts as a plain agent. Every staged key therefore lives only under the dedicated `resolved_data_root / "staging"` directory, and startup scrubs that directory: any file not referenced by an uncommitted-repair candidate (an unfinalized `staged_key_journal` row or a live `pending_hub_authority` row) is deleted before any role service starts, so the pre-commit crash window ends with the plaintext destroyed rather than orphaned. Test both journal windows for all three writers (creation, transfer activation, promotion), plus the pre-commit window for both stagers (kill after the key file is written but before the commit, restart, and prove the scrub removed it). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. @@ -1022,7 +1022,7 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. -A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds a listener set differing from the addresses advertised in its own `runtime_state.hub_announcement_json` mints a higher-generation fleet-signed announcement of itself carrying the new set, installs it as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule, persists the announcement columns atomically, and redirects transport. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule, persists the announcement columns atomically, and redirects transport. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. From 2ebfec9e55f629ed9520856508d7ac274943cddb Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:09:19 -0400 Subject: [PATCH 46/65] docs: close the forty-second-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 0a053cb..51164c5 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -377,7 +377,7 @@ used_nonces(principal_type, principal_id, nonce_hash, expires_at) enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, agent_claim_json, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) agent_enrollment_state(enrollment_id, agent_claim_json, hub_claim_digest, record_digest, finalize_result_json, state, created_at) integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) -integration_pairing_codes(pairing_code_id, code_hash, created_at, expires_at, consumed_at, integration_id, client_public_key) +integration_pairing_codes(pairing_code_id, code_hash, approved_name, approved_scopes_json, created_at, expires_at, consumed_at, integration_id, client_public_key) hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) hub_transfer_aborts(transfer_id, announcement_generation, announcement_digest, abort_json, state, attempt_count, next_attempt_at, acknowledged_at) pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staging_hub_machine_id, staging_hub_public_key, staged_snapshot_path, staged_key_path, prior_settings_json, state, created_at, activated_at) @@ -586,7 +586,7 @@ git commit -m "feat: add owner sessions and fleet creation" **Interfaces:** * Produces: `EnrollmentService.create_code() -> EnrollmentCode` * Produces: `EnrollmentService.claim(code: str, hub: HubClaim) -> AgentClaim` -* Produces: `EnrollmentService.finalize(enrollment_id: UUID, nonce: str, record: SignedEnrollmentRecord) -> RuntimeState` +* Produces: `EnrollmentService.finalize(enrollment_id: UUID, nonce: str, record: SignedEnrollmentRecord, announcement: HubAnnouncement) -> RuntimeState` * Produces: `EnrollmentRepository.store_claimed_attempt(record: SignedEnrollmentRecord, address: str, claim_nonce: str) -> None` * Produces: `sign_request(identity, fleet_id, machine_id, method, path, body, now, nonce) -> SignedHeaders` * Produces: `verify_internal_request(request, machine_record) -> None` @@ -649,7 +649,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity, carried in the wire model itself: `AgentClaim.prior_identity_proof` is an Ed25519 signature by the previously registered machine signing key over the claim's canonical compact JSON with the proof field omitted (the same sorted-key canonicalization every other signed object here uses), present exactly when the claim names an already-registered machine ID. The Rejoin flow holds that key, since machine identity survives revocation, and the hub verifies the proof against the registered public key before keeping or replacing anything. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement, which the agent verifies and persists verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. The agent verifies it and persists it verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. @@ -1265,7 +1265,7 @@ Assert ten-minute one-use pairing, independent credential revocation, no owner o - [ ] **Step 2: Implement integration request authentication** -Task 6 already revalidates the owner session on every event and heartbeat write and cancels open streams at session revocation. This task extends exactly that mechanism to the second principal kind: the events service revalidates an integration principal the same way, and deleting an integration publishes the same eager cancellation. Test that an open integration stream dies at revocation before the next heartbeat lands. Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema), and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. +Task 6 already revalidates the owner session on every event and heartbeat write and cancels open streams at session revocation. This task extends exactly that mechanism to the second principal kind: the events service revalidates an integration principal the same way, and deleting an integration publishes the same eager cancellation. Test that an open integration stream dies at revocation before the next heartbeat lands. Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema) carrying the owner-approved grant in `approved_name` and `approved_scopes_json`, decided by the owner at generation time, and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential, copying the name and `scopes_json` from those approved columns and nothing else. The exchange request never carries a name or scope set, because an unauthenticated caller offered that field would grant itself access, and the durable row is what lets a hub restarted between code generation and exchange still know exactly what the owner approved. Test that an exchange body attempting to smuggle scopes is rejected and the minted credential holds only the approved set. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. - [ ] **Step 3: Publish a machine-readable contract fixture** @@ -1273,7 +1273,7 @@ Add one versioned `tests/fixtures/slipshell/contract.json` containing fleet disc - [ ] **Step 4: Add integration administration UI and evidence** -Show paired clients, scopes, creation time, and revoke. Generate a short-lived pairing code without exposing stored credentials. Canary by attempting a write with an integration identity and require `403`. +Show paired clients, scopes, creation time, and revoke. Generate a short-lived pairing code without exposing stored credentials, with the owner naming the client and selecting its scope set at generation (persisted on the pairing row per Step 2). Canary by attempting a write with an integration identity and require `403`. - [ ] **Step 5: Verify and commit** From 1d1daadcdb5c4e3401a4ddaca84f12109cbfc3fa Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:18:21 -0400 Subject: [PATCH 47/65] docs: close the forty-third-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 15 +++++++++------ .../2026-08-11-humwatch-2.1-fleet-hub-design.md | 6 +++--- 2 files changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 51164c5..aedec5a 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -171,6 +171,7 @@ class EnrolledMachine(BaseModel): class HubAnnouncement(BaseModel): fleet_id: UUID hub_machine_id: UUID + recovery_epoch: int generation: int addresses: list[str] hub_signing_public_key: str @@ -178,6 +179,8 @@ class HubAnnouncement(BaseModel): signature: str ``` +`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. + Internal signed requests use these headers: ```text @@ -362,7 +365,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, hub_generation, generation_floor, hub_announcement_json, floor_announcement_json, hub_signing_public_key, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, recovery_epoch, hub_generation, generation_floor, hub_announcement_json, floor_announcement_json, hub_signing_public_key, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -535,11 +538,11 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the generation-1 fleet-signed `HubAnnouncement` built from that listener (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Every attached server task is configured with lifespan disabled: the process's original server already ran the app's startup and owns the shared database connection, the collectors, and the retention and delivery loops, and a second server invocation running the lifespan again would reopen or replace that connection and start duplicate copies of every background task the moment the fleet is created. Attached servers only serve requests, and the one lifecycle they participate in is the original server's shutdown, which detaches them. Test it by asserting exactly one collector tick source and one database connection before and after an attach. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the fleet-signed `HubAnnouncement` built from that listener at recovery epoch 1, generation 1 (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Every attached server task is configured with lifespan disabled: the process's original server already ran the app's startup and owns the shared database connection, the collectors, and the retention and delivery loops, and a second server invocation running the lifespan again would reopen or replace that connection and start duplicate copies of every background task the moment the fleet is created. Attached servers only serve requests, and the one lifecycle they participate in is the original server's shutdown, which detaches them. Test it by asserting exactly one collector tick source and one database connection before and after an attach. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key, an in-process attach test proving a request on the newly advertised listener succeeds with no process restart, and a kill between commit and attach proving the restarted process serves the stored listener. -Do not return the printable recovery key as the transaction's only copy, and do not put the plaintext in the database at all: a SQLite `DELETE` removes a row logically while its bytes survive in pages, the freelist, and WAL files, and from there ride into backups and transfer snapshots, which would break the promise that the key is unrecoverable after acknowledgement. The pending key lives in a service-owned mode `0600` file under `resolved_data_root / "identity"`, written with the same atomic discipline as the identity keys, and the `pending_recovery_key` row (Task 2 schema) persists only its path and metadata inside the create-fleet transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) reads and returns the printable key from that file for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` overwrites the file's bytes best-effort, unlinks it, and deletes the row, which is the only deletion path. The identity directory is already excluded from database backups and transfer snapshots, and a test scans the raw database, WAL, and staged transfer payload bytes to prove the printable key never appears in any of them and the file is gone after acknowledgement. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. +Do not return the printable recovery key as the transaction's only copy, and do not put the plaintext in the database at all: a SQLite `DELETE` removes a row logically while its bytes survive in pages, the freelist, and WAL files, and from there ride into backups and transfer snapshots, which would break the promise that the key is unrecoverable after acknowledgement. The pending key lives in a service-owned mode `0600` file under `resolved_data_root / "identity"`, written with the same atomic discipline as the identity keys, and the `pending_recovery_key` row (Task 2 schema) persists only its path and metadata inside the create-fleet transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) reads and returns the printable key from that file for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes the row in a committed transaction first, then overwrites the file's bytes best-effort and unlinks it, which is the only deletion path, ordered so no crash window can leave a durable row pointing at a missing file to block hub transfer indefinitely. The acknowledgement is idempotent: an ack with no row present returns success. Startup reconciles both partial states without owner intervention: a pending key file in the identity directory with no `pending_recovery_key` row is an acknowledgement that crashed mid-deletion, so startup overwrites and unlinks it, and a row whose file is missing (the impossible order, defended anyway) is completed as acknowledged by deleting the row rather than blocking transfer until someone edits the database. Test both windows: kill after the row delete but before the unlink and prove startup destroys the file, and delete the file out from under a live row and prove startup clears the row and transfer proceeds. The identity directory is already excluded from database backups and transfer snapshots, and a test scans the raw database, WAL, and staged transfer payload bytes to prove the printable key never appears in any of them and the file is gone after acknowledgement. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. - [ ] **Step 6: Mount `/api/v2` without weakening v1** @@ -1022,7 +1025,7 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. -A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule, persists the announcement columns atomically, and redirects transport. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies a strictly higher recovery epoch in that evidence stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. @@ -1198,13 +1201,13 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and mints its fresh self-announcement at floor plus one only when that evidence is a strictly higher generation than its own, or an equal generation whose announcement digest names a different hub lineage. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum with its evidence announcement retained in `floor_announcement_json` while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a generation above its own, or an equal one from a different lineage, never its own current announcement echoed back) mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum with its evidence announcement retained in `floor_announcement_json` while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a generation above its own, or an equal one from a different lineage, never its own current announcement echoed back) mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 704ac7c..8b91c9c 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -322,9 +322,9 @@ If validation fails before draining, the old hub remains authoritative and the t ### Split brain prevention -Every hub announcement contains the fleet UUID, hub machine UUID, addresses, and monotonically increasing generation. It is signed by the fleet key. +Every hub announcement contains the fleet UUID, hub machine UUID, addresses, a recovery epoch, and a monotonically increasing generation. It is signed by the fleet key. -Agents accept only a valid announcement with a higher generation than the one stored locally. An old hub that returns later cannot reclaim the fleet with stale state. +Agents accept only a valid announcement whose `(recovery epoch, generation)` pair is lexicographically higher than the one stored locally. Ordinary announcements (address changes, transfers) carry the current epoch forward and increment the generation. Dead-hub recovery is the one path that increments the epoch. An old hub that returns later cannot reclaim the fleet with stale state, even though it still holds the fleet signing key: anything it mints carries its old epoch, so no generation it can reach is ever accepted, and the rejection evidence (the agent's accepted announcement, fleet-signed) drives it into a visible retired state on its local recovery page. One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. Adoption never lowers the machine's monotonic generation floor, so the dead lineage's higher-generation announcement cannot be replayed to redirect the adopted machine, and the recovered hub learns of the floor and re-announces itself above it, converging the fleet past the dead lineage. @@ -360,7 +360,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. -6. Promote the PC to hub at a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its fleet-signed floor evidence announcement, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. +6. Promote the PC to hub at a recovery epoch one above the highest epoch the bundle or any verified probe response carries, and a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its fleet-signed floor evidence announcement, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. 7. Revoke browser sessions and integration credentials. 8. Accept reconnecting agents and rebuild central history from their retained buffers. From 69fc105a27060b224cc202eba335aa1a245e1328 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:25:40 -0400 Subject: [PATCH 48/65] docs: close the forty-fourth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 8 ++++---- .../specs/2026-08-11-humwatch-2.1-fleet-hub-design.md | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index aedec5a..b6c4bbe 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -388,8 +388,8 @@ hub_publication_obligations(announcement_generation, announcement_digest, announ hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) pending_recovery_key(id, key_path, created_at) staged_key_journal(journal_id, purpose, staged_path, created_at, finalized_at) -recovery_bundles(generation, bundle_revision, created_at, encrypted_payload, fleet_signature) -recovery_deliveries(generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) +recovery_bundles(recovery_epoch, generation, bundle_revision, created_at, encrypted_payload, fleet_signature) +recovery_deliveries(recovery_epoch, generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. @@ -1195,9 +1195,9 @@ Cover wrong keys, tampering, a forged bundle built with the legitimate recovery - [ ] **Step 2: Implement bundle creation and replication** -Serialize fleet identity, machine registry, settings, integrations, schema version, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind both generation and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the generation, revision, ciphertext digest, and associated data. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. +Serialize fleet identity, machine registry, settings, integrations, schema version, recovery epoch, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind recovery epoch, generation, and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the recovery epoch, generation, revision, ciphertext digest, and associated data. The epoch rides every layer (payload, associated data, signature, the `recovery_bundles` storage key, and the replica ordering tuple) for the same reason announcements carry it: a promotion forced to rely on its local bundle alone (every peer unreachable) must still mint at one above an epoch the bundle actually proves, and a stale former hub producing fleet-signed bundles at its old epoch must be fenced by ordering, not trusted by revision arithmetic. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. -Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. A non-`accepting` `transfer_state` pauses the loop rather than ending it: the tick skips its work and the loop keeps polling, so when a failed transfer's cleanup lifts the freeze back to `accepting` the very next tick resumes periodic bundles and queued delivery rows with no process restart. Only a role change away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting) exits the loop for good, since that database is no longer its to replicate. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple has a higher generation or the same generation with a higher revision. They reject an older or equal revision with a different digest, retain the newest valid bundle across restart, and return the accepted generation, revision, and digest so the hub cannot acknowledge stale ciphertext. +Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. A non-`accepting` `transfer_state` pauses the loop rather than ending it: the tick skips its work and the loop keeps polling, so when a failed transfer's cleanup lifts the freeze back to `accepting` the very next tick resumes periodic bundles and queued delivery rows with no process restart. Only a role change away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting) exits the loop for good, since that database is no longer its to replicate. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(recovery_epoch, generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name recovery epoch, generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(recovery_epoch, generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple is lexicographically higher on `(recovery_epoch, generation, bundle_revision)`, so a stale former hub's fleet-signed bundles at its old epoch are fenced by the same ordering that fences its announcements. They reject an older or equal tuple with a different digest, retain the newest valid bundle across restart, and return the accepted epoch, generation, revision, and digest so the hub cannot acknowledge stale ciphertext. - [ ] **Step 3: Implement local recovery preview and promotion** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 8b91c9c..051962a 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -340,7 +340,7 @@ Stable virtual addresses supplied by external network products are compatible bu The hub periodically creates a compact recovery bundle containing: -* Fleet UUID and current generation. +* Fleet UUID, current recovery epoch, and current generation. * Fleet signing key. * Machine registry and public keys. * Listener and fleet settings. @@ -357,7 +357,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 1. Choose **Recover Fleet** on another installation. 2. Enter the printable recovery key. -3. Select the valid recovery bundle with the highest `(generation, bundle revision)` tuple found locally. +3. Select the valid recovery bundle with the highest `(recovery epoch, generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. 6. Promote the PC to hub at a recovery epoch one above the highest epoch the bundle or any verified probe response carries, and a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its fleet-signed floor evidence announcement, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. From 8e2d8b0d9d452f5a2a3879e4e121c6a790d70597 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:35:05 -0400 Subject: [PATCH 49/65] docs: close the forty-fifth-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index b6c4bbe..8825300 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -392,7 +392,7 @@ recovery_bundles(recovery_epoch, generation, bundle_revision, created_at, encryp recovery_deliveries(recovery_epoch, generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` -Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(generation, bundle_revision)` for recovery bundles, `(generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(recovery_epoch, generation, bundle_revision)` for recovery bundles, `(recovery_epoch, generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. `enrollment_attempts.agent_claim_json` holds the complete claim Task 4 promises to reconstruct retries from, not merely its digest, and `agent_enrollment_state` is the agent-side mirror: the exact claim bytes the agent sent, the canonical digest of the hub claim it accepted at claim time, the record digest, and the stored finalize result that the digest-scoped `/finalize` retry serves, so either process restarting between claim and acknowledged finalize resumes from durable rows even when mutable inputs like hostname or detected addresses have since changed. The stored `hub_claim_digest` is what keeps finalization bound to the hub claim the pairing code authenticated across that restart: the signed record finalize presents must embed a hub claim whose canonical digest matches the stored one, a record carrying any other hub claim under the same enrollment ID is rejected as changed bytes, and a restart test proves it (claim, restart the agent, finalize with a substituted hub claim, expect rejection, then finalize with the original and expect success). `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. @@ -619,7 +619,8 @@ async def enroll_machine(address: str, code: str) -> EnrolledMachine: claim = await hub_client.claim(address, code, hub_claim) record = sign_enrollment_record(hub_claim, claim) await repository.store_claimed_attempt(record, address, claim.claim_nonce) - await hub_client.finalize(address, claim.enrollment_id, claim.claim_nonce, record) + announcement = current_hub_announcement() + await hub_client.finalize(address, claim.enrollment_id, claim.claim_nonce, record, announcement) return await repository.mark_finalized(claim.enrollment_id) ``` @@ -743,7 +744,7 @@ The reports route verifies content length, the outer request signature, the uplo Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Retention on a hub also ages out central fleet history: the existing service deletes only the legacy `metrics` and `process_snapshots` tables, so extend it to purge `fleet_metrics`, `fleet_processes`, and `report_batches` presence rows at or below each machine's persisted watermark older than `retention_days` in one transaction, with tests covering all three fleet tables (presence rows above the watermark are kept, since contiguity extends from the watermark through them). Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. -The same retention transaction also owns purging every other table nothing else claims an owner for: `used_nonces` rows past `expires_at`, `owner_sessions` rows past `expires_at`, `report_receipts` rows older than `retention_days`, `data_gaps` rows fully covered by history already purged above, and `recovery_bundles` rows below the newest `(generation, bundle_revision)` while keeping one prior bundle so an in-flight recovery preview against the second-newest revision does not fail mid-purge. Add tests for each: an expired nonce and an expired session are both gone after the pass, an old receipt is purged, a fully-covered gap range is dropped while a partially-covered one survives, and exactly one prior recovery bundle remains alongside the newest. +The same retention transaction also owns purging every other table nothing else claims an owner for: `used_nonces` rows past `expires_at`, `owner_sessions` rows past `expires_at`, `report_receipts` rows older than `retention_days`, `data_gaps` rows fully covered by history already purged above, and `recovery_bundles` rows below the newest `(recovery_epoch, generation, bundle_revision)` (lexicographic over the full triple, so a stale former hub's higher-generation bundle at an older epoch is the one purged, never the current lineage's) while keeping one prior bundle so an in-flight recovery preview against the second-newest revision does not fail mid-purge. Add tests for each: an expired nonce and an expired session are both gone after the pass, an old receipt is purged, a fully-covered gap range is dropped while a partially-covered one survives, and exactly one prior recovery bundle remains alongside the newest. - [ ] **Step 6: Verify collection, reporting, and full regression** From 0eea308f9ae313002885ececf47ba6761dcf926a Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:43:14 -0400 Subject: [PATCH 50/65] docs: close the forty-sixth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 14 +++++++------- .../2026-08-11-humwatch-2.1-fleet-hub-design.md | 4 ++-- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 8825300..cd9701d 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -179,7 +179,7 @@ class HubAnnouncement(BaseModel): signature: str ``` -`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. +`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. The persisted authority floor is the same kind of pair (`floor_recovery_epoch`, `generation_floor` in the Task 2 schema), compared lexicographically everywhere it is read, and promotion restarts the generation at 1 in its fresh epoch, which the pair rule makes strictly higher than every lower-epoch value however large. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. Internal signed requests use these headers: @@ -365,7 +365,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. `agent/fleet/schema.py` owns idempotent SQL for: ```sql -runtime_state(id, role, fleet_id, machine_id, hub_machine_id, recovery_epoch, hub_generation, generation_floor, hub_announcement_json, floor_announcement_json, hub_signing_public_key, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, recovery_epoch, hub_generation, floor_recovery_epoch, generation_floor, hub_announcement_json, floor_announcement_json, hub_signing_public_key, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) @@ -398,7 +398,7 @@ Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id `report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. The journal also cannot cover the window before it exists: a transfer import or recovery promotion decrypts the fleet key to its staging path before the transaction that records any reference commits, and a crash there would otherwise leave an untracked, directly usable plaintext key on a machine that restarts as a plain agent. Every staged key therefore lives only under the dedicated `resolved_data_root / "staging"` directory, and startup scrubs that directory: any file not referenced by an uncommitted-repair candidate (an unfinalized `staged_key_journal` row or a live `pending_hub_authority` row) is deleted before any role service starts, so the pre-commit crash window ends with the plaintext destroyed rather than orphaned. Test both journal windows for all three writers (creation, transfer activation, promotion), plus the pre-commit window for both stagers (kill after the key file is written but before the commit, restart, and prove the scrub removed it). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. -`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. `runtime_state.generation_floor` is the monotonic maximum of every generation this machine has ever accepted, and no path lowers it: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only generations strictly above the floor, so a dead hub's old higher-generation announcement cannot replay an adopted machine back to the dead lineage. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes, the loopback recovery page, and the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. The loopback recovery page stays reachable because the old hub can die between `commit_unannounced_hub` and its activation request, leaving no machine able to authorize activation or abort: its Cancel Pending Authority action runs the abort cleanup locally for an authority whose state was never `activated` (staged files deleted, prior settings restored, role back to `AGENT`), refuses to touch an activated one, and frees the machine to run ordinary dead-hub recovery from its replicated bundle, which a two-machine fleet's sole survivor depends on. Task 10's tests cover the stranded-target cancel followed by a successful promotion. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. The `runtime_state` pair `(floor_recovery_epoch, generation_floor)` is the machine's authority floor, the lexicographic maximum of every `(recovery_epoch, generation)` pair this machine has ever accepted, persisted as a pair together with its evidence announcement (whose own epoch and generation must equal the persisted pair) precisely because a scalar floor cannot survive an epoch transition: a machine moving from `(1, 100)` to a recovered `(2, 2)` has a floor of `(2, 2)`, not `100`, and comparing raw generations across epochs would either inflate the recovered lineage from obsolete evidence or wedge recovery at the storage ceiling. No path lowers the floor pair: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only pairs lexicographically above the floor, so a dead hub's old announcement cannot replay an adopted machine back to the dead lineage no matter how high its generation. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes, the loopback recovery page, and the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. The loopback recovery page stays reachable because the old hub can die between `commit_unannounced_hub` and its activation request, leaving no machine able to authorize activation or abort: its Cancel Pending Authority action runs the abort cleanup locally for an authority whose state was never `activated` (staged files deleted, prior settings restored, role back to `AGENT`), refuses to touch an activated one, and frees the machine to run ordinary dead-hub recovery from its replicated bundle, which a two-machine fleet's sole survivor depends on. Task 10's tests cover the stranded-target cancel followed by a successful promotion. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. It also owns the transaction helper used by every fleet repository: @@ -653,7 +653,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity, carried in the wire model itself: `AgentClaim.prior_identity_proof` is an Ed25519 signature by the previously registered machine signing key over the claim's canonical compact JSON with the proof field omitted (the same sorted-key canonicalization every other signed object here uses), present exactly when the claim names an already-registered machine ID. The Rejoin flow holds that key, since machine identity survives revocation, and the hub verifies the proof against the registered public key before keeping or replacing anything. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. The agent verifies it and persists it verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting `generation_floor` to that announcement's generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. The agent verifies it and persists it verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting the floor pair `(floor_recovery_epoch, generation_floor)` to that announcement's epoch and generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. @@ -1202,13 +1202,13 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The prober additionally rejects any verified generation whose increment would leave the storage-safe integer range, failing preview loudly rather than minting an unpersistable generation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation `max(bundle_generation, generation_floor, highest_probed_generation) + 1`, where the second term is the promoting machine's own durable `generation_floor` (which by definition is at or above its accepted generation, covering both the post-transfer window where the accepted generation exceeds the newest bundle's and the case where this machine manually adopted a lower generation and dies again before the floor-report convergence ran), and the third term is the highest generation learned by the preview probe, re-run at promotion time. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted generation, its `generation_floor`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its generation to equal the claimed floor, and mints its fresh self-announcement at floor plus one only when that evidence is a strictly higher generation than its own, or an equal generation whose announcement digest names a different hub lineage. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and mints its fresh self-announcement at its own epoch and floor generation plus one only when that evidence is a lexicographically higher pair at the hub's own epoch, or an equal pair whose announcement digest names a different hub lineage. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers `runtime_state.generation_floor`: the floor stays at the machine's pre-adoption maximum with its evidence announcement retained in `floor_announcement_json` while `hub_generation` drops to the adopted value, and remote announcement delivery keeps requiring a generation strictly above the floor, so a delayed publication retry or a replay of the dead lineage's valid higher-generation announcement cannot redirect the adopted machine back. The floor debt then clears itself fleet-wide: the signed watermark request after adoption carries the agent's `generation_floor` in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a generation above its own, or an equal one from a different lineage, never its own current announcement echoed back) mints a fresh fleet-signed announcement of itself at floor plus one and publishes it through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet above the dead lineage so no future replay of it can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 051962a..1db3781 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -326,7 +326,7 @@ Every hub announcement contains the fleet UUID, hub machine UUID, addresses, a r Agents accept only a valid announcement whose `(recovery epoch, generation)` pair is lexicographically higher than the one stored locally. Ordinary announcements (address changes, transfers) carry the current epoch forward and increment the generation. Dead-hub recovery is the one path that increments the epoch. An old hub that returns later cannot reclaim the fleet with stale state, even though it still holds the fleet signing key: anything it mints carries its old epoch, so no generation it can reach is ever accepted, and the rejection evidence (the agent's accepted announcement, fleet-signed) drives it into a visible retired state on its local recovery page. -One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. Adoption never lowers the machine's monotonic generation floor, so the dead lineage's higher-generation announcement cannot be replayed to redirect the adopted machine, and the recovered hub learns of the floor and re-announces itself above it, converging the fleet past the dead lineage. +One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. Adoption never lowers the machine's authority floor, itself a `(recovery epoch, generation)` pair compared lexicographically, so the dead lineage's announcement cannot be replayed to redirect the adopted machine whatever its generation, and a hub that reads verified floor evidence of unresolved authority at its own epoch re-announces above it, converging the fleet past the dead lineage. ### Address change @@ -360,7 +360,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 3. Select the valid recovery bundle with the highest `(recovery epoch, generation, bundle revision)` tuple found locally. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. -6. Promote the PC to hub at a recovery epoch one above the highest epoch the bundle or any verified probe response carries, and a generation above the bundle's generation, this machine's own monotonic generation floor, and the highest generation any reachable fleet member reports through its verified announcement or its fleet-signed floor evidence announcement, announcing the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. +6. Promote the PC to hub at a recovery epoch one above the highest epoch the bundle, this machine's own floor evidence, or any verified probe response carries, and generation 1: the fresh epoch outranks every `(epoch, generation)` pair any machine holds, so no generation arithmetic against the dead lineage is needed, and a dead lineage's generation at the storage ceiling cannot block recovery. Announce the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. 7. Revoke browser sessions and integration credentials. 8. Accept reconnecting agents and rebuild central history from their retained buffers. From 26a2817321a4e921b1a04b39dbe3442e9367d43a Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:49:52 -0400 Subject: [PATCH 51/65] docs: close the forty-seventh-round fleet hub findings Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index cd9701d..aeed272 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1192,7 +1192,7 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by `(generation, bundle_revision)`, promotion generation selection as `max(bundle_generation, generation_floor, highest_probed_generation) + 1` including recovery during the post-transfer replication window where the accepted generation exceeds the newest bundle's, promotion by a machine that had manually adopted a lower generation and recovers again before the floor-report convergence ran (its own floor lifts the new generation above every value it ever accepted), partial announcement delivery before hub death (a reachable machine accepted `G+1` while the recovering machine and its bundle sit at `G`, so the probe pushes promotion to `G+2` and that machine follows the new announcement), probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge floor never reaches the generation formula), preview failing loudly when a verified generation sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-generation lower- and equal-revision rejection, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), a bind failure leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, that same machine completing the full path (rejection, Rejoin under its proven identity, finalize-triggered watermark rewind with gap disclosure, retained history acknowledged on the recovered hub), zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by lexicographic `(recovery_epoch, generation, bundle_revision)` including an older-epoch bundle of far higher generation losing to a newer-epoch bundle of lower generation, promotion minting at `(max_epoch + 1, 1)` where the epoch term is the highest epoch proven by the bundle, the machine's own floor evidence, or any verified probe response, including recovery during the post-transfer replication window (a reachable machine accepted `(E, G+1)` while the recovering machine and its bundle sit at `(E, G)`, and promotion at `(E+1, 1)` outranks both so that machine follows the new announcement), promotion by a machine that had manually adopted a lower pair and recovers again before the floor-report convergence ran (its own floor evidence contributes its epoch to the epoch term), a second recovery on top of a first (epoch 2 to epoch 3) proving the generation reset repeats cleanly, probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge epoch or generation never reaches the promotion formula), preview failing loudly when a verified recovery epoch sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-pair lower- and equal-revision rejection plus an old-epoch bundle rejected outright by the replica replacement rule, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), promotion selecting the very address and port the machine's own agent listener already serves and succeeding through the `ListenerManager` reuse path instead of failing address-in-use, a bind failure on a genuinely new endpoint leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, that same machine completing the full path (rejection, Rejoin under its proven identity, finalize-triggered watermark rewind with gap disclosure, retained history acknowledged on the recovered hub), zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. - [ ] **Step 2: Implement bundle creation and replication** @@ -1202,7 +1202,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, exactly like Create Fleet: bind and hold the socket first, fail visibly without changing role when binding fails, attach the held socket through Task 3's `ListenerManager` after the promotion commit so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and never carry the dead hub's listeners or certificate paths forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and mints its fresh self-announcement at its own epoch and floor generation plus one only when that evidence is a lexicographically higher pair at the hub's own epoch, or an equal pair whose announcement digest names a different hub lineage. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. From 2a5be3f852da2105e629c8fd3edfadb1779943e4 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 11:57:11 -0400 Subject: [PATCH 52/65] docs: close the forty-eighth-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 11 ++++++----- .../specs/2026-08-11-humwatch-2.1-fleet-hub-design.md | 2 +- 2 files changed, 7 insertions(+), 6 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index aeed272..6603e89 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -161,6 +161,7 @@ class AgentClaim(BaseModel): class SignedEnrollmentRecord(BaseModel): enrollment_id: UUID agent_claim_digest: str + hub_announcement_digest: str hub_claim: HubClaim signature: str @@ -179,7 +180,7 @@ class HubAnnouncement(BaseModel): signature: str ``` -`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. The persisted authority floor is the same kind of pair (`floor_recovery_epoch`, `generation_floor` in the Task 2 schema), compared lexicographically everywhere it is read, and promotion restarts the generation at 1 in its fresh epoch, which the pair rule makes strictly higher than every lower-epoch value however large. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. +`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. Two partitioned machines can still recover the same fleet into the same fresh epoch, so same-epoch conflicts between distinct lineages resolve through Task 11's deterministic tie-break (the lexicographically smaller hub machine UUID wins and the loser retires) rather than through generation racing. The persisted authority floor is the same kind of pair (`floor_recovery_epoch`, `generation_floor` in the Task 2 schema), compared lexicographically everywhere it is read, and promotion restarts the generation at 1 in its fresh epoch, which the pair rule makes strictly higher than every lower-epoch value however large. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. Internal signed requests use these headers: @@ -653,7 +654,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity, carried in the wire model itself: `AgentClaim.prior_identity_proof` is an Ed25519 signature by the previously registered machine signing key over the claim's canonical compact JSON with the proof field omitted (the same sorted-key canonicalization every other signed object here uses), present exactly when the claim names an already-registered machine ID. The Rejoin flow holds that key, since machine identity survives revocation, and the hub verifies the proof against the registered public key before keeping or replacing anything. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. The agent verifies it and persists it verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting the floor pair `(floor_recovery_epoch, generation_floor)` to that announcement's epoch and generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. Self-authenticating is not self-binding, though: a first-enrolling agent has no floor yet, so an on-path host replaying the valid record and nonce could swap in any older fleet-signed announcement and finalize the agent onto a stale hub. The record therefore pins the delivery: `SignedEnrollmentRecord.hub_announcement_digest` is the canonical digest of the announcement finalize must carry, covered by the record's signature, and finalize verifies the carried announcement's digest against it and its fleet UUID, hub machine ID, authority pair, and `hub_signing_public_key` against the authenticated `HubClaim` before committing, rejecting any mismatch. Test the downgrade: a finalize carrying an older fleet-signed announcement with an otherwise valid record and nonce must be refused. The agent verifies all of that and persists the announcement verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting the floor pair `(floor_recovery_epoch, generation_floor)` to that announcement's epoch and generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. @@ -1026,7 +1027,7 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. -A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies a strictly higher recovery epoch in that evidence stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller hub machine UUID), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. @@ -1204,11 +1205,11 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and mints its fresh self-announcement at its own epoch and floor generation plus one only when that evidence is a lexicographically higher pair at the hub's own epoch, or an equal pair whose announcement digest names a different hub lineage. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose hub machine UUID is lexicographically smaller wins, a property fixed at promotion time that minting further generations cannot change. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. And test simultaneous recovery: two partitioned machines promote from the same bundle, both minting the same fresh epoch at generation 1, connectivity returns, the deterministic tie-break retires the larger-UUID lineage visibly while the winner re-announces above every pair the loser distributed, all agents converge on the winner, and repeated evidence exchanges after convergence mint nothing further. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 1db3781..ec2b825 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -324,7 +324,7 @@ If validation fails before draining, the old hub remains authoritative and the t Every hub announcement contains the fleet UUID, hub machine UUID, addresses, a recovery epoch, and a monotonically increasing generation. It is signed by the fleet key. -Agents accept only a valid announcement whose `(recovery epoch, generation)` pair is lexicographically higher than the one stored locally. Ordinary announcements (address changes, transfers) carry the current epoch forward and increment the generation. Dead-hub recovery is the one path that increments the epoch. An old hub that returns later cannot reclaim the fleet with stale state, even though it still holds the fleet signing key: anything it mints carries its old epoch, so no generation it can reach is ever accepted, and the rejection evidence (the agent's accepted announcement, fleet-signed) drives it into a visible retired state on its local recovery page. +Agents accept only a valid announcement whose `(recovery epoch, generation)` pair is lexicographically higher than the one stored locally. Ordinary announcements (address changes, transfers) carry the current epoch forward and increment the generation. Dead-hub recovery is the one path that increments the epoch. Two machines recovering the same fleet while partitioned can mint the same fresh epoch, so same-epoch conflicts between distinct hub lineages resolve deterministically: the lineage with the lexicographically smaller hub machine UUID wins, the loser stops publishing and retires visibly, and the winner re-announces above everything the loser distributed. An old hub that returns later cannot reclaim the fleet with stale state, even though it still holds the fleet signing key: anything it mints carries its old epoch, so no generation it can reach is ever accepted, and the rejection evidence (the agent's accepted announcement, fleet-signed) drives it into a visible retired state on its local recovery page. One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. Adoption never lowers the machine's authority floor, itself a `(recovery epoch, generation)` pair compared lexicographically, so the dead lineage's announcement cannot be replayed to redirect the adopted machine whatever its generation, and a hub that reads verified floor evidence of unresolved authority at its own epoch re-announces above it, converging the fleet past the dead lineage. From 32b5645533691234d2376c50231d239832378f3a Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:04:28 -0400 Subject: [PATCH 53/65] docs: close the forty-ninth-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 13 ++++++++----- .../2026-08-11-humwatch-2.1-fleet-hub-design.md | 4 ++-- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 6603e89..1ebccf4 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -172,6 +172,7 @@ class EnrolledMachine(BaseModel): class HubAnnouncement(BaseModel): fleet_id: UUID hub_machine_id: UUID + lineage_id: UUID recovery_epoch: int generation: int addresses: list[str] @@ -180,7 +181,7 @@ class HubAnnouncement(BaseModel): signature: str ``` -`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. Two partitioned machines can still recover the same fleet into the same fresh epoch, so same-epoch conflicts between distinct lineages resolve through Task 11's deterministic tie-break (the lexicographically smaller hub machine UUID wins and the loser retires) rather than through generation racing. The persisted authority floor is the same kind of pair (`floor_recovery_epoch`, `generation_floor` in the Task 2 schema), compared lexicographically everywhere it is read, and promotion restarts the generation at 1 in its fresh epoch, which the pair rule makes strictly higher than every lower-epoch value however large. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. +`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. Two partitioned machines can still recover the same fleet into the same fresh epoch, so same-epoch conflicts between distinct lineages resolve through Task 11's deterministic tie-break rather than through generation racing. Lineage identity is its own field because `hub_machine_id` does not survive a transfer: `lineage_id` is a UUID minted fresh at fleet creation and at each promotion, carried verbatim through every same-lineage mint and every transfer, so the tie-break stays pinned to the promotion event no matter which machine currently holds the role, and transfer timing can never change the winner. The persisted authority floor is the same kind of pair (`floor_recovery_epoch`, `generation_floor` in the Task 2 schema), compared lexicographically everywhere it is read, and promotion restarts the generation at 1 in its fresh epoch, which the pair rule makes strictly higher than every lower-epoch value however large. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. Internal signed requests use these headers: @@ -539,7 +540,7 @@ Keep the existing generic v1 request limiter. Add a login-specific limiter with - [ ] **Step 5: Implement create-fleet as one transaction** -Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the fleet-signed `HubAnnouncement` built from that listener at recovery epoch 1, generation 1 (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Every attached server task is configured with lifespan disabled: the process's original server already ran the app's startup and owns the shared database connection, the collectors, and the retention and delivery loops, and a second server invocation running the lifespan again would reopen or replace that connection and start duplicate copies of every background task the moment the fleet is created. Attached servers only serve requests, and the one lifecycle they participate in is the original server's shutdown, which detaches them. Test it by asserting exactly one collector tick source and one database connection before and after an attach. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the fleet-signed `HubAnnouncement` built from that listener at recovery epoch 1, generation 1, with a freshly minted `lineage_id` (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Every attached server task is configured with lifespan disabled: the process's original server already ran the app's startup and owns the shared database connection, the collectors, and the retention and delivery loops, and a second server invocation running the lifespan again would reopen or replace that connection and start duplicate copies of every background task the moment the fleet is created. Attached servers only serve requests, and the one lifecycle they participate in is the original server's shutdown, which detaches them. Test it by asserting exactly one collector tick source and one database connection before and after an attach. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key, an in-process attach test proving a request on the newly advertised listener succeeds with no process restart, and a kill between commit and attach proving the restarted process serves the stored listener. @@ -1027,7 +1028,9 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. -A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and nothing else ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller hub machine UUID), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and only a verified announcement, pushed by the hub or fetched by the agent's own discovery loop below, ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller `lineage_id`), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own, and a transfer announcement carries the source hub's `lineage_id` verbatim: the transfer replaces `hub_machine_id` with the target machine, never the lineage, so the same-epoch tie-break stays pinned to the original promotion event across any number of transfers. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. + +Hub push cannot be the only redirection path, because it needs the registry address to still reach the agent: an agent that was offline while the hub moved, and whose own DHCP or Tailscale address changed in the same window, leaves the delivery worker retrying a stale address while the agent retries a retired hub, and neither side can complete either the announcement delivery or the signed address update. The spec's discovery promise closes that loop from the agent side. When agent transport has failed to reach every address in its accepted announcement through its full backoff schedule, it enters a jittered discovery cycle: enumerate candidate hub addresses from the same LAN and Tailscale sources `discover_private_listeners()` draws from, fetch `GET /api/internal/v1/announcement` from each candidate, and feed any response through exactly the acceptance path a pushed delivery uses, fleet signature verification plus the strict above-floor pair rule, so discovery grants no trust that push did not already grant (a candidate returning garbage, a foreign fleet's announcement, or a dead lineage's old pair changes nothing). An accepted announcement redirects transport and triggers the standard signed address-update push so the registry learns where the agent lives now. The cycle repeats on the reconnect schedule until transport succeeds, and manual adoption on the local recovery page remains the fallback for a hub discovery cannot reach across subnets. Test the double-move: hub relocates and mints while the agent is offline and the agent's own address changes, the agent comes back, discovery finds the new hub, the pair rule accepts the announcement, transport redirects, the address update lands in the registry, and reporting resumes with no manual step. A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. @@ -1205,11 +1208,11 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose hub machine UUID is lexicographically smaller wins, a property fixed at promotion time that minting further generations cannot change. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose `lineage_id` is lexicographically smaller wins, a property minted at promotion time and carried verbatim through every same-lineage mint and transfer, so neither minting further generations nor transferring the role to another machine can change it. Different lineage means a differing `lineage_id`. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. And test simultaneous recovery: two partitioned machines promote from the same bundle, both minting the same fresh epoch at generation 1, connectivity returns, the deterministic tie-break retires the larger-UUID lineage visibly while the winner re-announces above every pair the loser distributed, all agents converge on the winner, and repeated evidence exchanges after convergence mint nothing further. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. And test simultaneous recovery: two partitioned machines promote from the same bundle, both minting the same fresh epoch at generation 1, connectivity returns, the deterministic tie-break retires the larger-`lineage_id` lineage visibly while the winner re-announces above every pair the loser distributed, all agents converge on the winner, repeated evidence exchanges after convergence mint nothing further, and a variant where one side transfers its hub role to a third machine before convergence proves the tie-break outcome is unchanged because the transfer carried the `lineage_id` verbatim. - [ ] **Step 4: Add the Recover Fleet page and evidence** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index ec2b825..5b9542e 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -322,9 +322,9 @@ If validation fails before draining, the old hub remains authoritative and the t ### Split brain prevention -Every hub announcement contains the fleet UUID, hub machine UUID, addresses, a recovery epoch, and a monotonically increasing generation. It is signed by the fleet key. +Every hub announcement contains the fleet UUID, hub machine UUID, a lineage UUID minted at fleet creation or promotion and preserved across transfers, addresses, a recovery epoch, and a monotonically increasing generation. It is signed by the fleet key. -Agents accept only a valid announcement whose `(recovery epoch, generation)` pair is lexicographically higher than the one stored locally. Ordinary announcements (address changes, transfers) carry the current epoch forward and increment the generation. Dead-hub recovery is the one path that increments the epoch. Two machines recovering the same fleet while partitioned can mint the same fresh epoch, so same-epoch conflicts between distinct hub lineages resolve deterministically: the lineage with the lexicographically smaller hub machine UUID wins, the loser stops publishing and retires visibly, and the winner re-announces above everything the loser distributed. An old hub that returns later cannot reclaim the fleet with stale state, even though it still holds the fleet signing key: anything it mints carries its old epoch, so no generation it can reach is ever accepted, and the rejection evidence (the agent's accepted announcement, fleet-signed) drives it into a visible retired state on its local recovery page. +Agents accept only a valid announcement whose `(recovery epoch, generation)` pair is lexicographically higher than the one stored locally. Ordinary announcements (address changes, transfers) carry the current epoch forward and increment the generation. Dead-hub recovery is the one path that increments the epoch. Two machines recovering the same fleet while partitioned can mint the same fresh epoch, so same-epoch conflicts between distinct hub lineages resolve deterministically: the lineage with the lexicographically smaller lineage UUID wins (a value fixed at promotion and unchanged by later transfers), the loser stops publishing and retires visibly, and the winner re-announces above everything the loser distributed. An old hub that returns later cannot reclaim the fleet with stale state, even though it still holds the fleet signing key: anything it mints carries its old epoch, so no generation it can reach is ever accepted, and the rejection evidence (the agent's accepted announcement, fleet-signed) drives it into a visible retired state on its local recovery page. One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. Adoption never lowers the machine's authority floor, itself a `(recovery epoch, generation)` pair compared lexicographically, so the dead lineage's announcement cannot be replayed to redirect the adopted machine whatever its generation, and a hub that reads verified floor evidence of unresolved authority at its own epoch re-announces above it, converging the fleet past the dead lineage. From e684ad8edd012ef4fc0c1099c6b11e3c5ce4635e Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:11:39 -0400 Subject: [PATCH 54/65] docs: close the fiftieth-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 1ebccf4..deb1964 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -866,11 +866,11 @@ assert.equal(sessionStorage.getItem("humwatch_auth_token"), null); Run: `.venv/bin/python -m pytest tests/test_frontend_auth.py tests/test_frontend_session_auth.py tests/test_frontend_setup.py -q` -Expected: FAIL while bearer headers, token storage, and the token prompt remain. +Expected: FAIL while bearer headers, token storage, and the token prompt remain reachable outside `RuntimeRole.LEGACY` (the role-scoped legacy client Step 3 retains is asserted present in `legacy` role, absent everywhere else). - [ ] **Step 3: Convert the browser client to owner sessions** -Remove bearer token storage and remote fetch helpers. Set `credentials: "same-origin"`. Load `/api/v2/session/me` before protected navigation. Send the returned CSRF token only on state-changing requests. +Convert the v2 roles to owner sessions without stranding upgraded installations that are still in `RuntimeRole.LEGACY`, whose protected v1 routes keep bearer authentication until Task 13's migration wizard converts them, several PRs from now. The client therefore branches on the role probe before choosing an auth path: in `legacy` role it keeps a minimal role-scoped token client (token storage, bearer header, and token prompt, exactly today's behavior, reachable only in that role), while every v2 role uses sessions with no bearer path at all. For the session path, set `credentials: "same-origin"`, load `/api/v2/session/me` before protected navigation, and send the returned CSRF token only on state-changing requests. Test both sides of the branch: a legacy-role installation's dashboard authenticates with its stored token end to end after this task, and a v2-role page contains no token storage, bearer header, or prompt. The legacy client is deleted in Task 13 alongside the wizard that ends the need for it. - [ ] **Step 4: Implement setup and login pages** @@ -1028,7 +1028,7 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. -A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and only a verified announcement, pushed by the hub or fetched by the agent's own discovery loop below, ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller `lineage_id`), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own, and a transfer announcement carries the source hub's `lineage_id` verbatim: the transfer replaces `hub_machine_id` with the target machine, never the lineage, so the same-epoch tie-break stays pinned to the original promotion event across any number of transfers. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and only a verified announcement, pushed by the hub or fetched by the agent's own discovery loop below, ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller `lineage_id`), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. The winning side of that same comparison acts too, because an agent pointed at a retired loser can never send the watermark request that drives Task 11's convergence: a delivering hub that verifies in the rejection evidence a different lineage, at its own epoch, at an equal or higher pair, that loses the tie-break to it, mints a fresh self-announcement above the rejected evidence's pair and republishes through the normal obligations, so agents holding the losing lineage accept the winner under the strict rule on the very next delivery. Test both branches from the same simultaneous-recovery fixture: the loser's delivery rejection retires it, and the winner's delivery rejection drives the mint that converges the loser's agents. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own, and a transfer announcement carries the source hub's `lineage_id` verbatim: the transfer replaces `hub_machine_id` with the target machine, never the lineage, so the same-epoch tie-break stays pinned to the original promotion event across any number of transfers. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. Hub push cannot be the only redirection path, because it needs the registry address to still reach the agent: an agent that was offline while the hub moved, and whose own DHCP or Tailscale address changed in the same window, leaves the delivery worker retrying a stale address while the agent retries a retired hub, and neither side can complete either the announcement delivery or the signed address update. The spec's discovery promise closes that loop from the agent side. When agent transport has failed to reach every address in its accepted announcement through its full backoff schedule, it enters a jittered discovery cycle: enumerate candidate hub addresses from the same LAN and Tailscale sources `discover_private_listeners()` draws from, fetch `GET /api/internal/v1/announcement` from each candidate, and feed any response through exactly the acceptance path a pushed delivery uses, fleet signature verification plus the strict above-floor pair rule, so discovery grants no trust that push did not already grant (a candidate returning garbage, a foreign fleet's announcement, or a dead lineage's old pair changes nothing). An accepted announcement redirects transport and triggers the standard signed address-update push so the registry learns where the agent lives now. The cycle repeats on the reconnect schedule until transport succeeds, and manual adoption on the local recovery page remains the fallback for a hub discovery cannot reach across subnets. Test the double-move: hub relocates and mints while the agent is offline and the agent's own address changes, the agent comes back, discovery finds the new hub, the pair rule accepts the announcement, transport redirects, the address update lands in the registry, and reporting resumes with no manual step. @@ -1355,7 +1355,7 @@ Fresh installations have only public v1 health. Upgraded installations default p - [ ] **Step 5: Implement the migration wizard and peer suggestions** -Read `humwatch_machines` from browser local storage as untrusted enrollment suggestions. Clear it only after owner confirmation. Present Create Fleet and Join Fleet with data counts and exact archive location. +Read `humwatch_machines` from browser local storage as untrusted enrollment suggestions. Clear it only after owner confirmation. Present Create Fleet and Join Fleet with data counts and exact archive location. This task also deletes the role-scoped legacy token client Task 7 retained (token storage, bearer header, and prompt), since the wizard ends the last role that needs it, with the Task 7 role-branch test flipped to assert the token path is gone everywhere. - [ ] **Step 6: Produce migration evidence and canary** From d8e7a4bcfcaaf4b5534c17a45949d9cf575f48c2 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:16:08 -0400 Subject: [PATCH 55/65] docs: close the fifty-first-round fleet hub findings Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index deb1964..491afb8 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1028,7 +1028,7 @@ Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public- `PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. -A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and only a verified announcement, pushed by the hub or fetched by the agent's own discovery loop below, ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller `lineage_id`), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. The winning side of that same comparison acts too, because an agent pointed at a retired loser can never send the watermark request that drives Task 11's convergence: a delivering hub that verifies in the rejection evidence a different lineage, at its own epoch, at an equal or higher pair, that loses the tie-break to it, mints a fresh self-announcement above the rejected evidence's pair and republishes through the normal obligations, so agents holding the losing lineage accept the winner under the strict rule on the very next delivery. Test both branches from the same simultaneous-recovery fixture: the loser's delivery rejection retires it, and the winner's delivery rejection drives the mint that converges the loser's agents. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own, and a transfer announcement carries the source hub's `lineage_id` verbatim: the transfer replaces `hub_machine_id` with the target machine, never the lineage, so the same-epoch tie-break stays pinned to the original promotion event across any number of transfers. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and only a verified announcement, pushed by the hub or fetched by the agent's own discovery loop below, ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the lineage tie-break before any pair arithmetic, then applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. The tie-break coming first is load-bearing: an announcement at the accepted announcement's epoch from a different lineage enters the pair comparison only when its lineage wins the tie-break against the accepted one, and a losing lineage's announcement is rejected with evidence no matter how high its pair climbs, because a partitioned loser that raced ahead in generations would otherwise be accepted outright by the floor rule and redirect the winning side's agents without the tie-break handler ever seeing evidence. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller `lineage_id`), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. The winning side of that same comparison acts too, because an agent pointed at a retired loser can never send the watermark request that drives Task 11's convergence: a delivering hub that verifies in the rejection evidence a different lineage, at its own epoch, at an equal or higher pair, that loses the tie-break to it, mints a fresh self-announcement above the rejected evidence's pair and republishes through the normal obligations, so agents holding the losing lineage accept the winner under the strict rule on the very next delivery. Test both branches from the same simultaneous-recovery fixture: the loser's delivery rejection retires it, and the winner's delivery rejection drives the mint that converges the loser's agents. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own, and a transfer announcement carries the source hub's `lineage_id` verbatim: the transfer replaces `hub_machine_id` with the target machine, never the lineage, so the same-epoch tie-break stays pinned to the original promotion event across any number of transfers. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. Hub push cannot be the only redirection path, because it needs the registry address to still reach the agent: an agent that was offline while the hub moved, and whose own DHCP or Tailscale address changed in the same window, leaves the delivery worker retrying a stale address while the agent retries a retired hub, and neither side can complete either the announcement delivery or the signed address update. The spec's discovery promise closes that loop from the agent side. When agent transport has failed to reach every address in its accepted announcement through its full backoff schedule, it enters a jittered discovery cycle: enumerate candidate hub addresses from the same LAN and Tailscale sources `discover_private_listeners()` draws from, fetch `GET /api/internal/v1/announcement` from each candidate, and feed any response through exactly the acceptance path a pushed delivery uses, fleet signature verification plus the strict above-floor pair rule, so discovery grants no trust that push did not already grant (a candidate returning garbage, a foreign fleet's announcement, or a dead lineage's old pair changes nothing). An accepted announcement redirects transport and triggers the standard signed address-update push so the registry learns where the agent lives now. The cycle repeats on the reconnect schedule until transport succeeds, and manual adoption on the local recovery page remains the fallback for a hub discovery cannot reach across subnets. Test the double-move: hub relocates and mints while the agent is offline and the agent's own address changes, the agent comes back, discovery finds the new hub, the pair rule accepts the announcement, transport redirects, the address update lands in the registry, and reporting resumes with no manual step. @@ -1212,7 +1212,7 @@ Rebuilding history needs an explicit rewind, because the imported registry carri The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. -The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. And test simultaneous recovery: two partitioned machines promote from the same bundle, both minting the same fresh epoch at generation 1, connectivity returns, the deterministic tie-break retires the larger-`lineage_id` lineage visibly while the winner re-announces above every pair the loser distributed, all agents converge on the winner, repeated evidence exchanges after convergence mint nothing further, and a variant where one side transfers its hub role to a third machine before convergence proves the tie-break outcome is unchanged because the transfer carried the `lineage_id` verbatim. +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. And test simultaneous recovery: two partitioned machines promote from the same bundle, both minting the same fresh epoch at generation 1, connectivity returns, the deterministic tie-break retires the larger-`lineage_id` lineage visibly while the winner re-announces above every pair the loser distributed, a variant where the loser out-minted the winner during the partition (loser at generation 2, winner at 1) proves winning-lineage agents reject the loser's higher pair outright through the tie-break-first acceptance rule and the returned evidence still drives the winner's converging mint, all agents converge on the winner, repeated evidence exchanges after convergence mint nothing further, and a variant where one side transfers its hub role to a third machine before convergence proves the tie-break outcome is unchanged because the transfer carried the `lineage_id` verbatim. - [ ] **Step 4: Add the Recover Fleet page and evidence** @@ -1355,7 +1355,7 @@ Fresh installations have only public v1 health. Upgraded installations default p - [ ] **Step 5: Implement the migration wizard and peer suggestions** -Read `humwatch_machines` from browser local storage as untrusted enrollment suggestions. Clear it only after owner confirmation. Present Create Fleet and Join Fleet with data counts and exact archive location. This task also deletes the role-scoped legacy token client Task 7 retained (token storage, bearer header, and prompt), since the wizard ends the last role that needs it, with the Task 7 role-branch test flipped to assert the token path is gone everywhere. +Read `humwatch_machines` from browser local storage as untrusted enrollment suggestions. Clear it only after owner confirmation. Present Create Fleet and Join Fleet with data counts and exact archive location. This task also deletes the general role-scoped legacy token client Task 7 retained, but not the wizard's own: a legacy installation migrated from a browser on another LAN machine has no owner account yet, and its inspection and conversion requests must carry the existing bearer token, so the wizard keeps a wizard-scoped token prompt and client used only by the migration pages against a still-legacy installation and discarded the moment conversion commits. The flipped Task 7 role-branch test asserts the token path survives nowhere except inside the wizard flow, and a test drives a full remote conversion through the wizard-scoped token from a browser that never had the general client. - [ ] **Step 6: Produce migration evidence and canary** From 316b44c8703274e5e68c06e670de060511ca02d3 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:21:26 -0400 Subject: [PATCH 56/65] docs: close the fifty-second-round fleet hub findings Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 491afb8..c6bd2f9 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1071,7 +1071,7 @@ Run focused tests, static asset verification, browser smoke, full pytest, then c **Interfaces:** * Produces: `HubAnnouncement.sign(fleet_identity) -> HubAnnouncement` -* Produces: `verify_announcement(announcement, fleet_public_key, current_generation) -> bool` +* Produces: `verify_announcement(announcement, fleet_public_key, accepted: HubAnnouncement | None, floor_recovery_epoch: int, generation_floor: int) -> bool` (the accepted announcement supplies the current epoch, generation, and `lineage_id` the tie-break-first rule compares against, alongside the floor pair, and is `None` only before first acceptance) * Produces: `HubTransferService.preflight(target_id) -> TransferPreflight` (includes the target's detected private listener candidates) * Produces: `HubTransferService.transfer(target_id, listener: ListenerSelection) -> TransferResult` * Produces: `FleetStateRepository.enqueue_signed_abort(announcement) -> None` (durable row in `hub_transfer_aborts`, Task 2 schema, resumed from startup like `recovery_deliveries`) @@ -1206,7 +1206,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` all advance atomically with the role change, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, `recovery_epoch`, `hub_generation`, and the complete floor pair `floor_recovery_epoch` and `generation_floor` all advance atomically with the role change, the floor columns always matching the pair the evidence announcement proves, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's, the watermark verifier never sees a floor claim its evidence contradicts, and the above-floor rule cannot admit a high-generation announcement from the fenced old epoch through a stale `floor_recovery_epoch`), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose `lineage_id` is lexicographically smaller wins, a property minted at promotion time and carried verbatim through every same-lineage mint and transfer, so neither minting further generations nor transferring the role to another machine can change it. Different lineage means a differing `lineage_id`. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. From e5d85fde5f16f26a277bedc28af9dbd6b8d14ae0 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:26:24 -0400 Subject: [PATCH 57/65] docs: close the fifty-third-round fleet hub finding Co-Authored-By: Claude Fable 5 --- .../plans/2026-08-13-humwatch-2.1-fleet-hub.md | 14 +++++++------- .../2026-08-11-humwatch-2.1-fleet-hub-design.md | 4 ++-- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index c6bd2f9..046ea84 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -390,11 +390,11 @@ hub_publication_obligations(announcement_generation, announcement_digest, announ hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) pending_recovery_key(id, key_path, created_at) staged_key_journal(journal_id, purpose, staged_path, created_at, finalized_at) -recovery_bundles(recovery_epoch, generation, bundle_revision, created_at, encrypted_payload, fleet_signature) -recovery_deliveries(recovery_epoch, generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) +recovery_bundles(lineage_id, recovery_epoch, generation, bundle_revision, created_at, encrypted_payload, fleet_signature) +recovery_deliveries(lineage_id, recovery_epoch, generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) ``` -Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(recovery_epoch, generation, bundle_revision)` for recovery bundles, `(recovery_epoch, generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(lineage_id, recovery_epoch, generation, bundle_revision)` for recovery bundles, `(lineage_id, recovery_epoch, generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. `enrollment_attempts.agent_claim_json` holds the complete claim Task 4 promises to reconstruct retries from, not merely its digest, and `agent_enrollment_state` is the agent-side mirror: the exact claim bytes the agent sent, the canonical digest of the hub claim it accepted at claim time, the record digest, and the stored finalize result that the digest-scoped `/finalize` retry serves, so either process restarting between claim and acknowledged finalize resumes from durable rows even when mutable inputs like hostname or detected addresses have since changed. The stored `hub_claim_digest` is what keeps finalization bound to the hub claim the pairing code authenticated across that restart: the signed record finalize presents must embed a hub claim whose canonical digest matches the stored one, a record carrying any other hub claim under the same enrollment ID is rejected as changed bytes, and a restart test proves it (claim, restart the agent, finalize with a substituted hub claim, expect rejection, then finalize with the original and expect success). `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. @@ -746,7 +746,7 @@ The reports route verifies content length, the outer request signature, the uplo Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Retention on a hub also ages out central fleet history: the existing service deletes only the legacy `metrics` and `process_snapshots` tables, so extend it to purge `fleet_metrics`, `fleet_processes`, and `report_batches` presence rows at or below each machine's persisted watermark older than `retention_days` in one transaction, with tests covering all three fleet tables (presence rows above the watermark are kept, since contiguity extends from the watermark through them). Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. -The same retention transaction also owns purging every other table nothing else claims an owner for: `used_nonces` rows past `expires_at`, `owner_sessions` rows past `expires_at`, `report_receipts` rows older than `retention_days`, `data_gaps` rows fully covered by history already purged above, and `recovery_bundles` rows below the newest `(recovery_epoch, generation, bundle_revision)` (lexicographic over the full triple, so a stale former hub's higher-generation bundle at an older epoch is the one purged, never the current lineage's) while keeping one prior bundle so an in-flight recovery preview against the second-newest revision does not fail mid-purge. Add tests for each: an expired nonce and an expired session are both gone after the pass, an old receipt is purged, a fully-covered gap range is dropped while a partially-covered one survives, and exactly one prior recovery bundle remains alongside the newest. +The same retention transaction also owns purging every other table nothing else claims an owner for: `used_nonces` rows past `expires_at`, `owner_sessions` rows past `expires_at`, `report_receipts` rows older than `retention_days`, `data_gaps` rows fully covered by history already purged above, and `recovery_bundles` rows below the newest `(recovery_epoch, generation, bundle_revision)` (tie-break first, then lexicographic over the full triple, so a stale former hub's higher-generation bundle at an older epoch and a same-epoch losing lineage's are what get purged, never the current winning lineage's) while keeping one prior bundle so an in-flight recovery preview against the second-newest revision does not fail mid-purge. Add tests for each: an expired nonce and an expired session are both gone after the pass, an old receipt is purged, a fully-covered gap range is dropped while a partially-covered one survives, and exactly one prior recovery bundle remains alongside the newest. - [ ] **Step 6: Verify collection, reporting, and full regression** @@ -1196,13 +1196,13 @@ def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> Recove - [ ] **Step 1: Write cryptographic and recovery tests** -Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by lexicographic `(recovery_epoch, generation, bundle_revision)` including an older-epoch bundle of far higher generation losing to a newer-epoch bundle of lower generation, promotion minting at `(max_epoch + 1, 1)` where the epoch term is the highest epoch proven by the bundle, the machine's own floor evidence, or any verified probe response, including recovery during the post-transfer replication window (a reachable machine accepted `(E, G+1)` while the recovering machine and its bundle sit at `(E, G)`, and promotion at `(E+1, 1)` outranks both so that machine follows the new announcement), promotion by a machine that had manually adopted a lower pair and recovers again before the floor-report convergence ran (its own floor evidence contributes its epoch to the epoch term), a second recovery on top of a first (epoch 2 to epoch 3) proving the generation reset repeats cleanly, probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge epoch or generation never reaches the promotion formula), preview failing loudly when a verified recovery epoch sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-pair lower- and equal-revision rejection plus an old-epoch bundle rejected outright by the replica replacement rule, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), promotion selecting the very address and port the machine's own agent listener already serves and succeeding through the `ListenerManager` reuse path instead of failing address-in-use, a bind failure on a genuinely new endpoint leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, that same machine completing the full path (rejection, Rejoin under its proven identity, finalize-triggered watermark rewind with gap disclosure, retained history acknowledged on the recovered hub), zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by the tie-break-first rule then lexicographic `(recovery_epoch, generation, bundle_revision)`, including an older-epoch bundle of far higher generation losing to a newer-epoch bundle of lower generation and a same-epoch losing lineage's higher-revision bundle rejected outright at both replica acceptance and recovery selection, promotion minting at `(max_epoch + 1, 1)` where the epoch term is the highest epoch proven by the bundle, the machine's own floor evidence, or any verified probe response, including recovery during the post-transfer replication window (a reachable machine accepted `(E, G+1)` while the recovering machine and its bundle sit at `(E, G)`, and promotion at `(E+1, 1)` outranks both so that machine follows the new announcement), promotion by a machine that had manually adopted a lower pair and recovers again before the floor-report convergence ran (its own floor evidence contributes its epoch to the epoch term), a second recovery on top of a first (epoch 2 to epoch 3) proving the generation reset repeats cleanly, probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge epoch or generation never reaches the promotion formula), preview failing loudly when a verified recovery epoch sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-pair lower- and equal-revision rejection plus an old-epoch bundle rejected outright by the replica replacement rule, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), promotion selecting the very address and port the machine's own agent listener already serves and succeeding through the `ListenerManager` reuse path instead of failing address-in-use, a bind failure on a genuinely new endpoint leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, that same machine completing the full path (rejection, Rejoin under its proven identity, finalize-triggered watermark rewind with gap disclosure, retained history acknowledged on the recovered hub), zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. - [ ] **Step 2: Implement bundle creation and replication** -Serialize fleet identity, machine registry, settings, integrations, schema version, recovery epoch, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind recovery epoch, generation, and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the recovery epoch, generation, revision, ciphertext digest, and associated data. The epoch rides every layer (payload, associated data, signature, the `recovery_bundles` storage key, and the replica ordering tuple) for the same reason announcements carry it: a promotion forced to rely on its local bundle alone (every peer unreachable) must still mint at one above an epoch the bundle actually proves, and a stale former hub producing fleet-signed bundles at its old epoch must be fenced by ordering, not trusted by revision arithmetic. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. +Serialize fleet identity, machine registry, settings, integrations, schema version, `lineage_id`, recovery epoch, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind `lineage_id`, recovery epoch, generation, and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the `lineage_id`, recovery epoch, generation, revision, ciphertext digest, and associated data. The epoch rides every layer (payload, associated data, signature, the `recovery_bundles` storage key, and the replica ordering tuple) for the same reason announcements carry it: a promotion forced to rely on its local bundle alone (every peer unreachable) must still mint at one above an epoch the bundle actually proves, and a stale former hub producing fleet-signed bundles at its old epoch must be fenced by ordering, not trusted by revision arithmetic. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. -Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. A non-`accepting` `transfer_state` pauses the loop rather than ending it: the tick skips its work and the loop keeps polling, so when a failed transfer's cleanup lifts the freeze back to `accepting` the very next tick resumes periodic bundles and queued delivery rows with no process restart. Only a role change away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting) exits the loop for good, since that database is no longer its to replicate. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(recovery_epoch, generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name recovery epoch, generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(recovery_epoch, generation, bundle_revision, digest)` and atomically replace a replica only when the incoming authenticated tuple is lexicographically higher on `(recovery_epoch, generation, bundle_revision)`, so a stale former hub's fleet-signed bundles at its old epoch are fenced by the same ordering that fences its announcements. They reject an older or equal tuple with a different digest, retain the newest valid bundle across restart, and return the accepted epoch, generation, revision, and digest so the hub cannot acknowledge stale ciphertext. +Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. A non-`accepting` `transfer_state` pauses the loop rather than ending it: the tick skips its work and the loop keeps polling, so when a failed transfer's cleanup lifts the freeze back to `accepting` the very next tick resumes periodic bundles and queued delivery rows with no process restart. Only a role change away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting) exits the loop for good, since that database is no longer its to replicate. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(recovery_epoch, generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name recovery epoch, generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(lineage_id, recovery_epoch, generation, bundle_revision, digest)` and replace a replica under the same discipline announcements use, tie-break first, then ordering: a same-epoch bundle from a different lineage is admitted to the tuple comparison only when its lineage wins the deterministic tie-break against the retained one (a losing lineage's bundle is rejected whatever its generation or revision, because both partitioned recoveries hold the fleet signing key and the loser can sign arbitrarily high tuples), and an admitted bundle replaces atomically only when lexicographically higher on `(recovery_epoch, generation, bundle_revision)`, so a stale former hub's fleet-signed bundles at its old epoch are fenced by ordering and a same-epoch loser's by lineage, exactly like its announcements. Without the bundle-side tie-break, a winner hub dying before its next replication would let recovery resurrect the losing lineage's registry and settings, including machines revoked only on the winner. They reject an older or equal tuple with a different digest, retain the newest valid bundle across restart, and return the accepted epoch, generation, revision, and digest so the hub cannot acknowledge stale ciphertext. - [ ] **Step 3: Implement local recovery preview and promotion** diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md index 5b9542e..9f14763 100644 --- a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -340,7 +340,7 @@ Stable virtual addresses supplied by external network products are compatible bu The hub periodically creates a compact recovery bundle containing: -* Fleet UUID, current recovery epoch, and current generation. +* Fleet UUID, lineage UUID, current recovery epoch, and current generation. * Fleet signing key. * Machine registry and public keys. * Listener and fleet settings. @@ -357,7 +357,7 @@ Each bundle is encrypted to the recovery public key with an ephemeral X25519 key 1. Choose **Recover Fleet** on another installation. 2. Enter the printable recovery key. -3. Select the valid recovery bundle with the highest `(recovery epoch, generation, bundle revision)` tuple found locally. +3. Select the valid recovery bundle with the highest `(recovery epoch, generation, bundle revision)` tuple found locally, after the same lineage tie-break announcements use: a same-epoch bundle from a losing lineage is never selected over a winning lineage's, whatever its generation or revision. 4. Set a new owner username and password. 5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. 6. Promote the PC to hub at a recovery epoch one above the highest epoch the bundle, this machine's own floor evidence, or any verified probe response carries, and generation 1: the fresh epoch outranks every `(epoch, generation)` pair any machine holds, so no generation arithmetic against the dead lineage is needed, and a dead lineage's generation at the storage ceiling cannot block recovery. Announce the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. From 1451e7d198372a0d6cfc5521cef8d4cbe766112c Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:33:00 -0400 Subject: [PATCH 58/65] docs: close the fifty-fourth-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 046ea84..4a8ef57 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -125,6 +125,7 @@ class SignedReportUpload(BaseModel): upload_id: UUID sequence_start: int sequence_end: int + outbox_high_water: int batches: list[ReportBatch] gaps: list[GapRange] = Field(default_factory=list) payload_digest: str @@ -368,7 +369,7 @@ Expected: collection FAIL for missing `agent.fleet` modules. ```sql runtime_state(id, role, fleet_id, machine_id, hub_machine_id, recovery_epoch, hub_generation, floor_recovery_epoch, generation_floor, hub_announcement_json, floor_announcement_json, hub_signing_public_key, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) -fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence) +fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence, outbox_high_water) owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) fleet_settings(id, listeners_json, https_cert_path, https_key_path, allow_public_listener, updated_at) @@ -715,6 +716,8 @@ async def ingest(self, upload: SignedReportUpload) -> IngestAck: return IngestAck(highest_contiguous_sequence=highest) ``` +Two of `mark_reported`'s writes are monotonic maxima, never plain replacements, because oldest-first backfill interleaves with live uploads: `last_seen` stores the greater of its existing value and `upload.latest_activity_at()`, since a historical batch landing after a current one would otherwise flip an actively reporting machine to stale for the entire backfill (Task 6 computes status from this column), and `fleet_machines.outbox_high_water` stores the greater of its existing value and the upload's `outbox_high_water`. The agent stamps that field at upload build time from its persisted `sqlite_sequence` high-water mark for `report_outbox`, riding inside the signed payload like every other field, so the hub durably knows the newest sequence the agent has allocated even when the 100-batch chunk limit means `sequence_end` trails it by thousands, and Task 6's backfill object computes its newest local sequence and pending count from these columns instead of guessing from chunk edges. + - [ ] **Step 1: Write outbox and idempotency tests** ```python @@ -726,7 +729,7 @@ async def test_duplicate_upload_is_stored_once(hub_ingestion, enrolled_machine, assert await count_metrics(upload.machine_id, upload.sequence_start) == len(upload.batches[0].metrics) ``` -Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic, a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Test sequence allocation after total purge: acknowledge every outbox row, delete them all through the retention path, restart the repository, and prove the next append allocates the prior maximum plus one from the persisted `sqlite_sequence` high-water mark rather than reusing an acknowledged sequence. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state.transfer_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. Also cover retention purging expired `used_nonces`, expired `owner_sessions`, aged `report_receipts`, fully-covered `data_gaps`, and all but the newest-plus-one `recovery_bundles` rows. +Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic (asserting `last_seen` and `outbox_high_water` never regress when a historical chunk lands after a live upload), a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Test sequence allocation after total purge: acknowledge every outbox row, delete them all through the retention path, restart the repository, and prove the next append allocates the prior maximum plus one from the persisted `sqlite_sequence` high-water mark rather than reusing an acknowledged sequence. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state.transfer_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. Also cover retention purging expired `used_nonces`, expired `owner_sessions`, aged `report_receipts`, fully-covered `data_gaps`, and all but the newest-plus-one `recovery_bundles` rows. - [ ] **Step 2: Run focused tests and confirm missing outbox behavior** From 4e68b1718d88d160917a506ca21bae3531cd2d95 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:38:44 -0400 Subject: [PATCH 59/65] docs: close the fifty-fifth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 4a8ef57..00bb3a1 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -656,7 +656,7 @@ The local setup surface first reads private LAN and Tailscale candidates, then w The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity, carried in the wire model itself: `AgentClaim.prior_identity_proof` is an Ed25519 signature by the previously registered machine signing key over the claim's canonical compact JSON with the proof field omitted (the same sorted-key canonicalization every other signed object here uses), present exactly when the claim names an already-registered machine ID. The Rejoin flow holds that key, since machine identity survives revocation, and the hub verifies the proof against the registered public key before keeping or replacing anything. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. -Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. Self-authenticating is not self-binding, though: a first-enrolling agent has no floor yet, so an on-path host replaying the valid record and nonce could swap in any older fleet-signed announcement and finalize the agent onto a stale hub. The record therefore pins the delivery: `SignedEnrollmentRecord.hub_announcement_digest` is the canonical digest of the announcement finalize must carry, covered by the record's signature, and finalize verifies the carried announcement's digest against it and its fleet UUID, hub machine ID, authority pair, and `hub_signing_public_key` against the authenticated `HubClaim` before committing, rejecting any mismatch. Test the downgrade: a finalize carrying an older fleet-signed announcement with an otherwise valid record and nonce must be refused. The agent verifies all of that and persists the announcement verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting the floor pair `(floor_recovery_epoch, generation_floor)` to that announcement's epoch and generation in the same transaction, so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. Self-authenticating is not self-binding, though: a first-enrolling agent has no floor yet, so an on-path host replaying the valid record and nonce could swap in any older fleet-signed announcement and finalize the agent onto a stale hub. The record therefore pins the delivery: `SignedEnrollmentRecord.hub_announcement_digest` is the canonical digest of the announcement finalize must carry, covered by the record's signature, and finalize verifies the carried announcement's digest against it and its fleet UUID, hub machine ID, authority pair, and `hub_signing_public_key` against the authenticated `HubClaim` before committing, rejecting any mismatch. Test the downgrade: a finalize carrying an older fleet-signed announcement with an otherwise valid record and nonce must be refused. The agent verifies all of that and persists the announcement verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting the floor pair `(floor_recovery_epoch, generation_floor)` to that announcement's epoch and generation in the same transaction (under the never-lower rule every floor write obeys: a rejoining machine that retains a higher floor from its earlier tenure keeps it and its evidence), so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. @@ -1141,7 +1141,7 @@ Create a SQLite backup through the backup API. Package fleet config, owner hash, The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target secures that socket before `commit_unannounced_hub`: when the selection names an address the target's `ListenerManager` already serves (the normal case, since Task 4 keeps the enrolled agent's listener bound for announcement and recovery routes), staging reuses that existing bound socket and atomically widens its route set at commit instead of binding the same address a second time, which would fail address-in-use on every ordinary transfer. Only a selection naming a genuinely new address is freshly bound and held, then attached through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure on such a new address fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. -The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, installs the transfer announcement as the target's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, and `generation_floor` advance together, so the target's probe endpoint and any later enrollment serve the new authority rather than the old hub's), and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report under the committed proposed generation, but that report check is non-committing by design: the old hub constructs a dedicated probe payload rather than handing over a real outbox batch, the target runs the full ingestion path (signature, admission, inserts, contiguity) inside a transaction it rolls back deliberately, and the result leaves the target as a check digest, never as an `IngestAck` any agent outbox could consume. Nothing about the probe persists, so a draining commit failure that aborts the still-staged target discards no acknowledged sample and no machine sequence was ever consumed, with a test asserting the staged database is byte-identical before and after the readiness probe. The target returns a signed readiness receipt containing the generation and each check digest. +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, installs the transfer announcement as the target's own accepted announcement, and advances the floor with the same never-lower rule every other floor write obeys: the floor columns and `floor_announcement_json` move to the transfer announcement's pair only when that pair is lexicographically above the target's retained floor, and otherwise keep the prior floor and its evidence untouched, because a target that manually adopted a lower pair while convergence was still pending has a floor above its accepted announcement, and blindly installing the transfer announcement as floor evidence would lower the anti-replay floor or desynchronize it from `floor_recovery_epoch` (`hub_announcement_json`, `recovery_epoch`, `hub_generation`, and the floor columns all write in this one transaction, so the target's probe endpoint and any later enrollment serve the new authority rather than the old hub's, with a test transferring to a target immediately after a lower-pair manual adoption proving the retained floor survives activation), and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report under the committed proposed generation, but that report check is non-committing by design: the old hub constructs a dedicated probe payload rather than handing over a real outbox batch, the target runs the full ingestion path (signature, admission, inserts, contiguity) inside a transaction it rolls back deliberately, and the result leaves the target as a check digest, never as an `IngestAck` any agent outbox could consume. Nothing about the probe persists, so a draining commit failure that aborts the still-staged target discards no acknowledged sample and no machine sequence was ever consumed, with a test asserting the staged database is byte-identical before and after the readiness probe. The target returns a signed readiness receipt containing the generation and each check digest. Preflight also rejects the transfer outright while an unacknowledged `pending_recovery_key` row exists, with an explicit "acknowledge the recovery key before moving the hub" error and no state change: transferring the fleet signing key to the target while the owner has never confirmed the printable recovery key is saved would silently destroy the only other recovery path this fleet has, so the plan picks the branch of refusing the transfer rather than the branch of carrying the pending key across it. From 1d51717193b2ba17848bc9df390445928f8094a9 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:45:21 -0400 Subject: [PATCH 60/65] docs: close the fifty-sixth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- .../superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 00bb3a1..c5bcd90 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -132,7 +132,10 @@ class SignedReportUpload(BaseModel): signature: str class IngestAck(BaseModel): + upload_id: UUID + payload_digest: str highest_contiguous_sequence: int + signature: str class EnrollmentCode(BaseModel): value: str @@ -713,9 +716,11 @@ async def ingest(self, upload: SignedReportUpload) -> IngestAck: await self.machines.mark_reported(upload.machine_id, highest, upload.latest_activity_at()) await self.receipts.record(upload) await self.events.publish_machine_update(upload.machine_id, upload.sequence_end) - return IngestAck(highest_contiguous_sequence=highest) + return sign_ingest_ack(upload.upload_id, upload.payload_digest, highest) ``` +`sign_ingest_ack` exists because request signing proves the agent to the hub while doing nothing for the response: over plain HTTP an on-path host could suppress the upload entirely and answer with an arbitrarily high watermark, and the agent would mark unsaved samples acknowledged and let retention destroy them. The returned `IngestAck` is therefore signed by the hub machine key over the echoed `upload_id`, `payload_digest`, and `highest_contiguous_sequence`, and the agent verifies that signature against its persisted `runtime_state.hub_signing_public_key` and both echoed identifiers against the upload it actually sent before advancing its local watermark, treating any mismatch or bad signature exactly like a failed upload (retry, nothing acknowledged). Test the forgery: suppress the upload, return a fabricated ack with a huge watermark, and prove the outbox advances nothing. + Two of `mark_reported`'s writes are monotonic maxima, never plain replacements, because oldest-first backfill interleaves with live uploads: `last_seen` stores the greater of its existing value and `upload.latest_activity_at()`, since a historical batch landing after a current one would otherwise flip an actively reporting machine to stale for the entire backfill (Task 6 computes status from this column), and `fleet_machines.outbox_high_water` stores the greater of its existing value and the upload's `outbox_high_water`. The agent stamps that field at upload build time from its persisted `sqlite_sequence` high-water mark for `report_outbox`, riding inside the signed payload like every other field, so the hub durably knows the newest sequence the agent has allocated even when the 100-batch chunk limit means `sequence_end` trails it by thousands, and Task 6's backfill object computes its newest local sequence and pending count from these columns instead of guessing from chunk edges. - [ ] **Step 1: Write outbox and idempotency tests** From 4ce750ad2d5f8be4fb00d25198415715b6bc4211 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:49:15 -0400 Subject: [PATCH 61/65] docs: close the fifty-seventh-round fleet hub finding Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index c5bcd90..0439ea1 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -708,7 +708,8 @@ async def ingest(self, upload: SignedReportUpload) -> IngestAck: await self.state.require_accepting_reports() stored = await self.receipts.stored_ack_or_reject_conflict(upload) if stored is not None: - return stored + return sign_ingest_ack(stored.upload_id, stored.payload_digest, + stored.highest_contiguous_sequence) await self.samples.insert_upload_if_absent(upload) await self.batches.insert_presence_if_absent(upload) await self.gaps.insert_if_absent(upload.machine_id, upload.gaps) @@ -719,7 +720,7 @@ async def ingest(self, upload: SignedReportUpload) -> IngestAck: return sign_ingest_ack(upload.upload_id, upload.payload_digest, highest) ``` -`sign_ingest_ack` exists because request signing proves the agent to the hub while doing nothing for the response: over plain HTTP an on-path host could suppress the upload entirely and answer with an arbitrarily high watermark, and the agent would mark unsaved samples acknowledged and let retention destroy them. The returned `IngestAck` is therefore signed by the hub machine key over the echoed `upload_id`, `payload_digest`, and `highest_contiguous_sequence`, and the agent verifies that signature against its persisted `runtime_state.hub_signing_public_key` and both echoed identifiers against the upload it actually sent before advancing its local watermark, treating any mismatch or bad signature exactly like a failed upload (retry, nothing acknowledged). Test the forgery: suppress the upload, return a fabricated ack with a huge watermark, and prove the outbox advances nothing. +`sign_ingest_ack` exists because request signing proves the agent to the hub while doing nothing for the response: over plain HTTP an on-path host could suppress the upload entirely and answer with an arbitrarily high watermark, and the agent would mark unsaved samples acknowledged and let retention destroy them. The returned `IngestAck` is therefore signed by the hub machine key over the echoed `upload_id`, `payload_digest`, and `highest_contiguous_sequence`, and the agent verifies that signature against its persisted `runtime_state.hub_signing_public_key` and both echoed identifiers against the upload it actually sent before advancing its local watermark, treating any mismatch or bad signature exactly like a failed upload (retry, nothing acknowledged). Test the forgery: suppress the upload, return a fabricated ack with a huge watermark, and prove the outbox advances nothing. Receipts persist only the tuple, never a signature: the identical-retry path reconstructs and re-signs the stored `(upload_id, payload_digest, highest_contiguous_sequence)` with the current hub machine key at response time, keeping the receipt's original watermark, because a signature stored at first ingest would go stale the moment a transfer rotates the authorized hub key and leave the retrying agent unable to verify a perfectly good acknowledgement. Test the lost-response retry across a hub transfer: ingest commits, the response is lost, the fleet transfers, and the retried upload's ack verifies under the new hub's key with the original watermark. Two of `mark_reported`'s writes are monotonic maxima, never plain replacements, because oldest-first backfill interleaves with live uploads: `last_seen` stores the greater of its existing value and `upload.latest_activity_at()`, since a historical batch landing after a current one would otherwise flip an actively reporting machine to stale for the entire backfill (Task 6 computes status from this column), and `fleet_machines.outbox_high_water` stores the greater of its existing value and the upload's `outbox_high_water`. The agent stamps that field at upload build time from its persisted `sqlite_sequence` high-water mark for `report_outbox`, riding inside the signed payload like every other field, so the hub durably knows the newest sequence the agent has allocated even when the 100-batch chunk limit means `sequence_end` trails it by thousands, and Task 6's backfill object computes its newest local sequence and pending count from these columns instead of guessing from chunk edges. From 2aaef5e7e7a66272107612baa30a258cf83cd0f0 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 12:54:31 -0400 Subject: [PATCH 62/65] docs: close the fifty-eighth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 0439ea1..dd171fd 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1215,7 +1215,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the - [ ] **Step 3: Implement local recovery preview and promotion** -Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, `recovery_epoch`, `hub_generation`, and the complete floor pair `floor_recovery_epoch` and `generation_floor` all advance atomically with the role change, the floor columns always matching the pair the evidence announcement proves, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's, the watermark verifier never sees a floor claim its evidence contradicts, and the above-floor rule cannot admit a high-generation announcement from the fenced old epoch through a stale `floor_recovery_epoch`), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview applies the lineage tie-break against announcements, not merely against competing bundles, because an agent can be holding only the losing lineage's bundle: it stores lineage B's replica before convergence, then accepts winning lineage A's announcement, and A dies before its first replication ever reaches it, leaving no A bundle for the bundle-versus-bundle tie-break to prefer. A same-epoch bundle whose lineage loses the tie-break against the lineage of this machine's own accepted announcement, its floor evidence announcement, or any verified probe response is therefore refused automatically, whatever tuple it carries, with preview stating plainly that this bundle belongs to a retired lineage and that promoting it would resurrect a superseded registry (including machines revoked on the winner). One escape hatch matches the manual-adoption pattern: when no winning-lineage bundle exists anywhere, the owner may explicitly confirm promotion from the losing bundle at the machine's loopback recovery page, which is safe against split brain because promotion mints the next epoch above both lineages either way, and the confirmation is only accepting the loser's stale registry as the best data that still exists. Test it: retain B's bundle, accept A's announcement, kill A, and prove preview refuses automatically, states the reason, and proceeds only through the explicit confirmation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, `recovery_epoch`, `hub_generation`, and the complete floor pair `floor_recovery_epoch` and `generation_floor` all advance atomically with the role change, the floor columns always matching the pair the evidence announcement proves, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's, the watermark verifier never sees a floor claim its evidence contradicts, and the above-floor rule cannot admit a high-generation announcement from the fenced old epoch through a stale `floor_recovery_epoch`), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose `lineage_id` is lexicographically smaller wins, a property minted at promotion time and carried verbatim through every same-lineage mint and transfer, so neither minting further generations nor transferring the role to another machine can change it. Different lineage means a differing `lineage_id`. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. From 914269d3b3fffb4772a0128c3a78169253e65367 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 13:02:59 -0400 Subject: [PATCH 63/65] docs: close the fifty-ninth-round fleet hub finding Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index dd171fd..3d0f7cd 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -526,7 +526,7 @@ Run: `.venv/bin/python -m pytest tests/test_owner_passwords.py tests/test_owner_ - [ ] **Step 3: Implement versioned scrypt and opaque sessions** -Use a random 32-byte session token and store only `SHA256(token)`. Return the raw token only in the cookie. Generate a separate CSRF token, store only its digest, and return the raw value from `/api/v2/session/me`. +Use a random 32-byte session token and store only `SHA256(token)`. Return the raw token only in the cookie. Generate a separate CSRF token and store only its digest, which means the server can never hand back the original raw value later, so retrieval is rotation: login returns the raw CSRF token in its response body (a `200` with a JSON body, not a bare `204`), and every `/api/v2/session/me` call mints a fresh CSRF token, atomically replaces `csrf_hash` on the session row, and returns the new raw value, so a fresh page load or a post-restart worker always ends up holding a token whose digest the database currently has. A second tab that rotated the token out from under an older tab costs one round trip, not a login: a state-changing request failing CSRF validation returns a distinguishable `403`, and the client re-reads `/me` once and retries with the fresh token. Test the rotation chain (login token works, `/me` rotates it, the old token's `403` triggers exactly one re-read and the retry succeeds) and test that a hub restart between login and the first state-changing request still validates after a `/me` read. ```python PASSWORD_VERSION = 1 From 30a2d7c642a3c26d2c6852b80c1b87bf20cb1d16 Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 13:07:20 -0400 Subject: [PATCH 64/65] docs: close the sixtieth-round fleet hub findings Co-Authored-By: Claude Fable 5 --- .../superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 3d0f7cd..64624b4 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -515,7 +515,8 @@ def test_login_sets_server_side_session_cookie(hub_client, owner_password): "username": "static", "password": owner_password, }) - assert response.status_code == 204 + assert response.status_code == 200 + assert response.json()["csrf_token"] assert "HttpOnly" in response.headers["set-cookie"] assert "SameSite=strict" in response.headers["set-cookie"] ``` @@ -623,9 +624,9 @@ def canonical_request_bytes( async def enroll_machine(address: str, code: str) -> EnrolledMachine: hub_claim = local_hub_claim() claim = await hub_client.claim(address, code, hub_claim) - record = sign_enrollment_record(hub_claim, claim) - await repository.store_claimed_attempt(record, address, claim.claim_nonce) announcement = current_hub_announcement() + record = sign_enrollment_record(hub_claim, claim, announcement) + await repository.store_claimed_attempt(record, address, claim.claim_nonce) await hub_client.finalize(address, claim.enrollment_id, claim.claim_nonce, record, announcement) return await repository.mark_finalized(claim.enrollment_id) ``` From d21e4c4fe55417224a32c93c31a6f071eb21e2dc Mon Sep 17 00:00:00 2001 From: Static Date: Fri, 14 Aug 2026 13:13:01 -0400 Subject: [PATCH 65/65] docs: close the sixty-first-round fleet hub finding Co-Authored-By: Claude Fable 5 --- docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md index 64624b4..1e4835a 100644 --- a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -1218,7 +1218,7 @@ Start `run_replication_loop()` from the hub lifespan. Every tick, including the Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview applies the lineage tie-break against announcements, not merely against competing bundles, because an agent can be holding only the losing lineage's bundle: it stores lineage B's replica before convergence, then accepts winning lineage A's announcement, and A dies before its first replication ever reaches it, leaving no A bundle for the bundle-versus-bundle tie-break to prefer. A same-epoch bundle whose lineage loses the tie-break against the lineage of this machine's own accepted announcement, its floor evidence announcement, or any verified probe response is therefore refused automatically, whatever tuple it carries, with preview stating plainly that this bundle belongs to a retired lineage and that promoting it would resurrect a superseded registry (including machines revoked on the winner). One escape hatch matches the manual-adoption pattern: when no winning-lineage bundle exists anywhere, the owner may explicitly confirm promotion from the losing bundle at the machine's loopback recovery page, which is safe against split brain because promotion mints the next epoch above both lineages either way, and the confirmation is only accepting the loser's stale registry as the best data that still exists. Test it: retain B's bundle, accept A's announcement, kill A, and prove preview refuses automatically, states the reason, and proceeds only through the explicit confirmation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, `recovery_epoch`, `hub_generation`, and the complete floor pair `floor_recovery_epoch` and `generation_floor` all advance atomically with the role change, the floor columns always matching the pair the evidence announcement proves, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's, the watermark verifier never sees a floor claim its evidence contradicts, and the above-floor rule cannot admit a high-generation announcement from the fenced old epoch through a stale `floor_recovery_epoch`), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. -Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's generation and stored watermark for that machine. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose `lineage_id` is lexicographically smaller wins, a property minted at promotion time and carried verbatim through every same-lineage mint and transfer, so neither minting further generations nor transferring the role to another machine can change it. Different lineage means a differing `lineage_id`. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's authority pair and stored watermark for that machine in a hub-signed response, the same discipline `IngestAck` follows and for the same reason: request signing proves nothing about the response, and an on-path host suppressing the request and echoing a watermark at or above the agent's local acknowledgement would cancel the one-time rewind and strand the recovered hub at watermark zero. The response signs the requesting machine ID, the request's nonce, the hub's current `(recovery_epoch, generation)` pair, and the watermark under the hub machine key, the agent verifies it against its persisted `runtime_state.hub_signing_public_key` before acting, any mismatch or bad signature is a failed request that retries, and a test proves a forged high-watermark response cannot suppress the rewind. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose `lineage_id` is lexicographically smaller wins, a property minted at promotion time and carried verbatim through every same-lineage mint and transfer, so neither minting further generations nor transferring the role to another machine can change it. Different lineage means a differing `lineage_id`. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's.