diff --git a/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md new file mode 100644 index 0000000..1e4835a --- /dev/null +++ b/docs/superpowers/plans/2026-08-13-humwatch-2.1-fleet-hub.md @@ -0,0 +1,1592 @@ +# HumWatch 2.1.0 Fleet Hub Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace HumWatch's fragmented per-PC dashboards, shared browser token, mandatory TLS, and CORS setup with one movable fleet hub, one owner login, one-time PC enrollment, buffered agent reporting, optional HTTPS, and a stable read-only integration boundary for SlipShell. + +**Architecture:** One Python package runs in `unconfigured`, `hub`, `agent`, or temporary `legacy` mode. Every installation keeps a stable machine identity and local sample buffer. The active hub owns owner sessions, the fleet registry, central history, `/api/v2`, and the dashboard. Agents authenticate signed reports with independent device keys. Hub transfer uses signed generations, while dead hub recovery uses encrypted bundles copied to agents. + +**Tech Stack:** Python 3.10+, FastAPI, Uvicorn, SQLite through aiosqlite, Pydantic 2, httpx, `cryptography` Ed25519/X25519/AES-GCM, vanilla JavaScript, Chart.js, pytest, PowerShell, systemd, Tailscale as an optional transport. + +## Global Constraints + +* The approved design is `docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md`. +* The normal hub address is `http://:9100`. +* Ordinary private LAN works without Tailscale. +* Tailscale works when installed, but is never required. +* HTTPS is optional and operator supplied. +* Fresh installs never require a shared bearer token, CORS list, or certificate. +* Browser authentication uses one owner username and password plus server-side sessions. +* Pairing codes are ten-minute, single-use values shown only on the target PC. +* Each machine owns separate Ed25519 signing and X25519 encryption key pairs. +* Each integration owns an independent revocable read-only credential. +* Every machine remains capable of becoming the hub. +* The public fleet API is `/api/v2`. The agent protocol is `/api/internal/v1`. +* Fresh installs disable protected `/api/v1` routes unless legacy compatibility is explicitly enabled. +* Central and local history retain seven days by default. +* Never delete legacy databases, tokens, certificates, or migration archives automatically. +* No task may project beyond roughly 800 lines of production code or 25 touched files. Split before implementation if either estimate is crossed. +* One independent completion review is required for every task. Apply fixes, then run a fresh review against the new HEAD until it returns zero new findings. +* UI and endpoint tasks require running surface evidence plus a deliberate-failure canary before PR creation. +* Before every push run `~/.local/bin/gitleaks git --log-opts="--all --not --remotes"` and stop on findings. +* Before every PR, write `/review-receipt.json` for code tasks with the current HEAD, zero new findings, and the evidence path. + +## Delivery shape and dependency order + +This design is too large for one safe branch. Each numbered task below is one reviewable PR and one bounded implementation session. Merge tasks in order because later interfaces depend on earlier schemas. + +| PR | Deliverable | Production estimate | Maximum file count | +|---:|---|---:|---:| +| 1 | Safe 2.0 database preservation and migration log | 180 lines | 6 | +| 2 | Runtime roles, schemas, and machine or fleet identities | 760 lines | 14 | +| 3 | Owner account, browser sessions, and create-fleet API | 760 lines | 14 | +| 4 | Single-use agent enrollment and signed internal requests | 760 lines | 14 | +| 5 | Local outbox, hub ingestion, acknowledgement, and backfill | 780 lines | 14 | +| 6 | Fleet query API and hub event stream | 650 lines | 12 | +| 7 | First-run setup and owner session UI | 650 lines | 14 | +| 8 | Fleet overview and in-app machine selection | 740 lines | 19 | +| 9 | Fleet administration, listener settings, and optional HTTPS | 680 lines | 16 | +| 10 | Planned hub transfer and signed generation handling | 760 lines | 13 | +| 11 | Encrypted recovery bundles and dead hub promotion | 740 lines | 12 | +| 12 | Read-only integration pairing and SlipShell contract | 680 lines | 16 | +| 13 | 2.0 migration wizard and legacy compatibility retirement | 780 lines | 17 | +| 14 | Windows and Linux installer transition plus duplicate-service detection | 720 lines | 16 | +| 15 | End-to-end acceptance, release docs, and 2.1.0 version | 500 lines | 18 | + +The total program is expected to span 15 PRs. That is deliberate. Combining transfer, recovery, migration, UI, and installers would violate the batch ceiling and make review evidence meaningless. + +## Spec coverage map + +| Approved design area | Implemented by | +|---|---| +| Unconfigured, hub, agent, and legacy runtime states | Tasks 2, 3, 13 | +| Collector boundary and seven-day local history buffer | Tasks 2, 5 | +| Agent transport, signed reporting, offline buffering, and ordered backfill | Tasks 4, 5 | +| Fleet registry, central history, current data, and event stream | Tasks 2, 5, 6 | +| Owner authentication, 30-day browser sessions, CSRF, and recovery password reset | Tasks 3, 11 | +| One-time PC enrollment and independent revocation | Tasks 4, 8 | +| Ordinary LAN, optional Tailscale, explicit listeners, and optional HTTPS | Tasks 9, 13 | +| First-run setup, one dashboard, fleet overview, and in-app machine selection | Tasks 7, 8 | +| Planned hub transfer, higher generation enforcement, and split-brain prevention | Task 10 | +| Encrypted recovery replicas and dead hub promotion | Task 11 | +| Read-only SlipShell integration boundary | Task 12 | +| Protected v1 compatibility and explicit 2.0 upgrade choices | Task 13 | +| Database preservation, rollback archive, and issue #14 | Tasks 1, 13 | +| Duplicate service detection and installer transition | Task 14 | +| Real Windows and Linux acceptance plus release evidence | Task 15 | + +## Shared names and wire rules + +These names are fixed for every task. Change them only by updating this plan and every consumer in the same planning commit. + +```python +class RuntimeRole(str, Enum): + UNCONFIGURED = "unconfigured" + HUB = "hub" + AGENT = "agent" + LEGACY = "legacy" + PENDING_HUB = "pending_hub" # transfer target holds staged or unannounced authority, Task 10 + +class MachineStatus(str, Enum): + ENROLLING = "enrolling" + ONLINE = "online" + OFFLINE = "offline" + REVOKED = "revoked" + MIGRATING = "migrating" + +class MetricValue(BaseModel): + category: str + metric_name: str + value: float | int | str | None + unit: str | None = None + +class ProcessValue(BaseModel): + pid: int + name: str + cpu_percent: float + memory_mb: float + +class GapRange(BaseModel): + sequence_start: int + sequence_end: int + reason: Literal["local-retention", "source-corruption", "recovery"] + reported_at: datetime + +class ReportBatch(BaseModel): + sequence: int + batch_id: UUID + captured_at: datetime + metrics: list[MetricValue] + processes: list[ProcessValue] + +class SignedReportUpload(BaseModel): + fleet_id: UUID + machine_id: UUID + upload_id: UUID + sequence_start: int + sequence_end: int + outbox_high_water: int + batches: list[ReportBatch] + gaps: list[GapRange] = Field(default_factory=list) + payload_digest: str + signature: str + +class IngestAck(BaseModel): + upload_id: UUID + payload_digest: str + highest_contiguous_sequence: int + signature: str + +class EnrollmentCode(BaseModel): + value: str + expires_at: datetime + +class HubClaim(BaseModel): + enrollment_id: UUID + fleet_id: UUID + hub_machine_id: UUID + hub_generation: int + hub_addresses: list[str] + fleet_signing_public_key: str + hub_signing_public_key: str + recovery_public_key: str + +class AgentClaim(BaseModel): + enrollment_id: UUID + claim_nonce: str + machine_id: UUID + hostname: str + signing_public_key: str + encryption_public_key: str + addresses: list[str] + agent_version: str + prior_identity_proof: str | None = None + +class SignedEnrollmentRecord(BaseModel): + enrollment_id: UUID + agent_claim_digest: str + hub_announcement_digest: str + hub_claim: HubClaim + signature: str + +class EnrolledMachine(BaseModel): + machine_id: UUID + status: MachineStatus + +class HubAnnouncement(BaseModel): + fleet_id: UUID + hub_machine_id: UUID + lineage_id: UUID + recovery_epoch: int + generation: int + addresses: list[str] + hub_signing_public_key: str + issued_at: datetime + signature: str +``` + +`recovery_epoch` exists because generations alone cannot fence a dead hub that still retains the fleet signing key: a hub that locally installed an undistributed generation, died, and was recovered past can restart, notice an endpoint change, and mint what verifies as a legitimate generation above everything the recovered fleet has accepted, redirecting agents back to its obsolete registry. Every announcement ordering rule in this plan therefore compares the fleet-signed pair `(recovery_epoch, generation)` lexicographically: the strict above-floor delivery rule, the recovery probe's highest-generation selection, the promotion formula, and the convergence rule all order by the pair, and "higher generation" throughout this document is shorthand for a higher pair. Ordinary minting (endpoint changes, transfers, convergence bumps) carries the current epoch forward and increments the generation. Exactly one path increments the epoch: dead-hub promotion in Task 11, which mints at one above the highest epoch seen in the bundle and every verified probe response. A former hub restarting from retained state signs its old epoch, so every remote path rejects its announcements no matter how high their generation climbs, and the recovered lineage is fenced from the dead one without rotating the fleet key. Two partitioned machines can still recover the same fleet into the same fresh epoch, so same-epoch conflicts between distinct lineages resolve through Task 11's deterministic tie-break rather than through generation racing. Lineage identity is its own field because `hub_machine_id` does not survive a transfer: `lineage_id` is a UUID minted fresh at fleet creation and at each promotion, carried verbatim through every same-lineage mint and every transfer, so the tie-break stays pinned to the promotion event no matter which machine currently holds the role, and transfer timing can never change the winner. The persisted authority floor is the same kind of pair (`floor_recovery_epoch`, `generation_floor` in the Task 2 schema), compared lexicographically everywhere it is read, and promotion restarts the generation at 1 in its fresh epoch, which the pair rule makes strictly higher than every lower-epoch value however large. The loopback manual adoption flow stays the only path that may accept a non-higher pair, unchanged in scope. + +Internal signed requests use these headers: + +```text +X-HumWatch-Machine: +X-HumWatch-Fleet: +X-HumWatch-Timestamp: +X-HumWatch-Nonce: +X-HumWatch-Signature: +``` + +`SignedHeaders` is the typed mapping containing those five headers. The signed bytes are `METHOD + "\n" + PATH + "\n" + FLEET_ID + "\n" + MACHINE_ID + "\n" + SHA256(BODY) + "\n" + TIMESTAMP + "\n" + NONCE`. The hub rejects timestamps beyond 120 seconds, reused nonces, unknown machines, revoked machines, relabeled fleet or machine headers, and invalid signatures. + +Agents authenticate the hub's side of this envelope with `hub_signing_public_key`: enrollment finalize persists the key carried on `HubClaim`, and every later fleet-signed `HubAnnouncement` the agent accepts (Task 10 transfer, Task 11 recovery) carries the current hub's `hub_signing_public_key` too, so the agent's authorized hub key rotates automatically the moment a higher-generation announcement passes fleet-key verification, with no separate distribution step. Inbound hub-originated signed requests to an agent (recovery replica delivery and every other push) are verified against that persisted key rather than trusting the claimed machine ID alone. Announcement delivery is the one deliberate exception, because it is the rotation mechanism itself: during a transfer or recovery the new hub signs with its own machine key, which no agent has persisted yet, and requiring the old key would make redirection impossible. The announcement payload is fleet-signed and self-authenticating, so the delivery route verifies the fleet signature and the generation rule on the payload alone, and acceptance atomically rotates the persisted `hub_signing_public_key` to the announcement's, after which every other inbound push from the new hub verifies normally. A forged delivery envelope around a legitimate announcement gains nothing: the payload either passes fleet-key verification or is discarded. + +Browser mutations send the CSRF token in `X-HumWatch-CSRF`. Session cookies are named `humwatch_session`. + +--- + +### Task 1: Preserve the real 2.0 database before any fleet migration + +**Purpose:** Close issue #14 and make data preservation fail visibly before 2.1 adds more migration paths. + +**Files:** +* Modify: `agent/config.py` +* Modify: `agent/migrations.py` +* Modify: `agent/main.py` +* Modify: `installer/service-setup.ps1` +* Modify: `tests/test_database_migration.py` +* Modify: `tests/test_windows_service_security.py` + +**Interfaces:** +* Produces: `HumWatchConfig.legacy_db_path: Optional[str]` +* Produces: `HumWatchConfig.resolved_legacy_db_path -> Optional[Path]` +* Produces: `MigrationResult(source: Path | None, target: Path, status: str, detail: str, archive_manifest: Path | None)` +* Produces: `migrate_legacy_database(config: HumWatchConfig) -> MigrationResult` + +- [ ] **Step 1: Pin the Windows installer failure with a test** + +Add a test that gives the runtime an absolute destination plus an explicit live SQLite source. Put `machine_info` and `metrics` in the database, enable WAL, commit one more metric through the WAL connection, and leave the connection open while migration runs: + +```python +def test_installer_absolute_destination_migrates_explicit_legacy_source(tmp_path): + legacy = tmp_path / "legacy" / "humwatch.db" + target = tmp_path / "runtime" / "humwatch.db" + legacy.parent.mkdir() + source = sqlite3.connect(legacy) + source.execute("PRAGMA journal_mode=WAL") + source.execute("CREATE TABLE machine_info (machine_id TEXT PRIMARY KEY)") + source.execute("CREATE TABLE metrics (timestamp TEXT, metric_name TEXT, value REAL)") + source.execute("INSERT INTO machine_info VALUES ('machine-a')") + source.execute("INSERT INTO metrics VALUES ('2026-08-13T12:00:00Z', 'cpu', 42)") + source.commit() + source.execute("INSERT INTO metrics VALUES ('2026-08-13T12:01:00Z', 'cpu', 43)") + source.commit() + config = HumWatchConfig( + db_path=str(target), + legacy_db_path=str(legacy), + data_root=str(tmp_path / "runtime"), + ) + + result = migrate_legacy_database(config) + + assert result.status == "migrated" + migrated = sqlite3.connect(target) + assert migrated.execute("PRAGMA quick_check").fetchone() == ("ok",) + assert migrated.execute("SELECT machine_id FROM machine_info").fetchone() == ("machine-a",) + assert migrated.execute("SELECT COUNT(*) FROM metrics").fetchone() == (2,) + assert result.archive_manifest.is_file() + assert legacy.is_file() +``` + +- [ ] **Step 2: Run the focused tests and confirm the old absolute-path guard fails** + +Run: `.venv/bin/python -m pytest tests/test_database_migration.py -q` + +Expected: FAIL because `legacy_db_path` and `MigrationResult` do not exist. + +- [ ] **Step 3: Implement explicit source selection and verifiable preservation** + +Add `legacy_db_path` to config and use it before the old inferred path: + +```python +@dataclass(frozen=True) +class MigrationResult: + source: Path | None + target: Path + status: str + detail: str + archive_manifest: Path | None + +def legacy_database_path(config: HumWatchConfig) -> Optional[Path]: + if config.resolved_legacy_db_path is not None: + return config.resolved_legacy_db_path + if Path(config.db_path).is_absolute(): + return None + return PROJECT_ROOT / Path(config.db_path) +``` + +Open the source read-only, use `sqlite3.Connection.backup()` into a staging file so committed WAL pages are included, then run `PRAGMA quick_check`, verify the required 2.0 tables, verify at least one `machine_info` row, and compare source and staged row counts. Only then rename the staging database atomically to the target, and the proof protocol brackets that rename so no crash window can strand a completed copy without evidence: before the rename, write and fsync the manifest with status `pending`, carrying the staged file's digest and verified row counts, and after the rename rewrite it to its final status. + +The manifest lives at `resolved_data_root / "migration-archive" / / "manifest.json"` with source and target paths, SHA256 digests, verified table row counts, status, and time. Return and log `migrated`, `source_missing`, `target_exists`, `same_path`, or `failed`. `target_exists` is not automatically safe to continue on, because the absolute-path bug this task exists for can have already created a newer empty or partial database at the target while the explicit legacy source still holds the missing history: a successful migration records itself in the target archive manifest, and `target_exists` proceeds only when a manifest proves this target came from a prior successful migration of this source. A `pending` manifest is proof too, once verified against the disk: a crash between the rename and the manifest finalize leaves exactly that pair behind, so `target_exists` with a `pending` manifest for this source recomputes the target's digest and row counts, and a match finalizes the manifest and continues while a mismatch is treated as no proof at all. A pre-existing target with no such proof while an explicit source exists raises `MigrationSafetyError` with both paths, their row counts, and the manual merge guidance in the message, stopping visibly instead of quietly adopting the empty database and leaving issue #14 alive. Test the bug-shaped case: an empty target, a populated explicit source, and no manifest must refuse to start. Test the interruption window too: kill the migration after the rename but before the manifest finalize, and the next startup must verify the target against the `pending` manifest, finalize it, and start normally instead of raising. Never unlink or rewrite the source database, its WAL or shared-memory sidecars, or an existing archive. Startup must raise `MigrationSafetyError` when an explicit source exists but backup or proof fails. + +- [ ] **Step 4: Make the installer pass the old path explicitly** + +Set `HUMWATCH_LEGACY_DB` to `C:\HumWatch\humwatch.db` only when that file exists. Preserve existing upgrade behavior when it does not. + +- [ ] **Step 5: Verify the focused migration and installer contracts** + +Run: `.venv/bin/python -m pytest tests/test_database_migration.py tests/test_windows_service_security.py -q` + +Expected: PASS with explicit assertions for every decline status, WAL continuity, manifest proof, source preservation, and no silent branch. + +- [ ] **Step 6: Run the full task gate and commit** + +Run: `.venv/bin/python -m pytest -q` + +Commit: + +```bash +git add agent/config.py agent/migrations.py agent/main.py installer/service-setup.ps1 tests/test_database_migration.py tests/test_windows_service_security.py +git commit -m "fix: preserve Windows history during upgrades" +``` + +Use issue #14 as the PR close target. + +--- + +### Task 2: Add runtime roles, fleet schema, and protected identities + +**Purpose:** Establish durable types and storage without changing the current dashboard behavior. + +**Files:** +* Create: `agent/fleet/__init__.py` +* Create: `agent/fleet/models.py` +* Create: `agent/fleet/schema.py` +* Create: `agent/fleet/identity.py` +* Create: `agent/fleet/state.py` +* Modify: `agent/config.py` +* Modify: `agent/database.py` +* Modify: `agent/main.py` +* Modify: `config.json` +* Modify: `tests/conftest.py` +* Create: `tests/test_fleet_schema.py` +* Create: `tests/test_fleet_identity.py` +* Create: `tests/test_runtime_roles.py` +* Modify: `tests/test_security_config.py` + +**Interfaces:** +* Produces: `RuntimeRole`, `MachineStatus`, `MetricValue`, `ProcessValue`, `ReportBatch`, `HubAnnouncement` +* Produces: `MachineIdentity.load_or_create(root: Path) -> MachineIdentity` +* Produces: `FleetIdentity.create(root: Path) -> tuple[FleetIdentity, str]` +* Produces: `FleetStateRepository.get_runtime_state() -> RuntimeState` +* Produces: `async FleetStateRepository.set_role(role: RuntimeRole) -> None` +* Produces: `transaction(db: aiosqlite.Connection) -> AsyncContextManager[None]` +* Produces: `detect_runtime_role(config: HumWatchConfig) -> RuntimeRole` +* Produces: `RoleServices.reconcile() -> None` (starts and stops role services on every persisted role change, no restart required) + +- [ ] **Step 1: Write schema and identity failure tests** + +Cover stable UUIDs across restart, mode defaulting, key file permissions, signature verification, X25519 round trip, and seven-day default retention: + +```python +async def test_new_database_starts_unconfigured(db): + state = await FleetStateRepository(db).get_runtime_state() + assert state.role is RuntimeRole.UNCONFIGURED + +def test_machine_identity_survives_restart(tmp_path): + first = MachineIdentity.load_or_create(tmp_path) + second = MachineIdentity.load_or_create(tmp_path) + assert first.machine_id == second.machine_id + assert first.sign(b"sample") == second.sign(b"sample") +``` + +- [ ] **Step 2: Run the new test files and confirm missing modules fail** + +Run: `.venv/bin/python -m pytest tests/test_fleet_schema.py tests/test_fleet_identity.py tests/test_runtime_roles.py -q` + +Expected: collection FAIL for missing `agent.fleet` modules. + +- [ ] **Step 3: Add the v2 tables** + +`agent/fleet/schema.py` owns idempotent SQL for: + +```sql +runtime_state(id, role, fleet_id, machine_id, hub_machine_id, recovery_epoch, hub_generation, floor_recovery_epoch, generation_floor, hub_announcement_json, floor_announcement_json, hub_signing_public_key, fleet_signing_public_key, recovery_public_key, transfer_state, transfer_id, recovery_bundle_revision, legacy_api_enabled, updated_at) +fleet_machines(machine_id, display_name, hostname, signing_public_key, encryption_public_key, status, addresses_json, enrolled_at, last_seen, revoked_at, last_acked_sequence, outbox_high_water) +owner_account(id, username, password_hash, password_salt, password_version, scrypt_n, scrypt_r, scrypt_p, updated_at) +owner_sessions(session_hash, csrf_hash, created_at, expires_at, last_seen_at, source_label) +fleet_settings(id, listeners_json, https_cert_path, https_key_path, allow_public_listener, updated_at) +fleet_metrics(machine_id, sequence, timestamp, category, metric_name, value, unit) +fleet_processes(machine_id, sequence, timestamp, pid, name, cpu_percent, memory_mb) +report_outbox(sequence, batch_id, captured_at, payload_json, acknowledged_at) +report_batches(machine_id, sequence, batch_id, batch_digest, captured_at, accepted_at) +report_receipts(machine_id, upload_id, payload_digest, sequence_start, sequence_end, highest_contiguous_sequence, accepted_at) +data_gaps(machine_id, sequence_start, sequence_end, reason, reported_at) +used_nonces(principal_type, principal_id, nonce_hash, expires_at) +enrollment_attempts(enrollment_id, machine_id, agent_claim_digest, agent_claim_json, hub_claim_json, signed_record_json, claim_nonce_ciphertext, address, state, created_at, finalized_at) +agent_enrollment_state(enrollment_id, agent_claim_json, hub_claim_digest, record_digest, finalize_result_json, state, created_at) +integration_clients(integration_id, name, public_key, scopes_json, created_at, revoked_at) +integration_pairing_codes(pairing_code_id, code_hash, approved_name, approved_scopes_json, created_at, expires_at, consumed_at, integration_id, client_public_key) +hub_transfers(transfer_id, target_machine_id, announcement_json, state, created_at, draining_at, retired_at) +hub_transfer_aborts(transfer_id, announcement_generation, announcement_digest, abort_json, state, attempt_count, next_attempt_at, acknowledged_at) +pending_hub_authority(transfer_id, announcement_generation, announcement_digest, announcement_json, staging_hub_machine_id, staging_hub_public_key, staged_snapshot_path, staged_key_path, prior_settings_json, state, created_at, activated_at) +hub_publication_obligations(announcement_generation, announcement_digest, announcement_json, state, created_at, activated_at) +hub_announcement_deliveries(announcement_generation, machine_id, announcement_digest, state, attempt_count, next_attempt_at, acknowledged_at) +pending_recovery_key(id, key_path, created_at) +staged_key_journal(journal_id, purpose, staged_path, created_at, finalized_at) +recovery_bundles(lineage_id, recovery_epoch, generation, bundle_revision, created_at, encrypted_payload, fleet_signature) +recovery_deliveries(lineage_id, recovery_epoch, generation, bundle_revision, machine_id, bundle_digest, state, attempt_count, next_attempt_at, acknowledged_at) +``` + +Add unique keys on `(machine_id, sequence, category, metric_name)`, `(machine_id, sequence, pid)`, `(machine_id, sequence)` for batch presence, `(machine_id, upload_id)`, `enrollment_attempts.enrollment_id`, `agent_enrollment_state.enrollment_id`, `(machine_id, sequence_start, sequence_end)` for `data_gaps` (matching the insertion identity `insert_if_absent` checks, so a disclosed gap resent under a fresh upload ID cannot duplicate availability ranges), `(principal_type, principal_id, nonce_hash)` for `used_nonces`, `(transfer_id, announcement_digest)` for `hub_transfer_aborts`, `pending_hub_authority.announcement_digest`, `integration_pairing_codes.pairing_code_id`, `(lineage_id, recovery_epoch, generation, bundle_revision)` for recovery bundles, `(lineage_id, recovery_epoch, generation, bundle_revision, machine_id)` for recovery delivery, and `(announcement_generation, machine_id, announcement_digest)` for publication delivery. + +`enrollment_attempts.agent_claim_json` holds the complete claim Task 4 promises to reconstruct retries from, not merely its digest, and `agent_enrollment_state` is the agent-side mirror: the exact claim bytes the agent sent, the canonical digest of the hub claim it accepted at claim time, the record digest, and the stored finalize result that the digest-scoped `/finalize` retry serves, so either process restarting between claim and acknowledged finalize resumes from durable rows even when mutable inputs like hostname or detected addresses have since changed. The stored `hub_claim_digest` is what keeps finalization bound to the hub claim the pairing code authenticated across that restart: the signed record finalize presents must embed a hub claim whose canonical digest matches the stored one, a record carrying any other hub claim under the same enrollment ID is rejected as changed bytes, and a restart test proves it (claim, restart the agent, finalize with a substituted hub claim, expect rejection, then finalize with the original and expect success). `used_nonces.principal_type` distinguishes `machine` from `integration` (Task 12): the header envelope reuses one machine-shaped slot for both principal kinds, so replay protection keys on the principal's own ID within its own type rather than assuming every signer is a fleet machine. + +`report_outbox.sequence` is declared `INTEGER PRIMARY KEY AUTOINCREMENT`, not a plain rowid or a `MAX(sequence) + 1` allocator: Task 5 purges acknowledged rows after seven days, so a fully caught-up agent can hold an empty outbox, and a reusable allocator would restart at 1 below the hub's watermark, leaving every later sample treated as an already-acknowledged old sequence. AUTOINCREMENT persists the high-water mark in `sqlite_sequence` independently of surviving rows, so allocation continues monotonically across a full purge and across restarts. `report_batches` records one row for every batch sequence the hub accepts, including a batch whose metric and process lists are both empty, so acknowledgement contiguity never depends on sample rows existing. The row also pins the batch's identity: `batch_id` and the canonical `batch_digest`, and ingestion compares them before accepting a repeated sequence, because receipts are keyed by upload ID and a fresh upload ID carrying an already-accepted sequence with different bytes would otherwise merge new samples into settled history and still acknowledge. An identical repeat is an idempotent no-op, a conflicting repeat rejects the upload, and both cases have tests in Task 5. `staged_key_journal` is the durable reference for fleet signing keys staged outside the identity directory: Create Fleet (Task 3), transfer activation on the target (Task 10), and dead-hub promotion (Task 11) insert a row inside their commit transaction, startup repairs any unfinalized row, and finalization stamps `finalized_at`. The repair is idempotent across both crash windows: when the staged path still exists it completes the move, and when the staged path is already gone it verifies the key installed in the identity location against the stored fleet public key and stamps the row finalized, so a crash after the move but before the stamp never strands a committed hub behind a journal row pointing at nothing. A missing staged path with no matching installed key is the one unrepairable state and fails startup loudly. The journal also cannot cover the window before it exists: a transfer import or recovery promotion decrypts the fleet key to its staging path before the transaction that records any reference commits, and a crash there would otherwise leave an untracked, directly usable plaintext key on a machine that restarts as a plain agent. Every staged key therefore lives only under the dedicated `resolved_data_root / "staging"` directory, and startup scrubs that directory: any file not referenced by an uncommitted-repair candidate (an unfinalized `staged_key_journal` row or a live `pending_hub_authority` row) is deleted before any role service starts, so the pre-commit crash window ends with the plaintext destroyed rather than orphaned. Test both journal windows for all three writers (creation, transfer activation, promotion), plus the pre-commit window for both stagers (kill after the key file is written but before the commit, restart, and prove the scrub removed it). `hub_transfers` and `hub_publication_obligations` retain one durable row per transfer or announcement digest. Check constraints require ordered sequence ranges, nonnegative attempts, known state values, and a positive recovery bundle revision. + +`runtime_state.transfer_state` (values `accepting|frozen|draining`, defaulting to `accepting`) is the single column every admission predicate reads on the source hub side of a transfer: `require_accepting_reports()` (Task 5) and every fleet-mutation route reject while it is anything but `accepting`, and the freeze, drain, and lift transactions (Task 10) are what write it. `runtime_state.transfer_id` names the current `hub_transfers` row so a restarted process finds the transfer it was mid-step on. `runtime_state.hub_announcement_json` stores the exact fleet-signed announcement bytes this machine last accepted (signature and `issued_at` included), written atomically with acceptance: the generation and hub machine ID alone cannot reproduce that object without the fleet private key, and Task 11's recovery probe must be able to serve it verbatim after any restart. `runtime_state.fleet_signing_public_key` and `runtime_state.recovery_public_key` are the machine's durable trust anchors, written in the same transaction that installs fleet state (create-fleet on a hub, enrollment finalize on an agent, Task 4) and read back at every startup: the fleet key is what authenticates every announcement, recovery bundle, signed abort, and readiness receipt this machine will ever verify, and `hub_announcement_json` is only as trustworthy as the key that checked it, so neither value may live solely in process memory. The `runtime_state` pair `(floor_recovery_epoch, generation_floor)` is the machine's authority floor, the lexicographic maximum of every `(recovery_epoch, generation)` pair this machine has ever accepted, persisted as a pair together with its evidence announcement (whose own epoch and generation must equal the persisted pair) precisely because a scalar floor cannot survive an epoch transition: a machine moving from `(1, 100)` to a recovered `(2, 2)` has a floor of `(2, 2)`, not `100`, and comparing raw generations across epochs would either inflate the recovered lineage from obsolete evidence or wedge recovery at the storage ceiling. No path lowers the floor pair: `hub_generation` can drop below it through Task 11's loopback manual adoption, but remote announcement delivery admits only pairs lexicographically above the floor, so a dead hub's old announcement cannot replay an adopted machine back to the dead lineage no matter how high its generation. The floor is always provable, never merely asserted: a floor only ever rises by accepting a fleet-signed announcement, and `floor_announcement_json` retains the exact announcement whose generation equals the current floor. Ordinary acceptance of a higher generation updates both announcement columns together, while manual adoption of a lower one updates only `hub_announcement_json`, keeping the floor's evidence intact. Any protocol that consumes a reported floor (Task 11's recovery probe and watermark convergence) verifies the accompanying evidence announcement with the fleet key and uses its generation, so a compromised machine cannot advance fleet authority by signing an arbitrary scalar the fleet key never endorsed. `hub_transfers` itself keeps the transfer's own journal detail (target machine, announcement, timestamps) rather than duplicating it onto `runtime_state`. `hub_transfer_aborts` is the durable queue for a signed abort the old hub could not deliver inline: it carries the same retry shape as `recovery_deliveries` (`attempt_count`, `next_attempt_at`, `acknowledged_at`), is resumed from startup the same way, and is keyed uniquely by `(transfer_id, announcement_digest)` so a transfer that fails staging twice cannot queue two aborts for the same announcement. `pending_hub_authority` is the target's mirror: it is the durable home for staged or unannounced hub authority a transfer or recovery target has not yet activated, one row per announcement digest, holding the staged snapshot and key paths so a target restart before activation resumes from this table instead of guessing whether it is a full hub, a plain agent, or something in between. The target's `runtime_state.role` is `RuntimeRole.PENDING_HUB` for exactly as long as a `pending_hub_authority` row is not yet `activated`, and a restart in that role serves only the internal activation, abort, and status routes, the loopback recovery page, and the pre-activation readiness surface Task 10 probes: the static login page, session login (readiness proves an invalid login gets the generic `401`), and signed report ingestion at the committed proposed generation. The loopback recovery page stays reachable because the old hub can die between `commit_unannounced_hub` and its activation request, leaving no machine able to authorize activation or abort: its Cancel Pending Authority action runs the abort cleanup locally for an authority whose state was never `activated` (staged files deleted, prior settings restored, role back to `AGENT`), refuses to touch an activated one, and frees the machine to run ordinary dead-hub recovery from its replicated bundle, which a two-machine fleet's sole survivor depends on. Task 10's tests cover the stranded-target cancel followed by a successful promotion. That readiness surface is reachable in practice only by the old hub's probe, because no agent redirects to a new hub before accepting an announcement, and the announcement is only published after activation. A pending hub never serves the authenticated dashboard data routes or any fleet mutation route. + +It also owns the transaction helper used by every fleet repository: + +The application shares one `aiosqlite` connection, and aiosqlite serializes statements but not transaction spans, so two coroutines could interleave `BEGIN IMMEDIATE` calls. The helper therefore holds one per-connection `asyncio.Lock` across the whole transaction: + +```python +_transaction_locks: WeakKeyDictionary = WeakKeyDictionary() + +@asynccontextmanager +async def transaction(db: aiosqlite.Connection): + lock = _transaction_locks.setdefault(db, asyncio.Lock()) + async with lock: + await db.execute("BEGIN IMMEDIATE") + try: + yield + except BaseException: + await db.rollback() + raise + else: + await db.commit() +``` + +Add a test that runs overlapping transactions from concurrent coroutines on one connection and proves they serialize with no `cannot start a transaction within a transaction` error and no lost write. + +- [ ] **Step 4: Implement machine and fleet identity storage** + +Use raw private key bytes stored below `resolved_data_root / "identity"`. Create files atomically with mode `0600` on POSIX. Windows ACL enforcement remains the installer task. + +```python +@dataclass(frozen=True) +class MachineIdentity: + machine_id: UUID + signing: Ed25519PrivateKey + encryption: X25519PrivateKey + + def sign(self, payload: bytes) -> bytes: + return self.signing.sign(payload) +``` + +`FleetIdentity.create()` returns the stored fleet signing identity plus a printable grouped base64url recovery private key with a checksum. Persist only the recovery public key. + +- [ ] **Step 5: Detect the role before applying any network security gate** + +Move role detection ahead of the current unconditional `validate_security_config()` calls in `agent/main.py`. Existing databases with an auth token start as `legacy`. Fresh databases start as `unconfigured`. Unconfigured mode starts its setup surface on `127.0.0.1` and does not require a token or certificate. Task 4 adds a loopback-selected, restricted private enrollment listener before its hub-orchestrated claim runs. That listener may expose only `/api/internal/v1/enrollment/claim` and `/finalize`, never the setup page, pairing-code endpoint, dashboard, or any v1 route. Legacy mode keeps the existing token, TLS, and CORS validation exactly. Hub and agent modes use loopback or explicitly stored private listeners until Task 9 adds the owner-managed settings surface, and Task 3's Create Fleet stores and binds the first private hub listener inside its transaction, so a new hub is reachable for enrollment claims and agent reports from the moment the fleet exists rather than waiting for Task 9. No role may fall back to `0.0.0.0`. + +`hub` starts collector, retention, and hub services. `agent` starts collector, retention, and agent transport. Role services are not a startup-only branch: the lifespan owns a `RoleServices` supervisor whose `reconcile()` starts and stops each role's services against the currently persisted role, and `FleetStateRepository.set_role()` triggers it after every in-process role change, because Create Fleet, enrollment finalize, transfer activation, abort back to agent, promotion, and Rejoin all change the role without a process restart. A fresh process that started `unconfigured` therefore begins collecting locally the moment Create Fleet commits, and begins collecting and reporting the moment finalize commits, with a no-restart test for each flow (Tasks 3 and 4). Until those services exist, role branches log the selected mode and start only the services already implemented for that role. Add startup tests proving a fresh config with no token and no TLS reaches the unconfigured app, while an unsafe legacy config still fails closed. + +Set both `HumWatchConfig.retention_days` and the checked-in `config.json` value to `7` here so later outbox and hub retention tests share the approved default. + +- [ ] **Step 6: Verify schema idempotency and the full suite** + +Run: + +```bash +.venv/bin/python -m pytest tests/test_fleet_schema.py tests/test_fleet_identity.py tests/test_runtime_roles.py tests/test_security_config.py -q +.venv/bin/python -m pytest -q +``` + +- [ ] **Step 7: Commit the foundation** + +```bash +git add agent/fleet agent/config.py agent/database.py agent/main.py config.json tests/conftest.py tests/test_fleet_schema.py tests/test_fleet_identity.py tests/test_runtime_roles.py tests/test_security_config.py +git commit -m "feat: add fleet runtime foundation" +``` + +--- + +### Task 3: Create the owner account and server-side browser sessions + +**Purpose:** Replace the normal bearer-token prompt for hub mode and create a fleet from local setup. + +**Files:** +* Create: `agent/security/passwords.py` +* Create: `agent/security/sessions.py` +* Create: `agent/fleet/setup.py` +* Create: `agent/fleet/listeners.py` +* Create: `agent/routes/v2_session.py` +* Create: `agent/routes/v2_setup.py` +* Modify: `agent/fleet/schema.py` +* Modify: `agent/security/limits.py` +* Modify: `agent/main.py` +* Modify: `tests/conftest.py` +* Create: `tests/test_owner_passwords.py` +* Create: `tests/test_owner_sessions.py` +* Create: `tests/test_create_fleet.py` +* Create: `tests/test_v2_csrf.py` +* Create: `tests/test_login_throttle.py` +* Modify: `tests/test_api_auth.py` + +**Interfaces:** +* Consumes: `FleetIdentity.create`, `MachineIdentity.load_or_create`, `FleetStateRepository` +* Produces: `hash_password(password: str) -> PasswordRecord` +* Produces: `verify_password(password: str, record: PasswordRecord) -> bool` +* Produces: `require_owner_session(request: Request) -> OwnerSession` +* Produces: `require_csrf(request: Request, session: OwnerSession) -> None` +* Produces: `ListenerManager.attach(held_socket) -> None` and `ListenerManager.detach(listener) -> None` +* Produces: `POST /api/v2/setup/create-fleet` +* Produces: `GET /api/v2/setup/recovery-key`, `POST /api/v2/setup/recovery-key/ack` +* Produces: `POST /api/v2/session/login`, `GET /me`, `POST /logout`, `GET /sessions`, `DELETE /sessions/{session_hash}` + +- [ ] **Step 1: Write failing password, session, and setup tests** + +Test `hashlib.scrypt`, parameter persistence, rehash after successful login from an older parameter version, generic login failures, a 30-day expiry, `HttpOnly`, `SameSite=Strict`, conditional `Secure`, CSRF rejection, logout, individual revocation, all-session revocation, and loopback-only fleet creation. Also test that create-fleet stores and binds the requested private hub listener, rejects creation with no private address, that the recovery key remains retrievable through the pending-setup record after a simulated lost response, and that acknowledgement deletes it permanently. + +```python +def test_login_sets_server_side_session_cookie(hub_client, owner_password): + response = hub_client.post("/api/v2/session/login", json={ + "username": "static", + "password": owner_password, + }) + assert response.status_code == 200 + assert response.json()["csrf_token"] + assert "HttpOnly" in response.headers["set-cookie"] + assert "SameSite=strict" in response.headers["set-cookie"] +``` + +- [ ] **Step 2: Run the focused tests and confirm missing endpoints fail** + +Run: `.venv/bin/python -m pytest tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py -q` + +- [ ] **Step 3: Implement versioned scrypt and opaque sessions** + +Use a random 32-byte session token and store only `SHA256(token)`. Return the raw token only in the cookie. Generate a separate CSRF token and store only its digest, which means the server can never hand back the original raw value later, so retrieval is rotation: login returns the raw CSRF token in its response body (a `200` with a JSON body, not a bare `204`), and every `/api/v2/session/me` call mints a fresh CSRF token, atomically replaces `csrf_hash` on the session row, and returns the new raw value, so a fresh page load or a post-restart worker always ends up holding a token whose digest the database currently has. A second tab that rotated the token out from under an older tab costs one round trip, not a login: a state-changing request failing CSRF validation returns a distinguishable `403`, and the client re-reads `/me` once and retries with the fresh token. Test the rotation chain (login token works, `/me` rotates it, the old token's `403` triggers exactly one re-read and the retry succeeds) and test that a hub restart between login and the first state-changing request still validates after a `/me` read. + +```python +PASSWORD_VERSION = 1 +SCRYPT_N_CHOICES = (1 << 14, 1 << 15, 1 << 16) +SCRYPT_R = 8 +SCRYPT_P = 1 +SESSION_TTL = timedelta(days=30) +``` + +On first fleet creation, benchmark the allowed `N` choices with an injected clock and choose the strongest value that stays at or below 200 ms and 64 MiB. Store `password_version`, `N`, `r`, and `p` with the hash. Do not calibrate on every startup. On a successful login, rehash inside a transaction when the stored version or parameters are below the current policy. Tests use an injected deterministic calibrator and never depend on workstation speed. + +- [ ] **Step 4: Add two-dimensional login throttling** + +Keep the existing generic v1 request limiter. Add a login-specific limiter with three counters: 5 attempts per username and source pair per 60 seconds, 15 attempts per username across all sources per 5 minutes, and 30 attempts per source across all usernames per 5 minutes. Normalize usernames before keying, hash stored counter keys, return the same public failure for unknown users and bad passwords, and add tests for one username across multiple addresses plus many usernames from one address. + +- [ ] **Step 5: Implement create-fleet as one transaction** + +Validate that role is `unconfigured`, request source is loopback, username is nonempty, and password is at least 12 characters. The request also names the first hub listener, chosen from the machine's detected non-loopback private addresses and defaulting to the primary private address with the current port. Reject creation when no private address exists rather than silently creating an unreachable hub. Bind and hold the requested socket before the transaction commits: a bind failure (the address disappeared, the port is taken) returns a visible error with the installation still `unconfigured` and freely retryable, instead of committing a hub role that can never come up on its stored listener. Only with the socket held does the transaction create fleet identity, owner record, self machine row, the stored hub listener, the fleet-signed `HubAnnouncement` built from that listener at recovery epoch 1, generation 1, with a freshly minted `lineage_id` (persisted into `runtime_state.hub_announcement_json`, so the recovery probe and enrollment delivery below have a real signed object from the fleet's first minute), and runtime role `hub`, after which the held socket serves enrollment claims and agent reports without a restart. `FleetIdentity.create()` stages its key files in a temporary directory, and the create-fleet transaction records that staged path in a `staged_key_journal` row (Task 2 schema) alongside everything else it commits. Only after the commit does finalization move the staged files into the identity location and stamp the journal row finalized. A rollback deletes the staged files and the journal row rolls back with the transaction, so a commit failure leaves neither stale identity files nor a vanished recovery key behind a still-`unconfigured` database. A crash between the commit and the move is not retryable through create-fleet (the role is already `hub`), so startup repairs it instead: any unfinalized journal row has its staged files moved into place before the hub serves anything, the same repair path Task 11's promotion relies on. Binding and holding the socket does not by itself make the running server accept requests on it: the process is already serving the loopback setup surface through its original listener. `agent/fleet/listeners.py` therefore provides a `ListenerManager` owned by the application lifespan. `attach(held_socket)` starts serving the same ASGI app on an already-bound socket from inside the running event loop (one additional in-process server task per listener) and `detach` stops one. Every attached server task is configured with lifespan disabled: the process's original server already ran the app's startup and owns the shared database connection, the collectors, and the retention and delivery loops, and a second server invocation running the lifespan again would reopen or replace that connection and start duplicate copies of every background task the moment the fleet is created. Attached servers only serve requests, and the one lifecycle they participate in is the original server's shutdown, which detaches them. Test it by asserting exactly one collector tick source and one database connection before and after an attach. Create-fleet finalization attaches the held socket immediately after the commit, alongside the key finalization, so enrollment claims and agent reports are reachable on the advertised listener without a restart. A crash between the commit and the attach needs no journal entry: startup in `hub` role binds and attaches every listener stored in `fleet_settings`, the same path every normal hub restart takes (Task 9 later adds owner-managed editing of that row, and Task 10's transfer target and Task 11's promotion reuse the same manager). + +Cover the bind-failure path with a test proving the role stays `unconfigured`, a database commit failure after key creation proving the identity directory stays clean and a retry succeeds, a post-commit interruption before the move proving a restarted process finishes finalization from the journal and comes up as a hub with a usable signing key, an in-process attach test proving a request on the newly advertised listener succeeds with no process restart, and a kill between commit and attach proving the restarted process serves the stored listener. + +Do not return the printable recovery key as the transaction's only copy, and do not put the plaintext in the database at all: a SQLite `DELETE` removes a row logically while its bytes survive in pages, the freelist, and WAL files, and from there ride into backups and transfer snapshots, which would break the promise that the key is unrecoverable after acknowledgement. The pending key lives in a service-owned mode `0600` file under `resolved_data_root / "identity"`, written with the same atomic discipline as the identity keys, and the `pending_recovery_key` row (Task 2 schema) persists only its path and metadata inside the create-fleet transaction. `GET /api/v2/setup/recovery-key` (owner session, loopback source) reads and returns the printable key from that file for as long as the record exists, and `POST /api/v2/setup/recovery-key/ack` deletes the row in a committed transaction first, then overwrites the file's bytes best-effort and unlinks it, which is the only deletion path, ordered so no crash window can leave a durable row pointing at a missing file to block hub transfer indefinitely. The acknowledgement is idempotent: an ack with no row present returns success. Startup reconciles both partial states without owner intervention: a pending key file in the identity directory with no `pending_recovery_key` row is an acknowledgement that crashed mid-deletion, so startup overwrites and unlinks it, and a row whose file is missing (the impossible order, defended anyway) is completed as acknowledged by deleting the row rather than blocking transfer until someone edits the database. Test both windows: kill after the row delete but before the unlink and prove startup destroys the file, and delete the file out from under a live row and prove startup clears the row and transfer proceeds. The identity directory is already excluded from database backups and transfer snapshots, and a test scans the raw database, WAL, and staged transfer payload bytes to prove the printable key never appears in any of them and the file is gone after acknowledgement. A create-fleet response lost before the browser renders the key therefore never strands dead-hub recovery: the owner logs in at the hub PC and retrieves it, then acknowledges. After acknowledgement the private key is unrecoverable from the hub, matching the design's retention rule. + +- [ ] **Step 6: Mount `/api/v2` without weakening v1** + +Session login and setup routes are public but rate limited. All fleet routes added later depend on `require_owner_session`. Keep existing v1 behavior unchanged for `legacy` mode. + +- [ ] **Step 7: Verify auth behavior and commit** + +Run: + +```bash +.venv/bin/python -m pytest tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py tests/test_login_throttle.py tests/test_api_auth.py tests/test_request_limits.py -q +.venv/bin/python -m pytest -q +``` + +Commit: + +```bash +git add agent/security/passwords.py agent/security/sessions.py agent/fleet/setup.py agent/fleet/listeners.py agent/fleet/schema.py agent/routes/v2_session.py agent/routes/v2_setup.py agent/security/limits.py agent/main.py tests/conftest.py tests/test_owner_passwords.py tests/test_owner_sessions.py tests/test_create_fleet.py tests/test_v2_csrf.py tests/test_login_throttle.py tests/test_api_auth.py +git commit -m "feat: add owner sessions and fleet creation" +``` + +--- + +### Task 4: Enroll agents with single-use codes and signed requests + +**Purpose:** Add a PC without distributing a shared token. + +**Files:** +* Create: `agent/fleet/request_signing.py` +* Create: `agent/fleet/enrollment.py` +* Create: `agent/fleet/hub_client.py` +* Create: `agent/routes/internal_enrollment.py` +* Create: `agent/routes/v2_machines.py` +* Modify: `agent/fleet/models.py` +* Modify: `agent/fleet/state.py` +* Modify: `agent/security/limits.py` +* Modify: `agent/main.py` +* Create: `tests/test_internal_request_signing.py` +* Create: `tests/test_agent_enrollment.py` +* Create: `tests/test_enrollment_limits.py` +* Modify: `tests/test_request_limits.py` +* Modify: `tests/test_security_contract.py` + +**Interfaces:** +* Produces: `EnrollmentService.create_code() -> EnrollmentCode` +* Produces: `EnrollmentService.claim(code: str, hub: HubClaim) -> AgentClaim` +* Produces: `EnrollmentService.finalize(enrollment_id: UUID, nonce: str, record: SignedEnrollmentRecord, announcement: HubAnnouncement) -> RuntimeState` +* Produces: `EnrollmentRepository.store_claimed_attempt(record: SignedEnrollmentRecord, address: str, claim_nonce: str) -> None` +* Produces: `sign_request(identity, fleet_id, machine_id, method, path, body, now, nonce) -> SignedHeaders` +* Produces: `verify_internal_request(request, machine_record) -> None` +* Produces: hub `POST /api/v2/fleet/machines/enroll` +* Produces: local-only `POST /api/v2/setup/pairing-code` +* Produces: local-only `GET/PUT /api/v2/setup/enrollment-listeners` +* Produces: agent `POST /api/internal/v1/enrollment/claim` and `/finalize` +* Produces: the agent's persisted post-enrollment listener set (`fleet_settings.listeners_json`, Task 2 schema) + +Use this one canonical implementation boundary: + +```python +def canonical_request_bytes( + fleet_id: UUID, + machine_id: UUID, + method: str, + path: str, + body: bytes, + timestamp: str, + nonce: str, +) -> bytes: + digest = hashlib.sha256(body).hexdigest() + return f"{method.upper()}\n{path}\n{fleet_id}\n{machine_id}\n{digest}\n{timestamp}\n{nonce}".encode("utf-8") + +async def enroll_machine(address: str, code: str) -> EnrolledMachine: + hub_claim = local_hub_claim() + claim = await hub_client.claim(address, code, hub_claim) + announcement = current_hub_announcement() + record = sign_enrollment_record(hub_claim, claim, announcement) + await repository.store_claimed_attempt(record, address, claim.claim_nonce) + await hub_client.finalize(address, claim.enrollment_id, claim.claim_nonce, record, announcement) + return await repository.mark_finalized(claim.enrollment_id) +``` + +- [ ] **Step 1: Write signing and enrollment protocol tests** + +Cover body tampering, clock skew, nonce replay, expired code, reused code, loopback-only listener selection, selected-private-listener binding, rejection of every non-enrollment route on that listener, private-address validation, interrupted finalization, a hub crash after its durable claim record, idempotent finalize retry, first-report pending status, fleet or machine header relabeling with an otherwise valid signature, a second claim attempt against an unconsumed code superseding the earlier unfinalized claim for that code, and revoking a previously enrolled machine then re-enrolling it under the same stable machine identity. + +```python +def test_pairing_code_expires_after_ten_minutes(clock, enrollment): + issued = enrollment.create_code() + clock.advance(minutes=10, seconds=1) + with pytest.raises(EnrollmentExpired): + enrollment.claim(issued.value, sample_hub_claim()) +``` + +- [ ] **Step 2: Run focused tests and observe the missing protocol** + +Run: `.venv/bin/python -m pytest tests/test_internal_request_signing.py tests/test_agent_enrollment.py tests/test_enrollment_limits.py -q` + +- [ ] **Step 3: Implement canonical request signing** + +Keep canonicalization in one module. Reject duplicate signature headers and decode base64url strictly. Store nonce digests until their 120-second replay window expires. + + +- [ ] **Step 4: Implement local-only listener selection, code display, and network claim endpoints** + +The local setup surface first reads private LAN and Tailscale candidates, then writes the exact selected bootstrap listeners through `PUT /api/v2/setup/enrollment-listeners`. Reject that route unless the actual peer address is loopback, including when forwarding headers claim loopback. Rebind only those selected private addresses before the hub can invoke claim, and reject every path there except `/api/internal/v1/enrollment/claim` and `/finalize`. `create_code()` is likewise callable only from the local setup surface. No HTTP response from `/api/internal/v1` returns the code. Claim binds the `enrollment_id`, hub claim digest, code, and agent claim. A retry with the same enrollment ID and identical hub claim returns the same `AgentClaim`. Any changed claim under that ID fails. Task 9 later replaces this bootstrap-only selection with the persisted owner-managed listener settings. + +- [ ] **Step 5: Implement hub-orchestrated enrollment** + +The hub validates `http` or `https` private addresses and calls claim. Claim is scoped to the pairing code, not to one attempt in progress: an unconsumed code accepts a new `enrollment_id`, and the fresh claim supersedes any earlier unfinalized claim for that same code, so a retried or restarted enrollment tool never races its own abandoned attempt for the last write to `enrollment_attempts`. Before finalize, it durably inserts the hub claim JSON, the complete agent claim JSON (signing and encryption public keys, hostname, addresses), signed record JSON, address, agent claim digest, and machine ID in `enrollment_attempts` with state `claimed`, and in the same transaction upserts the `fleet_machines` row from that agent claim: a machine ID with no existing row is created with status `enrolling`, and a machine ID that already exists (a previously revoked machine re-enrolling under its stable identity) has its addresses, hostname, and status replaced, `revoked_at` cleared, and `last_acked_sequence` left untouched so backfill contiguity survives the revoke and re-enroll cycle. That replacement path demands proof of the existing identity, carried in the wire model itself: `AgentClaim.prior_identity_proof` is an Ed25519 signature by the previously registered machine signing key over the claim's canonical compact JSON with the proof field omitted (the same sorted-key canonicalization every other signed object here uses), present exactly when the claim names an already-registered machine ID. The Rejoin flow holds that key, since machine identity survives revocation, and the hub verifies the proof against the registered public key before keeping or replacing anything. A claim naming a registered machine ID without that proof is rejected outright, because a modified client holding nothing but a valid pairing code could otherwise claim a visible machine UUID, take over its history and watermark, and break the real machine's reporting. A machine that lost its identity files enrolls as a fresh UUID like any new PC, leaving the old row revoked. Test the takeover attempt and the wiped-machine fresh-UUID path. A hub restarted after this commit can therefore both retry finalize and authenticate the agent's first report, instead of holding a digest it cannot rebuild a machine record from. Encrypt the raw claim nonce with AES-GCM under a key derived from the hub machine identity and enrollment ID, then store only nonce, ciphertext, and authentication tag in `claim_nonce_ciphertext`. The unique enrollment ID permits an identical retry and rejects changed bytes. Finalize verifies the fleet signature and agent claim digest, installs fleet state atomically, and changes the attempt to `finalized`. Finalize is idempotent by `enrollment_id` and record digest. The agent consumes the pairing code only after the transaction commits. The hub then marks its attempt finalized and clears the nonce ciphertext. A retry after either process restarts decrypts the nonce and reconstructs the next request entirely from the stored row. The machine remains `enrolling` until Task 5 commits its first signed report. + +Finalize also persists the trust the agent needs for everything after enrollment, all in the fleet-state installation transaction and all into columns Task 2's schema already declares, so this task touches neither `schema.py` nor `identity.py`. The claim's `fleet_signing_public_key` and `recovery_public_key` land in their `runtime_state` columns as the machine's durable verification anchors, and a restart test proves a subsequent fleet-signed announcement and a recovery replica both verify from the reloaded database with no in-memory state surviving. First, it stores `hub_claim.hub_signing_public_key` in `runtime_state.hub_signing_public_key` as the agent's authorized hub key, the same key every later fleet-signed `HubAnnouncement` is expected to carry and refresh, and it delivers the hub's current fleet-signed announcement in the finalize payload itself, a `HubAnnouncement` carried alongside the signed record per the finalize interface, because the record's embedded `HubClaim` has neither the announcement's `issued_at` nor its fleet signature and no announcement-delivery route exists at this task's HEAD, while the announcement is fleet-signed and self-authenticating so carrying it in the request costs nothing in trust. Self-authenticating is not self-binding, though: a first-enrolling agent has no floor yet, so an on-path host replaying the valid record and nonce could swap in any older fleet-signed announcement and finalize the agent onto a stale hub. The record therefore pins the delivery: `SignedEnrollmentRecord.hub_announcement_digest` is the canonical digest of the announcement finalize must carry, covered by the record's signature, and finalize verifies the carried announcement's digest against it and its fleet UUID, hub machine ID, authority pair, and `hub_signing_public_key` against the authenticated `HubClaim` before committing, rejecting any mismatch. Test the downgrade: a finalize carrying an older fleet-signed announcement with an otherwise valid record and nonce must be refused. The agent verifies all of that and persists the announcement verbatim into its own `runtime_state.hub_announcement_json` and `floor_announcement_json` together, setting the floor pair `(floor_recovery_epoch, generation_floor)` to that announcement's epoch and generation in the same transaction (under the never-lower rule every floor write obeys: a rejoining machine that retains a higher floor from its earlier tenure keeps it and its evidence), so the floor carries fleet-signed evidence from the very first generation the machine ever accepts (an agent enrolled at generation `G` that later adopts a recovered hub below `G` must still be able to prove `G` to the convergence rule). `hub_announcement_json` is the durable copy every later acceptance replaces atomically and Task 11's probe endpoint serves back after any restart. Second, it persists the same private addresses used for the bootstrap enrollment listener into the agent's own `fleet_settings.listeners_json` row (Task 2 schema), and those addresses populate `AgentClaim.addresses` so the hub's registry reflects where the agent can actually be reached. From finalize onward the agent's listener serves exactly `GET /api/internal/v1/announcement` (Task 11), the announcement delivery route (Task 9), the recovery replica delivery route (Task 11), and the loopback-only recovery page routes (Task 11), rejecting every other path, the same allowlist discipline the bootstrap listener used for claim and finalize. One digest-scoped exception keeps the handshake closable: the agent's finalize commit can succeed while its response is lost, and Step 5 promises the hub retries from its durable `claimed` row, so `/finalize` remains answerable for exactly the completed enrollment ID and record digest, idempotently returning the stored finalize result, until the hub acknowledges it or the attempt expires. A finalize request for any other identifier is rejected like every other closed route, and a lost-response retry test proves the hub marks the attempt finalized against an agent that already switched roles. The claim and finalize routes are removed, not gone forever: the loopback recovery page carries a Rejoin Fleet action that re-enables exactly those two routes on the private listener for one pairing window (they close again at finalize or at the ten-minute expiry), so a machine the owner revoked, or one missing from a recovered bundle's registry, re-enrolls under its existing machine identity with its retained outbox intact instead of being wiped. Test the full cycle: revoke, rejoin from the local page with a fresh code, and prove the machine keeps its identity and backfills its retained history through the Step 5 upsert. + +The enrolled listener set is not frozen at enrollment, because DHCP, Tailscale, and interface changes routinely move an agent's addresses. The listener row stores intent, not bare addresses: each entry names an enabled interface and transport (the LAN interface, the Tailscale interface, an explicit static address) alongside its currently resolved address, and anything the owner disabled stays recorded as disabled. On startup and on a periodic check, the agent binds its saved listeners with the same degraded discipline Task 9 gives the hub (loopback always, unbindable non-loopback entries dropped with a logged state), re-resolves only the enabled entries against its current private addresses, and never re-adds a disabled interface no matter what detection finds, so a refresh cannot undo an explicit listener policy. An adopted change persists to its `fleet_settings.listeners_json` row and pushes through `POST /api/internal/v1/machines/addresses`, a signed internal request whose authenticated body carries the new address list, with a test that a disabled interface stays disabled and unadvertised across a refresh that would have re-detected it. The hub verifies the envelope and updates `fleet_machines.addresses_json`, so announcement deliveries, recovery replica deliveries, the recovery probe, and transfer target preflight all reach the agent where it actually lives. Test identity continuity across an address change: the agent moves addresses, refreshes the registry, and a hub push reaches the new address with the machine ID and history unchanged. Without this, Tasks 10 and 11's hub-to-agent pushes and the recovery generation probe have no listener to reach and no key to verify against. The owner edits an agent's listener intent at the machine itself: the loopback-only local recovery page carries a listener editor over the same `fleet_settings` row with the same save rules as the hub's Task 9 surface (test-bind new or changed addresses, loopback always retained, disabling an interface records it disabled), and a saved edit takes the normal refresh path through the signed address-update route so the registry follows. Task 9's owner-session settings route edits only the hub's own listeners, so this local editor is the one path that changes an agent's, with a test disabling an interface from the local page and proving the registry and the periodic refresh both respect it. + +- [ ] **Step 6: Verify enrollment and commit** + +Run the three new files, `tests/test_request_limits.py`, `tests/test_security_contract.py`, then the full suite. + +Commit message: `feat: enroll agents with one-time codes` + +--- + +### Task 5: Buffer reports locally and ingest them at the hub + +**Purpose:** Make agent telemetry reliable across hub downtime. + +**Files:** +* Create: `agent/fleet/outbox.py` +* Create: `agent/fleet/ingestion.py` +* Create: `agent/fleet/transport.py` +* Create: `agent/routes/internal_reports.py` +* Modify: `agent/fleet/models.py` +* Modify: `agent/collector.py` +* Modify: `agent/services/retention.py` +* Modify: `agent/main.py` +* Create: `tests/test_report_outbox.py` +* Create: `tests/test_hub_ingestion.py` +* Create: `tests/test_agent_backfill.py` +* Create: `tests/test_collection_to_outbox.py` +* Modify: `tests/test_request_limits.py` +* Modify: `tests/test_database_migration.py` + +**Interfaces:** +* Produces: `OutboxRepository.append(captured_at, metrics, processes) -> ReportBatch` +* Produces: `OutboxRepository.pending(after: int, limit: int) -> list[ReportBatch]` +* Produces: `OutboxRepository.acknowledge(sequence: int) -> None` +* Produces: `HubIngestion.ingest(upload: SignedReportUpload) -> IngestAck` +* Produces: `AgentTransport.run() -> None` +* Produces: `POST /api/internal/v1/reports` + +The ingestion transaction has this shape. Both admission predicates run inside the serialized transaction, not before it: enrollment, revocation, and the Task 10 transfer-freeze all commit through the same per-connection transaction lock, so checking them outside the transaction would let a batch pass the check, lose the lock race, and be stored and acknowledged after a revocation or freeze committed. `require_accepting_reports()` reads `runtime_state.transfer_state` and rejects unless it is `accepting`. In this task nothing ever sets it to `frozen` or `draining`, and Task 10 relies on the predicate reading that exact column being in place before its freeze and drain transactions exist. `stored_ack_or_reject_conflict` is the identical-retry exit: a matching upload ID and digest returns the persisted acknowledgement from the receipt row before any recomputation, so a retry of upload A after a later upload B advanced the watermark answers with A's original acknowledgement, never B's, and skips the event publish since nothing changed (a same-ID different-digest upload raises the `409` here instead). + +```python +async def ingest(self, upload: SignedReportUpload) -> IngestAck: + async with transaction(self.database): + await self.machines.require_active(upload.machine_id) + await self.state.require_accepting_reports() + stored = await self.receipts.stored_ack_or_reject_conflict(upload) + if stored is not None: + return sign_ingest_ack(stored.upload_id, stored.payload_digest, + stored.highest_contiguous_sequence) + await self.samples.insert_upload_if_absent(upload) + await self.batches.insert_presence_if_absent(upload) + await self.gaps.insert_if_absent(upload.machine_id, upload.gaps) + highest = await self.batches.highest_contiguous(upload.machine_id) + await self.machines.mark_reported(upload.machine_id, highest, upload.latest_activity_at()) + await self.receipts.record(upload) + await self.events.publish_machine_update(upload.machine_id, upload.sequence_end) + return sign_ingest_ack(upload.upload_id, upload.payload_digest, highest) +``` + +`sign_ingest_ack` exists because request signing proves the agent to the hub while doing nothing for the response: over plain HTTP an on-path host could suppress the upload entirely and answer with an arbitrarily high watermark, and the agent would mark unsaved samples acknowledged and let retention destroy them. The returned `IngestAck` is therefore signed by the hub machine key over the echoed `upload_id`, `payload_digest`, and `highest_contiguous_sequence`, and the agent verifies that signature against its persisted `runtime_state.hub_signing_public_key` and both echoed identifiers against the upload it actually sent before advancing its local watermark, treating any mismatch or bad signature exactly like a failed upload (retry, nothing acknowledged). Test the forgery: suppress the upload, return a fabricated ack with a huge watermark, and prove the outbox advances nothing. Receipts persist only the tuple, never a signature: the identical-retry path reconstructs and re-signs the stored `(upload_id, payload_digest, highest_contiguous_sequence)` with the current hub machine key at response time, keeping the receipt's original watermark, because a signature stored at first ingest would go stale the moment a transfer rotates the authorized hub key and leave the retrying agent unable to verify a perfectly good acknowledgement. Test the lost-response retry across a hub transfer: ingest commits, the response is lost, the fleet transfers, and the retried upload's ack verifies under the new hub's key with the original watermark. + +Two of `mark_reported`'s writes are monotonic maxima, never plain replacements, because oldest-first backfill interleaves with live uploads: `last_seen` stores the greater of its existing value and `upload.latest_activity_at()`, since a historical batch landing after a current one would otherwise flip an actively reporting machine to stale for the entire backfill (Task 6 computes status from this column), and `fleet_machines.outbox_high_water` stores the greater of its existing value and the upload's `outbox_high_water`. The agent stamps that field at upload build time from its persisted `sqlite_sequence` high-water mark for `report_outbox`, riding inside the signed payload like every other field, so the hub durably knows the newest sequence the agent has allocated even when the 100-batch chunk limit means `sequence_end` trails it by thousands, and Task 6's backfill object computes its newest local sequence and pending count from these columns instead of guessing from chunk edges. + +- [ ] **Step 1: Write outbox and idempotency tests** + +```python +async def test_duplicate_upload_is_stored_once(hub_ingestion, enrolled_machine, upload): + first = await hub_ingestion.ingest(upload) + second = await hub_ingestion.ingest(upload) + assert first.highest_contiguous_sequence == upload.sequence_end + assert second.highest_contiguous_sequence == upload.sequence_end + assert await count_metrics(upload.machine_id, upload.sequence_start) == len(upload.batches[0].metrics) +``` + +Also cover a sequence gap, stale acknowledgement, oldest-first batching, reconnect jitter, a gap older than retention, simultaneous live plus backfill traffic (asserting `last_seen` and `outbox_high_water` never regress when a historical chunk lands after a live upload), a forged payload digest, a mismatched sequence range, and reuse of one upload ID with different bytes. `ReportBatch` permits empty `metrics` and `processes` lists (every collector source can fail for one tick), so test that an empty batch is accepted, recorded in `report_batches`, and advances `highest_contiguous_sequence` exactly like a populated one instead of reading as a missing sequence that stalls acknowledgement forever. Test sequence allocation after total purge: acknowledge every outbox row, delete them all through the retention path, restart the repository, and prove the next append allocates the prior maximum plus one from the persisted `sqlite_sequence` high-water mark rather than reusing an acknowledged sequence. Add races: a revocation committing between an upload's arrival and its ingestion transaction must reject the upload, and a `runtime_state.transfer_state` freeze committing in that window must do the same, both proven with overlapping coroutines on one connection. Also cover retention purging expired `used_nonces`, expired `owner_sessions`, aged `report_receipts`, fully-covered `data_gaps`, and all but the newest-plus-one `recovery_bundles` rows. + +- [ ] **Step 2: Run focused tests and confirm missing outbox behavior** + +Run: `.venv/bin/python -m pytest tests/test_report_outbox.py tests/test_hub_ingestion.py tests/test_agent_backfill.py tests/test_collection_to_outbox.py -q` + +- [ ] **Step 3: Add one collector persistence boundary** + +Refactor the end of each collection tick to build one `ReportBatch`. In `legacy`, keep the existing `_write_metrics` and `_write_processes`. In `agent` or `hub`, append the same normalized data to `report_outbox`. A hub immediately ingests its own batch through `HubIngestion`, then acknowledges its own outbox row only after that transaction commits. The self-ingest loop does not start with the newest batch: whenever the process enters `hub` role (normal startup, transfer activation in Task 10, recovery promotion in Task 11), it first drains every unacknowledged `report_outbox` row oldest first through the same `HubIngestion` path before live batches, because a machine that buffered samples while it could not reach a hub (a transfer target collecting through the frozen span, or any hub restarted mid-outage) holds rows that no live tick will ever revisit and that agent transport, stopped in `hub` role, will never send. Test that rows appended in `agent` role appear in central history after the machine becomes a hub with no process restart between. + +- [ ] **Step 4: Build and verify the durable upload envelope** + +Build canonical compact JSON with sorted keys for `{batches, gaps}` and set `payload_digest` to its SHA256. Sign canonical compact JSON containing `fleet_id`, `machine_id`, `upload_id`, `sequence_start`, `sequence_end`, and `payload_digest` with the machine key. The outer HTTP request is also signed through Task 4, binding the exact body, fleet, and machine headers. + +The reports route verifies content length, the outer request signature, the upload signature, the digest, the exact sequence bounds, and ordered unique batch sequences before ingesting. An upload must contain at least one batch or one gap range, and a gap-only upload (retention removed the unacknowledged history, leaving nothing else to send) commits and acknowledges normally: `latest_activity_at()` falls back to `GapRange.reported_at`, the signed timestamp the agent stamps when it persists the disclosure (also what ingestion writes into `data_gaps.reported_at`), so the agent can advance past the missing range and the machine's activity value stays defined with no server-side guessing. Test a gap-only upload end to end. Insert metrics, processes, one `report_batches` presence row per accepted batch sequence, gap ranges, the machine watermark, and `report_receipts` in one transaction. Compute `highest_contiguous_sequence` by extending the persisted machine watermark through `report_batches` presence rows and disclosed gap ranges, never by scanning sample rows, so an empty batch and a retention-purged prefix both count as covered. Persist the computed `highest_contiguous_sequence` in the receipt. A retry with the same upload ID and digest returns that stored acknowledgement without recalculating against newer uploads. The same upload ID with a different digest returns `409`. Unique sample and gap keys make identical retries harmless. Return the highest contiguous stored or explicitly gapped sequence, not merely the submitted end. + +- [ ] **Step 5: Implement transport and retention** + +Use exponential reconnect delays of 1, 2, 4, 8, 16, 30 seconds with up to 20 percent jitter. Send a `SignedReportUpload` containing at most 100 report batches or 1 MiB per request. Delete acknowledged outbox rows only after seven days. Retention on a hub also ages out central fleet history: the existing service deletes only the legacy `metrics` and `process_snapshots` tables, so extend it to purge `fleet_metrics`, `fleet_processes`, and `report_batches` presence rows at or below each machine's persisted watermark older than `retention_days` in one transaction, with tests covering all three fleet tables (presence rows above the watermark are kept, since contiguity extends from the watermark through them). Persist and upload an exact `GapRange` when local retention removed unacknowledged history. Do not advance the local acknowledgement beyond a missing sequence until the hub acknowledges the persisted gap. + +The same retention transaction also owns purging every other table nothing else claims an owner for: `used_nonces` rows past `expires_at`, `owner_sessions` rows past `expires_at`, `report_receipts` rows older than `retention_days`, `data_gaps` rows fully covered by history already purged above, and `recovery_bundles` rows below the newest `(recovery_epoch, generation, bundle_revision)` (tie-break first, then lexicographic over the full triple, so a stale former hub's higher-generation bundle at an older epoch and a same-epoch losing lineage's are what get purged, never the current winning lineage's) while keeping one prior bundle so an in-flight recovery preview against the second-newest revision does not fail mid-purge. Add tests for each: an expired nonce and an expired session are both gone after the pass, an old receipt is purged, a fully-covered gap range is dropped while a partially-covered one survives, and exactly one prior recovery bundle remains alongside the newest. + +- [ ] **Step 6: Verify collection, reporting, and full regression** + +Run the new tests plus existing current, history, SSE, retention, and full suite tests. + +Commit message: `feat: buffer and ingest fleet telemetry` + +--- + +### Task 6: Expose machine-scoped fleet data through `/api/v2` + +**Purpose:** Give the hub dashboard and SlipShell seam one stable read model. + +**Files:** +* Create: `agent/fleet/queries.py` +* Create: `agent/fleet/events.py` +* Create: `agent/routes/v2_fleet.py` +* Create: `agent/routes/v2_history.py` +* Create: `agent/routes/v2_events.py` +* Modify: `agent/routes/v2_machines.py` +* Modify: `agent/main.py` +* Create: `tests/test_v2_fleet_api.py` +* Create: `tests/test_v2_history_api.py` +* Create: `tests/test_v2_events.py` +* Modify: `tests/test_request_limits.py` +* Modify: `tests/test_frontend_sse.py` + +**Interfaces:** +* Produces: `GET /api/v2/fleet` +* Produces: `GET /api/v2/machines` +* Produces: `GET /api/v2/machines/{machine_id}` +* Produces: `GET /api/v2/machines/{machine_id}/current` +* Produces: `GET /api/v2/machines/{machine_id}/history` +* Produces: `GET /api/v2/machines/{machine_id}/processes` +* Produces: `GET /api/v2/machines/{machine_id}/availability` +* Produces: `GET /api/v2/events` +* Produces: `MachineAvailabilityResponse(status, last_report_at, stale_after_seconds, data_gaps, backfill)` + +Route handlers stay thin and require the owner session at the router boundary: + +```python +@router.get("/machines/{machine_id}/current", response_model=MachineCurrentResponse) +async def current_machine( + machine_id: UUID, + _session: OwnerSession = Depends(require_owner_session), +): + return await get_fleet_queries().current(machine_id) +``` + +- [ ] **Step 1: Write machine isolation and stale-data tests** + +Assert stable UUID routing, no cross-machine data, explicit `data_gap` ranges, last report timestamps, stale threshold, backfill watermark and pending count, revoked machine visibility to the owner, history bounds, and SSE capacity limits. + +- [ ] **Step 2: Run focused tests and confirm v2 reads are absent** + +Run: `.venv/bin/python -m pytest tests/test_v2_fleet_api.py tests/test_v2_history_api.py tests/test_v2_events.py -q` + +- [ ] **Step 3: Implement query services over v2 tables only** + +Do not expose table rows directly. Return typed response models. Current values come from the newest sequence per machine. Offline status is computed from the last report time and configured collection interval. Availability returns `status`, nullable `last_report_at`, `stale_after_seconds`, ordered persisted gap ranges, and a typed backfill object with acknowledged sequence, newest local sequence when known, and pending count. + +- [ ] **Step 4: Implement a hub-owned event stream** + +Emit `machine_metrics`, `machine_status`, `backfill_progress`, `machine_revoked`, and heartbeat events. Every event includes `machine_id`. Use owner session cookie authentication, and not only at connection time: the stream revalidates its owner session before every event and heartbeat write, and session revocation publishes a cancellation that closes any open stream bound to that session, because a dependency that runs once while the response is constructed would let a revoked session keep receiving telemetry until it chose to disconnect. Test that an open stream dies at session revocation before the next heartbeat, in this task's PR, since every task merges as a separately safe HEAD. Task 12 later extends the same revalidation to integration principals. + +- [ ] **Step 5: Verify endpoint surface with a running test server** + +Start a temporary hub, log in, fetch two machine IDs, prove their current and history responses differ, then capture the command output under `artifacts/task-6-api-surface.txt`. + +Run a deliberate canary with a revoked session and require `401` before accepting the evidence. + +- [ ] **Step 6: Run full tests and commit** + +Commit message: `feat: expose fleet data through api v2` + +--- + +### Task 7: Replace the token gate with owner login and first-run setup + +**Purpose:** Make Create Fleet, Join Fleet, owner login, and logout usable before changing the monitoring pages. + +**Files:** +* Create: `static/js/pages/login.js` +* Create: `static/js/pages/setup.js` +* Create: `scripts/update-static-digests.py` +* Create: `tests/test_frontend_session_auth.py` +* Create: `tests/test_frontend_setup.py` +* Modify: `static/index.html` +* Modify: `static/js/auth.js` +* Modify: `static/js/api.js` +* Modify: `static/js/app.js` +* Modify: `static/css/components.css` +* Modify: `static/css/layout.css` +* Modify: `static/vendor/SHA256SUMS` +* Modify: `tests/test_frontend_auth.py` +* Modify: `tests/test_static_asset_policy.py` + +**Interfaces:** +* Consumes: Task 3 session and create-fleet endpoints +* Consumes: Task 4 local pairing-code endpoint +* Produces: `HumWatch.auth.loadSession() -> Promise` +* Produces: `HumWatch.auth.csrfHeaders(method) -> Headers` +* Produces: setup routes `#/setup`, `#/login`, and `#/recovery` + +- [ ] **Step 1: Write browser harness tests for setup and session login** + +Assert no `sessionStorage` token, credentials included on fetch, CSRF header on mutations, loopback Create Fleet, local pairing-code display for Join Fleet, normal logout, and generic login failures. + +```javascript +await HumWatch.auth.loadSession(); +assert.equal(fetchCalls[0].url, "/api/v2/session/me"); +assert.equal(fetchCalls[0].options.credentials, "same-origin"); +assert.equal(sessionStorage.getItem("humwatch_auth_token"), null); +``` + +- [ ] **Step 2: Run frontend tests and confirm the token flow fails them** + +Run: `.venv/bin/python -m pytest tests/test_frontend_auth.py tests/test_frontend_session_auth.py tests/test_frontend_setup.py -q` + +Expected: FAIL while bearer headers, token storage, and the token prompt remain reachable outside `RuntimeRole.LEGACY` (the role-scoped legacy client Step 3 retains is asserted present in `legacy` role, absent everywhere else). + +- [ ] **Step 3: Convert the browser client to owner sessions** + +Convert the v2 roles to owner sessions without stranding upgraded installations that are still in `RuntimeRole.LEGACY`, whose protected v1 routes keep bearer authentication until Task 13's migration wizard converts them, several PRs from now. The client therefore branches on the role probe before choosing an auth path: in `legacy` role it keeps a minimal role-scoped token client (token storage, bearer header, and token prompt, exactly today's behavior, reachable only in that role), while every v2 role uses sessions with no bearer path at all. For the session path, set `credentials: "same-origin"`, load `/api/v2/session/me` before protected navigation, and send the returned CSRF token only on state-changing requests. Test both sides of the branch: a legacy-role installation's dashboard authenticates with its stored token end to end after this task, and a v2-role page contains no token storage, bearer header, or prompt. The legacy client is deleted in Task 13 alongside the wizard that ends the need for it. + +- [ ] **Step 4: Implement setup and login pages** + +Create Fleet submits username, password, and the hub listener selection pre-filled from the detected private addresses, then displays the recovery key until the owner confirms it is saved, which sends the acknowledgement that deletes the pending record. If the create response is lost, the setup surface offers the retrieval path from Task 3 instead of a dead end. Join Fleet requests a code from the local endpoint and displays the selected private addresses. Login accepts username and password. Recovery is a routed shell until Task 11 adds its backend. + +- [ ] **Step 5: Add the deterministic digest updater** + +Create `scripts/update-static-digests.py`. It reads the existing manifest, refuses missing files, duplicate paths, paths outside `static`, or malformed rows, recomputes only the listed local assets, sorts output by path, and replaces the manifest atomically. Add a test that copies the static tree, changes one CSS file, runs the updater against the copy, and proves only that file's digest changes. + +Run: `.venv/bin/python scripts/update-static-digests.py` + +- [ ] **Step 6: Update pinned assets and produce surface evidence** + +Capture unconfigured setup, recovery key display, login failure, successful login, and logout at desktop and mobile widths in default, light, and terminal themes under `artifacts/task-7-auth-ui/`. + +Canary: return `200` from login without a session cookie in the browser fixture and require the dashboard to remain locked. + +- [ ] **Step 7: Verify and commit** + +Run focused browser tests, `.venv/bin/python -m pytest tests/test_static_asset_policy.py -q`, `.venv/bin/python scripts/verify-static-assets.py`, `node tests/browser_globals_smoke.js`, and the full pytest suite. + +Commit message: `feat: add fleet setup and owner login` + +--- + +### Task 8: Add fleet overview and in-app machine selection + +**Purpose:** Deliver one dashboard for every enrolled PC with no new tabs or cross-origin browser calls. + +**Files:** +* Create: `static/js/fleet-state.js` +* Create: `tests/test_frontend_fleet_state.py` +* Create: `tests/test_frontend_machine_switching.py` +* Modify: `static/index.html` +* Modify: `static/js/api.js` +* Modify: `static/js/app.js` +* Modify: `static/js/sse.js` +* Modify: `static/js/pages/machines.js` +* Modify: `static/js/pages/overview.js` +* Modify: `static/js/pages/cpu.js` +* Modify: `static/js/pages/gpu.js` +* Modify: `static/js/pages/memory.js` +* Modify: `static/js/pages/disk.js` +* Modify: `static/js/pages/network.js` +* Modify: `static/js/pages/battery.js` +* Modify: `static/js/pages/processes.js` +* Modify: `static/css/components.css` +* Modify: `static/css/layout.css` +* Modify: `static/vendor/SHA256SUMS` + +**Interfaces:** +* Consumes: Task 6 `/api/v2` responses and events +* Produces: `HumWatch.fleet.selectMachine(machineId)` +* Produces: `HumWatch.fleet.getSelectedMachineId()` +* Produces: `HumWatch.api.machinePath(suffix)` + +- [ ] **Step 1: Write browser harness tests for switching** + +Assert stable selected machine, same-origin URLs only, no `window.open`, per-machine current and history paths, process routing, selected-machine SSE filtering, and page refresh after selection. + +```javascript +HumWatch.fleet.selectMachine("machine-b"); +await HumWatch.pages.cpu.init(container); +assert.equal(fetchCalls[0], "/api/v2/machines/machine-b/current"); +``` + +- [ ] **Step 2: Run focused tests and confirm the current new-tab behavior fails** + +Run: `.venv/bin/python -m pytest tests/test_frontend_fleet_state.py tests/test_frontend_machine_switching.py tests/test_frontend_sse.py -q` + +- [ ] **Step 3: Add persistent machine selection** + +Put the selector at the top of the desktop sidebar and mobile header. Store only the selected machine UUID in `localStorage`. Every data page obtains its URL through `machinePath()`. + +- [ ] **Step 4: Replace Machines with the fleet overview** + +Render enrolled machine cards from the hub. Clicking a card calls `selectMachine()` and navigates within the current origin. Show status, last report, current highlights, gaps, and backfill progress. + +- [ ] **Step 5: Route live events and every data page by machine UUID** + +The hub event stream remains one connection. Dispatch each event only to handlers for its `machine_id`. Refresh selected-machine availability after a selection changes. + +- [ ] **Step 6: Update pinned assets and produce surface evidence** + +Run `.venv/bin/python scripts/update-static-digests.py` after the final JavaScript and CSS edits, then run the verifier before capturing evidence. + +Capture fleet overview and two selected machines at desktop and mobile widths in default, light, and terminal themes under `artifacts/task-8-fleet-ui/`. + +Canary: break the selected-machine API path in the browser fixture, confirm the switching test fails, restore it, then rerun. + +- [ ] **Step 7: Verify and commit** + +Run focused browser tests, static asset verification, browser smoke, and the full pytest suite. + +Commit message: `feat: unify fleet access in one dashboard` + +--- + +### Task 9: Add fleet administration, listeners, and optional HTTPS + +**Purpose:** Make enrollment, removal, sessions, network access, and TLS manageable from one settings surface. + +**Files:** +* Modify: `agent/fleet/listeners.py` +* Create: `agent/fleet/announcements.py` +* Create: `agent/routes/internal_announcements.py` +* Create: `agent/routes/v2_settings.py` +* Create: `static/js/pages/fleet-settings.js` +* Create: `tests/test_listener_policy.py` +* Create: `tests/test_v2_fleet_admin.py` +* Create: `tests/test_frontend_fleet_settings.py` +* Modify: `agent/config.py` +* Modify: `agent/security/tls.py` +* Modify: `agent/main.py` +* Modify: `static/index.html` +* Modify: `static/js/api.js` +* Modify: `static/js/pages/settings.js` +* Modify: `static/css/components.css` +* Modify: `static/css/layout.css` +* Modify: `static/vendor/SHA256SUMS` +* Modify: `tests/test_security_config.py` + +**Interfaces:** +* Produces: `discover_private_listeners() -> list[ListenerCandidate]` +* Produces: `GET/PUT /api/v2/fleet/settings/listeners` +* Produces: `GET/PUT /api/v2/fleet/settings/https` +* Produces: `DELETE /api/v2/fleet/machines/{machine_id}` +* Produces: `ListenerSettingsResponse(saved, active, restart_required)` + +Listener selection is an explicit typed configuration: + +```python +class ListenerCandidate(BaseModel): + address: IPvAnyAddress + kind: Literal["loopback", "lan", "tailscale", "public"] + enabled: bool + warning_required: bool + +def bind_listener_sockets(listeners: list[ListenerCandidate], port: int) -> list[socket.socket]: + enabled = [item for item in listeners if item.enabled] + if not enabled: + raise SecurityConfigurationError("at least one listener is required") + return [bind_tcp_socket(str(item.address), port) for item in enabled] +``` + +Pass every returned socket to one `uvicorn.Server.run(sockets=sockets)` call so application startup, the collector, and database initialization run once. + +- [ ] **Step 1: Write listener and settings tests** + +Cover loopback, RFC1918, Tailscale `100.64.0.0/10`, IPv6 private ranges, public-address warnings, optional cert pairs, missing half-pairs, CSRF, self-removal rejection, independent revocation, persisted listeners across restart, a saved change that does not affect active sockets before restart, a `PUT` that rejects a requested address failing its test bind, and startup with one previously saved non-loopback listener that can no longer bind. + +- [ ] **Step 2: Change the security rule from mandatory TLS to explicit listeners** + +`validate_security_config()` must allow HTTP for selected private listeners without requiring a legacy bearer token. Public listeners require an explicit `allow_public_listener` acknowledgement and still show a warning. Store the exact list, HTTPS paths, and acknowledgement in the Task 2 `fleet_settings` row. Startup loads that row and binds exactly those addresses. Never silently widen to `0.0.0.0` after a saved specific-listener configuration. + +`PUT` validates and saves settings but does not mutate live sockets. Before saving, it test-binds only the addresses that are new or changed relative to the currently active set (open the socket, then close it immediately, never holding it across the response) and rejects the request with the specific address that failed if any bind fails, so a typo or an address that has already moved off this machine never becomes a saved-but-unbindable listener. Addresses already being served count as validated by the fact that Uvicorn holds them, and test-binding one would fail with a false address-in-use error, so resaving the current configuration, adding a listener next to it, or changing an HTTPS path never trips over the sockets the server itself owns. It returns saved settings, current active settings, and `restart_required=true` when they differ. The UI shows the exact service restart action for the platform. After restart, `GET` must report the saved list as active and `restart_required=false`. Use one Uvicorn server with all pre-bound sockets so app startup and collectors still run once. + +A listener change is also an authority change the fleet has to hear about, because agents transport reports to the addresses in their accepted announcement and only a verified announcement, pushed by the hub or fetched by the agent's own discovery loop below, ever redirects them: a hub startup that successfully binds an endpoint set differing from what its own `runtime_state.hub_announcement_json` advertises mints a higher-generation fleet-signed announcement of itself carrying the new set, and that comparison is over full advertised endpoints (scheme, address, and port), not bare addresses, because enabling or disabling HTTPS on the same listener IPs changes the URLs agents must use just as surely as a moved address does, and agents left holding `http://` URLs against an HTTPS socket would retry forever. The hub installs the new announcement as its own accepted announcement and floor evidence, and creates the durable per-agent delivery obligation rows (Task 2 tables). Task 9 itself implements the minimal machinery that makes this work at its own HEAD, since the redirect test must pass in this PR and Task 10 does not exist yet: `agent/fleet/announcements.py` with the canonical signing and verification helpers, the retrying delivery worker over `hub_announcement_deliveries` (the standard backoff schedule, resumed at startup), and the agent-side delivery route in `agent/routes/internal_announcements.py` that verifies the fleet signature, applies the lineage tie-break before any pair arithmetic, then applies the strict above-floor rule over the `(recovery_epoch, generation)` pair, persists the announcement columns atomically, and redirects transport. The tie-break coming first is load-bearing: an announcement at the accepted announcement's epoch from a different lineage enters the pair comparison only when its lineage wins the tie-break against the accepted one, and a losing lineage's announcement is rejected with evidence no matter how high its pair climbs, because a partitioned loser that raced ahead in generations would otherwise be accepted outright by the floor rule and redirect the winning side's agents without the tie-break handler ever seeing evidence. A rejection for a stale pair carries the agent's own accepted announcement in the rejection body, fleet-signed and self-authenticating like every announcement, and a delivering hub that verifies in that evidence a strictly higher recovery epoch, or its own epoch under a different lineage that wins Task 11's deterministic tie-break (the lexicographically smaller `lineage_id`), stops all publication obligations and surfaces the fenced state on its local recovery page with Rejoin Fleet as the stated remedy, so a returning dead hub converges to owner-visible retirement instead of retrying its obsolete authority forever. The winning side of that same comparison acts too, because an agent pointed at a retired loser can never send the watermark request that drives Task 11's convergence: a delivering hub that verifies in the rejection evidence a different lineage, at its own epoch, at an equal or higher pair, that loses the tie-break to it, mints a fresh self-announcement above the rejected evidence's pair and republishes through the normal obligations, so agents holding the losing lineage accept the winner under the strict rule on the very next delivery. Test both branches from the same simultaneous-recovery fixture: the loser's delivery rejection retires it, and the winner's delivery rejection drives the mint that converges the loser's agents. Task 10 reuses and extends exactly this machinery for transfer announcements rather than introducing its own, and a transfer announcement carries the source hub's `lineage_id` verbatim: the transfer replaces `hub_machine_id` with the target machine, never the lineage, so the same-epoch tie-break stays pinned to the original promotion event across any number of transfers. Old addresses keep serving until the new set is bound, so agents follow the announcement rather than losing the hub. Test at Task 9 HEAD that a saved listener replacement redirects an enrolled agent's transport to the new address through the published announcement, and test the scheme case the address comparison would miss: enable HTTPS on the same listener IPs, restart, and prove the hub mints a new announcement and the enrolled agent's transport moves to the `https://` URLs. + +Hub push cannot be the only redirection path, because it needs the registry address to still reach the agent: an agent that was offline while the hub moved, and whose own DHCP or Tailscale address changed in the same window, leaves the delivery worker retrying a stale address while the agent retries a retired hub, and neither side can complete either the announcement delivery or the signed address update. The spec's discovery promise closes that loop from the agent side. When agent transport has failed to reach every address in its accepted announcement through its full backoff schedule, it enters a jittered discovery cycle: enumerate candidate hub addresses from the same LAN and Tailscale sources `discover_private_listeners()` draws from, fetch `GET /api/internal/v1/announcement` from each candidate, and feed any response through exactly the acceptance path a pushed delivery uses, fleet signature verification plus the strict above-floor pair rule, so discovery grants no trust that push did not already grant (a candidate returning garbage, a foreign fleet's announcement, or a dead lineage's old pair changes nothing). An accepted announcement redirects transport and triggers the standard signed address-update push so the registry learns where the agent lives now. The cycle repeats on the reconnect schedule until transport succeeds, and manual adoption on the local recovery page remains the fallback for a hub discovery cannot reach across subnets. Test the double-move: hub relocates and mints while the agent is offline and the agent's own address changes, the agent comes back, discovery finds the new hub, the pair rule accepts the announcement, transport redirects, the address update lands in the registry, and reporting resumes with no manual step. + +A test bind at save time proves the address was bindable then, not that it stays bindable (Tailscale can drop, DHCP can reassign an address between save and the next restart). Startup therefore never refuses to start over a saved listener it cannot bind: it always binds and attaches loopback first, then attempts every saved non-loopback listener, and any that fails is dropped with a logged degraded-startup entry rather than raising `SecurityConfigurationError`. `GET /api/v2/fleet/settings/listeners` reports the dropped listener as saved but not active so the settings page can surface it, and the owner can fix the address or remove it without the hub having refused to come up in the meantime. + +- [ ] **Step 3: Implement Settings sections** + +Add Owner and Sessions, Fleet and Enrolled PCs, Add PC, Network Listeners, Optional HTTPS, Integrations status, and Legacy Compatibility. Destructive actions use plain confirmation copy. Network Listeners shows any saved listener startup could not bind as degraded, distinct from an active one, with the plain remediation of fixing or removing it. + +- [ ] **Step 4: Produce running surface evidence and canary** + +Capture settings on default, light, and terminal themes. Exercise add PC, revoke PC, session revoke, LAN listener selection, detected Tailscale selection, and optional HTTPS validation. + +Canary: select a public test address without acknowledgement and confirm the API and UI block saving. + +- [ ] **Step 5: Verify and commit** + +Run focused tests, static asset verification, browser smoke, full pytest, then commit as `feat: add fleet and network administration`. + +--- + +### Task 10: Move the hub safely to another enrolled PC + +**Purpose:** Transfer hub ownership without creating two authoritative hubs. + +**Files:** +* Modify: `agent/fleet/announcements.py` +* Create: `agent/fleet/transfer.py` +* Create: `agent/routes/internal_transfer.py` +* Create: `agent/routes/v2_transfer.py` +* Modify: `agent/fleet/models.py` +* Modify: `agent/fleet/state.py` +* Modify: `agent/fleet/transport.py` +* Modify: `agent/main.py` +* Modify: `static/js/pages/fleet-settings.js` +* Create: `tests/test_hub_announcements.py` +* Create: `tests/test_hub_transfer.py` +* Create: `tests/test_transfer_rollback.py` +* Create: `tests/test_frontend_hub_transfer.py` + +**Interfaces:** +* Produces: `HubAnnouncement.sign(fleet_identity) -> HubAnnouncement` +* Produces: `verify_announcement(announcement, fleet_public_key, accepted: HubAnnouncement | None, floor_recovery_epoch: int, generation_floor: int) -> bool` (the accepted announcement supplies the current epoch, generation, and `lineage_id` the tie-break-first rule compares against, alongside the floor pair, and is `None` only before first acceptance) +* Produces: `HubTransferService.preflight(target_id) -> TransferPreflight` (includes the target's detected private listener candidates) +* Produces: `HubTransferService.transfer(target_id, listener: ListenerSelection) -> TransferResult` +* Produces: `FleetStateRepository.enqueue_signed_abort(announcement) -> None` (durable row in `hub_transfer_aborts`, Task 2 schema, resumed from startup like `recovery_deliveries`) +* Produces: `POST /api/v2/fleet/hub-transfer/preflight` and `/execute` +* Produces: `GET /api/v2/fleet/hub-transfer/status` +* Produces: `HubTransferService.reconcile_draining_transfers() -> None` +* Produces: internal `POST /api/internal/v1/transfer/stage`, `/commit`, `/activate`, `/abort`, and `GET /status` on the target, backed by `pending_hub_authority` + +Keep the authority switch behind one method: + +```python +async def transfer(self, target_id: UUID, listener: ListenerSelection) -> TransferResult: + preflight = await self.preflight(target_id) + preflight.require_ready() + preflight.require_listener_candidate(listener) + preflight.require_no_pending_recovery_key() + await self.state.enter_transfer_freeze(require_active_target=preflight.target) + announcement = None + try: + staged = await self.stage_encrypted_snapshot(preflight.target) + announcement = await self.prepare_new_generation(preflight.target, listener) + await self.target_client.stage_pending_hub(staged, announcement, listener) + await self.target_client.commit_unannounced_hub(announcement) + await self.target_client.require_ready(announcement) + await self.state.enter_draining_transfer(announcement) + except BaseException: + async with transaction(self.database): + if announcement is not None: + await self.state.enqueue_signed_abort(announcement) + await self.state.lift_transfer_freeze() + raise + return TransferResult(announcement=announcement, status="draining") + + +async def reconcile_draining_transfers(self) -> None: + for row in await self.state.due_draining_transfers(): + try: + publication = await self.target_client.activate_publication_obligation(row.announcement) + await self.target_client.require_publication_persisted(publication) + except TransferRetryable: + continue + async with transaction(self.database): + await self.sessions.revoke_all() + await self.state.retire_local_hub(row.announcement, remove_signing_key=True) +``` + +`transfer()` returns as soon as draining commits, with `status="draining"`, instead of calling activation inline. Activation sat outside the original `try`/`except` in an earlier draft of this method: a network error there would leave the hub durably draining with no in-process code left to finish it, recoverable only by an eventual restart. `reconcile_draining_transfers()` closes that gap as a background task started from the hub lifespan (the same pattern Task 11's replication loop uses), polling `hub_transfers` rows in state `draining` and retrying activation until `require_publication_persisted()` succeeds, at which point it performs the same session revocation and local retirement the inline call used to. The `/execute` route and the transfer UI observe progress through `GET /api/v2/fleet/hub-transfer/status` rather than blocking on the HTTP request that started the transfer. + +- [ ] **Step 1: Write generation, rollback, and interruption tests** + +Cover lower generation rejection, equal generation rejection from another hub, tampered address lists, insufficient storage, incompatible version, target startup failure, target login surface failure, a target bind failure on the selected listener failing staging with the target still an agent and the old hub aborting with the freeze lifted, the activated target listening on the owner's selection with the announcement advertising it and never the old hub's imported addresses or certificate paths, an aborted target holding no staged fleet key file, a crash between the target's activation commit and key finalization repaired at startup into a hub that can sign, signed report rejection at the proposed generation, signed abort, agent announcement delivery, offline agent discovery, browser session revocation, and old hub demotion only after all readiness proof. Add crash tests for target publication obligation creation, a lost activation response after the target commits, target restart after the old hub retires, and old-hub restart while draining. Prove no acknowledged sample is lost across a transfer: a report accepted after the freeze would predate the snapshot, so assert reports are rejected between freeze and snapshot, buffered by the agent, and delivered to the new hub. Assert fleet mutations (enrollment, revocation, listener and integration changes) are rejected for the whole frozen and draining span. Cover freeze lift after a signed abort, freeze lift when the target is unreachable so abort delivery itself fails (the local lift must still commit and the abort must retry from its durable queue), freeze lift when the snapshot itself or generation preparation fails before any target contact, and old-hub restart while frozen. Add a target continuity test: the target collects samples into its own outbox during staging and readiness, activation preserves that outbox and the target's identity, and those samples appear in the new hub's history afterward, alongside the demoted old hub delivering the outbox rows it kept out of the snapshot. The old hub may remove its signing key only after the target has durably activated its own retryable announcement obligation. + +Add a preflight test that an unacknowledged `pending_recovery_key` row blocks transfer with an explicit error and changes nothing, and a follow-up transfer succeeding once the key is acknowledged. Add abort-scoping tests: a signed abort for a digest the target has already activated is rejected and reported so the old hub drops it from `hub_transfer_aborts` without retrying forever, and a stale abort queued for a failed transfer #1 does not touch a target that went on to activate a successful transfer #2 to the same machine. Add abort listener-restore tests: after an abort of a transfer that selected a genuinely new address, the target serves announcement and recovery pushes on its original enrolled address (in the running process and again after a restart) with the transfer socket detached, and after an abort of a transfer that reused the enrolled socket, that socket answers only the post-enrollment allowlist with every readiness route refused. Add a `pending_hub_authority` restart test: kill the target between `commit_unannounced_hub` and activation, restart it, and prove it comes back in `RuntimeRole.PENDING_HUB` serving the internal activation, abort, and status routes plus the readiness surface (login page, generic `401` login, proposed-generation report ingestion), and nothing else: no authenticated dashboard data route and no fleet mutation route. Add a background reconciler test: raise inside `activate_publication_obligation` after draining commits, prove `transfer()` itself returns `status="draining"` rather than raising, then run `reconcile_draining_transfers()` and prove it completes the retirement from the persisted `hub_transfers` row with no further caller involvement. Add a test that recovery bundle creation and replication (Task 11) are rejected for the whole frozen and draining span, matching every other fleet mutation. + +- [ ] **Step 2: Implement signed announcements** + +Canonicalize announcement JSON with sorted keys and compact separators before signing. Store the accepted generation atomically with new hub addresses. + +- [ ] **Step 3: Implement encrypted transfer staging** + +Create a SQLite backup through the backup API. Package fleet config, owner hash, registry, history, integrations, and fleet signing key. Exclude sessions, the old hub's machine-local `report_outbox`, and its `sqlite_sequence` allocator row for that table: `report_outbox.sequence` is `INTEGER PRIMARY KEY AUTOINCREMENT` (Task 2), and importing the old hub's `sqlite_sequence` entry over the target's own would rewind the target's outbox allocator below its already-acknowledged watermark, silently making every later local sample look like an old already-acknowledged sequence and stopping that machine's own history permanently. Encrypt to the target X25519 public key with HKDF-SHA256 and AES-256-GCM. + +- [ ] **Step 4: Implement the staged transfer and explicit abort path** + +The target imports into staging, inserts a `pending_hub_authority` row (Task 2 schema) keyed on the announcement digest with the staged snapshot and key paths, persists the signed proposed generation, initializes that generation's `recovery_bundle_revision` to zero, and sets `runtime_state.role` to `RuntimeRole.PENDING_HUB` for unannounced hub mode. An import assertion compares the target's own `sqlite_sequence` row for `report_outbox` before and after the import and fails loudly if it changed, catching a packaging regression on the old hub's side before it can silently rewind the target's allocator. It does not start on the imported listener settings: the snapshot's `fleet_settings` row carries the old PC's addresses and machine-local certificate paths, which the target may be unable to bind at readiness or after its next restart. The owner's preflight-validated `ListenerSelection` from the target's own detected private addresses travels with `stage_pending_hub`, and the target secures that socket before `commit_unannounced_hub`: when the selection names an address the target's `ListenerManager` already serves (the normal case, since Task 4 keeps the enrolled agent's listener bound for announcement and recovery routes), staging reuses that existing bound socket and atomically widens its route set at commit instead of binding the same address a second time, which would fail address-in-use on every ordinary transfer. Only a selection naming a genuinely new address is freshly bound and held, then attached through Task 3's `ListenerManager` when unannounced hub mode starts, so readiness probes reach the staged hub without a restart, exactly like Create Fleet and recovery: a bind failure on such a new address fails staging visibly with the target still an agent, which the old hub's cleanup path catches and aborts with the freeze lifted. The import replaces the machine-local listener and certificate settings with the selection, `prepare_new_generation` builds the announcement from that same selection, and the announcement never advertises the old hub's addresses. The import otherwise applies only hub-authority tables. + +The transferred fleet signing key follows the same staged-file discipline as fleet creation and recovery: the target decrypts it to a staging path during import, never into the identity location, and records that path in `pending_hub_authority.staged_key_path`. Holding staged or unannounced authority never makes the key usable (readiness verifies reports with the fleet public key and the announcement was signed by the old hub, so nothing before activation needs the private key). The target's activation transaction, the one that persists the publication obligation, sets `pending_hub_authority.state` to `activated`, flips `runtime_state.role` to `RuntimeRole.HUB`, installs the transfer announcement as the target's own accepted announcement, and advances the floor with the same never-lower rule every other floor write obeys: the floor columns and `floor_announcement_json` move to the transfer announcement's pair only when that pair is lexicographically above the target's retained floor, and otherwise keep the prior floor and its evidence untouched, because a target that manually adopted a lower pair while convergence was still pending has a floor above its accepted announcement, and blindly installing the transfer announcement as floor evidence would lower the anti-replay floor or desynchronize it from `floor_recovery_epoch` (`hub_announcement_json`, `recovery_epoch`, `hub_generation`, and the floor columns all write in this one transaction, so the target's probe endpoint and any later enrollment serve the new authority rather than the old hub's, with a test transferring to a target immediately after a lower-pair manual adoption proving the retained floor survives activation), and inserts a `staged_key_journal` row referencing the staged key, and finalization moves it into the identity location after commit, stamping the row finalized. That same activation transaction also deletes every `owner_sessions` row on the target: the target may itself have been a hub earlier (a prior transfer moved authority away from it), and an old cookie from that earlier tenure must not authenticate against the new fleet the moment this machine becomes authoritative again. A signed abort must name the announcement digest it is aborting: the target checks it against `pending_hub_authority.announcement_digest` and rejects (with a status the old hub uses to drop the queue row rather than retry) an abort for a digest it never staged or one whose `pending_hub_authority.state` is already `activated`, so a late-delivered abort from a failed earlier transfer can never tear down a different, successfully activated authority for the same machine. An abort that does match a still-staged digest must undo the listener surgery staging performed, not only the staged files: the `pending_hub_authority` row records the pre-transfer `fleet_settings` listener state in `prior_settings_json` when staging first replaces it, and the abort transaction restores that saved state, detaches any socket the transfer newly attached, narrows a reused agent socket back to its post-enrollment route allowlist, and disables the pending-hub readiness routes in the running process, so the target answers announcement and recovery pushes on the address the hub's registry still advertises rather than binding only the abandoned transfer address after its next restart. The same transaction deletes the staged key file, the staged snapshot, and the `pending_hub_authority` row, and returns `runtime_state.role` to `RuntimeRole.AGENT`, so an aborted target resumes agent mode holding no authoritative key material. A crash between the activation commit and the key move is repaired by the Task 3 startup journal repair, so an activated hub can always sign announcements, bundles, and future transfers. None of this replaces the target's machine-local state: the target is still a collecting agent during staging, so its own `report_outbox`, machine identity, and `runtime_state` row survive activation untouched, and the samples it buffered while the old hub was frozen are ingested by its own hub service after the switch through Task 5's role-entry outbox drain, which runs at activation and sweeps every unacknowledged row oldest first before live self-ingest begins (agent transport stops in `hub` role and live self-ingest only handles newly collected batches, so without the drain those staged-period rows would sit in the outbox until retention destroyed them). The snapshot side has the mirror rule: the old hub's local-only `report_outbox` rows are excluded from the transferred payload because they belong to the old machine, which keeps them and delivers them to the new hub as a normal agent after demotion. Readiness must fetch the real static login page, prove an invalid login gets the expected generic `401`, and submit a valid signed report under the committed proposed generation, but that report check is non-committing by design: the old hub constructs a dedicated probe payload rather than handing over a real outbox batch, the target runs the full ingestion path (signature, admission, inserts, contiguity) inside a transaction it rolls back deliberately, and the result leaves the target as a check digest, never as an `IngestAck` any agent outbox could consume. Nothing about the probe persists, so a draining commit failure that aborts the still-staged target discards no acknowledged sample and no machine sequence was ever consumed, with a test asserting the staged database is byte-identical before and after the readiness probe. The target returns a signed readiness receipt containing the generation and each check digest. + +Preflight also rejects the transfer outright while an unacknowledged `pending_recovery_key` row exists, with an explicit "acknowledge the recovery key before moving the hub" error and no state change: transferring the fleet signing key to the target while the owner has never confirmed the printable recovery key is saved would silently destroy the only other recovery path this fleet has, so the plan picks the branch of refusing the transfer rather than the branch of carrying the pending key across it. + +The old hub durably enters a transfer freeze before the snapshot is taken, and that freeze transaction is the write to `runtime_state.transfer_state` (`frozen`) and `runtime_state.transfer_id` (naming the new `hub_transfers` row) that the shared-names schema section describes: everything from here through draining reads that one column. The freeze transaction re-reads the target's machine row and requires it active: preflight ran before the freeze, so a revocation landing between them would otherwise ride into the snapshot while the transfer promotes the machine it just revoked. The same transaction rejects the transfer while any enrollment attempt is `claimed`: a claimed attempt's retry nonce is encrypted to the old hub's machine identity, which never transfers, so the new hub could not finish it and the enrollment would strand. Claims resolve or expire within the ten-minute pairing window, and expired or failed attempts do not block. The freeze rejects new report batches and every fleet mutation: enrollment, revocation, listener changes, integration changes, and recovery bundle creation or replication (Task 11), all reading the same `transfer_state` column. Rejecting reports means no acknowledgement watermark can advance past the snapshot, so every sample the old hub ever acknowledged is present on the target, and agents treat the rejection like any hub outage, buffering batches in the local outbox until delivery resumes against whichever hub wins. Rejecting mutations means the snapshot cannot go stale while staging and readiness run, so demotion can never restore a revoked machine or drop a post-snapshot enrollment or setting. Rejecting recovery bundle work means no bundle naming the pre-transfer generation and address set can be created or handed to an agent mid-transfer, which would otherwise let a promotion race the transfer to a stale outcome. Every fallible step after `enter_transfer_freeze()` and up to and including the `enter_draining_transfer()` commit runs inside the cleanup path that lifts the freeze on failure, so a failed transfer, including a draining transition whose journal transaction cannot commit, never leaves the old hub rejecting work until someone restarts it. Activation is only requested after draining commits, so every failure the cleanup path catches is still safely abortable. The cleanup never performs network work before the local lift: sending the abort to a target that just proved unreachable can itself fail, so the failure path inserts the row into `hub_transfer_aborts` and lifts the freeze (writing `transfer_state` back to `accepting`) in one local transaction, and a delivery loop retries the abort from that durable row, on the same 1, 2, 4, 8, 16, then 30 minute backoff Task 11's delivery loops use, until the target acknowledges it or reports the digest already resolved, either of which stamps `acknowledged_at` and stops the retry. A target holding staged or unannounced authority for a digest whose activation was never requested simply waits: it never announces on its own, so the old hub stays authoritative even while the abort is still undelivered. An old hub restarted while frozen consults its transfer journal (`runtime_state.transfer_id` into `hub_transfers`) to either continue the transfer or lift the freeze and resume. Draining subsumes the freeze from readiness onward. + +If staging, activation, or readiness fails before `enter_draining_transfer()` commits, the old hub sends a signed abort. The target deletes the staged authority and resumes agent mode while the old hub remains active with the freeze lifted. After readiness, `enter_draining_transfer()` writes `runtime_state.transfer_state` to `draining` in the same transaction that records the announcement on the `hub_transfers` row, so the old hub durably enters a draining state that rejects new reports and fleet mutations but retains its key while the target activates a target-owned publication obligation. Activation is idempotent by the signed announcement digest, and the authorization to drive it survives the authority flip: the target's internal transfer routes authenticate the staging hub's requests against `pending_hub_authority.staging_hub_machine_id` and `staging_hub_public_key`, persisted in the row itself so a target restarted after activation can still reconstruct exactly this authorization, scoped to that announcement digest, and that digest-scoped authorization remains answerable after activation until the old hub acknowledges the receipt. A retried `activate_publication_obligation` for an already-activated digest therefore returns the stored persistence receipt even though the target is already `HUB` and fleet-wide trust has rotated to the new key, so a lost first activation response can never leave the old hub draining forever against a target that no longer recognizes it. Once activation is requested, a timeout or lost response never calls `abort_pending_hub()`: the old hub remains draining, and `reconcile_draining_transfers()` (the background task described above) is what retries the same activation request against the persisted `hub_transfers` row until it obtains the persisted receipt, rather than a caller blocking inline on the original `/execute` request. It may send a signed abort only after the target reports that the digest was never activated. That obligation persists the signed announcement and a per-agent delivery outbox before any send attempt, retries after target restart, and returns a signed persistence receipt. Only after that receipt does the old hub atomically revoke sessions, persist retired agent state at the new generation, and remove its usable fleet signing key. If the old hub restarts while draining, its transfer journal requires the same status reconciliation before it can finish the accepted publication path or send the signed abort. A restarted old hub cannot mint a later generation, and a crash after local retirement leaves the target's durable delivery retry as the recovery path rather than relying on the retired hub. + +- [ ] **Step 5: Add Move Hub UI and evidence** + +Show preflight results including the target's detected private listener candidates and the explicit pending-recovery-key block when it applies, require the owner to pick the new hub's listener from them before execute, then poll `GET /api/v2/fleet/hub-transfer/status` for progress (draining, publication pending, complete) rather than blocking on the execute request, and show the exact new IP and port with a link to open it once complete. Capture successful transfer and forced rollback. Canary by failing the target validation probe and confirming the old hub remains active. + +- [ ] **Step 6: Verify and commit** + +Run focused transfer tests, full pytest, static assets, and the task surface. Commit as `feat: move fleet hub between enrolled PCs`. + +--- + +### Task 11: Recover a dead hub from encrypted bundles + +**Purpose:** Promote another PC without continuous replication. + +**Files:** +* Create: `agent/fleet/recovery.py` +* Create: `agent/routes/internal_recovery.py` +* Create: `agent/routes/v2_recovery.py` +* Create: `static/js/pages/recovery.js` +* Modify: `agent/fleet/models.py` +* Modify: `agent/fleet/transport.py` +* Modify: `agent/main.py` +* Modify: `static/index.html` +* Create: `tests/test_recovery_bundle.py` +* Create: `tests/test_dead_hub_recovery.py` +* Create: `tests/test_recovery_history_limits.py` +* Create: `tests/test_frontend_recovery.py` + +**Interfaces:** +* Produces: `RecoveryService.create_bundle() -> EncryptedRecoveryBundle` +* Produces: `RecoveryService.store_replica(bundle) -> ReplicaAcceptance` +* Produces: `RecoveryService.inspect(recovery_key: str) -> RecoveryPreview` +* Produces: `RecoveryService.promote(recovery_key: str, owner: NewOwner, listener: ListenerSelection) -> HubAnnouncement` +* Produces: `RecoveryService.run_replication_loop(interval: timedelta = timedelta(hours=6)) -> None` +* Produces: `GET /api/internal/v1/announcement` (every fleet member returns its accepted signed hub announcement for the recovery generation probe) +* Produces: `POST /api/v2/recovery/adopt-hub` (loopback-only manual hub adoption on any fleet member) + +Recovery decryption must authenticate before parsing: + +```python +def decrypt_bundle(bundle: EncryptedRecoveryBundle, recovery_key: str) -> RecoveryPayload: + private_key = decode_checked_recovery_key(recovery_key) + shared = private_key.exchange(bundle.ephemeral_public_key) + key = HKDF(algorithm=hashes.SHA256(), length=32, salt=bundle.salt, info=b"humwatch-recovery-v1").derive(shared) + plaintext = AESGCM(key).decrypt(bundle.nonce, bundle.ciphertext, bundle.associated_data()) + return RecoveryPayload.model_validate_json(plaintext) +``` + +- [ ] **Step 1: Write cryptographic and recovery tests** + +Cover wrong keys, tampering, a forged bundle built with the legitimate recovery public key but without the fleet signing key rejected at both replica acceptance and recovery preview, newest-valid selection by the tie-break-first rule then lexicographic `(recovery_epoch, generation, bundle_revision)`, including an older-epoch bundle of far higher generation losing to a newer-epoch bundle of lower generation and a same-epoch losing lineage's higher-revision bundle rejected outright at both replica acceptance and recovery selection, promotion minting at `(max_epoch + 1, 1)` where the epoch term is the highest epoch proven by the bundle, the machine's own floor evidence, or any verified probe response, including recovery during the post-transfer replication window (a reachable machine accepted `(E, G+1)` while the recovering machine and its bundle sit at `(E, G)`, and promotion at `(E+1, 1)` outranks both so that machine follows the new announcement), promotion by a machine that had manually adopted a lower pair and recovers again before the floor-report convergence ran (its own floor evidence contributes its epoch to the epoch term), a second recovery on top of a first (epoch 2 to epoch 3) proving the generation reset repeats cleanly, probe responses rejected when their announcement signature fails fleet-key verification, a probe response carrying a fabricated floor claim without fleet-signed evidence contributing nothing beyond its verified announcements (an attacker-supplied huge epoch or generation never reaches the promotion formula), preview failing loudly when a verified recovery epoch sits at the storage-safe ceiling, a probe answered correctly by a machine restarted after accepting a partially delivered announcement (served verbatim from its persisted `runtime_state.hub_announcement_json`), an unreachable machine listed in preview with the manual re-point warning, same-pair lower- and equal-revision rejection plus an old-epoch bundle rejected outright by the replica replacement rule, new owner password, session absence (including any `owner_sessions` rows already present on the promoting machine from an earlier tenure as hub, before promotion runs), integration revocation, promotion binding a listener owned by the recovering PC with the announcement advertising it (never the dead hub's addresses), promotion selecting the very address and port the machine's own agent listener already serves and succeeding through the `ListenerManager` reuse path instead of failing address-in-use, a bind failure on a genuinely new endpoint leaving the machine an agent, a crash between promotion commit and announcement delivery resuming the persisted obligation at startup, the watermark rewind handshake replaying retained acknowledged batches idempotently after generation adoption, a purged acknowledged prefix (sequences one through the oldest surviving batch minus one already deleted by Task 5 retention) producing a persisted recovery gap so the replay and later reports still acknowledge, an agent whose outbox is completely empty at rewind time (every acknowledged row already purged) still computing a disclosable gap from its persisted `sqlite_sequence` high-water mark and resuming acknowledgement instead of stalling forever, the promoting machine's own retained history replaying into its new hub through the same rewind and gap-disclosure logic driven against the local self-ingest path (covered in `tests/test_recovery_history_limits.py`), missing history disclosure, a machine enrolled after the newest surviving bundle's creation timestamp getting the distinguishable `unknown_machine` rejection surfaced on its local recovery page instead of an indefinite silent retry, that same machine completing the full path (rejection, Rejoin under its proven identity, finalize-triggered watermark rewind with gap disclosure, retained history acknowledged on the recovered hub), zero stored recovery private key bytes, immediate startup replication, fleet-change replication, periodic cadence, retry backoff, replica replacement on agents, the replication loop pausing while `transfer_state` is not `accepting` and resuming its periodic bundles and queued deliveries on the next tick after a lifted freeze with no restart, and the loop exiting for good once role leaves `RuntimeRole.HUB`. + +- [ ] **Step 2: Implement bundle creation and replication** + +Serialize fleet identity, machine registry, settings, integrations, schema version, `lineage_id`, recovery epoch, generation, a transactionally incremented positive `bundle_revision`, and timestamp. Bind `lineage_id`, recovery epoch, generation, and bundle revision into authenticated associated data as well as the encrypted payload. Encrypt with ephemeral X25519, HKDF-SHA256, and AES-GCM to the recovery public key. The encryption alone does not authenticate the sender, because every agent holds the recovery public key and could forge a valid higher-revision ciphertext, so the hub also signs each bundle with the fleet signing key over the `lineage_id`, recovery epoch, generation, revision, ciphertext digest, and associated data. The epoch rides every layer (payload, associated data, signature, the `recovery_bundles` storage key, and the replica ordering tuple) for the same reason announcements carry it: a promotion forced to rely on its local bundle alone (every peer unreachable) must still mint at one above an epoch the bundle actually proves, and a stale former hub producing fleet-signed bundles at its old epoch must be fenced by ordering, not trusted by revision arithmetic. Agents verify that signature with their retained fleet public key before replacing a replica, and recovery preview verifies it again before decrypting. Copy the ciphertext and signature to enrolled agents through signed internal requests. + +Start `run_replication_loop()` from the hub lifespan. Every tick, including the immediate startup tick, first checks `runtime_state.role` and `runtime_state.transfer_state`: it creates or replicates a bundle only while role is `RuntimeRole.HUB` and `transfer_state` is `accepting`, the same admission rule Task 10's freeze and drain transactions enforce on reports and mutations, so a bundle naming a pre-transfer generation and address set can never be created or handed to an agent mid-transfer. A non-`accepting` `transfer_state` pauses the loop rather than ending it: the tick skips its work and the loop keeps polling, so when a failed transfer's cleanup lifts the freeze back to `accepting` the very next tick resumes periodic bundles and queued delivery rows with no process restart. Only a role change away from `RuntimeRole.HUB` (transfer retirement or, on an agent that never held the role, simply never starting) exits the loop for good, since that database is no longer its to replicate. Create and distribute a bundle immediately on hub startup, after enrollment, revocation, listener or integration changes, and after a completed transfer. Also refresh every six hours. In one transaction, increment `runtime_state.recovery_bundle_revision`, insert the `(recovery_epoch, generation, bundle_revision)` bundle, and create one delivery row per active agent. An unchanged current bundle digest preserves its delivery state. A newer revision creates fresh pending rows, while acknowledgements name recovery epoch, generation, revision, and digest and are accepted only for that exact delivery. Failed deliveries durably update attempt count and next attempt time for retry after 1, 2, 4, 8, 16, then 30 minutes. Startup resumes every pending or due row. Agents persist their accepted `(lineage_id, recovery_epoch, generation, bundle_revision, digest)` and replace a replica under the same discipline announcements use, tie-break first, then ordering: a same-epoch bundle from a different lineage is admitted to the tuple comparison only when its lineage wins the deterministic tie-break against the retained one (a losing lineage's bundle is rejected whatever its generation or revision, because both partitioned recoveries hold the fleet signing key and the loser can sign arbitrarily high tuples), and an admitted bundle replaces atomically only when lexicographically higher on `(recovery_epoch, generation, bundle_revision)`, so a stale former hub's fleet-signed bundles at its old epoch are fenced by ordering and a same-epoch loser's by lineage, exactly like its announcements. Without the bundle-side tie-break, a winner hub dying before its next replication would let recovery resurrect the losing lineage's registry and settings, including machines revoked only on the winner. They reject an older or equal tuple with a different digest, retain the newest valid bundle across restart, and return the accepted epoch, generation, revision, and digest so the hub cannot acknowledge stale ciphertext. + +- [ ] **Step 3: Implement local recovery preview and promotion** + +Preview reports bundle age, generation, known PCs, and the oldest recoverable local sample, and states the bundle's creation timestamp with an explicit line that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion rather than expected to reconnect on its own: the bundle can only carry the machines the hub knew about when it last replicated. The recovered hub's own rejection of a report from an unregistered machine returns a status distinguishable from an ordinary signature or revocation failure (`unknown_machine`, not `unauthorized`), and that unregistered agent's local recovery page surfaces it plainly with the Task 4 Rejoin Fleet action as the stated remedy, since silently retrying forever would look identical to a network problem to the owner watching it. Preview also probes the generation actually accepted across the fleet, because announcement delivery can be partial before a hub dies: after a transfer, machine A may have accepted `G+1` while the recovering machine and its newest bundle are both still at `G`, and promoting at `G+1` would be rejected by A as an equal generation, leaving it pointed at the dead hub forever. Add `GET /api/internal/v1/announcement` on every fleet member, returning the machine's currently accepted signed hub announcement verbatim from `runtime_state.hub_announcement_json` (Task 2 schema), which acceptance writes atomically and which therefore survives restart: a machine that accepted a partially delivered `G+1` and rebooted still answers the probe with the exact `G+1` object. The response needs no additional authentication: the announcement is fleet-signed, so the prober verifies each returned announcement with its retained fleet public key and an attacker without the fleet key can neither forge a higher generation nor gain anything by replaying a lower one. The probe response carries two fleet-signed objects rather than any bare number: the machine's accepted announcement (`hub_announcement_json`) and its floor evidence announcement (`floor_announcement_json`), which differ only on a machine that manually adopted a lower generation. Both authenticate themselves under the fleet key, so the prober needs no trust in the responder at all: a LAN host or compromised machine can replay only announcements the fleet key actually signed, and no fabricated scalar, however large, ever reaches the generation formula. The storage-ceiling guard now lives where increments actually happen: promotion increments only the epoch (generation restarts at 1), so preview rejects an epoch whose increment would leave the storage-safe integer range, and the within-epoch generation increments (endpoint-change mints, transfer mints, convergence bumps) each reject a generation increment that would leave it, failing loudly rather than minting an unpersistable value. Preview applies the lineage tie-break against announcements, not merely against competing bundles, because an agent can be holding only the losing lineage's bundle: it stores lineage B's replica before convergence, then accepts winning lineage A's announcement, and A dies before its first replication ever reaches it, leaving no A bundle for the bundle-versus-bundle tie-break to prefer. A same-epoch bundle whose lineage loses the tie-break against the lineage of this machine's own accepted announcement, its floor evidence announcement, or any verified probe response is therefore refused automatically, whatever tuple it carries, with preview stating plainly that this bundle belongs to a retired lineage and that promoting it would resurrect a superseded registry (including machines revoked on the winner). One escape hatch matches the manual-adoption pattern: when no winning-lineage bundle exists anywhere, the owner may explicitly confirm promotion from the losing bundle at the machine's loopback recovery page, which is safe against split brain because promotion mints the next epoch above both lineages either way, and the confirmation is only accepting the loser's stale registry as the best data that still exists. Test it: retain B's bundle, accept A's announcement, kill A, and prove preview refuses automatically, states the reason, and proceeds only through the explicit confirmation. Preview probes every address in the imported registry, keeps the highest generation across every verified announcement in every response, and lists the machines that did not respond with the warning that an unreachable machine which accepted a newer generation must be re-pointed through the manual hub address entry on its local recovery page after promotion. The bundle's listener settings describe the dead hub's addresses and possibly its machine-local certificate paths, none of which belong to the recovering PC, so promotion requires a listener selection from this machine's detected private addresses, with the bind-before-commit discipline of Create Fleet adjusted for the machine promotion actually runs on: the promoting PC is normally an enrolled agent whose own Task 4 private listener already owns the selected address and port, so a naive fresh bind would fail address-in-use against the machine's own socket and leave recovery unusable. The selection therefore goes through Task 3's `ListenerManager` the way Task 10's transfer target does: an already-attached compatible socket counts as held (Uvicorn owning it is the proof it binds, the same rule Task 9's save-time validation applies), only a genuinely new endpoint is bound and held fresh, and a bind failure on a new endpoint fails visibly without changing role. The promotion commit then attaches any newly held socket so the recovered hub serves without a restart (a crash before the attach is repaired by hub startup binding the stored listener), and the dead hub's listeners and certificate paths are never carried forward. Promotion then creates a new owner password, deletes every `owner_sessions` row already present on this machine (it may have been a hub before, or briefly held transfer authority, and an old cookie from either must not authenticate against the recovered fleet), revokes integrations, zeroes every imported `last_acked_sequence`, switches to hub mode, and in the same transaction persists Task 10's durable per-agent announcement obligation for a signed announcement built from the newly bound listener, installing that same announcement as this machine's own accepted announcement and floor evidence (`hub_announcement_json`, `floor_announcement_json`, `recovery_epoch`, `hub_generation`, and the complete floor pair `floor_recovery_epoch` and `generation_floor` all advance atomically with the role change, the floor columns always matching the pair the evidence announcement proves, so the probe and manual adoption serve the recovered hub's announcement, never the dead hub's, the watermark verifier never sees a floor claim its evidence contradicts, and the above-floor rule cannot admit a high-generation announcement from the fenced old epoch through a stale `floor_recovery_epoch`), at recovery epoch one above the highest epoch carried by the bundle or by any verified probe response (the single epoch-incrementing path in this design, fencing the dead lineage's key-retaining hub from ever minting acceptable authority again) and generation 1, because the fresh epoch outranks every pair any machine holds lexicographically, so no generation arithmetic against the dead lineage is needed or wanted: the old formula's max over bundle, floor, and probed generations mixed epochs, could inflate the new lineage from obsolete evidence, and could refuse recovery outright when a dead lineage's generation sat at the storage ceiling, while a per-epoch restart at 1 is strictly higher than all of them under the pair rule. The bundle, the promoting machine's own durable floor pair, and every verified probe response (re-run at promotion time) still participate, but only through the epoch term: the minted epoch is one above the highest epoch any of those sources proves. The decrypted fleet signing key follows the same staged-file discipline as fleet creation: write it to a staging path first, have the promotion transaction insert a `staged_key_journal` row (Task 2 schema, so no schema change is needed here) referencing that staged key, and move it into the identity location as finalization, stamping the row finalized. A rollback deletes the staged file and the journal row rolls back, so a failed promotion leaves the agent holding no authoritative key material, and a crash between commit and finalization is repaired by the Task 3 startup journal repair, so a committed hub can always sign bundles and transfers. Cover failures on both sides of the role commit. Publication runs from that obligation, so a crash or lost response after the role commit retries the exact signed announcement at startup instead of leaving agents pointed at the dead hub. + +Rebuilding history needs an explicit rewind, because the imported registry carries the dead hub's acknowledgement watermarks while the recovered hub imports no central history: agents would treat their retained acknowledged outbox rows as delivered and the history would stay empty. Add `POST /api/internal/v1/reports/watermark`, a Task 4 signed internal request whose body carries the calling machine's accepted `(recovery_epoch, generation)` pair, its floor pair `(floor_recovery_epoch, generation_floor)`, and the fleet-signed floor evidence announcement from `floor_announcement_json` (the envelope's body digest authenticates the transport, and a tampering test proves a modified body fails verification), returning the hub's authority pair and stored watermark for that machine in a hub-signed response, the same discipline `IngestAck` follows and for the same reason: request signing proves nothing about the response, and an on-path host suppressing the request and echoing a watermark at or above the agent's local acknowledgement would cancel the one-time rewind and strand the recovered hub at watermark zero. The response signs the requesting machine ID, the request's nonce, the hub's current `(recovery_epoch, generation)` pair, and the watermark under the hub machine key, the agent verifies it against its persisted `runtime_state.hub_signing_public_key` before acting, any mismatch or bad signature is a failed request that retries, and a test proves a forged high-watermark response cannot suppress the rewind. The convergence rule consumes the evidence, not the scalar, and it bumps only for unresolved authority: the hub verifies the evidence announcement with the fleet key, requires its `(recovery_epoch, generation)` pair to equal the claimed floor pair, and resolves same-epoch conflicts by lineage rather than generation racing, because two machines recovering the same dead fleet while partitioned from each other can both legitimately mint the same fresh epoch. Evidence at the hub's own epoch naming a different hub lineage, at any pair equal to or above its own, invokes a deterministic tie-break: the lineage whose `lineage_id` is lexicographically smaller wins, a property minted at promotion time and carried verbatim through every same-lineage mint and transfer, so neither minting further generations nor transferring the role to another machine can change it. Different lineage means a differing `lineage_id`. A hub that loses the tie-break stops publishing and retires through exactly the fenced stale-epoch path (surfaced on its local recovery page with Rejoin as the remedy), and a hub that wins mints its fresh self-announcement at its own epoch and the evidence generation plus one, publishing through the normal obligations so every machine converges, including those that had accepted the losing lineage. The loser retiring instead of answering back is what makes leapfrog oscillation impossible. Same-lineage evidence strictly above the hub's own pair at its own epoch also mints at evidence generation plus one. Evidence carrying a strictly higher epoch than the hub's own is the fenced-hub case: the hub does not bump past it, it stops publishing and retires as Task 9's rejection handling describes. Evidence at a lower epoch, whatever its generation, is settled history and mints nothing, which is exactly the case the scalar floor got wrong: an old lineage's generation 100 reported to a recovered hub at `(2, 2)` must not drive it to 101. Evidence that is simply the hub's own current announcement is the healthy steady state after every ordinary enrollment, transfer, or recovery, and mints nothing, because bumping on equality would loop forever (each accepted bump re-triggers the watermark re-read at a floor equal to the new generation, climbing straight to the storage ceiling). The rule still rejects any floor whose increment would leave the storage-safe integer range. A compromised machine signing an enormous or maximal floor without fleet-signed evidence changes nothing, and a floor claim above its evidence is discarded, with tests for both. Agent transport re-reads it whenever its accepted generation changes and after every enrollment finalize, including a Rejoin: a machine missing from the recovery bundle re-enrolls against a hub whose announcement it may already have adopted, so the generation need not change while its fresh registry row still starts at watermark zero, and without the finalize trigger its retained acknowledged batches would never replay and contiguity would sit at zero forever. When the hub's watermark is lower than the agent's local acknowledgement, the agent resends every still-retained batch from the oldest recoverable sequence. `oldest_recoverable_sequence` is defined as the oldest surviving `report_outbox` row's sequence when the outbox retains at least one row, and as `last_allocated_sequence + 1` (the persisted `sqlite_sequence` high-water mark for `report_outbox`) when the outbox is completely empty, which Task 5's retention path makes possible for any agent that has been fully caught up and idle past the purge window. Without that empty-outbox case, an agent with nothing retained has no batch to anchor a gap disclosure on and its acknowledgement would never resume after promotion. The replay must first disclose what retention already destroyed: Task 5 deletes acknowledged outbox rows after seven days, so a mature agent's oldest surviving batch usually starts well above the recovered hub's zeroed watermark, and replaying from there alone would leave `highest_contiguous_sequence` stuck at zero with nothing ever acknowledged again. Before the replay, the agent persists and uploads a recovery `GapRange` from the hub watermark plus one through `oldest_recoverable_sequence - 1` (skipped entirely when the oldest retained batch is already contiguous with the hub watermark, and covering the whole prior range up to the high-water mark in the empty-outbox case), riding the same disclosed-gap path Task 5 built for retention, so contiguity advances across the purged prefix and the replayed batches acknowledge normally. Task 5's upload idempotency makes replayed previously-acknowledged rows harmless. + +The promoting machine's own history needs the identical treatment, because it never goes through agent transport at all: Task 5 has the hub ingest its own collected batches through a self-ingest path rather than an HTTP round trip, and that path only ever handles newly collected batches, so the promoting machine's pre-promotion retained history would otherwise never reach the fresh hub it just became. Promotion therefore runs the same watermark-rewind and gap-disclosure logic described above against the local self-ingest path for the self machine, using the same zeroed watermark and the same `oldest_recoverable_sequence` rule, so the promoting PC's own buffered samples land in central history exactly like every other agent's. + +The manual re-point warning from preview has a concrete flow behind it, because a machine that was unreachable during the probe and had accepted a generation above the recovered hub's announcement would otherwise reject that announcement forever. Every installation's local recovery page (loopback only, like the setup surface) carries manual hub address entry backed by `POST /api/v2/recovery/adopt-hub`: the agent fetches `GET /api/internal/v1/announcement` from the entered address, verifies the fleet signature and matching fleet UUID, and shows the owner the generation comparison. On explicit confirmation it adopts the announcement even when the generation is not higher than its locally accepted one. This loopback flow is the only path that may accept a non-higher generation: it is gated on the same physical access to the machine that could wipe and re-enroll it outright, while network announcement delivery keeps the strict higher-generation rule unchanged, so split-brain protection is not weakened for any remote path. Adoption persists the accepted hub and generation and triggers the transport's watermark re-read, so the recovery gap and replay handshake above run unchanged and the machine's retained history lands on the recovered hub. Adoption never lowers the floor pair: `(floor_recovery_epoch, generation_floor)` becomes the lexicographic maximum of its pre-adoption value and the adopted announcement's pair, with the corresponding evidence announcement in `floor_announcement_json`, so adopting a recovered hub at `(2, 2)` from a dead-lineage floor of `(1, 100)` advances the floor to `(2, 2)` while a same-epoch adoption of a lower generation keeps the higher pre-adoption floor. Remote announcement delivery keeps requiring a pair lexicographically above the floor, so a delayed publication retry or a replay of the dead lineage's announcement cannot redirect the adopted machine back, whatever its generation. Any remaining floor debt clears itself fleet-wide: the signed watermark request after adoption carries the agent's floor pair in its authenticated body, and a hub that reads verified floor evidence representing unresolved authority (a higher pair at its own epoch, or an equal pair from a different lineage, never its own current announcement echoed back, and never a lower epoch's settled history) re-announces above it per the convergence rule and publishes through the normal per-agent obligation, which every machine including the adopted one accepts under the strict rule, converging the fleet so no future replay of the dead lineage can ever be admitted anywhere. Test it end to end with an unprobed agent whose accepted generation exceeds the recovered hub's: automatic delivery is rejected, a loopback adoption with a wrong-fleet or tampered announcement is refused, a confirmed adoption succeeds, the replayed history acknowledges, a subsequent replay of the pre-adoption higher-generation announcement is rejected by the floor, and the floor report drives the recovered hub to re-announce above it with every machine converging on the new generation. Test the fencing scenario codex-style too: a hub locally installs an undistributed generation, dies, the fleet recovers past it at the next epoch, the former hub restarts with an endpoint change and mints an old-epoch announcement of arbitrarily higher generation, every agent rejects it under the pair rule, and the rejection evidence drives the returning hub into its fenced retirement state. And test simultaneous recovery: two partitioned machines promote from the same bundle, both minting the same fresh epoch at generation 1, connectivity returns, the deterministic tie-break retires the larger-`lineage_id` lineage visibly while the winner re-announces above every pair the loser distributed, a variant where the loser out-minted the winner during the partition (loser at generation 2, winner at 1) proves winning-lineage agents reject the loser's higher pair outright through the tie-break-first acceptance rule and the returned evidence still drives the winner's converging mint, all agents converge on the winner, repeated evidence exchanges after convergence mint nothing further, and a variant where one side transfers its hub role to a third machine before convergence proves the tie-break outcome is unchanged because the transfer carried the `lineage_id` verbatim. + +- [ ] **Step 4: Add the Recover Fleet page and evidence** + +Keep recovery copy literal. Capture valid preview, invalid key, recovery confirmation, and post-recovery login. Canary by flipping one ciphertext byte and proving preview refuses it. + +- [ ] **Step 5: Verify and commit** + +Run focused recovery tests, full pytest, static assets, and browser smoke. Commit as `feat: recover fleets after hub loss`. + +--- + +### Task 12: Add scoped integration pairing for future SlipShell monitoring + +**Purpose:** Ship the HumWatch half of the integration without adding SlipShell UI work. + +**Files:** +* Create: `agent/fleet/integrations.py` +* Create: `agent/routes/v2_integrations.py` +* Create: `agent/security/integration_auth.py` +* Create: `tests/test_integration_pairing.py` +* Create: `tests/test_integration_scopes.py` +* Create: `tests/test_slipshell_contract.py` +* Create: `tests/fixtures/slipshell/contract.json` +* Modify: `agent/fleet/models.py` +* Modify: `agent/main.py` +* Modify: `agent/routes/v2_fleet.py` +* Modify: `agent/routes/v2_history.py` +* Modify: `agent/routes/v2_events.py` +* Modify: `agent/routes/v2_machines.py` +* Modify: `static/js/pages/fleet-settings.js` +* Modify: `static/js/api.js` +* Modify: `static/vendor/SHA256SUMS` + +**Interfaces:** +* Produces: `POST /api/v2/integrations/pairing-code` +* Produces: `POST /api/v2/integrations/exchange` +* Produces: `GET /api/v2/integrations` +* Produces: `DELETE /api/v2/integrations/{integration_id}` +* Produces read-only scopes: `fleet:read`, `machines:read`, `metrics:read`, `history:read`, `events:read` + +Public read routes accept either an owner session or the named scope: + +```python +def owner_or_scope(required_scope: IntegrationScope): + async def dependency(request: Request) -> RequestPrincipal: + session = await optional_owner_session(request) + if session is not None: + return RequestPrincipal.owner(session) + integration = await verify_integration_request(request) + integration.require_scope(required_scope) + return RequestPrincipal.integration(integration) + return dependency +``` + +- [ ] **Step 1: Write pairing and scope tests** + +Assert ten-minute one-use pairing, independent credential revocation, no owner or agent credential reuse, read-only enforcement, stable machine UUIDs, and access to current, history, availability, and events. Exercise every existing Task 6 route through its real router with an owner session, its required integration scope, a missing scope, and a revoked integration. Assert exchange idempotency: an identical retry of `POST /api/v2/integrations/exchange` with the same pairing code and the same client public key returns the same integration record rather than minting a second one, and a retry against a code already consumed by a different client public key returns `409`. + +- [ ] **Step 2: Implement integration request authentication** + +Task 6 already revalidates the owner session on every event and heartbeat write and cancels open streams at session revocation. This task extends exactly that mechanism to the second principal kind: the events service revalidates an integration principal the same way, and deleting an integration publishes the same eager cancellation. Test that an open integration stream dies at revocation before the next heartbeat lands. Use the same Ed25519 signed request envelope as agents, but look up `integration_clients` and enforce scope per route: an integration signs with its `integration_id` in the machine slot of that envelope (`X-HumWatch-Machine`) and `principal_type` `integration` in the corresponding `used_nonces` row (Task 2 schema), so integration replay protection never shares key space with a fleet machine's nonces even though the wire shape is identical. `POST /api/v2/integrations/exchange` is idempotent on `(pairing_code_id, client public key)`, and that binding is durable, not in-memory: creating a pairing code inserts an `integration_pairing_codes` row (Task 2 schema) carrying the owner-approved grant in `approved_name` and `approved_scopes_json`, decided by the owner at generation time, and exchange stamps `consumed_at`, `integration_id`, and `client_public_key` on that row in the same transaction that creates the `integration_clients` credential, copying the name and `scopes_json` from those approved columns and nothing else. The exchange request never carries a name or scope set, because an unauthenticated caller offered that field would grant itself access, and the durable row is what lets a hub restarted between code generation and exchange still know exactly what the owner approved. Test that an exchange body attempting to smuggle scopes is rejected and the minted credential holds only the approved set. A retry presenting the same client public key against the same still-valid or already-consumed code looks the binding up from that row and returns the existing integration record unchanged, surviving a hub restart between the commit and the retry, while a different client public key against a code that has already been consumed by some other key returns `409` rather than silently minting a competing credential. The retention pass purges expired unconsumed rows with the other aged tables. Change the Task 6 router dependencies to `owner_or_scope(...)`: `fleet:read` for fleet discovery, `machines:read` for machine metadata and availability, `metrics:read` for current metrics and processes, `history:read` for history, and `events:read` for events. Integration credentials never receive CSRF cookies and every state-changing route remains owner-only. + +- [ ] **Step 3: Publish a machine-readable contract fixture** + +Add one versioned `tests/fixtures/slipshell/contract.json` containing fleet discovery, machine current state, history, availability, and event frames. `tests/test_slipshell_contract.py` validates every fixture against the live Pydantic response models and asserts UUID, timestamp, units, stale threshold, gap range, and backfill field names. This is the stable HumWatch half of the future SlipShell integration. + +- [ ] **Step 4: Add integration administration UI and evidence** + +Show paired clients, scopes, creation time, and revoke. Generate a short-lived pairing code without exposing stored credentials, with the owner naming the client and selecting its scope set at generation (persisted on the pairing row per Step 2). Canary by attempting a write with an integration identity and require `403`. + +- [ ] **Step 5: Verify and commit** + +Run focused integration tests, v2 API tests, full pytest, static assets, and browser smoke. Commit as `feat: add read-only monitoring integrations`. + +--- + +### Task 13: Migrate 2.0 installations through an explicit wizard + +**Purpose:** Convert existing nodes without silently keeping the broken security and peer model. + +**Files:** +* Create: `agent/fleet/upgrade.py` +* Create: `agent/routes/v2_migration.py` +* Create: `static/js/pages/migration.js` +* Create: `tests/test_fleet_upgrade.py` +* Create: `tests/test_legacy_api_compatibility.py` +* Create: `tests/test_frontend_migration.py` +* Create: `tests/fixtures/v2_0_database.py` +* Modify: `agent/fleet/state.py` +* Modify: `agent/migrations.py` +* Modify: `agent/main.py` +* Modify: `agent/routes/health.py` +* Modify: `static/index.html` +* Modify: `static/js/app.js` +* Modify: `static/js/auth.js` +* Modify: `static/js/pages/setup.js` +* Modify: `static/vendor/SHA256SUMS` +* Modify: `tests/test_database_migration.py` + +**Interfaces:** +* Produces: `UpgradeService.inspect() -> UpgradePreview` +* Produces: `UpgradeService.create_fleet(owner: NewOwner, listener: ListenerSelection) -> UpgradeResult` +* Produces: `UpgradeService.prepare_join(listener: ListenerSelection) -> PairingDisplay` +* Produces: `UpgradeService.enable_legacy_api(enabled: bool) -> None` + +Inspection returns proof, not a boolean: + +```python +class UpgradePreview(BaseModel): + source_database: str + source_sha256: str + metric_rows: int + process_rows: int + oldest_sample: datetime | None + newest_sample: datetime | None + archive_path: str + safe_to_continue: bool + blocking_errors: list[str] +``` + +- [ ] **Step 1: Build a real 2.0 fixture and continuity tests** + +The fixture contains machine information, metrics, process history, token paths, certificate paths, and CORS origins. Assert row counts and sample values before and after Create Fleet and Join Fleet conversions. + +- [ ] **Step 2: Implement safety inspection and archive** + +Use SQLite backup, validate schema and machine rows, copy config and security path metadata, and record a manifest with source paths and SHA256 digests. Stop migration with a visible error when proof fails. + +- [ ] **Step 3: Implement Create Fleet and Join Fleet conversions** + +Create Fleet imports history under the stable self machine UUID. Join Fleet converts history to ordered outbox batches and uploads through Task 5. Neither path deletes original runtime files. + +Both conversions take a listener selection and follow the same bind-before-commit rule as fresh setup: the legacy listener may be loopback-only, a forbidden wildcard, or archived with the legacy security settings, so Create Fleet reserves and persists a private hub listener (Task 3's rules) and Join Fleet selects the restricted private enrollment listener Task 4 requires, in each case securing the socket before the role transition and failing visibly without changing role when that fails. Securing does not always mean a fresh bind, because the migration wizard itself is being served by the live legacy socket on port 9100 and a typical 2.0 `agent/main.py` bound it as a wildcard: when the selected address and port are already covered by the process's own active specific-address socket, the conversion adopts that socket through the `ListenerManager` handoff instead of attempting a second bind that would fail address-in-use, and the role commit narrows its route set. A wildcard socket cannot be adopted, because a bound socket's kernel local address is immutable and `0.0.0.0` would leave the converted role reachable on every interface against the exact-listener rule, so the wildcard case is a coordinated replacement inside the conversion: stop accepting on the legacy wildcard listener (the wizard's own already-accepted connection survives socket closure and completes normally), bind the selected private addresses on the freed port, and only then commit the role. A bind failure in that window rebinds the wildcard and fails the conversion visibly with the role unchanged, so the wizard stays reachable for a retry. Only a selection the process does not cover at all is freshly bound with no replacement dance. Test all three paths: conversion from a wizard served on the legacy wildcard replaces it with no address-in-use error and the post-commit listener answers only on the selected addresses (a request to a non-selected interface fails), a bind failure mid-replacement restores the wildcard wizard with the role unchanged, and a specific-address adoption narrows routes without rebinding. The wizard offers the machine's detected private addresses for the choice. + +The migration router is not open: an upgraded legacy installation is reachable on its existing LAN listener, and these routes replace runtime authority and credentials before any owner account exists. Inspection and every conversion mutation require the installation's existing legacy bearer token, exactly as the protected v1 routers do, or an actual loopback peer. Add unauthorized-route tests proving a tokenless LAN request cannot inspect or convert. + +- [ ] **Step 4: Gate protected v1 endpoints by mode** + +Fresh installations have only public v1 health. Upgraded installations default protected v1 off, can explicitly enable it with the archived bearer token, and can remove compatibility permanently. + +- [ ] **Step 5: Implement the migration wizard and peer suggestions** + +Read `humwatch_machines` from browser local storage as untrusted enrollment suggestions. Clear it only after owner confirmation. Present Create Fleet and Join Fleet with data counts and exact archive location. This task also deletes the general role-scoped legacy token client Task 7 retained, but not the wizard's own: a legacy installation migrated from a browser on another LAN machine has no owner account yet, and its inspection and conversion requests must carry the existing bearer token, so the wizard keeps a wizard-scoped token prompt and client used only by the migration pages against a still-legacy installation and discarded the moment conversion commits. The flipped Task 7 role-branch test asserts the token path survives nowhere except inside the wizard flow, and a test drives a full remote conversion through the wizard-scoped token from a browser that never had the general client. + +- [ ] **Step 6: Produce migration evidence and canary** + +Run one Create Fleet and one Join Fleet conversion against copied 2.0 fixtures. Canary by corrupting the backup and proving the source remains untouched and the wizard blocks completion. + +- [ ] **Step 7: Verify and commit** + +Run focused migration tests, all v1 and v2 auth tests, full pytest, static assets, and browser smoke. Commit as `feat: migrate v2 nodes into fleets`. + +--- + +### Task 14: Transition Windows and Linux installers to the fleet model + +**Purpose:** Make fresh setup boring and detect duplicate HumWatch services without harming unrelated processes. + +**Files:** +* Modify: `installer/service-setup.ps1` +* Modify: `installer/HumWatch.iss` +* Modify: `installer/linux/humwatch.service` +* Modify: `scripts/install-service.ps1` +* Modify: `scripts/provision-security.ps1` +* Modify: `scripts/provision-security.sh` +* Modify: `scripts/build-deb.sh` +* Modify: `run.bat` +* Modify: `run-no-admin.bat` +* Modify: `run.sh` +* Modify: `tests/test_windows_service_security.py` +* Modify: `tests/test_linux_service_security.py` +* Modify: `tests/test_security_provisioning.py` +* Modify: `tests/test_installation_docs.py` +* Create: `tests/test_duplicate_service_detection.py` +* Modify: `.github/workflows/build-windows-installer.yml` + +**Interfaces:** +* Produces installer state under the existing protected runtime root +* Produces `Find-HumWatchServiceConflicts -Port 9100` and `Disable-ObsoleteHumWatchService` +* Produces `find_humwatch_service_conflicts 9100` for Linux package post-install + +The Windows detector returns records and never mutates during discovery: + +```powershell +function Find-HumWatchServiceConflicts { + param([int]$Port = 9100) + $services = Get-CimInstance Win32_Service | + Where-Object { $_.Name -like '*HumWatch*' } | + ForEach-Object { + [pscustomobject]@{ + Kind = 'service' + Name = $_.Name + ProcessId = $_.ProcessId + State = $_.State + PathName = $_.PathName + KnownHumWatchPath = Test-KnownHumWatchPath $_.PathName + } + } + $owners = Get-NetTCPConnection -State Listen -LocalPort $Port -ErrorAction SilentlyContinue | + ForEach-Object { + $process = Get-Process -Id $_.OwningProcess -ErrorAction SilentlyContinue + [pscustomobject]@{ + Kind = 'port-owner' + Port = $Port + ProcessId = $_.OwningProcess + ProcessName = $process.ProcessName + PathName = $process.Path + KnownHumWatchPath = Test-KnownHumWatchPath $process.Path + } + } + @($services) + @($owners) +} +``` + +The Linux detector parses `systemctl --user`, system `systemctl`, and `ss -H -ltnp "sport = :$port"` into the same service and port-owner concepts. Treat missing process metadata as unknown, never as HumWatch. + +- [ ] **Step 1: Write installer behavior tests first** + +Fresh install must create identity directories, no bearer token, no certificate, and `unconfigured` mode. Upgrade must preserve the archive. Duplicate detection must identify service name, executable path, user or system scope, and port conflict. + +- [ ] **Step 2: Remove mandatory security provisioning from fresh setup** + +Keep protected directories and ACLs. Stop generating token and self-signed certificate files by default. Preserve explicit operator TLS files during upgrades. + +- [ ] **Step 3: Add duplicate service detection with confirmation** + +Discovery never mutates. Offer to disable only a service whose executable or working directory resolves below a known HumWatch installation root. Port ownership alone is never enough. Report unrelated and unknown port owners with PID, executable when readable, and scope without terminating or disabling them. Tests cover a renamed HumWatch service on 9100, a stale known service on another port, and an unrelated process on 9100 on both platforms. + +- [ ] **Step 4: Update CI installation contracts** + +Exercise fresh Windows portable, Windows service, Linux package, upgrade preservation, duplicate user plus system service, and unrelated port owner cases. + +- [ ] **Step 5: Run packaging evidence and canary** + +Build the deb and Windows installer workflow fixtures. Canary with a fake unrelated process on 9100 and prove setup reports it without stopping it. + +- [ ] **Step 6: Verify and commit** + +Run installer-focused tests, package builds, full pytest, workflow YAML parsing, and script syntax checks. Commit as `feat: install fleet-ready HumWatch nodes`. + +--- + +### Task 15: Prove the complete 2.1.0 flow and prepare the release + +**Purpose:** Validate the product on real Windows and Linux machines before changing the version. + +**Files:** +* Create: `scripts/verify-fleet-e2e.py` +* Create: `tests/test_fleet_e2e_harness.py` +* Create: `docs/2.1-UPGRADE.md` +* Create: `docs/2.1-ACCEPTANCE.md` +* Modify: `README.md` +* Create: `docs/START-HERE.md` +* Modify: `HumWatch-Spec.md` +* Modify: `agent/__init__.py` +* Modify: `tests/test_installation_docs.py` +* Modify: `tests/test_security_contract.py` +* Modify: `tests/test_supply_chain_policy.py` +* Modify: `.github/workflows/build-windows-installer.yml` +* Modify: `scripts/build-release.ps1` +* Modify: `scripts/build-linux-portable.sh` +* Modify: `scripts/verify-static-assets.py` +* Modify: `static/vendor/SHA256SUMS` + +**Interfaces:** +* Produces: `python scripts/verify-fleet-e2e.py --hub URL --owner USER` +* Produces: release version `2.1.0` + +The verifier emits fixed sanitized checkpoints: + +```python +CHECKPOINTS = ( + "login", + "enrollment", + "live-report", + "machine-switch", + "offline-buffer", + "ordered-backfill", + "hub-transfer", + "recovery-preview", + "integration-read", + "revocation", +) +``` + +- [ ] **Step 1: Write the deterministic E2E harness test** + +The harness accepts secrets through hidden prompts or protected files, never command arguments. It prints sanitized checkpoints for login, enrollment, live report, machine switch, offline buffer, ordered backfill, transfer, recovery preview, integration read, and revocation. + +- [ ] **Step 2: Run the complete automated gate** + +```bash +.venv/bin/python -m pytest -q +.venv/bin/python scripts/verify-static-assets.py +node tests/browser_globals_smoke.js +python -m compileall -q agent scripts +bash -n run.sh setup.sh scripts/provision-security.sh scripts/build-deb.sh scripts/build-linux-portable.sh +``` + +- [ ] **Step 3: Prove the failure canaries** + +Corrupt one static digest and require static verification failure. Tamper with one signed report and require ingestion rejection. Corrupt one recovery bundle and require preview failure. Restore all fixtures and rerun the green gate. + +- [ ] **Step 4: Run real Windows and Linux acceptance** + +Use at least one Windows PC and one Linux PC. Record: + +1. Hub creation over ordinary LAN. +2. Agent enrollment using one temporary code. +3. Optional Tailscale reconnection. +4. One browser login and in-app machine switching. +5. Hub outage, continued collection, and ordered backfill. +6. Planned hub transfer to the other platform. +7. Controlled dead hub recovery. +8. Real 2.0 database upgrade with sample continuity. +9. Independent machine and integration revocation. +10. HTTP default plus optional HTTPS. + +Save logs and screenshots below `artifacts/2.1.0-acceptance/`. Redact addresses only if the artifact will be public. Never include secrets. + +- [ ] **Step 5: Rewrite setup and security documentation** + +Make IP plus port, owner login, Create Fleet, Join Fleet, recovery key, optional Tailscale, and optional HTTPS the primary flow. Move shared bearer token, CORS, certificate trust, and peer browser calls into a clearly labeled 2.0 compatibility section. + +- [ ] **Step 6: Change the version only after acceptance passes** + +Set `agent.__version__` to `2.1.0`, rebuild release artifacts, and rerun the complete gate. + +- [ ] **Step 7: Complete the review receipt and release PR** + +Run the independent review loop against current HEAD until zero new findings. Write the receipt with the real acceptance evidence directory. Run gitleaks. Push and open a ready-for-review PR. Do not create the `v2.1.0` tag until CI, Windows installer proof, Linux package proof, and the receipt all pass on the exact release commit. + +Commit message: `release: prepare HumWatch 2.1.0` + +## Per-task completion gate + +Every task ends with this sequence after its implementation commit: + +1. Run all focused tests named in the task. +2. Run the full pytest suite. +3. Run static asset and browser checks when browser files changed. +4. Run package or script checks when installer files changed. +5. Produce running surface evidence for every touched UI or endpoint. +6. Prove the relevant checker with one deliberate failure, then restore and rerun. +7. Run one independent review against current HEAD. +8. Fix every actionable finding and commit. +9. Repeat review against the new HEAD until zero new findings. +10. Write `/review-receipt.json` with current HEAD and evidence path. +11. Run `~/.local/bin/gitleaks git --log-opts="--all --not --remotes"`. +12. Push and open a ready-for-review PR. + +Do not carry a receipt across a fix commit. Do not treat a merged dependency PR as proof that the next task passes. Each task begins from freshly fetched and verified `origin/main` after its dependencies merge. + +## New-session reentry + +The implementation session should begin with: + +```bash +cd /projects/HumWatch +git fetch origin +git status --short +git worktree list --porcelain +``` + +Then inspect current open PRs and issue #14. The planning branch is `codex/humwatch-2.1-fleet-hub-design` at or after commit `32fc1b4`. Do not implement on the existing `docs/start-here-guide` checkout. Create a fresh worktree and branch for Task 1 from the latest merged `origin/main` after the design and plan PR is merged. + +Open PR #16 changes the old token gate. Recheck it before Task 7. If it is still open and Task 7 replaces that gate, close it as superseded instead of carrying dead CSS into 2.1.0. Open PR #15 documents the 2.0 setup path. Recheck it before Task 15 and rewrite any merged content that still makes mandatory TLS or shared tokens the normal path. diff --git a/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md new file mode 100644 index 0000000..9f14763 --- /dev/null +++ b/docs/superpowers/specs/2026-08-11-humwatch-2.1-fleet-hub-design.md @@ -0,0 +1,548 @@ +# HumWatch 2.1.0 Fleet Hub Design + +**Status:** Approved by Jeff on 2026-08-11 +**Target release:** 2.1.0 +**Scope:** Replace the fragmented multi-machine browser flow with one movable fleet hub, one owner login, and one-time agent enrollment. + +## Outcome + +HumWatch 2.1.0 has one normal entry point: + +```text +http://:9100 +``` + +The owner signs in with a username and password, then selects any enrolled PC inside the same dashboard. Ordinary LAN access works without Tailscale. Tailscale access works when available. HTTPS is optional. + +Tokens are not part of normal browser use. A short-lived pairing code is used only when a PC or future integration joins the hub. Each enrolled participant receives its own cryptographic identity and can be revoked without affecting the rest of the fleet. + +The hub role is movable. Every HumWatch installation is capable of becoming the hub through the web application. + +## Why 2.0.x regressed + +The 2.0 security model protected the network surface, but it split the user experience into separate browser origins and manual deployment work: + +* The browser bearer token lives in `sessionStorage`, which is scoped to one origin and browser session. +* The Machines page opens another PC's dashboard in a new tab instead of switching the active machine inside one application. +* Each installer creates a random token, while multi-machine discovery expects every machine to share the same token. +* Browser access to peers depends on CORS origin lists. +* Peer discovery requires certificate trust that the default per-machine self-signed certificates do not provide across LAN and Tailscale addresses. +* Machine lists are stored in each browser rather than owned by the fleet. + +A live deployment check found multiple healthy 2.0.2 agents reachable through Tailscale, while discovery still returned no peers because their certificates and credentials did not form a usable fleet trust model. + +The fix is architectural. More token documentation or certificate setup would preserve the regression. + +## Product decisions + +1. One installation package supports unconfigured, agent, and hub states. +2. One hub owns the dashboard, owner authentication, fleet registry, and central history. +3. The browser communicates only with the hub during normal use. +4. Agents collect data, keep a local rolling buffer, and report to the hub. +5. One owner account is supported in 2.1.0. +6. HTTP over a trusted private LAN is the default. +7. Tailscale is detected and supported, but never required. +8. HTTPS is optional. +9. Enrollment codes are short-lived and single use. +10. Every agent has an independent device identity. +11. The hub can move to another enrolled PC. +12. A printable recovery key supports recovery after hub loss. +13. The fleet API is a stable boundary for a future SlipShell integration. + +## System states + +### Unconfigured + +First run offers two actions: + +* **Create Fleet** makes this PC the hub. +* **Join Fleet** keeps this PC as an agent and shows its address plus a pairing code. + +An upgraded 2.0.x installation enters a migration version of this screen after its existing data is secured. + +### Hub + +The hub performs five jobs: + +* Serves the fleet dashboard. +* Authenticates the owner and manages browser sessions. +* Owns the machine registry and fleet settings. +* Accepts signed reports and history backfill from agents. +* Stores central fleet history. + +The hub also runs the collector locally. Its hardware appears as an ordinary machine in the fleet. + +### Agent + +An agent performs four jobs: + +* Collects local hardware data. +* Stores a rolling local history buffer. +* Sends live reports and ordered backfill to the hub. +* Serves a small local setup and recovery page. + +An agent does not serve the full fleet dashboard during normal operation. + +## Component boundaries + +The implementation must separate these responsibilities even if they continue to run in one Python process: + +### Collector + +Owns sensor discovery, sampling, normalization, and the local measurement stream. It does not know about browser sessions or fleet presentation. + +### Local history buffer + +Persists samples before hub acknowledgement. It owns sequence allocation, retention, and backfill reads. The default retention remains seven days unless a later product decision changes it. + +### Agent transport + +Packages measurements into signed, ordered batches. It owns reconnect behavior, backoff, acknowledgement handling, and hub announcements. + +### Fleet registry + +Owns fleet identity, machine identity, display names, enrollment status, revocation state, hub addresses, and hub generation. + +### Hub ingestion + +Verifies agent identity, rejects replayed batches, stores accepted samples, and advances each machine's acknowledgement watermark. + +### Owner authentication + +Owns password verification, browser sessions, recovery, login throttling, logout, and session revocation. It does not authenticate agents. + +### Fleet web API + +Exposes the stable `/api/v2` browser and integration contract. It queries the fleet registry and central history through service boundaries rather than reading their tables directly. + +### Local setup surface + +Owns first run, pairing code display, migration choices, and recovery actions. It remains small and literal. + +## Identity model + +### Fleet identity + +Creating a fleet generates: + +* A random fleet UUID. +* A fleet signing key pair. +* A recovery encryption key pair. +* Hub generation `1`. +* A stored private hub listener, selected from the machine's detected private addresses and bound before setup reports success, so enrollment and agent reports are reachable immediately. +* A printable recovery key that encodes the recovery private key. It is shown to the owner until they confirm it is saved. That acknowledgement deletes the protected pending record holding it, and a setup response lost in transit is recovered by logging in at the hub PC and retrieving the key before acknowledging. + +Agents retain the fleet public signing key and recovery public key. Only the active hub holds the fleet private signing key in usable form. After the owner's acknowledgement, HumWatch does not retain the recovery private key. + +### Machine identity + +Each installation generates a stable random machine UUID, an Ed25519 signing key pair, and an X25519 encryption key pair. The private keys stay on that PC in protected service-owned files. Enrollment registers the machine UUID and both public keys with the hub. + +Agent report envelopes include: + +* Fleet UUID. +* Machine UUID. +* Monotonic sequence range. +* Batch identifier. +* Capture timestamps. +* Payload digest. +* Agent signature. + +The hub verifies the signature, fleet membership, revocation state, sequence range, and batch identifier before accepting data. + +Signatures authenticate data and hub movement. They do not hide telemetry from someone who can observe an ordinary HTTP LAN connection. HTTPS or Tailscale supplies confidentiality when required. + +### Owner identity + +2.1.0 supports one owner account with a username and password. The password is stored with Python's `scrypt`, a random salt, and versioned work parameters. Parameters are calibrated for supported hardware and stored with the hash so they can be upgraded after a successful login. + +No recoverable password is stored. + +### Browser sessions + +Successful login creates a random server-side session. The browser receives an `HttpOnly`, `SameSite=Strict` cookie. Sessions default to 30 days and can be revoked individually or all at once. + +The cookie gains the `Secure` attribute when HTTPS is enabled. State-changing browser requests require CSRF protection. Login responses use generic failure text and rate limiting by account plus source address. + +Hub transfer and dead hub recovery revoke every browser session. + +### Integration identity + +Future integrations use their own revocable, read-only identity. They never receive the owner's password, browser cookie, agent private key, or fleet signing key. + +## Agent enrollment + +1. An unconfigured agent generates a random single-use pairing code. +2. The setup page or installer on that PC shows the agent address and pairing code locally. +3. The code expires after ten minutes and after one successful use. +4. The owner opens **Add PC** on the hub and enters the address plus code. +5. The hub confirms the agent's machine UUID and public key. +6. The agent confirms the fleet UUID, fleet public key, hub addresses, and signed enrollment record. +7. The hub adds the machine to the fleet registry. +8. The agent sends its current state and available buffered history. +9. The hub marks enrollment complete only after a signed report is stored. + +Pairing attempts are rate limited. The agent must show whether a code is active, expired, or already used without disclosing fleet secrets. + +The pairing code is never returned by a network API. The unconfigured agent exposes its enrollment endpoint on selected private listeners, while code display remains local to the target PC. + +Plain HTTP enrollment assumes a trusted LAN. The setup screen must state that limitation. Tailscale or optional HTTPS protects the exchange on networks that are not trusted. + +Removing a machine revokes its public key and rejects later reports. Re-enrollment creates a new enrollment record without changing other machines. + +## Network behavior + +### Default access + +The hub presents ordinary LAN and Tailscale addresses as IP plus port. The primary display format is: + +```text +http://192.168.x.x:9100 +http://100.x.x.x:9100 +``` + +The first run screen lists reachable private addresses and lets the owner choose which listeners to enable. + +### Listener policy + +Fresh installs enable localhost plus selected private LAN listeners. A detected Tailscale address is offered as another private listener. HumWatch must not silently expose the hub through a public address. + +Custom listeners remain available for advanced setups and carry a clear warning when they are not private, loopback, or Tailscale ranges. + +### HTTP and HTTPS + +HTTP is supported and is the default for trusted LAN use. The UI must not imply that HTTP encrypts credentials or telemetry. + +HTTPS is an optional setting. Existing operator-provided certificate support remains available. Tailscale certificate or Serve integration can be documented as an optional path, but it is not part of the required setup. + +### Discovery + +Discovery helps find candidates. It does not establish trust. + +HumWatch can use LAN discovery and Tailscale peer information to suggest addresses, but a machine joins only through its active pairing code. Browser CORS is not part of discovery because the browser never calls agents directly. + +Enrolled agents also use discovery to recover from a hub address change. A candidate hub exposes a signed announcement containing the fleet UUID, hub machine UUID, addresses, and generation. Agents verify that announcement with the fleet public key before changing destinations. The new hub actively sends the same announcement to discovered enrolled agents. Manual hub address entry remains available on the local recovery page. + +## Data flow + +### Live reporting + +1. The collector emits normalized measurements. +2. The local buffer commits the sample and allocates its sequence number. +3. Agent transport sends a signed batch to the active hub. +4. Hub ingestion verifies and stores the batch in one transaction. +5. The hub returns the highest contiguous stored sequence. +6. The agent advances its acknowledgement watermark. +7. The dashboard receives the update through the hub event stream. + +### Offline buffering and backfill + +If the hub is unavailable, collection and local storage continue. Reconnect uses exponential backoff with jitter. After reconnect, the agent uploads missing ranges oldest first while continuing to send current data. + +Batch identifiers and sequence ranges make retries idempotent. The hub may accept duplicated transport attempts, but it must store each sample once. + +If the offline interval exceeds local retention, the agent reports the exact missing range. The hub displays a data gap rather than inventing continuity. + +### Central history + +The hub stores measurements under stable machine UUIDs. Hostname, IP, and display name changes do not create a new machine. + +Central history keeps the existing seven-day default in 2.1.0. Changing product retention is outside this redesign. + +The existing per-machine database is imported into the local buffer during migration. After enrollment, that history uploads through the same ordered backfill path as new data. + +## Dashboard behavior + +### Login + +The hub address opens a normal username and password form. There is no shared fleet token field. + +### Fleet overview + +The landing page shows every enrolled machine with: + +* Display name and current hostname. +* Online, offline, enrolling, revoked, or migration status. +* Last report time. +* CPU load and temperature when available. +* GPU temperature when available. +* Memory use. +* Battery state when available. +* Data gap or backfill status. + +### Machine selection + +A persistent machine selector changes the active machine inside the existing dashboard shell. CPU, GPU, memory, disk, network, battery, process, and history pages read the selected machine UUID from application state. + +Selecting a PC does not navigate the browser to that PC and does not open another tab. + +### Settings + +Settings gains these sections: + +* Owner and sessions. +* Fleet and enrolled machines. +* Add PC. +* Move Hub. +* Recovery key status. +* Network listeners. +* Optional HTTPS. +* Integrations. +* Legacy compatibility. + +## Hub transfer + +### Planned transfer + +The owner chooses **Settings > Fleet > Move Hub** and selects an enrolled target. + +Preflight verifies: + +* Target is online and not revoked. +* Target runs a compatible HumWatch version. +* Target has enough free storage. +* Target can listen on at least one selected private address. +* The current database can produce a consistent snapshot. +* No unacknowledged recovery key is pending. An owner who has not yet confirmed the printable recovery key is saved must acknowledge it first, since transferring the fleet signing key without that acknowledgement would destroy the fleet's only other recovery path. + +Transfer then: + +1. Pauses fleet configuration changes and report ingestion, so no acknowledgement can advance past the snapshot. Agents buffer rejected batches in their local outboxes, and a failed transfer lifts the pause. +2. Creates a consistent database and fleet state snapshot. +3. Encrypts the transfer to the target machine's X25519 public key with an ephemeral key and authenticated encryption. +4. Transfers owner hash, fleet registry, history, settings, integration records, and the fleet signing key. +5. Excludes browser sessions. +6. Starts the target in pending hub mode. +7. Increments the hub generation and signs the new hub announcement. +8. Confirms that the target accepts signed reports and serves the owner login. +9. Makes the old hub durably enter a draining state, then has the target persist and activate a retryable announcement-delivery obligation. +10. Demotes the old hub to agent state only after the target acknowledges that durable publication obligation. + +If validation fails before draining, the old hub remains authoritative and the target discards pending hub state. Once draining starts, the old hub cannot resume normal hub service until it either completes the target-owned publication path or sends a signed abort. A lost activation response is reconciled by the announcement digest and never permits a blind abort of the target. A target restart resumes the persisted delivery obligation, so a crash after old-hub retirement cannot leave agents permanently pointed at the retired hub. + +### Split brain prevention + +Every hub announcement contains the fleet UUID, hub machine UUID, a lineage UUID minted at fleet creation or promotion and preserved across transfers, addresses, a recovery epoch, and a monotonically increasing generation. It is signed by the fleet key. + +Agents accept only a valid announcement whose `(recovery epoch, generation)` pair is lexicographically higher than the one stored locally. Ordinary announcements (address changes, transfers) carry the current epoch forward and increment the generation. Dead-hub recovery is the one path that increments the epoch. Two machines recovering the same fleet while partitioned can mint the same fresh epoch, so same-epoch conflicts between distinct hub lineages resolve deterministically: the lineage with the lexicographically smaller lineage UUID wins (a value fixed at promotion and unchanged by later transfers), the loser stops publishing and retires visibly, and the winner re-announces above everything the loser distributed. An old hub that returns later cannot reclaim the fleet with stale state, even though it still holds the fleet signing key: anything it mints carries its old epoch, so no generation it can reach is ever accepted, and the rejection evidence (the agent's accepted announcement, fleet-signed) drives it into a visible retired state on its local recovery page. + +One deliberate exception exists: the manual hub address entry on a machine's loopback-only recovery page may, after showing the generation comparison and getting explicit confirmation, adopt a fleet-signed announcement whose generation is not higher. It exists for the machine that accepted a generation the dead hub never finished distributing, and it is gated on physical access to that machine. No remote or automatic path may do this. Adoption never lowers the machine's authority floor, itself a `(recovery epoch, generation)` pair compared lexicographically, so the dead lineage's announcement cannot be replayed to redirect the adopted machine whatever its generation, and a hub that reads verified floor evidence of unresolved authority at its own epoch re-announces above it, converging the fleet past the dead lineage. + +### Address change + +Moving the hub normally changes its IP. The completion screen shows and opens the new IP plus port. The owner signs in again because browser sessions are revoked and the origin changed. + +Stable virtual addresses supplied by external network products are compatible but not required. + +## Dead hub recovery + +### Recovery bundle + +The hub periodically creates a compact recovery bundle containing: + +* Fleet UUID, lineage UUID, current recovery epoch, and current generation. +* Fleet signing key. +* Machine registry and public keys. +* Listener and fleet settings. +* Integration registration records. +* Schema and bundle version. +* A monotonically increasing bundle revision within the current hub generation. +* Creation timestamp. + +The recovery bundle does not contain browser sessions or full central history. + +Each bundle is encrypted to the recovery public key with an ephemeral X25519 key, key derivation, and authenticated encryption, and is signed by the fleet key so no agent can forge a replica with the widely held recovery public key. Agents verify the signature before storing a replica, and recovery verifies it before decrypting. The printable recovery key holds the corresponding private key. It is shown until the owner confirms it is saved and is never retained after that acknowledgement. The encrypted bundle is replicated to enrolled agents. + +### Recovery flow + +1. Choose **Recover Fleet** on another installation. +2. Enter the printable recovery key. +3. Select the valid recovery bundle with the highest `(recovery epoch, generation, bundle revision)` tuple found locally, after the same lineage tie-break announcements use: a same-epoch bundle from a losing lineage is never selected over a winning lineage's, whatever its generation or revision. +4. Set a new owner username and password. +5. Select a listener from this machine's private addresses. The dead hub's stored listeners and certificate paths are not reused. +6. Promote the PC to hub at a recovery epoch one above the highest epoch the bundle, this machine's own floor evidence, or any verified probe response carries, and generation 1: the fresh epoch outranks every `(epoch, generation)` pair any machine holds, so no generation arithmetic against the dead lineage is needed, and a dead lineage's generation at the storage ceiling cannot block recovery. Announce the newly bound listener. Machines that cannot be probed are listed with a warning that they may need the manual hub address entry after promotion. +7. Revoke browser sessions and integration credentials. +8. Accept reconnecting agents and rebuild central history from their retained buffers. + +The UI states how much history can be recovered before promotion. History older than all agent buffers is not promised. The UI also states the recovered bundle's creation time and that any PC enrolled after that time is not in this registry and must be re-enrolled after promotion, not merely reconnected. A machine reporting to the recovered hub without a registry entry gets a rejection distinguishable from an ordinary authentication failure, and its own local recovery page surfaces that it needs re-enrollment. + +## SlipShell compatibility boundary + +SlipShell monitoring is not implemented in HumWatch 2.1.0, but the contract required for it is. + +The hub provides a versioned `/api/v2` fleet API with stable machine UUIDs. The future SlipShell flow is: + +1. The owner creates a short-lived SlipShell pairing code in HumWatch. +2. SlipShell pairs with the hub and receives a revocable read-only integration identity. +3. SlipShell associates a saved SSH server with a HumWatch machine UUID. +4. SlipShell reads current metrics, history, and availability from the hub. +5. SlipShell can subscribe to a fleet event stream while active. + +SlipShell does not run another monitoring service, read HumWatch's database, or reuse the owner login. + +The API separates public fleet resources from the private hub-to-agent protocol: + +* `/api/v2/session/*` for browser login and sessions. +* `/api/v2/fleet/*` for fleet metadata and administration. +* `/api/v2/machines/*` for machine data and history. +* `/api/v2/integrations/*` for pairing and revocation. +* `/api/internal/v1/*` for agent enrollment, reports, backfill, and hub announcements. + +## `/api/v1` compatibility + +The public v1 health endpoint remains available for version and capability checks. + +Fresh 2.1.0 installs do not create a legacy bearer token. Protected v1 endpoints are disabled unless legacy compatibility is explicitly enabled. + +Upgraded installations can keep their protected v1 endpoints temporarily with the existing bearer token. The migration wizard defaults this option off, labels it as legacy, and provides a removal action. The fleet dashboard and `/api/v2` never depend on it. + +## Upgrade from 2.0.x + +### Safety before choices + +Before changing roles, the upgrade must: + +1. Stop the service cleanly. +2. Locate the active database, including legacy Windows paths. +3. Copy the database plus live WAL state through SQLite's supported backup mechanism. +4. Validate that the backup opens and contains the expected schema and machine record. +5. Archive config, token paths, certificate paths, and CORS origins. +6. Log every migration decision and source path. + +When the migration screen runs in a browser that has a 2.0.x manual peer list in `localStorage`, it reads that list as enrollment suggestions. It clears the old browser list only after the owner confirms migration. + +Issue #14, where Windows upgrades silently orphan the existing database, is a release blocker for 2.1.0. A migration that cannot prove which database it preserved must stop with a visible error. + +### Create Fleet migration + +Choosing **Create Fleet**: + +* Creates the fleet and owner account. +* Imports existing history under this installation's stable machine UUID. +* Makes this PC the hub and first agent. +* Shows the recovery key. +* Disables the browser token gate. +* Archives legacy peer, CORS, token, and certificate configuration unless the owner explicitly enables temporary v1 compatibility. + +### Join Fleet migration + +Choosing **Join Fleet**: + +* Preserves existing history in the local buffer. +* Shows an enrollment code and reachable addresses. +* Joins after the hub proves enrollment. +* Uploads preserved history through ordered backfill. +* Archives legacy peer, CORS, token, and certificate configuration unless the owner explicitly enables temporary v1 compatibility. + +### Rollback + +Legacy files are not deleted automatically. The owner can remove the migration archive after the fleet is verified. Rollback instructions must state that data collected after role conversion may require export before returning to 2.0.x. + +### Duplicate service detection + +Installers check for multiple HumWatch services, conflicting service definitions, and port ownership. When an obsolete development or user service conflicts with the installed service, setup identifies both paths and offers to disable the obsolete service after confirmation. + +The installer does not stop an unrelated process merely because it uses port 9100. + +## Failure behavior + +### Hub unavailable + +Agents continue collecting, buffer locally, and reconnect with bounded backoff. The dashboard is unavailable until the hub returns or recovery promotes another PC. + +### Agent unavailable + +The hub retains its last known state and shows the last report time. It does not label stale data as live. + +### Enrollment interrupted + +The hub does not show an agent as enrolled until it stores the first signed report. A partially registered machine can be retried or removed without changing the fleet identity. + +### Backfill interrupted + +The hub acknowledgement watermark resumes the same range. Duplicate batches remain idempotent. + +### Transfer interrupted + +The old hub stays authoritative until the new hub passes validation. If demotion completed, signed generation rules prevent the old hub from returning as leader. + +### Recovery key unavailable + +Dead hub recovery is not possible. The owner must create a new fleet and re-enroll each PC. This limitation is stated when the recovery key is shown. + +### Version mismatch + +The hub can reject an agent whose internal protocol is incompatible while still showing its address and version. Upgrade instructions must name which side needs updating. + +## Security boundary + +HumWatch 2.1.0 protects against unauthorized application access with owner authentication, independent machine identities, signed data, replay controls, revocation, throttling, and restricted default listeners. + +HTTP on ordinary LAN does not provide confidentiality. Anyone able to observe that network path may read credentials and telemetry. HumWatch must say this plainly. Tailscale or optional HTTPS is the path for untrusted networks. + +HumWatch does not expose the hub to the public internet by default. Public exposure, reverse proxies, and cloud relays are outside 2.1.0. + +## Verification and acceptance + +### Automated coverage + +* Fresh create-fleet flow. +* Fresh join-fleet flow. +* Owner password hashing and migration. +* Login persistence, logout, expiry, throttling, and session revocation. +* CSRF protection for state changes. +* Pairing code expiry, one-use behavior, and rate limiting. +* Machine key registration, report signatures, replay rejection, and revocation. +* Ordered ingestion, duplicate delivery, gaps, and acknowledgement watermarks. +* Offline buffering and reconnect backfill. +* Stable machine identity across hostname and address changes. +* Planned hub transfer success and rollback. +* Stale hub generation rejection. +* Recovery bundle encryption, validation, and promotion. +* Integration scope enforcement. +* Protected `/api/v1` disabled on fresh installs. +* Windows and Linux 2.0.x database migration. +* Duplicate service detection without unrelated process termination. +* HTTP listeners and optional HTTPS listeners. + +### Real system acceptance + +Before release, verify with at least one Windows agent and one Linux agent: + +1. Create a hub on one platform. +2. Enroll the other platform over ordinary LAN. +3. Add or reconnect one agent through Tailscale when available. +4. Use one browser login to switch among all machines. +5. Disconnect the hub, collect data, reconnect it, and verify ordered backfill. +6. Move the hub to the other platform and verify agent reconnection. +7. Recover from an encrypted bundle in a controlled destructive test. +8. Upgrade a real 2.0.x database with history and prove sample continuity. +9. Confirm no manual shared token, CORS, or certificate setup is required. + +### Release acceptance statement + +HumWatch 2.1.0 is accepted when a person can install it on several Windows and Linux PCs, choose one hub, sign in once, enroll each other PC with one temporary code, switch among them inside one dashboard, survive hub downtime, and move or recover the hub without manually distributing credentials. + +## Explicitly outside 2.1.0 + +* Multiple human accounts or roles. +* Cloud relay or public internet publishing. +* Continuous hub replication. +* Automatic high availability or leader election. +* SlipShell user interface work. +* Mobile push notifications. +* Required Tailscale setup. +* Required HTTPS setup. +* Long-term compatibility for the shared bearer token model. + +## Implementation constraints + +* Preserve existing collector behavior unless the hub protocol requires a narrow boundary change. +* Keep the collector usable without the fleet web layer. +* Do not let browser code call agents directly. +* Do not make SQLite table layouts part of `/api/v2`. +* Keep the internal agent protocol versioned separately from the public fleet API. +* Use transactional, resumable migrations with explicit logs. +* Never delete a legacy database, token, certificate, or recovery archive automatically. +* Keep secrets out of URLs, logs, process arguments, and browser storage. +* Keep destructive setup and recovery copy literal and plain.