feat(reporting): persist managed delivery reconciliation records

[bokelley]

Refs #1167

This adds the durable storage foundation for seller reporting reconciliation. Trusted publishers and future destination writers/receipt handlers can retain immutable destination bindings, materialization evidence, readability observations, and revision/adjustment receipts with the same state machine in memory and PostgreSQL.

The audit corrections reject evidence attached to a different frozen obligation, give reconciliation its own isolated incremental feed, and enforce the exact evidence graph in PostgreSQL. The buyer selects the obligation's unique current revision first: an official revision wins when present; otherwise the unique unsuperseded snapshot is current. An unmaterialized, pending, or failed current revision never falls back to an older materialized snapshot. Multiple current candidates fail closed.

A second independent adversarial review then reproduced ten further in-scope defects and fixed each with a regression that fails on the previous head. The database now recomputes every retained payload digest under a SQL implementation of canonical_json_utf8_v1, refuses any payload field outside the frozen record types, and re-derives the financial acceptance predicate itself; the buyer selector judges snapshot topology independently of the official close; and the reference validators separate credential shapes from operational names that merely contain a credential word.

Stack

• Base: conductor/reporting-obligation-currency-1171 / fix(reporting): freeze currency on reporting obligations #1175, ff584b6fbe3436e7afa4980a14e2e5e4004a06a3.
• Preceding dependency: fix(reporting)!: scope configuration generations by account #1174 / bug(reporting): scope configuration generations and leases by account #1169, account-qualified configuration generations.
• Head: 3c405a21f978ed9d3208611bb4a7a8434a056933, tree f81e8437de09ce341b3892bb17cc3edc069c661f.
• Preserved commits: eac9b1b000c8faa56c19cc4cd9df83ba1ad688e9 (storage foundation), 89674803f128868e4c8069f4b22f97dccf5836ac (independent revision histories and receipt races).
• Audit correction: c44095e3b457c44931eacc8a0da21ddf442229fd (fix(reporting): enforce exact reconciliation graph and isolated feeds).
• Second audit correction: 3c405a21f978ed9d3208611bb4a7a8434a056933 (fix(reporting): bind reconciliation evidence in the database and judge snapshot topology).

Architecture and corrected findings

• Exact frozen ownership. Attempts, outcomes, checks, revision receipts and adjustment receipts retain the exact principal/account, configuration generation, obligation and publication identity. Matching logical content does not authorize cross-obligation fan-out. Snapshot and later official histories coexist independently, including equal attempt numbers, separate rejected replacements and terminal acceptances.
• Independent incremental reconciliation. Optional ReportingReconciliationFeedStore supplies separate record kinds, typed snapshot/cursor/checkpoint tokens and filters. Each account/consumer has a dense append-only sequence/head. Evidence, its caller-visible sequence and its feed row commit together. Core LedgerRecordKind, LedgerPage, change feed and required public store protocols remain unchanged.
• Scoped, resumable pagination. Caller scope and filters apply before count/limit; keyset pages freeze upper bounds and counts. Tokens bind account, consumer, feed version, filter, bounds and last sequence/key. Partial pages expose a continuation cursor; only the final page exposes a changes checkpoint. Foreign reconciliation writes do not affect Core or another consumer's snapshot ID, count, cursor, checkpoint or next sequence. Reads fail closed on mismatched/missing feed identities.
• Database-enforced graph. Composite references, insert/immutability/ordinal guards and atomic receipt-head triggers enforce outcome → exact attempt, check → successful materialization, receipt → exact materialization/revision, adjustment receipt → exact adjustment/official revision, and current rejected-leaf replacement. Bidirectional record/feed references and a deferred head reference enforce co-commit even for direct SQL writes. All 15 denormalized identity columns are checked against decoded retained payloads on reads; referenced Core evidence stays immutable while leases/readability remain operational.
• Database-enforced payload and financial content. reporting_canonical_json implements the canonical_json_utf8_v1 profile in SQL, so every insert recomputes its own content_sha256, and a closed key allowlist refuses any field outside the frozen record types. Ordinary direct SQL — satisfying every column constraint with no trigger disabled — can therefore no longer retain a credential or provider blob under a chosen digest, nor leave a principal's feed permanently unreadable. reporting_reconciliation_evidence re-derives the predicate that decides money: revision row count, typed totals and canonical digest; binding format, readers, method, success status, verification path and retention floor; checksum-to-object binding; receipt profile, totals, profile-specific evidence and readable window; and an adjustment digest recomputed from the retained adjustment columns. A retained payload that disagrees with its own fingerprint fails every read for that principal, including pages whose filter would have skipped the damaged row.
• Publication topology judged independently. Snapshot leaves are counted on their own and every leaf is walked back through its predecessors, so an official close can no longer mask a forked snapshot history or a supersession cycle. ReportingRevision carries no obligation reference on the wire, so candidates exclude a revision only another obligation has materialized; revisions nobody materialized — an unmaterialized official included — are never excluded, illegal fan-out is still reported, and an unresolvable owner still fails closed. The buyer also requires immutability="native_version" for native_commit, matching the seller.
• Achievable verification contracts. native_commit requires native_version immutability, exact native references and the exact observation path. The complete method × profile × billing matrix is tested on both stores: dataset/warehouse bindings reject manifest-only verification, and billing requires canonical digests. Success also binds typed/unit-bearing expected totals, row count, format/readers, checksums and retention floors. Managed canonical evidence remains distinct from Core's unchanged digest contract.
• Provider-native references. Field-specific, secret-safe validators preserve wire-valid operational IDs containing spaces, slash, plus, equals and Unicode. Generated-model → retained-record → wire round trips cover GAM, FreeWheel and warehouse identifiers. Recognizable credentials, tokens, URLs and encoded forms remain rejected without echoing their contents. Credential detection now matches shapes rather than substrings: a keyword condemns a value only when it stands alone as a word and introduces something after it, so tokenized_inventory_daily, secretariat-report-v17 and authorization_metrics_v2 persist unchanged while token=… and Bearer … do not. Independently, any ?name=/&name= query-parameter pair is refused in every reference field, so presigned S3/GCS and Azure SAS material such as report.csv?sv=…&sig=… never reaches storage even though no keyword appears in it.

Exact retries return retained evidence and append nothing; changed retries conflict. Receipt IDs share one namespace, accepted leaves are terminal, and expiry/corruption does not erase acceptance. Adjustment acceptance binds the complete immutable adjustment digest and its official revision; revision and adjustment acceptance remain independent facts that completion must require together.

Migration

create_schema() applies the four bundled SQL resources in one transaction under the shared schema advisory lock. The reconciliation migration is itself atomic in autocommit mode and idempotent. Literal beta.15 and #1171 fixtures preserve existing rows, hashes, currency, leases, issue history and Core feed sequence numbers. The literal initial reconciliation upgrade also preserves records, receipt heads and legacy feed bytes/timestamps while projecting retained order into independent dense account/consumer sequences. Incomplete or inconsistent history is rejected atomically; unknown historical currency/digests/totals remain unknown. Start a new reconciliation snapshot/feed walk after upgrading; legacy Core tokens are not reconciliation tokens. Drain older writers before applying the migrations.

Validation

Python 3.12.14, real PostgreSQL 16.14. The authoritative evidence below is from head 3c405a21f978ed9d3208611bb4a7a8434a056933, tree f81e8437de09ce341b3892bb17cc3edc069c661f, with a clean checkout before and after every run. Earlier c44095e3 evidence is superseded.

• Full make ci-local on the exact committed head: 9,076 passed, 34 skipped, 9 deselected, 1 expected failure; 85.14% coverage (80% required), pytest 1,075.97s. Lint, mypy over 1,359 source files, mypy --strict over 28 adopter fixtures with no typing suppressions, and generated-code validation all passed. All commit hooks passed, including Black, Ruff, mypy, Bandit and conventional subject validation.
• Focused reporting gate on that head: 976 passed in 94.62s, up from 893 at c44095e3. It now carries 495 reconciliation conformance regressions (boundaries 23, changes 62, migration 20, profiles 188, SQL 104, store 50, transitions 48) plus 52 buyer-selector cases — +83 new cases from 16 new test functions.
• Each of the ten defects was reproduced before fixing and each regression fails against the parent's src/. Replacing only src/ with c44095e3 and re-running the new tests yields: signed-query rejection 20 failed/20 passed (the 20 passes are values the old keyword list already caught), benign-name acceptance 10 failed, caller-boundary classification 4 failed, closed-payload 3 failed, payload-digest 1 failed, outcome-evidence 6 failed, receipt-evidence 4 failed, adjustment-digest 1 failed, snapshot-topology 2 failed, identical-scope ownership 1 failed, buyer native_commit immutability 1 failed.
• Wheel and sdist built from that commit. Both fresh, non-editable installs match all 4 SQL resources byte-for-byte and ship py.typed; no workspace artifacts are packaged. Isolated python -I core import with an empty snapshot/feed/checkpoint probe passes in both installs without any PostgreSQL dependency (PG_AVAILABLE is False).
• The built wheel with [pg] passed a real PostgreSQL 16.14 exercise across two fresh Python processes: three idempotent create_schema() calls, then a five-record walk paged [1, 2] in the seed process and resumed [3, 4, 5] from the saved cursor in a second process, with the snapshot reading all five. The isolated schema was dropped and the checkout stayed unchanged.
• Direct feed/token probes on both stores confirmed memory/PostgreSQL parity: Core open_snapshot byte-identical before and after higher-tier writes; inter-page writes deferred past the frozen upper bound; exclusive changes_after at head; replay appending no sequence; and unknown, malformed, overlong, cross-principal, filter-mismatched and ahead tokens all failing indistinguishably as INVALID_CHECKPOINT.
• reporting_iso_utc and reporting_canonical_json were checked byte-for-byte against iso() and canonical_json_utf8_v1, including sub-millisecond, trailing-zero and non-UTC-offset timestamps, because the recomputed adjustment digest depends on that parity.
• Wheel SHA-256: e63c5aaa2635cfecf6411824d0a38c0d93270ec84a593c95b509f07efe08993b.
• Sdist SHA-256: 6b89262c5e06fb4fcb6e0f8eb8f6fba1e9192ccdecc05e23be8a02382c80bea0.

The focused command was:

.venv/bin/pytest tests/conformance/reporting \
  tests/test_reporting_canonical_json.py tests/test_reporting_consumer_loop.py \
  tests/test_reporting_inline_source.py tests/test_reporting_ledger.py \
  tests/test_reporting_reconciliation.py tests/test_reporting_source_contract.py \
  tests/test_reporting_webhook_operation_id.py -q --no-cov

The full gate used the configured real PostgreSQL URL and make ci-local. Full CI on the committed head is the authoritative evidence.

Remaining boundaries

These are trusted low-level storage contracts. Future handlers must derive account/consumer scope from authenticated transport, and future publishers/writers must compute expected evidence and verify actual destination content outside retained records. Core status remains Core-only. Higher-tier handlers, destination writers, capabilities, reconciliation health, bounded wire projections and #1168's atomic notification outbox remain completion work; #1167 stays open.

The reference stores load a consumer's retained graph for transition validation. Feed pages use indexed keyset reads with scoped integrity/count scans and the existing account transaction lock. The integrity scan now recomputes one canonical digest per retained record for the calling principal, and each insert recomputes its own, so both costs grow with a principal's history. Large-history throughput and production-sized migration lock duration remain unbenchmarked. Details are in the storage architecture.

Three review findings were deliberately not changed here, with reasons:

• Cursors and checkpoints stay unsigned, matching Core's documented design. They bind feed version, account, consumer, filter fingerprint, frozen bounds and last key, and the store re-validates all of it on use. A principal can present a boundary it built itself, but a checkpoint at the current head is byte-identical to the one a completed walk issues, so the store cannot distinguish forgery from a legitimate full walk, and the caller gains nothing it could not reach through the supported API. Foreign-principal tokens are refused. A durable per-reader issuance record would be needed to do better, and that changes the stateless-reader contract.
• ledger_as_of is not bound into snapshot_id. Binding a clock-derived value would change the snapshot ID on every read and destroy the "foreign writes do not move my snapshot ID" property this slice is required to have. ledger_as_of is echoed-only and affects no row selection.
• An authoritative page-local revision→obligation ownership projection belongs to the status slice. ReportingRevision has no obligation field and the response's ext extension is not yet a protocol-sanctioned alias model, so this PR narrows candidates by materialization ownership and otherwise fails closed. Two Core-only obligations that share a definition, profile, campaign set and period still read as ambiguous rather than guessing; resolving that needs the status projection planned for #1167C.

A residual, unchanged boundary: no syntax check can decide whether an arbitrary opaque string is a secret. A high-entropy value with no keyword, scheme, userinfo, PEM header, AWS key shape, JWT shape or query-parameter pair still persists, and provenance remains the trusted adapter's responsibility. One cosmetic nit is also left alone: a ledger with both multiple officials and multiple snapshot leaves reports AMBIGUOUS_REVISION_CHAIN twice, which changes no decision.

[gitguardian]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them.
While these secrets were previously flagged, we no longer have a reference to the
specific commits where they were detected. Once a secret has been leaked into a git
repository, you should consider it compromised, even if it was deleted immediately.
Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[aao-secretariat]

Ladon cannot review this PR until merge conflicts are resolved.

[aao-secretariat]

Ladon verdict: Approve

Approve — additive seller reconciliation storage contracts (#1167 storage half).

Checked:

• No no-auto-approve team match; gated_paths false; high_risk false — no hard gates apply.
• Reviewer found no blocking or Medium findings across all 26 files.
• Verified via reviewer notes: multi-tenant isolation, credential/echo boundary (no credentials in ctx_metadata), append-only immutability, transaction atomicity, forward-compat on discriminated unions, and type-system import layering all clean.
• Semver signal correctly feat: — the change is purely additive (new opt-in stores, managed evidence fields, a bundled migration, DB triggers, wire projections), no public-surface removals/renames/required-flips.

No blocking findings, no medium findings, no escalation triggers — falls through to row 9 (approve).

[aao-secretariat]

Ladon verdict: Approve

Approve — reconciliation storage foundation (second pass).

No blocking findings. No medium findings. Reviewer verified memory/PG store parity, the fail-closed transition contract, scoped independent feeds, official-wins revision selection, and an additive public surface carrying the correct feat(reporting) semver signal. Full CI green (8,993 passed).

Gate checks:

• No no-auto-approve team match.
• gated_paths: false; review_decision: APPROVED.
• high_risk: false (no deleted/modified high-risk trigger).
• Prior decision was approve — this pass is consistent, no new findings.

None of rows 1–8 fire (zero findings), so the outcome falls through to row 9: approve.

[aao-secretariat]

Ladon verdict: Approve

Approve.

Delta pass on adversarial-hardening commit 3c405a2 (subsequent review; prior decision was approve with no findings). The reviewer examined all five source deltas — _reconcile._select_current cycle/fork detection, evidence.py credential-shape validators, delivery/delivery_pg boundary attribution, and the +262-line reconciliation SQL evidence graph — plus operational readiness and public-surface/layering. No Critical/High/Medium findings; changes are fail-closed and covered by new regression tests.

Gate checks:

• No no-auto-approve team match.
• gated_paths: false (row 2 N/A); review_decision: APPROVED regardless.
• high_risk: false — no (deleted)/(modified) escalation triggers (rows 3, 5 N/A).
• No medium findings (rows 4, 8 N/A).
• Prior decision was approve, not escalate (row 6 N/A).

None of rows 1–8 fire; falls through to row 9 → approve.