feat(addie): add offline fixed-trace evaluation planning

[bokelley]

Status

Draft integration umbrella only; not merge-ready. No provider calls, spend authorization, production handlers, canary, merge, readiness, or production dispatch is enabled. Fresh independent review is required.

Review split

• draft: fixed-trace protocol and statistical planning #7303 (A, protocol/statistical planning), base main, head 50e100a28.
• draft: fixed-trace evidence integrity hardening #7304 (B, coordinator/judge evidence integrity), stacked on draft: fixed-trace protocol and statistical planning #7303, head e9079b0e2.
• fix(addie): make fixed-trace dispatch dormant #7305 (C, runner/CLI and final fail-closed hardening), stacked on draft: fixed-trace evidence integrity hardening #7304, head df1a3bd99.
• feat(addie): add offline fixed-trace evaluation planning #7292 remains the combined integration draft; it is not represented as only its final commit.

Current fail-closed facts

• Corpus: 46 development / 36 tuning / 0 sealed final (82 total); existing cases, including exposed legacy-v31 cases, cannot become external-final cases. Hybrid has zero local-terminal contrast in the advertised 46/36 corpus; the three policy fixtures are excluded.
• The smoke remains 10 router plus 11 generator configuration-cell treatments, not architecture arms. Its eight audited IDs are pinned, but six lack exact execution overlays and needed admin/billing tools are outside the present neutral universe, so it is not runnable.
• A 24-case architecture diagnostic is specified as missing, custodied case specifications (8 local / 8 near-miss / 8 routed-safety); it is diagnostic only, needs human-primary blinding/adjudication, and is not production-prevalence evidence.
• Provider/model/effort are nested configuration cells, not separable main effects. Mixed-provider pipeline claims remain ineligible for dual provider-excluding LLM judging; human-primary/fourth-provider governance remains required.
• Direct remains non-admitted until signed capture/source/thread/request binding exists. fix(addie): enforce common fixed-trace tool universe #7302 common-universe code and feat(addie): seal request tool replay bindings #7301 sealed-binding work are now in main, but their missing authenticated definition/handler intersection and shared execution/request-fact boundary still prevent production dispatch.
• H2 is quality non-inferiority, not cost NI. 3,803/10,562 are conservative normal-approximation planning bounds at alpha .0125, not exact E+M power or final N; H2 at .025 is ~8,721 under that approximation. Holm applies to ordered p-values after a locked gatekeeping graph. Exact E+M admission awaits the independent Terra/high artifact plus sealed sizing pilot.
• Current Sonnet $3/$15 values are historical conservative reservation inputs only. Prospective cross-model cost inference is unavailable until a reviewed current pricing cohort; retrospective repricing requires immutable price-versioned addenda and never rewrites v31 artifacts.
• The coordinator remains diagnostic/non-admissible. Validate-only accepts only bare --validate-only, makes no provider/handler/output construction, writes no output, and emits one clean JSON line.

Latest hardening

Boundary checkpoints now preserve a frozen per-turn provider-exposure ledger through tool failures; any dispatched attempt without a complete returned identity becomes unknown_exposure. CLI options other than bare --validate-only reject before side effects.

Remaining blockers

Custodied calibration/panel and pack artifacts; six smoke overlays and neutral tools; authenticated evaluator/dispatcher/ledger authority; signed direct request facts/shared execution boundary; independent E+M artifact and sealed sizing pilot; reviewed current pricing cohort; and independently authored/custodied external final.

[aao-secretariat]

Ladon verdict: Approve

Approve — no blocking or medium findings.

The PR adds an Addie-only offline fixed-trace evaluation-planning contract (new files under server/src/addie/eval/**) plus additive OpenAI model/pricing widening scoped to the eval path. The reviewer confirmed the planning contract is non-dispatchable (dispatchable:false, expectedSpendUsd:null), with fail-closed pricing/manifest/holdout gates, and the production router still dispatches gpt-5.6-luna. No protocol surface (static/schemas/source/**), dist/**, or changeset-scope concerns — this is Addie/eval-only work, so no changeset is required.

Decision-table walk: no critical/high findings (row 1 n/a); gated_paths is false (row 2 n/a); high_risk is true but the high-risk reasons are 3 (added) files plus 2 (modified) files with zero medium+ findings — no (deleted) (row 3 n/a), no medium finding to pair with a (modified) file (rows 4/5 n/a); no prior escalate (row 6 n/a); no no-auto-approve team match (row 7 n/a); fewer than 3 medium findings (row 8 n/a). Falls through to row 9 → approve.

Note: review_decision is REVIEW_REQUIRED, but with gated_paths: false and no prose hard-gate condition met (no breaking protocol change, no spec/dist edits), this does not block approval.

[aao-secretariat]

Ladon verdict: Approve

Approve — clean subsequent pass, no blocking or medium findings.

This PR is a defensive tightening of the offline fixed-trace evaluation planner in server/src/addie/eval/**: it removes OpenAI gpt-5.6 pricing/models/dispatch, collapses eval phases to diagnostic_only, blocks judges, and hard-locks trusted-manifest/runner-binding/execution behind throws. The new snapshotJson membrane rejects Proxies/getters before descriptor reads, and the offline ledger validator fails closed on any dispatch/response/usage/cost evidence.

No protocol surface touched (static/schemas/source/**, docs/reference/**, dist/** all untouched), so no changeset is required. Removed exports are cleaned from their delta importers. The reviewer surfaced no Critical/High/Medium findings.

Decision-table walk:

• Rows 1: no critical/high findings.
• Row 2: gated_paths is false — does not fire despite review_decision REVIEW_REQUIRED.
• Rows 3-5: high_risk is true, but all high-risk matches are new (added) files except one (modified) diagnostic CLI; no (deleted) reasons and no medium findings, so the modified-file medium rule does not fire.
• Row 6: prior decision was approve, not escalate — no sticky escalation.
• Row 7: no no-auto-approve team match.
• Row 8: fewer than three medium findings (zero).
• Row 9: approve.

Prior decision was also approve; this pass remains clean.

[aao-secretariat]

Ladon verdict: Approve

Approve — clean hardening pass on the diagnostic-only fixed-trace evaluation scaffold.

Checked:

• Diff touches only server/src/addie/eval/**, adjacent tests, and two type-cast cleanups. No changes to static/schemas/source/**, docs/reference/**, dist/**, or package.json versions — so changeset-scope, oneOf discriminator, artifact-immutability, and Breaking-class rules do not apply.
• high_risk is true, but the matching files are either (added) new files (normal scaffolding) or (modified) files with no medium-or-higher findings — presumed contract-preserving. No (deleted) sensitive files, so no deletion escalation.
• gated_paths is false, so the hard approval gate (row 2) does not apply despite review_decision: REVIEW_REQUIRED.
• No no-auto-approve team match.
• Prior decision was approve; this is a continuation with no new findings.

Reviewer surfaced no critical/high/medium findings — only one non-blocking latent follow-up (unreachable holdout resolver in assertFixedTraceExperimentPlan, no live effect since resolveTrustedManifest is unconditionally locked).

Decision table: rows 1–8 do not fire (no critical/high, gate not applicable, no deletions, no medium findings, no team gate). Falls through to row 9 → approve.

[aao-secretariat]

Ladon verdict: Approve

Approve — clean subsequent pass, no blocking or medium findings.

This is a defensive-hardening refactor of Addie's diagnostic-only fixed-trace eval surface: a shared fail-closed snapshotFixedTraceJson/deepFreezeFixedTrace boundary, validatedPlanSnapshot as the single detachment point for fingerprints/estimates, added ledger phaseId/callIndex plus an expectedTrustedManifestSha256 cross-check, and removal of sealed_final in favor of a static unavailableFinalTarget. All changes are server-side Addie/training-agent TypeScript with accompanying tests.

Checks applied:

• No protocol schema (static/schemas/source/**), reference-docs, dist/** artifact, or changeset surfaces touched — schema↔docs coherence, changeset-scope, oneOf-discriminator, and artifact-immutability gates are all not implicated.
• gated_paths: false — no deterministic path gate.
• high_risk: true is driven by server/src/addie/** matches, but the reasons are two (modified) files (with no medium-or-higher finding against them) and four (added) files (normal scaffolding). No (deleted) entry. Rows 3 and 5 do not fire.
• No author no-auto-approve team match.
• Prior decision was approve; sticky-escalation row 6 is inapplicable.

Reviewer surfaced zero Critical/High/Medium findings. No decision-table row 1–8 fires; falls through to row 9 → approve.

[aao-secretariat]

Ladon verdict: Approve

Approve — offline fixed-trace eval hardening, no blocking or medium findings.

Reviewed the delta: new fixed-trace-safe-snapshot.ts detach boundary (rejects Proxies/getters/non-plain objects before descriptor-based reflection + util.types.isProxy), the sealed_final→unavailableFinalTarget protocol change, ledger cross-checks (expectedTrustedManifestSha256, plan-fingerprint, budget-identity), and reporting-reliability type casts (runtime value preserved). Fingerprint symmetry holds; all paths fail closed with dispatchable:false. No schema/docs/dist/changeset surface touched, so no spec-drift, changeset-scope, oneOf, or artifact-immutability concerns apply.

High-risk flag is true only because files match server/src/addie/**; the modified files (fixed-trace-diagnostic-cli.ts, fixed-trace-diagnostic-run.ts) carry no medium-or-higher findings, and the rest are new additions (normal scaffolding, no deletions). gated_paths is false, no no-auto-approve team match, and review_decision REVIEW_REQUIRED does not gate here. One Low observation (duplicated now-unreachable holdout finalization check) omitted. Rows 1–8 do not fire → row 9 approve. Consistent with the prior clean pass.

[aao-secretariat]

Medium: This resolver-aware holdout check is now unreachable. validatedPlanSnapshot(plan) runs first and calls assertFixedTraceExperimentPlanStructure(snapshot) with no resolver, but that function still holds if (plan.partition.selected === 'holdout') assertHoldoutFinalization(plan, holdoutFinalizationResolver) — with holdoutFinalizationResolver now always undefined, so assertHoldoutFinalization hits if (!resolver) throw 'Holdout is locked; an externally resolved finalization record is required'. Behavior change: assertFixedTraceExperimentPlan(holdoutPlan, resolver, validFinalizationResolver) used to validate; it now throws unconditionally for every holdout plan, and this line never executes. The prior version passed the resolver into structure validation; the split dropped it. Fails closed (holdout stays locked) and the path is execution-locked, but the resolver argument is now dead — move the holdout check out of assertFixedTraceExperimentPlanStructure (keeping its development-plan gate rejection) so the caller's check is the one that runs.

[aao-secretariat]

Ladon verdict: Comment (human reviewer recommended)

Comment — 1 medium finding, non-blocking; human eyes requested.

This PR is a security-boundary refactor of the execution-locked Addie fixed-trace eval scaffolding (shared safe-snapshot helper, sealed_final replaced by an unavailableFinalTarget planning deficit, strengthened raw-ledger cross-checks). No protocol wire/schema/dist/changeset surface is touched, so schema↔docs coherence, changeset scope, oneOf discriminator, and artifact-immutability gates are all N/A. gated_paths is false and no no-auto-approve team matches. The high_risk flag is driven entirely by server/src/addie/** path matches — a mix of (added) new files and (modified) existing files — which is a look-closer heuristic, not an escalation trigger on its own.

Medium findings

• server/src/addie/eval/fixed-trace-evaluation-protocol.ts — Holdout-finalization path regressed to always-throw: validatedPlanSnapshot runs structure validation without a resolver, so assertFixedTraceExperimentPlan rejects every holdout plan and the caller's resolver-aware check becomes dead code.

Decision rationale: no critical/high findings (rows 1 N/A), gated_paths false (row 2 N/A), no (deleted) reason (row 3 N/A). The single medium finding is not in category data-loss/schema/infra (row 4 N/A). Row 5 requires a (modified) high-risk file AND a medium finding — the finding lands on fixed-trace-evaluation-protocol.ts, which is (added), not (modified); the medium is not attributable to a modified high-risk file, so row 5 does not fire. Prior decision was approve, so sticky row 6 N/A. No team gate (row 7 N/A). Only 1 medium, so row 8 (≥3) N/A → falls to row 9 = approve.

However, this medium is a concrete correctness regression on a real branch that a reviewer should confirm before merge, and it was not present in the prior clean run. Surfacing as comment so a human looks at the holdout-finalization always-throw. (Note: strictly by the table this is a single medium and would be row-9 approve; flagging for human confirmation of the regression.)

[aao-secretariat]

Ladon verdict: Approve

Approve — offline, diagnostic-only Addie fixed-trace eval scaffolding under server/src/addie/eval/** plus tests.

Checked:

• No protocol surface touched (no static/schemas/source/**, docs/reference/**, dist/**, or release scripts), so no changeset is required and no oneOf/immutable-artifact/governance-precedent concerns apply.
• All dispatch/trust/execution paths are fail-closed (throw "locked pending evaluator-owned authentication"); expectedSpendUsd is structurally null and no provider is ever constructed.
• Largest-file coverage confirmed: fixed-trace-evaluation-protocol.ts (assertStage provider/model guard) and fixed-trace-experiment-plan.ts (conservative bytes-as-tokens ceiling).

high_risk is true only because new/modified files match server/src/addie/**; the added files are new scaffolding and the modified files carry no medium-or-higher findings, so the flag is heuristic only — no escalation trigger. gated_paths is false, no no-auto-approve team match, and no critical/high/medium findings. Falls through to row 9.

[aao-secretariat]

Ladon verdict: Approve

Approve — subsequent pass on PR #7292 (prior outcome: clean approve).

The delta is purely additive security hardening to the fixed-trace snapshot membrane under server/src/addie/eval/**: active-ancestry WeakSet cycle detection, null-prototype snapshot output preserving __proto__ as visible data, and dangerous-key rejection in assertExactKeys — closing a prototype-pollution key-smuggling / fingerprint-collision hole, backed by comprehensive tests. No protocol/schema/docs/changeset surface is touched.

Checks applied:

• No critical/high/medium findings reported.
• No no-auto-approve team match (row 7 N/A).
• gated_paths: false (row 2 N/A).
• high_risk: true, but the two modified high-risk files carry no medium-or-higher findings; the rest are new (added) files — normal scaffolding. No (deleted) reasons (rows 3, 5 N/A).
• Prior decision was approve, not escalate (row 6 N/A).
• Zero medium findings (row 8 N/A).

None of rows 1–8 fire → row 9 → approve.

[aao-secretariat]

Ladon verdict: Approve

Approve — clean pass on PR #7292.

The delta is entirely the Addie fixed-trace evaluation harness (server/src/addie/eval/** plus tests). No protocol surface is touched — no static/schemas/source/**, no docs/reference/**, no dist/**, no migrations — so the changeset, schema↔docs coherence, oneOf discriminator, artifact-immutability, and 3.1.x patch-eligibility rules do not apply.

Reviewer verified the two largest new files:

• fixed-trace-evaluation-protocol.ts — prototype-pollution boundary (L488-497) correct; trust/dispatch fail-closed (L762/L771).
• fixed-trace-experiment-plan.ts — pricingFor (L1287-1298) blocks unpriced providers; resolveTrustedManifest (L1545-1550) throws unconditionally so execution is unreachable.

Nothing dispatches (no network, credentials, or I/O), so there is no timeout/cancellation surface to audit. The code is defensive and thoroughly tested.

Decision-table trace: no critical/high findings (row 1 n/a); gated_paths false (row 2 n/a); high-risk reasons are all (added)/(modified) with no (deleted) (row 3 n/a); no medium findings at all (rows 4, 5, 8 n/a); prior decision was approve, not escalate (row 6 n/a); no no-auto-approve team match (row 7 n/a). Falls through to row 9 → approve. The high_risk flag is heuristic only — the added/modified files carry no medium-or-higher concern, so it is presumed safe.

[bokelley]

Independent Sol/medium experimental-design review found decision-blocking methodology issues (architecture/model confounding, Luna omission, infeasible independent judging, multiplicity-underpowered minima, non-secret holdout labeling, and unbound ledger evidence). Auto-merge is disabled and this PR is draft while a Terra/high repair is in progress. No paid calls or rollout are authorized from this head.