Skip to content

feat(proof): the Harbor runner authors its whole set (DRAFT, HOLD) - #304

Draft
echobt wants to merge 14 commits into
droid/2edcb0c8-100-rlm-autonomous-strip-tbefrom
droid/8bcbefa3-sn100-stay-lit-harbor-propos
Draft

echobt wants to merge 14 commits into
droid/2edcb0c8-100-rlm-autonomous-strip-tbefrom
droid/8bcbefa3-sn100-stay-lit-harbor-propos

Conversation

@echobt

@echobt echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor

Stacked on #301 at 80bc2cdd (DRAFT HOLD — do not merge).

The gap this closes

Control-plane authorship was done: the wire carries a whole TopicAuthoring,
the store persists it, the install applies it, the journal records per-part
provenance. The guest half had no propose_rules entrypoint, so
--drive-rlm failed closed with NO_RLM_RULES (503, no row) and no topic
could reach authorship: rlm. The publish gate refuses to open a topic whose
behavior its own RLM never wrote, so authorship evidence box 2 could not go
LIVE green.

What ships

deploy/guest/runners/rlm_fc_in_guest_harbor/propose_rules +
harness/authoring_set.py write a complete authoring.json
(schema_version 1: topic_id, rules, migrations, apis,
submission_format, pin_policy). It never writes rules.json — a fragment
is recorded with honest rlm provenance and refused by name.

Every part is topic data, not a compiled list and not a copy of the operator's
bundle:

Part Authored from Why it is the RLM's
rules signed checklist (ids/order) + the signed inspect policy the text is the RLM's own statement of what it will prove and how; the topic's sentence is quoted as the declaration it enforces. None invented, none dropped
migrations the topic's own namespace (-_) the RLM's own state table, inside its namespace. Prior entries retained on re-author
apis the topic's own prefix the row the install registers; prior routes retained
submission_format this runtime's real intake staged cap, base-proof-submit-v1, single-use nonce
pin_policy the signed document's own knobs a restatement (scoring reads the document, so divergence either way is a threshold nobody is judged by). eval_image_digest / gpu_class are pin equalities the VM does not hold, so they stay absent rather than invented

Re-authoring reads PROOF_CURRENT_AUTHORING_FILE to retain the parts it is
not changing; rules and pin_policy are re-derived, because a retained rule
could be one the re-signed document dropped.

Tests

  • crates/proof-vm-guest/tests/reference_adaptor_authoring.rs runs the shipped
    entrypoint and holds its output to the real gates: TopicAuthoring::validate,
    PinPolicy::agrees_with_document, and validate_against_pin with the shipped pin.
  • agent_tests::the_reference_adaptor_propose_rules_is_discovered_and_authors_the_whole_set
    drives the adaptor through the actual guest agent. Non-vacuous: removing the
    entrypoint fails it (the reference adaptor ships no propose_rules).
  • tests/test_authoring_set.py — 25 cases (complete set, both refusals, re-authoring
    retention, bounds).
  • The bake gate now requires propose_rules in the adaptor tree and checks it is
    executable (100755), so a lost bit fails CI.

Gates run on this tip

cargo fmt --check · clippy -p proof-vm-guest --all-targets -D warnings ·
cargo test --workspace · xtask loc-cap / consensus-lint / spec-check /
design-check / external-docs-check · the adaptor's own suite
(tests/run.sh, all passed). cargo deny not re-run here (see the evidence
pack's pre-existing RUSTSEC-2026-0285 caveat).

Operator follow-up (documented, not run here)

A guest rebake is required: propose_rules is an operator artefact copied
by bake-rootfs.sh --runner. Tipping proof-challenge alone leaves
/opt/proof/runners on the old pin and --drive-rlm keeps failing closed.
Runbook § 2b has the staging ceremony, the per-part journal check, and the
clone-diff against the legacy document.

HOLD: do not merge. This stacks on #301, which is held pending Mathis GO.

echobt and others added 2 commits September 16, 2026 15:07
The control-plane half of authorship was done; the guest half had no
propose_rules entrypoint. A runner without one is NO_RLM_RULES (503, no
row), so `--drive-rlm` failed closed and no topic could reach
`authorship: rlm` — the publish gate will not open a topic whose behavior
its own RLM never wrote.

Adds the entrypoint to the reference adaptor:

- `propose_rules` + `harness/authoring_set.py` write a complete
  `authoring.json` (schema_version 1, all five parts). Never `rules.json`:
  a fragment is recorded honestly and refused by name.
- Every part is topic data. The rule vector is the signed checklist with
  the RLM's own framing of what it will prove (the declared sentence is
  quoted as the declaration it enforces); migrations sit in the topic's
  namespace; the route is under its own prefix; the submission format is
  the host's real intake; the pin policy **restates** the signed document
  and invents neither `eval_image_digest` nor `gpu_class`, which are pin
  equalities the VM does not hold.
- Re-authoring retains migrations and routes from
  `PROOF_CURRENT_AUTHORING_FILE`; rules and the policy are re-derived,
  because a retained rule could be one the re-signed document dropped.
- Two refusals, both deliberate: a declared rule with no signed inspect
  policy (no invented check, no silent drop), and a marker policy for an
  undeclared rule.

Tests, each verified non-vacuous:

- `reference_adaptor_authoring` runs the shipped entrypoint and holds its
  output to the **real** gates — `TopicAuthoring::validate`,
  `PinPolicy::agrees_with_document`, and `validate_against_pin` with the
  shipped pin.
- `the_reference_adaptor_propose_rules_is_discovered_and_authors_the_whole_set`
  drives the adaptor through the actual guest agent; removing the
  entrypoint fails it.
- `test_authoring_set.py` (25 cases) and the bake gate now require
  `propose_rules` and check it is executable.

Docs: the runner contract, the adaptor README, and the install runbook
(the ceremony on cortex-staging, the rebake requirement, the per-part
journal check, the clone-diff against the legacy document).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Records the entrypoint at 18a2532 as §2h: the gap it closes (a runner
without propose_rules is NO_RLM_RULES, so no topic could reach
authorship: rlm), the two tests that pin it (the set against the real
Rust gates, and discovery through the actual guest agent, each verified
non-vacuous), the refusals, and the guest rebake that makes it live.

Tip tables, the local gate run, and the PR stack move to 18a2532 / #304;
the summary's item 2 no longer reads as blocked on a missing adaptor.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review

@greptile-apps

greptile-apps Bot commented Sep 16, 2026

Copy link
Copy Markdown

Greptile Summary

This update corrects proof-slice preflight reporting so it follows the guest’s normalized selection result rather than raw shell selector values. The existing Harbor authorship fixes remain in place, including retained migration handling, current submission metadata derivation, and corrected evidence-tip tracking.

Confidence Score: 5/5

Safe to merge; there are no outstanding blocking issues.

All four prior findings are fixed in the current code. The retained topic-scoped DELETE migration check now skips SQL keywords and reaches the scoped table identifier. Submission metadata is always derived from the active runtime contract. The evidence pack identifies the reviewed branch head rather than an earlier parent commit. The preflight now uses the guest-produced normalized selection source, so whitespace-only selectors report the same selection branch the guest executes.

Reviews (11): Last reviewed commit: "fix(proof): read the selector from the g..." | Re-trigger Greptile

Comment on lines +442 to +452
while cursor < len(words) and words[cursor].upper() in (
"IF",
"NOT",
"EXISTS",
"OR",
"REPLACE",
"ONLY",
"INTO",
"UNIQUE",
"CONCURRENTLY",
):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Accept DELETE FROM migrations

The migration scope check treats FROM as the table name in a standard DELETE FROM <topic-scoped-table> statement. This rejects retained migrations such as DELETE FROM fixture_topic_v0_kept ... before re-authoring can complete, even though the authoritative SQL guard accepts the same topic-scoped statement. Skip FROM before validating the table identifier.

Artifacts

Evidence from the check

  • This authored Python script imports and executes the actual authoring preflight with a topic-scoped DELETE statement, showing the parser path under review.

Command output from the check

  • Running the authored reproduction shows tokenization and the actual rejection of `FROM` as an unscoped table name, confirming the defect.

Command output from the check

  • Running the narrow Rust namespace test passes and includes its accepted `DELETE FROM tb4_scratch` case, establishing the authoritative guard behavior.

Command output from the check

  • The captured Git status confirms validation added only evidence artifacts and did not modify product code.

View artifacts

T-Rex Ran code and verified through T-Rex

Comment on lines +688 to +693
retained_format = prior.get("submission_format")
submission_format = (
retained_format
if isinstance(retained_format, dict) and retained_format
else derive_submission_format()
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Refresh submission metadata

A populated prior submission_format is copied into the newly authored set instead of using the active format derivation. Re-authoring can therefore publish an obsolete intake contract—for example, a one-byte artifact limit and stale nonce or signature details while the current format allows 5 MiB—causing clients to prepare submissions that the current installation does not support. Derive the format for every run from the authoritative current intake configuration.

Artifacts

Evidence from the check

  • This authored Python script invokes the shipped propose_rules entrypoint with its environment contract and compares its output to the active format derivation, establishing the retention behavior.

Command output from the check

  • The fresh entrypoint execution completed with exit code 0 and authored the current 5 MiB format exactly matching derive_submission_format, establishing the baseline.

Command output from the check

  • The prior-state entrypoint execution completed with exit code 0 and copied the supplied one-byte stale format rather than the active derivation, confirming the defect.

Command output from the check

  • The focused shipped unittest suite completed with 25 passing tests, showing the existing tests currently encode rather than detect retention of the stale format.

View artifacts

T-Rex Ran code and verified through T-Rex

Comment on lines +5 to +7
**Tip under review:** `droid/8bcbefa3-sn100-stay-lit-harbor-propos` @ **`18a2532c`**
(PR [#304](https://github.com/CortexLM/cortex/pull/304), draft — the stack head,
stacked on #301 at `80bc2cdd`).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Update the reviewed tip

The evidence header identifies 18a2532c as the PR tip, but that commit is the direct parent of the reviewed head 7f226e00459b3b86d65366abc6b698af8e5104bb. This is non-blocking, but it makes it unclear whether the recorded gate results apply to the reviewed change and reduces the evidence pack's audit value.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Artifacts

Evidence from the check

  • Authored executable script resolves the two commits, prints their metadata, and performs ancestry and equality checks; it is the exact source executed for this verification.

Command output from the check

  • Captured `sed` and `git rev-parse HEAD` output shows the documentation names `18a2532c` while the checkout is `7f226e00459b3b86d65366abc6b698af8e5104bb`; the stated tip is already inconsistent with the reviewed patch.

Command output from the check

  • Complete output of the authored script, with command, working directory, and exit code, shows the supplied head equals HEAD and `18a2532c` is its direct ancestor; the docs tip is stale for this patch.

View artifacts

T-Rex Ran code and verified through T-Rex

Two defects Greptile reproduced on the guest-side entrypoint, both in
re-authoring:

- `check_migration_scope` skipped modifiers after a table keyword but not
  `FROM`, so a retained `DELETE FROM <topic>_kept …` was refused for
  "touching FROM" — a topic that prunes its own table could not re-author.
  The scan now skips SQL keywords in that position, exactly as the guard's
  own `is_sql_keyword` does, so the outer loop reaches the real name.
- a populated prior `submission_format` was copied into the new set, so a
  re-authoring run published a **previous host's** intake contract as the
  current one (Greptile's run emitted a 1-byte cap). That part is a fact
  about the runtime this run executes on, not a decision to keep: it is now
  always derived. Only `migrations` / `apis` are retained.

Each fix has a regression test verified non-vacuous by neutering it:
`test_a_retained_topic_scoped_delete_is_not_refused` (SystemExit on the old
`FROM` handling) and `test_the_submission_format_is_re_derived_not_retained`
(the stale object in the assertion), plus a Rust-side case that runs the
shipped entrypoint and holds the retained set to `TopicAuthoring::validate`.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — both P1s are fixed in 3bc31a2e with regression tests verified non-vacuous by neutering each fix, and the P2 (evidence header) is updated to name this tip. Details: the scope check now skips SQL keywords in the table-name position (so DELETE FROM <topic>_kept resolves to the table, exactly as the guard's own is_sql_keyword does), and submission_format is always derived rather than inherited — only migrations / apis are retained on re-authoring.

echobt and others added 3 commits September 16, 2026 15:31
The § 2b commands now carry the staging host's real paths and flags: the
rebake step with the image staging and the sha256sum that becomes the pin,
the CA file, the owner key path the overlay actually names, and the
master's own --admin-url (the publish call is
/challenge/proof/v1/admin/proof/topics).

The clone-diff against the legacy document is a table of what to expect
per part and what a red flag would look like — including the one that
matters: a `topic_document` source on the newest row means the ceremony
did not do what it is for. The B1 FIXED YAML is explicitly not the SoT
and is not re-run.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
A header naming its own commit goes stale the moment the commit exists.
The pack now lists the four commits it is made against, so a later
doc-only commit cannot make the header wrong.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — the evidence header no longer names its own commit (it lists the four commits the claims rest on), so the P2 cannot recur on a doc-only push. The two P1s are fixed in 3bc31a2e.

The contract README said what an adaptor is but not that changing one has
no effect until the image is re-baked — the failure mode is fail-closed and
therefore looks like a missing entrypoint rather than a stale pin
(NO_RLM_RULES for propose_rules, a missing results.json for a stale run).
Names both, and the re-pin step.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — final docs-only commit on the head: the guest runner contract README now states that a runner-tree change needs a rebake (fail-closed looks like a missing entrypoint, not a stale pin). No code changed since the 5/5 review of dea78bde.

The pack recorded the counts from before the Greptile fixes landed: the
Rust authorship gate is 5 tests (the retained-DELETE case was added with
the fix) and the Python suite is 26 cases. Both are re-run here.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — final commit is a docs-only count correction: the evidence pack now records 5 Rust authorship tests (the retained-DELETE case added with the P1 fix) and 26 Python cases, both re-run on this tip. No source changed since the 5/5 review.

The Owner LIVE run failed: the rebaked image carried the tip's runner tree
(`propose_rules` writing the whole set) but a **stale Sep-13 guest agent**
that read only `rules.json`. The adaptor wrote a full `authoring.json` (8
rules / 1 migration / 1 route) and no `rules.json`, so the orchestrator
answered 502 `adaptor wrote no rules.json`, the control plane 503, and no
row landed. The drive never bound authorship parts.

The agent and the adaptor are baked into the same image but come from two
sources: the agent is compiled from this repo, the adaptor tree is copied by
`bake-rootfs.sh --runner`. A rebake that refreshes one and not the other is
exactly this failure, and nothing compared them.

Two halves, neither alone enough:

- **Harvest the set.** `read_authored_set` reads `authoring.json` first and
  answers `AuthoredSet::Complete`; a `rules.json` beside it is the compat
  copy and must carry the same vector (`DUAL_EMIT_RULES_DISAGREE`). Neither
  file present is `NO_AUTHORING_OR_RULES` — a run that wrote nothing authored
  nothing, and the signed `checklist` is the operator's vector. This is the
  half that survives once every image is rebaked: a complete `authoring.json`
  is now sufficient on any agent.
- **Dual-emit.** The reference adaptor writes `rules.json` (the set's own
  `rules`, verbatim) before `authoring.json`, so one run answers a tip agent
  and a pre-set agent — the compatibility window, not the contract.

A disagreement is refused rather than resolved by preference: otherwise which
guest harvested the run would decide the topic's anti-cheat surface, and the
same run would score under one vector on a rebaked image and another on the
old one. The fragment is written first, so a run cut between the writes
leaves a fragment (which opens nothing) rather than a set a stale agent
cannot read.

Tests, each verified non-vacuous by neutering the fix: 9 reader tests in
`runner.rs` (set-first precedence, agreeing pair, disagreeing pair, neither
file, fragment-only, wrong topic, malformed fragment, redaction), the same
three cases through the real guest agent in `agent_tests.rs`, the adaptor's
own Python tests (fragment is the set's vector; fragment written first; a
refused run writes neither file), and the Rust adaptor gate now compares the
pair instead of asserting the fragment is absent.

Refusal strings are documented in the runbook table, the runner contract,
both adaptor READMEs, and §2i of the evidence pack.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — new head 01770fd5: the LIVE FAIL fix.

What happened live: the rebaked image carried the tip runner tree (propose_rules writing the whole set) but a stale Sep-13 guest agent that read only rules.json. The adaptor wrote a full authoring.json (8 rules / 1 mig / 1 route) and no rules.json → orchestrator 502 adaptor wrote no rules.json → 503, no row. The drive never bound authorship parts.

Two halves, neither alone enough:

  1. Harvest the set (crates/proof-vm-guest/src/runner.rs, read_authored_set): authoring.json is read first and answers AuthoredSet::Complete, held to the same shape / deny-list / policy-vs-document gates as before. A rules.json beside it is the compat copy and must carry the same vector, or the pair is refused (DUAL_EMIT_RULES_DISAGREE) — otherwise which guest harvested the run would decide the topic anti-cheat surface. Neither file present is NO_AUTHORING_OR_RULES (a run that wrote nothing authored nothing; the signed checklist is never a substitute). This is the half that survives once every image is rebaked.
  2. Dual-emit (…/rlm_fc_in_guest_harbor/harness/authoring_set.py): the adaptor writes rules.json (the set own rules, verbatim) before authoring.json, so one run answers a tip agent and a pre-set agent. The compatibility window, not the contract.

Tests, each verified non-vacuous by neutering the fix:

  • 9 reader tests in runner.rs — set-first precedence, agreeing pair, disagreeing pair, neither file, fragment-only stays a fragment, wrong topic, malformed fragment, secret redaction. Neutering the set-first precedence fails 5; neutering the pair comparison fails the disagreement case with expected Failed, got Done { output: Authored(…) }.
  • the same three cases through the real guest agent (agent_tests.rs::a_dual_emit_pair_whose_vectors_disagree_is_refused).
  • the adaptor Python suite: the fragment is the set vector; the fragment is written first; a refused run writes neither file.
  • the Rust adaptor gate (reference_adaptor_authoring.rs) now compares the pair instead of asserting the fragment is absent.

Gates run locally on the tip: cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test -p proof-vm-guest -p proof-vm-agent -p proof-rlm -p proof-topic-setup (green apart from 4 permission-dependent tests that fail identically at pristine 173ce178), loc-cap (guest 1340/1500), consensus-lint, spec-check, design-check, external-docs-check.

Not claimed: that a live image is fixed — the rebake carries this agent to the host, and that is Dev/Owner step. Not merged; the HOLD stands.

…skew

Arch GO's finding: the LIVE FAIL is host<->guest wire skew. The guest emits
`VmJobOutput::Authored`; a host binary built before `945e143f` cannot decode
that variant. `API_VERSION` stayed `1`, so the coarse guard on `Hello` does
not cover it.

Three gaps let it ship silently, each verified in the tree:

- `jobs_name_their_topic_and_outputs_round_trip` enumerated **every**
  `VmJobOutput` variant except `Authored` — the one variant whose payload is
  a whole document was the one never serialised.
- `proof-vm-agent`'s `FakeHypervisor` answered `ProposeRules` with `Rules`
  always, so the orchestrator's HTTP hop was never exercised with the variant
  that actually travels.
- `read_frame` mapped a serde failure to a bare `Decode(msg)`, so an unknown
  variant read like a corrupt frame instead of a build mismatch.

What this changes:

- `proof-vm-proto::guest::decode_skew` (public, shared by both host
  decoders) names an unknown variant as a **build skew** with both remedies
  and the variant it extracted; anything else stays a decode error. The
  control plane's orchestrator client uses it too, so both sides say the same
  thing.
- `Authored` is round-tripped through the frame codec with every part, and
  `the_authored_tag_is_a_wire_contract` pins the tag `authored` — what two
  separately built binaries agree on — with `rules` for the fragment.
- `the_whole_authored_set_survives_a_frame` and
  `the_whole_authored_set_crosses_the_job_hop` drive the real frame codec and
  the real `POST /v1/vms/{id}/jobs` router, asserting the tag, all five parts
  in the body, and completeness.
- The orchestrator's fake emits `Authored` by default
  (`set_authored_complete(false)` for the fragment path).

The full-set bind was already correct and is pinned where it happens:
`proof-topic-setup` binds `Authored` -> `validate_against_pin` -> the whole
set; `proof-topic-install` applies the RLM's set in place of the bundle's
section and journals each part as `source: rlm` with its own digest
(`assert_authorship_journal_names_every_part`, against a real Postgres).

Two bugs found and fixed in my own new code, both caught by neutering:
the variant extraction split on a double quote where serde uses backticks
(the test had passed on the frame preview alone), and the first tag assertion
was too weak to fail on a wrong tag. Both tests now pin the extracted value.

Tests verified non-vacuous by neutering: the tag assertion (wrong tag fails,
printing the full set), the skew diagnosis (`if false` fails), and the hop
test (fake forced back to `Rules` fails).

Docs: the orchestrator runbook gains "the wire is a contract between two
separately built binaries" with the promotion order; the evidence pack gains
section 2j with the three gaps, the per-layer fix, and the bind sites.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — new head bc02cc89: Arch GO's wire-skew finding.

The skew: the guest emits VmJobOutput::Authored; a host binary built before 945e143f cannot decode that variant, and API_VERSION stayed 1, so the Hello guard does not cover it. Verified against the real decoder in this container.

Why it shipped silently — three gaps, each verified in the tree:

  1. jobs_name_their_topic_and_outputs_round_trip enumerated every VmJobOutput variant except Authored — the one variant whose payload is a whole document was the one never serialised.
  2. proof-vm-agent's FakeHypervisor answered ProposeRules with Rules always, so the orchestrator's HTTP hop was never exercised with the variant that actually travels.
  3. read_frame mapped a serde failure to a bare Decode(msg), so an unknown variant read like a corrupt frame rather than a build mismatch.

Fix, per layer:

  • proof-vm-proto::guest::decode_skew (public, shared by both host decoders) names an unknown variant as a build skew with both remedies and the extracted variant; the control plane's orchestrator client (proof-vm-fc) uses it too. A genuinely malformed body stays a plain decode error.
  • Authored round-trips through the frame codec with every part; the_authored_tag_is_a_wire_contract pins the tag authored (what two separately built binaries agree on) and rules for the fragment.
  • the_whole_authored_set_survives_a_frame + the_whole_authored_set_crosses_the_job_hop drive the real frame codec and the real POST /v1/vms/{id}/jobs router, asserting the tag, all five parts, and completeness.
  • The orchestrator fake emits Authored by default (set_authored_complete(false) for the fragment path).

The full-set bind was already correct, and is pinned where it happens: proof-topic-setup:435 binds Authoredvalidate_against_pin → the whole set (a Rules answer becomes IncompleteAuthoring); proof-topic-install:274 applies the RLM's set in place of the bundle section; :489 journals each part as source: rlm with its own digest; install_engine.rs:521 walks PARTS against a real Postgres.

Two bugs in my own new code, both caught by neutering: the variant extraction split on a double quote where serde uses backticks (the test had passed on the frame preview alone — the assertion now pins the extracted value), and my first tag assertion was too weak to fail on a wrong tag. Both fixed.

Tests verified non-vacuous by neutering: the tag assertion (wrong tag fails, printing the full set), the skew diagnosis (if false fails), the hop test (fake forced back to Rules fails).

Gates on the tip: cargo fmt --check, workspace clippy -D warnings, cargo test --workspace (227 suites green; the same 4 permission-dependent tests fail identically at pristine 173ce178), loc-cap (guest 1340/1500), consensus-lint, spec-check, design-check, external-docs-check.

Dual-emit kept (authoring.json + rules.json, fragment written first). Not merged — the HOLD stands.

@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Answer to Arch GO: host-only rebuild is NOT enough — the guest must be rebaked. Verified from the tree, not inferred.

Two distinct errors, two distinct layers:

Error Emitted by Meaning
adaptor wrote no rules.json guest agent (baked in the image, runner.rs:732) the agent read only rules.json; it never knew authoring.json
unknown variant \authored`` host decoder (orchestrator + CP, proof-vm-proto) the host has no Authored variant

The failure matrix:

Guest agent Host binaries Result
Sep-13 (reads only rules.json) tip dual-emit gives it the fragment → VmJobOutput::RulesSetupError::IncompleteAuthoring { missing: [migrations, apis, submission_format, pin_policy] }. Topic cannot open.
tip (emits Authored) stale decode fails → 502 (now names the build skew instead of looking like corruption)
tip tip works; the topic opens

Why a host rebuild alone cannot fix it: only Authored satisfies the publish gate. proof-topic-setup:449 turns a Rules answer into missing = PARTS − rules, and :704 returns IncompleteAuthoring. The Sep-13 agent returns Result<Vec<ChecklistRule>, String> (runner.rs:975 at 7bf20a4a) — it is structurally incapable of emitting the set. So with 48c00e5e kept, the drive still refuses to open the topic; the 502 becomes a named refusal, not a success.

What the digest 48c00e5e must contain for the guest to stay: the tip agent (AuthoredSet, this stack) and the tip runner tree (propose_rules writing authoring.json, 18a2532c). If it is the Sep-13 surgical rebake, it has the runner tree but not the agent, and it must be rebaked from bc02cc89.

Detectability is the gap I could not close here: AGENT_NAME is proof-vm-guest-agent/<workspace version>, the workspace version is 0.1.0 and has not changed, and the host logs the reported name without checking it (proof-fc-host/src/lib.rs:470check_version only covers api_version, which stayed 1). So an operator cannot tell from the handshake which agent generation an image carries. I did not bump api_version or the version string: both are wire/contract changes with their own blast radius, and they are a decision for the GO, not a silent fix inside a DRAFT. Flagging it as the follow-up.

Order matters: rebuild the host binaries from bc02cc89 before an image whose agent emits Authored reaches the host, or accept the (now clearly named) 502 in between. Not merged; HOLD stands.

… host does

The in-process agent tests assert `VmJobOutput::Authored` on the Rust type, and
the wire tests drive the codec with a fixture. Neither ran the **real** agent's
answer through a frame: the live FAIL fell through exactly that gap, where a
guest that emits the set and a host that cannot decode it never meet in a test.

`the_real_agent_frames_the_whole_set_the_host_decodes` drives the shipped
`GuestAgent` over a real duplex channel with the reference adaptor's shape
(`authoring.json` plus its compat `rules.json`), reads the `Done` frame back
with the wire types, and requires the whole set with every part — plus the
`authored` tag on the raw JSON.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — final head 87466d4a (one test file on top of the reviewed bc02cc89).

The addition closes the last coverage gap the wire-skew RCA exposed: the in-process agent tests asserted VmJobOutput::Authored on the Rust type, and my wire tests drove the codec with a fixture — neither ran the real agent answer through a frame. the_real_agent_frames_the_whole_set_the_host_decodes drives the shipped GuestAgent over a real duplex with the reference adaptor shape (authoring.json + its compat rules.json), reads the Done frame back with the wire types, and requires the whole set with every part plus the authored tag on the raw JSON.

bc02cc89 (the wire-contract fix) is unchanged; this is additive test-only. Gates on the tip: fmt, clippy -p proof-vm-guest -D warnings, the five xtask gates, and the affected suites (proof-vm-guest 36 passed with the same 2 permission-dependent tests that fail at pristine 173ce178, proof-vm-proto 14, proof-vm-agent 41, proof-vm-fc 11+5+2).

@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

For Dev's parallel rebuild — the discriminating check, because "current /opt/base tip" is ambiguous and one reading of it silently does nothing.

remote-deploy.sh rsyncs the local tree to the host (rsync -az --delete "$ROOT/" "$HOST:$REMOTE_DIR/", line ~169), so /opt/base's "tip" is whatever checkout was rsynced. Verified ancestry:

origin/main (aabd1724)          -> 945e143f is NOT an ancestor  -> cannot decode `authored`
droid/8bcbefa3-…@87466d4a       -> 945e143f IS an ancestor      -> decodes it

So a host rebuilt from a main-based checkout does not gain the authored variant, and the 502 stays — just now with a clear build-skew message instead of an opaque one. The rsynced tree must come from this stack.

One-line check before rebuilding, on whichever box holds the checkout:

git merge-base --is-ancestor 945e143f HEAD && echo "can decode authored" || echo "CANNOT — use the stack tip"

Or grep the source, which is what actually compiles:

grep -c authored crates/proof-rlm/src/vm.rs   # 0 on main, >0 on this stack

The two binaries that must be rebuilt from 87466d4a (or bc02cc89):

Binary Where it runs Why
proof-challenge CP/master (compose) decodes the orchestrator's HTTP answer (proof-vm-fcRunJobResponse)
proof-vm-orchestrator KVM host (systemd unit, not compose) decodes the guest's vsock frame (proof-fc-hostRlmToHost)

Both pull the same proof-vm-proto types, so both need it. The orchestrator is a systemd unit on the KVM host (cargo build --release -p proof-vm-orchestrator-bin, runbook § Build), not something remote-deploy.sh reaches — it has to be copied over and the unit restarted separately.

Reminder of the other half: the host rebuild is necessary but not sufficient. With the Sep-13 agent still in 48c00e5e, the dual-emit gives that agent the fragment, which the CP records honestly and refuses (IncompleteAuthoring) — the topic cannot open. The guest rebake is what makes the set exist; the host rebuild is what lets the set be read. Not merged; HOLD stands.

The promotion-order paragraph said "rebuild the host binaries from the tip"
without saying which checkout that is. It matters: `remote-deploy.sh` rsyncs
the **local tree** (`"$ROOT/"` -> `$REMOTE_DIR/`), so `/opt/base`'s tip is
whatever checkout was rsynced — and `Authored` (`945e143f`) is not an ancestor
of `main` (`aabd1724`), so a host rebuilt from a main-based checkout gains
nothing and keeps 502ing.

Adds the discriminating probe (`git merge-base --is-ancestor 945e143 HEAD`,
and `grep -c '"authored"' crates/proof-rlm/src/vm.rs`), verified against both
trees in this container (2 on the stack, 0 on main), and names the two
binaries that need it — `proof-challenge` (compose, CP) and
`proof-vm-orchestrator` (systemd unit on the KVM host, which remote-deploy
does not reach).

Also states what the rebuild alone does not do: with an agent that predates
the set, the dual-emit gives it the fragment and the control plane refuses
(`IncompleteAuthoring`). The rebake is what makes the set exist.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — final head ca58c60b (docs-only on top of the reviewed 87466d4a).

The promotion-order paragraph in the orchestrator runbook said "rebuild the host binaries from the tip" without naming which checkout. That gap is real and load-bearing: remote-deploy.sh rsyncs the local tree ("$ROOT/"$REMOTE_DIR/), so /opt/base's tip is whatever checkout was rsynced — and Authored (945e143f) is not an ancestor of main (aabd1724). A host rebuilt from a main-based checkout gains nothing.

The commit adds the discriminating probe and names the two binaries that need it (proof-challenge in compose on the CP; proof-vm-orchestrator as a systemd unit on the KVM host, which remote-deploy.sh does not reach). Probe verified against both trees in this container: grep -c authored crates/proof-rlm/src/vm.rs2 on the stack, 0 on main. It also states what the rebuild alone does not do — the rebake is what makes the set exist.

Docs-only; no source changed since the 0-comment review of 87466d4a. Five xtask gates green. Not merged; HOLD stands.

…pin the early refusal

Arch GO's Gate-1 add-on: fail closed early when `task_slice` is set but
unresolved and `params.tasks` is empty, and prefer an explicit
`params.tasks` / `n_tasks` set for RLM-emitted SoT installs.

**The literal refusal already lands on this stack** (`b2c8fe9e`, "an
unresolved task_slice fails closed, always"; on the stack, not on `main`).
Verified by driving the shipped `filter_tasks.py` against throwaway packs
rather than assuming: a lone unresolved label is refused whether the pack has
no slices, an `allow` list, or a `n_tasks` bound beside it, and a resolvable
label still resolves with `task_slice_resolved: true`.

**What was actually broken: the preflight refused the escape.** The same
matrix through `proof-slice-preflight.sh` — the tool that exists to catch a
bad selection before a VM — rejected the documented escape
(`--task-slice <stale> --tasks t-one,t-two`) because it asserted any set label
must have resolved. The guest reads `params.tasks` **instead of** the label,
so the run is correct and the preflight was wrong: it blocked the exact shape
the Owner's re-sign produces. The assertion now follows the guest, both
directions pinned.

**The ordering property is pinned, not assumed.** Gate 1's cost was a
provision + boot + a Harbor run that overran its wall clock, so "fail-closed"
is only half of it. `test_adaptor.sh` drives the real entrypoint with a fake
`harbor` that records its own invocation and requires a refused selection to
leave no invocation, no jobs dir, and no report — with a positive control (the
escape reaches Harbor) so it cannot pass because the entrypoint is broken.

**The stale advice is corrected.** `docs/PROOF.md` and the smoke runbook still
said a label the pack does not define "stays an informational label" — the
pre-`b2c8fe9e` behavior, and the fail-open that produced Gate 1. Both now state
the refusal and name the supported path (`params.tasks` + `n_tasks`), with the
preflight command to verify it before a re-sign.

Tests verified non-vacuous: restoring the preflight's unconditional label
assertion fails the new case, and reintroducing the fail-open in the adaptor
fails the suite. Two bugs in my own new test found and fixed by running it:
the invocation marker was read from the wrong directory (making the refusal
assertion vacuous) and the control case lacked the pack/artefact fixtures.

Gates: fmt, workspace clippy -D warnings, the five xtask gates, the adaptor
suites (180 Python + the shell suite), the preflight suite, and the affected
Rust crates (the same 2 permission-dependent guest tests fail as at pristine
`173ce178`).

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — new head 46c32399: the Gate-1 class (FAIL #3).

Checked before changing anything, because the literal refusal already lands here. b2c8fe9e ("an unresolved task_slice fails closed, always") is on this stack and not on main (git merge-base --is-ancestor b2c8fe9e origin/main → no). I drove the shipped filter_tasks.py against throwaway packs rather than assuming:

Selection Pack Result
task_slice=tb4-first-5 alone no slices refused, names "defines no slices"
task_slice=tb4-first-5 + n_tasks=5 no slices refused (a count is not a selector)
task_slice=tb4-first-5 alone allow list refused (no fall-through to allow)
task_slice=five alone defines slices/five.json resolves, resolved: true
task_slice=tb4-first-5 + params.tasks any runs on the tasks; the label is not read

What was actually broken: the preflight refused the escape. The same matrix through proof-slice-preflight.sh — the tool that exists to catch a bad selection before a VM — rejected the documented escape (--task-slice <stale> --tasks t-one,t-two), because it asserted any set label must have resolved. The guest reads params.tasks instead of the label, so the run is correct and the preflight was wrong: it blocked the exact shape the Owner's re-sign produces. The assertion now follows the guest, both directions pinned.

The ordering property is pinned, not assumed. Gate 1's cost was a provision + boot + a Harbor run that overran its wall clock, so "fail-closed" is only half of it. test_adaptor.sh drives the real entrypoint with a fake harbor that records its own invocation and requires a refused selection to leave no invocation, no jobs dir, no report — with a positive control (the escape reaches Harbor) so it cannot pass because the entrypoint is broken.

The stale advice is corrected. docs/PROOF.md and the smoke runbook still said a label the pack does not define "stays an informational label" — the pre-b2c8fe9e behavior and the fail-open that produced Gate 1. Both now state the refusal and name the supported path (params.tasks + n_tasks), with the preflight command to verify before a re-sign.

Tests verified non-vacuous: restoring the preflight's unconditional label assertion fails the new case; reintroducing the fail-open in the adaptor fails the suite. Two bugs in my own new test, found by running it: the invocation marker was read from the wrong directory (making the refusal assertion vacuous) and the control case lacked the pack/artefact fixtures.

Gates on the tip: fmt, workspace clippy -D warnings, the five xtask gates, the adaptor suites (180 Python + the shell suite), the preflight suite, and the affected Rust crates (the same 2 permission-dependent guest tests fail as at pristine 173ce178).

No merge, no staging tip — the HOLD stands.

@greptile-apps

greptile-apps Bot commented Sep 16, 2026

Copy link
Copy Markdown

P1 Fix the positive-control fixture's task name

The new control case cannot reach Harbor. The pack created above contains only t-one, t-two, and t-three, but the control exports PROOF_PARAM_TASKS=alpha at deploy/guest/runners/rlm_fc_in_guest_harbor/tests/test_adaptor.sh:466. filter_tasks.py therefore refuses alpha as absent from the staged pack before the fake harbor is invoked, so the assertion that the escape reaches Harbor fails. The fake result is also hard-coded as alpha__1; once the selector is corrected, that trial name must match the selected task or summarize.py will reject it as outside the filtered set. Use the same available task name in the pack, PROOF_PARAM_TASKS, and the fake Harbor result fixture.

Tip: You can customize Greptile's behavior for this repo with .greptile/rules.md and .greptile/config.json.

@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — pinging for 46c32399 (the head above; the previous review was on ca58c60b). Docs + two shell test suites + one shell script; no Rust changed in this commit.

Comment thread deploy/scripts/proof-slice-preflight.sh Outdated
Comment on lines +185 to +191
if [ -n "$task_slice" ] && [ -z "$tasks" ]; then
case "$resolved" in
true) pass "task_slice '$task_slice' resolved through the pack (resolved=true)" ;;
*) die "task_slice '$task_slice' did not resolve (resolved=$resolved) — the guest would refuse this run" ;;
esac
elif [ -n "$task_slice" ] && [ -n "$tasks" ]; then
pass "task_slice '$task_slice' is not read: params.tasks named the set (source=$source)"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Normalize selector checks

Whitespace-only selector values do not have the same meaning here as they do in the guest. If both --tasks and --task-slice contain whitespace, the guest normalizes them as absent and runs the full task directory, but this branch succeeds while claiming that params.tasks selected the set. A whitespace-only --tasks beside a valid slice similarly causes the preflight to claim explicit-task selection even though the guest selects the slice. This can hide an unexpectedly broad Harbor run or give an operator the wrong selection explanation; use the filter summary's normalized source and task_slice_resolved values for these checks.

Artifacts

Whitespace selector reproduction

  • The authored differential reproduction exercises guest and preflight handling of whitespace-only selector values.

Guest selector output

  • The guest output shows whitespace-only selectors are normalized as absent, so it selects the slice or full task directory.

Preflight selector output

  • The preflight output shows the conflicting rejection and incorrect explicit-task explanation.

Targeted preflight suite

  • The passing targeted suite confirms existing coverage does not include whitespace-only selector cases.

View artifacts

T-Rex Ran code and verified through T-Rex

…'s variables

Greptile P2 on the previous commit. That version decided "was the label the
selector?" from the shell's own `--tasks` / `--task-slice` values. The guest
normalizes a whitespace-only value as **absent** (`present()` trims, then
rejects empty), so `--tasks "   "` beside a resolvable slice selects the
**slice** — while the preflight passed and told the operator "params.tasks
named the set". That is the wrong explanation attached to a run that took a
different branch, on the exact shape a re-sign produces.

Reproduced independently before changing anything (throwaway packs, both
directions), then fixed: the branch now reads the summary's normalized
`source`, which is what the guest actually used. A `params.tasks` source says
the label was not read; a `pack slice <label>` source says it resolved; any
other source with a label set is the LIVE refusal, asserted rather than
assumed because reaching it means the guest and the preflight disagree.

Both whitespace directions are pinned, and reinstating the raw-variable
branch fails the suite (verified non-vacuous). The preflight and adaptor
suites both pass.

Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
@echobt

echobt commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

@greptileai review — b6c6bf25 addresses the P2 from the 46c32399 review (the only comment it added).

The finding was real and I reproduced it before changing anything. My first version decided "was the label the selector?" from the shell's own --tasks / --task-slice values. The guest normalizes a whitespace-only value as absent (present() trims, then rejects empty), so --tasks " " beside a resolvable slice selects the slice — while the preflight passed and reported "params.tasks named the set". Wrong explanation, different branch, on the exact shape a re-sign produces.

Fix: the branch now reads the summary's normalized source — what the guest actually used. params.tasks means the label was not read; pack slice <label> means it resolved; any other source with a label set is the LIVE refusal, asserted rather than assumed (reaching it means the guest and the preflight disagree).

Both whitespace directions pinned: whitespace --tasks beside a slice reports the slice as the selector; whitespace --task-slice beside real tasks reports no label read. Reinstating the raw-variable branch fails the suite (verified non-vacuous). Preflight suite, adaptor suite (180 Python + shell), and the five xtask gates all pass.

Still no merge, no staging tip.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant