fix(proof): attach miner harbor agent on evaluate - #266
Conversation
Live Harbor evaluate ignored PROOF_ARTIFACT_DIR and the miner BYOK file, always running terminus-2 on the operator pack with the owner key. Ship a versioned rlm_fc_in_guest_harbor adaptor that resolves a Harbor module:Class from the artefact (Harbor -a is not a path), refuses topic-agent fallback on evaluate, and never falls back to the owner key when miner_byok is set. Co-authored-by: Mathis <echobt@users.noreply.github.com>
|
@greptileai review |
Greptile SummaryThis PR adds a versioned in-guest Harbor adaptor for Proof evaluation, including miner-provided agent selection, BYOK credential handling, result summarization, artifact inspection, deployment documentation, and adaptor tests. No code has changed since the previous review, and the previously identified blocking evaluation-integrity issues remain unresolved. Confidence Score: 0/5Not merge-safe: unresolved evaluation-integrity and artifact-isolation issues can affect paid Proof results. Five outstanding previous findings remain unresolved. A nonzero Harbor exit is still suppressed before a report is written, allowing partial results to be treated as a successful evaluation. Result collection still stops after 256 measured trials, so the reported score can omit completed results. Artifact scanning still stops at its file limit and treats the scanned subset as clean, which can produce false-green prohibited-content evidence. Miner-controlled import paths are still accepted without confirming that the imported module originates within the staged artifact. Unsupported rule IDs still pass when an artifact directory exists rather than being evaluated. The bounded scan and external import-path issues are unresolved security findings. Reviews (2): Last reviewed commit: "fix(proof): attach miner harbor agent on..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| set +e | ||
| "${harbor_cmd[@]}" >"$log" 2>&1 | ||
| harbor_exit=$? | ||
| set -e | ||
|
|
||
| python3 "$HERE/summarize.py" \ |
There was a problem hiding this comment.
When Harbor writes some trial results and then exits unsuccessfully, this runner suppresses that failure and still writes a successful report. A run that wrote one result before exiting with status 23 produced primary_value: 0.6 and claim_holds: true; the guest runner accepts that finite report and forwards its claim status. This can publish and pay a score derived from an incomplete evaluation. Reject nonzero Harbor exits, or require verified completion of the expected trial set before producing a passing report.
Knowledge Base Used: Proof challenge network and evaluation
Artifacts
Partial Harbor validation script
- This authored executable creates an empty-result control and a fake Harbor that writes one reward before exiting 23, then runs the real harness and asserts the resulting report, showing the tested failure condition.
- The control invocation of the real summarizer exited 2 and refused to create report.json when no measured results existed, establishing the baseline rejection behavior.
Partial-results failure output
- The real harness returned exit code 0 after fake Harbor exited 23, and its emitted report contains primary_value 0.6 and claim_holds true, confirming partial output is accepted.
Downstream report contract output
- The focused proof-vm-guest contract test passed, corroborating that a finite report and claim_holds field are accepted and forwarded downstream.
| for result_path in sorted(jobs_dir.rglob("result.json")): | ||
| if len(trials) >= MAX_EVIDENCE_TRIALS: | ||
| break |
There was a problem hiding this comment.
The summarizer stops after the first 256 lexically sorted result files, then calculates both primary_value and n_measured from that subset while still setting claim_holds to true. With 260 valid results, the four lexically later rewards of 100 were omitted: the report scored 0.0, while all 260 results average 1.5384615384615385. This makes the authoritative score depend on file ordering rather than all completed trials. Score every measured result and bound only serialized evidence, or fail closed when collection is incomplete.
Knowledge Base Used: Proof challenge network and evaluation
Artifacts
Result-cap reproduction script
- Authored Python harness creates exact-cap and 260-file Harbor-shaped job trees and invokes the real summarizer, demonstrating whether all measured files affect the report; it proves the test scope.
- Executed baseline command in `/home/user/repo` with 256 measured results, showing all 256 are reported and score to zero; it establishes the cap boundary.
- Executed overflow command in `/home/user/repo` with 260 measured results, showing only the first 256 are reported, the score remains zero, and `claim_holds` remains true; it confirms silent lexicographic truncation.
| names.append(rel) | ||
| n_files += 1 | ||
| if n_files > MAX_FILES or total > MAX_TOTAL_BYTES: | ||
| break |
There was a problem hiding this comment.
The scanner stops reading after its file limit but treats the absence of a marker from the truncated text as proof that the off-limits rule passes. An artifact containing 256 lexically earlier benign files and a later z-forbidden.txt containing no_tb4_hardcoding completed successfully with that rule marked passing. This can create false-green integrity evidence at the paid evaluation boundary and allow prohibited hardcoding to be evaluated or promoted. Mark bounded scans as incomplete and fail the relevant rule rather than treating them as clean.
How this was verified: A 257-file artifact with the forbidden marker after the file limit emitted a passing off-limits checklist entry.
Knowledge Base Used: Proof challenge network and evaluation
Artifacts
Scan-limit reproduction script
- Authored and executed Python reproduction that builds the same-size baseline and lexically later forbidden-marker cases, invokes the real scanner, and prints its emitted checklist state; it confirms the tested path.
- Captured output of `python3 trex-artifacts/inspect-scan-limit-repro.py baseline` from `/home/user/repo`, including command, working directory, exit code, and emitted passing checklist for the clean baseline; it establishes expected clean behavior.
Marker after scan limit output
- Captured output of `python3 trex-artifacts/inspect-scan-limit-repro.py marker` from `/home/user/repo`, including command, working directory, exit code, and emitted passing checklist despite `z-forbidden.txt` containing `no_tb4_hardcoding`; it confirms the bypass.
| if ":" not in line or line.startswith("acp:"): | ||
| _fail( | ||
| f"{path} must name a Python import path module.path:ClassName; " | ||
| f"got {line!r} (a built-in Harbor name is not miner code)" | ||
| ) | ||
| if ".." in line or "/" in line or "\\" in line: | ||
| _fail(f"{path} is not a Python import path: {line!r}") | ||
| return line |
There was a problem hiding this comment.
If the Harbor environment exposes a matching external Python module, a miner-controlled import_path can select it instead of code staged in the artifact. The resolver accepts module:Class based only on syntax and passes it to Harbor without resolving or checking the module origin. In the evaluation flow, outside_agent:ExternalAgent was accepted and Harbor imported /tmp/.../external/outside_agent.py, outside the staged artifact. This permits scoring code that was not supplied by the miner artifact while it is treated as the selected agent. Resolve the module in the evaluation import environment and reject any origin outside the resolved artifact directory.
How this was verified: An artifact-controlled import path selected a module whose loaded Python file was outside the staged artifact.
Knowledge Base Used: Proof challenge network and evaluation
Artifacts
External import-path validation script
- This authored script creates a staged artifact whose import_path names a module supplied only outside the artifact, runs resolver-only and Harbor-flow modes, and asserts the imported origin is external; it is the reproducible proof harness.
- This executed resolver capture shows the artifact import_path `outside_agent:ExternalAgent` was returned with exit code 0 before any Harbor invocation; the resolver does not enforce module-origin containment.
External module execution output
- This executed fake-Harbor capture shows the adapter passed `outside_agent:ExternalAgent` to Harbor and Harbor imported `/tmp/.../external/outside_agent.py` outside the staged artifact; containment was not enforced before invocation.
Harbor adaptor regression output
- This executed repository adapter suite completed successfully after the focused validation artifacts were added; the existing tests pass but do not cover external module-origin containment.
| return { | ||
| "id": rid, | ||
| "pass": True, | ||
| "evidence": _clip( | ||
| f"rule {rid}: artefact present ({n_files} files); no paid inference; " | ||
| "no off-limits short-circuit/tb4 markers beyond the dedicated probes" | ||
| ), | ||
| } |
There was a problem hiding this comment.
Any substantive rule not explicitly handled by the scanner passes whenever an artifact directory exists, without evaluating the rule ID, its text, or supporting evidence. The unknown requirement must_provide_reproducible_benchmark changed from failing with no artifact directory to passing after adding one unrelated non-proving file. A newly introduced artifact requirement can therefore generate green checklist evidence without being checked, allowing evaluation to proceed without satisfying that requirement. Fail closed for unsupported rule IDs until a rule-specific evaluator is implemented.
Knowledge Base Used: Proof challenge network and evaluation
Artifacts
Unsupported-rule reproduction script
- Runs inspect_scan.py with the same unknown benchmark rule before and after staging an artifact directory, capturing the checklist output; it reproduces the unsupported-rule fallback.
No-artifact unknown-rule output
- Executed `trex-artifacts/unsupported-artifact-rule-repro.sh before` in `/home/user/repo` with exit code 0; the unknown rule is marked `pass: false` because no artifact tree was staged.
Artifact-present unknown-rule output
- Executed `trex-artifacts/unsupported-artifact-rule-repro.sh after` in `/home/user/repo` with exit code 0; the same unknown substantive rule is marked `pass: true` merely because one artifact file exists.
2 participants
Summary
Live tbench Harbor evaluate on prod metal ignored the miner artefact and the miner BYOK file:
harbor run … -a terminus-2always used the operator pack agent.$PROOF_ARTIFACT_DIRwas never read.PROOF_SECRETS_DIR/$PROOF_PARAM_INFERENCE_KEY_FILE, violatingminer_byok_openrouter.Inspect already walked the artefact. Evaluate did not. This PR ships a versioned reference adaptor operators can bake/deploy so miner code actually affects scoring.
Harbor
-a/--agent(verified against Harbor CLI +AgentFactory) accepts a built-in name or a Python import pathmodule.path:ClassName— not a filesystem path. Evaluate therefore:$PROOF_ARTIFACT_DIR/agent, else$PROOF_ARTIFACT_DIR/recipe/agentPYTHONPATHand passes-a module:Class(AST resolve; optional one-lineimport_pathfile)recipe/run.shwithout an agent dir (classic marker is not a Harbor-atarget)PROOF_PARAM_HARBOR_AGENT(terminus-2) on evaluate when an artefact was stagedPROOF_ARTIFACT_DIRunsetBYOK: evaluate exports
$PROOF_MINER_ENV_DIR/$PROOF_PARAM_MINER_BYOKand refuses the owner key. Owner key remains for baseline when miner_byok is unset / no miner file is staged.report.jsonis the mean of Harbor trialverifier_result.rewards.rewardonly. A trial with no such field is not a measurement.Greptile P1s (follow-up on this branch)
Fail-closed scoring / integrity fixes for the T-Rex-verified P1s (commit
fix(proof): fail closed on incomplete harbor evaluate):report.json;summarize.pyalso refusesharbor_exit != 0.module:Classis resolved in the evaluate import env (artefact parent only) and rejected when the origin is outside the staged artefact.Owner semantics unchanged: Harbor
-ais an import path; evaluate FAIL CLOSED without an agent at$PROOF_ARTIFACT_DIR/agentor…/recipe/agent; miner BYOK required (no owner-key fallback);terminus-2only for baseline with no miner artefact.How operators bake / deploy
Preferred (guest image):
--runnercopies this tree to/opt/proof/runners/rlm_fc_in_guest_harbor/.runexecs the in-treeharness/run-harbor. Do not keep the old overlay/opt/proof/harness/run-harboras the evaluate path — that is the bug.Metal copy (then re-bake/remount so the guest actually contains it):
Harbor CLI/venv and the task pack stay operator overlays. Topic
baseline_runnermust berlm_fc_in_guest_harbor.Miner attach layout:
docs/external-miner/proof-tbench.mdanddeploy/guest/runners/rlm_fc_in_guest_harbor/README.md.Test plan
deploy/guest/runners/rlm_fc_in_guest_harbor/tests/run.sh(no Harbor, no podman): agent selection, evaluate refusesterminus-2/recipe/run.shfallback, BYOK evaluate vs owner key, summarize mean / fail-closed / redact, inspect off-limits markersimport_pathcargo run -p xtask -- external-docs-checkcargo fmt --all -- --check(CI)Greptile
@greptileaiposted after the pushRisk
Operators must re-bake or replace the live overlay/adaptor for this to affect scoring. Shipping the tree in git does not change prod until the guest image (or
/opt/proof/runners/rlm_fc_in_guest_harbor) is updated. NoBASE_*env, deployed path, or crypto domain tag changes.Naming
I did not rename
BASE_*environment variables, deployed host paths(
/opt/base,/run/base, …), GHCRbaseintelligence/basepackage names, orbase-*-v1cryptographic domain tags.