diff --git a/README.md b/README.md index b89c475..eec199c 100644 --- a/README.md +++ b/README.md @@ -84,16 +84,61 @@ Legacy Eval v1/v2 state is neither read nor migrated. ## Host experiment policy The experiment population is the next 20 eligible root Codex tasks after the -managed host policy becomes active. Codex loads global `~/.codex/AGENTS.md` +managed host policy becomes active. The Host launches every candidate with the +preflighted `ward` permission profile. Launching a candidate with another +profile is a provisioning failure that aborts the experiment run; it is not a +reason to exclude that task afterward. Codex loads global `~/.codex/AGENTS.md` guidance at session start, so only fresh tasks are eligible after installation. See the [OpenAI AGENTS.md documentation](https://learn.chatgpt.com/docs/agent-configuration/agents-md). -Eligible terminal outcomes are `completed`, `failed`, and `abandoned`. Eval -development, pure Q&A, and subagent child tasks are excluded in advance. Ward -or Seal usage and the quality of the result never exclude a task afterward. +Before activation, the Host adds only the Eval state directory +`$XDG_STATE_HOME/jgoneit/eval-experiment` (or the platform fallback shown +above) as an explicit writable workspace root for the selected permission +profile, creates its private directories with mode `0700`, ensures any files it +creates use mode `0600` on Darwin/Linux, and verifies writes against a separate +preflight state root. For a managed permission profile, this workspace-root +rule is authoritative and must not be combined with legacy +`sandbox_workspace_write` settings. See the [OpenAI Permissions documentation](https://learn.chatgpt.com/docs/permissions). + +A preflight always passes a disposable absolute child path beneath that granted +Eval state directory through `--state-root` and confirms that its journal path +differs from the production journal before it runs. It must not create or +modify the production journal. If state provisioning or lifecycle calibration +fails, the Host marks that experiment run aborted and restarts with a new +activation time; it does not silently reuse the failed run or migrate its +state. If an aborted run already contains rows, the Host quarantines that exact +journal subtree before starting the replacement run rather than appending new +samples to it. + +The end of an assistant turn is not by itself a terminal task outcome. The +same primary objective remains nonterminal only while an identified, in-scope +continuation can resume it after user input, approval, configuration, or +permission becomes available. A recoverable Ward or Seal stop, clarification +request, or resumable handoff is therefore nonterminal and must not be mapped +to `failed` or `abandoned`. + +Eligible terminal outcomes have these narrower meanings: + +- `completed`: the requested objective is achieved and no required work remains; +- `failed`: the objective cannot be completed within the current scope and + granted authority, no identified in-scope continuation remains pending, and + the task is being closed; +- `abandoned`: the Host explicitly supplies the outcome; the Agent never infers + abandonment from user silence. + +Without an explicit Host terminal callback, an abandoned task contributes only +to the Host denominator and missing-attempt count; it cannot produce an Agent +observation. + +Eval development, pure Q&A, and subagent child tasks are excluded in advance. +Ward or Seal usage and the quality of the result never exclude a task afterward. For an eligible task, the primary Agent makes one silent `evalctl observe` -attempt after the task outcome is fixed. It does not: +attempt only after the eventual terminal outcome is fixed. It makes no attempt +during a nonterminal turn; if the same task resumes, it preserves eligibility +for the single terminal attempt. The attempt pipes one complete JSON object to +standard input in the same shell command; bare `evalctl observe` is invalid and +must never be called. It does not: - run Ward or Seal to discover a metric or version; - ask a question or request approval; @@ -102,7 +147,12 @@ attempt after the task outcome is fixed. It does not: The Host task history supplies the population denominator and missing-attempt count. The journal intentionally does not store skipped rows or task identity. -The managed host block is removed after the twentieth eligible task. +The managed host block is removed after the twentieth eligible task. If the +same objective reopens after a terminal attempt, the Agent makes no second +attempt. The Host treats that reopen as a lifecycle-calibration failure, aborts +the experiment run, and restarts with a new activation time rather than +continuing with the stale row. This minimal experiment does not correct or +supersede recorded rows. ## Decision boundary