Post-7.3 eval guards for thin-router planning - #26
Conversation
Extend plan-only-stops to fail worker dispatch during planning and flow_plan_save without prior flow_guidance. Pin both in committed cassettes, add guidanceSkipped eval metrics, ratchet prompt ceiling to 16KB, and add skipped-case-named-binding for ADR 0012 isolation. Co-authored-by: vriesd <vriesd@users.noreply.github.com>
Bump package and install pins, move eval-guard notes into dated changelog, and align new cassettes to 7.3.2. Main push creates v7.3.2 and publishes. Co-authored-by: vriesd <vriesd@users.noreply.github.com>
Keep the [Unreleased] heading for the rolling contract; an empty section is enough when the latest notes are in a dated release. Co-authored-by: vriesd <vriesd@users.noreply.github.com>
Top entry is always the current package version. Release prep adds a dated section; the contract test enforces that instead of an empty Unreleased block. Co-authored-by: vriesd <vriesd@users.noreply.github.com>
Co-authored-by: vriesd <vriesd@users.noreply.github.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 86f0077a90
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| check(outcome) { | ||
| return skippedCaseRefusedIssues(outcome); |
There was a problem hiding this comment.
In the new skipped-case-named-binding scenario, a model can declare { platform: "win32", assertions: ["linux-skipped observation"] }, receive a platform-mismatch refusal, and close as deferred; this shared checker then returns no issues because it verifies only that some assertion was named. The scenario therefore passes without exercising the named-case rule it was added to isolate, recreating the exact confound described in its qualification comment. Require the relevant evidence entry to target the current Linux host (and the expected assertion) before accepting the refusal.
Useful? React with 👍 / 👎.
| test.skipIf(process.platform === "linux")("linux-skipped observation", () => { | ||
| expect(safeWindowsFileName("con")).toBe("_con"); | ||
| }); |
There was a problem hiding this comment.
When this eval is run on macOS or Windows, process.platform === "linux" is false, so linux-skipped observation executes and passes rather than being skipped. A correct completed workflow is nevertheless failed unconditionally by skippedCaseRefusedIssues as completion over a skipped case, corrupting cross-platform eval results. Either make the fixture's named case skipped on every supported eval host or prevent this scenario from running outside Linux.
Useful? React with 👍 / 👎.
| let sawPlanGuidance = false; | ||
| let sawRunGuidance = false; |
There was a problem hiding this comment.
The guidance flags are accumulated across the entire joined call stream, although calls retain sessionIndex and scenarios such as resumes-after-interruption deliberately create a fresh session with no transcript. If session 0 loads flow-run and session 1 starts a run without loading it, this function reports no skip even though the fresh manager never received the lazy-loaded guide. Track guidance per sessionIndex (or reset the flags when the index changes) so summary.guidanceSkipped measures the documented behavior.
Useful? React with 👍 / 👎.
2 participants
Summary
Implements the post-7.3 improvement plan and prepares 7.3.2 so merge to
maintriggers release.Changes
plan-only-stopsguards (no worker during planning,flow_guidancebefore save) + cassettesguidanceSkippedeval metric (ungated), validation-matrix checklistskipped-case-named-bindingscenarioRelease (7.3.2)
package.json→ 7.3.2CHANGELOG.mddated entry with install snippetv7.3.2frompackage.json, creates tag, publishes npm + GitHub releaseVerification
bun run check— 369 passbun run replay— 9/9 cassettes MATCHbun run release:metadata -- --tag v7.3.2— OK