test(agent): the four conversation red paths, and an example that shows why - #597
Open
HappyDevs1 wants to merge 2 commits into
Open
HappyDevs1 wants to merge 2 commits into
HappyDevs1 wants to merge 2 commits into
Conversation
This was referenced Sep 15, 2026
…ws why #375 asked for red-path tests covering a changed later user turn, a missing turn, an extra turn, and a tool-call divergence after turn one. What existed was adjacent but weaker: a unit test over validate_delivery_windows, which checks the validator rather than a replay, and an early-exit test. None of them edit a flow file against a fixed recording, which is what a real regression looks like. These four do, and each asserts WHERE the run failed, not just that it did. That distinction is the whole value: a test asserting is_err() would pass just as happily if every drift were reported at delivery 1. A changed turn 2 must fail at turn 2, having replayed turn 1 cleanly first. The example is the other gap. cancel-order.flow.yaml is the canonical case the feature exists for - the agent must ask before it cancels, and cancel only once confirmed - and it is the only shipped flow using `conversation:`, so the grammar now has a user. Its guard is delivery-local on purpose: across the whole conversation the agent DOES cancel, so a flow-wide guard would certify nothing. The test pins that shape, because an example that quietly stops demonstrating its point is worse than no example. No cassette is committed: recording one needs a real model, and hand-writing a trace would mint exactly the kind of unearned recording this tool exists to prevent. Same posture as weather.flow.yaml.
HappyDevs1
force-pushed
the
test/conversation-red-paths-and-example
branch
from
September 15, 2026 13:40
d78ed1e to
7023e13
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 2 of 3 closing out #375. Stacks conceptually on #596 but is independent — this touches no schema.
Why
#375 asks for red-path tests covering a changed later user turn, a missing turn, an extra turn, and a tool-call divergence after turn one, plus a runnable example. What existed was adjacent but weaker:
conversation_windows_require_every_delivery_and_every_callis a unit test overvalidate_delivery_windows(it checks the validator, not a replay), plus an early-exit test. Nothing edited a flow file against a fixed recording — which is what an actual regression looks like.The four red paths
Each edits the spec against a fixed cassette and asserts where the run failed:
user:editedturn 2: message 2 (user) content changedthe agent made 1 model calls, the recording has 2turn 3: the system under test made 3 model calls, the recording has 2turn 2: tools offered changedAsserting the location is the point —
is_err()alone would pass just as happily if every drift were reported at delivery 1. The changed-turn case must get past delivery 0 cleanly and fail at turn 2.Falsifiability checked, not assumed: I removed the drift from the changed-turn test and confirmed it then fails (replay succeeds,
expect_errpanics) — so it detects that specific edit rather than failing for an incidental reason.The example
examples/agent-demo/cancel-order.flow.yaml+cancellation_agent.py— a real multi-turn OpenAI-SDK agent (delivery 0 viaFLOWPROOF_PROMPT, later deliveries as JSON lines on stdin, history kept across turns). It's the canonical confirmation-gate case, and the only shipped flow usingconversation:.Its
assert_no_tool_callis delivery-local on purpose: across the whole conversation the agent does callcancel_order, so a flow-wide guard would certify nothing.the_conversation_example_keeps_its_delivery_local_guardpins that shape.No cassette is committed — recording one needs a real model, and hand-writing a trace would mint exactly the unearned recording this tool exists to prevent. Same posture as
weather.flow.yaml.Test plan
cargo test -p flowproof-cli --lib— 126 pass, including 11 conversation tests.cargo test -p flowproof-cli --test examples_resolve— 9 pass.cargo fmt --check,cargo clippy -p flowproof-cli --all-targets -- -D warningsclean.doctor_ai_e2efails on this machine only because it has a real API key stored (api key: configured), while the test asserts the no-key path. Untouched by this PR; expected green in CI.