test(agent): cover a multi-turn conversation in the Anthropic dialect - #598
Open
HappyDevs1 wants to merge 1 commit into
Open
HappyDevs1 wants to merge 1 commit into
HappyDevs1 wants to merge 1 commit into
Conversation
Plan 012 shipped delivery gating with OpenAI-shaped fixtures only and recorded the rest as a known gap: the gating code never inspects Turn.protocol, so there was "no known reason Anthropic would behave differently - but that is confidence, not a fixture." This is the fixture, and it pins more than "it works". Delivery 1 here produces TWO turns - a tool_use and the text that follows its result - while delivery 0 produces one. A grouping bug that stamped one turn per delivery, or closed a delivery at the first response rather than at settle, would still assemble the same replies and still satisfy every reply assertion in the spec. So the assertions are on the per-delivery turn counts and the turn->delivery mapping, which only correct grouping produces.
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3a of closing out #375. Independent of #596 and #597 (different files).
Why
#375 requires multi-turn record/replay for both the OpenAI-compatible and Anthropic dialects.
plans/012-agent-multiturn-conversations.mdshipped delivery gating with OpenAI-shaped fixtures only and logged the rest explicitly as a known gap:This is the fixture.
What it actually pins
Not just "it works". The trajectory is deliberately asymmetric: delivery 0 produces one turn (the agent asks for confirmation), delivery 1 produces two (a
tool_use, then the text that follows its tool result).That matters because a grouping bug — one turn stamped per delivery, or a delivery closed at first response rather than at settle — would still assemble the same replies and still satisfy every
assert: reply containsin the spec. So the assertions are on:deliveries[].turn_count==[1, 2][0, 1, 1]"protocol": "anthropic"…which only correct grouping can produce.
Falsifiability checked: I asserted
turn_count == 3and confirmed the failure readsleft: Number(2)from the real recorded trace — so the assertion reads recorded data, and delivery 1 genuinely owns both turns.Also asserts the volatile real tool result (
cancelled_at) never reaches disk in this dialect, and that the whole conversation replays with every upstream env handle unset.Test plan
cargo test -p flowproof-cli --test agent_conversation_anthropic_e2e— passes (1.3s).cargo fmt --check,cargo clippy -p flowproof-cli --all-targets -- -D warningsclean.