test(agent): cover a multi-turn conversation that streams - #599
Open
HappyDevs1 wants to merge 2 commits into
Open
HappyDevs1 wants to merge 2 commits into
HappyDevs1 wants to merge 2 commits into
Conversation
Streaming is the combination most worth a real fixture rather than an argument from construction. A delivery settles when the proxy has no in-flight request and stays quiet, and an SSE response is in flight for longer than a buffered one - releasing the next delivery while the previous stream was still draining would be a real bug, and nothing proved it did not happen. Asserting on assembled text could not catch it: a conversation whose streams were collapsed into buffered bodies, or cut short by an early release, would still assemble the same replies and satisfy every `assert: reply contains` in the spec. So this asserts the frame boundaries per model call - three complete, separately terminated streams - at record AND at replay, from a cassette that holds no stream at all. With the Anthropic fixture alongside it, the coverage plan 012 deferred is built, so its "known gap" note is now a record of what closed it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3b — the last of the series closing out #375. Pairs with #598 (Anthropic dialect); independent of #596 and #597.
Why streaming is the one that needed a real fixture
A delivery settles when the proxy has no in-flight request and stays quiet for 150ms. An SSE response is in flight for longer than a buffered one, so "release the next delivery while the previous stream is still draining" is a genuinely plausible bug — and nothing proved it wasn't happening. This is the combination where the argument-from-construction in
plans/012was weakest.Why it asserts frames, not text
A conversation whose streams were collapsed into buffered bodies, or cut short by an early release, would still assemble the same replies and still satisfy every
assert: reply containsin the spec. Asserting on text would be a test that cannot fail for its own bug.So it asserts the frame boundaries per model call — three complete, separately terminated streams — at record and at replay, from a cassette holding no stream at all:
Every list terminating in
DONEis the evidence that no delivery was released while the previous one's stream was still draining. The agent tolerates a buffered answer rather than erroring on it, so the frame log is the only place a collapsed stream shows up.Falsifiability checked: I changed one expected frame to
content-type:application/jsonand confirmed the failure prints the real recorded frames — three genuinetext/event-streamlists — so the assertion reads real data.Also asserts transport stays out of the recording (no
event-stream, nochat.completion.chunk, no"stream"key) and that delivery grouping survives streaming (turn_count[1, 2]).plans/012
Its "Known gap, not blocking" note said the dialect/streaming fixtures were deferred and that the safety argument was "confidence, not a fixture". With #598 and this PR both fixtures exist and the assumption held, so the note now records what closed it.
Test plan
cargo test -p flowproof-cli --test agent_conversation_streaming_e2e— passes (1.2s).cargo fmt --check,cargo clippy -p flowproof-cli --all-targets -- -D warningsclean.