Skip to content

test(agent): cover a multi-turn conversation that streams - #599

Open
HappyDevs1 wants to merge 2 commits into
mainfrom
test/conversation-streaming
Open

HappyDevs1 wants to merge 2 commits into
mainfrom
test/conversation-streaming

Conversation

@HappyDevs1

Copy link
Copy Markdown
Collaborator

Part 3b — the last of the series closing out #375. Pairs with #598 (Anthropic dialect); independent of #596 and #597.

Why streaming is the one that needed a real fixture

A delivery settles when the proxy has no in-flight request and stays quiet for 150ms. An SSE response is in flight for longer than a buffered one, so "release the next delivery while the previous stream is still draining" is a genuinely plausible bug — and nothing proved it wasn't happening. This is the combination where the argument-from-construction in plans/012 was weakest.

Why it asserts frames, not text

A conversation whose streams were collapsed into buffered bodies, or cut short by an early release, would still assemble the same replies and still satisfy every assert: reply contains in the spec. Asserting on text would be a test that cannot fail for its own bug.

So it asserts the frame boundaries per model call — three complete, separately terminated streams — at record and at replay, from a cassette holding no stream at all:

["content-type:text/event-stream", "role:assistant", "content:Are you sure? This cannot be undone.", "finish:stop", "DONE"]
["content-type:text/event-stream", "role:assistant", "tool:cancel_order:{\"order\":\"A-4471\"}", "finish:tool_calls", "DONE"]
["content-type:text/event-stream", "role:assistant", "content:Order A-4471 is cancelled.", "finish:stop", "DONE"]

Every list terminating in DONE is the evidence that no delivery was released while the previous one's stream was still draining. The agent tolerates a buffered answer rather than erroring on it, so the frame log is the only place a collapsed stream shows up.

Falsifiability checked: I changed one expected frame to content-type:application/json and confirmed the failure prints the real recorded frames — three genuine text/event-stream lists — so the assertion reads real data.

Also asserts transport stays out of the recording (no event-stream, no chat.completion.chunk, no "stream" key) and that delivery grouping survives streaming (turn_count [1, 2]).

plans/012

Its "Known gap, not blocking" note said the dialect/streaming fixtures were deferred and that the safety argument was "confidence, not a fixture". With #598 and this PR both fixtures exist and the assumption held, so the note now records what closed it.

Test plan

  • cargo test -p flowproof-cli --test agent_conversation_streaming_e2e — passes (1.2s).
  • cargo fmt --check, cargo clippy -p flowproof-cli --all-targets -- -D warnings clean.

Note: an earlier iteration of the fake upstream matched only one JSON spacing of "role": "tool", which hung the run — the in-tree fakes check both forms for exactly this reason. Fixed before commit.

Streaming is the combination most worth a real fixture rather than an
argument from construction. A delivery settles when the proxy has no
in-flight request and stays quiet, and an SSE response is in flight for
longer than a buffered one - releasing the next delivery while the
previous stream was still draining would be a real bug, and nothing
proved it did not happen.

Asserting on assembled text could not catch it: a conversation whose
streams were collapsed into buffered bodies, or cut short by an early
release, would still assemble the same replies and satisfy every
`assert: reply contains` in the spec. So this asserts the frame
boundaries per model call - three complete, separately terminated
streams - at record AND at replay, from a cassette that holds no stream
at all.

With the Anthropic fixture alongside it, the coverage plan 012 deferred
is built, so its "known gap" note is now a record of what closed it.
Copilot AI lite review requested due to automatic review settings September 15, 2026 13:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants