Skip to content

Stream large workflow history exports - #2732

Open
MacHatter1 wants to merge 1 commit into
get-bb:mainfrom
MacHatter1:bb/evaluate-bb-workflows-extension-thr_b27t6knfps-v2
Open

Stream large workflow history exports#2732
MacHatter1 wants to merge 1 commit into
get-bb:mainfrom
MacHatter1:bb/evaluate-bb-workflows-extension-thr_b27t6knfps-v2

Conversation

@MacHatter1

Copy link
Copy Markdown

Summary

  • Stream large bb workflows history <run-id> exports without truncation.
  • Preserve the buffered 1 MiB safety limit and explicit --cursor/--limit compatibility.
  • Make stream cancellation, plugin reload, and candidate activation rollback resource-safe.

Problem

Valid workflow history output could exceed the buffered plugin CLI response limit. A real dev-app run produced a valid 1,049,375-byte JSONL export and exited successfully on the streaming path, while packaged BB 0.40.0 rejected the same boundary as plugin_cli_output_too_large.

Implementation

  • Adds an experimental streamed stdout result to the plugin CLI boundary.
  • Streams plugin output through the server and CLI with backpressure, cancellation, and 64 KiB UTF-8 chunks.
  • Makes default Workflows history follow all pages automatically; explicit cursor/limit requests remain single-page compatibility behavior.
  • Keeps buffered stdout/stderr bounded at 1 MiB.
  • Protects plugin reload from active streams, failed candidate activation, schedule persistence failures, and pending iterator cancellation.
  • Updates API inventory, authoring guidance, Workflows documentation, and regression coverage.

Validation

  • Workflows tests: 228 passed.
  • Focused CLI tests: 36 passed.
  • Focused server/plugin tests: 42 passed.
  • Plugin API-map checks: 4 passed.
  • Affected Turbo typechecks and builds passed.
  • git diff --check passed.
  • Live dev-app verification confirmed a JSONL export larger than 1 MiB is lossless and parseable.

No host-daemon protocol change is included.

@MacHatter1
MacHatter1 marked this pull request as ready for review August 30, 2026 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant