Skip to content

Add native agent control for BB's visible Browser - #2765

Open
divyesh-puri wants to merge 2 commits into
get-bb:mainfrom
divyesh-puri:bb/lean-agent-browser-native-automation
Open

Add native agent control for BB's visible Browser#2765
divyesh-puri wants to merge 2 commits into
get-bb:mainfrom
divyesh-puri:bb/lean-agent-browser-native-automation

Conversation

@divyesh-puri

@divyesh-puri divyesh-puri commented Aug 31, 2026

Copy link
Copy Markdown

Summary

Adds an experiment-gated, agent-facing Browser automation surface that controls BB's visible in-app Browser through native Electron/Chromium primitives.

This contribution was developed after reviewing #1869 (dd66e0893) and is intended to supersede #1869. It retains that PR's useful bounded-lifecycle, correlation, cancellation, and navigation-epoch concepts, while replacing the synthetic DOM action/custom snapshot approach with native CDP accessibility and input primitives.

Visual demo

A provider-backed agent controlling Wikipedia in BB’s visible in-app Browser

This is the full BB desktop running this PR branch, not a mock. A provider-backed agent used bb browser to open Wikipedia, search for Lean theorem prover, verify the result, scroll with PageDown, follow the homotopy type theory link from a fresh accessibility snapshot, and capture a screenshot. The Browser chrome shows the owning thread and the user-visible Stop control.

What this adds

  • Fresh, visible automation-owned Browser tabs only; existing user tabs cannot be adopted
  • bb browser open|list|navigate|wait|snapshot|click|type|press|select|screenshot|close
  • Typed public API and SDK surface
  • Accessibility-tree snapshots with atomic generations and stale-reference rejection
  • Native pointer, keyboard, scrolling, and text input
  • Bounded exact-value <select> operation without a generic evaluation surface
  • Visible Stop/cancellation, timeout recovery, reconnect/disconnect cleanup, and four-target limit
  • Thread/host-confined screenshot artifacts with bounded retention and private atomic writes
  • Server-backed browserAutomation experiment, disabled by default
  • Built-in bb-browser skill, injected only while the experiment is enabled
  • Privacy-safe aggregate metrics
  • Native Electron and built-CLI/Electron macOS CI gates

Security and ownership model

This stays within BB's existing trusted local/Connect-owner boundary; it does not claim cryptographic desktop attestation.

  • Ownership is derived server-side and correlated across socket, window, tab, request, and command IDs
  • A one-time desktop reservation is required before registering a fresh target
  • Cross-thread and cross-host public access is rejected
  • No generic JavaScript evaluation, caller-supplied scripts, synthetic DOM actions, custom DOM snapshots, plugin Browser API, or arbitrary tab discovery
  • Snapshot/action payloads, request bodies, accessibility trees, screenshots, artifacts, target counts, and recovery windows are bounded
  • Clicks use native hit testing and reject occluded/intercepted targets instead of reporting false success

Validation

  • All affected package typechecks passed
  • Focused domain, server, renderer, desktop, CLI, contract, artifact, route, and lifecycle tests passed
  • Native Electron fixture passed all scenarios, including:
    • controlled and rich-text input
    • exact native select
    • offscreen pointer interaction
    • nested hit targets and overlay rejection
    • Enter default form action
    • SPA/full navigation and stale refs
    • timeout, Stop, and cleanup
  • Built CLI journey passed through the real boundary:
    • built bb binary → HTTP API → renderer WebSocket → preload/IPC → Electron WebContentsView → Chromium CDP → visible page
    • owner isolation, navigation, interaction, screenshot materialization, and no debugger/view leaks
  • Three consecutive runs of each deterministic Electron gate passed during development
  • Provider-backed Claude Fable 5 graduation passed 10/10 sequential runs with snapshot/wait evidence and empty target lists before and after each run:
    • complex fixture workflows
    • scrolling and offscreen interaction
    • expected overlay/stale-ref rejection and recovery
    • Wikipedia search/article navigation
    • Hacker News list/discussion navigation

Key commands:

npm exec -- pnpm exec turbo run test:browser-automation-electron \
  --filter=@bb/desktop --force

npm exec -- pnpm exec turbo run test:browser-cli-electron \
  --filter=@bb/integration-tests --force

Review order

The change is cohesive but cross-layer. Suggested review order:

  1. packages/domain and packages/*-contract
  2. server Browser lifecycle/routes/artifacts
  3. desktop native driver and reservation/IPC
  4. renderer ownership client and Stop UI
  5. CLI/SDK/skill
  6. Electron fixtures and CI

Known gap

Native Electron acceptance is currently validated on macOS. Linux native Electron behavior remains explicitly unclaimed and is a follow-up.

Add experiment-gated Browser automation across the desktop, renderer, server, SDK, and CLI with fresh-tab ownership, native CDP input, AX snapshots, screenshots, cancellation, and bounded lifecycle cleanup.\n\nInclude deterministic Electron boundary coverage and a provider-backed trial harness. Supersedes the approach in get-bb#1869 while retaining its bounded lifecycle and correlation concepts.
amitbtcai added a commit to amitbtcai/axeai-macos that referenced this pull request Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant