Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
# Changelog

## Unreleased

- Select attached or locally launched browser tabs explicitly from a title/URL
picker. Clear stale frames and pending input when the selected tab closes;
require another selection before forwarding input.
- Preserve whitespace, Unicode, tabs, and newlines in the text prompt with
bracketed paste. Confirm insertion separately without sending Enter to the page.

## 0.8.0

- Add declarative saved browser QA scenarios with isolated desktop/mobile
Expand Down
28 changes: 24 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,10 +151,10 @@ Use these controls to drive the shared session directly:
| `u` | Open the address prompt; `https://` is assumed when omitted |
| `a` | Attach to a CDP endpoint (`http://host:port` or `ws://…`) |
| `l` | Launch a local Chromium the pane owns and attach to it |
| `t` | Attach mode: cycle the pane between the browser's page targets (tabs) |
| `t` | Attach/launch mode: open the tab picker; arrows or `j`/`k` highlight, Enter selects, `t` refreshes, Esc cancels |
| `o` | Toggle observe-only: pane input is dropped instead of forwarded |
| Click the screenshot | Send real Chrome mouse move/down/up events at that page coordinate |
| `i` | Type into the currently focused page element |
| `i` | Enter or paste text for the focused page element; Enter inserts it, Esc cancels |
| `b` / `f` | Navigate backward / forward |
| `r` | Reload |
| `j` / `k` | Scroll down / up |
Expand All @@ -167,6 +167,13 @@ Clicks are mapped through the rendered-frame geometry to page pixels, so they
work with overlays, canvas content, and shadow DOM. Live sessions usually
repaint immediately; polling fallback can take up to the configured interval.

The text prompt preserves leading/trailing spaces and Unicode. In a terminal
that supports bracketed paste, pasted tabs and newlines remain text in the
prompt until you press Enter to insert it. Newlines appear as `\n` in the
preview. Inserting text does not send an Enter key to the page or submit a
form. Paste into the `i` prompt; pasted text outside a prompt is ignored.
Each paste is limited to 1 MiB. URL and endpoint prompts still trim whitespace.

## Rendering and streaming

### Automatic mode selection
Expand Down Expand Up @@ -251,8 +258,8 @@ printf 'http://127.0.0.1:9222\n' > "$(herdr plugin config-dir structupath.browse
Press `a` in the pane to attach at runtime. `u` still means "navigate" — the
keys are separate because `localhost:9222` is a valid destination as well as a
valid endpoint. While attached, the pane header shows the endpoint's
`host:port` instead of a session name, `t` cycles between the browser's page
targets when your automation has more than one tab open, and `o` toggles
`host:port` instead of a session name, `t` opens a tab picker with titles and
URLs, and `o` toggles
**observe-only**: every pane click, wheel event, keystroke, and navigation —
including Cmd/Ctrl+click link handoffs in attach mode — is dropped at the
pane instead of forwarded, so watching a live run cannot blur the field your
Expand All @@ -265,6 +272,19 @@ navigates the session daemon directly, outside the pane. Set
watch-the-agent workspaces — the pane starts observe-only *and* the open
action itself refuses link navigation, closing that gap in both modes.

On first connection, a single tab is selected automatically. With multiple
tabs, input waits for an explicit choice in the picker. Selection uses the
tab's stable identity, so duplicate titles and changing tab order do not
redirect it. The picker changes only what this pane observes; it does not
activate the tab in another client's UI and remains available in observe-only
mode. Shared agent-browser mode continues following that session's active tab.

If the selected tab closes or detaches, the pane clears its cached image and
pending text, drops queued input, and keeps the browser connection open.
Press `t` to select another tab, even if only one remains. It never switches
automatically to a surviving tab. Reconnection restores the same tab only
when the browser identity and tab identity both still match.

Launcher recipes: Playwright `chromium.launch({args:['--remote-debugging-port=9222']})`,
Puppeteer the same `args`, Browser Use its `chrome_remote_debugging_port` option.
A default launch often uses a pipe transport with no TCP port — the port has to
Expand Down
122 changes: 78 additions & 44 deletions bin/cdp.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,13 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
let lastMeta = null; // latest frame metadata (deviceWidth/Height for input scaling)
let handler = null; // onMessage subscriber (the Renderer)
let attachTimeMs = 0;
let connectedOnce = false;
let selection = 0;
const targetError = () => Object.assign(new Error("select a tab with t before sending input"), { code: "TARGET_GONE" });
const requireTarget = (expected = pageSessionId) => {
if (!expected || expected !== pageSessionId || !pinnedTargetId || session?.dead) throw targetError();
return expected;
};
const emit = (m) => {
try {
handler?.(m);
Expand All @@ -272,13 +279,21 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
const { targetInfos } = await session.send("Target.getTargets");
return targetInfos.filter((t) => t.type === "page");
};
const clearTarget = (type) => {
selection++;
gen++;
pinnedTargetId = null;
pageSessionId = null;
lastMeta = null;
emit({ type });
};

const startScreencast = async () => {
const startScreencast = async (sid = requireTarget()) => {
const g = ++gen;
await session.send(
"Page.startScreencast",
{ format: "jpeg", quality, maxWidth: maxDim, maxHeight: maxDim, everyNthFrame: 1 },
pageSessionId,
requireTarget(sid),
);
return g;
};
Expand All @@ -294,12 +309,18 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
await session.send("Runtime.enable", {}, sessionId);
};

const pinTarget = async (targetId) => {
const pinTarget = async (target) => {
const { targetId } = target;
const version = selection;
pinnedTargetId = targetId;
const { sessionId } = await session.send("Target.attachToTarget", {
targetId,
flatten: true,
});
pinnedTargetId = targetId;
if (version !== selection || pinnedTargetId !== targetId) {
await session.send("Target.detachFromTarget", { sessionId }).catch(() => {});
throw targetError();
}
pageSessionId = sessionId;
await session.send("Page.enable", {}, sessionId);
await enableFeed(sessionId);
Expand All @@ -316,6 +337,8 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
} catch {
/* older engines: page-level feed only */
}
requireTarget(sessionId);
emit({ type: "target_selected", targetId, url: target.url, title: target.title });
try {
await startScreencast();
} catch (err) {
Expand All @@ -330,7 +353,7 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
};

const onCdpEvent = (m) => {
if (m.method === "Page.screencastFrame" && m.sessionId === pageSessionId) {
if (m.method === "Page.screencastFrame" && pageSessionId && m.sessionId === pageSessionId) {
lastMeta = m.params.metadata ?? null;
emit({
type: "frame",
Expand All @@ -352,16 +375,11 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
}
if (m.method === "Target.targetDestroyed") {
if (m.params.targetId !== pinnedTargetId) return;
pinnedTargetId = null;
pageSessionId = null;
// Re-pin only on destruction of OUR target — never follow creation.
pageTargets()
.then(async (pages) => {
if (!pages.length) return emit({ type: "target_gone" });
await pinTarget(pages[0].targetId);
emit({ type: "url", url: pages[0].url, title: pages[0].title });
})
.catch(() => emit({ type: "target_gone" }));
clearTarget("target_gone");
return;
}
if (m.method === "Target.detachedFromTarget" && pageSessionId && m.params.sessionId === pageSessionId) {
clearTarget("target_gone");
return;
}
if (m.method === "Inspector.targetCrashed" && m.sessionId === pageSessionId) {
Expand Down Expand Up @@ -428,24 +446,33 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
// Identity of what we're attached to; the Renderer compares guid across
// reattaches so a reused port can't silently swap browsers underneath.
async connect() {
const previousTarget = pinnedTargetId;
const previousGuid = endpoint?.guid;
if (session) {
session.close();
clearTarget("target_changing");
}
endpoint = await discoverEndpoint(endpointInput, opts);
session = makeCdpSession(endpoint.wsUrl, opts);
await session.opened;
session.onEvent(onCdpEvent);
session.onClose(() => emit({ type: "endpoint_gone" }));
await session.send("Target.setDiscoverTargets", { discover: true });
const pages = await pageTargets();
if (!pages.length) throw new Error("endpoint has no page targets");
attachTimeMs = Date.now();
await pinTarget(pages[0].targetId);
const target = !connectedOnce && pages.length === 1 ? pages[0]
: previousGuid && previousGuid === endpoint.guid ? pages.find((p) => p.targetId === previousTarget) : null;
connectedOnce = true;
if (target) await pinTarget(target);
return {
host: endpoint.host,
port: endpoint.port,
browser: endpoint.browser,
guid: endpoint.guid,
rediscoverable: endpoint.rediscoverable,
url: pages[0].url,
title: pages[0].title,
targetId: pinnedTargetId,
url: target?.url ?? "",
title: target?.title ?? "",
};
},
onMessage(fn) {
Expand All @@ -464,57 +491,64 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
}
},
async restartScreencast() {
const sid = requireTarget();
try {
await session.send("Page.stopScreencast", {}, pageSessionId);
await session.send("Page.stopScreencast", {}, sid);
} catch {
/* already stopped */
}
await startScreencast();
await startScreencast(sid);
},
async cycleTarget() {
async listTargets() {
return (await pageTargets()).map(({ targetId, url, title }) => ({ targetId, url, title, selected: targetId === pinnedTargetId }));
},
async selectTarget(targetId) {
const pages = await pageTargets();
if (pages.length < 2) return false;
const i = pages.findIndex((t) => t.targetId === pinnedTargetId);
const next = pages[(i + 1) % pages.length];
try {
await session.send("Page.stopScreencast", {}, pageSessionId);
} catch {
/* old session may be gone */
const next = pages.find((p) => p.targetId === targetId);
if (!next) throw Object.assign(new Error("that tab closed — press t to refresh the list"), { code: "TARGET_GONE" });
if (pinnedTargetId === targetId && pageSessionId) return;
const oldSession = pageSessionId;
clearTarget("target_changing");
if (oldSession) {
await session.send("Page.stopScreencast", {}, oldSession).catch(() => {});
await session.send("Target.detachFromTarget", { sessionId: oldSession }).catch(() => {});
}
await pinTarget(next.targetId);
emit({ type: "url", url: next.url, title: next.title });
return true;
try { await pinTarget(next); }
catch (err) { clearTarget("target_gone"); throw err; }
},
async open(u) {
await session.send("Page.navigate", { url: u }, pageSessionId);
await session.send("Page.navigate", { url: u }, requireTarget());
},
async back() {
const h = await session.send("Page.getNavigationHistory", {}, pageSessionId);
const sid = requireTarget();
const h = await session.send("Page.getNavigationHistory", {}, sid);
if (h.currentIndex <= 0) return;
await session.send(
"Page.navigateToHistoryEntry",
{ entryId: h.entries[h.currentIndex - 1].id },
pageSessionId,
requireTarget(sid),
);
},
async forward() {
const h = await session.send("Page.getNavigationHistory", {}, pageSessionId);
const sid = requireTarget();
const h = await session.send("Page.getNavigationHistory", {}, sid);
if (h.currentIndex >= h.entries.length - 1) return;
await session.send(
"Page.navigateToHistoryEntry",
{ entryId: h.entries[h.currentIndex + 1].id },
pageSessionId,
requireTarget(sid),
);
},
async reload() {
await session.send("Page.reload", {}, pageSessionId);
await session.send("Page.reload", {}, requireTarget());
},
// x/y arrive in page CSS pixels — the Renderer scales pane cells ->
// frame pixels -> CSS via the per-frame metadata before calling.
async click(x, y) {
const sid = requireTarget();
const base = { x, y, button: "left", clickCount: 1 };
await session.send("Input.dispatchMouseEvent", { type: "mousePressed", ...base }, pageSessionId);
await session.send("Input.dispatchMouseEvent", { type: "mouseReleased", ...base }, pageSessionId);
await session.send("Input.dispatchMouseEvent", { type: "mousePressed", ...base }, sid);
await session.send("Input.dispatchMouseEvent", { type: "mouseReleased", ...base }, requireTarget(sid));
},
async scroll(dir, px) {
const m = lastMeta;
Expand All @@ -523,24 +557,24 @@ export function makeCdpBrowser(endpointInput, opts = {}) {
await session.send(
"Input.dispatchMouseEvent",
{ type: "mouseWheel", x: cx, y: cy, deltaX: 0, deltaY: dir === "down" ? px : -px },
pageSessionId,
requireTarget(),
);
},
async type(text) {
await session.send("Input.insertText", { text }, pageSessionId);
await session.send("Input.insertText", { text }, requireTarget());
},
async screenshot(file) {
const { data } = await session.send(
"Page.captureScreenshot",
{ format: "png" },
pageSessionId,
requireTarget(),
15_000,
);
const fs = await import("node:fs");
fs.writeFileSync(file, Buffer.from(data, "base64"));
},
async sessionExists() {
if (!session || session.dead || !pinnedTargetId) return false;
if (!session || session.dead) return false;
return session.ping();
},
close() {
Expand Down
Loading
Loading