feat: quiet window recovery, scroll-into-view and non-activating software cursor

[wicgee]

Summary

Three fixes for driving a GUI with computer-use, aligned with Codex behavior:

• Focus stealing — operating a target app used to pull the user's foreground focus and interrupt their work.
• Missed clicks — targets outside the viewport were not scrolled into view, so clicks landed on nothing.
• Cursor feedback — the software cursor jumped on every call, jittered while idle, and disappeared between actions.

Stacking note: this branch is based on commit 244c7ed (feat(select_text)), an ancestor that is not yet in main; select_text is carried by PR #68 (wicgee, feat/select-text → main, OPEN/MERGEABLE, +564/−21, not yet merged into main), which this branch is stacked on. The PR therefore contains 12 commits / 35 files; the 11 commits unique to this branch are 7dbcd03 (exec-plan docs), 497afa9 (quiet window recovery + scroll-into-view + cursor, 12 files), 2b63fe5 (choreographer ordering), a021c00 (cursor travel coalescing), 7b9c477 (idle stillness), bacf376 + 21daf46 (skill docs), c434426 (cursor stays on screen for the whole turn, zero-activation default), 50df06e (removes the target-highlight feature, 17 files, −1815/+241), 29934cf (cross-screen re-anchor, 12 files, +1220/−33) and 6bc1e9f (records the measured risk paths in the platform-neutral skill text, 1 file, +2/−1). HEAD is 6bc1e9f.

Changes

P2 — Quiet window recovery (new default)

• Snapshot restore now defaults to .readOnly instead of .allowActivation.
• unhide / activate / open -b / AXRaise / Main / Focused run only on explicit opt-in:
  • tool parameter allow_window_recovery=true, or
  • env OPEN_COMPUTER_USE_ALLOW_WINDOW_RECOVERY=1.
• When no window is available it returns Apple event error -10005: cgWindowNotFound, with a hint to move the window to the current Space or un-minimize it.
• sky_click re-capture stays always read-only.

P1 — Automatic scroll-into-view

• New ensureElementVisible: AXScrollToVisible → up to 6 ancestor levels → nearest AXScrollArea paging (≤ 8 scrolls, frame re-read every round).
• Invoked before click(element_index), set_value and select_text.
• After scrolling, the element handle is re-resolved by the same index and the frame is read back; if the frame cannot be read back or is still out of bounds, the action fails closed.
• Env OPEN_COMPUTER_USE_SCROLL_TARGET_INTO_VIEW (default on; =0 / false / no / off disables).

P3 — auto tries sky_click first (gradual rollout)

• With OPEN_COMPUTER_USE_AUTO_SKY_CLICK=1, auto tries sky_click once before postToPid, and only when: left button, 1–2 clicks, valid windowID/bounds, and the SPI is available.
• On failure it falls back to the existing path; it never escalates to global.

P4 — Target element highlight: removed

• TargetHighlightOverlay.swift, TargetHighlightLifetime.swift and TargetHighlightStyle.swift are physically deleted, together with the OPEN_COMPUTER_USE_TARGET_HIGHLIGHT_STYLE / OPEN_COMPUTER_USE_DEBUG_HIGHLIGHT switches, the [open-computer-use] highlight ... diagnostics, and the highlight step in VisualInteractionChoreographer (its record / snapshot parameters are gone too).
• debug-highlight → debug-cursor (cursor + click pulse only; single screen, --seconds cap, deterministic cleanup retained).
• Rationale: the official Codex Computer Use runtime has no target-highlight concept — grep -in highlight over the reverse-engineering reference and the runtime assets is empty, and its only visual surface is the software cursor (126×126 canvas, fog halo, Bezier travel, click pulse). Keeping a codex|plain style switch for a concept the official client does not have is dead code, so the feature was deleted outright rather than defaulted off. docs/ARCHITECTURE.md records the removal so it is not reintroduced.

P5 — Non-activating software cursor (the only visual surface)

• Travel is coalesced: repeating an action on the element the cursor already covers only keeps the cursor in place; actions inside a 400 ms window merge into a single travel animation (OPEN_COMPUTER_USE_VISUAL_CURSOR_COALESCE_MS, default 400, 0 restores one animation per action).
• Idle is absolutely static: after the last action a bounded 1 s micro-wobble, then the overlay freezes outright — no frame, level or stacking updates at all (OPEN_COMPUTER_USE_VISUAL_CURSOR_IDLE_SWAY_MS, default 1000; 0 freezes immediately; OPEN_COMPUTER_USE_VISUAL_CURSOR_DEBUG_STATS=1 dumps the write/tick counters when the cursor hides).
• The cursor stays on screen for the whole turn: it never hides on an inactivity timer, keeps a .floating panel level so another app taking focus cannot bury it, and only turn-ended / an explicit reset / OPEN_COMPUTER_USE_VISUAL_CURSOR=0 removes it.
• Zero activation by default: the cursor panel is non-activating (canBecomeKey / canBecomeMain = false, ignoresMouseEvents = true), so a default click never raises, mains or focuses a window.
• Single display only: exactly one cursor panel exists; NSScreen.screens is used only for coordinate mapping, availability checks and screen(containing:) clamping, never to build per-screen panels.
• Ordering fix (2b63fe5) — the official log order moves the software cursor and signals completion first, and only then performs the action (docs/references/codex-computer-use-reverse-engineering/software-cursor-overlay.md:190-196,243), so the sequence is now move → settle (120 ms) → action, funneled through VisualInteractionChoreographer; perform_secondary_action / scroll gained the move stage too. A click pulse is played only when the coalescer reports that a travel animation actually happened (approach.playsClickPulse).
• Self-check without touching another app: open-computer-use debug-cursor [--seconds N] [--display N] paints the cursor and its click pulse on one screen (1-based --display, main by default; capped at 60 s; defer cleanup) and prints the screen-state rect for screencapture -R.

P6 — Cross-screen / window-move re-anchor

• The snapshot no longer freezes window geometry at get_app_state time: before every action ComputerUseService.currentSnapshot re-reads the target window with CGWindowListCopyWindowInfo(.optionIncludingWindow). Same window and same size (move only) → only windowBounds is patched, because element frames are window-relative; a size change or a different window → the snapshot is fully refreshed; if the window cannot be resolved → the previous snapshot is kept.
• VisualCursorTarget now carries the screen-state point together with the window frame of that moment, and the overlay keeps a CursorRestingAnchor (window-local offset + screen-state frame), so a move to another display re-derives the tip from the live frame and re-places the cursor immediately.
• AXWindowMotionObserver (CursorWindowMotionWatch.swift) is a read-only accessibility watch on AXWindowMoved / AXWindowResized that re-places the cursor as soon as a move or resize ends. It is on by default; OPEN_COMPUTER_USE_WINDOW_MOVE_WATCH=0 disables it.
• The pre-action geometry refresh is the fallback when no event arrives: the cursor's screen is compared with the target window's screen and any mismatch recomputes and repaints the anchor. Neither path hides the cursor, and neither adds an activation path, so focus behaviour is unchanged.
• Measured on a real session: with the target window moved from the external display to the built-in one and back, the overlay X followed -1453 → 256/259 → -1453 with no further click in between. Before the fix the cursor stayed on the display the window had just left until the next action refreshed the snapshot.
• Files: new SnapshotWindowGeometry.swift and CursorWindowMotionWatch.swift, changes in AccessibilitySnapshot.swift / ComputerUseService.swift / SoftwareCursorOverlay.swift / VisualInteractionChoreographer.swift, 9 new cases in CursorCrossScreenTests.swift, plus docs/ARCHITECTURE.md and the history note docs/histories/2026-09/20260912-1510-cursor-cross-screen-reanchor.md.

P7 — Measured behaviour of the three opt-in risk paths (defaults unchanged)

• allow_window_recovery: true unminimizes or unhides the target window at its previous frame and the click then succeeds, but it also activates that app: frontmost changes to it, which interrupts the user's foreground. It stays opt-in / off by default for that reason.
• An explicit click_method: "sky_click" is safe for in-page content buttons (no error, unchanged Chromium renderer PIDs, unchanged frontmost), but must not be used on app navigation or sidebar links: it crashed the Chromium renderer with error code 5 (recovered with a toolbar reload). Navigation keeps the default AXPress click.
• click_method: "global" (requires OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1) really moves the system pointer — measured from (1392.56, 991.32) on one display to (-1050, 453.5) on another, where it stayed — and changes frontmost. Keep it disabled by default and validate only in an isolated instance (separate agent socket namespace plus a separate agent, cleaned up immediately), never in a shared or production session.

Verification

• swift build — passes
• swift test — 231 tests, 1 skipped (live sky_click, opt-in), 0 failures, measured on 6bc1e9f when this update was pushed: Executed 231 tests, with 1 test skipped and 0 failures. That is +9 over the 222 reported after the highlight removal, accounted for by the CursorCrossScreenTests cases added in 29934cf.
• ./scripts/check-docs.sh — passes
• Render self-check — debug-cursor --seconds 2 --display 1 prints the target/capture/screen rects and exits deterministically after 2 s; the 2 s capture shows the fog cursor only, with no ring anywhere around the target rect.

Risks / not covered

• No full end-to-end turn against an arbitrary third-party app yet; the real-GUI evidence so far is limited to the measured paths recorded in P6/P7.
• P3 was not verified against a real Chromium.
• The AXScrollArea paging branch of ensureElementVisible only has semantic-level unit tests.
• The existing AX activation-only fallback in click (AXWindow role only) is not covered by the opt-in.
• Lark/Electron pull-forward scenarios were not re-tested; the P7 measurement now shows the concrete cost of that path (frontmost changes), so P2 keeps the activating path behind an opt-in.

Restoring previous behavior

• OPEN_COMPUTER_USE_ALLOW_WINDOW_RECOVERY=1 — re-enables unhide/activate/raise.
• OPEN_COMPUTER_USE_SCROLL_TARGET_INTO_VIEW=0 — disables automatic scroll-into-view.
• OPEN_COMPUTER_USE_WINDOW_MOVE_WATCH=0 — disables the read-only window-move watch; the pre-action screen-consistency re-anchor stays active.
• Unset OPEN_COMPUTER_USE_AUTO_SKY_CLICK — disables the sky_click attempt.
• Unset OPEN_COMPUTER_USE_VISUAL_CURSOR — hides the software cursor (there is no element highlight left to configure; that feature was removed in P4).

Upstream overlap: PR #65 (@hyprcat, open) independently makes background operation complete: key_method=sky_key, covered/other-Space window resolution via WindowCapture.resolve, occlusion keep-alive and window_placement=agent_display. It shares this PR's goal (never activate/raise the user's foreground app) and does not conflict semantically with P1/P2/P5; P3 (auto tries sky_click once, OPEN_COMPUTER_USE_AUTO_SKY_CLICK=1, default off, always falls back to postToPid) is orthogonal to #65's sky_key and keeps #65's "explicit opt-in" convention. Merge conflicts with #65 are expected in AccessibilitySnapshot.swift, ComputerUseService.swift, ComputerUseToolDispatcher.swift, ToolDefinitions.swift and docs/ARCHITECTURE.md (both sides add optional tool parameters); resolution is a union, keeping both window_placement/key_method and allow_window_recovery, with sky_click/sky_key result refreshes staying read-only unless allow_window_recovery is explicitly passed. We are happy to rebase on #65 once it lands.

[wicgee]

Final state of this branch (HEAD 50df06e)

• Quiet by default, no focus stealing: snapshot restore defaults to .readOnly; unhide / activate / open -b / AXRaise run only with an explicit opt-in - the per-call allow_window_recovery: true parameter, or OPEN_COMPUTER_USE_ALLOW_WINDOW_RECOVERY=1 for the server process (default off).
• Automatic scroll-into-view: AXScrollToVisible -> ancestor levels -> AXScrollArea paging before click / set_value / select_text; the element handle is re-resolved after scrolling and the action fails closed if the frame cannot be read back. OPEN_COMPUTER_USE_SCROLL_TARGET_INTO_VIEW defaults on.
• Cursor travel coalescing + idle stillness: actions inside the OPEN_COMPUTER_USE_VISUAL_CURSOR_COALESCE_MS window (default 400) merge into one travel animation; after the last action the overlay keeps a bounded 1 s micro-wobble and then freezes outright (no frame / level / stacking writes). OPEN_COMPUTER_USE_VISUAL_CURSOR_IDLE_SWAY_MS default 1000.
• Zero activation by default: the cursor panel is non-activating (canBecomeKey / canBecomeMain = false) and click-through (ignoresMouseEvents = true), so a default click never raises, mains or focuses a window.
• Cursor stays visible for the whole turn: .floating baseline level and no inactivity timer; only turn-ended / an explicit reset / OPEN_COMPUTER_USE_VISUAL_CURSOR=0 removes it.
• The only visual surface is the software cursor + click pulse - no element highlight: TargetHighlightOverlay / TargetHighlightLifetime / TargetHighlightStyle, both env switches, the highlight ... diagnostics and the choreographer step are physically deleted; no TargetHighlight* file remains in the tree and no highlight switch is left to configure.
• Tests on this HEAD: swift test -> 222 tests, 1 skipped (live sky_click, opt-in), 0 failures.
• Self-check entry point: open-computer-use debug-cursor [--seconds N] [--display N] - cursor + click pulse only, single screen (1-based --display, main by default), capped at 60 s with deterministic cleanup, prints the screen-state rect for screencapture -R.
• Switch defaults: allow_window_recovery / OPEN_COMPUTER_USE_ALLOW_WINDOW_RECOVERY off; OPEN_COMPUTER_USE_SCROLL_TARGET_INTO_VIEW on; OPEN_COMPUTER_USE_AUTO_SKY_CLICK off; OPEN_COMPUTER_USE_VISUAL_CURSOR on.

[wicgee]

Follow-up pushed to this branch: 50df06e..6bc1e9f, remote tip 6bc1e9f.

Commits

1. 29934cf — fix(overlay): window-move / cross-screen re-anchor. The target window frame is re-read from the window server before every action (CGWindowListCopyWindowInfo(.optionIncludingWindow); move-only → patch windowBounds, resize or window change → full snapshot refresh, unresolvable → keep the previous snapshot); the overlay keeps a CursorRestingAnchor and re-derives the tip from the live frame; a read-only AXWindowMotionObserver (AXWindowMoved / AXWindowResized) follows moves event-style. No new activation path.
2. 6bc1e9f — docs(skill): records the measured conclusions below in the platform-neutral skill text (+2/−1, no source change).

Switch defaults

• OPEN_COMPUTER_USE_WINDOW_MOVE_WATCH — window-move watch on by default; =0 disables it. The pre-action "cursor screen == target window screen" re-anchor stays active either way.
• allow_window_recovery / OPEN_COMPUTER_USE_ALLOW_WINDOW_RECOVERY, OPEN_COMPUTER_USE_AUTO_SKY_CLICK, OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS — unchanged, off by default.

Measured on a real session

• Window moved to the other display and back: overlay X -1453 → 256/259 → -1453, with no further click in between.
• allow_window_recovery=true: recovers the window at its previous frame and the click succeeds, but it activates the app (frontmost changes) → recommendation: keep it opt-in.
• Explicit click_method="sky_click": safe on in-page content buttons (no error, unchanged Chromium renderer PIDs, unchanged frontmost); crashes the Chromium renderer (error code 5) on navigation / sidebar links → keep navigation on the default AXPress path.
• click_method="global": really moves the system pointer (measured (1392.56, 991.32) → (-1050, 453.5), where it stayed) and changes frontmost → recommendation: keep it disabled and validate only in an isolated instance.

swift test on 6bc1e9f: 231 tests, 1 skipped (opt-in live sky_click), 0 failures.

[wicgee]

Merge-order note: this branch is stacked on #68 (feat/select-text).

feat/quiet-mode-scroll-into-view contains 244c7ed as an ancestor, so the commit list and diff shown here include that PR's select_text change as well. Merging #68 first keeps each PR reviewable on its own; merging this one alone would also bring select_text into main.

Both are independently mergeable against main today — this is only about keeping the two changesets separate.