fix(overlay): keep the software cursor on the window when it moves mid-action

[wicgee]

Summary

Real-machine report: after dragging a window from the main display to the built-in one, the software cursor stayed on the main display and kept being animated there, and only returned to the built-in display once the whole animation had finished.

29934cf (in #69) already re-places the cursor onto the live window frame, from both the accessibility move notification and the pre-action fallback. The symptom still reproduced, so something else was overriding that re-placement.

Root cause: SoftwareCursorOverlay.animateMove runs a synchronous main-thread loop that writes the cursor on every frame from a path sampled up front. When the window is dragged during a travel, the notification-driven re-place is overwritten by the loop's next frame, which keeps writing samples towards the screen the window just left — which is exactly why the cursor looked like it was being driven across the old display.

Fix

• SnapshotWindowGeometry gains a pure decision, cursorTravelMustAbort(startFrame:liveFrame:): abort as soon as the live frame changed. An unreadable frame (minimized, hidden, another Space) is not a change, matching snapshotWindowReanchorAction, so the accessibility action paths keep working.
• animateMove re-reads the target window frame on every frame; on a change it lands the cursor on the live frame and returns, instead of finishing the stale path.

Verification

• Two new cases: the decision's boundaries (including both unreadable-frame directions), and an end-to-end case that drives the real travel loop with the frame changing between the starting frame and the first drawn frame.
• Negative proof: with the fix disabled the new case fails — the cursor ends at (140, 370), still on the stale path; with the fix it passes at (840, 370), derived from the live frame.
• swift test: 264 tests / 1 skipped / 0 failures. make check-docs passes.

Stacking

This branch is based on #69 (feat/quiet-mode-scroll-into-view) because the cross-screen cursor code it fixes only exists there. Review the last commit only (fix(overlay): …); the earlier commits belong to #69, which should merge first.

Not covered

• The real drag (pointer held down) was not reproduced end to end: scratch windows on this machine land in a Space the OCU agent does not drive (cgWindowNotFound), so the loop behaviour is pinned by an injected frame sequence instead.
• The other half of the race remains: if the window server has not yet published the new frame at the moment an action starts, that action can still aim at the old point. This change covers "changed mid-travel" only.

[wicgee]

Real-machine verification: this does not fix the reported bug

I installed the build from this branch on the machine that reported the issue (two displays: 2048x1152 main + 1512x982 built-in) and ran a controlled A/B, driving a scratch Finder window with click.

Baseline (abort disabled): the window was moved to the main display mid-action and the cursor ended on the built-in display at the stale target (x=-977). The reported symptom reproduced.

This branch: same result — the cursor still ended at x=-977 on the built-in display.

Why. A temporary instrumentation of the travel loop shows the abort condition does fire, but far too late:

travel-start targetWindow=10009 startFrame=(-1400.0, 200.0, 518.0, 400.0)
frame-check live=(-1400.0, 200.0, 518.0, 400.0) abort=false   x83
frame-check live=(600.0, 200.0, 518.0, 400.0) abort=true      <- last frame of the travel

The window reached the main display about 0.45s into the travel, yet the re-read returned the old frame for 83 consecutive frames (~1.3s — the whole travel). So the chain environment.windowBounds → currentWindowBounds(for:) → windowGeometry(forWindowID:) → CGWindowListCopyWindowInfo([.optionIncludingWindow], id) reports the old frame for roughly a second after the window has actually moved.

The frame signal this patch relies on therefore arrives after the travel has already finished, which is exactly why the cursor still finishes on the old screen.

What should work instead. The accessibility move notification is timely: pumpFrame() runs the run loop every 1/120s, and in the same trace the notification path had already re-placed the cursor ~0.26s after the move. Aborting on that notification (a travel generation counter bumped from targetWindowDidMove()) uses a signal that arrives while the travel is still running, and the final placement should take the window origin from the AX element (kAXPositionAttribute) instead of the lagging window list.

Marking this draft until it is re-verified on the same rig. The unit tests here remain valid — the pure decision, and the loop behaviour under an injected frame change — but they cannot catch the real-world timing of the frame source.

[wicgee]

Follow-up: the abort works, but a later placement re-applies the stale point

I iterated on this on the same rig. The notification-driven abort behaves exactly as intended, and the instrumentation shows why it is still not enough:

abort-by-notification after 1 notifications
abort ts=1789470020.608591
  ax  =(600.0, 200.0, 518.0, 400.0)     <- fresh: the window is already on the main display
  list=(-1400.0, 200.0, 518.0, 400.0)   <- stale: the window list still reports the old origin
  anchor=10009                           <- the resting anchor resolves and matches
  tip=(-765.6, 734.3)                    <- the cursor was mid-travel on the old display

So: the accessibility notification arrives while the travel is still running, the AX frame read is correct at that instant (and CGWindowListCopyWindowInfo is not), and the abort lands the cursor on the live frame. The cursor still finishes on the old display, because the action pipeline places it again afterwards: VisualInteractionChoreographer runs move → settle → action, and that settle (and the click pulse) use the cursor target computed from the snapshot taken before the window moved. The abort fixes the travel and is then overwritten by a stale settle.

A real fix therefore has to invalidate the pending cursor target when the window moves, not only stop the travel — the recomputation belongs in the choreographer/service path, next to where the target is derived from the snapshot. I have not pushed that change; the branch is reverted to the reviewed commit and this PR stays draft until a change is verified on the rig.

Two things worth keeping regardless, both measured here:

• The window list can lag a window move by ~1s (83 consecutive frames), so anything derived from it during a move is stale.
• The accessibility element position (kAXPositionAttribute) reflects the move immediately.

[wicgee]

Fixed and verified on the reporting machine

The earlier failures here were partly a false negative: another session was driving Open Computer Use at the same time. After a passive 10-second check that the overlay was not moving on its own, the scenario was re-run cleanly.

Result: three consecutive passes. Window dragged from the built-in display to the main display during the action → the cursor ends on the main display at x=1023, inside the window (600–1118). A control build with only the window-list fallback leaves the cursor on the old display at -977 in the same scenario.

Two things were still missing after the first commit here. Both are now in:

1. The signal was too late. CGWindowListCopyWindowInfo keeps returning the pre-move frame — 83 consecutive travel frames (~1.3s) identical, measured. The window's accessibility element (kAXPositionAttribute) already reports the new origin at that moment. The abort and the re-derivation now read the accessibility frame, with the window list only as a fallback.
2. The travel abort was being undone by the action pipeline itself. VisualInteractionChoreographer.approach is moveCursor(target) → settleCursorArrival(target), and that settle re-places the cursor using the same target computed before the window moved. settle and pulseClick now re-derive their point from the live frame first (element frames are window-relative: keep the window-local offset, move the origin).

The abort trigger is now the accessibility move notification (a generation counter bumped in targetWindowDidMove) instead of polling: the notification arrives while the travel is still running, because pumpFrame() runs the run loop every 1/120s.

Tests: three new cases — notification abort mid-travel, settle, pulseClick — full suite 267 tests / 1 skipped / 0 failures; make check-docs passes. Negative proof: without the settle re-derivation the new case fails with the cursor at (140, 370), i.e. the old display.

Still not covered: a real pointer drag (button held) was not reproduced end to end — the window move is scripted, so the frame-change timing is representative but not identical.

[wicgee]

Added a documentation commit (docs(skill)) to this branch while verifying the fix on the reporting machine.

skills/open-computer-use/references/usage.md now has a Focus Behaviour section: the default path does not take focus (auto click → accessibility action then app_post; type_text → AXValue write, with PID-directed keyboard events only as a fallback), while raising or activating the target app is confined to three opt-in paths — activateClickTarget and recoverVisibleWindow behind allow_window_recovery, and prepareAppForGlobalPointerInput behind click_method: "global" plus OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1. It also records that a target app can raise itself in response to an accessibility action — Chromium does — which no client-side setting prevents.

It ends with the two-display verification recipe used here, including one testing discipline worth writing down: never synthesize the "user is typing" side of the test. Injected key events go to whatever is frontmost at that instant, which is the real desktop; while building this verification an emulation harness guessed the wrong window and typed a 45-character probe into the user's own chat input. Only a human types; the agent only measures.

Measured on the reporting machine (two displays, default settings, no allow_window_recovery):

• AI clicking a Chromium button while the user typed on the other display: the frontmost app never changed across the whole run (8s sampling), click effect confirmed by a page counter.
• AI typing into a Chromium field while the user typed on the other display: frontmost app unchanged for 45s, and the field contained exactly the probe string — no cross-contamination in either direction, confirmed by the person typing.