Skip to content

Add global OCR text capture (Ctrl/Cmd+Shift+T) - #2095

Open
x1xhlol wants to merge 7 commits into
CapSoftware:mainfrom
x1xhlol:feat/ocr-text-capture
Open

Add global OCR text capture (Ctrl/Cmd+Shift+T)#2095
x1xhlol wants to merge 7 commits into
CapSoftware:mainfrom
x1xhlol:feat/ocr-text-capture

Conversation

@x1xhlol

@x1xhlol x1xhlol commented Aug 6, 2026

Copy link
Copy Markdown

Summary

Adds a privacy-first "copy text from screen" feature: press Ctrl+Shift+T (Windows) / Cmd+Shift+T (macOS), drag over any screen region with the existing area selector, and the recognized text is copied straight to the clipboard. Nothing is written to disk by default.

Reuses Cap's existing native OCR engines (Apple Vision on macOS, Windows.Media.Ocr on Windows) — no new dependency, free, fast, on-device.

How it fits together:

  • HotkeyAction::OcrArea — new hotkey action, seeded with a default binding on startup via seed_default_hotkeys (a seeded list in HotkeysStore ensures it's only auto-added once and never overrides a user's existing binding). Pressing it opens the target-select overlay in a new RecordingTargetMode::Ocr.
  • target-select-overlay.tsx"ocr" target mode reuses the area-selector UI (screenshot-style: 1×1 min size, no recording controls, toolbar hint "Draw an area to copy text"). On mouse-up it invokes the new command instead of takeScreenshot; on error the overlay is restored with a toast.
  • capture_ocr_text (recording.rs) — new Tauri command:
    capture region in-memory → recognize_text_from_dynamic_image (native OCR)
      → clipboard.write_text → optional save_screenshot_project → optional notification
    
  • take_screenshot was split into capture_screen_image + save_screenshot_project so OCR shares the capture/persistence code without duplicating it.
  • Settings:
    • General → new "Text capture (OCR)" section with two default-off toggles: ocr_keep_screenshot (save the region as a normal screenshot project) and ocr_show_notification (system notification with a preview of the copied text).
    • Shortcuts → "Copy text from area (OCR)" entry, rebindable like any other hotkey.

Showcase

Pressing the shortcut opens the familiar area selector, in text-capture mode:

OCR capture overlay

Drag over any text and it lands on the clipboard (nothing written to disk):

OCR text copied to clipboard

The capture is a background gesture — the app you were in keeps keyboard focus afterwards:

Focus stays on the previous app

Settings — rebindable shortcut and the two default-off toggles:

Shortcut entry

Text capture settings

Testing (Windows, end-to-end on a live dev build)

Test Result
Ctrl+Shift+T opens OCR overlay ("Draw an area to copy text")
Drag region → OCR text copied to clipboard (exact match)
No screenshot persisted by default
Shortcuts: "Copy text from area (OCR)" = Ctrl Shift T
General: "Text capture (OCR)" toggles present, default OFF
"Keep a screenshot" ON → .cap screenshot project created

Known minor limitation: a very thin selection (~30 px tall) can return "No text was found in the selected area" (Windows.Media.Ocr small-region behavior); handled gracefully with a toast, slightly larger selections work.

macOS path uses the same Vision-based OCR code Cap already ships for the screenshot editor; it compiles for both platforms but was runtime-tested on Windows only.

Greptile Summary

Adds a global area-selection OCR workflow that captures screen pixels in memory, copies recognized text to the clipboard, and optionally saves the capture or displays a notification.

  • Adds and seeds a configurable OCR global hotkey.
  • Extends the target-selection overlay with an OCR-specific immediate-capture mode.
  • Refactors screenshot capture so OCR can reuse capture and project-persistence logic.
  • Adds persisted OCR settings and generated frontend IPC bindings.

Confidence Score: 4/5

The unrelated-overlay restoration defect should be fixed before merging because invoking OCR can leave existing Cap UI windows hidden.

The new global OCR path reuses a capture helper that hides several classes of Cap windows, while its completion and recovery logic restores only target-selector overlays.

Files Needing Attention: apps/desktop/src-tauri/src/recording.rs, apps/desktop/src/routes/target-select-overlay.tsx

Important Files Changed

Filename Overview
apps/desktop/src-tauri/src/recording.rs Adds the OCR command and shared capture helper, but the helper leaves unrelated Cap overlay windows hidden after capture.
apps/desktop/src/routes/target-select-overlay.tsx Adds OCR selection and IPC sequencing; its cleanup only restores target-selector overlays and therefore does not compensate for the backend window-state issue.
apps/desktop/src-tauri/src/hotkeys.rs Adds the OCR hotkey action and one-time conflict-aware default seeding without overriding existing bindings.
apps/desktop/src-tauri/src/general_settings.rs Adds backward-compatible, default-off OCR persistence and notification settings.
apps/desktop/src-tauri/src/screenshot_editor.rs Extracts the existing native OCR implementation to accept an in-memory DynamicImage.
apps/desktop/src-tauri/src/lib.rs Registers the new OCR command in the Tauri command bridge.
apps/desktop/src/routes/(window-chrome)/settings/general.tsx Adds controls for retaining OCR screenshots and showing notification previews.
apps/desktop/src/utils/tauri.ts Updates generated IPC bindings and shared types for the new command, settings, hotkey action, and target mode.

Comments Outside Diff (1)

  1. apps/desktop/src-tauri/src/recording.rs, line 2878-2881 (link)

    P1 Capture leaves overlays hidden

    When OCR runs while another Cap overlay is visible, capture_screen_image hides that window without restoring its visibility or cursor-event state, causing recordings, mode-selection, capture-area, or occluder overlays to remain inaccessible after the capture succeeds or fails.

    Knowledge Base Used:

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: apps/desktop/src-tauri/src/recording.rs
    Line: 2878-2881
    
    Comment:
    **Capture leaves overlays hidden**
    
    When OCR runs while another Cap overlay is visible, `capture_screen_image` hides that window without restoring its visibility or cursor-event state, causing recordings, mode-selection, capture-area, or occluder overlays to remain inaccessible after the capture succeeds or fails.
    
    **Knowledge Base Used:**
    - [Desktop Tauri App (Rust Backend)](https://app.greptile.com/cap/-/custom-context/knowledge-base/capsoftware/cap/-/docs/desktop-tauri-app.md)
    - [Desktop Frontend (apps/desktop/src)](https://app.greptile.com/cap/-/custom-context/knowledge-base/capsoftware/cap/-/docs/desktop-frontend.md)
    
    ---
    
    For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Prompt To Fix All With AI
### Issue 1
apps/desktop/src-tauri/src/recording.rs:2878-2881
**Capture leaves overlays hidden**

When OCR runs while another Cap overlay is visible, `capture_screen_image` hides that window without restoring its visibility or cursor-event state, causing recordings, mode-selection, capture-area, or occluder overlays to remain inaccessible after the capture succeeds or fails.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Reviews (1): Last reviewed commit: "Restore macOS diagnostics types in gener..." | Re-trigger Greptile

Context used:

@x1xhlol
x1xhlol force-pushed the feat/ocr-text-capture branch from 00b2e6a to b26aeff Compare August 6, 2026 13:44
@x1xhlol
x1xhlol force-pushed the feat/ocr-text-capture branch from ce81b7b to 6ecef8b Compare August 6, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant