Aeth3randClaude Opus 5.5
ca7e62ac8d
A cross-platform C++23 library with a platform-neutral public API and runtime-selected backends: - Input: Wayland virtual pointer/keyboard (Hyprland, wlroots), libei via the RemoteDesktop portal (GNOME, KDE), uinput, and Win32 SendInput - Capture: ScreenCast portal + PipeWire, and Win32 GDI - Windows: Hyprland IPC and Win32 enumeration - Monitors via wl_output/xdg-output and EnumDisplayMonitors - Optional OpenCV template matching (alvin::vision) Also adds the `alvin` CLI, Catch2 unit tests, install rules with find_package(Alvin) support, API docs (docs/API.md and the Obsidian vault), and README instructions for both ways of consuming the library. Windows backends are written but not yet compiled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
15 lines
1.8 KiB
Markdown
15 lines
1.8 KiB
Markdown
# Screen Capture & Computer Vision (Decision Making)
|
|
To "see" the screen and make complex logical evaluations, combine low-level frame grabbers with computer vision wrappers.
|
|
|
|
**Linux Capture (Wayland/Hyprland):** Implement **PipeWire** or utilize the **xdg-desktop-portal (ScreenCast)** API. Unprivileged apps cannot take screenshots directly in Wayland, so you must stream frames via PipeWire. [[1](https://www.reddit.com/r/kde/comments/113wt14/wayland_input_emulation_org_kde_kwin_fake_input/), [2](https://github.com/can1357/oh-my-pi/issues/7886)]
|
|
|
|
**Cross-Platform Vision Processing:** **[OpenCV (C++)](https://opencv.org/)** is the industry standard for processing raw frame arrays. It allows for image template matching (e.g., finding a specific button on screen), color analysis, and image thresholding. [[1](https://www.reddit.com/r/hyprland/comments/1rwxfae/i_built_a_scrolling_longscreenshot_tool_for/)]
|
|
|
|
**Text Recognition (OCR):** Integrate **[Tesseract OCR](https://github.com/tesseract-ocr/tesseract)** if the bot needs to read on-screen text, menus, or labels to inform its choices.
|
|
|
|
|
|
# Cross-Platform Window Management & Coordinates
|
|
A major pitfall of Wayland automation is that clients do not know where other windows are located on the global coordinate space. [[1](https://github.com/developer239/robot-cpp)]
|
|
|
|
- **Linux (Hyprland IPC):** Use **`hyprctl`** JSON sockets or the **`hyprland-ipc`** protocol to query window positions, bounding boxes, and active window titles. Because this mechanism is compositor-dependent, implement it cleanly under a modular interface.
|
|
- **Windows/macOS:** Use Win32 (`GetWindowRect`, `GetForegroundWindow`) and the macOS `CGWindowListCopyWindowInfo` API respectively to achieve identical window tracking capabilities on alternative platforms. |