Aeth3randClaude Opus 5.5
ca7e62ac8d
A cross-platform C++23 library with a platform-neutral public API and runtime-selected backends: - Input: Wayland virtual pointer/keyboard (Hyprland, wlroots), libei via the RemoteDesktop portal (GNOME, KDE), uinput, and Win32 SendInput - Capture: ScreenCast portal + PipeWire, and Win32 GDI - Windows: Hyprland IPC and Win32 enumeration - Monitors via wl_output/xdg-output and EnumDisplayMonitors - Optional OpenCV template matching (alvin::vision) Also adds the `alvin` CLI, Catch2 unit tests, install rules with find_package(Alvin) support, API docs (docs/API.md and the Obsidian vault), and README instructions for both ways of consuming the library. Windows backends are written but not yet compiled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1.8 KiB
Screen Capture & Computer Vision (Decision Making)
To "see" the screen and make complex logical evaluations, combine low-level frame grabbers with computer vision wrappers.
Linux Capture (Wayland/Hyprland): Implement PipeWire or utilize the xdg-desktop-portal (ScreenCast) API. Unprivileged apps cannot take screenshots directly in Wayland, so you must stream frames via PipeWire. [1, 2]
Cross-Platform Vision Processing: OpenCV (C++) is the industry standard for processing raw frame arrays. It allows for image template matching (e.g., finding a specific button on screen), color analysis, and image thresholding. [1]
Text Recognition (OCR): Integrate Tesseract OCR if the bot needs to read on-screen text, menus, or labels to inform its choices.
Cross-Platform Window Management & Coordinates
A major pitfall of Wayland automation is that clients do not know where other windows are located on the global coordinate space. [1]
- Linux (Hyprland IPC): Use
hyprctlJSON sockets or thehyprland-ipcprotocol to query window positions, bounding boxes, and active window titles. Because this mechanism is compositor-dependent, implement it cleanly under a modular interface. - Windows/macOS: Use Win32 (
GetWindowRect,GetForegroundWindow) and the macOSCGWindowListCopyWindowInfoAPI respectively to achieve identical window tracking capabilities on alternative platforms.