Add Alvin library: input, capture, windows and vision
A cross-platform C++23 library with a platform-neutral public API and runtime-selected backends: - Input: Wayland virtual pointer/keyboard (Hyprland, wlroots), libei via the RemoteDesktop portal (GNOME, KDE), uinput, and Win32 SendInput - Capture: ScreenCast portal + PipeWire, and Win32 GDI - Windows: Hyprland IPC and Win32 enumeration - Monitors via wl_output/xdg-output and EnumDisplayMonitors - Optional OpenCV template matching (alvin::vision) Also adds the `alvin` CLI, Catch2 unit tests, install rules with find_package(Alvin) support, API docs (docs/API.md and the Obsidian vault), and README instructions for both ways of consuming the library. Windows backends are written but not yet compiled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
76ff506fd0
commit
ca7e62ac8d
59 files changed
+6515
-1
No files matched your search
@@ -0,0 +1,15 @@
|
||||
# Screen Capture & Computer Vision (Decision Making)
|
||||
To "see" the screen and make complex logical evaluations, combine low-level frame grabbers with computer vision wrappers.
|
||||
|
||||
**Linux Capture (Wayland/Hyprland):** Implement **PipeWire** or utilize the **xdg-desktop-portal (ScreenCast)** API. Unprivileged apps cannot take screenshots directly in Wayland, so you must stream frames via PipeWire. [[1](https://www.reddit.com/r/kde/comments/113wt14/wayland_input_emulation_org_kde_kwin_fake_input/), [2](https://github.com/can1357/oh-my-pi/issues/7886)]
|
||||
|
||||
**Cross-Platform Vision Processing:** **[OpenCV (C++)](https://opencv.org/)** is the industry standard for processing raw frame arrays. It allows for image template matching (e.g., finding a specific button on screen), color analysis, and image thresholding. [[1](https://www.reddit.com/r/hyprland/comments/1rwxfae/i_built_a_scrolling_longscreenshot_tool_for/)]
|
||||
|
||||
**Text Recognition (OCR):** Integrate **[Tesseract OCR](https://github.com/tesseract-ocr/tesseract)** if the bot needs to read on-screen text, menus, or labels to inform its choices.
|
||||
|
||||
|
||||
# Cross-Platform Window Management & Coordinates
|
||||
A major pitfall of Wayland automation is that clients do not know where other windows are located on the global coordinate space. [[1](https://github.com/developer239/robot-cpp)]
|
||||
|
||||
- **Linux (Hyprland IPC):** Use **`hyprctl`** JSON sockets or the **`hyprland-ipc`** protocol to query window positions, bounding boxes, and active window titles. Because this mechanism is compositor-dependent, implement it cleanly under a modular interface.
|
||||
- **Windows/macOS:** Use Win32 (`GetWindowRect`, `GetForegroundWindow`) and the macOS `CGWindowListCopyWindowInfo` API respectively to achieve identical window tracking capabilities on alternative platforms.
|
||||
Reference in new issue
Block a user