# Screen Capture & Computer Vision (Decision Making)
To "see" the screen and make complex logical evaluations, combine low-level frame grabbers with computer vision wrappers.

 **Linux Capture (Wayland/Hyprland):** Implement **PipeWire** or utilize the **xdg-desktop-portal (ScreenCast)** API. Unprivileged apps cannot take screenshots directly in Wayland, so you must stream frames via PipeWire. [[1](https://www.reddit.com/r/kde/comments/113wt14/wayland_input_emulation_org_kde_kwin_fake_input/), [2](https://github.com/can1357/oh-my-pi/issues/7886)]

 **Cross-Platform Vision Processing:** **[OpenCV (C++)](https://opencv.org/)** is the industry standard for processing raw frame arrays. It allows for image template matching (e.g., finding a specific button on screen), color analysis, and image thresholding. [[1](https://www.reddit.com/r/hyprland/comments/1rwxfae/i_built_a_scrolling_longscreenshot_tool_for/)]

**Text Recognition (OCR):** Integrate **[Tesseract OCR](https://github.com/tesseract-ocr/tesseract)** if the bot needs to read on-screen text, menus, or labels to inform its choices.


# Cross-Platform Window Management & Coordinates
A major pitfall of Wayland automation is that clients do not know where other windows are located on the global coordinate space. [[1](https://github.com/developer239/robot-cpp)]

- **Linux (Hyprland IPC):** Use **`hyprctl`** JSON sockets or the **`hyprland-ipc`** protocol to query window positions, bounding boxes, and active window titles. Because this mechanism is compositor-dependent, implement it cleanly under a modular interface.
- **Windows/macOS:** Use Win32 (`GetWindowRect`, `GetForegroundWindow`) and the macOS `CGWindowListCopyWindowInfo` API respectively to achieve identical window tracking capabilities on alternative platforms.