Alvin/README.md

# Alvin

Agent Leveraging Vision and Input. Nice.

A cross-platform C++23 library that lets a program **see the screen** (capture monitors or windows),
**act on it** (synthesize mouse and keyboard input), and **find things on it** (optional OpenCV template
matching). All of this sits behind one platform-neutral API, with swappable backends for each desktop.

```cpp
#include <alvin/alvin.hpp>
#include <alvin/vision.hpp>

auto input  = alvin::Input::create().value();
auto screen = alvin::Capture::create().value();   // portal picker appears the first time only

auto frame  = screen->grab().value();
auto button = alvin::vision::loadImage("ok_button.png").value();
if (auto hit = alvin::vision::find(frame, button).value())
    input->clickAt(frame.toDesktop(hit->rect.center()));

input->chord("ctrl+s");
input->typeText("héllo wörld ✓");
```

**Full API reference: [docs/API.md](docs/API.md)**

## Layout

```
include/alvin/      public API (the only headers users include)
  input.hpp         Input: moveTo/moveBy/click/drag/scroll/keyDown/keyUp/chord/typeText
  capture.hpp       Capture: grab() latest frame, nextFrame() wait for a new one
  frame.hpp         Frame: BGRA pixels + where they sit on the desktop
  window.hpp        WindowManager: list/active/find/focus top-level windows
  types.hpp         Point/Rect/Monitor, listMonitors(), desktopBounds()
  keys.hpp          Key enum (physical keys), MouseButton, chord parsing
  vision.hpp        optional OpenCV helpers (alvin::vision target)
src/common/         platform-neutral code + the internal platform interface
src/linux/          Linux backends
src/windows/        Windows backends
src/vision/         OpenCV integration
protocols/          vendored Wayland protocol XML
examples/           the `alvin` CLI
tests/              Catch2 unit tests
```

Every public type is platform-neutral; backend headers (libei, PipeWire, Wayland, Win32) never leak
into `include/`. Each platform implements four entry points in `src/common/platform.hpp`
(`createInput`, `createCapture`, `createWindowManager`, `listMonitors`), which pick a backend at runtime.

## Backends

| Area    | Backend | Where it works | Status |
|---------|---------|----------------|--------|
| Input   | `wlr` — `zwlr_virtual_pointer_v1` + `zwp_virtual_keyboard_v1` | Hyprland, sway, wlroots compositors. No dialog. | Tested on Hyprland (motion, multi-monitor, fractional scale) |
| Input   | `libei` — via the RemoteDesktop portal | GNOME, KDE Plasma | Builds; untested (Hyprland's portal has no RemoteDesktop) |
| Input   | `uinput` — kernel virtual devices | Any Linux session incl. X11/TTY; needs write access to `/dev/uinput` | Tested on Hyprland (absolute + relative motion) |
| Input   | `sendinput` — Win32 `SendInput` | Windows | Written; not yet compiled |
| Capture | `portal` — ScreenCast portal + PipeWire (SHM buffers) | Any modern Wayland desktop | Portal handshake tested; frame path awaiting first run |
| Capture | `gdi` — BitBlt / PrintWindow | Windows | Written; not yet compiled |
| Windows | `hyprland` — IPC socket | Hyprland | Tested |
| Windows | `win32` — EnumWindows + DWM | Windows | Written; not yet compiled |

With `InputBackend::Auto`, Linux tries `wlr` → `libei` → `uinput`. Override with
`ALVIN_INPUT_BACKEND=wlr|libei|uinput` or `InputOptions::backend`.

### Things worth knowing

- **Coordinates** are global desktop coordinates in logical pixels on Linux (what `hyprctl` reports)
  and physical pixels on Windows (Alvin makes the process per-monitor DPI aware). A `Frame` from a
  scaled monitor has more pixels than its logical size; use `frame.toDesktop()` / `fromDesktop()`.
- **Portal restore tokens**: `Capture::restoreToken()` / `Input::restoreToken()` return a token; pass
  it back in the options next run to skip the picker. The CLI caches them in `~/.cache/alvin/`.
- **Typing text**: `wlr` uploads a keymap containing exactly the characters needed, so any Unicode
  text works regardless of layout. `uinput` and `libei` go through the layout (`libei` uses the
  server's keymap; `uinput` uses `XKB_DEFAULT_LAYOUT`, defaulting to `us`).
- **uinput** devices take ~2.5 s to be adopted by Hyprland; `Input::create()` waits for the compositor
  to open them. Relative motion through uinput is subject to pointer acceleration.
- **Windows UIPI** blocks injecting input into elevated windows unless Alvin runs elevated too.

## Building

Arch Linux dependencies (most are already on a Hyprland system):

```sh
sudo pacman -S --needed cmake ninja pkgconf wayland wayland-protocols libxkbcommon \
    glib2 pipewire libei opencv catch2
```

```sh
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build
```

Options (`ON`/`OFF`/`AUTO`, where `AUTO` enables a backend when its dependencies are found):
`ALVIN_WITH_WAYLAND`, `ALVIN_WITH_PORTAL`, `ALVIN_WITH_LIBEI`, `ALVIN_WITH_VISION`, plus
`ALVIN_WITH_UINPUT`, `ALVIN_BUILD_EXAMPLES`, `ALVIN_BUILD_TESTS`.

On Windows, build with a recent MSVC (C++23 `std::expected`) and CMake; OpenCV is optional.

## Using Alvin in your project

There are two ways to use Alvin from another project. Either way you link the same two targets:

| Target | What it gives you |
|--------|-------------------|
| `alvin::alvin` | Core library: input, capture, windows, monitors (`#include <alvin/alvin.hpp>`) |
| `alvin::vision` | OpenCV helpers (`#include <alvin/vision.hpp>`). Only exists if Alvin was built with OpenCV. |

Your project must be built as C++23 or newer. Linking `alvin::alvin` requests that for you.

### Option 1: build Alvin with your project (`add_subdirectory`)

Your build compiles Alvin from source alongside your own code. There's nothing to install, and
changes to Alvin are picked up on your next build. This is the easiest choice while Alvin is still
changing.

1. Put Alvin somewhere your project can reach, e.g. as a git submodule:

   ```sh
   cd my_agent
   git submodule add https://github.com/<you>/Alvin.git external/Alvin
   ```

   Or just point at your existing checkout (step 2 accepts an absolute path).

2. In your project's `CMakeLists.txt`:

   ```cmake
   cmake_minimum_required(VERSION 3.24)
   project(my_agent CXX)

   # Optional: skip Alvin's own tests and CLI while building your project.
   set(ALVIN_BUILD_TESTS OFF CACHE BOOL "" FORCE)
   set(ALVIN_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)

   add_subdirectory(external/Alvin)
   # With a checkout outside your source tree, give a build directory name as well:
   # add_subdirectory(/home/you/gitRepos/Alvin alvin)

   add_executable(my_agent main.cpp)
   target_link_libraries(my_agent PRIVATE alvin::alvin alvin::vision)
   ```

3. Build as normal:

   ```sh
   cmake -S . -B build -G Ninja
   cmake --build build
   ```

### Option 2: install Alvin once, then `find_package(Alvin)`

This builds and installs Alvin as a compiled library (static `.a`/`.lib` or shared `.so`/`.dll`)
with its headers. Any number of projects can then find it with `find_package`, without rebuilding
Alvin.

1. Build and install Alvin. Pick an install prefix: `~/.local` for just you (no sudo), or
   `/usr/local` for the whole system.

   ```sh
   cd Alvin
   cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
         -DCMAKE_INSTALL_PREFIX=$HOME/.local \
         -DBUILD_SHARED_LIBS=ON        # OFF (the default) for a static library
   cmake --build build
   cmake --install build             # prefix with sudo for /usr/local
   ```

   This installs:

   ```
   <prefix>/include/alvin/*.hpp                  headers
   <prefix>/lib/libalvin.so (or .a)              library, plus libalvin_vision if built
   <prefix>/lib/cmake/Alvin/AlvinConfig.cmake    what find_package reads
   <prefix>/bin/alvin                            the CLI
   ```

2. In your project's `CMakeLists.txt`:

   ```cmake
   cmake_minimum_required(VERSION 3.24)
   project(my_agent CXX)

   find_package(Alvin 0.1 REQUIRED)                      # core only
   # find_package(Alvin 0.1 REQUIRED COMPONENTS vision)  # fail early if Alvin was built without OpenCV

   add_executable(my_agent main.cpp)
   target_link_libraries(my_agent PRIVATE alvin::alvin alvin::vision)
   ```

3. Configure your project, telling CMake where Alvin is installed. `/usr/local` and `/usr` are
   searched automatically; for anything else, pass the prefix:

   ```sh
   cmake -S . -B build -G Ninja -DCMAKE_PREFIX_PATH=$HOME/.local
   cmake --build build
   ```

**Notes for option 2**

- **After updating Alvin**, re-run the `cmake --build` and `cmake --install` commands from step 1.
  Projects linked against the shared library use the new version the next time they run, but only
  if the API didn't change; otherwise rebuild them. Projects linked against the static library
  always need rebuilding.
- **Static vs shared:** a static install bakes Alvin into your executable. Alvin's own dependencies
  (Wayland, PipeWire, libei, …) must still be installed wherever your program is *built*;
  `AlvinConfig.cmake` finds them for you. A shared install puts all of that in `libalvin.so`.
- **"cannot open shared object file: libalvin.so"** at runtime means the library's folder isn't on
  the loader's search path. This is common with `~/.local` or `/usr/local` prefixes. Either run
  `export LD_LIBRARY_PATH=$HOME/.local/lib`, install to `/usr`, or add the folder to
  `/etc/ld.so.conf.d/` and run `sudo ldconfig`.
- **Versioning:** `find_package(Alvin 0.1)` accepts any 0.1.x. Before 1.0, a new minor version
  (0.2) may change the API, so it won't be accepted as a match for 0.1.
- **Windows:** the same steps should work with MSVC, but they haven't been tested yet. Use
  `--config Release` on the build and install commands, and put the installed `bin` folder (where
  `alvin.dll` goes) on `PATH` when running.

### Without CMake

If your project doesn't use CMake, do an option 2 install with `-DBUILD_SHARED_LIBS=ON`, then add
`<prefix>/include` to your include path and link `-lalvin` (plus `-lalvin_vision` and OpenCV if you
use vision). For example:

```sh
g++ -std=c++23 main.cpp -I$HOME/.local/include -L$HOME/.local/lib -lalvin -o my_agent
LD_LIBRARY_PATH=$HOME/.local/lib ./my_agent
```

## The `alvin` CLI

```sh
alvin monitors
alvin windows
alvin focus firefox
alvin move 1440 810
alvin clickat 1440 810 right
alvin key ctrl+shift+t
alvin type "hello"
alvin capture shot.png              # first run shows the share picker
alvin find button.png 0.85          # prints desktop coordinates of the match
alvin --input uinput move 100 100
```

## Roadmap

- Promptless Hyprland capture (`hyprland-toplevel-export-v1`, `ext-image-copy-capture-v1`)
  so an agent can capture any window by id without a picker
- DMA-BUF import for zero-copy PipeWire frames
- X11 backends (XTest input, XShm capture)
- Windows: DXGI Desktop Duplication / Windows.Graphics.Capture; CI build on Windows
- macOS: CGEvent input, ScreenCaptureKit capture
- OCR (Tesseract) in the vision module