Alvin/README.md

Alvin

Agent Leveraging Vision and Input. Nice.

A cross-platform C++23 library that lets a program see the screen (capture monitors or windows), act on it (synthesize mouse and keyboard input), and find things on it (optional OpenCV template matching). All of this sits behind one platform-neutral API, with swappable backends for each desktop.

#include <alvin/alvin.hpp>
#include <alvin/vision.hpp>

auto input  = alvin::Input::create().value();
auto screen = alvin::Capture::create().value();   // portal picker appears the first time only

auto frame  = screen->grab().value();
auto button = alvin::vision::loadImage("ok_button.png").value();
if (auto hit = alvin::vision::find(frame, button).value())
    input->clickAt(frame.toDesktop(hit->rect.center()));

input->chord("ctrl+s");
input->typeText("héllo wörld ✓");

Full API reference: docs/API.md

Layout

include/alvin/      public API (the only headers users include)
  input.hpp         Input: moveTo/moveBy/click/drag/scroll/keyDown/keyUp/chord/typeText
  capture.hpp       Capture: grab() latest frame, nextFrame() wait for a new one
  frame.hpp         Frame: BGRA pixels + where they sit on the desktop
  window.hpp        WindowManager: list/active/find/focus top-level windows
  types.hpp         Point/Rect/Monitor, listMonitors(), desktopBounds()
  keys.hpp          Key enum (physical keys), MouseButton, chord parsing
  vision.hpp        optional OpenCV helpers (alvin::vision target)
src/common/         platform-neutral code + the internal platform interface
src/linux/          Linux backends
src/windows/        Windows backends
src/vision/         OpenCV integration
protocols/          vendored Wayland protocol XML
examples/           the `alvin` CLI
tests/              Catch2 unit tests

Every public type is platform-neutral; backend headers (libei, PipeWire, Wayland, Win32) never leak into include/. Each platform implements four entry points in src/common/platform.hpp (createInput, createCapture, createWindowManager, listMonitors), which pick a backend at runtime.

Backends

Area	Backend	Where it works	Status
Input	wlr — zwlr_virtual_pointer_v1 + zwp_virtual_keyboard_v1	Hyprland, sway, wlroots compositors. No dialog.	Tested on Hyprland (motion, multi-monitor, fractional scale)
Input	libei — via the RemoteDesktop portal	GNOME, KDE Plasma	Builds; untested (Hyprland's portal has no RemoteDesktop)
Input	uinput — kernel virtual devices	Any Linux session incl. X11/TTY; needs write access to /dev/uinput	Tested on Hyprland (absolute + relative motion)
Input	sendinput — Win32 SendInput	Windows	Written; not yet compiled
Capture	portal — ScreenCast portal + PipeWire (SHM buffers)	Any modern Wayland desktop	Portal handshake tested; frame path awaiting first run
Capture	gdi — BitBlt / PrintWindow	Windows	Written; not yet compiled
Windows	hyprland — IPC socket	Hyprland	Tested
Windows	win32 — EnumWindows + DWM	Windows	Written; not yet compiled

With InputBackend::Auto, Linux tries wlr → libei → uinput. Override with ALVIN_INPUT_BACKEND=wlr|libei|uinput or InputOptions::backend.

Things worth knowing

• Coordinates are global desktop coordinates in logical pixels on Linux (what hyprctl reports) and physical pixels on Windows (Alvin makes the process per-monitor DPI aware). A Frame from a scaled monitor has more pixels than its logical size; use frame.toDesktop() / fromDesktop().
• Portal restore tokens: Capture::restoreToken() / Input::restoreToken() return a token; pass it back in the options next run to skip the picker. The CLI caches them in ~/.cache/alvin/.
• Typing text: wlr uploads a keymap containing exactly the characters needed, so any Unicode text works regardless of layout. uinput and libei go through the layout (libei uses the server's keymap; uinput uses XKB_DEFAULT_LAYOUT, defaulting to us).
• uinput devices take ~2.5 s to be adopted by Hyprland; Input::create() waits for the compositor to open them. Relative motion through uinput is subject to pointer acceleration.
• Windows UIPI blocks injecting input into elevated windows unless Alvin runs elevated too.

Building

Arch Linux dependencies (most are already on a Hyprland system):

sudo pacman -S --needed cmake ninja pkgconf wayland wayland-protocols libxkbcommon \
    glib2 pipewire libei opencv catch2

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
ctest --test-dir build

Options (ON/OFF/AUTO, where AUTO enables a backend when its dependencies are found): ALVIN_WITH_WAYLAND, ALVIN_WITH_PORTAL, ALVIN_WITH_LIBEI, ALVIN_WITH_VISION, plus ALVIN_WITH_UINPUT, ALVIN_BUILD_EXAMPLES, ALVIN_BUILD_TESTS.

On Windows, build with a recent MSVC (C++23 std::expected) and CMake; OpenCV is optional.

Using Alvin in your project

There are two ways to use Alvin from another project. Either way you link the same two targets:

Target	What it gives you
alvin::alvin	Core library: input, capture, windows, monitors (#include <alvin/alvin.hpp>)
alvin::vision	OpenCV helpers (#include <alvin/vision.hpp>). Only exists if Alvin was built with OpenCV.

Your project must be built as C++23 or newer. Linking alvin::alvin requests that for you.

Option 1: build Alvin with your project (add_subdirectory)

Your build compiles Alvin from source alongside your own code. There's nothing to install, and changes to Alvin are picked up on your next build. This is the easiest choice while Alvin is still changing.

1. Put Alvin somewhere your project can reach, e.g. as a git submodule:

   cd my_agent
   git submodule add https://github.com/<you>/Alvin.git external/Alvin
   

   Or just point at your existing checkout (step 2 accepts an absolute path).

2. In your project's CMakeLists.txt:

   cmake_minimum_required(VERSION 3.24)
   project(my_agent CXX)
   
   # Optional: skip Alvin's own tests and CLI while building your project.
   set(ALVIN_BUILD_TESTS OFF CACHE BOOL "" FORCE)
   set(ALVIN_BUILD_EXAMPLES OFF CACHE BOOL "" FORCE)
   
   add_subdirectory(external/Alvin)
   # With a checkout outside your source tree, give a build directory name as well:
   # add_subdirectory(/home/you/gitRepos/Alvin alvin)
   
   add_executable(my_agent main.cpp)
   target_link_libraries(my_agent PRIVATE alvin::alvin alvin::vision)
   

3. Build as normal:

   cmake -S . -B build -G Ninja
   cmake --build build
   

Option 2: install Alvin once, then find_package(Alvin)

This builds and installs Alvin as a compiled library (static .a/.lib or shared .so/.dll) with its headers. Any number of projects can then find it with find_package, without rebuilding Alvin.

1. Build and install Alvin. Pick an install prefix: ~/.local for just you (no sudo), or /usr/local for the whole system.

   cd Alvin
   cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
         -DCMAKE_INSTALL_PREFIX=$HOME/.local \
         -DBUILD_SHARED_LIBS=ON        # OFF (the default) for a static library
   cmake --build build
   cmake --install build             # prefix with sudo for /usr/local
   

   This installs:

   <prefix>/include/alvin/*.hpp                  headers
   <prefix>/lib/libalvin.so (or .a)              library, plus libalvin_vision if built
   <prefix>/lib/cmake/Alvin/AlvinConfig.cmake    what find_package reads
   <prefix>/bin/alvin                            the CLI
   

2. In your project's CMakeLists.txt:

   cmake_minimum_required(VERSION 3.24)
   project(my_agent CXX)
   
   find_package(Alvin 0.1 REQUIRED)                      # core only
   # find_package(Alvin 0.1 REQUIRED COMPONENTS vision)  # fail early if Alvin was built without OpenCV
   
   add_executable(my_agent main.cpp)
   target_link_libraries(my_agent PRIVATE alvin::alvin alvin::vision)
   

3. Configure your project, telling CMake where Alvin is installed. /usr/local and /usr are searched automatically; for anything else, pass the prefix:

   cmake -S . -B build -G Ninja -DCMAKE_PREFIX_PATH=$HOME/.local
   cmake --build build
   

Notes for option 2

• After updating Alvin, re-run the cmake --build and cmake --install commands from step 1. Projects linked against the shared library use the new version the next time they run, but only if the API didn't change; otherwise rebuild them. Projects linked against the static library always need rebuilding.
• Static vs shared: a static install bakes Alvin into your executable. Alvin's own dependencies (Wayland, PipeWire, libei, …) must still be installed wherever your program is built; AlvinConfig.cmake finds them for you. A shared install puts all of that in libalvin.so.
• "cannot open shared object file: libalvin.so" at runtime means the library's folder isn't on the loader's search path. This is common with ~/.local or /usr/local prefixes. Either run export LD_LIBRARY_PATH=$HOME/.local/lib, install to /usr, or add the folder to /etc/ld.so.conf.d/ and run sudo ldconfig.
• Versioning: find_package(Alvin 0.1) accepts any 0.1.x. Before 1.0, a new minor version (0.2) may change the API, so it won't be accepted as a match for 0.1.
• Windows: the same steps should work with MSVC, but they haven't been tested yet. Use --config Release on the build and install commands, and put the installed bin folder (where alvin.dll goes) on PATH when running.

Without CMake

If your project doesn't use CMake, do an option 2 install with -DBUILD_SHARED_LIBS=ON, then add <prefix>/include to your include path and link -lalvin (plus -lalvin_vision and OpenCV if you use vision). For example:

g++ -std=c++23 main.cpp -I$HOME/.local/include -L$HOME/.local/lib -lalvin -o my_agent
LD_LIBRARY_PATH=$HOME/.local/lib ./my_agent

The alvin CLI

alvin monitors
alvin windows
alvin focus firefox
alvin move 1440 810
alvin clickat 1440 810 right
alvin key ctrl+shift+t
alvin type "hello"
alvin capture shot.png              # first run shows the share picker
alvin find button.png 0.85          # prints desktop coordinates of the match
alvin --input uinput move 100 100

Roadmap

• Promptless Hyprland capture (hyprland-toplevel-export-v1, ext-image-copy-capture-v1) so an agent can capture any window by id without a picker
• DMA-BUF import for zero-copy PipeWire frames
• X11 backends (XTest input, XShm capture)
• Windows: DXGI Desktop Duplication / Windows.Graphics.Capture; CI build on Windows
• macOS: CGEvent input, ScreenCaptureKit capture
• OCR (Tesseract) in the vision module