# Alvin API Reference Everything public lives in namespace `alvin` (vision helpers in `alvin::vision`). Include the umbrella header for the core library: ```cpp #include // Input, Capture, WindowManager, Frame, keys, types #include // optional, needs the alvin::vision target (OpenCV) ``` - [Using Alvin in your project](#using-alvin-in-your-project) - [Error handling](#error-handling) - [Coordinates and monitors](#coordinates-and-monitors) - [Input](#input) - [Keys and chords](#keys-and-chords) - [Capture](#capture) - [Frame](#frame) - [Windows](#windows) - [Vision (OpenCV)](#vision-opencv) - [Putting it together](#putting-it-together) --- ## Using Alvin in your project Either build Alvin as part of your project, or install it once and use `find_package`. The README's "Using Alvin in your project" section has step-by-step instructions for both. ```cmake add_subdirectory(external/Alvin) # option 1: build from source # find_package(Alvin 0.1 REQUIRED) # option 2: use an installed Alvin target_link_libraries(my_agent PRIVATE alvin::alvin) # core target_link_libraries(my_agent PRIVATE alvin::vision) # + OpenCV helpers, if built ``` Alvin requires C++23 (it uses `std::expected`). --- ## Error handling Alvin never throws. Every call that can fail returns a `Result`, which is `std::expected`. Calls with no value return `Status` (`Result`). ```cpp struct Error { ErrorCode code; std::string message; // human-readable detail }; enum class ErrorCode { Unknown, Unsupported, NotAvailable, PermissionDenied, Cancelled, Timeout, InvalidArgument, NotFound, Io, Protocol, }; std::string_view toString(ErrorCode code); ``` | Code | Typical cause | |------|---------------| | `Unsupported` | Backend not compiled in, or the feature isn't possible on this platform | | `NotAvailable` | Backend compiled in but its service is missing (no Wayland display, no portal interface) | | `PermissionDenied` | `/dev/uinput` not writable, portal request denied, Windows UIPI block | | `Cancelled` | The user dismissed a portal dialog | | `Timeout` | No answer from a portal dialog, or no frame arrived in time | | `NotFound` | A window or monitor that doesn't exist | Usage patterns: ```cpp auto input = alvin::Input::create(); if (!input) { std::cerr << alvin::toString(input.error().code) << ": " << input.error().message << '\n'; return 1; } (*input)->moveTo({100, 100}); // Status; check it or ignore it if (auto s = (*input)->click(); !s) // check a Status std::cerr << s.error().message << '\n'; auto in = alvin::Input::create().value(); // or throw std::bad_expected_access on failure ``` --- ## Coordinates and monitors `#include ` All positions are **global desktop coordinates**: - **Linux:** logical pixels, after monitor scaling. These are the numbers `hyprctl` reports. - **Windows:** physical pixels. Alvin makes the process per-monitor DPI aware. ```cpp struct Point { int x, y; }; struct Size { int width, height; }; struct Rect { int x, y, width, height; int right() const; // x + width int bottom() const; // y + height Point center() const; bool empty() const; bool contains(Point p) const; Rect united(const Rect& o) const; // bounding box of both }; struct Monitor { std::string name; // "DP-2", "\\.\DISPLAY1" std::string description; // make/model Rect bounds; // position and size on the desktop Size pixelSize; // actual mode, e.g. 3840x2160 double scale; // pixelSize / logical size, e.g. 1.333 bool primary; }; Result> listMonitors(); Result desktopBounds(); // bounding box of all monitors ``` ```cpp for (const auto& m : alvin::listMonitors().value()) std::printf("%s at %d,%d (%dx%d logical, scale %.2f)\n", m.name.c_str(), m.bounds.x, m.bounds.y, m.bounds.width, m.bounds.height, m.scale); ``` --- ## Input `#include ` ### Creating ```cpp static Result> Input::create(const InputOptions& options = {}); struct InputOptions { InputBackend backend = InputBackend::Auto; std::string restoreToken; // libei only: skip the portal dialog (see restoreToken()) std::chrono::milliseconds permissionTimeout = std::chrono::minutes(2); }; enum class InputBackend { Auto, WlrVirtual, LibEi, UInput, SendInput }; ``` | Backend | Platform | Notes | |---------|----------|-------| | `WlrVirtual` | Hyprland, sway, wlroots | No dialog. Types any Unicode text. | | `LibEi` | GNOME, KDE (RemoteDesktop portal) | Shows a permission dialog unless given a restore token. | | `UInput` | Any Linux session | Needs write access to `/dev/uinput`. Takes about 2.5 s to create on Hyprland. | | `SendInput` | Windows | Can't inject into elevated windows unless Alvin runs elevated too. | With `Auto`, Linux tries `WlrVirtual`, then `LibEi`, then `UInput`. You can force a backend with the environment variable `ALVIN_INPUT_BACKEND=wlr|libei|uinput` (Linux only). Create one `Input` and keep it for the life of your program. Creation is the expensive part (connecting to the compositor, creating devices, portal dialogs). ### Mouse | Call | Description | |------|-------------| | `Status moveTo(Point p)` | Move the pointer to an absolute desktop position. | | `Status moveBy(int dx, int dy)` | Relative motion. With `uinput`, the compositor applies pointer acceleration. | | `Status buttonDown(MouseButton b)` / `buttonUp(b)` | Press or release a button. | | `Status click(MouseButton b = Left, int count = 1)` | Click in place. `count = 2` double-clicks. | | `Status clickAt(Point p, MouseButton b = Left, int count = 1)` | Move, then click. | | `Status drag(Point from, Point to, MouseButton b = Left)` | Press at `from`, move to `to` in 10 steps, release. | | `Status scroll(int dx, int dy)` | Wheel notches. Positive `dy` scrolls **down**, positive `dx` scrolls **right**. | `MouseButton` is `Left`, `Right`, `Middle`, `Back` or `Forward`. ```cpp input->clickAt({1440, 810}); // left click input->clickAt({1440, 810}, alvin::MouseButton::Right); input->click(alvin::MouseButton::Left, 2); // double-click in place input->drag({100, 100}, {600, 400}); // e.g. select or move something input->scroll(0, 3); // 3 notches down ``` ### Keyboard | Call | Description | |------|-------------| | `Status keyDown(Key k)` / `keyUp(Key k)` | Press or release one physical key. | | `Status keyPress(Key k)` | Down, then up. | | `Status chord(std::initializer_list)` | Press in order, release in reverse. | | `Status chord(std::string_view)` | Same, from text like `"ctrl+shift+t"`. | | `Status typeText(std::string_view utf8)` | Type text. `\n` presses Enter and `\t` presses Tab. | ```cpp using alvin::Key; input->keyPress(Key::Enter); input->chord({Key::LeftCtrl, Key::C}); input->chord("ctrl+shift+t"); input->chord("super+enter"); input->typeText("Hello, wörld ✓\n"); ``` `chord()` always releases every key it pressed, even when an earlier call fails, so a key is never left held down. **How `typeText` handles layouts:** - **`wlr`:** builds a keymap containing exactly the characters needed, so any Unicode text works. - **`libei`:** uses the server's keyboard layout. If the server supports libei's text capability, it sends the text directly. - **`uinput`:** uses the layout named in `XKB_DEFAULT_LAYOUT`/`XKB_DEFAULT_VARIANT` (default `us`). Characters that aren't on that layout fail with `Unsupported` before anything is typed. - **`sendinput`:** sends Unicode directly, so any text works. ### Other | Call | Description | |------|-------------| | `InputBackend backend() const` | Which backend was chosen. | | `std::string restoreToken() const` | libei: save this and pass it in `InputOptions::restoreToken` next run to skip the dialog. Empty for other backends. | | `void setEventDelay(std::chrono::milliseconds)` | Pause between the down and up halves of clicks, key presses and typed characters. The default is 8 ms. Raise it for applications that miss fast input. | | `std::string_view toString(InputBackend)` | `"wlr"`, `"libei"`, `"uinput"`, `"sendinput"`, `"auto"` | --- ## Keys and chords `#include ` `Key` names **physical key positions** on a US QWERTY layout, the same way evdev and Win32 scan codes do. Use `typeText()` when you want particular characters. ``` A–Z, Num0–Num9, F1–F24, Escape, Enter, Tab, Backspace, Space, Minus, Equal, LeftBracket, RightBracket, Backslash, Semicolon, Apostrophe, Grave, Comma, Period, Slash, CapsLock, NumLock, ScrollLock, LeftShift, RightShift, LeftCtrl, RightCtrl, LeftAlt, RightAlt, LeftSuper, RightSuper, Menu, Insert, Delete, Home, End, PageUp, PageDown, Left, Right, Up, Down, PrintScreen, Pause, Kp0–Kp9, KpDecimal, KpDivide, KpMultiply, KpSubtract, KpAdd, KpEnter, VolumeMute, VolumeDown, VolumeUp, MediaPlayPause, MediaStop, MediaNext, MediaPrevious ``` ```cpp std::optional keyFromName(std::string_view name); // case-insensitive std::string_view keyName(Key key); std::optional> parseChord(std::string_view chord); bool isModifier(Key key); ``` Names accepted by `keyFromName` and in chord strings: | Key | Names | |-----|-------| | Letters and digits | `a`…`z`, `0`…`9` | | Function keys | `f1`…`f24` | | Modifiers | `shift`/`lshift`, `rshift`, `ctrl`/`control`/`lctrl`, `rctrl`, `alt`/`lalt`, `ralt`/`altgr`, `super`/`win`/`meta`/`cmd`/`lsuper`, `rsuper` | | Editing | `enter`/`return`, `esc`/`escape`, `tab`, `backspace`, `space`, `insert`, `delete`/`del` | | Navigation | `home`, `end`, `pageup`/`pgup`, `pagedown`/`pgdn`, `left`, `right`, `up`, `down` | | Punctuation | `minus`/`-`, `equal`/`=`, `[`, `]`, `\`, `;`, `'`, `` ` ``, `,`, `.`, `/` (or their word names) | | Keypad | `kp0`…`kp9`, `kpdecimal`, `kpdivide`, `kpmultiply`, `kpsubtract`, `kpadd`, `kpenter` | | Media | `mute`, `volumeup`, `volumedown`, `playpause`, `stop`, `next`, `prev`/`previous` | | Other | `capslock`, `numlock`, `scrolllock`, `printscreen`/`print`, `pause`, `menu` | Chords are joined with `+`, and spaces around keys are ignored. `"ctrl++"` means Ctrl plus the `+`/`=` key. --- ## Capture `#include ` ### Creating ```cpp static Result> Capture::create(const CaptureOptions& options = {}); struct CaptureOptions { CaptureBackend backend = CaptureBackend::Auto; // Auto, Portal, Gdi CaptureSource source = CaptureSource::Monitor; // Monitor, Window, Any int monitorIndex = 0; // Windows: which monitor (index into listMonitors()) std::optional window; // Windows: capture this window directly bool showCursor = false; std::string restoreToken; // Portal: skip the picker (see below) std::chrono::milliseconds permissionTimeout = std::chrono::minutes(2); }; ``` **Linux (`Portal`):** `create()` opens the desktop's share picker. On Hyprland that's `hyprland-share-picker`. `source` controls whether the picker offers monitors, windows or both. When the call returns, a PipeWire stream is running in the background. **Windows (`Gdi`):** there's no dialog. Set `monitorIndex`, or set `window` to capture one window. ### Skipping the picker next time ```cpp alvin::CaptureOptions opts; opts.restoreToken = loadSavedToken(); // empty the first time auto cap = alvin::Capture::create(opts).value(); saveToken(cap->restoreToken()); // new token each session; always re-save it ``` The `alvin` CLI does this for you and stores tokens in `~/.cache/alvin/`. > Don't leave a picker unanswered. While it's open, xdg-desktop-portal-hyprland > blocks every other portal request on the system, screen sharing in other > apps included. Use `permissionTimeout` to limit how long `create()` waits. ### Getting frames | Call | Description | |------|-------------| | `Result grab(timeout = 5s)` | The most recent frame. Blocks only until the first frame exists. | | `Result nextFrame(timeout = 5s)` | Waits for a frame newer than the last one you received. | | `std::string restoreToken() const` | Portal token for next time. | | `CaptureBackend backend() const` | Which backend was chosen. | PipeWire streams only send a frame when the screen changes, so `grab()` can return an image that's a few seconds old if nothing has moved. That's usually what you want. After you act (click, type), call `nextFrame()` to see the result: ```cpp input->clickAt(button); auto after = cap->nextFrame(std::chrono::seconds(2)); if (!after && after.error().code == alvin::ErrorCode::Timeout) { // nothing on screen changed } ``` A `Capture` holds a live stream, so keep one and reuse it. Don't create one per frame. --- ## Frame `#include ` A `Frame` is an owned BGRA8 image (4 bytes per pixel: B, G, R, A, with no row padding). It also records where it sits on the desktop. | Member | Description | |--------|-------------| | `Frame(int w, int h)` | New zero-filled frame. | | `int width()`, `int height()`, `Size size()` | Size in **pixels**. | | `std::size_t stride()` | Bytes per row (`width * 4`). | | `uint8_t* data()`, `row(y)`, `std::span bytes()` | Raw pixel access. | | `Color pixel(int x, int y)` | `{r, g, b, a}`. Returns all zeros if out of bounds. | | `Rect desktopRect()` | The area of the desktop this frame shows, in desktop coordinates. | | `Point toDesktop(Point px)` | Frame pixel to desktop coordinate. | | `Point fromDesktop(Point p)` | Desktop coordinate to frame pixel. | | `Frame crop(Rect r)` | Sub-image (`r` is in frame pixels and gets clipped). `desktopRect()` is adjusted to match. | | `int64_t timestampNs()` | Capture timestamp. | | `Status savePpm(path)` | Save without dependencies (PPM). With vision, use `vision::saveImage` for PNG/JPG. | **Always convert through `toDesktop()` before clicking.** On a monitor scaled to 1.33, a 3840×2160 frame covers 2880×1620 desktop units, so frame pixels aren't desktop coordinates: ```cpp auto frame = cap->grab().value(); alvin::Point pixel{1920, 1080}; // something found in the image input->clickAt(frame.toDesktop(pixel)); // correct on any scale ``` ```cpp alvin::Color c = frame.pixel(10, 10); if (c.r > 200 && c.g < 50 && c.b < 50) { /* red */ } ``` --- ## Windows `#include ` Wayland hides other windows from normal clients, so each compositor needs its own backend. Right now that's Hyprland (IPC socket) and Win32. ```cpp static Result> WindowManager::create(WindowBackend b = WindowBackend::Auto); struct WindowInfo { WindowId id; // opaque handle; pass it to focus() std::string title; std::string appId; // Hyprland class / Win32 executable name Rect bounds; // desktop coordinates int64_t pid; bool focused, visible, minimized, fullscreen; std::string workspace; // Hyprland workspace name }; ``` | Call | Description | |------|-------------| | `Result> list()` | All mapped top-level windows. | | `Result> active()` | The focused window, or none. | | `Result> find(std::string_view text)` | First window whose title or appId contains `text` (case-insensitive). | | `Status focus(WindowId id)` | Focus it. On Hyprland this switches to its workspace. | ```cpp auto wm = alvin::WindowManager::create().value(); if (auto w = wm->find("firefox").value()) { wm->focus(w->id); input->clickAt(w->bounds.center()); // click 40px in from the top-left of the window input->clickAt({w->bounds.x + 40, w->bounds.y + 40}); } ``` --- ## Vision (OpenCV) `#include `, link `alvin::vision`. | Call | Description | |------|-------------| | `cv::Mat view(const Frame&)` | Zero-copy `CV_8UC4` view. The Frame must outlive the Mat. | | `cv::Mat toBgr(const Frame&)` | Deep copy as `CV_8UC3`, the layout most OpenCV code expects. | | `Result fromMat(const cv::Mat&)` | 8-bit gray, BGR or BGRA to a Frame. | | `Result loadImage(path)` | Load a PNG/JPG as BGR. | | `Status saveImage(const Frame&, path)` | Save as PNG/JPG/… (by extension). | | `Result> find(frame, needle, opts)` | Best template match at or above the threshold. | | `Result> findAll(frame, needle, opts)` | Every non-overlapping match, best first. | ```cpp struct Match { Rect rect; double score; }; // rect in FRAME PIXELS, score 0..1 struct MatchOptions { double threshold = 0.9; // lower it (0.7–0.85) if the target is anti-aliased or slightly different bool grayscale = false; // faster; ignores colour int maxResults = 16; // findAll only }; ``` Take template images from Alvin's own captures, e.g. `alvin capture` followed by a crop. That keeps the scale and colours identical, which template matching needs. ```cpp auto needle = alvin::vision::loadImage("save_button.png").value(); auto frame = cap->grab().value(); if (auto hit = alvin::vision::find(frame, needle, {.threshold = 0.85}).value()) input->clickAt(frame.toDesktop(hit->rect.center())); for (const auto& m : alvin::vision::findAll(frame, needle, {.threshold = 0.8}).value()) std::printf("match at %d,%d score %.2f\n", m.rect.x, m.rect.y, m.score); ``` For anything else OpenCV can do, use `vision::view(frame)` to get a `cv::Mat`. --- ## Putting it together A small agent loop: wait for a button to appear, click it, then check that the screen changed. ```cpp #include #include #include #include #include using namespace std::chrono_literals; int main() { auto input = alvin::Input::create(); if (!input) { std::cerr << input.error().message << '\n'; return 1; } auto cap = alvin::Capture::create(); // the picker appears on first run if (!cap) { std::cerr << cap.error().message << '\n'; return 1; } auto button = alvin::vision::loadImage("accept.png"); if (!button) { std::cerr << button.error().message << '\n'; return 1; } for (int attempt = 0; attempt < 30; ++attempt) { auto frame = (*cap)->grab(); if (!frame) { std::cerr << frame.error().message << '\n'; return 1; } auto hit = alvin::vision::find(*frame, *button, {.threshold = 0.85}); if (hit && *hit) { (void)(*input)->clickAt(frame->toDesktop((*hit)->rect.center())); if ((*cap)->nextFrame(2s)) std::cout << "clicked; screen updated\n"; return 0; } std::this_thread::sleep_for(1s); } std::cerr << "button never appeared\n"; return 1; } ``` See [`examples/alvin_cli.cpp`](../examples/alvin_cli.cpp) for a complete program that uses every part of the API.