Alvin/docs/API.md

Alvin API Reference

Everything public lives in namespace alvin (vision helpers in alvin::vision). Include the umbrella header for the core library:

#include <alvin/alvin.hpp>      // Input, Capture, WindowManager, Frame, keys, types
#include <alvin/vision.hpp>     // optional, needs the alvin::vision target (OpenCV)

• Using Alvin in your project
• Error handling
• Coordinates and monitors
• Input
• Keys and chords
• Capture
• Frame
• Windows
• Vision (OpenCV)
• Putting it together

---

Using Alvin in your project

Either build Alvin as part of your project, or install it once and use find_package. The README's "Using Alvin in your project" section has step-by-step instructions for both.

add_subdirectory(external/Alvin)        # option 1: build from source
# find_package(Alvin 0.1 REQUIRED)      # option 2: use an installed Alvin

target_link_libraries(my_agent PRIVATE alvin::alvin)   # core
target_link_libraries(my_agent PRIVATE alvin::vision)  # + OpenCV helpers, if built

Alvin requires C++23 (it uses std::expected).

---

Error handling

Alvin never throws. Every call that can fail returns a Result<T>, which is std::expected<T, alvin::Error>. Calls with no value return Status (Result<void>).

struct Error {
    ErrorCode code;
    std::string message;   // human-readable detail
};

enum class ErrorCode {
    Unknown, Unsupported, NotAvailable, PermissionDenied, Cancelled,
    Timeout, InvalidArgument, NotFound, Io, Protocol,
};

std::string_view toString(ErrorCode code);

Code	Typical cause
Unsupported	Backend not compiled in, or the feature isn't possible on this platform
NotAvailable	Backend compiled in but its service is missing (no Wayland display, no portal interface)
PermissionDenied	/dev/uinput not writable, portal request denied, Windows UIPI block
Cancelled	The user dismissed a portal dialog
Timeout	No answer from a portal dialog, or no frame arrived in time
NotFound	A window or monitor that doesn't exist

Usage patterns:

auto input = alvin::Input::create();
if (!input) {
    std::cerr << alvin::toString(input.error().code) << ": " << input.error().message << '\n';
    return 1;
}
(*input)->moveTo({100, 100});        // Status; check it or ignore it

if (auto s = (*input)->click(); !s)  // check a Status
    std::cerr << s.error().message << '\n';

auto in = alvin::Input::create().value();   // or throw std::bad_expected_access on failure

---

Coordinates and monitors

#include <alvin/types.hpp>

All positions are global desktop coordinates:

• Linux: logical pixels, after monitor scaling. These are the numbers hyprctl reports.
• Windows: physical pixels. Alvin makes the process per-monitor DPI aware.

struct Point { int x, y; };
struct Size  { int width, height; };

struct Rect {
    int x, y, width, height;
    int right() const;               // x + width
    int bottom() const;              // y + height
    Point center() const;
    bool empty() const;
    bool contains(Point p) const;
    Rect united(const Rect& o) const;   // bounding box of both
};

struct Monitor {
    std::string name;          // "DP-2", "\\.\DISPLAY1"
    std::string description;   // make/model
    Rect bounds;               // position and size on the desktop
    Size pixelSize;            // actual mode, e.g. 3840x2160
    double scale;              // pixelSize / logical size, e.g. 1.333
    bool primary;
};

Result<std::vector<Monitor>> listMonitors();
Result<Rect> desktopBounds();      // bounding box of all monitors

for (const auto& m : alvin::listMonitors().value())
    std::printf("%s at %d,%d (%dx%d logical, scale %.2f)\n",
                m.name.c_str(), m.bounds.x, m.bounds.y, m.bounds.width, m.bounds.height, m.scale);

---

Input

#include <alvin/input.hpp>

Creating

static Result<std::unique_ptr<Input>> Input::create(const InputOptions& options = {});

struct InputOptions {
    InputBackend backend = InputBackend::Auto;
    std::string restoreToken;      // libei only: skip the portal dialog (see restoreToken())
    std::chrono::milliseconds permissionTimeout = std::chrono::minutes(2);
};

enum class InputBackend { Auto, WlrVirtual, LibEi, UInput, SendInput };

Backend	Platform	Notes
WlrVirtual	Hyprland, sway, wlroots	No dialog. Types any Unicode text.
LibEi	GNOME, KDE (RemoteDesktop portal)	Shows a permission dialog unless given a restore token.
UInput	Any Linux session	Needs write access to /dev/uinput. Takes about 2.5 s to create on Hyprland.
SendInput	Windows	Can't inject into elevated windows unless Alvin runs elevated too.

With Auto, Linux tries WlrVirtual, then LibEi, then UInput. You can force a backend with the environment variable ALVIN_INPUT_BACKEND=wlr|libei|uinput (Linux only).

Create one Input and keep it for the life of your program. Creation is the expensive part (connecting to the compositor, creating devices, portal dialogs).

Mouse

Call	Description
Status moveTo(Point p)	Move the pointer to an absolute desktop position.
Status moveBy(int dx, int dy)	Relative motion. With uinput, the compositor applies pointer acceleration.
Status buttonDown(MouseButton b) / buttonUp(b)	Press or release a button.
Status click(MouseButton b = Left, int count = 1)	Click in place. count = 2 double-clicks.
Status clickAt(Point p, MouseButton b = Left, int count = 1)	Move, then click.
Status drag(Point from, Point to, MouseButton b = Left)	Press at from, move to to in 10 steps, release.
Status scroll(int dx, int dy)	Wheel notches. Positive dy scrolls down, positive dx scrolls right.

MouseButton is Left, Right, Middle, Back or Forward.

input->clickAt({1440, 810});                         // left click
input->clickAt({1440, 810}, alvin::MouseButton::Right);
input->click(alvin::MouseButton::Left, 2);           // double-click in place
input->drag({100, 100}, {600, 400});                 // e.g. select or move something
input->scroll(0, 3);                                 // 3 notches down

Keyboard

Call	Description
Status keyDown(Key k) / keyUp(Key k)	Press or release one physical key.
Status keyPress(Key k)	Down, then up.
Status chord(std::initializer_list<Key>)	Press in order, release in reverse.
Status chord(std::string_view)	Same, from text like "ctrl+shift+t".
Status typeText(std::string_view utf8)	Type text. \n presses Enter and \t presses Tab.

using alvin::Key;
input->keyPress(Key::Enter);
input->chord({Key::LeftCtrl, Key::C});
input->chord("ctrl+shift+t");
input->chord("super+enter");
input->typeText("Hello, wörld ✓\n");

chord() always releases every key it pressed, even when an earlier call fails, so a key is never left held down.

How typeText handles layouts:

• wlr: builds a keymap containing exactly the characters needed, so any Unicode text works.
• libei: uses the server's keyboard layout. If the server supports libei's text capability, it sends the text directly.
• uinput: uses the layout named in XKB_DEFAULT_LAYOUT/XKB_DEFAULT_VARIANT (default us). Characters that aren't on that layout fail with Unsupported before anything is typed.
• sendinput: sends Unicode directly, so any text works.

Other

Call	Description
InputBackend backend() const	Which backend was chosen.
std::string restoreToken() const	libei: save this and pass it in InputOptions::restoreToken next run to skip the dialog. Empty for other backends.
void setEventDelay(std::chrono::milliseconds)	Pause between the down and up halves of clicks, key presses and typed characters. The default is 8 ms. Raise it for applications that miss fast input.
std::string_view toString(InputBackend)	"wlr", "libei", "uinput", "sendinput", "auto"

---

Keys and chords

#include <alvin/keys.hpp>

Key names physical key positions on a US QWERTY layout, the same way evdev and Win32 scan codes do. Use typeText() when you want particular characters.

A–Z, Num0–Num9, F1–F24,
Escape, Enter, Tab, Backspace, Space,
Minus, Equal, LeftBracket, RightBracket, Backslash, Semicolon, Apostrophe, Grave, Comma, Period, Slash,
CapsLock, NumLock, ScrollLock,
LeftShift, RightShift, LeftCtrl, RightCtrl, LeftAlt, RightAlt, LeftSuper, RightSuper, Menu,
Insert, Delete, Home, End, PageUp, PageDown, Left, Right, Up, Down, PrintScreen, Pause,
Kp0–Kp9, KpDecimal, KpDivide, KpMultiply, KpSubtract, KpAdd, KpEnter,
VolumeMute, VolumeDown, VolumeUp, MediaPlayPause, MediaStop, MediaNext, MediaPrevious

std::optional<Key> keyFromName(std::string_view name);            // case-insensitive
std::string_view keyName(Key key);
std::optional<std::vector<Key>> parseChord(std::string_view chord);
bool isModifier(Key key);

Names accepted by keyFromName and in chord strings:

Key	Names
Letters and digits	a…z, 0…9
Function keys	f1…f24
Modifiers	shift/lshift, rshift, ctrl/control/lctrl, rctrl, alt/lalt, ralt/altgr, super/win/meta/cmd/lsuper, rsuper
Editing	enter/return, esc/escape, tab, backspace, space, insert, delete/del
Navigation	home, end, pageup/pgup, pagedown/pgdn, left, right, up, down
Punctuation	minus/-, equal/=, [, ], \, ;, ', `, ,, ., / (or their word names)
Keypad	kp0…kp9, kpdecimal, kpdivide, kpmultiply, kpsubtract, kpadd, kpenter
Media	mute, volumeup, volumedown, playpause, stop, next, prev/previous
Other	capslock, numlock, scrolllock, printscreen/print, pause, menu

Chords are joined with +, and spaces around keys are ignored. "ctrl++" means Ctrl plus the +/= key.

---

Capture

#include <alvin/capture.hpp>

Creating

static Result<std::unique_ptr<Capture>> Capture::create(const CaptureOptions& options = {});

struct CaptureOptions {
    CaptureBackend backend = CaptureBackend::Auto;   // Auto, Portal, Gdi
    CaptureSource source = CaptureSource::Monitor;   // Monitor, Window, Any
    int monitorIndex = 0;                            // Windows: which monitor (index into listMonitors())
    std::optional<WindowId> window;                  // Windows: capture this window directly
    bool showCursor = false;
    std::string restoreToken;                        // Portal: skip the picker (see below)
    std::chrono::milliseconds permissionTimeout = std::chrono::minutes(2);
};

Linux (Portal): create() opens the desktop's share picker. On Hyprland that's hyprland-share-picker. source controls whether the picker offers monitors, windows or both. When the call returns, a PipeWire stream is running in the background.

Windows (Gdi): there's no dialog. Set monitorIndex, or set window to capture one window.

Skipping the picker next time

alvin::CaptureOptions opts;
opts.restoreToken = loadSavedToken();            // empty the first time
auto cap = alvin::Capture::create(opts).value();
saveToken(cap->restoreToken());                  // new token each session; always re-save it

The alvin CLI does this for you and stores tokens in ~/.cache/alvin/.

Don't leave a picker unanswered. While it's open, xdg-desktop-portal-hyprland blocks every other portal request on the system, screen sharing in other apps included. Use permissionTimeout to limit how long create() waits.

Getting frames

Call	Description
Result<Frame> grab(timeout = 5s)	The most recent frame. Blocks only until the first frame exists.
Result<Frame> nextFrame(timeout = 5s)	Waits for a frame newer than the last one you received.
std::string restoreToken() const	Portal token for next time.
CaptureBackend backend() const	Which backend was chosen.

PipeWire streams only send a frame when the screen changes, so grab() can return an image that's a few seconds old if nothing has moved. That's usually what you want. After you act (click, type), call nextFrame() to see the result:

input->clickAt(button);
auto after = cap->nextFrame(std::chrono::seconds(2));
if (!after && after.error().code == alvin::ErrorCode::Timeout) {
    // nothing on screen changed
}

A Capture holds a live stream, so keep one and reuse it. Don't create one per frame.

---

Frame

#include <alvin/frame.hpp>

A Frame is an owned BGRA8 image (4 bytes per pixel: B, G, R, A, with no row padding). It also records where it sits on the desktop.

Member	Description
Frame(int w, int h)	New zero-filled frame.
int width(), int height(), Size size()	Size in pixels.
std::size_t stride()	Bytes per row (width * 4).
uint8_t* data(), row(y), std::span<const uint8_t> bytes()	Raw pixel access.
Color pixel(int x, int y)	{r, g, b, a}. Returns all zeros if out of bounds.
Rect desktopRect()	The area of the desktop this frame shows, in desktop coordinates.
Point toDesktop(Point px)	Frame pixel to desktop coordinate.
Point fromDesktop(Point p)	Desktop coordinate to frame pixel.
Frame crop(Rect r)	Sub-image (r is in frame pixels and gets clipped). desktopRect() is adjusted to match.
int64_t timestampNs()	Capture timestamp.
Status savePpm(path)	Save without dependencies (PPM). With vision, use vision::saveImage for PNG/JPG.

Always convert through toDesktop() before clicking. On a monitor scaled to 1.33, a 3840×2160 frame covers 2880×1620 desktop units, so frame pixels aren't desktop coordinates:

auto frame = cap->grab().value();
alvin::Point pixel{1920, 1080};                  // something found in the image
input->clickAt(frame.toDesktop(pixel));          // correct on any scale

alvin::Color c = frame.pixel(10, 10);
if (c.r > 200 && c.g < 50 && c.b < 50) { /* red */ }

---

Windows

#include <alvin/window.hpp>

Wayland hides other windows from normal clients, so each compositor needs its own backend. Right now that's Hyprland (IPC socket) and Win32.

static Result<std::unique_ptr<WindowManager>> WindowManager::create(WindowBackend b = WindowBackend::Auto);

struct WindowInfo {
    WindowId id;              // opaque handle; pass it to focus()
    std::string title;
    std::string appId;        // Hyprland class / Win32 executable name
    Rect bounds;              // desktop coordinates
    int64_t pid;
    bool focused, visible, minimized, fullscreen;
    std::string workspace;    // Hyprland workspace name
};

Call	Description
Result<std::vector<WindowInfo>> list()	All mapped top-level windows.
Result<std::optional<WindowInfo>> active()	The focused window, or none.
Result<std::optional<WindowInfo>> find(std::string_view text)	First window whose title or appId contains text (case-insensitive).
Status focus(WindowId id)	Focus it. On Hyprland this switches to its workspace.

auto wm = alvin::WindowManager::create().value();
if (auto w = wm->find("firefox").value()) {
    wm->focus(w->id);
    input->clickAt(w->bounds.center());
    // click 40px in from the top-left of the window
    input->clickAt({w->bounds.x + 40, w->bounds.y + 40});
}

---

Vision (OpenCV)

#include <alvin/vision.hpp>, link alvin::vision.

Call	Description
cv::Mat view(const Frame&)	Zero-copy CV_8UC4 view. The Frame must outlive the Mat.
cv::Mat toBgr(const Frame&)	Deep copy as CV_8UC3, the layout most OpenCV code expects.
Result<Frame> fromMat(const cv::Mat&)	8-bit gray, BGR or BGRA to a Frame.
Result<cv::Mat> loadImage(path)	Load a PNG/JPG as BGR.
Status saveImage(const Frame&, path)	Save as PNG/JPG/… (by extension).
Result<std::optional<Match>> find(frame, needle, opts)	Best template match at or above the threshold.
Result<std::vector<Match>> findAll(frame, needle, opts)	Every non-overlapping match, best first.

struct Match { Rect rect; double score; };     // rect in FRAME PIXELS, score 0..1

struct MatchOptions {
    double threshold = 0.9;    // lower it (0.7–0.85) if the target is anti-aliased or slightly different
    bool grayscale = false;    // faster; ignores colour
    int maxResults = 16;       // findAll only
};

Take template images from Alvin's own captures, e.g. alvin capture followed by a crop. That keeps the scale and colours identical, which template matching needs.

auto needle = alvin::vision::loadImage("save_button.png").value();
auto frame  = cap->grab().value();
if (auto hit = alvin::vision::find(frame, needle, {.threshold = 0.85}).value())
    input->clickAt(frame.toDesktop(hit->rect.center()));

for (const auto& m : alvin::vision::findAll(frame, needle, {.threshold = 0.8}).value())
    std::printf("match at %d,%d score %.2f\n", m.rect.x, m.rect.y, m.score);

For anything else OpenCV can do, use vision::view(frame) to get a cv::Mat.

---

Putting it together

A small agent loop: wait for a button to appear, click it, then check that the screen changed.

#include <alvin/alvin.hpp>
#include <alvin/vision.hpp>
#include <chrono>
#include <iostream>
#include <thread>

using namespace std::chrono_literals;

int main() {
    auto input = alvin::Input::create();
    if (!input) { std::cerr << input.error().message << '\n'; return 1; }

    auto cap = alvin::Capture::create();          // the picker appears on first run
    if (!cap) { std::cerr << cap.error().message << '\n'; return 1; }

    auto button = alvin::vision::loadImage("accept.png");
    if (!button) { std::cerr << button.error().message << '\n'; return 1; }

    for (int attempt = 0; attempt < 30; ++attempt) {
        auto frame = (*cap)->grab();
        if (!frame) { std::cerr << frame.error().message << '\n'; return 1; }

        auto hit = alvin::vision::find(*frame, *button, {.threshold = 0.85});
        if (hit && *hit) {
            (void)(*input)->clickAt(frame->toDesktop((*hit)->rect.center()));
            if ((*cap)->nextFrame(2s)) std::cout << "clicked; screen updated\n";
            return 0;
        }
        std::this_thread::sleep_for(1s);
    }
    std::cerr << "button never appeared\n";
    return 1;
}

See examples/alvin_cli.cpp for a complete program that uses every part of the API.