Alvin/docs/API.md

# Alvin API Reference

Everything public lives in namespace `alvin` (vision helpers in `alvin::vision`).
Include the umbrella header for the core library:

```cpp
#include <alvin/alvin.hpp>      // Input, Capture, WindowManager, Frame, keys, types
#include <alvin/vision.hpp>     // optional, needs the alvin::vision target (OpenCV)
```

- [Using Alvin in your project](#using-alvin-in-your-project)
- [Error handling](#error-handling)
- [Coordinates and monitors](#coordinates-and-monitors)
- [Input](#input)
- [Keys and chords](#keys-and-chords)
- [Capture](#capture)
- [Frame](#frame)
- [Windows](#windows)
- [Vision (OpenCV)](#vision-opencv)
- [Putting it together](#putting-it-together)

---

## Using Alvin in your project

Either build Alvin as part of your project, or install it once and use `find_package`.
The README's "Using Alvin in your project" section has step-by-step instructions for both.

```cmake
add_subdirectory(external/Alvin)        # option 1: build from source
# find_package(Alvin 0.1 REQUIRED)      # option 2: use an installed Alvin

target_link_libraries(my_agent PRIVATE alvin::alvin)   # core
target_link_libraries(my_agent PRIVATE alvin::vision)  # + OpenCV helpers, if built
```

Alvin requires C++23 (it uses `std::expected`).

---

## Error handling

Alvin never throws. Every call that can fail returns a `Result<T>`, which is
`std::expected<T, alvin::Error>`. Calls with no value return `Status`
(`Result<void>`).

```cpp
struct Error {
    ErrorCode code;
    std::string message;   // human-readable detail
};

enum class ErrorCode {
    Unknown, Unsupported, NotAvailable, PermissionDenied, Cancelled,
    Timeout, InvalidArgument, NotFound, Io, Protocol,
};

std::string_view toString(ErrorCode code);
```

| Code | Typical cause |
|------|---------------|
| `Unsupported` | Backend not compiled in, or the feature isn't possible on this platform |
| `NotAvailable` | Backend compiled in but its service is missing (no Wayland display, no portal interface) |
| `PermissionDenied` | `/dev/uinput` not writable, portal request denied, Windows UIPI block |
| `Cancelled` | The user dismissed a portal dialog |
| `Timeout` | No answer from a portal dialog, or no frame arrived in time |
| `NotFound` | A window or monitor that doesn't exist |

Usage patterns:

```cpp
auto input = alvin::Input::create();
if (!input) {
    std::cerr << alvin::toString(input.error().code) << ": " << input.error().message << '\n';
    return 1;
}
(*input)->moveTo({100, 100});        // Status; check it or ignore it

if (auto s = (*input)->click(); !s)  // check a Status
    std::cerr << s.error().message << '\n';

auto in = alvin::Input::create().value();   // or throw std::bad_expected_access on failure
```

---

## Coordinates and monitors

`#include <alvin/types.hpp>`

All positions are **global desktop coordinates**:

- **Linux:** logical pixels, after monitor scaling. These are the numbers `hyprctl` reports.
- **Windows:** physical pixels. Alvin makes the process per-monitor DPI aware.

```cpp
struct Point { int x, y; };
struct Size  { int width, height; };

struct Rect {
    int x, y, width, height;
    int right() const;               // x + width
    int bottom() const;              // y + height
    Point center() const;
    bool empty() const;
    bool contains(Point p) const;
    Rect united(const Rect& o) const;   // bounding box of both
};

struct Monitor {
    std::string name;          // "DP-2", "\\.\DISPLAY1"
    std::string description;   // make/model
    Rect bounds;               // position and size on the desktop
    Size pixelSize;            // actual mode, e.g. 3840x2160
    double scale;              // pixelSize / logical size, e.g. 1.333
    bool primary;
};

Result<std::vector<Monitor>> listMonitors();
Result<Rect> desktopBounds();      // bounding box of all monitors
```

```cpp
for (const auto& m : alvin::listMonitors().value())
    std::printf("%s at %d,%d (%dx%d logical, scale %.2f)\n",
                m.name.c_str(), m.bounds.x, m.bounds.y, m.bounds.width, m.bounds.height, m.scale);
```

---

## Input

`#include <alvin/input.hpp>`

### Creating

```cpp
static Result<std::unique_ptr<Input>> Input::create(const InputOptions& options = {});

struct InputOptions {
    InputBackend backend = InputBackend::Auto;
    std::string restoreToken;      // libei only: skip the portal dialog (see restoreToken())
    std::chrono::milliseconds permissionTimeout = std::chrono::minutes(2);
};

enum class InputBackend { Auto, WlrVirtual, LibEi, UInput, SendInput };
```

| Backend | Platform | Notes |
|---------|----------|-------|
| `WlrVirtual` | Hyprland, sway, wlroots | No dialog. Types any Unicode text. |
| `LibEi` | GNOME, KDE (RemoteDesktop portal) | Shows a permission dialog unless given a restore token. |
| `UInput` | Any Linux session | Needs write access to `/dev/uinput`. Takes about 2.5 s to create on Hyprland. |
| `SendInput` | Windows | Can't inject into elevated windows unless Alvin runs elevated too. |

With `Auto`, Linux tries `WlrVirtual`, then `LibEi`, then `UInput`. You can force a
backend with the environment variable `ALVIN_INPUT_BACKEND=wlr|libei|uinput` (Linux only).

Create one `Input` and keep it for the life of your program. Creation is the
expensive part (connecting to the compositor, creating devices, portal dialogs).

### Mouse

| Call | Description |
|------|-------------|
| `Status moveTo(Point p)` | Move the pointer to an absolute desktop position. |
| `Status moveBy(int dx, int dy)` | Relative motion. With `uinput`, the compositor applies pointer acceleration. |
| `Status buttonDown(MouseButton b)` / `buttonUp(b)` | Press or release a button. |
| `Status click(MouseButton b = Left, int count = 1)` | Click in place. `count = 2` double-clicks. |
| `Status clickAt(Point p, MouseButton b = Left, int count = 1)` | Move, then click. |
| `Status drag(Point from, Point to, MouseButton b = Left)` | Press at `from`, move to `to` in 10 steps, release. |
| `Status scroll(int dx, int dy)` | Wheel notches. Positive `dy` scrolls **down**, positive `dx` scrolls **right**. |

`MouseButton` is `Left`, `Right`, `Middle`, `Back` or `Forward`.

```cpp
input->clickAt({1440, 810});                         // left click
input->clickAt({1440, 810}, alvin::MouseButton::Right);
input->click(alvin::MouseButton::Left, 2);           // double-click in place
input->drag({100, 100}, {600, 400});                 // e.g. select or move something
input->scroll(0, 3);                                 // 3 notches down
```

### Keyboard

| Call | Description |
|------|-------------|
| `Status keyDown(Key k)` / `keyUp(Key k)` | Press or release one physical key. |
| `Status keyPress(Key k)` | Down, then up. |
| `Status chord(std::initializer_list<Key>)` | Press in order, release in reverse. |
| `Status chord(std::string_view)` | Same, from text like `"ctrl+shift+t"`. |
| `Status typeText(std::string_view utf8)` | Type text. `\n` presses Enter and `\t` presses Tab. |

```cpp
using alvin::Key;
input->keyPress(Key::Enter);
input->chord({Key::LeftCtrl, Key::C});
input->chord("ctrl+shift+t");
input->chord("super+enter");
input->typeText("Hello, wörld ✓\n");
```

`chord()` always releases every key it pressed, even when an earlier call fails,
so a key is never left held down.

**How `typeText` handles layouts:**

- **`wlr`:** builds a keymap containing exactly the characters needed, so any Unicode text works.
- **`libei`:** uses the server's keyboard layout. If the server supports libei's text capability, it sends the text directly.
- **`uinput`:** uses the layout named in `XKB_DEFAULT_LAYOUT`/`XKB_DEFAULT_VARIANT` (default `us`). Characters that aren't on that layout fail with `Unsupported` before anything is typed.
- **`sendinput`:** sends Unicode directly, so any text works.

### Other

| Call | Description |
|------|-------------|
| `InputBackend backend() const` | Which backend was chosen. |
| `std::string restoreToken() const` | libei: save this and pass it in `InputOptions::restoreToken` next run to skip the dialog. Empty for other backends. |
| `void setEventDelay(std::chrono::milliseconds)` | Pause between the down and up halves of clicks, key presses and typed characters. The default is 8 ms. Raise it for applications that miss fast input. |
| `std::string_view toString(InputBackend)` | `"wlr"`, `"libei"`, `"uinput"`, `"sendinput"`, `"auto"` |

---

## Keys and chords

`#include <alvin/keys.hpp>`

`Key` names **physical key positions** on a US QWERTY layout, the same way evdev
and Win32 scan codes do. Use `typeText()` when you want particular characters.

```
A–Z, Num0–Num9, F1–F24,
Escape, Enter, Tab, Backspace, Space,
Minus, Equal, LeftBracket, RightBracket, Backslash, Semicolon, Apostrophe, Grave, Comma, Period, Slash,
CapsLock, NumLock, ScrollLock,
LeftShift, RightShift, LeftCtrl, RightCtrl, LeftAlt, RightAlt, LeftSuper, RightSuper, Menu,
Insert, Delete, Home, End, PageUp, PageDown, Left, Right, Up, Down, PrintScreen, Pause,
Kp0–Kp9, KpDecimal, KpDivide, KpMultiply, KpSubtract, KpAdd, KpEnter,
VolumeMute, VolumeDown, VolumeUp, MediaPlayPause, MediaStop, MediaNext, MediaPrevious
```

```cpp
std::optional<Key> keyFromName(std::string_view name);            // case-insensitive
std::string_view keyName(Key key);
std::optional<std::vector<Key>> parseChord(std::string_view chord);
bool isModifier(Key key);
```

Names accepted by `keyFromName` and in chord strings:

| Key | Names |
|-----|-------|
| Letters and digits | `a`…`z`, `0`…`9` |
| Function keys | `f1`…`f24` |
| Modifiers | `shift`/`lshift`, `rshift`, `ctrl`/`control`/`lctrl`, `rctrl`, `alt`/`lalt`, `ralt`/`altgr`, `super`/`win`/`meta`/`cmd`/`lsuper`, `rsuper` |
| Editing | `enter`/`return`, `esc`/`escape`, `tab`, `backspace`, `space`, `insert`, `delete`/`del` |
| Navigation | `home`, `end`, `pageup`/`pgup`, `pagedown`/`pgdn`, `left`, `right`, `up`, `down` |
| Punctuation | `minus`/`-`, `equal`/`=`, `[`, `]`, `\`, `;`, `'`, `` ` ``, `,`, `.`, `/` (or their word names) |
| Keypad | `kp0`…`kp9`, `kpdecimal`, `kpdivide`, `kpmultiply`, `kpsubtract`, `kpadd`, `kpenter` |
| Media | `mute`, `volumeup`, `volumedown`, `playpause`, `stop`, `next`, `prev`/`previous` |
| Other | `capslock`, `numlock`, `scrolllock`, `printscreen`/`print`, `pause`, `menu` |

Chords are joined with `+`, and spaces around keys are ignored. `"ctrl++"` means
Ctrl plus the `+`/`=` key.

---

## Capture

`#include <alvin/capture.hpp>`

### Creating

```cpp
static Result<std::unique_ptr<Capture>> Capture::create(const CaptureOptions& options = {});

struct CaptureOptions {
    CaptureBackend backend = CaptureBackend::Auto;   // Auto, Portal, Gdi
    CaptureSource source = CaptureSource::Monitor;   // Monitor, Window, Any
    int monitorIndex = 0;                            // Windows: which monitor (index into listMonitors())
    std::optional<WindowId> window;                  // Windows: capture this window directly
    bool showCursor = false;
    std::string restoreToken;                        // Portal: skip the picker (see below)
    std::chrono::milliseconds permissionTimeout = std::chrono::minutes(2);
};
```

**Linux (`Portal`):** `create()` opens the desktop's share picker. On Hyprland
that's `hyprland-share-picker`. `source` controls whether the picker offers
monitors, windows or both. When the call returns, a PipeWire stream is running
in the background.

**Windows (`Gdi`):** there's no dialog. Set `monitorIndex`, or set `window` to
capture one window.

### Skipping the picker next time

```cpp
alvin::CaptureOptions opts;
opts.restoreToken = loadSavedToken();            // empty the first time
auto cap = alvin::Capture::create(opts).value();
saveToken(cap->restoreToken());                  // new token each session; always re-save it
```

The `alvin` CLI does this for you and stores tokens in `~/.cache/alvin/`.

> Don't leave a picker unanswered. While it's open, xdg-desktop-portal-hyprland
> blocks every other portal request on the system, screen sharing in other
> apps included. Use `permissionTimeout` to limit how long `create()` waits.

### Getting frames

| Call | Description |
|------|-------------|
| `Result<Frame> grab(timeout = 5s)` | The most recent frame. Blocks only until the first frame exists. |
| `Result<Frame> nextFrame(timeout = 5s)` | Waits for a frame newer than the last one you received. |
| `std::string restoreToken() const` | Portal token for next time. |
| `CaptureBackend backend() const` | Which backend was chosen. |

PipeWire streams only send a frame when the screen changes, so `grab()` can
return an image that's a few seconds old if nothing has moved. That's usually
what you want. After you act (click, type), call `nextFrame()` to see the result:

```cpp
input->clickAt(button);
auto after = cap->nextFrame(std::chrono::seconds(2));
if (!after && after.error().code == alvin::ErrorCode::Timeout) {
    // nothing on screen changed
}
```

A `Capture` holds a live stream, so keep one and reuse it. Don't create one per frame.

---

## Frame

`#include <alvin/frame.hpp>`

A `Frame` is an owned BGRA8 image (4 bytes per pixel: B, G, R, A, with no row
padding). It also records where it sits on the desktop.

| Member | Description |
|--------|-------------|
| `Frame(int w, int h)` | New zero-filled frame. |
| `int width()`, `int height()`, `Size size()` | Size in **pixels**. |
| `std::size_t stride()` | Bytes per row (`width * 4`). |
| `uint8_t* data()`, `row(y)`, `std::span<const uint8_t> bytes()` | Raw pixel access. |
| `Color pixel(int x, int y)` | `{r, g, b, a}`. Returns all zeros if out of bounds. |
| `Rect desktopRect()` | The area of the desktop this frame shows, in desktop coordinates. |
| `Point toDesktop(Point px)` | Frame pixel to desktop coordinate. |
| `Point fromDesktop(Point p)` | Desktop coordinate to frame pixel. |
| `Frame crop(Rect r)` | Sub-image (`r` is in frame pixels and gets clipped). `desktopRect()` is adjusted to match. |
| `int64_t timestampNs()` | Capture timestamp. |
| `Status savePpm(path)` | Save without dependencies (PPM). With vision, use `vision::saveImage` for PNG/JPG. |

**Always convert through `toDesktop()` before clicking.** On a monitor scaled
to 1.33, a 3840×2160 frame covers 2880×1620 desktop units, so frame pixels
aren't desktop coordinates:

```cpp
auto frame = cap->grab().value();
alvin::Point pixel{1920, 1080};                  // something found in the image
input->clickAt(frame.toDesktop(pixel));          // correct on any scale
```

```cpp
alvin::Color c = frame.pixel(10, 10);
if (c.r > 200 && c.g < 50 && c.b < 50) { /* red */ }
```

---

## Windows

`#include <alvin/window.hpp>`

Wayland hides other windows from normal clients, so each compositor needs its
own backend. Right now that's Hyprland (IPC socket) and Win32.

```cpp
static Result<std::unique_ptr<WindowManager>> WindowManager::create(WindowBackend b = WindowBackend::Auto);

struct WindowInfo {
    WindowId id;              // opaque handle; pass it to focus()
    std::string title;
    std::string appId;        // Hyprland class / Win32 executable name
    Rect bounds;              // desktop coordinates
    int64_t pid;
    bool focused, visible, minimized, fullscreen;
    std::string workspace;    // Hyprland workspace name
};
```

| Call | Description |
|------|-------------|
| `Result<std::vector<WindowInfo>> list()` | All mapped top-level windows. |
| `Result<std::optional<WindowInfo>> active()` | The focused window, or none. |
| `Result<std::optional<WindowInfo>> find(std::string_view text)` | First window whose title or appId contains `text` (case-insensitive). |
| `Status focus(WindowId id)` | Focus it. On Hyprland this switches to its workspace. |

```cpp
auto wm = alvin::WindowManager::create().value();
if (auto w = wm->find("firefox").value()) {
    wm->focus(w->id);
    input->clickAt(w->bounds.center());
    // click 40px in from the top-left of the window
    input->clickAt({w->bounds.x + 40, w->bounds.y + 40});
}
```

---

## Vision (OpenCV)

`#include <alvin/vision.hpp>`, link `alvin::vision`.

| Call | Description |
|------|-------------|
| `cv::Mat view(const Frame&)` | Zero-copy `CV_8UC4` view. The Frame must outlive the Mat. |
| `cv::Mat toBgr(const Frame&)` | Deep copy as `CV_8UC3`, the layout most OpenCV code expects. |
| `Result<Frame> fromMat(const cv::Mat&)` | 8-bit gray, BGR or BGRA to a Frame. |
| `Result<cv::Mat> loadImage(path)` | Load a PNG/JPG as BGR. |
| `Status saveImage(const Frame&, path)` | Save as PNG/JPG/… (by extension). |
| `Result<std::optional<Match>> find(frame, needle, opts)` | Best template match at or above the threshold. |
| `Result<std::vector<Match>> findAll(frame, needle, opts)` | Every non-overlapping match, best first. |

```cpp
struct Match { Rect rect; double score; };     // rect in FRAME PIXELS, score 0..1

struct MatchOptions {
    double threshold = 0.9;    // lower it (0.7–0.85) if the target is anti-aliased or slightly different
    bool grayscale = false;    // faster; ignores colour
    int maxResults = 16;       // findAll only
};
```

Take template images from Alvin's own captures, e.g. `alvin capture` followed by
a crop. That keeps the scale and colours identical, which template matching needs.

```cpp
auto needle = alvin::vision::loadImage("save_button.png").value();
auto frame  = cap->grab().value();
if (auto hit = alvin::vision::find(frame, needle, {.threshold = 0.85}).value())
    input->clickAt(frame.toDesktop(hit->rect.center()));

for (const auto& m : alvin::vision::findAll(frame, needle, {.threshold = 0.8}).value())
    std::printf("match at %d,%d score %.2f\n", m.rect.x, m.rect.y, m.score);
```

For anything else OpenCV can do, use `vision::view(frame)` to get a `cv::Mat`.

---

## Putting it together

A small agent loop: wait for a button to appear, click it, then check that the
screen changed.

```cpp
#include <alvin/alvin.hpp>
#include <alvin/vision.hpp>
#include <chrono>
#include <iostream>
#include <thread>

using namespace std::chrono_literals;

int main() {
    auto input = alvin::Input::create();
    if (!input) { std::cerr << input.error().message << '\n'; return 1; }

    auto cap = alvin::Capture::create();          // the picker appears on first run
    if (!cap) { std::cerr << cap.error().message << '\n'; return 1; }

    auto button = alvin::vision::loadImage("accept.png");
    if (!button) { std::cerr << button.error().message << '\n'; return 1; }

    for (int attempt = 0; attempt < 30; ++attempt) {
        auto frame = (*cap)->grab();
        if (!frame) { std::cerr << frame.error().message << '\n'; return 1; }

        auto hit = alvin::vision::find(*frame, *button, {.threshold = 0.85});
        if (hit && *hit) {
            (void)(*input)->clickAt(frame->toDesktop((*hit)->rect.center()));
            if ((*cap)->nextFrame(2s)) std::cout << "clicked; screen updated\n";
            return 0;
        }
        std::this_thread::sleep_for(1s);
    }
    std::cerr << "button never appeared\n";
    return 1;
}
```

See [`examples/alvin_cli.cpp`](../examples/alvin_cli.cpp) for a complete program
that uses every part of the API.