Tine

tine

**Drive a GNOME Wayland desktop from AI agents. CLI-first, no portals, no consent dialogs.**

Tine is a command-line bridge between an AI coding agent (Claude Code, Codex, etc.) and a running Linux desktop. It reads the screen, walks the accessibility tree, and injects keyboard and mouse events at the kernel level — no Wayland portal dialogs, no per-action consent prompts, no X11 fallback hacks.

$ tine describe
[screenshot + AT-SPI2 tree summary]

$ tine click ref_17           # click by accessibility-tree ref
$ tine click B3               # click by labeled grid cell
$ tine type "hello, world"    # kernel-level key injection
$ tine key ctrl+t             # modifier combos
$ tine focus Firefox          # raise + focus a window

**Status:** alpha. Tested on GNOME 49 Wayland / Arch Linux. API may change before 1.0.

---

Why

Anthropic's computer-use feature works on Windows and macOS. If you use Linux — and especially if you use Wayland, which is more locked-down than X11 and breaks most of the existing Linux automation stack — you're mostly out of luck. Tine is an attempt at a usable Wayland alternative.

It reads the screen three ways:

• AT-SPI2 accessibility tree. When an app exposes its widgets to the Linux a11y stack (GTK apps, Qt apps, most native GNOME stuff), tine walks the tree and gives the agent structured data — roles, names, bounding boxes, actions. The agent can say click ref_17 and tine clicks the center of the button with that ref.
• Labeled coordinate grid. When AT-SPI2 is sparse or missing (Chrome, Electron, most web content, games), tine overlays a labeled grid on the screenshot and the agent says click B3. Not fancy, but it works.
• OCR text refs. Run OCR on the screenshot (RapidOCR, local, CPU) and get refs like ref_t3 tied to detected text regions. The agent can click by on-screen text — tine click ref_t3 clicks the center of the OCR region with that ref. Optional, lazy-loaded, install with `pip install