Vercel’s Open-Source Desktop Tool Gives AI Agents Structure

Desktop agents just got a map. Vercel Labs released agent-desktop, an open-source Rust CLI and C library that lets AI agents read and operate apps on macOS, Windows, and Linux through each operating system’s accessibility tree. It returns structured JSON snapshots instead of asking a model to stare at screenshots and guess where to click.

Version 1.0.2 shipped on October 30, 2026, under the Apache-2.0 license. The tool is not an agent itself: it embeds no LLM, has no GUI, and gives shell-capable coding agents a way to drive native apps without a vision model.

Every command prints one JSON envelope and exits, leaving the observe, act, and verify loop inside the agent. That split matters because agent-desktop handles desktop access while the agent decides what to do next — a tool, not another opinionated assistant demanding a seat at the table.

Structured access instead of screenshot guesswork

Each snapshot gives the agent a ref for every element it can act on. A qualified ref might look like @s8f3k2p9:e8, with the snapshot ID making it self-describing; a bare @e8 is rejected with INVALID_ARGS.

When the interface changes, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent takes a fresh snapshot. Ref resolution fails closed instead of guessing, while every error includes a disposition that tells the agent whether retrying is safe.

Dense apps can expose thousands of accessibility nodes, so agent-desktop includes a –skeleton snapshot that reads only three levels deep. Cut containers report a children_count and provide a drill ref; snapshot –root then reads one region into the same snapshot. The result is less data for the model and fewer opportunities for it to lose the plot.

Self-reported figures claim a reduction of 78% to 96% fewer tokens on dense Electron apps through skeleton traversal. One Apple Maps skeleton fell from 3,811 tokens per look to 273, while a simulated 52-step task dropped from 198k to 31k tokens.

Those figures are self-reported, but the underlying distinction is clear. Model-side computer-use tools work from screenshots and pixel coordinates; Anthropic’s docs put each screenshot at roughly 1,000 to 1,800 input tokens, while its toolset definition adds about 4,500 input tokens per request. OpenAI’s tool returns click, type, and scroll actions that code executes, whereas agent-desktop returns structure, not images.

A fixed path to three operating systems

Each command follows the same chain: the CLI or C library calls agent-desktop-core, which reaches the operating system through the PlatformAdapter trait. One platform crate then communicates with the native accessibility API, while core never imports a platform crate. Each binary is compiled to drive exactly one operating system.

On macOS, that API is AX API; Windows uses UI Automation; Linux uses AT-SPI. Microsoft’s UFO² also uses UI Automation, but it ships as a full Windows agent, putting agent-desktop in a different category: infrastructure for agents rather than a complete Windows operator.

Ref actions travel through the accessibility API without moving the pointer or taking focus, allowing agents to work while a person keeps using the machine. The global –headed flag enables physical input when needed, and every ref action waits up to 5,000 ms for a visible, enabled, stable target.

Installation starts with npm install -g agent-desktop, though Bun, Yarn, and pnpm also work. The package bundles native binaries for x64 and ARM64, runs on plain CPU with no GPU, and supports macOS 13+, Windows 10 1809+ or Server 2019+, and Ubuntu 24.04+.

macOS requires Accessibility permission, with the guide available through agent-desktop permissions –guide. Windows needs no grant for apps at the terminal’s integrity level, while Linux needs a graphical session with at-spi2-core.

Agents can learn the tool from bundled documentation through agent-desktop skills get desktop, and a skill is also listed on skills.sh. The C library lets Python, Swift, Go, Ruby, and Node hosts call agent-desktop in process; for Electron web contents, –cdp opens a DevTools port for agent-browser.

With 60 command names, including 56 operational commands, agent-desktop is a sizable interface rather than a demo wrapped in a launch post. Its practical pitch is simple: give any agent computer use through the operating system’s accessibility tree, keep the control loop in the agent, and stop spending tokens rediscovering pixels that the operating system already understands.

Stay connected via Google News

Follow us for the latest AI News