Vercel Labs Open-Sources agent-desktop for Agent Computer Use

Vercel Labs has released agent-desktop , an open-source CLI for AI agent computer use on macOS, Windows and Linux. agent-desktop reads each app through the operating system’s accessibility tree, not through screenshots. An agent gets a JSON snapshot with a ref for every element it can act on. Version 1.0.2 shipped on October 30, 2026, under Apache-2.0. For developers, any shell-capable coding agent can now drive native apps without a vision model. TL;DR Library: Rust CLI and a C library. 60 command names, 56 operational . Runs on: macOS 13+, Windows 10 1809+ or Server 2019+, Ubuntu 24.04+ , on x64 and ARM64. Plain CPU, no GPU. Performance: Deterministic, retry-safe ref actions. No independent computer-use benchmark yet. Best: 78% to 96% fewer tokens on dense Electron apps via skeleton traversal ( self-reported ). What is agent-desktop? agent-desktop is a Rust CLI and C library from Vercel Labs for reading and operating desktop apps. It is a tool that agents call, not an agent . It embeds no LLM and has no GUI. Every command prints one JSON envelope and exits. The observe, act and verify loop stays in your agent. How does agent-desktop work? Each command flows through a fixed chain. The CLI or C library calls agent-desktop-core , which reaches the OS only through a PlatformAdapter trait. One platform crate then talks to the native accessibility API: macOS: AX API Windows: UI Automation Linux: AT-SPI Core never imports a platform crate. Each binary is compiled to drive exactly one OS. Snapshots and refs A snapshot assigns qualified refs such as @s8f3k2p9:e8 . The snapshot ID inside the ref makes it self-describing. A bare @e8 is rejected with INVALID_ARGS . When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET . The agent then takes a fresh snapshot. Every error carries a disposition . Agents should retry only when it says safe . Ref resolution is strict and fails closed rather than guessing. Skeleton and drill-down Dense apps can expose thousands of accessibility nodes. A --skeleton snapshot reads only 3 levels deep . Cut containers report a children_count and get a drill ref. snapshot --root then reads one region into the same snapshot. The builder reported a 78% to 96% token cut versus full-tree dumps in Slack, VS Code and Notion. In Apple Maps, he reported 3,811 tokens per look falling to 273 . A simulated 52-step task dropped from 198k to 31k tokens. These are self-reported figures, not an independent benchmark. Headless by default Ref actions go through the accessibility API without moving the pointer or taking focus . That is what lets agents work while you keep using the machine. The global --headed flag allows physical input when needed. Every ref action also auto-waits up to 5,000 ms for a visible, enabled, stable target. Trace a command</button><button class="b g" id="ad-step">Step</button></div> </div> <div class="pane" data-p="1"> <p class="lead">Map, drill, act, verify. Click a step. Example output is trimmed from the official docs (GNOME Calculator).</p> <div class="steps" id="ad-steps"> <div class="st on" data-s="0"><i>1</i><b>Skeleton</b></div> <div class="st" data-s="1"><i>2</i><b>Drill</b></div> <div class="st" data-s="2"><i>3</i><b>Act</b></div> <div class="st" data-s="3"><i>4</i><b>Verify</b></div> </div> <pre id="ad-code"></pre> <p class="note" id="ad-note"></p> <div class="btns"><button class="b" id="ad-next">Next step →</button></div> </div> <div class="pane" data-p="2"> <p class="lead">Builder-reported numbers for skeleton drilling, next to Anthropic's documented screenshot cost.</p> <div class="bars" id="ad-bars"> <div class="grp">Apple Maps, tokens per look (self-reported)</div> <div class="bar"><div class="lb"><span>Full window snapshot</span><em>3,811</em></div><div class="trk"><div class="fill" data-w="100" style="background:#ededed">3,811</div></div></div> <div class="bar"><div class="lb"><span>Skeleton, then drill</span><em>273</em></div><div class="trk"><div class="fill" data-w="7.2" style="background:#0070f3"></div></div></div> <div class="grp">Simulated 52-step task, total tokens (self-reported)</div> <div class="bar"><div class="lb"><span>Full window every step</span><em>198k</em></div><div class="trk"><div class="fill" data-w="100" style="background:#ededed">198k</div></div></div> <div class="bar"><div class="lb"><span>Skeleton, then drill</span><em>31k</em></div><div class="trk"><div class="fill" data-w="15.7" style="background:#0070f3">31k</div></div></div> <div class="grp">Reference: pixel-based computer use</div> <div class="bar"><div class="lb"><span>One Claude computer-use screenshot (Anthropic docs)</span><em>~1,000 to 1,800</em></div><div class="trk"><div class="fill" data-w="47" style="background:#f5a623">~1.8k max</div></div></div> </div> <div class="btns"><button class="b g" id="ad-replay">↻ Replay</button></div> <p class="note" style="margin-top:10px!important">Builder also reports 78% to 96% savings on Slack, VS Code and Notion. No independent benchmark yet.</p> </div> <div class="pane" data-p="3"> <p class="lead">Same 60 command names everywhere. Click a platform for requirements.</p> <div class="os"> <div class="oc"><h4>macOS</h4><div class="api">AX API</div><div class="more">macOS 13+, Intel or Apple Silicon. Grant Accessibility permission with <code>permissions --guide</code>. Shell surfaces still planned.</div></div> <div class="oc"><h4>Windows</h4><div class="api">UI Automation</div><div class="more">Windows 10 1809+ or Server 2019+, x64 or ARM64. No grant for apps at your terminal's integrity level. Higher-integrity apps refuse input.</div></div> <div class="oc"><h4>Linux</h4><div class="api">AT-SPI</div><div class="more">Ubuntu 24.04+, glibc 2.35 floor, x86_64 or aarch64. Needs <code>at-spi2-core</code>. Mouse input and cursor overlay need X11.</div></div> </div> <pre style="min-height:0;margin-top:14px!important"><span class="c"># one install for all 3</span> <span class="k">npm</span> install -g agent-desktop <span class="k">agent-desktop</span> skills get desktop</pre> </div> <div class="pane" data-p="4"> <p class="lead">Where it falls short. Click to expand.</p> <di