# trycua/cua

> Rust MCP driver for native apps and Chromium on macOS, Windows and Linux, plus a Python agent loop, sandboxes and benchmarks.

- Category: [Browser & computer control](https://llms-technical-reviews.com/browser-control/)
- Repository: https://github.com/trycua/cua (reviewed at commit `4f33cd6b119c50b11d96d82d023722fc2edf1bc0`, 2026-10-06)
- Stars: 28422 · Language: Rust · License: MIT
- Canonical page: https://llms-technical-reviews.com/p/cua/

## Overview

Cua is a large monorepo for giving AI agents a computer to operate. At the pinned commit it contains several products that share a name but not a runtime: **Cua Driver**, a Rust daemon and MCP server that inspects and drives native apps and Chromium browsers on macOS, Windows and Linux; **cua-agent**, a Python agent loop that adapts many computer-use models to one action space; **cua-sandbox** and the Rust `cua` SDK/CLI, which create local or cloud VMs and containers; **Lume** (Swift) for macOS VMs on Apple Silicon; **Cua Spaces**, a desktop app for hosting those machines; **CUA-S1**, small research models for element/action decisions; and **Cua Bench**, a task runner and evaluator.

For browser control, the interesting part is Cua Driver. It is not a Playwright wrapper. The browser is one surface of a desktop driver: the agent first finds a browser window by `pid` + `window_id` through the OS, and the driver then binds that window to a loopback CDP endpoint, refusing to act unless the binding is exact. Every page element is an opaque, session-scoped ref (`p<snapshot>:<index>`), and every mutation re-proves process, window, endpoint, tab and frame identity first. The code is strongly safety-minded. It refuses rather than guesses, and it keeps the user's own pointer and focus untouched by default ("background delivery").

The Python `ComputerAgent` is a separate, older layer. It runs a screenshot-and-coordinates loop against a `cua_sandbox.Sandbox` or a dict of callables, not against Cua Driver. Its own module docstring says that to control the local machine you use cua-driver's SDK or MCP server instead.

## Architecture

```mermaid
flowchart LR
  AG["MCP agent (Claude Code, Codex, ...)"] --> MCP["cua-driver mcp (stdio)"]
  APP["App via cua_driver / @trycua/cua-driver"] --> SDK["cua-driver-sdk (UniFFI)"]
  MCP --> SRV["server.rs: tools/list, tools/call"]
  MCP --> DMN["cua-driver serve (daemon)"]
  DMN --> SRV
  SDK --> REG["ToolRegistry"]
  SRV --> REG
  REG --> PLAT["platform-macos / -windows / -linux"]
  REG --> BR["BrowserEngine + browser tools"]
  PLAT --> OS["AX / UIA / AT-SPI, CGEvent / PostMessage / XSendEvent"]
  BR --> CDP["Loopback CDP WebSocket"]
  CDP --> CHR["Chromium tab"]
  PY["cua-agent ComputerAgent"] --> LOOP["Model loops via liteLLM"]
  PY --> SBX["cua-sandbox Sandbox"]
```

| Component | Path | Role |
|---|---|---|
| Driver binary | `libs/cua-driver/rust/crates/cua-driver/src/` | CLI, `mcp` stdio adapter, `serve` daemon, `call`, doctor, updater |
| Contract | `libs/cua-driver/rust/crates/cua-driver-contract/` | Typed inputs/outputs, JSON Schemas, MCP protocol version, action-result vocabulary |
| Core | `libs/cua-driver/rust/crates/cua-driver-core/src/` | MCP dispatch, `Tool` trait and `ToolRegistry`, sessions, authorization, element tokens, recording |
| Browser engine | `libs/cua-driver/rust/crates/cua-driver-core/src/browser/` | Binding, CDP pool, DOM/semantic snapshots, browser tools, isolated-profile launch |
| Platform backends | `libs/cua-driver/rust/crates/platform-macos`, `platform-windows`, `platform-linux` | Window enumeration, accessibility trees, screenshots, background input |
| SDK | `libs/cua-driver/rust/crates/cua-driver-sdk/` | In-process runtime exported to Python and TypeScript via UniFFI |
| Perception (optional) | `libs/cua-driver/rust/crates/cua-perception/` | Out-of-process ONNX OCR + icon detection over a retained screenshot |
| Agent loop | `libs/python/agent/cua_agent/` | `ComputerAgent`, model-specific loops, callbacks, computer handlers |
| Sandboxes | `libs/python/cua-sandbox`, `libs/cua/crates/cua-sdk` | Docker/QEMU/Lume/cloud machines with mouse, keyboard, screen interfaces |
| Models and benchmarks | `libs/cua-s1`, `libs/cua-bench` | Research decision models; task datasets and runner (`cb`) |

## How a request flows

Take an MCP agent that clicks a button in a Chromium tab.

1. **Start.** `cua-driver mcp` either runs the SDK runtime directly or proxies stdio JSON-RPC to a `serve` daemon, depending on platform and flags ([main.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver/src/main.rs#L846-L891)). The proxy keeps the ownership boundary invisible to the client; on macOS the daemon runs under LaunchServices for the right permission attribution ([proxy.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver/src/proxy.rs#L1-L60)).
2. **Dispatch.** `handle_request_inner` validates the call against the advertised tool list, strips reserved underscore arguments, checks `authorize_tool_call`, and hands off to the registry ([server.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/server.rs#L877-L940)). Each platform's `register_all` builds a `BrowserEngine` with its own platform adapter and registers the browser tools beside the native ones ([platform-macos tools/mod.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/platform-macos/src/tools/mod.rs#L950-L961), [browser/tools.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs#L30-L40)).
3. **Bind.** `get_browser_state` with `pid` + `window_id` calls `bind_native`, which asks the platform to classify the process and refuses anything that is not a CDP-capable browser ([engine.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs#L1237-L1253)). Attaching to a user's existing profile needs an explicit grant (`cua-driver mcp --grant existing-profile`). Alternatively `browser_prepare` launches a driver-owned Chromium with `--remote-debugging-port=0` and a marked profile directory, then reads `DevToolsActivePort` ([prepare.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs#L473-L488), [L650-L656](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs#L650-L656)).
4. **Snapshot.** The same tool with `target_id` + `tab_id` returns either `dom_refs_v1` or `semantic_v2`. The semantic form joins the AX tree, pierced DOM and layout evidence, caps output at 300 nodes, reports what it omitted and why, and supports `scope_ref`, `query` and continuation paging ([browser/tools.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs#L356-L470), [semantic.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs#L1-L28)).
5. **Revalidate.** `browser_click` takes a per-tab mutation lock, then `revalidate_for_mutation` rejects heuristic bindings, stale generations and ended grants before any page call ([tools.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs#L980-L1090), [engine.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs#L1416-L1470)). The ref is resolved and its frame identity re-proven.
6. **Act.** The default "trusted" route scrolls the node into view, takes the centre of `DOM.getBoxModel`, enables focus emulation and sends `Input.dispatchMouseEvent` press/release. It never silently falls back to a synthetic `el.click()`; that `dom_event` route is opt-in and its result is labelled `effect: unverifiable` ([tools.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs#L1200-L1310)).
7. **Return.** The result has a shared shape, `{"status": "ok" | "refused", ...}` with a stable refusal code, so an agent can branch on it and refresh state.

## Key components

### Native perception and element tokens

For any window, `get_window_state` returns a screenshot (as MCP image content), `tree_markdown`, a structured `elements` list and truncation flags ([windows.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs#L274-L323)). Elements are addressed by tokens like `s00001234:42`; a token from an old snapshot fails with "element_token is stale; call get_window_state again" ([element_token.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs#L1-L33)). That makes the observe-then-act contract explicit for both desktop and browser.

### Background input per OS

Each platform implements the same tools differently. On macOS, `click` performs an AX action on a token or posts CGEvent clicks to the target pid; `type_text` writes `AXSelectedText` and falls back to key events for Chromium, Electron and terminals ([click.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs#L1-L14), [type_text.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs#L1-L22)). On Linux, pointer and key events go through X11 `XSendEvent` and deliberately not XTest, because XTest targets the focused window ([input/mod.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs#L1-L12)). Windows uses UIA plus `PostMessage`. A `delivery_mode` of `background` (default) or `foreground` is accepted by the browser tools too.

### Browser engine

The browser module documents its rules up front: native entry point is `pid + window_id`, target/tab/ref ids are opaque session capabilities, mutation needs an exact binding, and frames whose identity cannot be proven are omitted, never guessed ([browser/mod.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs#L1-L36), [engine.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs#L1-L24)). Shadow DOM, same-process iframes and (when proven) out-of-process iframes are composed into one snapshot. CDP endpoints must be literal loopback `ws://` URLs ([cdp_ws.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/cdp_ws.rs#L83-L102)). Besides click, there are `browser_navigate`, `browser_type`, `browser_pointer`, dialog handling, file upload and download tools.

### Sessions, permissions, recording

Sessions are started with `start_session` and expire after a 5-minute idle TTL ([session.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs#L30-L30)); browser refs require an explicit session so cleanup has an owner. Permission modes are `standard`, `bounded` (a reviewed capability manifest) and `unrestricted` (behind `--dangerously-bypass-approvals`). The `ToolRegistry` also auto-records non-read-only calls for replay ([tool.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs#L682-L700)).

### Optional visual perception

`cua-perception` is a separate, separately licensed (AGPL icon detector) extension. It runs as a child process over a length-prefixed stdio protocol, hashes local models before loading ONNX Runtime, and caps inputs at 8192 px per side and 8 MB ([lib.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-perception/src/lib.rs#L27-L31)). A pixel click derived from its regions must carry the same one-use `capture_id`.

### cua-agent loop

`ComputerAgent` picks a loop class from a regex registry sorted by priority ([decorators.py](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/decorators.py#L13-L55)): Claude, OpenAI `computer-use-preview`/`gpt-5.4`, Gemini, UI-TARS, Qwen, GLM-4.5V, OpenCUA, Holo and more, with a catch-all `GenericVLM` at priority -100 and a composed `grounding_model+thinking_model` loop at priority 1 ([composed_grounded.py](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/loops/composed_grounded.py#L123-L130)). `run()` loops until the last item is an assistant message: call `predict_step` with exponential-backoff retries on transient errors ([agent.py](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/agent.py#L186-L247)), execute each `computer_call` by calling the same-named method on the computer handler, then take a fresh screenshot ([agent.py](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/agent.py#L717-L790), [L936-L1035](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/agent.py#L936-L1035)). Callbacks handle image retention, budgets and trajectory saving. Computers are a cua-sandbox `Sandbox`, a custom `AsyncComputerHandler`, or a dict of functions ([computers/__init__.py](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/computers/__init__.py#L33-L53)).

## Extending it

- **New tools** implement the `Tool` trait (`def`, `invoke`, plus optional ownership and scope hooks for protected resources) and register in a platform's `register_all` ([tool.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/tool.rs#L519-L530)). Public contract changes go through `cua-driver-contract` and, per the repo's rules, an RFC.
- **New browsers or OSes** implement `BrowserPlatform` for process identity, endpoint ownership and window metadata; the engine owns all CDP logic.
- **New models** add an `AsyncAgentConfig` class with `predict_step`, `predict_click` and `get_capabilities`, decorated with `@register_agent(models=..., priority=...)`.
- **Embedding:** Python and TypeScript apps can load the driver in-process through `cua_driver` / `@trycua/cua-driver` (UniFFI over a versioned C ABI), with no daemon.

## Running it

- **Driver:** install with the published shell or PowerShell script, then `cua-driver mcp-config --client claude-code` to wire an MCP client, or `cua-driver call <tool>` from a shell. macOS needs Accessibility and Screen Recording permissions; Linux backends cover X11, Sway/wlroots, GNOME and KDE with different limits.
- **Agent:** `pip install cua-agent` and `cua-sandbox`, then pass a `Sandbox` and a model string to `ComputerAgent`. Provider keys go through liteLLM.
- **Sandboxes:** local Docker, QEMU, Lume or Tart, or cloud machines through the `cua` SDK ([cua-sdk lib.rs](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua/crates/cua-sdk/src/lib.rs#L1-L31)).

## Strengths and caveats

- **Strength: one driver for desktop and browser.** The same session, token and refusal model covers a native Calculator window and a Chromium tab, so an agent can mix both in one task.
- **Strength: exact-or-refused safety.** Mutations re-prove identity, trusted input never degrades silently to synthetic events, and CDP is loopback-only. It is among the most defensive browser layers in this category.
- **Strength: background operation.** Driving apps without moving the user's pointer or stealing focus is a real engineering effort per OS, not a flag.
- **Caveat: it does not plan.** Cua Driver is a tool server. Planning comes from your MCP agent, and the Python `ComputerAgent` does not use the driver at all; it is a coordinates-and-screenshots loop over sandboxes.
- **Caveat: Chromium only for page tools.** The browser engine requires CDP; Firefox and Safari windows are reachable only through the generic accessibility and pixel tools.
- **Caveat: no stealth or CAPTCHA handling.** Driver-owned profiles are marked as automation profiles; there is no anti-detection code.
- **Caveat: size and churn.** Dozens of Rust crates plus Python, TypeScript, Swift and Kotlin packages, many verbose refusal paths, and several generations of APIs (legacy `page` tool vs. browser v2) make the codebase hard to learn from the outside.

*Sources: code at 4f33cd6, deepwiki-open wiki (9 pages), OpenDeepWiki wiki (50 pages), verified Q&A.*

## How trycua/cua answers the Browser & computer control questions

### How is the page represented to the model? (answered)

Page representation comes in three layers that can be used separately or together:

**Screenshots.** `get_window_state` and `get_desktop_state` return a PNG screenshot with its dimensions. The desktop capture has an optional `max_image_dimension` cap — if downscaled, the response reports `screenshot_original_width/height` and the `desktop_capture_scale` module records the ratio so subsequent desktop-frame actions are automatically scaled back to uncapped coordinates (`libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rs:32-56`). Screenshots travel in the MCP envelope as `SnapshotImage {mime_type, data_base64}` (`libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs:265-272`).

**Accessibility tree.** `get_window_state` returns a `tree_markdown` field — a formatted text representation of the platform's accessibility tree — plus a structured `elements: Vec<WindowElement>` array. Each element carries an `element_index` used to mint per-element tokens. The walk can be bounded by `max_elements`, `max_depth`, and `timeout_ms` (100–120000ms). Truncated walks are flagged with `truncated: true` and a `truncation_reason` rather than failing (`libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs:275-323`). On browsers, the semantic snapshot composes Chrome's AX tree with pierced DOM metadata and layout evidence (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs:1-16`).

**Element tokens (set-of-marks).** Each AX element gets a stable token formatted as `s{snapshot_id:08x}:{element_index}` (e.g. `s00001234:42`). Tokens are per-snapshot: they become stale after a new snapshot (returning the error `"element_token is stale; call get_window_state again to refresh"`) (`libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4-20`). The browser path additionally uses page refs in the `p<snapshot>:<index>` namespace (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs:103-110`).

**Visual regions (parse_visual_regions).** An optional ONNX-based perception subprocess (cua-perception crate) runs a YOLO icon detector and an OCR pipeline (Paddle-style detector + recognizer) on a screenshot, returning `VisualRegion` items with bounds, kind (text/icon), confidence, and optionally text content. This runs as a separate child process with a length-prefixed stdio protocol (`libs/cua-driver/rust/crates/cua-perception/src/runtime.rs:132-200`, `libs/cua-driver/rust/crates/cua-perception/src/lib.rs:28-31`). Max image size is 8192px on a side, 32M pixels, 8MB wire bytes (`libs/cua-driver/rust/crates/cua-perception/src/lib.rs:29-31`).


Citations: [libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs:275-323](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs#L275-L323) · [libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4-20](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs#L4-L20) · [libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rs:32-56](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rs#L32-L56) · [libs/cua-driver/rust/crates/cua-perception/src/runtime.rs:132-200](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-perception/src/runtime.rs#L132-L200) · [libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs:1-16](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs#L1-L16)

### How are actions executed and how are elements targeted? (answered)

Actions execute through a layered system of MCP tools dispatched to OS-native code on each platform.

**Tools & targeting.** Actions like `click`, `type_text`, `scroll`, `drag`, `move_cursor`, `hotkey`, `press_key` are registered MCP tools (`libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs:107-128`). Targets can be specified by:
- **Pixel coordinates** (`x`, `y`) in the window or desktop frame
- **Element token** (`element_token = "s00001234:42"`) — resolves to a cached accessible element from the last snapshot
- **Browser ref** (`"p<snapshot>:<index>"`) for browser DOM elements
- **PID + window_id** for window-scoped actions
- **Desktop scope** via `{kind: "desktop", display_id: "primary"}`
The `action_target.rs` module normalizes the tagged-union `ActionTarget` into the legacy flat fields before dispatch (`libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs:30-100`).

**Platform-specific delivery.** Each platform has its own input implementation behind the cross-platform contract:
- **macOS**: Uses Apple Accessibility (AX) API for most input. `click` performs AXAction on an element or synthesizes CGEvent mouse clicks posted to the target pid. `type_text` inserts text via `AXSetAttribute(kAXSelectedText)` for native Cocoa fields, falling back to CGEvent keystrokes for Chromium/Electron and terminals (`libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs:1-80`, `libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs:1-80`).
- **Linux X11**: Uses `XSendEvent` for background pointer events (no focus steal). Keyboard input uses evdev uinput virtual devices (`libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs:1-53`). AT-SPI for accessibility-backed actions.
- **Windows**: Uses UIA TextPattern + FindAll for accessibility, shared CDP for JS exec.
- **Browser**: CDP-based `browser_click`, `browser_type`, `browser_pointer` tools with session-scoped target/tab lifetimes (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs:30-40`).

**Delivery modes.** Actions have a `delivery_mode` of `background` (default, no activation of the target window) or `foreground` (accepts activation). Background delivery that would require activation is refused with `WouldRequireActivation` (`libs/cua-driver/rust/crates/cua-driver-core/src/interactive_input.rs:38-46`).

**Special actions.** `set_value` for direct value writes, `invoke_menu` for app menus (up to 16 menu items deep), `drag` with waypoint interpolation, and `type_text_chars` for per-character pacing. File upload for browsers uses `browser_set_input_files` (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs:37`).

> **Editor's note.** Correction: on Linux X11, background pointer and keyboard events both go through XSendEvent (platform-linux/src/input/mod.rs). The evdev uinput device is used only for the MPX virtual-keyboard path, and XTest is deliberately avoided.

Citations: [libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs:107-128](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs#L107-L128) · [libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs:30-100](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs#L30-L100) · [libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs:1-80](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs#L1-L80) · [libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs:1-80](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs#L1-L80) · [libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs:1-53](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs#L1-L53)

### How is the agent loop / planning implemented? (answered)

The agent loop lives in the Python `cua_agent` package (in the Python agent SDK at `libs/python/agent/`), with the TypeScript `@trycua/agent` package providing a client interface to the same proxy.

**Architecture.** `ComputerAgent` (`libs/python/agent/cua_agent/agent.py:250`) is the main class. It auto-selects an agent loop based on the model name via a decorator-based registry (`libs/python/agent/cua_agent/decorators.py:13-55`). Each loop implements the `AsyncAgentConfig` protocol with `predict_step`, `predict_click`, and `get_capabilities` methods (`libs/python/agent/cua_agent/loops/base.py:11-80`).

**Step loop.** The core loop in `ComputerAgent._run()` (around line 936) iterates: it sends messages + tool schemas to the model via the loop's `predict_step`, collects the response, handles any computer actions by executing them through a `ComputerHandler`, appends the results (screenshots, outputs) back to the message history, and repeats until the model produces an assistant message (no more tool calls). The loop is a single agent — no separate planner vs. executor distinction; the model itself decides actions and their sequence.

**Tool-call schema.** Messages follow OpenAI's Responses API schema: `ComputerCallMessage` (with `{type: "computer_call", action: {type, x, y, ...}}`) and `ComputerCallOutputMessage` (with screenshot) are the primary computer-use types (`libs/typescript/agent/src/types.ts:76-87`). Each agent loop maps these to its provider's format (Anthropic's `computer_use` beta tool, OpenAI's `computer_use_preview`, Gemini's `computer_use`, or custom function calls).

**Stop conditions.** The loop naturally terminates when the model returns a non-computer-call response (text assistant message). The `BudgetManagerCallback` can also stop based on token/cost limits. Per-request `timeout` (default 30s) limits individual API calls (`libs/typescript/agent/src/client.ts:160`).

**Memory between steps.** The full message history (old_items + new_items) accumulates across iterations. Callbacks like `ImageRetentionCallback` (optionally keeping only N most recent images) and `PromptInstructionsCallback` can trim context. `use_prompt_caching` enables Anthropic-style prompt caching (`libs/python/agent/cua_agent/agent.py:948`).

> **Editor's note.** Correction: ComputerAgent also ships a composed loop (loops/composed_grounded.py, model string "grounding_model+thinking_model", priority 1) that splits planning and grounding across two models. It drives a cua-sandbox Sandbox or custom handler, not Cua Driver.

Citations: [libs/python/agent/cua_agent/agent.py:250-275](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/agent.py#L250-L275) · [libs/python/agent/cua_agent/loops/base.py:11-80](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/loops/base.py#L11-L80) · [libs/python/agent/cua_agent/decorators.py:13-55](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/decorators.py#L13-L55) · [libs/python/agent/cua_agent/agent.py:936-1045](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/agent.py#L936-L1045) · [libs/typescript/agent/src/types.ts:76-87](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/typescript/agent/src/types.ts#L76-L87)

### How are failures, retries and self-healing handled? (answered)

**Retries.** The agent loop wraps every `predict_step` call in `_predict_step_with_retry` (`libs/python/agent/cua_agent/agent.py:213-247`) which provides exponential backoff (base 2s, doubling) on transient errors. Retryable errors include `asyncio.TimeoutError`, `RateLimitError`, `ServiceUnavailableError`, `APIConnectionError`, `InternalServerError`, and messages containing "timeout", "rate limit", "503", "502", "429", or "connection" (`libs/python/agent/cua_agent/agent.py:186-210`). Default max_retries is 3. Non-transient errors are raised immediately.

**Driver-level protection.** The driver's action tools have a `PidOnlyWindowTargetGuard` that resolves PID-only calls: if a pid has no windows it returns `NotFound`, if one window it targets it, if multiple it returns `Ambiguous(...)` with the candidates so the agent can disambiguate (`libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs:28-51`). Element tokens have staleness detection — using a stale token returns a structured `"element_token is stale"` refusal (`libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:5-6`).

**Driver-owned browsers.** The browser engine re-proves every identity before mutation — `revalidate_for_mutation` checks the full chain (process fingerprint → native ownership/bounds → endpoint → CDP target type/window) before any input, and `frame_session_for_mutation` additionally re-proves frame identity. Missing or changed identities are refused, never guessed (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs:1-23`).

**Timeouts.** Session idle TTL defaults to 5 minutes, configured via `DEFAULT_SESSION_IDLE_TTL` (`libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30`). Window observation after actions has a configurable deadline (default ~500ms, max 10s, min poll interval 5ms) (`libs/cua-driver/rust/crates/cua-driver-core/src/window_observation.rs:23-38`). Post-action observation detects menus/dialogs the action may have opened.

**Caching/session recovery.** The `capture_registry` retains recent screenshots by `capture_id` so re-parsing doesn't need re-capture. The `SnapshotStore` keeps an LRU per-pid (8 entries) for element snapshots (`libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4`). The TypeScript agent client has a `classifyError` method that categorizes errors (timeout, http, parse, network, peer) for appropriate handling (`libs/typescript/agent/src/client.ts:135-156`).


Citations: [libs/python/agent/cua_agent/agent.py:186-247](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/agent.py#L186-L247) · [libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4-7](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs#L4-L7) · [libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs:28-51](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs#L28-L51) · [libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs:1-23](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs#L1-L23) · [libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30-30](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs#L30-L30)

### Which models are supported and how are they called? (answered)

**Provider support.** The agent package (Python `cua_agent`) routes model calls through liteLLM, giving it access to hundreds of providers. Each model family gets a registered agent loop that maps the shared computer-use action space to the provider's API format. Supported model families include:
- **Anthropic Claude**: All Claude models via `anthropic` loop with beta version negotiation (`computer_20251124`, `computer_20250124`, `computer_20241022`). Screenshots proactively downscaled to 1024×768 to avoid API-side downscaling (`libs/python/agent/cua_agent/loops/anthropic.py:41-79`).
- **OpenAI**: `computer-use-preview` and `gpt-5.4` via native `computer_use_preview` tool or standard function calling (`libs/python/agent/cua_agent/loops/openai.py:42-80`).
- **Google Gemini**: `gemini-2.5/3/3.1-{flash,pro,computer-use-preview}` via `google.genai` SDK, with `thinking_level` and `media_resolution` parameters (`libs/python/agent/cua_agent/loops/gemini.py:688`).
- **CUA-S1**: Small specialized models — nano (855K-parameter option-attention classifier), 4B (Qwen3.5-based LoRA for element/action decisions), and form-v0 (text-only for form UIs) — all running locally via ONNX (nano) or Transformers+PEFT (4B) (`libs/cua-s1/README.md:1-100`).
- **Other**: Qwen3-VL, InternVL, GLM-4.5V, FARA-7B, UI-TARS, Yutori N1, Moondream3, UI-Ins, Holo1.5, OpenCUA, GTA1, Gelato for research/demo purposes. A catch-all `GenericVLM` loop at priority -100 handles any unrecognized model (`libs/python/agent/cua_agent/loops/generic_vlm.py:237`).

**Vision requirement.** All computer-use loops require vision — the loop sends screenshots as images and receives coordinate-based actions back. The Ollama path explicitly errors out if images are detected (`libs/python/agent/cua_agent/agent.py:964-990`).

**Structured output / tool calling.** Different loops use different mechanisms: Anthropic uses beta `computer_use` tool type, OpenAI uses `computer_use_preview` or `function` type, Gemini uses the `computer_use` config. The Python adapter `CUAAdapter` routes models through a Cua inference API that adds provider prefixes (`anthropic/`, `gemini/`, `openai/`) via liteLLM (`libs/python/agent/cua_agent/adapters/cua_adapter.py:27-34`).


Citations: [libs/python/agent/cua_agent/loops/anthropic.py:41-79](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/loops/anthropic.py#L41-L79) · [libs/python/agent/cua_agent/loops/openai.py:42-80](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/loops/openai.py#L42-L80) · [libs/python/agent/cua_agent/adapters/cua_adapter.py:27-34](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/adapters/cua_adapter.py#L27-L34) · [libs/cua-s1/README.md:1-100](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-s1/README.md#L1-L100) · [libs/python/agent/cua_agent/loops/generic_vlm.py:237-237](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/python/agent/cua_agent/loops/generic_vlm.py#L237-L237)

### How are browser sessions, profiles, auth and anti-bot handled? (answered)

**Session lifecycle.** Cua Driver has a full session lifecycle via MCP tools: `start_session`, `list_sessions`, `get_session`, `end_session`, and `escalate_session`. Sessions own agent cursor state, per-run configuration, recording, cleanup, and telemetry. Each session has a public label (optional, for multi-call coordination) and an implicit transport-bound id. The default idle TTL is 5 minutes (`libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30`). End reasons include explicit, idle timeout, process exit, and unknown (`libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:67-73`).

**Browser sessions (CDP).** The `BrowserEngine` manages pooled WebSocket connections to CDP endpoints. Browsers are classified by platform adapter into engine families (Chromium/Gecko/WebKit) and products (Chrome, Edge, Brave, etc.). Mutation requires exact bindings (bounds-correlated, optionally title-tie-broken) not heuristic ones (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs:1-37`). Browser preparation (`browser_prepare`) supports `isolated_new` (ephemeral) and `isolated_named` (persistent) profile modes with consent-based granting (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs:30-60`). `prepare_strategy` can be `OpenExisting`, `CreateNew`, or `CloseNewAfterAction`. 

**Local vs remote/cloud.** The SDK supports multiple topologies: embedded (in-process daemon), local daemon (Unix socket at `~/.cua/cua.sock`), and remote (loopback + token). Sandboxes can be local (VMs via QEMU, Tart, Lume) or cloud (Fleet, AWS, Google Cloud, Modal) (`libs/cua/crates/cua-sdk/src/lib.rs:5-28`). Cua Spaces app provides the remote desktop UI with Teleport (moving signed-in apps into sandboxes via Cua Keyvault).

**Profiles and auth.** The Cua Keyvault (`cua-keyvault` crate) stores app sessions encrypted on the Mac. Teleport moves a signed-in app (Chrome, Slack) into a Space still signed in. The `cua-auth` crate handles sign-in (browser PKCE or device code). Browser profiles can be isolated (ephemeral profiles flagged with `.cua-driver-owned-profile.json`) or reused (`isolated_named` with a stable name) (`libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs:30-60`).

**Anti-bot / stealth / CAPTCHA.** The repository does not implement anti-bot countermeasures or CAPTCHA handling. There is no stealth/heuristics module. The platform adapters classify browsers but that classification is used for capability negotiation (e.g. "does this pid support CDP") rather than concealing automation. The driver-owned browser profiles are explicitly marked as automation profiles. The codebase documents that CAPTCHAs and permission popups may be visible ("browser chrome may be incomplete in window scope") and routes through `recovery` instructions suggesting desktop-level capture for prompts the window cannot capture (`libs/cua-driver/rust/crates/cua-driver-core/src/window_inspection.rs:47-56`).


Citations: [libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30-100](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/session.rs#L30-L100) · [libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs:1-37](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs#L1-L37) · [libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs:30-60](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs#L30-L60) · [libs/cua/crates/cua-sdk/src/lib.rs:5-28](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua/crates/cua-sdk/src/lib.rs#L5-L28) · [libs/cua-driver/rust/crates/cua-driver-core/src/window_inspection.rs:36-56](https://github.com/trycua/cua/blob/4f33cd6b119c50b11d96d82d023722fc2edf1bc0/libs/cua-driver/rust/crates/cua-driver-core/src/window_inspection.rs#L36-L56)