Home / Browser & computer control / cua

trycua/cua

Rust MCP driver for native apps and Chromium on macOS, Windows and Linux, plus a Python agent loop, sandboxes and benchmarks.

GitHub ↗★ 28kRustMITcommit 4f33cd6 · 2026-10-06homepage ↗

Overview

Cua is a large monorepo for giving AI agents a computer to operate. At the pinned commit it contains several products that share a name but not a runtime: Cua Driver, a Rust daemon and MCP server that inspects and drives native apps and Chromium browsers on macOS, Windows and Linux; cua-agent, a Python agent loop that adapts many computer-use models to one action space; cua-sandbox and the Rust cua SDK/CLI, which create local or cloud VMs and containers; Lume (Swift) for macOS VMs on Apple Silicon; Cua Spaces, a desktop app for hosting those machines; CUA-S1, small research models for element/action decisions; and Cua Bench, a task runner and evaluator.

For browser control, the interesting part is Cua Driver. It is not a Playwright wrapper. The browser is one surface of a desktop driver: the agent first finds a browser window by pid + window_id through the OS, and the driver then binds that window to a loopback CDP endpoint, refusing to act unless the binding is exact. Every page element is an opaque, session-scoped ref (p<snapshot>:<index>), and every mutation re-proves process, window, endpoint, tab and frame identity first. The code is strongly safety-minded. It refuses rather than guesses, and it keeps the user’s own pointer and focus untouched by default (“background delivery”).

The Python ComputerAgent is a separate, older layer. It runs a screenshot-and-coordinates loop against a cua_sandbox.Sandbox or a dict of callables, not against Cua Driver. Its own module docstring says that to control the local machine you use cua-driver’s SDK or MCP server instead.

Architecture

flowchart LR
  AG["MCP agent (Claude Code, Codex, ...)"] --> MCP["cua-driver mcp (stdio)"]
  APP["App via cua_driver / @trycua/cua-driver"] --> SDK["cua-driver-sdk (UniFFI)"]
  MCP --> SRV["server.rs: tools/list, tools/call"]
  MCP --> DMN["cua-driver serve (daemon)"]
  DMN --> SRV
  SDK --> REG["ToolRegistry"]
  SRV --> REG
  REG --> PLAT["platform-macos / -windows / -linux"]
  REG --> BR["BrowserEngine + browser tools"]
  PLAT --> OS["AX / UIA / AT-SPI, CGEvent / PostMessage / XSendEvent"]
  BR --> CDP["Loopback CDP WebSocket"]
  CDP --> CHR["Chromium tab"]
  PY["cua-agent ComputerAgent"] --> LOOP["Model loops via liteLLM"]
  PY --> SBX["cua-sandbox Sandbox"]

Component	Path	Role
Driver binary	`libs/cua-driver/rust/crates/cua-driver/src/`	CLI, `mcp` stdio adapter, `serve` daemon, `call`, doctor, updater
Contract	`libs/cua-driver/rust/crates/cua-driver-contract/`	Typed inputs/outputs, JSON Schemas, MCP protocol version, action-result vocabulary
Core	`libs/cua-driver/rust/crates/cua-driver-core/src/`	MCP dispatch, `Tool` trait and `ToolRegistry`, sessions, authorization, element tokens, recording
Browser engine	`libs/cua-driver/rust/crates/cua-driver-core/src/browser/`	Binding, CDP pool, DOM/semantic snapshots, browser tools, isolated-profile launch
Platform backends	`libs/cua-driver/rust/crates/platform-macos`, `platform-windows`, `platform-linux`	Window enumeration, accessibility trees, screenshots, background input
SDK	`libs/cua-driver/rust/crates/cua-driver-sdk/`	In-process runtime exported to Python and TypeScript via UniFFI
Perception (optional)	`libs/cua-driver/rust/crates/cua-perception/`	Out-of-process ONNX OCR + icon detection over a retained screenshot
Agent loop	`libs/python/agent/cua_agent/`	`ComputerAgent`, model-specific loops, callbacks, computer handlers
Sandboxes	`libs/python/cua-sandbox`, `libs/cua/crates/cua-sdk`	Docker/QEMU/Lume/cloud machines with mouse, keyboard, screen interfaces
Models and benchmarks	`libs/cua-s1`, `libs/cua-bench`	Research decision models; task datasets and runner (`cb`)

How a request flows

Take an MCP agent that clicks a button in a Chromium tab.

Start. cua-driver mcp either runs the SDK runtime directly or proxies stdio JSON-RPC to a serve daemon, depending on platform and flags (main.rs). The proxy keeps the ownership boundary invisible to the client; on macOS the daemon runs under LaunchServices for the right permission attribution (proxy.rs).
Dispatch. handle_request_inner validates the call against the advertised tool list, strips reserved underscore arguments, checks authorize_tool_call, and hands off to the registry (server.rs). Each platform’s register_all builds a BrowserEngine with its own platform adapter and registers the browser tools beside the native ones (platform-macos tools/mod.rs, browser/tools.rs).
Bind. get_browser_state with pid + window_id calls bind_native, which asks the platform to classify the process and refuses anything that is not a CDP-capable browser (engine.rs). Attaching to a user’s existing profile needs an explicit grant (cua-driver mcp --grant existing-profile). Alternatively browser_prepare launches a driver-owned Chromium with --remote-debugging-port=0 and a marked profile directory, then reads DevToolsActivePort (prepare.rs, L650-L656).
Snapshot. The same tool with target_id + tab_id returns either dom_refs_v1 or semantic_v2. The semantic form joins the AX tree, pierced DOM and layout evidence, caps output at 300 nodes, reports what it omitted and why, and supports scope_ref, query and continuation paging (browser/tools.rs, semantic.rs).
Revalidate. browser_click takes a per-tab mutation lock, then revalidate_for_mutation rejects heuristic bindings, stale generations and ended grants before any page call (tools.rs, engine.rs). The ref is resolved and its frame identity re-proven.
Act. The default “trusted” route scrolls the node into view, takes the centre of DOM.getBoxModel, enables focus emulation and sends Input.dispatchMouseEvent press/release. It never silently falls back to a synthetic el.click(); that dom_event route is opt-in and its result is labelled effect: unverifiable (tools.rs).
Return. The result has a shared shape, {"status": "ok" | "refused", ...} with a stable refusal code, so an agent can branch on it and refresh state.

Key components

Native perception and element tokens

For any window, get_window_state returns a screenshot (as MCP image content), tree_markdown, a structured elements list and truncation flags (windows.rs). Elements are addressed by tokens like s00001234:42; a token from an old snapshot fails with “element_token is stale; call get_window_state again” (element_token.rs). That makes the observe-then-act contract explicit for both desktop and browser.

Background input per OS

Each platform implements the same tools differently. On macOS, click performs an AX action on a token or posts CGEvent clicks to the target pid; type_text writes AXSelectedText and falls back to key events for Chromium, Electron and terminals (click.rs, type_text.rs). On Linux, pointer and key events go through X11 XSendEvent and deliberately not XTest, because XTest targets the focused window (input/mod.rs). Windows uses UIA plus PostMessage. A delivery_mode of background (default) or foreground is accepted by the browser tools too.

Browser engine

The browser module documents its rules up front: native entry point is pid + window_id, target/tab/ref ids are opaque session capabilities, mutation needs an exact binding, and frames whose identity cannot be proven are omitted, never guessed (browser/mod.rs, engine.rs). Shadow DOM, same-process iframes and (when proven) out-of-process iframes are composed into one snapshot. CDP endpoints must be literal loopback ws:// URLs (cdp_ws.rs). Besides click, there are browser_navigate, browser_type, browser_pointer, dialog handling, file upload and download tools.

Sessions, permissions, recording

Sessions are started with start_session and expire after a 5-minute idle TTL (session.rs); browser refs require an explicit session so cleanup has an owner. Permission modes are standard, bounded (a reviewed capability manifest) and unrestricted (behind --dangerously-bypass-approvals). The ToolRegistry also auto-records non-read-only calls for replay (tool.rs).

Optional visual perception

cua-perception is a separate, separately licensed (AGPL icon detector) extension. It runs as a child process over a length-prefixed stdio protocol, hashes local models before loading ONNX Runtime, and caps inputs at 8192 px per side and 8 MB (lib.rs). A pixel click derived from its regions must carry the same one-use capture_id.

cua-agent loop

ComputerAgent picks a loop class from a regex registry sorted by priority (decorators.py): Claude, OpenAI computer-use-preview/gpt-5.4, Gemini, UI-TARS, Qwen, GLM-4.5V, OpenCUA, Holo and more, with a catch-all GenericVLM at priority -100 and a composed grounding_model+thinking_model loop at priority 1 (composed_grounded.py). run() loops until the last item is an assistant message: call predict_step with exponential-backoff retries on transient errors (agent.py), execute each computer_call by calling the same-named method on the computer handler, then take a fresh screenshot (agent.py, L936-L1035). Callbacks handle image retention, budgets and trajectory saving. Computers are a cua-sandbox Sandbox, a custom AsyncComputerHandler, or a dict of functions (computers/init.py).

Extending it

New tools implement the Tool trait (def, invoke, plus optional ownership and scope hooks for protected resources) and register in a platform’s register_all (tool.rs). Public contract changes go through cua-driver-contract and, per the repo’s rules, an RFC.
New browsers or OSes implement BrowserPlatform for process identity, endpoint ownership and window metadata; the engine owns all CDP logic.
New models add an AsyncAgentConfig class with predict_step, predict_click and get_capabilities, decorated with @register_agent(models=..., priority=...).
Embedding: Python and TypeScript apps can load the driver in-process through cua_driver / @trycua/cua-driver (UniFFI over a versioned C ABI), with no daemon.

Running it

Driver: install with the published shell or PowerShell script, then cua-driver mcp-config --client claude-code to wire an MCP client, or cua-driver call <tool> from a shell. macOS needs Accessibility and Screen Recording permissions; Linux backends cover X11, Sway/wlroots, GNOME and KDE with different limits.
Agent: pip install cua-agent and cua-sandbox, then pass a Sandbox and a model string to ComputerAgent. Provider keys go through liteLLM.
Sandboxes: local Docker, QEMU, Lume or Tart, or cloud machines through the cua SDK (cua-sdk lib.rs).

Strengths and caveats

Strength: one driver for desktop and browser. The same session, token and refusal model covers a native Calculator window and a Chromium tab, so an agent can mix both in one task.
Strength: exact-or-refused safety. Mutations re-prove identity, trusted input never degrades silently to synthetic events, and CDP is loopback-only. It is among the most defensive browser layers in this category.
Strength: background operation. Driving apps without moving the user’s pointer or stealing focus is a real engineering effort per OS, not a flag.
Caveat: it does not plan. Cua Driver is a tool server. Planning comes from your MCP agent, and the Python ComputerAgent does not use the driver at all; it is a coordinates-and-screenshots loop over sandboxes.
Caveat: Chromium only for page tools. The browser engine requires CDP; Firefox and Safari windows are reachable only through the generic accessibility and pixel tools.
Caveat: no stealth or CAPTCHA handling. Driver-owned profiles are marked as automation profiles; there is no anti-detection code.
Caveat: size and churn. Dozens of Rust crates plus Python, TypeScript, Swift and Kotlin packages, many verbose refusal paths, and several generations of APIs (legacy page tool vs. browser v2) make the codebase hard to learn from the outside.

Sources: code at 4f33cd6, deepwiki-open wiki (9 pages), OpenDeepWiki wiki (50 pages), verified Q&A.

How it answers the Browser & computer control questions

Each answer was drafted by a code-reading agent at commit 4f33cd6. Its citations were checked mechanically. Compare with the other browser & computer control →

How is the page represented to the model?

answered

Page representation comes in three layers that can be used separately or together:

Screenshots. get_window_state and get_desktop_state return a PNG screenshot with its dimensions. The desktop capture has an optional max_image_dimension cap — if downscaled, the response reports screenshot_original_width/height and the desktop_capture_scale module records the ratio so subsequent desktop-frame actions are automatically scaled back to uncapped coordinates (libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rs:32-56). Screenshots travel in the MCP envelope as SnapshotImage {mime_type, data_base64} (libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs:265-272).

Accessibility tree. get_window_state returns a tree_markdown field — a formatted text representation of the platform's accessibility tree — plus a structured elements: Vec<WindowElement> array. Each element carries an element_index used to mint per-element tokens. The walk can be bounded by max_elements, max_depth, and timeout_ms (100–120000ms). Truncated walks are flagged with truncated: true and a truncation_reason rather than failing (libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs:275-323). On browsers, the semantic snapshot composes Chrome's AX tree with pierced DOM metadata and layout evidence (libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs:1-16).

Element tokens (set-of-marks). Each AX element gets a stable token formatted as s{snapshot_id:08x}:{element_index} (e.g. s00001234:42). Tokens are per-snapshot: they become stale after a new snapshot (returning the error "element_token is stale; call get_window_state again to refresh") (libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4-20). The browser path additionally uses page refs in the p<snapshot>:<index> namespace (libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs:103-110).

Visual regions (parse_visual_regions). An optional ONNX-based perception subprocess (cua-perception crate) runs a YOLO icon detector and an OCR pipeline (Paddle-style detector + recognizer) on a screenshot, returning VisualRegion items with bounds, kind (text/icon), confidence, and optionally text content. This runs as a separate child process with a length-prefixed stdio protocol (libs/cua-driver/rust/crates/cua-perception/src/runtime.rs:132-200, libs/cua-driver/rust/crates/cua-perception/src/lib.rs:28-31). Max image size is 8192px on a side, 32M pixels, 8MB wire bytes (libs/cua-driver/rust/crates/cua-perception/src/lib.rs:29-31).

libs/cua-driver/rust/crates/cua-driver-contract/src/windows.rs:275-323 libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4-20 libs/cua-driver/rust/crates/cua-driver-core/src/desktop_capture_scale.rs:32-56 libs/cua-driver/rust/crates/cua-perception/src/runtime.rs:132-200 libs/cua-driver/rust/crates/cua-driver-core/src/browser/semantic.rs:1-16

How are actions executed and how are elements targeted?

answered

Actions execute through a layered system of MCP tools dispatched to OS-native code on each platform.

Tools & targeting. Actions like click, type_text, scroll, drag, move_cursor, hotkey, press_key are registered MCP tools (libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs:107-128). Targets can be specified by:

Pixel coordinates (x, y) in the window or desktop frame
Element token (element_token = "s00001234:42") — resolves to a cached accessible element from the last snapshot
Browser ref ("p<snapshot>:<index>") for browser DOM elements
PID + window_id for window-scoped actions
Desktop scope via {kind: "desktop", display_id: "primary"} The action_target.rs module normalizes the tagged-union ActionTarget into the legacy flat fields before dispatch (libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs:30-100).

Platform-specific delivery. Each platform has its own input implementation behind the cross-platform contract:

macOS: Uses Apple Accessibility (AX) API for most input. click performs AXAction on an element or synthesizes CGEvent mouse clicks posted to the target pid. type_text inserts text via AXSetAttribute(kAXSelectedText) for native Cocoa fields, falling back to CGEvent keystrokes for Chromium/Electron and terminals (libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs:1-80, libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs:1-80).
Linux X11: Uses XSendEvent for background pointer events (no focus steal). Keyboard input uses evdev uinput virtual devices (libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs:1-53). AT-SPI for accessibility-backed actions.
Windows: Uses UIA TextPattern + FindAll for accessibility, shared CDP for JS exec.
Browser: CDP-based browser_click, browser_type, browser_pointer tools with session-scoped target/tab lifetimes (libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs:30-40).

Delivery modes. Actions have a delivery_mode of background (default, no activation of the target window) or foreground (accepts activation). Background delivery that would require activation is refused with WouldRequireActivation (libs/cua-driver/rust/crates/cua-driver-core/src/interactive_input.rs:38-46).

Special actions. set_value for direct value writes, invoke_menu for app menus (up to 16 menu items deep), drag with waypoint interpolation, and type_text_chars for per-character pacing. File upload for browsers uses browser_set_input_files (libs/cua-driver/rust/crates/cua-driver-core/src/browser/tools.rs:37).

Editor's note. Correction: on Linux X11, background pointer and keyboard events both go through XSendEvent (platform-linux/src/input/mod.rs). The evdev uinput device is used only for the MPX virtual-keyboard path, and XTest is deliberately avoided.

libs/cua-driver/rust/crates/cua-driver-contract/src/lib.rs:107-128 libs/cua-driver/rust/crates/cua-driver-core/src/action_target.rs:30-100 libs/cua-driver/rust/crates/platform-macos/src/tools/click.rs:1-80 libs/cua-driver/rust/crates/platform-macos/src/tools/type_text.rs:1-80 libs/cua-driver/rust/crates/platform-linux/src/input/mod.rs:1-53

How is the agent loop / planning implemented?

answered

The agent loop lives in the Python cua_agent package (in the Python agent SDK at libs/python/agent/), with the TypeScript @trycua/agent package providing a client interface to the same proxy.

Architecture. ComputerAgent (libs/python/agent/cua_agent/agent.py:250) is the main class. It auto-selects an agent loop based on the model name via a decorator-based registry (libs/python/agent/cua_agent/decorators.py:13-55). Each loop implements the AsyncAgentConfig protocol with predict_step, predict_click, and get_capabilities methods (libs/python/agent/cua_agent/loops/base.py:11-80).

Step loop. The core loop in ComputerAgent._run() (around line 936) iterates: it sends messages + tool schemas to the model via the loop's predict_step, collects the response, handles any computer actions by executing them through a ComputerHandler, appends the results (screenshots, outputs) back to the message history, and repeats until the model produces an assistant message (no more tool calls). The loop is a single agent — no separate planner vs. executor distinction; the model itself decides actions and their sequence.

Tool-call schema. Messages follow OpenAI's Responses API schema: ComputerCallMessage (with {type: "computer_call", action: {type, x, y, ...}}) and ComputerCallOutputMessage (with screenshot) are the primary computer-use types (libs/typescript/agent/src/types.ts:76-87). Each agent loop maps these to its provider's format (Anthropic's computer_use beta tool, OpenAI's computer_use_preview, Gemini's computer_use, or custom function calls).

Stop conditions. The loop naturally terminates when the model returns a non-computer-call response (text assistant message). The BudgetManagerCallback can also stop based on token/cost limits. Per-request timeout (default 30s) limits individual API calls (libs/typescript/agent/src/client.ts:160).

Memory between steps. The full message history (old_items + new_items) accumulates across iterations. Callbacks like ImageRetentionCallback (optionally keeping only N most recent images) and PromptInstructionsCallback can trim context. use_prompt_caching enables Anthropic-style prompt caching (libs/python/agent/cua_agent/agent.py:948).

Editor's note. Correction: ComputerAgent also ships a composed loop (loops/composed_grounded.py, model string "grounding_model+thinking_model", priority 1) that splits planning and grounding across two models. It drives a cua-sandbox Sandbox or custom handler, not Cua Driver.

libs/python/agent/cua_agent/agent.py:250-275 libs/python/agent/cua_agent/loops/base.py:11-80 libs/python/agent/cua_agent/decorators.py:13-55 libs/python/agent/cua_agent/agent.py:936-1045 libs/typescript/agent/src/types.ts:76-87

How are failures, retries and self-healing handled?

answered

Retries. The agent loop wraps every predict_step call in _predict_step_with_retry (libs/python/agent/cua_agent/agent.py:213-247) which provides exponential backoff (base 2s, doubling) on transient errors. Retryable errors include asyncio.TimeoutError, RateLimitError, ServiceUnavailableError, APIConnectionError, InternalServerError, and messages containing "timeout", "rate limit", "503", "502", "429", or "connection" (libs/python/agent/cua_agent/agent.py:186-210). Default max_retries is 3. Non-transient errors are raised immediately.

Driver-level protection. The driver's action tools have a PidOnlyWindowTargetGuard that resolves PID-only calls: if a pid has no windows it returns NotFound, if one window it targets it, if multiple it returns Ambiguous(...) with the candidates so the agent can disambiguate (libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs:28-51). Element tokens have staleness detection — using a stale token returns a structured "element_token is stale" refusal (libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:5-6).

Driver-owned browsers. The browser engine re-proves every identity before mutation — revalidate_for_mutation checks the full chain (process fingerprint → native ownership/bounds → endpoint → CDP target type/window) before any input, and frame_session_for_mutation additionally re-proves frame identity. Missing or changed identities are refused, never guessed (libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs:1-23).

Timeouts. Session idle TTL defaults to 5 minutes, configured via DEFAULT_SESSION_IDLE_TTL (libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30). Window observation after actions has a configurable deadline (default ~500ms, max 10s, min poll interval 5ms) (libs/cua-driver/rust/crates/cua-driver-core/src/window_observation.rs:23-38). Post-action observation detects menus/dialogs the action may have opened.

Caching/session recovery. The capture_registry retains recent screenshots by capture_id so re-parsing doesn't need re-capture. The SnapshotStore keeps an LRU per-pid (8 entries) for element snapshots (libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4). The TypeScript agent client has a classifyError method that categorizes errors (timeout, http, parse, network, peer) for appropriate handling (libs/typescript/agent/src/client.ts:135-156).

libs/python/agent/cua_agent/agent.py:186-247 libs/cua-driver/rust/crates/cua-driver-core/src/element_token.rs:4-7 libs/cua-driver/rust/crates/cua-driver-core/src/window_target.rs:28-51 libs/cua-driver/rust/crates/cua-driver-core/src/browser/engine.rs:1-23 libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30

Which models are supported and how are they called?

answered

Provider support. The agent package (Python cua_agent) routes model calls through liteLLM, giving it access to hundreds of providers. Each model family gets a registered agent loop that maps the shared computer-use action space to the provider's API format. Supported model families include:

Anthropic Claude: All Claude models via anthropic loop with beta version negotiation (computer_20251124, computer_20250124, computer_20241022). Screenshots proactively downscaled to 1024×768 to avoid API-side downscaling (libs/python/agent/cua_agent/loops/anthropic.py:41-79).
OpenAI: computer-use-preview and gpt-5.4 via native computer_use_preview tool or standard function calling (libs/python/agent/cua_agent/loops/openai.py:42-80).
Google Gemini: gemini-2.5/3/3.1-{flash,pro,computer-use-preview} via google.genai SDK, with thinking_level and media_resolution parameters (libs/python/agent/cua_agent/loops/gemini.py:688).
CUA-S1: Small specialized models — nano (855K-parameter option-attention classifier), 4B (Qwen3.5-based LoRA for element/action decisions), and form-v0 (text-only for form UIs) — all running locally via ONNX (nano) or Transformers+PEFT (4B) (libs/cua-s1/README.md:1-100).
Other: Qwen3-VL, InternVL, GLM-4.5V, FARA-7B, UI-TARS, Yutori N1, Moondream3, UI-Ins, Holo1.5, OpenCUA, GTA1, Gelato for research/demo purposes. A catch-all GenericVLM loop at priority -100 handles any unrecognized model (libs/python/agent/cua_agent/loops/generic_vlm.py:237).

Vision requirement. All computer-use loops require vision — the loop sends screenshots as images and receives coordinate-based actions back. The Ollama path explicitly errors out if images are detected (libs/python/agent/cua_agent/agent.py:964-990).

Structured output / tool calling. Different loops use different mechanisms: Anthropic uses beta computer_use tool type, OpenAI uses computer_use_preview or function type, Gemini uses the computer_use config. The Python adapter CUAAdapter routes models through a Cua inference API that adds provider prefixes (anthropic/, gemini/, openai/) via liteLLM (libs/python/agent/cua_agent/adapters/cua_adapter.py:27-34).

libs/python/agent/cua_agent/loops/anthropic.py:41-79 libs/python/agent/cua_agent/loops/openai.py:42-80 libs/python/agent/cua_agent/adapters/cua_adapter.py:27-34 libs/cua-s1/README.md:1-100 libs/python/agent/cua_agent/loops/generic_vlm.py:237

How are browser sessions, profiles, auth and anti-bot handled?

answered

Session lifecycle. Cua Driver has a full session lifecycle via MCP tools: start_session, list_sessions, get_session, end_session, and escalate_session. Sessions own agent cursor state, per-run configuration, recording, cleanup, and telemetry. Each session has a public label (optional, for multi-call coordination) and an implicit transport-bound id. The default idle TTL is 5 minutes (libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30). End reasons include explicit, idle timeout, process exit, and unknown (libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:67-73).

Browser sessions (CDP). The BrowserEngine manages pooled WebSocket connections to CDP endpoints. Browsers are classified by platform adapter into engine families (Chromium/Gecko/WebKit) and products (Chrome, Edge, Brave, etc.). Mutation requires exact bindings (bounds-correlated, optionally title-tie-broken) not heuristic ones (libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs:1-37). Browser preparation (browser_prepare) supports isolated_new (ephemeral) and isolated_named (persistent) profile modes with consent-based granting (libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs:30-60). prepare_strategy can be OpenExisting, CreateNew, or CloseNewAfterAction.

Local vs remote/cloud. The SDK supports multiple topologies: embedded (in-process daemon), local daemon (Unix socket at ~/.cua/cua.sock), and remote (loopback + token). Sandboxes can be local (VMs via QEMU, Tart, Lume) or cloud (Fleet, AWS, Google Cloud, Modal) (libs/cua/crates/cua-sdk/src/lib.rs:5-28). Cua Spaces app provides the remote desktop UI with Teleport (moving signed-in apps into sandboxes via Cua Keyvault).

Profiles and auth. The Cua Keyvault (cua-keyvault crate) stores app sessions encrypted on the Mac. Teleport moves a signed-in app (Chrome, Slack) into a Space still signed in. The cua-auth crate handles sign-in (browser PKCE or device code). Browser profiles can be isolated (ephemeral profiles flagged with .cua-driver-owned-profile.json) or reused (isolated_named with a stable name) (libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs:30-60).

Anti-bot / stealth / CAPTCHA. The repository does not implement anti-bot countermeasures or CAPTCHA handling. There is no stealth/heuristics module. The platform adapters classify browsers but that classification is used for capability negotiation (e.g. "does this pid support CDP") rather than concealing automation. The driver-owned browser profiles are explicitly marked as automation profiles. The codebase documents that CAPTCHAs and permission popups may be visible ("browser chrome may be incomplete in window scope") and routes through recovery instructions suggesting desktop-level capture for prompts the window cannot capture (libs/cua-driver/rust/crates/cua-driver-core/src/window_inspection.rs:47-56).

libs/cua-driver/rust/crates/cua-driver-core/src/session.rs:30-100 libs/cua-driver/rust/crates/cua-driver-core/src/browser/mod.rs:1-37 libs/cua-driver/rust/crates/cua-driver-core/src/browser/prepare.rs:30-60 libs/cua/crates/cua-sdk/src/lib.rs:5-28 libs/cua-driver/rust/crates/cua-driver-core/src/window_inspection.rs:36-56