Home / Browser & computer control / typesafe-computer-use

awlevin/typesafe-computer-use

macOS computer-use loop in which a TypeSafe classifier picks each action from OCR and accessibility state; an LLM writes only free text.

GitHub ↗★ 1.2kPythonMITcommit 44ca11f · 2026-09-29

Overview

typesafe-computer-use, called “jev” in its own code, is a computer-use agent that drives a real Mac toward a goal typed in plain English. Its one idea is to take the frontier model out of the per-step loop. Each step reads the screen deterministically (Vision OCR plus the accessibility tree), code adds the facts a model would otherwise have to work out, and a small hosted classifier, TypeSafe, picks the next action from a fixed menu. An LLM, called the “writer”, runs only when a text field needs free text, when a URL outside a small catalog is needed, or when the classifier stops and someone has to read the screen and say what happened.

The project’s own measurement puts a decision at about $0.0002 and 0.13 to 0.38 s of model latency, against roughly $0.03 and 5 s for a frontier model reading a screenshot. The README is frank about the price of that: everything the big model would do for free, such as comparing event dates, has to be rebuilt as deterministic state in code.

It is not a browser library in the Playwright sense. The main path (clicker) clicks real pixels and presses real keys through Quartz, and it reaches websites by bringing Chrome to the front with AppleScript. There is a separate, opt-in browser backend that reads the DOM over raw CDP, and an OSWorld adapter that runs the same loop against an Ubuntu VM for benchmarking. Windows support is marked experimental and untested.

Architecture

flowchart LR
  CLI["clicker CLI"] --> RUN["runner.run step loop"]
  RUN --> CAP["perception.capture"]
  CAP --> AD["platform_adapter.desktop"]
  AD --> MAC["macos.py"]
  AD --> WIN["windows.py"]
  AD --> OSW["osworld/desktop.py"]
  RUN --> PER["perception.perceive: OCR + AX merge"]
  PER --> DEC["decide.decide"]
  DEC --> TS["TypeSafe system_one"]
  RUN --> ACT["actions.perform"]
  ACT --> AD
  ACT --> WR["writer: text and URLs"]
  RUN --> HO["hand_off: answer model"]
  HO --> WR
  BENCH["clicker-bench"] --> BR["browser/ runner over CDP"]
  BR --> TS

Component	Path	Role
CLI	`typesafe_computer_use/cli.py`	`clicker` (run or dry run) and `clicker-inspect` (capture and dump the exact payload)
Step loop	`typesafe_computer_use/runner.py`	Capture, perceive, decide, act; stop rules; hand-off to the writer; run folder
Perception	`typesafe_computer_use/perception.py`	Screenshot, cropped OCR with a tile cache, AX items, merge into one numbered item list
AX walk	`typesafe_computer_use/ax_walk.py`	Bounded, platform-free accessibility-tree walk with pruning rules
Decision	`typesafe_computer_use/decide.py`	State dict, criteria, one multi-`Choice` TypeSafe request, typed-text check
Actions	`typesafe_computer_use/actions.py`	One handler per action kind, each returning a history line
Writer	`typesafe_computer_use/writer.py`, `openai_writer.py`	Structured free-text calls: field text, URLs, the final answer with focus or question
Platform adapters	`platform_adapter.py`, `macos.py`, `windows.py`	The `Desktop` protocol and its macOS and Windows implementations
Browser backend	`typesafe_computer_use/browser/`	Opt-in DOM perception and real CDP input events on a throwaway Chrome
OSWorld agent	`typesafe_computer_use/osworld/`	`JevAgent` and a `Desktop` that turns inputs into pyautogui code for the VM
Tests	`tests/`	Pure-logic tests plus a simulated computer (`world.py`) driven by the real loop

How a request flows

Take uv run clicker "open the Playground" --act:

Start. main loads .env, refuses to run without TYPESAFE_API_KEY, refuses --act without Accessibility permission, builds the writer, and calls run with a RunConfig (cli.py). Without --act it is a one-step dry run.
Loop. run opens a TypeSafeClient, wraps both model clients in call meters, and calls run_step up to --steps times (default 100). When a step stops, hand_off decides whether the run ends or resumes (runner.py).
Capture. capture takes a screenshot (screencapture on macOS) and asks the adapter for the frontmost app and pid, window bounds, focused field and active tab URL (perception.py).
Perceive. perceive runs OCR on the frontmost window’s crop, walks the accessibility tree, drops AX nodes that are not actually drawn, and merges the two sources into one numbered list. It also fills side tables: AX refs for pressing, items covered by a popup, and labelled controls that exist but are off screen (perception.py).
Stall check. screen_moved compares a screen signature with the last one. Three actions in a row that changed nothing end the step as stalled (runner.py).
Decide. decide sends one system_one request with up to four Choice questions: kind, item, site and, when there are hidden controls, offscreen. The state carries the goal, the clock, app, URL, focused field, the last 8 actions, the actions already tried on this screen, and every item with its region, date hint, row-mates and popup (decide.py, L205-L256).
Gate. resolve stops on done or none, or when confidence is under --min-confidence (0.4). For a click, confidence is the minimum of the kind and item answers (runner.py, decide.py).
Act. perform routes the chosen key to click_item, press_offscreen or a handler table (actions.py). Every action returns a history line, and repeating stops the run after two actions in a row that were already tried on the same screen (runner.py).
Hand off. On any stop, hand_off asks the answer model to read the screenshot and screen text. It returns achieved, answer, and optionally a one-move focus (the classifier resumes with it) or a question for the user. The default limits are 10 hand-offs and 3 questions. A third stall with no new page is stuck and final (runner.py).

Key components

Perception: OCR plus accessibility

OCR reads only the frontmost window plus the menu-bar strip above it. A tile cache compares each capture with the previous one at 1/8 scale in 256 px tiles and re-reads only the changed areas, up to four rectangles. The accessibility walk is capped at 4000 nodes or 0.6 s, and it prunes nodes whose frames lie: off-display subtrees, slivers under 4 pt, closed AXMenu trees and nameless groups (ax_walk.py). merge_with_origins pairs each AX control with an overlapping OCR block of matching text. It drops symbol-only OCR blocks that sit on a button (an icon read as ←), cuts the list to 255 items, the TypeSafe Choice ceiling, and renumbers it in reading order (perception.py).

Decision: facts in code, choice in the classifier

The design rule, stated in CONTRIBUTING, is “the classifier picks; code decides facts”. So the state carries computed facts. dates.py adds “dated 2026-10-13 (in 27 days)” to any block with a date. row_mates tells three “Buy” buttons apart by the text in their rows (decide.py). The runner lists the actions that already led back to this screen. The action menu is kept mutually exclusive on purpose, because the author found that every stall came from two options meaning the same thing (decide.py). There is no “click the address bar”. A website is reached only through use_browser plus the site question, with an eight-site catalog and other for the writer (config.py).

Actions and input

click_item first tries AXPress on the element itself, which works under sticky headers. It falls back to a pixel click, closing a covering popup first with its close button or Escape, never with its other buttons (actions.py). Text goes in through AXValue with a read-back, then keystrokes after a clear (actions.py). Unsubmitted text is checked by a second TypeSafe Noul question. Below 0.5 the original value is restored through the same element (actions.py). On macOS, input is raw Quartz events, and every click and keystroke first checks the abort corner (macos.py).

The writer

All free text goes through _structured, which sends a JSON Schema as output_config and, for non-Anthropic endpoints, also writes the schema into the system prompt and turns thinking off (writer.py). Defaults are claude-haiku-4-5 for field text and URLs and claude-sonnet-5 for the answer model. CLICKER_WRITER_BASE_URL with CLICKER_WRITER_API=openai points it at any OpenAI-compatible server (writer.py). Proposed URLs must be https with no whitespace (writer.py). The answer prompt forbids answers from memory and limits a focus to one move the classifier can actually make (writer.py).

Platform adapters

Desktop is a Protocol with two dozen methods: abort, input, apps and windows, capture/OCR/AX, and element actions (platform_adapter.py). pick_host loads windows.py on Windows and macos.py elsewhere. Where that fails, as on Linux, it uses a NoDesktop that raises on any call (platform_adapter.py). using(adapter) swaps in another computer for a block, which is how the OSWorld agent drives a VM without changing the loop (platform_adapter.py).

Browser backend and OSWorld

browser/ is a second, independent loop. It starts a dedicated Chrome with a temporary profile, loopback CDP and one allowed origin (cdp.py), and reads elements with one Runtime.evaluate. It clicks with Input.dispatchMouseEvent at the element centre (act.py). Its request asks kind, element and a satisfied Noul (browser/decide.py). It is reached only through clicker-bench, not clicker. JevAgent runs the unchanged runner.run on a worker thread inside using(OSWorldDesktop(...)). Each step’s inputs are collected as pyautogui code, and the run always reports DONE, never FAIL (osworld/agent.py, desktop.py).

Extending it

A new action. Add it to fixed_actions and to _HANDLERS together (actions.py), and keep it exclusive of the existing ones. The contributing guide asks for a replay note showing the decision before and after.
A new site. Add a key to SITES. Anything not in the catalog is proposed by the writer.
A new platform. Implement every Desktop method (a test holds adapters to the protocol, parameter for parameter) and add a branch to pick_host. The AX walk takes children, attributes and actions as callables, so only those bindings change.
Another writer model. Set CLICKER_WRITER_* and CLICKER_ANSWER_*. CLICKER_WRITER_VISION=false sends a text-only answer model no screenshot.
A task the loop cannot do. Write it as an xfail scenario in tests/test_scenarios.py on the simulated computer in tests/world.py, which runs the real runner.run with a policy in place of the classifier.

Running it

Install. macOS 14+, Python 3.12+, uv sync, and a .env with TYPESAFE_API_KEY (required) and ANTHROPIC_API_KEY (needed for typing, writer URLs and the final answer). The terminal needs Screen Recording and Accessibility permissions.
Run. uv run clicker "goal" is a dry run of one step. --act drives the machine, with --steps, --delay, --min-confidence and --handoffs. Ctrl-C or the top-left screen corner aborts.
Inspect and replay. clicker-inspect captures after a countdown and opens the annotated screen and exact payload (cli.py). Every run writes runs/<timestamp>/ with raw captures, payloads and every probability, and clicker --image <capture> --app ... --url ... re-decides a step offline.
Benchmark. scripts/osworld setup and run-jev chrome/<task> --ocr rapidocr need a Linux host with KVM. The committed record holds 24 scored jev task runs, 19 of them passing, mostly Chrome tasks, with the 52 proxy-dependent tasks deferred.

Strengths and caveats

Strength: cost and latency. One small classifier request per step, with free output tokens and a calibrated confidence to gate on, is a different cost class from screenshot-per-step agents.
Strength: debuggability. Every step saves the capture, the exact payload and the full probability distributions, and a step can be replayed offline. This is better than most agents in this category.
Strength: disciplined boundaries. Facts are computed in code, free text comes only from the writer with a structured reply and a credential guard, and platform calls stay behind one protocol.
Caveat: it drives your real machine. The main path takes over the mouse and keyboard of the logged-in Mac. Using the computer during a run fights it for focus, and only the main display is captured.
Caveat: perception limits. OCR sees only text, and the AX tree only helps in apps that publish one. The README cites Spotify at 0% labelled controls. Identical labels without distinguishing rows split the vote.
Caveat: hard-coded reasoning. Every comparison the classifier cannot make (dates, rows, popups) must be rebuilt in code. New task types tend to need new state, not just a better prompt.
Caveat: vendor dependency. Every decision goes to the hosted TypeSafe API. There is no local or alternative classifier.
Caveat: weak browser story for this category. There are no stealth, proxy, CAPTCHA or session controls. The DOM backend is a benchmark tool, not the default path.

Sources: code at 44ca11f, deepwiki-open wiki (14 pages), OpenDeepWiki wiki (11 pages), verified Q&A.

How it answers the Browser & computer control questions

Each answer was drafted by a code-reading agent at commit 44ca11f. Its citations were checked mechanically. Compare with the other browser & computer control →

How is the page represented to the model?

answered

The page is represented to the model as a numbered list of Items, each carrying text, screen coordinates, and source (OCR, accessibility tree, or both). Perception is a two-source pipeline:

OCR path: perception.py:capture() takes a screenshot via screencapture, crops to the frontmost window plus an 8pt margin and the menu bar strip, then runs macOS Vision OCR (ocrmac's recognize_text). A change-detection system compares the current capture to the previous one at 1/8 scale in 256px tiles; unchanged tiles reuse their OCR lines. Changed tiles are clustered into contiguous blobs, each re-read as a single OCR crop — up to 4 rectangles max, preferring a full re-read above 60% change. Lines below MIN_OCR_CONFIDENCE (0.3) or matching the goal echo are filtered, then merged into blocks by vertical proximity and alignment. (perception.py:84–123, 258–291, 300–319)

Accessibility tree: ax_walk.py:walk_actionable() does a bounded BFS of the frontmost app's AX tree (cap 4000 nodes, 0.6s), collecting labelled on-screen controls. Nodes with real frames wholly off the display are pruned to screen.offscreen (cap 120) — reachable via AXPress, never as click targets. The walk prunes AXMenu subtrees, nameless AXGroup layout boxes, and sub-4pt slivers. (ax_walk.py:134–212)

Merge and budget: OCR blocks and AX controls are merged by box-overlap (≥50% of the smaller box) and text-match (containment or ≥50% shared words). An OCR block centered on a button/link with no alphanumerics is treated as that control's icon and dropped. The merged list is capped at 255 items (TypeSafe's Choice ceiling) — AX-sourced items are kept first, then OCR items by confidence. (perception.py:636–701, config.py:10)

Facts code adds: date-distance hints on blocks containing dates (via dates.py), row-mates for duplicated labels, popup coverage markers, the action already tried on this screen, the focused field, the URL, and the app name. All is rendered into decide.py:base_state() for the classifier. (decide.py:128–162, 91–105)

The browser backend replaces all of the above with a single CDP Runtime.evaluate call — a JS script collects interactive DOM elements, assigns them data-tscu attribute indices, checks occlusion via elementFromPoint, and returns an ordered list with exact text, role, viewport coordinates, and visible page text blocks. No OCR, no accessibility tree, no screenshot permission needed. (browser/perceive.py:29–168, docs/browser-backend.md:1–27)

typesafe_computer_use/perception.py:84-123 typesafe_computer_use/perception.py:258-291 typesafe_computer_use/ax_walk.py:134-212 typesafe_computer_use/decide.py:128-162 typesafe_computer_use/perception.py:636-662 typesafe_computer_use/browser/perceive.py:29-168

How are actions executed and how are elements targeted?

answered

Actions are dispatched through actions.py:perform(), which routes by the classifier's kind Choice into deterministic handlers. Elements are targeted by index into the item list, not by CSS selectors or raw coordinates — the index maps back to the item's pixel center or accessibility element ref.

MacOS native: macos.py posts real Quartz events through CGEventPost to kCGHIDEventTap. A click (click_at) moves the pointer then sends mouse down/up events. Keys (press) use CGEventCreateKeyboardEvent, supporting Command-prefixed keys via kCGEventFlagMaskCommand. Typing emits one unicode keystroke per character. Scrolling parks the pointer at the window center first. (macos.py:74–120)

Accessibility-first clicks: When an item came from the AX tree, click_item first tries ax_press(ref) — this sends AXPress to the control element itself, landing even if covered by a sticky header. If that fails or there's no AX ref, it falls back to a pixel-coordinate mouse click. If the item sits under a popup (tracked in screen.covered), the popup's close button (or Escape) is pressed first. (actions.py:52–70, 73–86)

Text input: fill_field first tries ax_set_value on the focused element — one call, no keystrokes. It verifies by reading the value back: if it ends with the typed text, done. Otherwise it clear_field (Cmd-A, Delete) then types character by character. After typing, a TypeSafe Noul check (verify_typed) scores whether the field holds a sensible value; below 0.5, the original value is restored via AX (restore_field). (actions.py:105–136)

Browser (CDP) path: The browser backend uses real Input.dispatchMouseEvent / Input.insertText CDP events. Elements are targeted by their data-tscu="<index>" attribute, set by the perception JS. A click uses scrollIntoView first if off-screen, then dispatches mouse move/press/release. Typing focuses the element, then calls Input.insertText. (browser/act.py:43–86)

Other actions: use_browser activates the named browser via AppleScript (tell app to activate), then opens a URL from the SITES catalog or from the writer. scroll_down/scroll_up set lines via Quartz scroll wheel events, or pyautogui.scroll in OSWorld. go_back sends Cmd-[ (or Alt-Left on Windows). wait sleeps for 3 seconds, plus the step's own settle delay. (actions.py:138–234)

The OSWorld adapter translates all inputs into pyautogui code strings, accumulated and sent as one step's action list. Clicks become pyautogui.click(x, y), keys become pyautogui.press(...). The text you type reaches the VM only through repr(), never as raw code. (osworld/desktop.py:160–180)

typesafe_computer_use/actions.py:34-49 typesafe_computer_use/macos.py:74-81 typesafe_computer_use/actions.py:52-70 typesafe_computer_use/actions.py:105-136 typesafe_computer_use/browser/act.py:43-64 typesafe_computer_use/osworld/desktop.py:160-180

How is the agent loop / planning implemented?

answered

The agent loop lives in runner.py:run(). It is a synchronous step loop: one capture + percept + decide + act cycle per iteration, up to --steps (default 100).

Step structure: run_step captures the screen (screenshot, app, window, field, URL), runs perceive() to produce the item list, constructs the state dict with goal, date context, previous actions, tried actions, and screen items, then calls decide(). That is a single TypeSafe system_one request with up to four Choice questions: kind (which action), item (which on-screen item — only for click_item), site (which website — only for use_browser), and offscreen (which hidden control — only for press_offscreen). The responses carry full probability distributions and calibrated confidence scores. (runner.py:284–337, decide.py:204–256)

Stop conditions are checked in resolve() after the decision: (1) done or none — goal achieved or impossible; (2) confidence below --min-confidence (0.4); (3) dry run (--act not passed). Then comes the deterministic action via perform(). (runner.py:340–369)

Stall detection uses a screen signature — the tuple of (app, URL, focused field, (text, row) pairs in reading order). Two captures differing by ≤1 line and ≤1-in-10 lines count as the same screen. Three consecutive idle actions (screen unchanged) or two consecutive repeated actions (same action on same screen) stop the run. Three stalls without reaching a new page produce stuck, which is final. (runner.py:372–417, models.py:91–117)

Writer handoff: When the classifier stops, hand_off() calls the writer model (the stronger CLICKER_ANSWER_MODEL, default claude-sonnet-5) with the screenshot, all screen text, earlier screens' text, the stop reason, and earlier stops with their focuses. The writer returns {achieved, answer, focus, question}. If the goal isn't reached, it can set a focus (one concrete next action for the classifier, e.g. "Click the 'Sort by price' button") and the loop resumes with that focus in state. Or it can ask a question for the user. Up to --handoffs 10 handoffs and 3 questions per run. A focus that produces no action leaves the prior answer standing. (runner.py:157–220, writer.py:332–423)

Memory between steps: The classifier receives the last 8 actions as previous_actions, the actions already tried on the current screen as already_tried_on_this_screen, and Guidance (current focus + user replies from the handoff). The writer gets the same plus earlier stops and earlier screens' text. The OCR cache carries one step's raw lines into the next via OcrCache. (decide.py:128–162, runner.py:87–96)

typesafe_computer_use/runner.py:99-145 typesafe_computer_use/runner.py:284-337 typesafe_computer_use/decide.py:204-256 typesafe_computer_use/runner.py:157-220 typesafe_computer_use/runner.py:372-417 typesafe_computer_use/models.py:91-117

How are failures, retries and self-healing handled?

answered

Jev uses deterministic code-side checks and escalation rules rather than action-level retries.

Stall detection (the primary reliability mechanism): Every step computes a signature — (app, URL, focused field, (text, row) tuples). same_screen() allows at most 1 differing text line and that line must be ≤1-in-10, so a clock tick or OCR jitter doesn't mask a stall. After 3 idle actions (screen unchanged — a refused click, a wait, a scroll at the bottom) or 2 repeated actions (same action on same screen — a cycle) the run stops with outcome stalled. After 3 stalls with no new page between them, the run escalates to stuck, which the writer cannot hand back from. (runner.py:24–27, 372–391, 399–417, models.py:91–117)

Error classes caught: Missed is raised when a click's pointer never reached the target — the action line says "click refused: … was not clicked". Abort (mouse in top-left corner or Ctrl-C) stops the loop, as does KeyboardInterrupt. The osworld adapter additionally catches BaseException in its worker thread and hands it back to predict. (models.py:36–41, macos.py:37–40, osworld/agent.py:195–221)

Popup auto-close: When an item sits under a popup (tracked by ax_walk as the covering window), click_item closes the popup first — using its close button when available, otherwise Escape — then clicks the item. Chrome's "Restore pages?" bubble is identified by name and never closed with its Restore button. Popup tracking only works via the OSWorld accessibility tree (the macOS tree doesn't report covering windows). (perception.py:114–119, actions.py:52–86)

Text verification and recovery: After typing into a field, verify_typed runs a TypeSafe Noul asking "did the typing succeed, and is the value sensible?". Below 0.5, restore_field sets the original value back through the same AX element, but only if it still holds exactly what was just typed — no keystroke fallback, since focus may have moved. (actions.py:191–199, 126–136, decide.py:259–276)

No retries, no caching of successful actions: There is no retry-on-failure — a refused action is recorded in history and counts toward the idle limit. The OCR cache (OcrCache) caches raw OCR lines across unchanged tiles, not action outcomes. The already_tried_on_this_screen list prevents re-picking the same fruitless action. The critic is the next capture: nothing in the action description says what came of it, so every action is verified by the screen it produced. (runner.py:393–396, perception.py:164–227)

typesafe_computer_use/runner.py:24-27 typesafe_computer_use/runner.py:372-417 typesafe_computer_use/models.py:91-117 typesafe_computer_use/actions.py:126-136 typesafe_computer_use/decide.py:259-276 typesafe_computer_use/perception.py:164-227

Which models are supported and how are they called?

answered

Jev uses two distinct model tiers for two distinct roles:

Classifier (every step, 86%+ of model calls): The TypeSafe SDK (typesafe.ai) answers a Choice over up to 255 options with a full probability distribution and calibrated confidence. It is a small, fast model: 130–380ms latency, ~$0.0002 per decision, no vision. The classifier never reads pixels or generates text — it receives the structured state dict (goal, items, previous actions, computed facts) and picks one option per Choice. Up to four Choices are answered in a single system_one request. (config.py:10, decide.py:204–256, calls.py:87–107)

Writer (when the classifier needs free text or stops): By default Claude via the Anthropic Messages API. The per-step writer model (CLICKER_WRITER_MODEL, default claude-haiku-4-5) generates field text and proposes URLs. The answer model (CLICKER_ANSWER_MODEL, default claude-sonnet-5) reads the screenshot whenever the classifier stops — a stronger reader for judging goal completion. (config.py:16–17, 51–53, 99–100, writer.py:332–423)

Custom endpoints: CLICKER_WRITER_BASE_URL can point at any Anthropic Messages API or (with CLICKER_WRITER_API=openai) OpenAI Chat Completions endpoint — LM Studio, Ollama, vLLM, DeepSeek, a LiteLLM proxy. The OpenAIWriter adapter steps down response_format (json_schema → json_object → none) and token_limit (max_tokens → max_completion_tokens) when endpoints refuse parameters. Thinking is explicitly disabled for non-Anthropic endpoints. Vision can be toggled off (CLICKER_WRITER_VISION=false) for text-only answer models. (config.py:55–96, openai_writer.py:1–70, docs/writer-endpoints.md:1–52)

Key separation: CLICKER_WRITER_API_KEY only goes to CLICKER_WRITER_BASE_URL. ANTHROPIC_API_KEY only goes to Anthropic's API. On the Anthropic API the key is sent in both x-api-key and Authorization for proxy compatibility. (writer.py:48–63)

Structured output: The writer is called with Anthropic's output_config/json_schema or OpenAI's response_format: json_schema — every call returns a validated, typed dict. Non-Anthropic endpoints also get the schema spelled out in the prompt as fallback. (writer.py:74–113, openai_writer.py:46–69)

The intent is that the classifier takes nearly every model call, and the writer only runs when free text is needed or the classifier stops. The calls: log line tracks the split. (runner.py:143, calls.py:1–84)

typesafe_computer_use/config.py:14-20 typesafe_computer_use/writer.py:37-63 typesafe_computer_use/writer.py:74-113 typesafe_computer_use/openai_writer.py:33-70 typesafe_computer_use/decide.py:204-256 docs/writer-endpoints.md:1-52

How are browser sessions, profiles, auth and anti-bot handled?

answered

The desktop adapter (macOS) drives the user's real machine — it activates apps via AppleScript, captures the screen, and sends synthetic input events through Quartz. There is no isolated browser session: the classifier decides to use_browser, which brings CLICKER_BROWSER (default Google Chrome) to the front via AppleScript activate, then optionally opens a URL via open location. The browser's own cookies, profiles, and extensions are untouched because the loop uses AppleScript's own activation — it never launches a separate Chrome instance. (macos.py:146–172, macos.py:126–128, cli.py:30–38)

Browser backend (CDP): cdp.py:Chrome starts a dedicated, ephemeral Chrome instance with a fresh temporary profile (tempfile.mkdtemp(prefix="tscu-chrome-")). The debugging websocket listens on loopback only (127.0.0.1) and restricts origins to itself. The profile is removed on close(). This never touches the user's profile. Headless mode is available (--headless=new) in addition to headed. (browser/cdp.py:80–87, 94–125, 148–159, docs/browser-backend.md:3)

OSWorld VM: JevAgent runs inside an Ubuntu VM via OSWorld's runner. The OSWorldDesktop adapter communicates through pyautogui code strings. The browser is started per task by OSWorld — activate calls wmctrl -xa google-chrome, starting the browser with subprocess.Popen if it isn't running. Each task gets a fresh VM snapshot via OSWorld's reset(), so no session persists. (osworld/desktop.py:196–201, 256–281, osworld/agent.py:98–106)

Windows adapter: Uses UI Automation's Windows API access — session follows whatever the current Windows desktop has. The browser URL is read from the address bar via UI Automation (Chrome/Edge show it without the scheme). (windows.py:71–73, 1–10, docs/windows.md:1–13)

Anti-bot / stealth / CAPTCHA: None of these are implemented. The README explicitly says passwords are never typed — the user should rely on the browser's password manager or an SSO button the OCR can read. There are no stealth measures, no proxy configuration, no CAPTCHA handling, no user-agent rotation, and no cookie persistence across runs. The CDP chrome starts with --disable-extensions, --disable-sync, --disable-background-networking — these are for determinism in benchmarking, not stealth. (docs/how-a-step-works.md:172, browser/cdp.py:104–112)

typesafe_computer_use/macos.py:146-172 typesafe_computer_use/browser/cdp.py:80-160 typesafe_computer_use/osworld/desktop.py:196-281 docs/browser-backend.md:1-8 docs/how-a-step-works.md:168-176 docs/windows.md:1-22