# browserbase/stagehand

> Browser automation SDK (TS, Python, Go) whose act/observe/extract run inside a Chrome extension that drives pages over CDP.

- Category: [Browser & computer control](https://llms-technical-reviews.com/browser-control/)
- Repository: https://github.com/browserbase/stagehand (reviewed at commit `c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a`, 2026-10-06)
- Stars: 25549 · Language: TypeScript · License: MIT
- Canonical page: https://llms-technical-reviews.com/p/stagehand/

## Overview

Stagehand is Browserbase's SDK for scripting a browser with a mix of normal code and natural-language steps. You call `act("click the login button")`, `observe()` or `extract(instruction, zodSchema)` on a page. Stagehand turns the page into a text snapshot, asks an LLM for one structured decision, and then performs that decision itself with Chrome DevTools Protocol (CDP) commands. The model never drives the mouse directly and never writes code.

At the pinned commit (the v4 line), the architecture is unusual. The application logic does not run in your Node process. It runs in a Manifest V3 Chrome extension, the "Stagehand Runtime". The extension's service worker opens its own CDP WebSocket to the browser, captures snapshots, calls the LLM provider from inside the browser, and dispatches input events. The TypeScript, Python and Go SDKs are thin JSON-RPC clients. They load or find the extension, attach to its service-worker target, and tunnel messages through CDP `Runtime.evaluate` and `Runtime.addBinding`. All three SDKs share one contract, `packages/protocol/stagehand.v4.json`.

Stagehand is a tool layer, not an autonomous agent. The SDK has no `agent()` method at this SHA. Multi-step agents are expected to come from outside: the `packages/integrations` folder exposes a `run` / `snapshot` / `screenshot` tool surface over MCP or native tools for Claude Code, Codex, Mastra, CrewAI, Vercel AI SDK and others.

## Architecture

```mermaid
flowchart LR
  U["Your code"] --> SDK["SDK: Stagehand class"]
  SDK --> RPC["RPCClient (JSON-RPC)"]
  RPC --> CDPC["CDPClient: Runtime.evaluate / addBinding"]
  CDPC --> SW["Extension service worker"]
  SW --> RT["RPCRouter -> controllers"]
  RT --> SVC["act / observe / extract services"]
  SVC --> SNAP["Hybrid snapshot (AX + DOM)"]
  SVC --> LLM["llmService (AI SDK / gateway / client LLM)"]
  SVC --> CACHE["cacheService (Browserbase API)"]
  SVC --> UND["understudy: Page, Locator"]
  UND --> CDP["CDP WebSocket to browser"]
  SNAP --> CDP
  CDP --> TAB["Browser tabs + content script"]
```

| Component | Path | Role |
|---|---|---|
| SDK facade | `packages/sdk-ts/src/stagehand.ts` | `Stagehand.create`, `act`, `observe`, `extract`, `experimentalBatch`; each call is one JSON-RPC request |
| SDK transport | `packages/sdk-ts/src/cdpClient.ts` | Loads/discovers the extension, attaches to its service worker, tunnels JSON-RPC over CDP |
| Browser factories | `packages/sdk-ts/src/browser/` | `localBrowser` (spawns Chrome with `--remote-debugging-port`) and `browserbase` (cloud sessions) |
| Protocol | `packages/protocol/` | Zod schemas, method registry, version check, `stagehand.v4.json` |
| Service worker | `packages/extension/service-worker.ts` | Composes the runtime, router and RPC client inside the extension |
| Router and controllers | `packages/extension/rpcRouter.ts`, `controllers/` | Maps `stagehand.*`, `page.*`, `locator.*`, `context.*` methods to handlers |
| Services | `packages/extension/services/` | `actService`, `observeService`, `extractService`, `cacheService`, `llmService` |
| Inference | `packages/extension/inference.ts`, `prompt.ts` | Zod response schemas and prompts for the three primitives |
| Understudy | `packages/extension/understudy/` | Playwright-like `Page`, `Frame`, `Locator` written directly on CDP; snapshot capture under `a11y/snapshot/` |
| Content script | `packages/extension/content-script.ts` | Installs locator helper scripts and world markers in every frame |
| Integrations | `packages/integrations/` | Agent-harness tool surface (MCP stdio server and native tools) |

## How a request flows

Take `await stagehand.act("click the sign-in button")`:

1. **Startup.** `Stagehand.create` claims the browser handle and sends `stagehand.init` over a new `RPCClient` ([stagehand.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/stagehand.ts#L89-L119), [L191-L230](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/stagehand.ts#L191-L230)). Before that, the CDP client loads the extension with `Extensions.loadUnpacked` (or finds it by id), waits for its `service_worker` target, attaches with `Target.attachToTarget`, and adds the host binding ([cdpClient.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/cdpClient.ts#L340-L399), [L686-L784](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/cdpClient.ts#L686-L784)).
2. **Send.** `act()` resolves the active page and sends `stagehand.act` with the page id ([stagehand.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/stagehand.ts#L234-L246)). `CDPClient.send` wraps the message in a `Runtime.evaluate` that calls `globalThis.__stagehandReceiveFromHost` in the service worker ([cdpClient.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/cdpClient.ts#L408-L430)).
3. **Route.** In the extension, `startStagehandServiceWorker` has wired `ChromeRuntimeClient` -> `RPCClient` -> `RPCRouter` ([service-worker.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/service-worker.ts#L27-L97)). `RPCRouter.route` dispatches `stagehand.act` to the controller ([rpcRouter.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/rpcRouter.ts#L140-L170)), which checks init state, picks the per-call or init-time model and calls `actService.act` ([stagehandController.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/controllers/stagehandController.ts#L64-L93)).
4. **Settle and cache.** `actWithProgress` waits for DOM/network quiet, then wraps the work in `cacheService.withCache` ([actService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L62-L151)). If a cached action list exists, it is replayed without an LLM.
5. **Snapshot and infer.** `runActPipeline` captures the hybrid snapshot, builds the act prompt and calls `getActionFromLLM`. The model returns an `elementId` such as `0-18372` (frame ordinal + backend node id), a method, arguments and a `twoStep` flag ([actService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L152-L246), [inference.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/inference.ts#L60-L85)). The id is mapped to an XPath through the snapshot's `xpathMap`.
6. **Execute.** `takeDeterministicAction` calls `performUnderstudyMethod` with the selector ([actService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L330-L397)). For a click, `Locator.click` scrolls the node into view, reads `DOM.getBoxModel`, and sends `Input.dispatchMouseEvent` moved/pressed/released at the centre ([locator.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/locator.ts#L385-L464)).
7. **Second step (optional).** If `twoStep` is true, a second snapshot is diffed against the first and a second inference picks the follow-up action, for example the option inside a custom dropdown.
8. **Return.** The `ActResult` (success, message, actions, token usage, cache metadata) travels back as a JSON-RPC response through the `Runtime.addBinding` channel.

## Key components

### SDK-to-extension tunnel

There is no extra port or server. The SDK talks CDP to the browser, attaches to the extension's service-worker target, and uses two CDP features as a duplex pipe: `Runtime.evaluate` (host to worker) and a `Runtime.addBinding` callback (worker to host). Readiness is checked by reading `globalThis.__stagehand_runtime`, and `checkProtocolCompatibility` rejects mismatched SDK and extension versions at `stagehand.init` ([stagehandController.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/controllers/stagehandController.ts#L43-L56)). If the service worker is not running yet, the SDK opens `wake-service-worker.html` to start it.

### Hybrid snapshot

`captureHybridSnapshot` builds per-frame indexes from `DOM.getDocument` and `Accessibility.getFullAXTree`, prunes structural roles, pierces shadow DOM by default, and stitches iframe outlines into one `combinedTree` with an id-to-XPath map ([capture.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/a11y/snapshot/capture.ts#L60-L95)). Callers can scope it with `locator` or exclude parts with `ignoreLocators`. Screenshots are not part of `act` or `observe`. Only `extract` can attach a viewport PNG, and only when `options.screenshot` is set ([extractService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/extractService.ts#L97-L150)).

### Inference

Every LLM call goes through `generateStructured`. It sends a JSON Schema built from a Zod schema as `responseFormat: { type: "json_schema" }` and parses the result back with Zod ([inference.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/inference.ts#L94-L125)). `extract` makes two calls: the extraction itself, then a metadata call that judges completeness.

### LLM routing

`llmService.generate` picks one of three paths ([llmService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/llmService.ts#L13-L36)):
- a client-side model, where the SDK registered a `generate` function and the extension calls back over RPC (`llm.generate`);
- the Browserbase Model Gateway, when there is no provider key but there is a Browserbase session;
- a direct Vercel AI SDK provider (OpenAI, Anthropic, Google, Groq or Cerebras; [LLMProvider.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/llm/LLMProvider.ts#L22-L28)).

Because requests leave from the extension, Anthropic calls set `anthropic-dangerous-direct-browser-access` ([aiSdkClient.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/llm/aiSdkClient.ts#L151-L163)).

### Understudy

`understudy/` is a small Playwright-shaped driver written directly on CDP: `BrowserContext`, `Page`, `Frame`, `Locator`, `deepLocator` for `>>` iframe hops, cookies, init scripts, network idle tracking and a `Progress` deadline object. It has no Playwright or Puppeteer dependency. The SDK's `page.*` and `locator.*` methods are RPC calls into these classes.

### Cache and self-heal

`withCache` sends the raw CDP accessibility tree plus request parameters to the Browserbase API, which computes the key. A hit replays stored actions. Any replay failure falls back to normal inference ([cacheService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/cacheService.ts#L206-L300)). The cache is enabled only with a Browserbase API key and session id ([cacheService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/cacheService.ts#L62-L73)). With `selfHeal` on, a failed action takes a fresh snapshot and one new inference before giving up. `TimeoutError` is always rethrown ([actService.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L399-L464)).

## Extending it

- **Your own loop.** Combine `act`/`observe`/`extract` with ordinary `page.goto`, `locator.click` and `page.evaluate` calls. `observe()` returns candidate actions that you can pass back to `act(action)` to execute without inference.
- **Batch code into the browser.** `experimentalBatch(callback, input)` serialises a function and runs it inside the extension against an in-process Stagehand object, so a sequence of page and `act` calls costs one SDK round trip ([stagehand.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/stagehand.ts#L133-L189)).
- **Bring your own model.** Pass `model.generate` to route every inference through your own function. This is how unsupported providers or local models are plugged in.
- **Agent harnesses.** `packages/integrations/core` defines the `run`/`snapshot`/`screenshot` contract and a `stagehand-facade` stdio MCP server. The other folders mount it into specific agent frameworks.
- **Other languages.** `packages/sdk-python` and `packages/sdk-go` are clients of the same protocol, so a change to an RPC method touches the protocol package first.

## Running it

- **Local.** `localBrowser.launch()` finds Chrome on the machine and starts it with `--remote-debugging-port`, a temp or given `--user-data-dir`, optional `--proxy-server` and `--headless` ([localBrowser.ts](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/browser/localBrowser.ts#L185-L215)). Then it loads the bundled extension from `dist/extension/`. Chrome must support `Extensions.loadUnpacked`. Otherwise launch with `--load-extension` and pass `extensionId`.
- **Browserbase.** `browserbase.launch()` creates a cloud session through `@browserbasehq/sdk`, uploads the extension zip, and exposes stealth, proxies, CAPTCHA solving, persistent contexts and regions as session options. A `BROWSERBASE_API_KEY` also enables the model gateway and the server-side cache.
- **Required.** Node.js, a Chromium-based browser that allows extensions, and either a provider API key, a Browserbase key, or a client `generate` function.

## Strengths and caveats

- **Strength: deterministic execution.** The LLM only picks an element id and method from a Zod-validated enum. Clicks and typing are plain CDP input events, so actions are easy to log, cache and replay.
- **Strength: one runtime, three SDKs.** Logic lives once in the extension, and the protocol package keeps the TS, Python and Go clients in step.
- **Strength: cheap prompts.** The text snapshot has no screenshot by default, and each primitive is one or two calls.
- **Caveat: no built-in agent.** Planning, memory and retry across steps are your job or the harness's. `prompt.ts` still contains operator-style system prompts, but nothing in the SDK uses them.
- **Caveat: extension required.** The browser must accept an unpacked or uploaded MV3 extension with the `debugger` permission. That rules out some locked-down or managed Chrome setups.
- **Caveat: some features are Browserbase-only.** Action caching, the model gateway, CAPTCHA solving and advanced stealth all depend on Browserbase services. A local run has proxy and profile flags, nothing more.
- **Caveat: fixed provider list.** Direct providers are the five AI SDK factories with allow-listed model ids. Anything else needs the client `generate` hook.

*Sources: code at c9c8a41, deepwiki-open wiki (12 pages), OpenDeepWiki wiki (17 pages), verified Q&A.*

## How browserbase/stagehand answers the Browser & computer control questions

### How is the page represented to the model? (answered)

The page is represented to the model as a **hybrid text snapshot** combining the Chrome accessibility tree and the DOM tree. The core mechanism lives in `capture.ts`, which calls `Accessibility.getFullAXTree` via CDP for each frame's a11y tree and `DOM.getDocument` for the DOM structure. These are merged into a single hierarchical text outline where each element is rendered as a line like `[0-18372] button: Submit` — the encoded ID (`frameOrdinal-backendNodeId`) serves as the element index that the LLM returns when choosing an action. An xpathMap (a `Record<string, string>`) maps each encoded ID to an absolute XPath, which the execution layer uses to locate the element for CDP interaction.

Iframes are handled via a multi-step pipeline: `buildSessionIndexes` calls `DOM.getDocument` once per CDP session; per-frame maps are sliced from the shared index; and `computeFramePrefixes` walks the frame tree to compute absolute XPath prefixes for child frames. The per-frame outlines are stitched into a single `combinedTree` via `injectSubtrees` in `treeFormatUtils.ts`. Shadow DOM is pierced by default (`pierceShadow: true`).

**Size limits and pruning**: The DOM tree retrieval adaptively retries with shallower depths when CDP's CBOR encoder stack overflows — `DOM_DEPTH_ATTEMPTS` in `domTree.ts` tries `[-1, 256, 128, 64, 32, 16, 8, 4, 2, 1]`, and each truncated node is hydrated individually via `DOM.describeNode` with its own depth fallback. Structural AX roles (`generic`, `none`, `InlineTextBox`) are pruned from the outline. Nodes can also be excluded via `ignoreLocators`.

**Screenshots** are optional and only used in `extract()` when `options.screenshot: true`. They are captured as raw PNG data (`page.screenshot` → `Page.screenshot`) and sent alongside the DOM text as an image content block. The screenshot is *not* annotated with bounding boxes or set-of-marks overlays — the `AnnotatedScreenshotText` constant in `LLMClient.ts` describes a planned annotation feature but the `screenshotScripts/index.ts` only exports `resolveMaskRect`.


Citations: [packages/extension/understudy/a11y/snapshot/capture.ts:60-90](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/a11y/snapshot/capture.ts#L60-L90) · [packages/extension/understudy/a11y/snapshot/domTree.ts:7-15](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/a11y/snapshot/domTree.ts#L7-L15) · [packages/extension/understudy/a11y/snapshot/a11yTree.ts:20-55](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/a11y/snapshot/a11yTree.ts#L20-L55) · [packages/extension/understudy/a11y/snapshot/treeFormatUtils.ts:8-15](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/a11y/snapshot/treeFormatUtils.ts#L8-L15) · [packages/extension/services/extractService.ts:107-139](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/extractService.ts#L107-L139) · [packages/extension/llm/LLMClient.ts:46-48](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/llm/LLMClient.ts#L46-L48)

### How are actions executed and how are elements targeted? (answered)

Actions are **never executed by the LLM directly** — the LLM returns a structured decision (`elementId`, `method`, `arguments`, `twoStep` flag), and `actService.ts` executes it deterministically via CDP primitives. The `twoStep` flag enables two-phase actions (e.g., clicking to expand a non-`<select>` dropdown, then choosing the option).

**From decision to execution**: The LLM returns an `ActInferenceSchema` object with an `elementId` (like `"0-18372"`). `normalizeActInferenceElement` looks up the ID in the `combinedXpathMap` to get an XPath, then wraps it as `xpath=/...`. The `takeDeterministicAction` function calls `performUnderstudyMethod` (`actHandlerUtils.ts`), which resolves the XPath to a `Locator` object (via `resolveLocatorWithHops` for cross-iframe support) and dispatches to the appropriate method handler.

**Method handlers** are mapped in `METHOD_HANDLER_MAP` and include: `click`, `doubleClick`, `fill`, `type`, `press` (key), `scrollTo`, `scrollIntoView`, `mouse.wheel`, `nextChunk`/`prevChunk` (scroll by element height), `selectOption`, `hover`, `dragAndDrop`.

**CDP-level execution**: The `Locator` class (`understudy/locator.ts`) resolves the selector to an `objectId` inside an isolated world (`Page.createIsolatedWorld`), then uses CDP commands: `DOM.scrollIntoViewIfNeeded`, `DOM.getBoxModel` (to find the element's center coordinates), and `Input.dispatchMouseEvent` for clicks (with mouseMoved/mousePressed/mouseReleased events). Typing uses `Input.insertText` (efficient) or per-character `Input.dispatchKeyEvent`. Filling uses a JavaScript function injected via `Runtime.callFunctionOn` that sets the element's value; if the element needs IME/character-level input, it falls back to `Input.insertText`. File uploads construct `File` objects in-page via `assignFilePayloadsToInputElement` and assign them to `<input type="file">`. Scrolling uses `Runtime.callFunctionOn` with JavaScript that scrolls the element/window by its height.

**Selectors**: The LLM returns encoded element IDs, which are resolved to `xpath=...` selectors. The `Locator` class supports CSS selectors, XPath expressions, and `>>`-delimited iframe hops (via `deepLocator.ts`). Coordinates are never sent to the LLM — they are computed server-side from `DOM.getBoxModel`.

**Tabs**: `page.goto`, `page.goBack`, `page.goForward` use CDP `Navigation.goto`/`Navigation.goBack`. The `BrowserContext` manages pages as top-level CDP targets.


Citations: [packages/extension/services/actService.ts:286-328](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L286-L328) · [packages/extension/handlers/handlerUtils/actHandlerUtils.ts:47-116](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/handlers/handlerUtils/actHandlerUtils.ts#L47-L116) · [packages/extension/understudy/locator.ts:393-464](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/locator.ts#L393-L464) · [packages/extension/understudy/locator.ts:680-727](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/locator.ts#L680-L727) · [packages/extension/types/private/handlers.ts:1-14](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/types/private/handlers.ts#L1-L14) · [packages/extension/handlers/handlerUtils/actHandlerUtils.ts:120-137](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/handlers/handlerUtils/actHandlerUtils.ts#L120-L137)

### How is the agent loop / planning implemented? (answered)

Stagehand does **not** implement a traditional autonomous agent loop (no planner/executor loop within the SDK itself). Instead, it provides a **tool-call API** designed to be used from external agent frameworks. The core operations are `act()`, `observe()`, and `extract()`, each making one (or two) LLM calls per invocation.

**`act()` service flow** (`actService.ts`):
1. Wait for DOM/network quiet via `waitForDomNetworkQuiet` (CDP Network events)
2. Capture a hybrid snapshot (`page.captureSnapshot`)
3. Call the LLM with `buildActPrompt` + the snapshot — the LLM returns an `ActInferenceSchema` (elementId, method, args, twoStep flag)
4. Execute the action deterministically via CDP (`takeDeterministicAction`)
5. If `twoStep` is true, diff the new snapshot against the old one (`diffCombinedTrees`), make a second LLM call (`buildStepTwoPrompt`), and execute a follow-up action

**LLM calling** (`inference.ts`): All calls use structured output via `responseFormat: { type: "json_schema", name, schema }`. The schemas are defined as Zod objects — `ActInferenceSchema`, `ObservationSchema`, `ExtractMetadataSchema` — and serialized to JSON Schema via `z.toJSONSchema()`. This is JSON Schema-driven structured generation (not tool calling).

**`observe()`** captures a snapshot and asks the LLM to return an array of candidate actions (element + method + args) matching the user's instruction.

**`extract()`** captures a snapshot, calls the LLM for structured data extraction against a user-provided Zod schema, then makes a second LLM call for a metadata judgment on whether extraction is complete.

**External agent loop**: The `prompt.ts` file contains `buildOperatorSystemPrompt` and `buildGoogleCUASystemPrompt` — system prompts for an external agent that calls `act()`/`extract()`/`goto()` as tools in a loop. But Stagehand itself is not that loop; it's the tool layer beneath it.

**Memory between steps**: There is no built-in memory system — each `act()`/`observe()`/`extract()` call is stateless with respect to the LLM. The full snapshot is re-captured on each call. Variables (`%varName%` placeholders) can be substituted into action arguments before execution.


Citations: [packages/extension/services/actService.ts:152-246](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L152-L246) · [packages/extension/inference.ts:60-85](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/inference.ts#L60-L85) · [packages/extension/inference.ts:94-125](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/inference.ts#L94-L125) · [packages/extension/prompt.ts:225-260](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/prompt.ts#L225-L260) · [packages/extension/prompt.ts:289-321](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/prompt.ts#L289-L321)

### How are failures, retries and self-healing handled? (answered)

**Self-healing**: When a deterministic action fails (any non-`TimeoutError` exception), and `selfHeal: true` is set at init, `actService.ts` re-captures the full page snapshot and re-asks the LLM for a new action decision (`selfHealAction`). This handles cases where the DOM changed between capture and execution. `TimeoutError` is NOT caught — it propagates up to the caller as a fatal timeout.

**Error classes** (`errors.ts`): `TimeoutError`, `StagehandProtocolCompatibilityError`, `DuplicatePageEventSubscriptionError`, `ShadowRootEvaluationError`, `ShadowRootEvaluationUnavailableError`.

**Caching** (`cacheService.ts`): Uses the Browserbase API's stateless cache routes. On each `act()`/`observe()`/`extract()`:
1. The raw CDP accessibility tree(s) are collected (`collectCdpTree`)
2. A cache key is computed server-side from the CDP tree + request params
3. On hit: the cached action array is replayed deterministically — if replay fails (e.g., stale selectors), it falls back to execution
4. On miss: the LLM inference runs and the result is persisted
5. The cache has a hit-count threshold and supports `bypass` for requests with locator scoping
6. Cache read/write failures are best-effort: they log warnings but never break the operation
7. No caching of successful workflows; caching is per single act/observe/extract

**Timeouts**: Each `act()`/`observe()`/`extract()` has a configurable `timeout` (passed through `options.timeout`). The `Progress` class in `progress.ts` enforces a deadline and provides an abort signal. `waitForDomNetworkQuiet` uses a 500ms quiet-after-last-network-request heuristic with a 2s stale-request sweep. The `timeoutConfig.ts` and `DEFAULT_LOCATOR_TIMEOUT_MS` set baseline defaults.

**DOM settle**: Before each snapshot capture, `waitForDomNetworkQuiet` uses CDP Network events (`Network.requestWillBeSent`, `Network.loadingFinished`, etc.) to wait until no requests are in-flight for 500ms. Stalled requests older than 2s are force-completed.


Citations: [packages/extension/services/actService.ts:370-397](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L370-L397) · [packages/extension/services/actService.ts:399-464](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/actService.ts#L399-L464) · [packages/extension/services/cacheService.ts:206-240](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/cacheService.ts#L206-L240) · [packages/extension/services/cacheService.ts:266-310](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/cacheService.ts#L266-L310) · [packages/extension/errors.ts:1-36](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/errors.ts#L1-L36) · [packages/extension/handlers/handlerUtils/actHandlerUtils.ts:459-613](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/handlers/handlerUtils/actHandlerUtils.ts#L459-L613)

### Which models are supported and how are they called? (answered)

Stagehand supports **five providers** with explicitly allowlisted model IDs defined in `packages/protocol/schemas.ts`:

- **OpenAI**: gpt-4.1 family, gpt-4o, o1/o3/o4-mini, gpt-5 family (gpt-5 through gpt-5.6)
- **Anthropic**: claude-3-haiku, claude-haiku-4-5, claude-opus-4 through 4.8, claude-sonnet-4 through 4.6, claude-fable-5, claude-sonnet-5
- **Google (Gemini)**: gemini-2.0-flash, gemini-2.5-pro/flash, gemini-3 variants, gemma-3
- **Groq**: llama-3.x, gemma2, mixtral, deepseek-r1-distill, qwen, kimi-k2
- **Cerebras**: llama3.1-8b, gpt-oss-120b, qwen-3 variants, zai-glm

**Provider resolution** (`LLMProvider.ts`): Model names use the format `{provider}/{modelId}` (e.g., `openai/gpt-4o`, `anthropic/claude-sonnet-4-5`). The provider is extracted from the prefix, and the model is created via the Vercel AI SDK (`@ai-sdk/openai`, `@ai-sdk/anthropic`, `@ai-sdk/google`, `@ai-sdk/groq`, `@ai-sdk/cerebras`).

**Structured output / tool calling**: All LLM calls use JSON Schema structured output (`responseFormat: { type: "json_schema", name, schema }`). The Zod schemas are converted to JSON Schema via `z.toJSONSchema()`. This is not tool calling (function calling) — it's native JSON Schema mode supported by OpenAI, Anthropic, and Gemini.

**Vision**: Vision is optional. When `extract(options.screenshot: true)`, the screenshot is sent as an image content block alongside the DOM text. The `vision` requirement is per-model (e.g., Gemini models generally support it). The `LLMClient` abstract class tracks `hasVision: boolean`.

**Model Gateway / Auto-selection**: When using Browserbase cloud sessions without an explicit API key, `createGatewayLanguageModel` routes through Browserbase's Model Gateway (an OpenAI-compatible proxy at `{apiUrl}/llm`). With `modelName: "auto"`, Browserbase selects the model automatically.

**Client-side LLM**: The SDK client can register a `model.generate` function. If a model with `source: "client"` is configured, `clientLlmClient.ts` delegates generation to the connected client rather than a built-in provider.

**Model resolution fallback**: In `stagehandController.ts`, the per-call `options.model` overrides the instance-level `initParams.model`. If neither is set, the Browserbase gateway must be available.


Citations: [packages/protocol/schemas.ts:9-76](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/protocol/schemas.ts#L9-L76) · [packages/protocol/schemas.ts:78-100](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/protocol/schemas.ts#L78-L100) · [packages/protocol/schemas.ts:182-213](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/protocol/schemas.ts#L182-L213) · [packages/extension/llm/LLMProvider.ts:22-28](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/llm/LLMProvider.ts#L22-L28) · [packages/extension/llm/LLMProvider.ts:78-113](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/llm/LLMProvider.ts#L78-L113) · [packages/extension/services/llmService.ts:13-36](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/services/llmService.ts#L13-L36)

### How are browser sessions, profiles, auth and anti-bot handled? (answered)

**Local browser** (`localBrowser.ts`): The SDK launches a Chrome/Chromium process with CDP remote debugging enabled (`--remote-debugging-port`). It auto-detects Chrome on the system (macOS: canonical paths; Windows: Program Files; Linux: PATH). Configurable via `LocalBrowserLaunchOptions`: `headless`, `viewport`, `proxy`, `locale`, `userDataDir` (persistent profile), `args` (extra flags). Default flags disable background networking, component updates, sync, hang monitor, and prompt-on-repost. A data directory is created per-session (temp dir) and cleaned up on close unless `preserveUserDataDir: true`. The sandbox is disabled on Linux root or CI.

**Browserbase cloud** (`browserbaseSession.ts`): Sessions are created via `@browserbasehq/sdk` with extensive configuration:
- **Stealth/fingerprint**: `advancedStealth`, `fingerprint` (browser/device/OS/locale/screen simulation)
- **Proxies**: Browserbase managed proxies (`BrowserbaseProxyConfigSchema` with geolocation) or external proxies (`ExternalProxyConfigSchema` with server/credentials)
- **Regions**: `us-west-2`, `us-east-1`, `eu-central-1`, `ap-southeast-1`
- **CAPTCHA handling**: `solveCaptchas`, `captchaInputSelector`, `captchaImageSelector`
- **Persistence**: `context.id` + `context.persist` for persistent browser contexts across sessions
- **Recording**: `recordSession`, `logSession`
- **Verification**: `verified` sessions for high-trust sites
- **BlockAds**, `os`, `viewport`

**Extension provisioning** (`browserbaseExtension.ts`): The Stagehand extension (packed as a `.zip` archive) is uploaded to Browserbase and attached to the session. Uploads retry up to 4 times with backoff.

**Cookies and headers**: `BrowserContext` in the understudy (`context.ts`) supports `addCookies`, `clearCookies`, `setExtraHTTPHeaders`, and `setDomainPolicy` (for domain-specific fetch interception). Cookies are manipulated via the CDP `Web.ARC` or native cookie stores.

**Connection modes**: The client can either create a new session (with an optionally provisioned extension) or reconnect to an existing session by session ID (`connectSession`). The CDP URL (`webSocketDebuggerUrl`) is used to establish the WebSocket transport.

**Lifecycle**: The `Stagehand.create()` to `stagehand.close()` lifecycle ensures browser claims are released and sessions are invalidated on ambiguous init failures (via `claimStagehandBrowser`/`releaseStagehandBrowser`/`invalidateStagehandBrowser` in `factories.ts`).


Citations: [packages/sdk-ts/src/browser/localBrowser.ts:14-43](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/browser/localBrowser.ts#L14-L43) · [packages/sdk-ts/src/browser/localBrowser.ts:104-167](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/browser/localBrowser.ts#L104-L167) · [packages/sdk-ts/src/browser/browserbaseSession.ts:72-133](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/browser/browserbaseSession.ts#L72-L133) · [packages/protocol/schemas.ts:888-963](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/protocol/schemas.ts#L888-L963) · [packages/sdk-ts/src/browser/factories.ts:244-256](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/sdk-ts/src/browser/factories.ts#L244-L256) · [packages/extension/understudy/context.ts:86-99](https://github.com/browserbase/stagehand/blob/c9c8a41778b2000c9a9bdfc4b68e6c0c4866ab1a/packages/extension/understudy/context.ts#L86-L99)