Home / Browser & computer control / stagehand

browserbase/stagehand

Browser automation SDK (TS, Python, Go) whose act/observe/extract run inside a Chrome extension that drives pages over CDP.

GitHub ↗★ 26kTypeScriptMITcommit c9c8a41 · 2026-10-06homepage ↗

Overview

Stagehand is Browserbase’s SDK for scripting a browser with a mix of normal code and natural-language steps. You call act("click the login button"), observe() or extract(instruction, zodSchema) on a page. Stagehand turns the page into a text snapshot, asks an LLM for one structured decision, and then performs that decision itself with Chrome DevTools Protocol (CDP) commands. The model never drives the mouse directly and never writes code.

At the pinned commit (the v4 line), the architecture is unusual. The application logic does not run in your Node process. It runs in a Manifest V3 Chrome extension, the “Stagehand Runtime”. The extension’s service worker opens its own CDP WebSocket to the browser, captures snapshots, calls the LLM provider from inside the browser, and dispatches input events. The TypeScript, Python and Go SDKs are thin JSON-RPC clients. They load or find the extension, attach to its service-worker target, and tunnel messages through CDP Runtime.evaluate and Runtime.addBinding. All three SDKs share one contract, packages/protocol/stagehand.v4.json.

Stagehand is a tool layer, not an autonomous agent. The SDK has no agent() method at this SHA. Multi-step agents are expected to come from outside: the packages/integrations folder exposes a run / snapshot / screenshot tool surface over MCP or native tools for Claude Code, Codex, Mastra, CrewAI, Vercel AI SDK and others.

Architecture

flowchart LR
  U["Your code"] --> SDK["SDK: Stagehand class"]
  SDK --> RPC["RPCClient (JSON-RPC)"]
  RPC --> CDPC["CDPClient: Runtime.evaluate / addBinding"]
  CDPC --> SW["Extension service worker"]
  SW --> RT["RPCRouter -> controllers"]
  RT --> SVC["act / observe / extract services"]
  SVC --> SNAP["Hybrid snapshot (AX + DOM)"]
  SVC --> LLM["llmService (AI SDK / gateway / client LLM)"]
  SVC --> CACHE["cacheService (Browserbase API)"]
  SVC --> UND["understudy: Page, Locator"]
  UND --> CDP["CDP WebSocket to browser"]
  SNAP --> CDP
  CDP --> TAB["Browser tabs + content script"]

Component	Path	Role
SDK facade	`packages/sdk-ts/src/stagehand.ts`	`Stagehand.create`, `act`, `observe`, `extract`, `experimentalBatch`; each call is one JSON-RPC request
SDK transport	`packages/sdk-ts/src/cdpClient.ts`	Loads/discovers the extension, attaches to its service worker, tunnels JSON-RPC over CDP
Browser factories	`packages/sdk-ts/src/browser/`	`localBrowser` (spawns Chrome with `--remote-debugging-port`) and `browserbase` (cloud sessions)
Protocol	`packages/protocol/`	Zod schemas, method registry, version check, `stagehand.v4.json`
Service worker	`packages/extension/service-worker.ts`	Composes the runtime, router and RPC client inside the extension
Router and controllers	`packages/extension/rpcRouter.ts`, `controllers/`	Maps `stagehand.`, `page.`, `locator.`, `context.` methods to handlers
Services	`packages/extension/services/`	`actService`, `observeService`, `extractService`, `cacheService`, `llmService`
Inference	`packages/extension/inference.ts`, `prompt.ts`	Zod response schemas and prompts for the three primitives
Understudy	`packages/extension/understudy/`	Playwright-like `Page`, `Frame`, `Locator` written directly on CDP; snapshot capture under `a11y/snapshot/`
Content script	`packages/extension/content-script.ts`	Installs locator helper scripts and world markers in every frame
Integrations	`packages/integrations/`	Agent-harness tool surface (MCP stdio server and native tools)

How a request flows

Take await stagehand.act("click the sign-in button"):

Startup. Stagehand.create claims the browser handle and sends stagehand.init over a new RPCClient (stagehand.ts, L191-L230). Before that, the CDP client loads the extension with Extensions.loadUnpacked (or finds it by id), waits for its service_worker target, attaches with Target.attachToTarget, and adds the host binding (cdpClient.ts, L686-L784).
Send. act() resolves the active page and sends stagehand.act with the page id (stagehand.ts). CDPClient.send wraps the message in a Runtime.evaluate that calls globalThis.__stagehandReceiveFromHost in the service worker (cdpClient.ts).
Route. In the extension, startStagehandServiceWorker has wired ChromeRuntimeClient -> RPCClient -> RPCRouter (service-worker.ts). RPCRouter.route dispatches stagehand.act to the controller (rpcRouter.ts), which checks init state, picks the per-call or init-time model and calls actService.act (stagehandController.ts).
Settle and cache. actWithProgress waits for DOM/network quiet, then wraps the work in cacheService.withCache (actService.ts). If a cached action list exists, it is replayed without an LLM.
Snapshot and infer. runActPipeline captures the hybrid snapshot, builds the act prompt and calls getActionFromLLM. The model returns an elementId such as 0-18372 (frame ordinal + backend node id), a method, arguments and a twoStep flag (actService.ts, inference.ts). The id is mapped to an XPath through the snapshot’s xpathMap.
Execute. takeDeterministicAction calls performUnderstudyMethod with the selector (actService.ts). For a click, Locator.click scrolls the node into view, reads DOM.getBoxModel, and sends Input.dispatchMouseEvent moved/pressed/released at the centre (locator.ts).
Second step (optional). If twoStep is true, a second snapshot is diffed against the first and a second inference picks the follow-up action, for example the option inside a custom dropdown.
Return. The ActResult (success, message, actions, token usage, cache metadata) travels back as a JSON-RPC response through the Runtime.addBinding channel.

Key components

SDK-to-extension tunnel

There is no extra port or server. The SDK talks CDP to the browser, attaches to the extension’s service-worker target, and uses two CDP features as a duplex pipe: Runtime.evaluate (host to worker) and a Runtime.addBinding callback (worker to host). Readiness is checked by reading globalThis.__stagehand_runtime, and checkProtocolCompatibility rejects mismatched SDK and extension versions at stagehand.init (stagehandController.ts). If the service worker is not running yet, the SDK opens wake-service-worker.html to start it.

Hybrid snapshot

captureHybridSnapshot builds per-frame indexes from DOM.getDocument and Accessibility.getFullAXTree, prunes structural roles, pierces shadow DOM by default, and stitches iframe outlines into one combinedTree with an id-to-XPath map (capture.ts). Callers can scope it with locator or exclude parts with ignoreLocators. Screenshots are not part of act or observe. Only extract can attach a viewport PNG, and only when options.screenshot is set (extractService.ts).

Inference

Every LLM call goes through generateStructured. It sends a JSON Schema built from a Zod schema as responseFormat: { type: "json_schema" } and parses the result back with Zod (inference.ts). extract makes two calls: the extraction itself, then a metadata call that judges completeness.

LLM routing

llmService.generate picks one of three paths (llmService.ts):

a client-side model, where the SDK registered a generate function and the extension calls back over RPC (llm.generate);
the Browserbase Model Gateway, when there is no provider key but there is a Browserbase session;
a direct Vercel AI SDK provider (OpenAI, Anthropic, Google, Groq or Cerebras; LLMProvider.ts).

Because requests leave from the extension, Anthropic calls set anthropic-dangerous-direct-browser-access (aiSdkClient.ts).

Understudy

understudy/ is a small Playwright-shaped driver written directly on CDP: BrowserContext, Page, Frame, Locator, deepLocator for >> iframe hops, cookies, init scripts, network idle tracking and a Progress deadline object. It has no Playwright or Puppeteer dependency. The SDK’s page.* and locator.* methods are RPC calls into these classes.

Cache and self-heal

withCache sends the raw CDP accessibility tree plus request parameters to the Browserbase API, which computes the key. A hit replays stored actions. Any replay failure falls back to normal inference (cacheService.ts). The cache is enabled only with a Browserbase API key and session id (cacheService.ts). With selfHeal on, a failed action takes a fresh snapshot and one new inference before giving up. TimeoutError is always rethrown (actService.ts).

Extending it

Your own loop. Combine act/observe/extract with ordinary page.goto, locator.click and page.evaluate calls. observe() returns candidate actions that you can pass back to act(action) to execute without inference.
Batch code into the browser. experimentalBatch(callback, input) serialises a function and runs it inside the extension against an in-process Stagehand object, so a sequence of page and act calls costs one SDK round trip (stagehand.ts).
Bring your own model. Pass model.generate to route every inference through your own function. This is how unsupported providers or local models are plugged in.
Agent harnesses. packages/integrations/core defines the run/snapshot/screenshot contract and a stagehand-facade stdio MCP server. The other folders mount it into specific agent frameworks.
Other languages. packages/sdk-python and packages/sdk-go are clients of the same protocol, so a change to an RPC method touches the protocol package first.

Running it

Local. localBrowser.launch() finds Chrome on the machine and starts it with --remote-debugging-port, a temp or given --user-data-dir, optional --proxy-server and --headless (localBrowser.ts). Then it loads the bundled extension from dist/extension/. Chrome must support Extensions.loadUnpacked. Otherwise launch with --load-extension and pass extensionId.
Browserbase. browserbase.launch() creates a cloud session through @browserbasehq/sdk, uploads the extension zip, and exposes stealth, proxies, CAPTCHA solving, persistent contexts and regions as session options. A BROWSERBASE_API_KEY also enables the model gateway and the server-side cache.
Required. Node.js, a Chromium-based browser that allows extensions, and either a provider API key, a Browserbase key, or a client generate function.

Strengths and caveats

Strength: deterministic execution. The LLM only picks an element id and method from a Zod-validated enum. Clicks and typing are plain CDP input events, so actions are easy to log, cache and replay.
Strength: one runtime, three SDKs. Logic lives once in the extension, and the protocol package keeps the TS, Python and Go clients in step.
Strength: cheap prompts. The text snapshot has no screenshot by default, and each primitive is one or two calls.
Caveat: no built-in agent. Planning, memory and retry across steps are your job or the harness’s. prompt.ts still contains operator-style system prompts, but nothing in the SDK uses them.
Caveat: extension required. The browser must accept an unpacked or uploaded MV3 extension with the debugger permission. That rules out some locked-down or managed Chrome setups.
Caveat: some features are Browserbase-only. Action caching, the model gateway, CAPTCHA solving and advanced stealth all depend on Browserbase services. A local run has proxy and profile flags, nothing more.
Caveat: fixed provider list. Direct providers are the five AI SDK factories with allow-listed model ids. Anything else needs the client generate hook.

Sources: code at c9c8a41, deepwiki-open wiki (12 pages), OpenDeepWiki wiki (17 pages), verified Q&A.

How it answers the Browser & computer control questions

Each answer was drafted by a code-reading agent at commit c9c8a41. Its citations were checked mechanically. Compare with the other browser & computer control →

How is the page represented to the model?

answered

The page is represented to the model as a hybrid text snapshot combining the Chrome accessibility tree and the DOM tree. The core mechanism lives in capture.ts, which calls Accessibility.getFullAXTree via CDP for each frame's a11y tree and DOM.getDocument for the DOM structure. These are merged into a single hierarchical text outline where each element is rendered as a line like [0-18372] button: Submit — the encoded ID (frameOrdinal-backendNodeId) serves as the element index that the LLM returns when choosing an action. An xpathMap (a Record<string, string>) maps each encoded ID to an absolute XPath, which the execution layer uses to locate the element for CDP interaction.

Iframes are handled via a multi-step pipeline: buildSessionIndexes calls DOM.getDocument once per CDP session; per-frame maps are sliced from the shared index; and computeFramePrefixes walks the frame tree to compute absolute XPath prefixes for child frames. The per-frame outlines are stitched into a single combinedTree via injectSubtrees in treeFormatUtils.ts. Shadow DOM is pierced by default (pierceShadow: true).

Size limits and pruning: The DOM tree retrieval adaptively retries with shallower depths when CDP's CBOR encoder stack overflows — DOM_DEPTH_ATTEMPTS in domTree.ts tries [-1, 256, 128, 64, 32, 16, 8, 4, 2, 1], and each truncated node is hydrated individually via DOM.describeNode with its own depth fallback. Structural AX roles (generic, none, InlineTextBox) are pruned from the outline. Nodes can also be excluded via ignoreLocators.

Screenshots are optional and only used in extract() when options.screenshot: true. They are captured as raw PNG data (page.screenshot → Page.screenshot) and sent alongside the DOM text as an image content block. The screenshot is not annotated with bounding boxes or set-of-marks overlays — the AnnotatedScreenshotText constant in LLMClient.ts describes a planned annotation feature but the screenshotScripts/index.ts only exports resolveMaskRect.

packages/extension/understudy/a11y/snapshot/capture.ts:60-90 packages/extension/understudy/a11y/snapshot/domTree.ts:7-15 packages/extension/understudy/a11y/snapshot/a11yTree.ts:20-55 packages/extension/understudy/a11y/snapshot/treeFormatUtils.ts:8-15 packages/extension/services/extractService.ts:107-139 packages/extension/llm/LLMClient.ts:46-48

How are actions executed and how are elements targeted?

answered

Actions are never executed by the LLM directly — the LLM returns a structured decision (elementId, method, arguments, twoStep flag), and actService.ts executes it deterministically via CDP primitives. The twoStep flag enables two-phase actions (e.g., clicking to expand a non-<select> dropdown, then choosing the option).

From decision to execution: The LLM returns an ActInferenceSchema object with an elementId (like "0-18372"). normalizeActInferenceElement looks up the ID in the combinedXpathMap to get an XPath, then wraps it as xpath=/.... The takeDeterministicAction function calls performUnderstudyMethod (actHandlerUtils.ts), which resolves the XPath to a Locator object (via resolveLocatorWithHops for cross-iframe support) and dispatches to the appropriate method handler.

Method handlers are mapped in METHOD_HANDLER_MAP and include: click, doubleClick, fill, type, press (key), scrollTo, scrollIntoView, mouse.wheel, nextChunk/prevChunk (scroll by element height), selectOption, hover, dragAndDrop.

CDP-level execution: The Locator class (understudy/locator.ts) resolves the selector to an objectId inside an isolated world (Page.createIsolatedWorld), then uses CDP commands: DOM.scrollIntoViewIfNeeded, DOM.getBoxModel (to find the element's center coordinates), and Input.dispatchMouseEvent for clicks (with mouseMoved/mousePressed/mouseReleased events). Typing uses Input.insertText (efficient) or per-character Input.dispatchKeyEvent. Filling uses a JavaScript function injected via Runtime.callFunctionOn that sets the element's value; if the element needs IME/character-level input, it falls back to Input.insertText. File uploads construct File objects in-page via assignFilePayloadsToInputElement and assign them to <input type="file">. Scrolling uses Runtime.callFunctionOn with JavaScript that scrolls the element/window by its height.

Selectors: The LLM returns encoded element IDs, which are resolved to xpath=... selectors. The Locator class supports CSS selectors, XPath expressions, and >>-delimited iframe hops (via deepLocator.ts). Coordinates are never sent to the LLM — they are computed server-side from DOM.getBoxModel.

Tabs: page.goto, page.goBack, page.goForward use CDP Navigation.goto/Navigation.goBack. The BrowserContext manages pages as top-level CDP targets.

packages/extension/services/actService.ts:286-328 packages/extension/handlers/handlerUtils/actHandlerUtils.ts:47-116 packages/extension/understudy/locator.ts:393-464 packages/extension/understudy/locator.ts:680-727 packages/extension/types/private/handlers.ts:1-14 packages/extension/handlers/handlerUtils/actHandlerUtils.ts:120-137

How is the agent loop / planning implemented?

answered

Stagehand does not implement a traditional autonomous agent loop (no planner/executor loop within the SDK itself). Instead, it provides a tool-call API designed to be used from external agent frameworks. The core operations are act(), observe(), and extract(), each making one (or two) LLM calls per invocation.

act() service flow (actService.ts):

Wait for DOM/network quiet via waitForDomNetworkQuiet (CDP Network events)
Capture a hybrid snapshot (page.captureSnapshot)
Call the LLM with buildActPrompt + the snapshot — the LLM returns an ActInferenceSchema (elementId, method, args, twoStep flag)
Execute the action deterministically via CDP (takeDeterministicAction)
If twoStep is true, diff the new snapshot against the old one (diffCombinedTrees), make a second LLM call (buildStepTwoPrompt), and execute a follow-up action

LLM calling (inference.ts): All calls use structured output via responseFormat: { type: "json_schema", name, schema }. The schemas are defined as Zod objects — ActInferenceSchema, ObservationSchema, ExtractMetadataSchema — and serialized to JSON Schema via z.toJSONSchema(). This is JSON Schema-driven structured generation (not tool calling).

observe() captures a snapshot and asks the LLM to return an array of candidate actions (element + method + args) matching the user's instruction.

extract() captures a snapshot, calls the LLM for structured data extraction against a user-provided Zod schema, then makes a second LLM call for a metadata judgment on whether extraction is complete.

External agent loop: The prompt.ts file contains buildOperatorSystemPrompt and buildGoogleCUASystemPrompt — system prompts for an external agent that calls act()/extract()/goto() as tools in a loop. But Stagehand itself is not that loop; it's the tool layer beneath it.

Memory between steps: There is no built-in memory system — each act()/observe()/extract() call is stateless with respect to the LLM. The full snapshot is re-captured on each call. Variables (%varName% placeholders) can be substituted into action arguments before execution.

packages/extension/services/actService.ts:152-246 packages/extension/inference.ts:60-85 packages/extension/inference.ts:94-125 packages/extension/prompt.ts:225-260 packages/extension/prompt.ts:289-321

How are failures, retries and self-healing handled?

answered

Self-healing: When a deterministic action fails (any non-TimeoutError exception), and selfHeal: true is set at init, actService.ts re-captures the full page snapshot and re-asks the LLM for a new action decision (selfHealAction). This handles cases where the DOM changed between capture and execution. TimeoutError is NOT caught — it propagates up to the caller as a fatal timeout.

Error classes (errors.ts): TimeoutError, StagehandProtocolCompatibilityError, DuplicatePageEventSubscriptionError, ShadowRootEvaluationError, ShadowRootEvaluationUnavailableError.

Caching (cacheService.ts): Uses the Browserbase API's stateless cache routes. On each act()/observe()/extract():

The raw CDP accessibility tree(s) are collected (collectCdpTree)
A cache key is computed server-side from the CDP tree + request params
On hit: the cached action array is replayed deterministically — if replay fails (e.g., stale selectors), it falls back to execution
On miss: the LLM inference runs and the result is persisted
The cache has a hit-count threshold and supports bypass for requests with locator scoping
Cache read/write failures are best-effort: they log warnings but never break the operation
No caching of successful workflows; caching is per single act/observe/extract

Timeouts: Each act()/observe()/extract() has a configurable timeout (passed through options.timeout). The Progress class in progress.ts enforces a deadline and provides an abort signal. waitForDomNetworkQuiet uses a 500ms quiet-after-last-network-request heuristic with a 2s stale-request sweep. The timeoutConfig.ts and DEFAULT_LOCATOR_TIMEOUT_MS set baseline defaults.

DOM settle: Before each snapshot capture, waitForDomNetworkQuiet uses CDP Network events (Network.requestWillBeSent, Network.loadingFinished, etc.) to wait until no requests are in-flight for 500ms. Stalled requests older than 2s are force-completed.

packages/extension/services/actService.ts:370-397 packages/extension/services/actService.ts:399-464 packages/extension/services/cacheService.ts:206-240 packages/extension/services/cacheService.ts:266-310 packages/extension/errors.ts:1-36 packages/extension/handlers/handlerUtils/actHandlerUtils.ts:459-613

Which models are supported and how are they called?

answered

Stagehand supports five providers with explicitly allowlisted model IDs defined in packages/protocol/schemas.ts:

OpenAI: gpt-4.1 family, gpt-4o, o1/o3/o4-mini, gpt-5 family (gpt-5 through gpt-5.6)
Anthropic: claude-3-haiku, claude-haiku-4-5, claude-opus-4 through 4.8, claude-sonnet-4 through 4.6, claude-fable-5, claude-sonnet-5
Google (Gemini): gemini-2.0-flash, gemini-2.5-pro/flash, gemini-3 variants, gemma-3
Groq: llama-3.x, gemma2, mixtral, deepseek-r1-distill, qwen, kimi-k2
Cerebras: llama3.1-8b, gpt-oss-120b, qwen-3 variants, zai-glm

Provider resolution (LLMProvider.ts): Model names use the format {provider}/{modelId} (e.g., openai/gpt-4o, anthropic/claude-sonnet-4-5). The provider is extracted from the prefix, and the model is created via the Vercel AI SDK (@ai-sdk/openai, @ai-sdk/anthropic, @ai-sdk/google, @ai-sdk/groq, @ai-sdk/cerebras).

Structured output / tool calling: All LLM calls use JSON Schema structured output (responseFormat: { type: "json_schema", name, schema }). The Zod schemas are converted to JSON Schema via z.toJSONSchema(). This is not tool calling (function calling) — it's native JSON Schema mode supported by OpenAI, Anthropic, and Gemini.

Vision: Vision is optional. When extract(options.screenshot: true), the screenshot is sent as an image content block alongside the DOM text. The vision requirement is per-model (e.g., Gemini models generally support it). The LLMClient abstract class tracks hasVision: boolean.

Model Gateway / Auto-selection: When using Browserbase cloud sessions without an explicit API key, createGatewayLanguageModel routes through Browserbase's Model Gateway (an OpenAI-compatible proxy at {apiUrl}/llm). With modelName: "auto", Browserbase selects the model automatically.

Client-side LLM: The SDK client can register a model.generate function. If a model with source: "client" is configured, clientLlmClient.ts delegates generation to the connected client rather than a built-in provider.

Model resolution fallback: In stagehandController.ts, the per-call options.model overrides the instance-level initParams.model. If neither is set, the Browserbase gateway must be available.

packages/protocol/schemas.ts:9-76 packages/protocol/schemas.ts:78-100 packages/protocol/schemas.ts:182-213 packages/extension/llm/LLMProvider.ts:22-28 packages/extension/llm/LLMProvider.ts:78-113 packages/extension/services/llmService.ts:13-36

How are browser sessions, profiles, auth and anti-bot handled?

answered

Local browser (localBrowser.ts): The SDK launches a Chrome/Chromium process with CDP remote debugging enabled (--remote-debugging-port). It auto-detects Chrome on the system (macOS: canonical paths; Windows: Program Files; Linux: PATH). Configurable via LocalBrowserLaunchOptions: headless, viewport, proxy, locale, userDataDir (persistent profile), args (extra flags). Default flags disable background networking, component updates, sync, hang monitor, and prompt-on-repost. A data directory is created per-session (temp dir) and cleaned up on close unless preserveUserDataDir: true. The sandbox is disabled on Linux root or CI.

Browserbase cloud (browserbaseSession.ts): Sessions are created via @browserbasehq/sdk with extensive configuration:

Stealth/fingerprint: advancedStealth, fingerprint (browser/device/OS/locale/screen simulation)
Proxies: Browserbase managed proxies (BrowserbaseProxyConfigSchema with geolocation) or external proxies (ExternalProxyConfigSchema with server/credentials)
Regions: us-west-2, us-east-1, eu-central-1, ap-southeast-1
CAPTCHA handling: solveCaptchas, captchaInputSelector, captchaImageSelector
Persistence: context.id + context.persist for persistent browser contexts across sessions
Recording: recordSession, logSession
Verification: verified sessions for high-trust sites
BlockAds, os, viewport

Extension provisioning (browserbaseExtension.ts): The Stagehand extension (packed as a .zip archive) is uploaded to Browserbase and attached to the session. Uploads retry up to 4 times with backoff.

Cookies and headers: BrowserContext in the understudy (context.ts) supports addCookies, clearCookies, setExtraHTTPHeaders, and setDomainPolicy (for domain-specific fetch interception). Cookies are manipulated via the CDP Web.ARC or native cookie stores.

Connection modes: The client can either create a new session (with an optionally provisioned extension) or reconnect to an existing session by session ID (connectSession). The CDP URL (webSocketDebuggerUrl) is used to establish the WebSocket transport.

Lifecycle: The Stagehand.create() to stagehand.close() lifecycle ensures browser claims are released and sessions are invalidated on ambiguous init failures (via claimStagehandBrowser/releaseStagehandBrowser/invalidateStagehandBrowser in factories.ts).

packages/sdk-ts/src/browser/localBrowser.ts:14-43 packages/sdk-ts/src/browser/localBrowser.ts:104-167 packages/sdk-ts/src/browser/browserbaseSession.ts:72-133 packages/protocol/schemas.ts:888-963 packages/sdk-ts/src/browser/factories.ts:244-256 packages/extension/understudy/context.ts:86-99