hyperbrowserai/HyperAgent

The page is represented to the model primarily through the CDP accessibility tree, serialized as a hierarchical text tree with encoded element IDs in the format frameIndex-backendNodeId (e.g., "0-5125"). The main entry point is getA11yDOM() (src/context-providers/a11y-dom/index.ts:908), which calls Accessibility.getFullAXTree via CDP for the main frame and all iframes — same-origin iframes via contentDocumentBackendNodeId, out-of-process iframes (OOPIFs) via separate CDP sessions discovered through FrameContextManager.captureOOPIFs (src/context-providers/a11y-dom/index.ts:993). Raw AXNode arrays are converted to AccessibilityNode objects by buildHierarchicalTree() (src/context-providers/a11y-dom/build-tree.ts:71), then serialized into a simplified text tree via formatSimplifiedTree — this text is what the LLM sees in the === Elements === section of the prompt (src/agent/messages/builder.ts:63).

When enableVisualMode is true, bounding boxes are collected in batch via batchCollectBoundingBoxesWithFailures and a visual overlay is rendered by renderA11yOverlay() (src/context-providers/a11y-dom/index.ts:1187), which composites bounding boxes with encoded IDs over a CDP screenshot using Jimp. The overlay is composited with the raw screenshot via compositeScreenshot() (src/agent/tools/agent.ts:65). The composited screenshot is passed to the LLM as a base64 PNG alongside the text tree (src/agent/messages/builder.ts:69).

DOM caching is handled by domSnapshotCache — a snapshot is cached when useCache is true and visual mode is off, and invalidated on navigation/frame events via markDomSnapshotDirty() (src/context-providers/a11y-dom/dom-cache.ts). The DOM capture has 3 retry attempts for transient CDP errors (src/agent/shared/dom-capture.ts:11), with recovery for "Execution context was destroyed" / "Target closed" errors. Streaming mode (enableDomStreaming) sends frame chunks progressively via FrameChunkEvent with ordered assembly in DomChunkAggregator (src/agent/shared/dom-capture.ts:28).

Actions are executed CDP-first, falling back to Playwright when CDP is unavailable. The central dispatch is performAction() (src/agent/actions/shared/perform-action.ts:20). When ctx.cdpActions is true and a backendNodeMap exists, it uses CDP: resolveElement() (src/cdp/element-resolver.ts:35) resolves an encoded ID (frameIndex-backendNodeId) to a {session, frameId, backendNodeId, objectId} via DOM.resolveNode, with XPath-based recovery when backendNodeIds go stale (calling DOM.describeNode after evaluating an XPath in the correct execution context). The resolved element is then dispatched by dispatchCDPAction() (src/cdp/interactions.ts) which uses CDP Input.dispatchMouseEvent / Input.dispatchKeyEvent / Input.dispatchDragEvent for clicks, typing, and fills, and JavaScript evaluation for scroll actions.

CDP methods (defined in src/cdp/interactions.ts:17-31) include: click, hover, fill, type, press, check, uncheck, selectOptionFromDropdown, scrollToElement, scrollToPercentage, nextChunk, prevChunk. When CDP is unavailable, the system builds a Playwright locator via getElementLocator() (src/agent/shared/element-locator.ts) using XPaths from xpathMap, resolving frame indices to Playwright frames, then calls executePlaywrightMethod() (src/agent/shared/execute-playwright-method.ts) with a 3500ms click timeout.

Element targeting uses the encoded ID scheme frameIndex-backendNodeId. The actElement action schema requires elementId, method (one of 12 CDP methods), and arguments (string array) (src/agent/actions/act-element.ts:12-36). Scrolling also has a specialized ScrollAction (src/agent/actions/scroll.ts:4) for up/down/left/right by viewport height via window.scrollBy. File upload, tab management, and page navigation actions (goToUrl, pageBack, pageForward, refreshPage) are defined as separate action definitions (src/agent/actions/index.ts:29-43). Tab-following is handled by setupHyperPage() which maintains a page stack — when the active page opens a new tab, the stack auto-pushes; when a tab closes it auto-splices (src/agent/index.ts:1482-1558).

The core agent loop is runAgentTask() in src/agent/tools/agent.ts:208. It is a single-step loop with no separate planner/executor distinction — each iteration captures DOM state, builds messages, invokes the LLM for one action, executes it, and repeats. Each LLM call returns a structured output validated via Zod: {thoughts: string, memory: string, action: union-schema} (src/types/agent/types.ts:6-21). The thoughts field captures the model's reasoning, memory accumulates state changes (e.g., "Clicked login button → login form appeared"), and action selects from available action types via a Zod discriminated union built by getActionSchema() (src/agent/tools/agent.ts:100).

Messages are constructed by buildAgentStepMessages() (src/agent/messages/builder.ts:9) which appends: the final goal, current URL, variables, previous action history (assistant thought/memory/action → user result), the DOM tree, and optionally a composited screenshot with scroll info. The system prompt (src/agent/messages/system-prompt.ts) describes input/output format, available actions, and guidelines (one action per step, use wait when unsure, avoid loops).

Stop conditions are checked at each iteration: TaskStatus.PAUSED sleeps 100ms and continues; endTaskStatuses (COMPLETED, FAILED, CANCELLED) breaks; maxSteps threshold cancels the task (src/agent/tools/agent.ts:303-306); and a consecutive failure/waits counter of 5 (MAX_CONSECUTIVE_FAILURES_OR_WAITS) fails the task if exceeded (src/agent/tools/agent.ts:267). The loop also tracks schema validation errors — up to 3 are fed back into the next step's messages for cross-step learning (src/agent/tools/agent.ts:400-413). LLM structured output has up to 3 retry attempts per step with Zod error feedback appended as user messages (src/agent/tools/agent.ts:424-508).

Reliability is handled through multiple layers of retries, error classification, and action caching. The general-purpose retry() utility (src/utils/retry.ts:2) provides exponential backoff (2^i * 1000ms) with up to 3 attempts — used for LLM invocations, DOM state capture, and scroll info retrieval. DOM capture has its own retry loop (src/agent/shared/dom-capture.ts:105) with 3 attempts for transient CDP errors ("Execution context was destroyed", "Cannot find context", "Target closed"), plus a placeholder-snapshot detector via isPlaceholderSnapshot() that retries on "Error: Could not extract accessibility tree".

LLM structured output failures get 3 attempts per step with Zod validation errors fed back as corrective user messages (src/agent/tools/agent.ts:424-508). Schema errors accumulate across steps (last 3 tracked) and are injected into future step messages to guide the model (src/agent/tools/agent.ts:400-413). The consecutive failure/waits counter terminates tasks after 5 consecutive waits or action failures, with a clear error message (src/agent/tools/agent.ts:267, 575-624).

Action caching records each step's XPath, frame index, method, and arguments via buildActionCacheEntry() (src/agent/shared/action-cache.ts:95). Successful tasks produce an ActionCacheOutput that can be replayed via runFromActionCache() (src/agent/index.ts:519-834), which replays cached actions by dispatching through perform* helpers with XPath retry support and fallback to instruction-based execution. Element resolution has its own recovery: if DOM.resolveNode fails with "Could not find node with given id", the system falls back to XPath-based recovery via recoverBackendNodeId() (src/cdp/element-resolver.ts:228) which evaluates the XPath in the correct frame's execution context, then calls DOM.describeNode to get a fresh backendNodeId. Page context switching is detected and triggers retry with a 500ms delay (src/agent/index.ts:1537-1558).

The project supports four LLM providers: OpenAI, Anthropic, Gemini, and DeepSeek, all implementing the common HyperAgentLLM interface (src/llm/types.ts:76) with three key methods: invoke(), invokeStructured(), getCapabilities(). Provider selection happens in createLLMClient() (src/llm/providers/index.ts:18) which routes LLMConfig.provider to the correct client constructor. Default is OpenAI (gpt-4o) if no LLM is configured but OPENAI_API_KEY is set (src/agent/index.ts:108-113).

Vision requirement: All providers must support multimodal (vision) input for the screenshot-in-prompt flow — getCapabilities().multimodal returns true for all four (src/llm/providers/openai.ts:218, src/llm/providers/anthropic.ts:127, etc.).

Structured output varies by provider:

• OpenAI uses response_format: {type: "json_schema"} with Zod-jsonSchema conversion via convertToOpenAIJsonSchema() (src/llm/providers/openai.ts:153-206). Temperature is omitted for GPT-5 models at temperature 1.
• Anthropic uses tool calling with two paths: invokeStructuredViaTools() converts action definitions to Anthropic tool schemas and forces tool use (tool_choice: "any") (src/llm/providers/anthropic.ts:133-230); invokeStructuredViaSimpleTool() wraps arbitrary Zod schemas as a single "structured_output" tool (src/llm/providers/anthropic.ts:236-283).
• Gemini uses responseMimeType: "application/json" with responseSchema for JSON mode (src/llm/providers/gemini.ts).
• DeepSeek reuses the OpenAI SDK (src/llm/providers/deepseek.ts:23) with baseURL: "https://api.deepseek.com" and supports response_format for JSON mode.

Custom baseURL is supported for OpenAI and DeepSeek, enabling proxy/compatible endpoints. The getCapabilities() method reports multimodal, toolCalling, and jsonMode flags — the agent loop uses invokeStructured() exclusively for deterministic action parsing with Zod validation.

Browser sessions are managed through the abstract BrowserProvider<T> base class (src/types/browser-providers/types.ts:3), with two concrete implementations: LocalBrowserProvider (src/browser-providers/local.ts) and HyperbrowserProvider (src/browser-providers/hyperbrowser.ts). Selection is controlled by HyperAgentConfig.browserProvider (default: "Local") (src/agent/index.ts:81-83, 124).

LocalBrowserProvider launches Playwright's Chromium with channel: "chrome", headless: false, and the stealth flag --disable-blink-features=AutomationControlled (src/browser-providers/local.ts:13-18). This flag helps evade anti-bot detection by suppressing the "Chrome is being controlled by automated software" banner. Additional launch options can be passed via localConfig.

HyperbrowserProvider (src/browser-providers/hyperbrowser.ts) connects to cloud browser sessions via the @hyperbrowser/sdk. It creates a remote session through the Hyperbrowser API (client.sessions.create()), then connects Playwright to it via chromium.connectOverCDP(session.wsEndpoint) (src/browser-providers/hyperbrowser.ts:35-41). Session details (live URL, session ID, info URL) are logged in debug mode. On close, it both closes the Playwright browser and calls client.sessions.stop() to terminate the cloud session (src/browser-providers/hyperbrowser.ts:58-63).

Profiles, cookies, and auth are managed through raw Playwright BrowserContext — the agent creates one context per browser session with viewport: null (src/agent/index.ts:160-162). The context is not configured with a persistent profile directory by default; users can pass a Playwright-provided context via initPage into executeTask()/executeTaskAsync() for session reuse. Anti-bot beyond the AutomationControlled flag is not implemented — the system prompt instructs the agent to "Accept or close" cookie banners and to try refreshing or alternative approaches for CAPTCHAs (src/agent/messages/system-prompt.ts:74-77). Proxies are not configured in the agent layer but can be passed through Playwright launch options via localConfig or hyperbrowserConfig, which are passed through to the underlying chromium.launch() / chromium.connectOverCDP() calls.