omdsh-dev/dsh-browser

Overview

dsh-browser connects DeepSeek Harness (dsh) to the Chrome or Firefox tabs you already have open. It ships as one pnpm workspace with two halves: a dsh plugin, @yuxianglin/dsh-bridge-browser, that registers 15 browser_* tools with the agent, and an MV3 extension that carries out those tools in your real browser profile. A side panel in the extension doubles as a chat client for dsh. Logins, cookies and open tabs are the user’s own, because no new browser is launched.

The design is deliberately text-only. A page reaches the model as a structured text snapshot: title, URL, a main-content excerpt, a numbered list of interactive elements and a list of form fields. The model then acts by number (“click 7”). No screenshots are taken for page control, and the extension does not use CDP or the debugger permission. Every action is a DOM operation performed by a content script.

This is a tool layer, not an agent. The planning loop, history, model choice and tool calling all belong to dsh. The repository adds the bridge, the extension, an approval system for state-changing actions, and a benchmark suite that compares the extension backend with a Playwright-based backend that uses the same tool contract.

Architecture

flowchart LR
  M["dsh agent loop"] --> T["browser_* tools (plugin)"]
  T --> S["BridgeServer /ext/bridge"]
  S -->|"tool.call over WebSocket"| B["Extension service worker"]
  B --> A["Approval policy + side panel"]
  B --> R["Tab affinity: controlled tab"]
  B -->|"chrome.tabs.sendMessage"| C["Content script per frame"]
  C --> SN["Text snapshot + stable ids"]
  C --> AC["DOM actions: click/type/press"]
  P["Side panel (React)"] -->|"rpc frames"| S
  S --> G["dsh Gateway: sessions, models"]

Component	Path	Role
Plugin entry	packages/browser/bridge-browser/src/index.ts	Cordis plugin: config, token, upgrade route, discovery route, tool registration, system-prompt hint
Tool set	packages/browser/bridge-browser/src/tools.ts	15 browser_* definitions; each forwards to the bridge and returns one text block
Bridge server	packages/browser/bridge-browser/src/server.ts	WebSocket auth (hello), single active connection, tool.call/tool.cancel, gateway RPC passthrough
Wire protocol	packages/browser/bridge-browser/src/protocol.ts	Frame types, error codes, BridgeCaps (text-only, snapshot budgets)
Session helpers	session-deferral.ts, session-workspace.ts, browser-context.ts	Lazy session creation, workspace grouping, injecting a followed tab’s snapshot into the agent
Service worker	extensions/dsh-browser/src/background/index.ts	Bridge client, settings, approvals, tab affinity, tool routing
Tool dispatch	extensions/dsh-browser/src/background/tools.ts	Frame discovery, approval checks, content-script messaging and re-injection, tab-level tools
Authorization	background/authorization.ts, src/security/	Read vs action prompts, origin trust, untrusted-content wrapping
Content script	extensions/dsh-browser/src/content/	Snapshot, stable element ids, privacy masking, DOM actions, settle detection
Side panel	extensions/dsh-browser/src/panel/	React chat UI, approval dialogs, sessions, image attachments
Benchmark	benchmark/	Paired extension vs Playwright runs on a local task site

How a request flows

Take a model call to browser_click with index: 7:

1. Tool call. dsh runs the tool registered in registerBrowserTools, which calls bridge.requestTool(name, args, signal, timeout, sessionId) (tools.ts).
2. Dispatch. requestTool fails fast with bridge-closed if no extension is connected, then sends a tool.call frame with an expiresAt deadline. On timeout or abort it sends tool.cancel, so a late approval click cannot run an expired action (server.ts).
3. Route. In the service worker, routeToolCall builds an abort controller and resolves the session’s controlled tab through resolveToolTab before calling dispatchToolCall (background/index.ts).
4. Validate and authorize. dispatchToolCall lists the tab’s frames, checks that index 7 still refers to a live element in the target document, and asks approvalPromptForCall whether a prompt is needed. If the user approves, it re-lists frames and refuses to proceed when the page changed while the dialog was open (background/tools.ts).
5. Execute. The worker sends a DSH_ACTION message to the right frame with chrome.tabs.sendMessage (background/tools.ts). If the content script is missing, for example in a tab that was open before the extension loaded, it injects it with chrome.scripting.executeScript and retries once.
6. Act. clickAction scrolls the element into view and calls el.click(). Links get special handling: the content script dispatches a cancelable click so SPA routers can intercept it, then sets location.href itself for ordinary same-frame links (actions.ts).
7. Return. After a MutationObserver-based settle wait, the result text includes a delta snapshot of what changed. It travels back as tool.result and becomes a single text content block for the model.

Key components

Text snapshot and stable ids

buildSnapshot collects visible interactive elements, sorts open-dialog elements first and in-viewport elements next, and caps the list at maxItems (60 by default) (snapshot.ts). Ids come from ElementIds, a WeakMap registry that gives each element a number once and keeps it for as long as the element exists. It also writes a data-dsh-el attribute on the element (ids.ts). Because ids survive re-snapshots, renderSnapshot can emit a delta that lists only changed and removed elements, and it warns the model when more than half the set was renumbered (snapshot.ts).

Privacy and prompt-injection boundary

isSensitiveField masks password inputs, cc-* autocomplete fields and any field whose id, name or aria-label matches patterns such as password, card or secret (privacy.ts). The match is coarse, so it also masks harmless fields whose names contain those words. Page text is wrapped in an UNTRUSTED_PAGE_CONTENT block with a fresh nonce, so page content cannot forge the closing tag (untrusted.ts). The code itself calls this defense in depth. The real enforcement is the approval step.

Approvals

approvalPromptForCall classifies reads (browser_snapshot, browser_get_text) and state-changing actions, and computes the origins involved. Cross-origin navigations and back/forward can never be added to the trust list (authorization.ts). authorizeToolCall skips the prompt when unrestricted access is on or the origin is trusted for the session or permanently. Otherwise it shows a side-panel dialog that times out after 60 seconds (background/index.ts). Page reads default to sharePageContent: 'auto', so out of the box only actions prompt (background/index.ts).

Bridge authentication

The first frame must be hello. A loopback socket whose Origin starts with chrome-extension:// may skip the token. Everything else, including Firefox’s per-install moz-extension:// origins, must present the 256-bit bearer token (server.ts). Privileged gateway methods such as settings.* and credentials.* stay loopback-only even with a valid token. Only one extension connection is active at a time, and a new one replaces the old.

Tab affinity and session context

TabAffinityController binds tools to one user-controlled tab per session. When the user switches tabs, dispatch pauses for a keep/follow decision rather than silently acting on the new tab. When the user follows a tab, BrowserContextInjector pushes a fresh snapshot into the agent’s inbox, replacing any older one, so the next turn can use its indices directly (browser-context.ts).

Extending it

• New tools. Add a definition in tools.ts and a matching action in the content script’s runAction switch. The tool name doubles as the wire action name.
• Budgets. snapshotMaxChars, maxInteractiveItems and toolTimeoutMs are plugin config. They are negotiated to the extension in hello.ok, so there is no shared config file (index.ts).
• Model guidance. The plugin adds a one-paragraph system-prompt section telling the model to snapshot on demand and reuse injected snapshots (index.ts).
• Other clients. The protocol is exported as @yuxianglin/dsh-bridge-browser/protocol, so a different extension or client could speak it.

Running it

• Requirements. Node.js ^22.19 or >=24, pnpm, Chrome 116+ or Firefox 140+, and dsh 0.2.0-rc.2 or a compatible newer version. The plugin refuses to mount if the gateway lacks wireStream (index.ts).
• Install. scripts/install.sh or install.ps1 builds the plugin, registers it in the dsh web profile and builds the unpacked Chrome extension, which you then load from chrome://extensions. pnpm start runs dsh web. Firefox needs a source build and a temporary add-on.
• Discovery. With no bridge URL set, the extension probes 127.0.0.1 on a fixed list of dsh web and desktop ports for /ext/bridge-config (background/index.ts).

Strengths and caveats

• Strength: the user’s real session. There is no headless copy and no profile import. Whatever the user is logged into, the model can reach, subject to approval.
• Strength: careful approval semantics. Approvals are origin-scoped, re-validated after the dialog closes, cancelled when the bridge times out, and denied if nobody answers within 60 seconds. With the panel closed, the user gets an OS notification instead of a silent pass.
• Strength: cheap, stable addressing. Indices persist across snapshots and delta output keeps follow-up turns small. The repo’s own paired benchmark shows fewer tool calls than its Playwright baseline (3.4 vs 4.7), though it is a six-task suite on a local site.
• Caveat: synthetic events. Clicks, key presses and typing are DOM calls (el.click(), dispatchEvent, native value setters), so isTrusted is false. Sites that check for trusted input, or that rely on real key events for typing, may not respond. browser_press dispatches only keydown/keyup.
• Caveat: dsh only. The tools are dsh tool definitions and the panel speaks the dsh gateway. Using it with another agent means reimplementing the plugin side.
• Caveat: broad loopback trust. The token-free path accepts any chrome-extension:// origin, not only this extension’s id. Any other installed Chrome extension can therefore connect to a local bridge without the token.
• Caveat: one connection. A second browser window or profile replaces the first connection, and non-loopback deployment is explicitly discouraged.

Sources: code at 23895d7, deepwiki-open wiki (12 pages), OpenDeepWiki wiki (15 pages), verified Q&A.