LLMs Technical Reviews

How are actions on the user's behalf gated?

Approval / human-in-the-loop flows; permission scopes; dry-run or draft modes; audit trail.

Verdict

This is where the two projects differ most.

OpenBot makes governance structural. A ComputerGateway is the only path to a Bot’s computer: it resolves the clicked element from the server’s own snapshot, evaluates CEL deny/allow rules (deny wins, errors deny, dry-run mode available), writes an audit row, and only then acts. An approvals service adds non-configurable safety hand-offs for credentials, security settings and payments, then team and personal rules (allow, pre_approved, ask, hand_off), then an optional LLM auto-review that fails closed.

Rakazo gates at the tool level. toolRequiresApproval exempts computer, browser, shell and file tools, always asks for destructive builtins, and judges connector tools by name prefix (unknown names ask). User rules by tool, connector or category override the default; Auto Review can only escalate to ask. Webhook-triggered runs are stricter. Each effect is recorded with an idempotency key.

OpenBot fits teams that need an audit trail and fine-grained control over what a browser agent clicks. Rakazo fits personal use where the sandbox is the boundary and approval is reserved for outward-facing tool calls.

More projects in this category are being researched.

Per-project answers

CopilotKit/OpenBot

answered

The gateway — mandatory choke point. Every action a Bot takes on its computer goes through ComputerGateway (server/src/computer/gateway.ts:1-15). The gateway has three jobs: (1) resolve the element ref from the server-fetched snapshot (never from what the model claims); (2) evaluate the action policy; (3) write an audit row and only then act. "An action that was not recorded did not happen, because there is no path that acts without writing the row first" is a stated invariant.

Action policy — CEL expressions. The policy (server/src/computer/policy.ts) uses cel-js to evaluate allow/deny rules against a PolicyContext that includes tool name, bot ID, page URL/host, actor ID, element role/name/type, and keyboard key. Deny beats allow. The policy supports dry-run mode (records decisions without blocking) for safe rule authoring. The three policy layers are: (1) CEL expressions (allow/deny lists), (2) approve/ask/hand-off decisions, (3) learning-based classification of action effects.

Approval gate — human-in-the-loop. The approvals service (server/src/approvals/service.ts) evaluates every action against a four-tier model: built-in safety requirements → team rules → personal rules → auto-review → default behavior. Outcomes are allow, ask, hand_off, or deny (server/src/approvals/types.ts:31-52). Safety requirements (server/src/approvals/policy.ts:83-108) automatically flag password changes, security settings, and payments for hand-off using regex patterns — these are non-configurable.

Auto-review model. Risky actions that are not caught by safety rules go through an LLM-based auto-review (server/src/approvals/policy.ts:210-218). The reviewer model gets the action context and user's request, then outputs a structured verdict (proceed | needs_approval | hand_off) with safety classification. It fails closed: errors become needs_approval. Actions marked as pre_approved skip approval if the person explicitly requested them.

Audit trail. Every policy decision, approval request, and action is recorded in the auditEvents table (server/src/audit.ts). Audit event types include everything from configuration.changed and credential.created to detailed agent action records. Sensitive fields (tokens, passwords, credentials, prompts, tool arguments/results) are identified by key name and stripped from audit logs (server/src/audit.ts:17-46).

elie222/rakazo

answered

Approval / human-in-the-loop. All tool calls by the agent go through a gating pipeline in packages/core/src/action-approval.ts. The toolRequiresApproval() function (line 92) classifies each tool into one of four buckets:

  1. Exempt tools (APPROVAL_EXEMPT_TOOLS — line 3-23): computer_observe, browser_navigate, shell, remember, write_file, etc. — run immediately.
  2. Builtin tools requiring approval (APPROVAL_REQUIRED_BUILTIN_TOOLS — line 25-35): delete_bot, archive_bot, forget_secret, forget_memory, cloud_agent_launch, etc. — must be approved.
  3. Explicit-approval tools (EXPLICIT_APPROVAL_BUILTIN_TOOLS — line 36): create_space — always prompts the user; cannot be pre-allowed.
  4. Connector (integration) tools — the connectorToolRequiresApproval() function (line 85) uses naming heuristics: tools with get_, list_, search_, find_, read_ prefixes are read-only (no approval); tools with create_, delete_, send_, update_, pay_, etc. require approval. A declared readOnly=false override always requires approval. Compound action patterns (_and_, _or_, _then_) are always gated.

Permission scopes. User-defined ActionApprovalRule records allow fine-grained overrides: rules match by exact tool name, connector kind, or category (email or purchase). The resolveActionApprovalDetail() function (line 182) applies deterministic resolution: most-specific matching rule wins. Always-allow and require-approval both beat the default. The planActionGate() function (line 220) adds a third outcome — "judge" — routing consequential default-rule tools to an Auto Review LLM judge when configured.

Auto Review. packages/adapters/src/auto-review.ts defines an LLM-based judge that reviews tool calls before execution. Configured via RAKAZO_AUTO_REVIEW=true env var, plus optional TypeSafe Jev integration (TYPESAFE_API_KEY). applyJudgeDecision() (line 240) maps verdicts: pass → allow, ask → human prompt, error → fail closed for consequential tools.

Unattended / webhook runs. unattendedTriggerToolRequiresApproval() (line 110) applies stricter rules: webhook-triggered runs can only use UNATTENDED_SAFE_BUILTIN_TOOLS (read-only: browser_snapshot, recall_memory, web_fetch, etc.) without approval. Any connector tool in a webhook run needs approval.

Audit trail. Every tool execution records an ExternalEffect via recordEffect() in executor.ts (line 7433) with idempotency keys to prevent replay. Effects transition through statuses (executing → intended → approved, or denied) and are the durable audit log of all actions.

← How is memory and user context stored and retrieved? · How are LLM providers selected and configured? →