savka777/jev-use
Native macOS voice/typed computer-use app that reads the Accessibility tree and has the Jev model pick each action from a closed list.
Overview
jev-use (the app is called “Desktop Voice”) is a menu-bar app for macOS 14.2+ that turns a spoken or typed sentence into actions on your own desktop. It controls any app, not only browsers: it opens apps, folders and sites, clicks controls, types into fields, scrolls, uses menus and tiles windows. Browsers are just apps with a large Accessibility tree.
Its design choice is narrow and deliberate: the model never writes anything. Each step, the app walks the front app’s Accessibility tree, builds a numbered table of controls and a closed list of operations, and sends them to TypeSafe’s Jev model (jev-latest), which only answers multiple-choice and yes/no questions. Even the text to type is selected, not generated: Jev picks the first and last word of the span inside your own sentence. Swift code performs the action with Accessibility calls or synthetic CGEvent input, then reads the screen again.
The project is small (about 4,500 lines of Swift in two modules) with no third-party dependencies. It reads as a harness tuned by trial: many thresholds in the code carry comments explaining which logged failure they fixed.
Architecture
flowchart LR
V["Voice: SFSpeechRecognizer / hotkey / wake phrase"] --> M["AppModel.run"]
T["Typed command / say.sh"] --> M
M -->|"default"| C["runCycles (up to 14)"]
M -->|"PlannerEnabled + key"| P["Planner.plan (OpenRouter)"]
P --> E["executePlanned per step"]
C --> D["Desktop.capture: AX walk"]
E --> D
C --> J1["JevClient.cycle"]
E --> J2["JevClient.ground"]
J1 --> TS["TypeSafe /v1/systemone"]
J2 --> TS
C --> X["Desktop.perform: AX actions + CGEvent"]
E --> X| Component | Path | Role |
|---|---|---|
| App model and loops | Sources/JevDesktop/JevDesktopApp.swift |
AppModel: speech/typed entry, run, runCycles, executePlanned, window-chain replay, SwiftUI widget and settings |
| Desktop layer | Sources/JevDesktop/Desktop.swift |
AX tree walk into DesktopSnapshot, perform for every DesktopAction, settling and change detection |
| Jev client | Sources/JevCore/Decision.swift |
Request builders (cycleBody, groundingBody), HTTP calls, Decision parsing |
| Planner | Sources/JevCore/Planner.swift |
Optional OpenRouter call that splits a sentence into PlanSteps |
| Command parsing | Sources/JevCore/CommandInput.swift |
Extracts URLs, counts and durations from the sentence |
| Voice input | Sources/JevDesktop/SpeechInput.swift, HotKey.swift, WakePhrase.swift |
Hold-to-talk, hands-free, “Hey Jev” |
| Secrets | Sources/JevDesktop/KeyStore.swift |
TypeSafe and OpenRouter keys in the Keychain |
How a request flows
Take “Go to youtube.com, search Rick Astley and play the first video” with default settings:
- Entry. Speech or the text field calls
run, which cancels any running task and chooses a path. The planner path runs only if a planner key exists and thePlannerEnableduser default is true. Otherwise it callsrunCycles(JevDesktopApp.swift). - Read. Each cycle calls
Desktop.capture, which walks the focused window off the main thread (Desktop.swift). After a navigation it re-reads for up to about 3 s until a browser page shows new controls (JevDesktopApp.swift). - Offer. Code builds an element table (index, role, label, value, page or toolbar) and an operation menu:
CLICK,TYPE_TEXT,OPEN_APP,OPEN_URL,MENU,PRESS_RETURN,SCROLL_*,SKIP_*,ARRANGE_WINDOWS,WAIT,DONE,BLOCKEDand others. Toolbar controls are hidden unless the sentence mentions tabs or the address bar. Lists are trimmed to stay under TypeSafe’s 255-option limit (JevDesktopApp.swift). - Choose.
JevClient.cyclesends one request: anoperationchoice, one speculative target choice per operation (click_target,type_target,type_from/type_toover the sentence’s words, …), and yes/no heads such asfinishes,create_first,countedandevery_window(Decision.swift). - Gate. Code may override the pick (for example
TYPE_TEXTbecomesCLICKorMENUwhen Jev says the wanted input is missing). An unsure Return (confidence < 0.5) is not pressed. A target below confidence 0.2, or 0.6 for destructive labels such as quit, delete or close, stops and asks “Which one?” (JevDesktopApp.swift). - Act.
Desktop.performbrings the target app back to the front, checks the window and the control’s label are unchanged, then acts (Desktop.swift). - Check. It waits for a fingerprint change, records the effect (“window is now …”, “no visible effect”) in
recentActions, and loops. It returns early when Jev’sfinisheshead was ≥ 0.8 and the action visibly worked. It stops onDONE,BLOCKED, three no-change actions, the same pick three times, or 14 cycles (JevDesktopApp.swift).
Key components
Accessibility capture
read fetches about 20 attributes per element in one AXUIElementCopyMultipleAttributeValues round trip and names unlabeled inputs from their title element or static-text value (Desktop.swift). The walk skips secure text fields and hidden nodes, clips to scroll areas and web areas, and drops controls more than 200 px outside the visible region. Lists with more than 50 children fall back to the app’s own visible-rows subset, which keeps very large tables fast (Desktop.swift). Chromium and Electron apps get extra waits because their web content builds the tree late.
Executing actions
DesktopAction is a closed enum: open app/folder/website, quit, AX press, select row, click at centre, focus, key, scroll, type, position and arrange windows. Opening goes through NSWorkspace. Clicks on controls that accept no AX press are posted as CGEvents, and the cursor is put back afterwards (Desktop.swift). Typing first tries setting kAXSelectedText, falls back to per-character Unicode key events, and reads the value back to report whether the text actually appears (Desktop.swift). Window tiling is pure arithmetic in code, with no model call per window.
Optional planner
Planner.plan sends the sentence, the front app and the running apps to OpenRouter (default inception/mercury-2.5, temperature 0, reasoning off, strict JSON schema). It gets back steps from an 11-kind vocabulary (Planner.swift). executePlanned runs keys, scrolls, skips and URLs directly in code. On-screen steps get one narrow JevClient.ground question (“which listed target does this step mean?” plus already_done) (JevDesktopApp.swift).
Every-window chains
When Jev’s every_window head says the goal applies to all windows, the steps that worked in the first window are recorded as role/label/ordinal descriptions. finish() replays them in the other windows without asking Jev again, unless the finish itself was uncertain (JevDesktopApp.swift).
Extending it
There is no plugin surface. Changes are code edits:
- New operation. Add a
DesktopActioncase and its branch inDesktop.perform, add the operation description inrunCycles, and map it to a target head. - Prompt rules. All Jev instructions are string constants in
Decision.swift(cycleRulesand the per-head instructions). - Planner model.
defaults write local.jev-use PlannerModel <openrouter id>. To turn the planner on at all:defaults write local.jev-use PlannerEnabled -bool true, plus a key saved in Settings. - Diagnostics.
defaults write local.jev-use DumpTree -bool truelogs every walked AX node. All logs are under thelocal.jev-usesubsystem.
Running it
- Build. Xcode and
bash build.sh, which builds a release binary, signs it locally and installs~/Applications/Desktop Voice.app. - Permissions. Accessibility, microphone and speech recognition. Keys are stored in the Keychain under service
local.jev-use(KeyStore.swift). - Services. A TypeSafe API key is required. An OpenRouter key is optional. Apple Speech may send audio to Apple’s servers.
- Use. Hold Control-Option-Space and speak, use hands-free mode (about 1.5 s of silence submits), say “Hey Jev”, or run
scripts/say.sh "Open Finder".
Strengths and caveats
- Strength: no free text from the model. Operations, targets and typed spans are all choices over offered options, which makes the agent cheap, fast and hard to talk into inventing actions.
- Strength: whole-desktop reach. The same loop drives Finder, native apps, Electron apps and browsers without per-app code.
- Strength: careful verification. Label re-checks before acting, read-back after typing, and effect strings fed into the next decision.
- Caveat: macOS only, and accessibility-limited. Apps or web pages with poor AX labelling (canvas UIs, unlabeled custom controls) are hard or impossible to target. There is no vision fallback.
- Caveat: hidden planner switch. The Settings text says an OpenRouter key enables planning. In code,
runalso needs aPlannerEnableddefault that no UI sets, so a saved key alone changes nothing. - Caveat: dead fallback path.
JevClient.decideand the olderexecuteloop (batched 251-option questions) are never called. Readers ofDecision.swiftshould start fromcycleandground. - Caveat: your real session. It acts on your logged-in apps and browser profile with only heuristic confirmation for destructive labels, and the control table (labels and values, not secure fields) goes to TypeSafe.
Sources: code at b22e568, deepwiki-open wiki (12 pages), verified Q&A.
How it answers the Browser & computer control questions
Each answer was drafted by a code-reading agent at commit b22e568. Its citations were checked mechanically. Compare with the other browser & computer control →
How is the page represented to the model?
answeredThe page is represented exclusively through the macOS Accessibility tree — no screenshots, no DOM serialization, no set-of-marks images are captured or sent. The Desktop.capture() function (Desktop.swift:187) walks the Accessibility tree starting from the frontmost application's window (AXUIElement root), recursing through AXUIElementCopyMultipleAttributeValues to read roles, names, values, positions, enabled/hidden state, children, and DOM identifiers for every control (method read(), Desktop.swift:112-151). The walk is bounded: children lists longer than 50 nodes are pruned to the app's reported visible subset (Desktop.swift:139-144). Controls outside a clipped frame within a −200 px margin are skipped as off-screen (Desktop.swift:514-517). Unnamed text-less controls that are not recognised interactive roles are dropped (Desktop.swift:633). Secure fields (kAXSecureTextFieldSubrole) are skipped entirely (Desktop.swift:504). The walk also performs a pointer-coordinate sweep (Desktop.swift:557-573) to catch elements the tree walk missed. The result is serialised as a flat list of Element structs (index, role, label, value, place, operations) plus available apps/folders/sites/menus (Decision.swift:198-222). This is sent alongside recent actions as a JSON payload to the TypeSafe API. The model sees numbered elements in screen order (top-to-bottom, left-to-right, Desktop.swift:678-683), each described with '[index] role 'label' value...' strings. Total candidates are trimmed to stay under TypeSafe's 255-option-per-question limit (Decision.swift:82); elements are batched into multiple parallel choice questions when needed (Decision.swift:95-124). The fingerprint function (Desktop.swift:203-218) provides a cheap structural hash (window title, focused control, first 400 nodes, window position/size) used to detect screen changes between cycles.
runCycles), oversized control lists are trimmed to about 250 options, preferring labels that share words with the command; the multi-batch splitting in Decision.swift requestBody/decide is only used by the execute loop, which nothing calls.How are actions executed and how are elements targeted?
answeredActions are executed through a closed vocabulary of DesktopAction types (Desktop.swift:13-30) and dispatched in Desktop.perform() (Desktop.swift:1089-1326). The action types are: application — opens/switches to an app via NSWorkspace.shared.openApplication(); quit — terminates a running app via NSRunningApplication.terminate(); folder — opens a Finder folder; website — opens a URL in a specific browser; press — performs an Accessibility press action on an AXUIElement (AXUIElementPerformAction); select — sets kAXSelectedAttribute on a row/cell (Desktop.swift:1036-1046); clickAt — synthesises a mouse click at the element's centre point using CGEvent posted via system tap (Desktop.swift:1051-1064); focus — sets kAXFocusedAttribute, falling back to a centre click if Accessibility focus fails (Desktop.swift:1067-1080); key — posts keyboard events via CGEvent (Desktop.swift:1223-1228); scroll — posts scroll wheel events (Desktop.swift:1229-1230); type — enters text by setting kAXSelectedTextAttribute or by posting per-character keyboard Unicode events (Desktop.swift:1231-1269); position — sets window position/size; arrange — tiles/columns/cascades all windows with arithmetic layout (Desktop.swift:1278-1323). Element targeting is by Accessibility element reference (AXUIElement), not by selector or coordinate index. The DesktopSnapshot maps candidate IDs to DesktopAction values that contain the actual AXUIElement pointer (Desktop.swift:361-362). Before acting, the code verifies the target element still exists and its label matches (Desktop.swift:1160-1162). Opening a website uses NSWorkspace.shared.open() which delegates to the OS, not CDP or Playwright.
How is the agent loop / planning implemented?
answeredThe agent loop lives in AppModel.run() (JevDesktopApp.swift:397) and takes two forms. Default loop (no planner key): runCycles() (JevDesktopApp.swift:613-1010) runs up to 14 cycles. Each cycle: captures the Accessibility tree → builds a CycleState with elements, available apps/folders/sites/menus, recent actions, and dictation → sends one JevClient.cycle() request (Decision.swift:289) that asks for an operation (choice) plus speculative targets (choices for click_target, type_target, app_target, etc.) and several yes/no questions (finishes, create_first, every_window, counted) in a single model call → executes the chosen action → detects screen change → loops until DONE, BLOCKED, WAIT, or exhaustion. Planner path (with OpenRouter key): Planner.plan() (Planner.swift:137) sends the utterance to OpenRouter with a JSON schema (response_format: json_schema, Planner.swift:110-111) to decompose into ordered PlanStep kinds (open_app, open_url, click, type_text, press_key, etc.). Then each step is grounded via a narrow JevClient.ground() call (JevDesktopApp.swift:560) that asks Jev to select the correct on-screen target. Deterministic steps (press key, scroll, skip, open URL) execute without any model call (JevDesktopApp.swift:489-512). Stop conditions: DONE, BLOCKED, WAIT; three consecutive no-change actions stop; same pick three times stops; a 14-cycle hard cap. Memory between steps: recentActions holds the last 10 actions (Decision.swift:206). priorCommand and priorAction carry context across clarification questions (JevDesktopApp.swift:89-90). The JevClient.session is a keep-alive TLS connection (Decision.swift:306-310).
PlannerEnabled user default (set only via defaults write), so the default for every user is the per-cycle loop. WAIT is not a stop condition: it sleeps 0.4 s and the loop continues.How are failures, retries and self-healing handled?
answeredError classes caught: DesktopError with a stale flag (Desktop.swift:6-11), DecisionError.invalidResponse (Decision.swift:388-393), ServiceError for TypeSafe HTTP errors (Decision.swift:373-385), PlannerError for OpenRouter failures (Planner.swift:156-173), CancellationError for user Escape. Retries: When stale=true fires on a DesktopError, the cycle retries the same action after a fresh capture, up to 2 times (JevDesktopApp.swift:931-933). In executePlanned(), if grounding returns no candidate, it retries up to 2 times with a 700 ms sleep (JevDesktopApp.swift:571-581). If stale fires during Desktop.perform(), the step re-captures and retries up to 2 times (JevDesktopApp.swift:601-607). In the fallback loop, unavailable retries twice (JevDesktopApp.swift:1146-1152). Screen-change detection: fingerprint() (Desktop.swift:203-218) gives a structural hash checked before and after each action. waitForChange() (Desktop.swift:242-250) polls it for change. waitForQuiet() (Desktop.swift:229-238) waits for the fingerprint to stabilise. Before execution, Desktop.perform() hit-tests the target centre point to confirm nothing is covering it (Desktop.swift:1176-1186). Chain replay: When Jev judges the goal applies to every window, replay() (JevDesktopApp.swift:1021-1086) repeats the recorded chain across all other windows without further model calls, matching targets by name, kind, place, and ordinal; a missing target after 12 attempts is reported, not guessed. Timeout: The TypeSafe HTTP request has a 20-second timeout (Decision.swift:309).
runCycles a stale-target error is not retried as the same action; it is recorded as 'NOT performed' and the next cycle captures again and lets Jev re-decide, bounded only by the 14-cycle cap and the no-change/same-pick stops. The cited 'fallback loop' (execute) is never called.Which models are supported and how are they called?
answeredTwo models serve different roles. The primary model is Jev (jev-latest), accessed through the TypeSafe API at https://api.typesafe.ai/v1/systemone (Decision.swift:90,177-178). Jev is a vision-free decision model: it receives element tables (not screenshots) and chooses from closed choice vocabularies. Calls use JSON-over-HTTPS with no streaming, no tool-calling, and no structured output schema — the response is always a Decision JSON object with choice/noul answer types (Decision.swift:15-22). Vision is not required because the page is represented entirely through the Accessibility tree. The optional planner model goes through OpenRouter (https://openrouter.ai/api/v1/chat/completions, Planner.swift:138). It defaults to inception/mercury-2.5 (Planner.swift:43), overridable via defaults write local.jev-use PlannerModel (Planner.swift:45). The planner uses JSON schema response_format: json_schema (Planner.swift:110-111) for structured step output, low temperature (0), no reasoning tokens (Planner.swift:106-108), and 900 max tokens. When the planner is absent, the system falls back to the per-cycle loop where Jev handles both sequencing and targeting in each request. How they are called: Both are plain HTTPS POST requests. TypeSafe receives a state+questions JSON (Decision.swift:88-140) and returns JSON answers. OpenRouter receives a standard chat completions request (Planner.swift:101-112). API keys are stored in the macOS Keychain (KeyStore.swift:6-41). The provider distinction is sharp: TypeSafe for the decision/grounding model, OpenRouter only for the initial planning step.
How are browser sessions, profiles, auth and anti-bot handled?
answeredLocal macOS-only. The app controls the user's own desktop — there is no remote/cloud browser, no Docker, no VNC. Browser control is through the macOS NSWorkspace.shared.open() API, which launches or activates the native browser binary (Desktop.swift:1133). The app reads a list of known browsers by bundle identifier (Desktop.swift:60): Brave, Chrome, Safari, Edge, Firefox, plus any Chromium/Electron app detected by the presence of a (Renderer).app helper (Desktop.swift:67-77). Persistent profiles and cookies are not managed by the app — it opens URLs through the OS, so the user's existing browser profile and cookies are used as-is. Stealth/anti-bot is not implemented: there is no user-agent manipulation, no browser fingerprint spoofing. Proxies are not configured by the app — the system proxy configured in macOS is used by default. CAPTCHA handling is not implemented and not claimed; if a site presents a CAPTCHA, the model cannot solve visual challenges. The code includes a waitForPageToSettle() function (Desktop.swift:970-993) that polls the Accessibility tree until the control count stabilises after page navigation. Auth/Keychain: API keys for TypeSafe and OpenRouter are stored in the macOS system Keychain using the SecItem API (KeyStore.swift:14-36) under the service name local.jev-use. Wake phrase detection is purely client-side text matching (WakePhrase.swift:5-33): the app matches "Hey Jev" (or the variant "Hey Jeff") at the start of an Apple Speech transcript fragment — no dedicated wake-word engine.