ekzhang/openjev-sglang

Overview

OpenJev is a small FastAPI service that reimplements the TypeSafe “Jev” decision API (POST /v1/systemone) on an open model. You send a state (text, JSON, or a chat transcript) and up to 64 named questions of three kinds: Noul (yes/no probability), Choice (pick one of 2-64 options) and Score (expected position on an ordered rubric). The server answers each question by reading the next-token probabilities of answer labels from Qwen3.6-35B-A3B running in SGLang. It never generates more than one token.

It sits in the browser-control category because several Jev-based agents (computer-use and browser agents) call exactly this API to pick their next action. OpenJev itself is not an agent. It has no browser, no screen reading, no action loop and no tools. It is a self-hostable backend you could point such an agent at. The README now says the experiment is superseded: SGLang ships a native decisions endpoint built the same way.

The code is compact (about 1,000 lines in src/openjev/) and carefully defensive: strict request validation, admission limits, cancellation of sibling requests, and a supervisor that kills the container when SGLang dies.

Architecture

flowchart LR
  C["Client (Jev-style request)"] --> G["RequestGuard: auth + body cap"]
  G --> API["FastAPI /v1/systemone"]
  API --> SVC["EvaluationService"]
  SVC --> PC["PromptCompiler: chat template + labels"]
  SVC --> BK["SGLangClient /generate"]
  BK --> SG["SGLang server (localhost)"]
  SG --> M["Qwen3.6-35B-A3B NVFP4"]
  SVC --> SC["scoring: softmax over label logprobs"]
  RT["runtime: launch + watchdog"] --> SG
  MOD["modal_app.py (B200)"] --> RT

Component	Path	Role
HTTP app	src/openjev/api.py	create_app, RequestGuard middleware, routes, error mapping, timing headers
Evaluation service	src/openjev/service.py	Admission control, token budgets, prefix warm-up, parallel branches
Prompt compiler	src/openjev/prompts.py	Renders the chat template once, splits prefix/suffix, picks single-token labels
Backend client	src/openjev/backend.py	One-token /generate calls with selected-token logprobs, abort on cancel
Scoring	src/openjev/scoring.py	Temperature softmax, Noul/Choice/Score answers, entropy confidence
Runtime	src/openjev/runtime.py, launch.py	Builds the SGLang command line, starts/stops the process group, watchdogs
Settings	src/openjev/config.py, profiles.py	OPENJEV_* env settings and the single qwen36 profile
Deployment	modal_app.py	Modal Server on one B200 with a persistent weights volume
Evals	evals/	BoolQ and MMLU-Pro comparisons against hosted Jev

How a request flows

1. Guard. RequestGuard checks the optional Bearer key with a constant-time compare and buffers the body, returning 413 once it passes max_body_bytes, even for chunked uploads (api.py).
2. Route. The systemone handler starts service.evaluate() and a disconnect watcher side by side. If the client leaves first, it raises 499 and cancels the work (api.py).
3. Admit. evaluate() accepts the configured model name, the jev-latest alias or any jev-* name. It rejects with 529 when max_concurrent_requests evaluations are active, and turns a deadline overrun into 504 (service.py).
4. Compile. PromptCompiler.prepare appends a user turn containing a random marker, renders the whole conversation once with thinking disabled, and splits on the marker. The prefix is tokenized once. Each question becomes a suffix with Question:, lettered Options: and Answer: (prompts.py).
5. Budget. Before any GPU work, _evaluate rejects a branch longer than max_input_tokens (counting the output token) or a total over max_total_input_tokens (service.py).
6. Warm, then fan out. It sends the bare prefix to SGLang and waits, so the radix cache holds it. Then all branches go out concurrently with asyncio.gather. Any failure cancels the siblings (service.py).
7. Read logprobs. Each generate() call asks for max_new_tokens=1 and token_ids_logprob for the label tokens only. On cancel or timeout it posts /abort_request (backend.py). parse_generation insists on exactly one output token and finite label values (backend.py).
8. Score and reply. answer() softmaxes the label logprobs (divided by temperature) and builds the typed answer. The response carries x-openjev-prefix-tokens, an optional x-openjev-cached-tokens header and a Server-Timing split of prepare, prefill and branches.

Key components

Single-token labels

Qwen splits numbers such as 10 into several tokens, so OpenJev does not number options. At startup the compiler walks A-Z, then AA, AB and so on, and keeps only strings that encode to exactly one distinct token and decode back unchanged. It refuses to start unless it finds 64 (prompts.py). Choice keys are hidden from the model unless an option’s description is null, in which case the key is shown as its meaning.

State handling

state_messages treats a list of {role, content} objects, or an object that is exactly {"messages": [...]}, as a real chat and keeps its roles. Anything else is serialized as one user message. Only text content is allowed (prompts.py). The classification instruction tells the model to treat instructions inside the state as material, not commands.

Scoring

normalize is a stable softmax over only the requested labels, so probabilities are conditional on the offered options. Noul returns P(true). Choice returns the argmax plus the full distribution. Score returns the probability-weighted zero-based index plus a legend. confidence is 1 - H(p)/log(n), a concentration measure, not calibration (scoring.py).

Runtime and supervision

backend_command builds the SGLang launch line: localhost only, FP8 KV cache, --mamba-radix-cache-strategy extra_buffer, breakable prefill CUDA graphs. It refuses extra flags that would disable radix caching or CUDA graphs (runtime.py). watch_process calls os._exit(1) if the child dies unexpectedly, so the platform replaces the container instead of serving a dead backend (launch.py). Every request, warm-ups included, asks for at least one token’s logprob, to stay clear of an SGLang crash when mixed batches occur (backend.py).

Extending it

• Another model. Add a ModelProfile to profiles.py (weights, revision, image, backend flags). The Settings.profile literal currently allows only qwen36, so that type must widen too (config.py). The tokenizer must provide 64 single-token labels and a chat template that keeps the marker intact.
• Existing backend. openjev serve --connect URL skips process management and talks to a running SGLang that has the same model and selected-token logprobs.
• Prompt changes. All prompt wording lives in PromptCompiler.prepare. evals/prompt_probe.py exists to compare variants.

Running it

• Modal. uv run modal deploy modal_app.py creates an unauthenticated Server on one B200 that scales to zero after five idle minutes. Weights and compile caches persist in a Modal Volume (modal_app.py).
• Own GPU. uv run openjev serve --sglang-python /path/to/sglang/bin/python starts SGLang 0.5.19 from a separate CUDA environment.
• Check. uv run openjev smoke URL exercises all three answer types, a 64-option question and the 65-option rejection.

Strengths and caveats

• Strength: efficient by design. N questions cost N+1 one-token calls over a shared, cached prefix. No decoding loop.
• Strength: honest outputs. Strict response parsing, explicit null for cache counts the backend omits, and documented meaning of confidence.
• Strength: drop-in. Jev request and response shapes, jev-* model aliases and TypeSafe-style /v1/models mean Jev clients can switch endpoints with little or no change.
• Caveat: not a browser tool. Nothing here perceives or controls a UI. It only matters to this category as a backend for Jev-driven agents.
• Caveat: one model, heavy hardware. The only profile is Qwen3.6-35B-A3B NVFP4 tuned for B200-class GPUs. Changing models means code changes.
• Caveat: open by default. The Modal deployment is unauthenticated unless you set OPENJEV_API_KEY.
• Caveat: superseded. The authors point users to SGLang’s native decisions endpoint instead.

Sources: code at bf6a53b, deepwiki-open wiki (10 pages), OpenDeepWiki wiki (10 pages), verified Q&A.