# khoj-ai/khoj

> Self-hosted multi-user chat server that answers from your pgvector-indexed notes, web search, sandboxed code and a computer-use agent.

- Category: [Open-source personal assistants](https://llms-technical-reviews.com/personal-assistants/)
- Repository: https://github.com/khoj-ai/khoj (reviewed at commit `ae229ca894c0b80ad84664afcfdde523b5e87057`, 2026-08-01)
- Stars: 37572 · Language: Python · License: AGPL-3.0
- Canonical page: https://llms-technical-reviews.com/p/khoj/

## Overview

Khoj is a self-hostable "second brain": a multi-user chat server that answers from your own documents, the web, a code sandbox and, optionally, a remote-controlled desktop. You upload or sync notes (Markdown, Org, PDF, DOCX, plaintext, images, Notion pages, GitHub repos). Khoj chunks and embeds them into PostgreSQL with pgvector. A chat turn then picks which sources to consult, gathers context, and streams a cited answer from OpenAI, Anthropic or Gemini models, or any OpenAI-compatible local endpoint.

It is a server product more than a library. One Python process runs FastAPI for the API and chat, mounts a Django app for the ORM, migrations and admin panel, and serves a prebuilt Next.js web client. The admin panel is the main configuration surface: chat models, API keys, search model, web scraper and server-wide settings are database rows, not config files. Around the chat loop sit the "assistant" features: custom agents with their own persona, model and tool set; automations that re-run a query on a cron schedule and email the result; a long-term memory of facts extracted from conversations; and a `/research` mode that iterates over tools for several rounds.

It suits people who want a private Perplexity-plus-notes assistant on their own box, or a small team that wants shared agents over shared documents. Besides the web client, the repo ships Obsidian, Emacs, desktop and Android clients that talk to the same HTTP API. It does not run long autonomous tasks on your machine: the only "computer use" happens inside a sandboxed container.

## Architecture

```mermaid
flowchart LR
  C["Web client / API caller"] --> API["FastAPI routers (/api/*)"]
  API --> AUTH["Auth middleware + rate limiters"]
  AUTH --> EG["event_generator (chat turn)"]
  EG --> SEL["Tool selection (LLM or /command)"]
  SEL --> DOCS["Document search (pgvector)"]
  SEL --> WEB["Web search + page reader"]
  SEL --> CODE["Code sandbox (Terrarium / E2B)"]
  SEL --> OP["Operator (computer / browser)"]
  SEL --> RES["Research loop + MCP tools"]
  EG --> GEN["agenerate_chat_response"]
  GEN --> LLM["OpenAI / Anthropic / Gemini"]
  EG --> SAVE["save_to_conversation_log"]
  SAVE --> MEM["Memory extraction"]
  SAVE --> PG["PostgreSQL + pgvector"]
  DOCS --> PG
  SCHED["APScheduler (automations)"] --> EG
```

| Component | Path | Role |
|---|---|---|
| Entry point | `src/khoj/main.py` | Django setup and migrations, FastAPI app, scheduler leader election, route and static mounts |
| Route wiring | `src/khoj/configure.py` | Registers routers, auth middleware, periodic jobs (`schedule` library) |
| Chat API | `src/khoj/routers/api_chat.py` | HTTP and WebSocket chat endpoints, `event_generator` turn orchestration |
| Helpers | `src/khoj/routers/helpers.py` | Command parsing, model fallback wrapper, response generation, memory updates, automations, content indexing |
| Research mode | `src/khoj/routers/research.py` | Multi-iteration tool loop with parallel tool execution and MCP tools |
| Tools | `src/khoj/processor/tools/` | Online search and webpage reading, sandboxed Python, MCP client |
| Operator | `src/khoj/processor/operator/` | Vision-model agent that drives a Docker desktop or a browser |
| Conversation backends | `src/khoj/processor/conversation/{openai,anthropic,google}/` | Provider-specific chat and structured-output calls |
| Content ingestion | `src/khoj/processor/content/` | Per-format parsers that turn files into `Entry` rows |
| Data model | `src/khoj/database/models/__init__.py`, `database/adapters/` | Django models and query adapters (users, chat models, entries, memories, agents) |
| Web client | `src/interface/web/` | Next.js app built with Bun and served as static files |
| Other clients | `src/interface/{obsidian,emacs,desktop,android}/` | Editor plugins and apps that sync files and chat over the HTTP API |

## How a request flows

Take a plain chat message, "what did I write about the Q3 budget?", sent to `POST /api/chat`:

1. **Admit.** The endpoint is wrapped in `@requires(["authenticated"])` and three FastAPI dependencies: 20 requests per minute, a daily cap (100 free, 600 subscribed), and an image size limiter. It then hands the body to `event_generator` and either streams the output or collects it into one JSON response ([api_chat.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/api_chat.py#L1733-L1761)). The WebSocket variant at `/api/chat/ws` first rejects origins outside the allowed hosts ([api_chat.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/api_chat.py#L1489-L1500)).
2. **Parse a command.** `get_conversation_command` maps a leading `/notes`, `/online`, `/webpage`, `/code`, `/research`, `/image`, `/diagram` or `/operator` to a command. Anything else is `Default` ([helpers.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L255-L277)).
3. **Load memory.** If memory is enabled, the turn pulls recent memories (last 7 days) and semantically similar ones from `UserMemory` ([adapters](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/adapters/__init__.py#L2295-L2353)).
4. **Choose tools.** For `Default`, one LLM call (`aget_data_sources_and_output_format`) returns the sources and the output mode. On failure it falls back to a general, tool-less answer ([api_chat.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/api_chat.py#L1000-L1016)). Per-command rate limits are then checked.
5. **Gather context.** `event_generator` runs the chosen tools one after another: research, document search, online search, webpage reading, code, operator, then image or diagram output. Each tool yields status events to the client.
6. **Answer.** `agenerate_chat_response` builds the final prompt from references, online and code results, memories and history, then dispatches on `chat_model.model_type` to `converse_openai`, `converse_anthropic` or `converse_gemini`.
7. **Persist and learn.** `save_to_conversation_log` appends both messages, with their context, to the conversation's JSON log. Unless the turn came from an automation, it then calls `ai_update_memories` ([utils.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/conversation/utils.py#L541-L630)). That function asks the LLM which facts to create or delete, and embeds new ones into `UserMemory` ([helpers.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L1038-L1066)).

## Key components

### Model routing and fallback

A `ChatModel` row has a name, a type (`openai`, `anthropic` or `google`), a price tier, a vision flag and an optional `AiModelApi` with key and base URL ([models](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L200-L240)). The base URL is what makes Ollama, vLLM or LM Studio work: they are configured as "openai" models. `ServerChatSettings` has six slots (default, advanced, and fast/deep variants for free and paid users), a web scraper choice, a priority and a server-level memory mode ([models](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L470-L530)). `send_message_to_model_wrapper` tries the primary model, swaps to a vision model if images are attached, then walks the same slot across lower-priority settings rows. It moves on only for retryable errors ([helpers.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L1635-L1728)).

### Document search

Each upload becomes `Entry` rows with a pgvector embedding. The embedding model is a local sentence-transformers model by default, or a remote inference endpoint. Search applies file, date and word filters, ranks by cosine distance with a configurable threshold, and can re-rank with a cross-encoder. Notion (OAuth) and GitHub (personal access token) are content sources fed into the same pipeline. A 22-25 hour job re-indexes them under a process lock ([configure.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/configure.py#L409-L424)).

### Web search

`search_online` builds a provider list from whatever keys are set: Serper, Exa, Firecrawl, Google Custom Search, then SearXNG. It uses the first provider that returns results ([online_search.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/tools/online_search.py#L100-L140)). Page reading uses the configured `WebScraper` type: Firecrawl, Olostep, Exa or direct fetch ([models](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L392-L420)).

### Research mode

`research()` runs up to `KHOJ_RESEARCH_ITERATIONS` (default 5) rounds. In each round `apick_next_tool` lets the model choose tool calls. The operator runs on its own because it streams. Everything else runs concurrently with `asyncio.gather`. A user can interrupt mid-run with a new instruction, which is folded into the history ([research.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/research.py#L476-L560)). MCP servers registered in the admin panel are exposed only here, as `server/tool` names. `MCPClient` uses SSE for `http(s)` paths and stdio for script paths ([mcp.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/tools/mcp.py#L1-L80)).

### Code and operator sandboxes

Generated Python runs in a Terrarium container (no network) or, with an API key, in an E2B sandbox. The operator is off unless `KHOJ_OPERATOR_ENABLED` is set ([helpers](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/utils/helpers.py#L778-L781)). It needs a vision model. It uses the Anthropic computer-use agent, or a UI-TARS grounding agent; the OpenAI operator branch is hard-disabled. It drives a Docker desktop or a Playwright browser for up to 100 steps. A `RequestUserAction` ends the loop and hands the question back to the user ([operator](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/operator/__init__.py#L170-L181)).

### Automations

An automation is an APScheduler cron job stored in Postgres via `DjangoJobStore`. Its id is derived from the user and query hash. It runs `scheduled_chat` under a process lock, with 60 seconds of jitter ([helpers.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L2655-L2696)). Only one worker executes jobs: `main.run` takes a `ProcessLock` to become schedule leader and starts the scheduler paused on the others ([main.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/main.py#L119-L204)).

## Extending it

- **Add a model.** Create an `AiModelApi` (key plus base URL) and a `ChatModel` in the admin panel, then put it in a `ServerChatSettings` slot. No code change is needed for any OpenAI-compatible server.
- **Add tools via MCP.** Register a `McpServer` (name, path or URL, optional key). Its tools appear in research mode.
- **Custom agents.** An `Agent` bundles a persona, a chat model, input tools, output modes and a privacy level, and can own its own knowledge files. Agents are created through `/api/agents` or the web UI.
- **New content types.** Add a processor under `processor/content/` that produces `Entry` objects through `TextToEntries`, then wire it into `configure_content`.
- **New routes.** `configure_routes` is a plain list of `include_router` calls. Billing and Twilio routers are added conditionally ([configure.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/configure.py#L319-L358)).

## Running it

- **Docker Compose.** The stock compose file runs `pgvector/pgvector:pg15`, the Terrarium sandbox, SearXNG, an optional `khoj-computer` desktop for the operator, and the server image. The server needs only Postgres credentials, a Django secret and an admin email and password. Add at least one LLM key, such as `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` or `GEMINI_API_KEY`, or point a model at a local endpoint.
- **pip.** `pip install khoj` gives a `khoj` command (`khoj.main:run`) with `--host`, `--port`, `--socket`, `--anonymous-mode` for single-user use without login, and `--non-interactive` ([cli.py](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/utils/cli.py#L10-L40)). You still need your own Postgres with pgvector.
- **Production.** `prod.Dockerfile` runs gunicorn and adds the Stripe, Twilio and S3 extras. Email login and automation emails use Resend.

## Strengths and caveats

- **Strength: complete product.** Retrieval, web search, code, memory, agents, automations, admin panel and a polished web client ship together and work multi-user.
- **Strength: model portability.** Three native SDKs, plus any OpenAI-compatible base URL, with a per-slot fallback chain that survives provider outages.
- **Strength: inspectable memory.** Memories are plain text rows with embeddings, listed and edited through `/api/memories`, and switchable per server and per user.
- **Caveat: heavy footprint.** Postgres with pgvector is mandatory, and the default embedding and cross-encoder models pull PyTorch into the image.
- **Caveat: shallow default loop.** Outside `/research`, a turn is one tool-selection call followed by a fixed sequence of tools. Iterative planning and MCP tools only happen in research mode.
- **Caveat: little action gating.** Searches and sandboxed code run without confirmation. The operator is the one place that can pause for the user, and it does so only when the model asks.
- **Caveat: SaaS shape.** Subscriptions, price tiers and per-plan rate limits are part of the routing code. With billing off, every user counts as subscribed ([adapters](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/adapters/__init__.py#L435-L447)), but the free/paid model slots and the hosted-service defaults still shape the admin setup.

*Sources: code at ae229ca, deepwiki-open wiki (12 pages), verified Q&A.*

## How khoj-ai/khoj answers the Open-source personal assistants questions

### How is the assistant architected? (answered)

Khoj is a Python application with a **FastAPI backend** and a **Next.js (React) frontend** served together in a single Docker image. The entry point is `main.py:run()` which initializes Django (for ORM, migrations, admin), sets up FastAPI with CORS middleware, mounts the Django app at `/server`, and starts a Uvicorn (dev) or Gunicorn (prod) server.

**Agent loop and runtime.** The core agent loop lives in `api_chat.py:event_generator()` — a large async generator that processes each user request step by step. It first detects the conversation command from the user's query (e.g., `/notes`, `/online`, `/code`, `/research`, `/operator`) via `helpers.py:get_conversation_command()`. For the "default" command, an LLM call determines which tools to use (`aget_data_sources_and_output_format`). Tools are then executed **sequentially in order**: (1) research iterations, (2) document/note search (semantic vector search over user content), (3) online search, (4) webpage reading, (5) code execution, (6) operator/computer-use. Each tool yields status events back to the client via SSE (HTTP) or a buffered WebSocket protocol. After all tools run, `agenerate_chat_response()` dispatches to the appropriate LLM provider (OpenAI, Anthropic, or Gemini), streams the response back, and saves the conversation turn.

**Frontend/backend split.** The frontend is a Next.js app (in `src/interface/web/`) built with Bun. FastAPI serves it as static files and also provides a Jinja2-based web client for the app and a landing page. Django provides the admin panel (via `django-unfold`) and the database ORM.

**Main packages.** The `khoj/` package contains: `routers/` (API endpoints), `processor/` (conversation models, content ingestion, operator agents), `database/` (Django models and adapters), `search_filter/` (date/file/word query filters), `search_type/` (text/image search backends), and `utils/` (config, helpers, state).

**User request to action flow.** A request arrives at an HTTP POST or WebSocket endpoint → FastAPI authenticates (Django session, bearer token, or anonymous mode) → `event_generator` is invoked → command detection → tool execution phase (each tool calls the LLM as needed) → final LLM call generates the natural-language response with all compiled context → conversation is saved to PostgreSQL via `save_to_conversation_log()`.


Citations: [src/khoj/routers/api_chat.py:676-1487](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/api_chat.py#L676-L1487) · [src/khoj/routers/helpers.py:255-277](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L255-L277) · [src/khoj/configure.py:319-356](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/configure.py#L319-L356) · [src/khoj/routers/helpers.py:1902-2022](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L1902-L2022) · [src/khoj/processor/conversation/utils.py:677-851](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/conversation/utils.py#L677-L851)

### How are integrations (email, calendar, chat, docs) implemented? (answered)

Khoj integrates with several external services, each implemented differently:

**Notion.** OAuth 2.0 flow handled in `routers/notion.py`. The user is redirected to Notion's authorization page; the callback exchanges the code for an access token (using Basic Auth with `NOTION_OAUTH_CLIENT_ID`/`SECRET`), stores it in the `NotionConfig` model (Django DB), and triggers `configure_content()` to index Notion pages. No refresh-token logic — the token is stored as-is.

**GitHub.** `processor/content/github/github_to_entries.py` uses a Personal Access Token (PAT) stored in `GithubConfig` to clone repo file trees via the GitHub REST API (e.g., `api.github.com/repos/{owner}/{name}/contents/`). Supports multiple repos per user. Rate-limit aware: waits on `X-RateLimit-Reset` headers.

**Email.** `routers/email.py` uses **Resend** (via the `resend` Python SDK) to send magic-link login emails, welcome emails, and feedback forms. Configured via `RESEND_API_KEY` env var. Falls back silently if not configured.

**Online search and web scraping.** `processor/tools/online_search.py` supports **six** search/read providers: Google Search API, SerperDev, SearXNG, Exa, Firecrawl, and Olostep. Each is configured via env vars (`GOOGLE_SEARCH_API_KEY`, `SERPER_DEV_API_KEY`, `KHOJ_SEARXNG_URL`, etc.) and the active scraper is a model-priority chain in `WebScraper`/`ServerChatSettings`. Direct HTTP fetching is used via `aiohttp` and `BeautifulSoup`/`markdownify` for webpages.

**MCP (Model Context Protocol).** `processor/tools/mcp.py` implements a generic MCP client that connects to external servers via stdio (for `.py`/`.js` scripts) or SSE (for HTTP URLs). MCP servers are stored in the `McpServer` Django model. The `MCPClient` class lists available tools and calls them by name.

**Phone (Twilio).** Optional phone integration via `routers/api_phone.py` / `routers/twilio.py`.

**Audio.** Text-to-speech via ElevenLabs (`processor/speech/text_to_speech.py`). Speech-to-text via OpenAI Whisper.

**Image generation.** Via OpenAI DALL-E, Replicate, or Google Imagen (`TextToImageModelConfig`).

All integrations are **on-demand**: triggered by the user query, not synced in the background. Content indexing (Notion, GitHub, local files) is triggered manually or via the APScheduler-based periodic indexer.


Citations: [src/khoj/database/models/__init__.py:375-391](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L375-L391) · [src/khoj/processor/content/github/github_to_entries.py:21-80](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/content/github/github_to_entries.py#L21-L80) · [src/khoj/processor/tools/online_search.py:1-55](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/tools/online_search.py#L1-L55) · [src/khoj/routers/email.py:1-100](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/email.py#L1-L100)

### How is memory and user context stored and retrieved? (answered)

Khoj has three memory layers, all stored in **PostgreSQL with pgvector** for vector embeddings:

**1. Conversation history.** Chat sessions are stored as JSON blobs in `Conversation.conversation_log` (a Django `JSONField`). Each user message and assistant response is saved via `save_to_conversation_log()` in `processor/conversation/utils.py`. Messages include references, online results, code context, operator trajectories, images, and thoughts — everything needed to reconstruct the agent's full state. Conversations are keyed per user and optionally per client application. The `generate_chatml_messages_with_context()` function reconstructs the prompt by reading the last N messages, truncating oldest ones to fit the model's context window.

**2. User content (notes/docs).** User-uploaded files (markdown, org-mode, plaintext, PDF, images, DOCX) are ingested by content processors in `processor/content/`. Each file is chunked into `Entry` objects with a vector embedding (`pgvector VectorField`), raw text, compiled text, heading, and file metadata. When the user asks `/notes`, semantic search is run against these entries using cosine distance via `EntryAdapters`.

**3. Long-term memories (UserMemory).** `UserMemory` is a vector-indexed store derived automatically from conversations. After every chat turn, `ai_update_memories()` (in `helpers.py`) calls the LLM to extract new facts from the conversation and saves them via `UserMemoryAdapters.save_memory()`. Two retrieval modes exist: `pull_memories()` (recent, time-windowed — "medium term") and `search_memories()` (semantic vector search — "long term"). Memories are injected into the chat prompt as a special `<retrieved_memories>` block right before the user's query.

**Memory configuration.** Server admins can disable memory entirely (`ServerChatSettings.memory_mode = DISABLED`) or set defaults on/off. Users can toggle it in `UserConversationConfig.enable_memory`.

Memories are accessible and editable via the `/api/memories` REST endpoints (list, update, delete).


Citations: [src/khoj/database/adapters/__init__.py:2295-2353](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/adapters/__init__.py#L2295-L2353) · [src/khoj/routers/helpers.py:1038-1066](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L1038-L1066) · [src/khoj/processor/conversation/utils.py:677-851](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/conversation/utils.py#L677-L851) · [src/khoj/routers/api_chat.py:978-983](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/api_chat.py#L978-L983) · [src/khoj/database/models/__init__.py:658-701](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L658-L701)

### How are actions on the user's behalf gated? (answered)

Khoj has **limited human-in-the-loop gating** — actions are generally executed autonomously once triggered, with a few specific safeguards:

**Operator/computer-use safety.** The most powerful actions (clicking, typing, scrolling, file editing, running terminal commands) are executed inside the operator loop (`processor/operator/__init__.py`). The loop has a key safety valve: it can yield a `RequestUserAction` action type (`operator_actions.py:RequestUserAction`) which pauses the loop and asks the user for input via the status event stream. When this happens, the environment is **left open** (not closed) to wait for user response. The loop also has a hard iteration limit (`KHOJ_OPERATOR_ITERATIONS`, default 100) and responds to client disconnect cancellation events.

**Rate limiting.** Three tiers: `ApiUserRateLimiter` (per-minute and per-day for chat), `ApiImageRateLimiter` (max images and size), and `ConversationCommandRateLimiter` (per-command, with trial=20 / subscribed=75 limits). These are enforced as FastAPI dependency injections on the chat endpoint.

**Subscription gating.** `ConversationAdapters.get_chat_model()` splits users by subscription: subscribed users can use any `ChatModel` including paid-tier models; free/trial users are restricted to `price_tier = FREE` models. Server admins configure which models are free/paid via the Django admin.

**Process locks.** The `ProcessLock` model with named operations (index_content, schedule_leader, apply_migrations) provides distributed mutual exclusion via PostgreSQL. Used to ensure only one worker indexes content at a time and only one worker acts as scheduler leader.

**No generic approval flow.** There is no general "confirm this action before execution" flow for note searches, web searches, or code execution — those run automatically. The operator's `RequestUserAction` is the only mechanism that explicitly pauses for user consent. There is no persistent audit trail of actions performed beyond conversation history (which records what the assistant chose to do).


Citations: [src/khoj/processor/operator/__init__.py:34-243](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/operator/__init__.py#L34-L243) · [src/khoj/processor/operator/operator_actions.py:115-120](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/operator/operator_actions.py#L115-L120) · [src/khoj/routers/api_chat.py:1026-1034](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/api_chat.py#L1026-L1034) · [src/khoj/database/adapters/__init__.py:1196-1210](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/adapters/__init__.py#L1196-L1210) · [src/khoj/database/models/__init__.py:351-363](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L351-L363) · [src/khoj/routers/helpers.py:159-188](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L159-L188)

### How are LLM providers selected and configured? (answered)

Khoj supports **three LLM providers** with a model-agnostic routing layer:

**Supported providers.** OpenAI (`ChatModel.ModelType.OPENAI`), Anthropic (`ANTHROPIC`), and Google Gemini (`GOOGLE`). Each `ChatModel` row stores the model name, a foreign key to `AiModelApi` (which holds `api_key` and optional `api_base_url`), a `model_type`, `vision_enabled` flag, `price_tier`, and `max_prompt_size`. The `AiModelApi` abstraction allows any OpenAI-compatible endpoint to serve as a provider — the base URL is fully configurable, enabling local models via Ollama, VLLM, LMStudio, etc.

**Config surface.** Server admins configure models via the Django admin panel (backed by `ServerChatSettings`). Models are slotted into positions: `chat_default` (free), `chat_advanced` (paid), plus fast/deep variants for free and paid users. Slots are resolved per-request via `aget_chat_model_slot()`. Users can override their model via `UserConversationConfig.setting`. Model fallback is chained: `send_message_to_model_wrapper()` tries the primary model first, then iterates through all fallbacks (from all `ServerChatSettings` entries ordered by priority), retrying on rate-limit/API/timeout errors. Non-retryable errors (auth, invalid model) propagate immediately.

**Tool-calling/structured output.** The `utils.py` for OpenAI (`processor/conversation/openai/utils.py`) detects `StructuredOutputSupport` level per model: TOOL (native function/tool calling), SCHEMA (JSON Schema in `response_format`), OBJECT (basic `json_object`), or NONE. For Anthropic, tool definitions are passed in the Anthropic tool-use format. Structured output schemas use the OpenAI `response_format` with `json_schema` and Anthropic's tool-use respectively. For non-schema-capable models, `clean_json()` strips markdown fences and falls back to string parsing.

**Local-model support.** Any OpenAI-compatible API endpoint works: set `KHOJ_DEFAULT_CHAT_MODEL` and `OPENAI_BASE_URL` (e.g., to `http://localhost:11434/v1/` for Ollama). The `is_local_api()` helper detects localhost URLs and skips HTTPS upgrades.

**Other model types:** Text-to-image (OpenAI DALL-E, Replicate, Google Imagen), speech-to-text (OpenAI Whisper), text-to-speech (ElevenLabs), and embedding models (sentence-transformers from HuggingFace, configurable via `SearchModelConfig`).


Citations: [src/khoj/database/models/__init__.py:206-239](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L206-L239) · [src/khoj/database/adapters/__init__.py:1196-1231](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/adapters/__init__.py#L1196-L1231) · [src/khoj/routers/helpers.py:1635-1728](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L1635-L1728) · [src/khoj/routers/helpers.py:1902-2022](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/helpers.py#L1902-L2022) · [src/khoj/processor/conversation/openai/utils.py:37-100](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/processor/conversation/openai/utils.py#L37-L100) · [src/khoj/database/models/__init__.py:470-540](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/database/models/__init__.py#L470-L540)

### How is it deployed and self-hosted? (answered)

Khoj is designed for **Docker-based self-hosting** with a single `docker-compose.yml` that orchestrates five services:

**Required dependencies.** PostgreSQL with pgvector (`pgvector/pgvector:pg15`) is the sole hard database dependency — it stores all data: users, conversations, vector embeddings, content entries, and scheduler state. A **SearxNG** container provides privacy-respecting meta web search. A **Terrarium** sandbox container (`ghcr.io/khoj-ai/terrarium`) provides isolated Python code execution. An optional **Docker-based computer** container (`ghcr.io/khoj-ai/khoj-computer`) enables the operator/computer-use feature via a VNC-accessible desktop environment.

**Docker/one-click paths.** The `Dockerfile` builds a single image containing both the Python server and the compiled Next.js frontend. It installs system deps (swig, musl, libsqlite), Pip-installs Python packages with CPU-only PyTorch, and runs `collectstatic`. The `prod.Dockerfile` does the same but uses `gunicorn` instead of Uvicorn and includes the `[prod]` extras (Stripe, Twilio, S3). The standard `docker-compose up` works with minimal env vars (only Postgres credentials and a Django secret key are required). The server automatically runs Django migrations on startup via `main.py`.

**Required external accounts.** To actually use the assistant, you need at least one LLM provider API key: **OpenAI** (`OPENAI_API_KEY`), **Anthropic** (`ANTHROPIC_API_KEY`), or **Google Gemini** (`GEMINI_API_KEY`). For web search, you need a **Google Search API** key (`GOOGLE_SEARCH_API_KEY` + engine ID) or a SerperDev/Firecrawl/Exa key. Email login requires a **Resend** API key. Production deployments should configure `KHOJ_DOMAIN` and a strong `KHOJ_DJANGO_SECRET_KEY`.

**Scheduler.** APScheduler (with Django job store) handles periodic tasks: content re-indexing (every 22-25 hours), telemetry upload (every 2 min), stale rate-limit cleanup (every 31 min). A leader-election mechanism via `ProcessLock` ensures only one worker executes scheduled tasks in multi-worker deployments.

**Storage.** Image assets can optionally be stored in **S3** (configured via `AWS_ACCESS_KEY`/`AWS_SECRET_KEY`). Logs go to stdout (Rich) and a log file. No Redis, no message queue — the scheduler piggybacks on the web worker.

> **Editor's note.** Correction: the periodic jobs (content re-index every 22-25 h, telemetry, request cleanup) run on the `schedule` library, polled every 60 s by `poll_task_scheduler`; APScheduler with the Django job store is used for user automations, and the ProcessLock leader election gates that scheduler.

Citations: [src/khoj/main.py:150-183](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/main.py#L150-L183) · [src/khoj/configure.py:417-422](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/configure.py#L417-L422) · [src/khoj/routers/storage.py:1-62](https://github.com/khoj-ai/khoj/blob/ae229ca894c0b80ad84664afcfdde523b5e87057/src/khoj/routers/storage.py#L1-L62)