AsyncFuncAI/deepwiki-open

A repository is ingested via the Repo class in api/repository.py (line 140). It supports GitHub, GitLab, Bitbucket, and local paths — each with its own _clone_from_* function that handles access tokens via URL injection (oauth2:token@ for GitLab, token@ for GitHub, x-bitbucket-api-token-auth for Bitbucket). Clones use git.Repo.clone_from with --depth=1 --single-branch for shallow clones. Local paths skip cloning entirely and are used in-place. The cloned repo path is ~/.deepwiki/repo/.

File filtering is centralized in api/config.py's iterate_files() (line 467). It walks the repo directory and applies: (1) extension whitelisting from code_extensions and doc_extensions config arrays, (2) exclusion mode using default file_filters.excluded_dirs (like node_modules, .venv, .git, dist) merged with request-level exclusions, or (3) inclusion mode when included_dirs/included_files are specified. Default excluded dirs are extensive (line 3-40 of repo.json).

Chunking happens in the RAG pipeline, not at ingestion. api/rag/pipeline.py's LineTrackingTextSplitter (line 174) uses adalflow's TextSplitter with split_by: word, chunk_size: 350, chunk_overlap: 100 (from embedder.json line 36-41). Each chunk gets annotated with its 1-based start_line/end_line within the original file. Files exceeding MAX_EMBEDDING_TOKENS * 10 (82K tokens) are skipped (line 126). The pipeline then embeds each chunk and saves to a local LocalDB pickle file at `/.adalflow/databases/(line 247-274). Large-repo limits are configurable viaMAX_CONCURRENT_WIKI_TASKS(defaults to half CPU cores) andWIKI_PAGE_CONCURRENCY(default 1) inapi/services/wiki/tasks.py` (lines 62-68).

Retrieval is implemented via a FAISS-based RAG pipeline using the adalflow framework. The core class is RAG in api/rag/rag.py (line 134), which wraps a FAISSRetriever (line 290) configured with top_k: 20 (from embedder.json). Documents are embedded using configurable embedders: OpenAI text-embedding-3-small (256 dims) by default, Google gemini-embedding-001, Ollama nomic-embed-text, or Bedrock amazon.titan-embed-text-v2:0 — selected via the DEEPWIKI_EMBEDDER_TYPE env var (config.py line 65).

The retrieval pipeline in research_chat() (api/services/research.py, line 60) first prepares the retriever index (loading from a .pkl pickle file or creating it fresh), then calls rag.acall(query) which invokes the FAISS retriever. Retrieved documents are grouped by file path and each chunk is annotated with its line range (line 178-184). The grouped context is assembled into a structured prompt that includes file-path headers and line-numbered chunks inside <START_OF_CONTEXT> / <END_OF_CONTEXT> tags (line 146-192).

The context is then injected into prompt_builder() (api/chat/_prompts.py, line 6), which builds a final prompt combining the system prompt, conversation history, context block, and user query. The response streamer (ChatStreamer in _stream.py) sends the full prompt to the LLM and streams back chunks. If the input exceeds ~7500 tokens, RAG context is omitted and a fallback note is inserted instead (line 27-29 of _prompts.py). The Memory class in rag.py (line 79) accumulates conversation turns for multi-turn dialogue.

The wiki structure (table of contents) is determined by an LLM call. In api/services/wiki/tasks.py, _determine_structure() (line 315) orchestrates the process. First it reads the cloned repo's file tree and README via read_repo_file_tree() in structure.py (line 20), which returns sorted file paths filtered through iterate_files() plus the README content. The repo's default branch is detected with git rev-parse --abbrev-ref HEAD (line 51).

These inputs are fed into build_structure_prompt() in prompts.py (line 213). The prompt constructs an XML schema asking the LLM to output a <wiki_structure> block containing a title, description, and individual <page> elements — each with an id, title, importance (high/medium/low), related pages, and a list of <file_path> citations. For "comprehensive" wikis, it also includes <sections> grouping pages and supporting nested subsections (line 135-185); "concise" omits sections and targets 4-6 pages instead of 8-12 (line 187-209).

The LLM's raw XML response is parsed by parse_wiki_structure() in structure.py (line 179). This function is robust: it strips markdown fences, escapes bare & characters, handles truncated responses (missing </wiki_structure>) by salvaging complete blocks, and falls back to regex-based extraction if strict xml.etree.ElementTree parsing fails. The result is a WikiStructureModel containing WikiPage and WikiSection objects (schemas in api/schemas/wiki.py).

Individual wiki pages are generated by _generate_pages() in api/services/wiki/tasks.py (line 288). Each page from the WikiStructureModel is processed by _generate_page() (line 369), which builds a per-page prompt via build_page_prompt() in prompts.py (line 25). The prompt includes the page title and a pre-built Markdown list of relevant file links with host-specific URLs (GitHub blob/{branch}, GitLab -/blob/{branch}, Bitbucket src/{branch}). The prompt instructs the model to: start with a <details>" summary="Relevant source files" block citing >=5 files, use Mermaid diagrams extensively (flowcharts, sequence diagrams, class diagrams — all top-down via graph TD), create markdown tables, and cite sources using empty-parenthesis citations like Sources: [path/to/file.py:10-25]() (line 108-118).

Generation streams the LLM response through the research_chat() pipeline (same as RAG chat), then post-processes it in post_process_wiki_content() (api/services/wiki/content.py, line 84). This function: (1) rebuilds the <details> block with real hyperlinks from the known file list, (2) resolves all empty-parenthesis citations into proper GitHub/GitLab/Bitbucket URLs with line anchors (#L10-L25), (3) handles generic file-path citations and basename-only references, and (4) strips redundant empty parens.

Pages are generated with configurable concurrency: WIKI_PAGE_CONCURRENCY defaults to 1 (sequential) but each page gets up to WIKI_PAGE_RETRIES + 1 attempts before falling back to an error-placeholder page (line 268-285). Once all pages are done, the wiki is saved as a JSON cache file (deepwiki_cache_{type}_{owner}_{repo}_{lang}.json) in the wiki cache directory via save_wiki_cache() (io.py, line 52). Existing cache is detected at submission time to skip regeneration entirely (tasks.py line 176).

Model providers are configured through JSON config files and a client registry pattern. The primary config is api/config/generator.json, which defines providers and their models. The default provider is google with gemini-2.5-flash as the default model; OpenAI (gpt-5-nano as default), OpenRouter, Ollama, Bedrock, Azure, DashScope, and LiteLLM are all supported (generator.json lines 4-198). A LiteLLM variant of the config lives in generator.litellm.json with litellm as the default provider.

Each provider has a corresponding ChatStreamer subclass in api/chat/_stream.py that registers itself via the provider class attribute: OllamaChatStreamer, OpenRouterChatStreamer, OpenAIChatStreamer, AzureChatStreamer, LiteLLMChatStreamer, BedrockChatStreamer, DashScopeChatStreamer, GoogleGenerativeChatStreamer, and AnthropicChatStreamer. The ChatStreamer.create() factory method (line 40) looks up the provider name in a registry dict and instantiates the right class.

For per-stage model selection, get_model_config() in config.py (line 364) looks up a provider's configuration, resolves the model client class, and extracts model-specific parameters (temperature, top_p, etc.). The same provider+model tuple is passed through all requests (ChatCompletionRequest and WikiTaskRequest via RepoRequestBase schema, api/schemas/base.py line 9). The embedder is configured separately via DEEPWIKI_EMBEDDER_TYPE env var (openai | google | ollama | bedrock) and its own config file (embedder.json).

OpenAI-compatible endpoints are supported via several paths. The LiteLLMClient (api/clients/litellm.py) extends the OpenAI client, connecting to a LiteLLM proxy at LITELLM_BASE_URL (default localhost:4000). The OpenAIClient uses the standard OpenAI API. Any OpenAI-compatible endpoint can be configured by pointing the client to a different base URL. Local models run through OllamaClient (ollama.py) or via the Ollama provider with local models like qwen3:1.7b, llama3:8b, and qwen3:8b (generator.json lines 116-141).

Interactive Q&A over a repository is implemented through a WebSocket-based chat system. The frontend's Ask component (src/components/Ask.tsx) provides the UI, with three modes: Fast, Deep Research, and Codemap. When a user submits a question, the frontend opens a WebSocket to /ws/chat (src/utils/websocketClient.ts, line 64), sends a ChatCompletionRequest as JSON, and receives streaming text chunks. On WebSocket failure, it falls back to HTTP POST at /api/chat/stream (src/app/api/chat/stream/route.ts).

The backend endpoint is handle_websocket_chat() in api/routers/chat.py (line 20). It delegates to research_chat() (api/services/research.py), which: (1) prepares or loads the FAISS RAG index for the repo, (2) retrieves relevant document chunks, (3) loads conversation history from the Memory class into XML-style <turn> blocks, (4) picks the appropriate iteration prompt, and (5) streams the LLM response via ChatStreamer.respond_stream(). If the prompt exceeds 7500 tokens, RAG context is dropped and the model answers from its training data alone (line 146-148).

Deep Research mode (api/prompts.py lines 60-151) runs up to 5 automated iterations: iteration 1 produces a research plan, iterations 2-4 investigate deeper aspects, and iteration 5 produces a final conclusion. Each iteration builds on the previous conversation history. The frontend's Ask.tsx auto-continues through iterations (line 620-637), extracting research stages from the response (## Research Plan, ## Research Update N, ## Final Conclusion) and displaying a stage navigator. A max of 5 iterations is hard-coded (research.py line 219), with force-complete logic on the frontend (Ask.tsx line 501-508).

Codemap mode (api/services/codemap.py) is a separate multi-phase pipeline: RAG retrieval → two-step JSON generation (skeleton + enrich with Mermaid diagrams) → citation grounding against real source files → NDJSON streamed over /ws/codemap. Results are displayed as structured step-by-step guides with a side-by-side CodeViewer for cited source files.

Conversation history is maintained client-side in the conversationTurns array (Ask.tsx line 92), with model/provider selection exposed through a modal. Unlike the wiki generation pipeline, Q&A does not cache results on the server.