sopaco/deepwiki-rs

Overview

deepwiki-rs, which calls itself Litho, is a single Rust binary. You point it at a local checkout and it writes a folder of Markdown architecture documents. The output has a fixed shape: an overview, an architecture page, a workflow page, a boundary-interfaces page, an optional database page, and one “deep exploration” page per domain module that the LLM finds. The documents follow the C4 levels (context, container, component) and contain Mermaid diagrams.

The tool is a batch generator. It has no server, no web UI, no vector store and no chat. Every stage is a call to a “StepForwardAgent” that packs earlier results into one prompt and gets back either typed JSON or Markdown. The results live in an in-process key-value Memory for the length of one run. The only state that survives between runs is an on-disk LLM response cache keyed by an MD5 hash of the full prompt.

It is aimed at teams that want a one-command architecture write-up of a repo in one of eight output languages, and that can run a hosted or local LLM. It is not aimed at people who want to ask questions about a codebase.

Architecture

flowchart TD
  CLI["cli.rs: Args -> Config"] --> L["workflow::launch"]
  L --> KS["KnowledgeSyncer (optional local docs)"]
  L --> P["PreProcessAgent"]
  P --> SE["StructureExtractor (walk + git ls-files)"]
  P --> DS["DirectorySummarizer (LLM per directory)"]
  P --> RA["RelationshipsAnalyze (LLM)"]
  P --> M[("Memory (in-process)")]
  L --> R["ResearchOrchestrator: 6-7 research agents"]
  R --> M
  L --> C["DocumentationComposer: 6 editors"]
  C --> M
  R --> LLM["LLMClient (rig providers)"]
  C --> LLM
  DS --> LLM
  LLM --> CACHE[("CacheManager: .litho/cache")]
  LLM --> TOOLS["file_explorer / file_reader tools"]
  L --> O["DiskOutlet -> output dir"]
  O --> MF["mermaid-fixer (external binary)"]

Component	Path	Role
CLI and config	src/cli.rs, src/config.rs	Parse flags, load litho.toml, apply overrides and defaults
Pipeline driver	src/generator/workflow.rs	launch: preprocess, research, compose, save, summary
Preprocessing	src/generator/preprocess/	File walk, README extraction, per-directory LLM dossiers, relationship analysis
Language processors	src/generator/preprocess/extractors/language_processors/	Regex extractors for 12 languages that pre-list interfaces and imports for the prompt
Research agents	src/generator/research/agents/	System context, domain modules, architecture, workflow, key modules, boundary, database
Editors	src/generator/compose/agents/	Turn research results into the final Markdown pages
Agent framework	src/generator/step_forward_agent.rs, agent_executor.rs	Data-source declaration, prompt assembly, compression, cached LLM calls
LLM client	src/llm/client/	rig provider clients, model choice, retries, tool-using agents
Agent tools	src/llm/tools/	file_explorer, file_reader, time
Output	src/generator/outlet/	DocTree, DiskOutlet, SummaryOutlet, MermaidFixer
External knowledge	src/integrations/	Optional ingestion of local PDFs, Markdown, SQL and similar files by category

How a request flows

Take deepwiki-rs -p ./my-repo -o ./litho.docs:

1. Configure. Args::to_config loads litho.toml from the working directory or --config, then lets CLI flags override provider, models, base URL, key, language and stage-skip flags (cli.rs).
2. Startup checks. launch optionally wipes the cache (--force-regenerate). It then refuses to run if the mermaid-fixer binary is not on PATH, builds the LLMClient, CacheManager and an empty Memory, and syncs external knowledge if configured (workflow.rs).
3. Preprocess. PreProcessAgent::execute reads the README, walks the tree with StructureExtractor (honouring git ls-files by default), asks the LLM for a DirectoryDossier per directory, and then runs RelationshipsAnalyze. All four results are stored in Memory under the PREPROCESS scope (preprocess/mod.rs).
4. Research. ResearchOrchestrator runs the agents in a fixed order: SystemContextResearcher (C1), DomainModulesDetector, ArchitectureResearcher and WorkflowResearcher (C2), KeyModulesInsight (C3-C4), BoundaryAnalyzer, and DatabaseOverviewAnalyzer only when a directory or file looks database-related (orchestrator.rs).
5. Per-agent call. Each agent’s execute checks that its required sources exist in Memory, builds a prompt from them, appends a target-language instruction, and calls extract (typed JSON), prompt (text) or prompt_with_tools (tool-using agent) depending on LLMCallMode. The result is written back to Memory (step_forward_agent.rs).
6. Compose. DocumentationComposer::execute runs the overview, architecture, workflow, key-modules, boundary and (optional) database editors in sequence (compose/mod.rs). KeyModulesInsightEditor fans out one editor per domain report, bounded by max_parallels, and registers each page under deep_exploration/<domain>.md (key_modules_insight_editor.rs).
7. Write. DiskOutlet::save deletes the output directory, writes every DocTree entry it finds in Memory, and then shells out to mermaid-fixer over the folder (outlet/mod.rs). SummaryOutlet adds a run report.

Key components

Directory dossiers

Preprocessing is directory-first. generate_directory_dossiers reads each directory’s files, sorts them by name, and splits them into batches of at most 256 KB of raw content (preprocess/mod.rs). The 256 KB limit does not mean the model sees 256 KB. build_summary_prompt sends only the first 500 characters of each file. Next to them it sends a list of up to 20 interfaces and 15 imports that the regex-based language processors extracted, plus line, function and class counts (directory_summary.rs). The model returns a summary, an importance score, key files and per-file insights. If a call fails, the directory gets an empty fallback dossier and the run continues.

The 12 language processors (Rust, JS, TS, React, Vue, Svelte, PHP, Kotlin, Python, Java, C#, Swift) are line-regex extractors. There is no tree-sitter or compiler-grade parsing (language_processors/mod.rs).

StepForwardAgent

Agents declare required_sources and optional_sources (memory keys, other agents’ results, or external-knowledge categories). GeneratorPromptBuilder formats each source, runs PromptCompressor on it when it is too large, then compresses the assembled prompt once more (step_forward_agent.rs). The default formatter keeps 25 code insights, omits source code, and lists only directories when there are more than 100 files (L111-L123). Editors put __CURRENT_UTC_TIME__ placeholders in the prompt, not the real time, and substitute the time after generation. That keeps the prompt stable, so the cache can hit.

LLM client and model choice

ProviderClient::new maps eight providers to rig clients: OpenAI-compatible, Moonshot, DeepSeek, Mistral, OpenRouter, Anthropic, Gemini and Ollama (providers.rs). Model choice is a size rule: if the system and user prompts together are at most 32 KB, the run uses model_efficient with model_powerful as fallback; otherwise it uses model_powerful only (utils.rs). On a failed extraction, the fallback call repeats the prompt with the previous error appended (client/mod.rs). For the OpenAI variant, a response-parsing error falls back to a raw /chat/completions POST (providers.rs).

Tool-using agents

Three kinds of stage run as tool-using agents: ArchitectureResearcher, WorkflowEditor, and the per-domain key-module editors. AgentBuilder attaches file_explorer and file_reader and adds “Do not fabricate non-existent code” to the system prompt (agent_builder.rs). rig runs the tool loop with max_turns (default 100). file_reader refuses paths outside the project root and returns at most 200 lines unless asked for a range (file_reader.rs). ReActExecutor always reports success without a max-depth flag (react_executor.rs), so the SummaryReasoner fallback in prompt_with_react never runs at this SHA.

Cache

CacheManager stores each reply as JSON under <cache_dir>/<scope>/<md5(prompt)>.json with a timestamp and token estimate. Entries expire after expire_hours, which defaults to 8760 (one year) (cache/mod.rs). Re-runs on an unchanged repo are almost free. Any change to an upstream summary changes every downstream prompt, so those stages miss the cache.

Extending it

• New provider. Add an LLMProvider variant and the matching arms in ProviderClient/ProviderAgent. The match blocks repeat for every provider, so this is a mechanical but wide change.
• New page. Implement StepForwardAgent for a research agent and an editor, add both to the orchestrator and composer, and give the editor a DocTree slot. The page list is code, not config.
• New language. Implement LanguageProcessor (dependencies and interfaces) and register it in LanguageProcessorManager::new.
• External knowledge. [knowledge.local_docs] in litho.toml lets you feed PDFs, Markdown, SQL, YAML or JSON by category to chosen agents. sync-knowledge chunks and caches them.

Running it

• Install. cargo install deepwiki-rs, plus cargo install mermaid-fixer. The run aborts at startup without mermaid-fixer (workflow.rs).
• Configure. Out of the box the provider is “openai” pointed at an OpenAI-compatible ModelScope endpoint with Qwen models, max_tokens 131072, max_parallels 3, and the key from LITHO_LLM_API_KEY (config.rs). Override with --llm-provider, --model-efficient, --model-powerful, --llm-api-base-url and --llm-api-key. Ollama defaults to localhost:11434.
• Scope. By default only git-tracked files are read. *.md, *.txt, lock files and .env are excluded, and test files are skipped.
• Output. ./litho.docs by default, with file names localised for the target language (zh, en, ja, ko, de, fr, ru, vi).

Strengths and caveats

• Strength: predictable structure. The page list and agent order are fixed in code, so two runs on similar repos give documents with the same layout.
• Strength: wide provider reach. Eight providers through rig, a two-model size rule with fallback, and an OpenAI HTTP fallback for picky compatible endpoints.
• Strength: grounded deep pages. The tool-using editors can open real files inside the project root, and they are told not to invent code.
• Strength: cheap re-runs. The prompt-hash cache plus placeholder timestamps make repeated runs on the same tree hit the cache.
• Caveat: shallow first pass. Directory dossiers see 500 characters per file plus regex-extracted signatures. Everything later builds on these summaries, so anything missed here is missed in the final pages unless a tool-using agent looks it up.
• Caveat: hard external dependency. mermaid-fixer must be installed, and it is called with the API key as a command-line argument (fixer.rs). Other users on the machine can see that argument in the process list.
• Caveat: destructive output. DiskOutlet runs remove_dir_all on the output path before writing. Point -o at a dedicated folder.
• Caveat: no incremental model. Memory is per-process, so --skip-preprocessing only works when the cache can rebuild the earlier stages. There is no diff-based update of single pages.
• Caveat: crude heuristics. The database stage triggers on any directory whose name contains “db”, and backend directories are explicitly rated above frontend ones in the dossier prompt.

Sources: code at 27dbd3e, verified Q&A.