weibaohui/openDeepWiki

Overview

This is weibaohui/openDeepWiki, a Go project. It is not AIDotNet/OpenDeepWiki, the larger C#/.NET project reviewed on this site under the slug opendeepwiki. The two share a name and a goal, but no code, stack or authors. This one is a single Go binary (Gin, GORM, CloudWeGo’s Eino ADK) with a React and Ant Design X front end embedded in it. Prompts, logs and generated wiki titles are in Chinese.

The design is agent-first. There is no file chunking, no embedding index and no static page plan. When a repository is added, the server shallow-clones it. A two-agent chain (toc_editor then toc_checker) then explores the checkout with file and shell tools and emits a YAML list of chapters with outlines and hints. Each chapter becomes a queued task. Another chain of three agents writes, lints and fact-checks that page. Agents are YAML files that are hot-reloaded from disk. Models are API-key rows in the database with priority-ordered failover.

Around that core sit a chat assistant over WebSocket, an MCP server for coding tools, incremental updates driven by git pull and diff, document versions and ratings, PDF export, and push-sync to another instance.

Architecture

flowchart LR
  UI["React UI (embedded)"] --> API["Gin /api router"]
  API --> RS["RepositoryService"]
  RS --> CLONE["git clone --depth 1"]
  RS --> BUS["Event buses"]
  BUS --> TS["TaskService"]
  TS --> SCHED["Pending scheduler (10 s)"]
  SCHED --> ORCH["Orchestrator (1 worker)"]
  ORCH --> W["Writers: toc / default / api / db / incremental"]
  W --> SEQ["Eino SequentialAgent"]
  SEQ --> TOOLS["list_dir, read_file, search_files, run_terminal_command"]
  SEQ --> PM["ProxyChatModel + model pool"]
  W --> DB["SQLite or MySQL (GORM)"]
  API --> CHAT["Chat WebSocket: chat_assistant"]
  CHAT --> PM
  MCP["/mcp/streamable"] --> DOCS["DocumentService keyword search"]
  DOCS --> DB

Component	Path	Role
Entry point	backend/cmd/server/main.go	Wires repos, services, writers, buses, orchestrator, MCP and router
Writers	backend/internal/domain/writers/	toc, default, api, dbmodel, incremental, title_rewriter, doc_rewriter, user_request
Agent runtime	backend/internal/pkg/adkagents/	YAML loader and watcher, Manager, BuildSequentialAgent, model pool, rate limiter
Agent tools	backend/internal/pkg/adkagents/tools/	File listing, reading and search, shell, skills, read_doc
Agent definitions	backend/agents/*.yaml	Instructions, tool lists, maxIterations, optional model per agent
Tasks	backend/internal/service/task*.go, orchestrator/	Task state machine, RunAfter dependencies, ants worker pool
Documents	backend/internal/service/document.go	Versions, keyword search, export
Chat	backend/internal/handler/chat_handler.go	WebSocket hub and streaming of agent events
MCP	backend/internal/mcp/server.go	Five read-only tools over the generated docs
Sync and activity	backend/internal/service/sync/, activity_scheduler.go	Push to a remote instance; scheduled incremental refresh

How a request flows

Adding a repository and getting its wiki:

1. Create. POST /api/repositories calls RepositoryService.Create. It normalises HTTPS or SSH URLs, rejects duplicates by host/owner/repo, and stores a pending row with a timestamped local path (repository.go).
2. Clone and plan. The repo-added event runs CloneRepository, which starts git clone --depth 1 in a goroutine (10-minute timeout). The same handler immediately publishes a TocWrite event (repo_event_subscriber.go, repository_clone.go, git.go).
3. Queue. Tasks are created as pending. Every 10 seconds, enqueuePendingTasks pushes them into the global orchestrator once their RunAfter dependency has succeeded (task.go). The orchestrator is an ants pool started with one worker, despite a comment that says two (main.go).
4. TOC. tocWriter.genDirList builds a SequentialAgent of toc_editor and toc_checker and asks for strict YAML with dirs and analysis_summary. parseDirList unmarshals the reply, and each dir becomes a DocWrite task with its outline and hints stored (toc.go).
5. Write a page. CreateDocWriteTask first inserts a placeholder document (title cut to 20 runes) (task_helper.go). When the task runs, defaultWriter.genDocument chains document_generator, markdown_checker and document_checker. Its prompt holds the repo path, title, outline and hints (default.go).
6. Run agents. RunAgentToLastContent iterates the Eino runner and keeps the last message. That text is the page (agent_factory.go).
7. Save. executeTaskLogic writes the content into the placeholder document. A rewrite task creates a new version and moves the IsLatest flag (task.go).

API and database-model chapters are not part of the automatic plan. They are separate buttons (/api-analyze, /db-model-analyze) that queue the apiWriter or dbModelWriter chains.

Key components

Agents as YAML

Manager.createADKAgent turns a definition into an Eino ChatModelAgent (manager.go). It picks a model: the named pool, one named key, or a dynamic proxy over all enabled keys. It resolves each tool name through ToolProvider, sets 3 retries on rate-limit errors, and appends the iteration cap to the instruction. Unknown tool names are skipped with only a log line. For example, chat_assistant.yaml lists git_log and git_show, which the provider does not implement (tools_providers.go). Every tool is rooted at the shared Data.RepoDir, not at the current repository’s folder. An agent writing about one repo can therefore read the others.

Shell access

Most writer agents get run_terminal_command. It runs sh -c <command> with a 30-second timeout. The empty allow-list means all commands are allowed, and the only guards are a working-directory check and a substring test for .. (cmd.go). The agents read untrusted repository content, so a prompt injection in a README can reach a shell on the server. Run it in a container.

Model pool

API keys are database rows (provider, base URL, model, priority, status). The provider anthropic gets Eino’s Claude client with MaxTokens: 4096. Everything else goes through the OpenAI-compatible client (model_provider.go). ProxyChatModel tries up to three models from the pool in turn and records token usage per task (proxy_model.go). The Claude client also overwrites request headers to pose as the Claude Code CLI (claude-cli user agent, claude-code-20250219 beta) (claude_header_transport.go).

Chat and MCP

The chat handler puts the repo’s metadata and a list of document titles and IDs into a system message, adds the last 20 messages, and streams the chat_assistant agent’s events over WebSocket (chat_handler.go). The agent can read_doc a generated page or read raw source with the file tools. The MCP server registers list_repositories, get_repository, search_documents, read_document and get_document_summary (server.go). Search is a case-insensitive substring scan over every latest document in memory, capped at 20 hits (document.go).

Incremental updates

The incremental writer runs git pull (unshallowing when needed), summarises the diff since the stored commit, and asks incremental_editor/incremental_checker which chapters to add or rewrite. It then queues DocWrite or content-rewrite tasks. An activity scheduler can trigger this automatically for repos in completed state (activity_scheduler.go).

Extending it

• Agents. Edit or add YAML under the agent directory (AGENT_DIR). Defaults are extracted from the binary on first start, and a polling file watcher reloads changes. Agent versions are tracked in the database.
• Skills. Agents can call list_skills and run scripts from SKILL_DIR through the shell tool. One sample skill ships: go-backend-stack-analyzer.
• Writers. Implement domain.Writer (Name, Generate) and register it with taskService.AddWriters in main.go.
• Models. Add rows on the API-key page. Any OpenAI-compatible endpoint works. A model field in an agent’s YAML pins that agent to one key.
• External tools. Point an MCP client at /mcp/streamable. /.well-known/openapi.yaml describes the REST API.

Running it

• Build. make build builds the front end, embeds it and the agents, and produces one Go binary. Docker images use Alpine and expose 8080. make dev runs the hot-reload setup.
• Config. config.yaml (or CONFIG_PATH) sets the port, the database (sqlite default, or mysql via DB_TYPE/DB_DSN) and the data and repo directories. LLM keys are not in the config file, whatever the README’s quick-start says. They are added in the UI and stored in the database.
• Requirements. git on the PATH, network access to the git host and the LLM endpoints. A github.token key appears in the example config, but no Go code reads it, so private repos depend on the host’s own git credentials.
• Security. No authentication on any route, and CORS * with credentials (router.go), plus the shell tool above. Treat it as a single-user tool behind a firewall.

Strengths and caveats

• Strength: grounded in the code. Pages come from agents that read the actual files. The generator prompt requires a source link on every code block and forbids invented code, and a separate document_checker pass reviews the draft.
• Strength: operational features. Task state machine, retries, stuck-task cleanup, per-task token accounting, document versions and ratings, model failover, incremental refresh and MCP. That is more operations tooling than most projects of this size have.
• Strength: configurable without recompiling. Prompts, tool sets and model choice per agent are plain YAML.
• Caveat: serial and slow. With one orchestrator worker, a repo’s chapters are written one at a time, and the generator agent alone may take up to 100 tool iterations per page.
• Caveat: plan races the clone. The TOC task is queued while the clone is still running, and nothing waits for ready. On a slow clone, the first plan can see a partial tree or fail and need a retry.
• Caveat: Chinese-only output. The TOC prompt asks for Chinese titles, and no language setting exists.
• Caveat: unsafe defaults. Unrestricted sh -c, cross-repo file access and an open API make it unsuitable for untrusted repos or shared hosting as shipped.
• Caveat: small details. The BuildSequentialAgent error in genDirList is never checked, and WriteComplete events are published with no subscriber.

Sources: code at bf64e72, verified Q&A.