crawlab-team/crawlab

Overview

Crawlab is not a scraper and has no AI features. It is a crawler management platform: a Go server, a MongoDB database and a web UI that run your spider programs on one or more machines. A spider can be a Scrapy project, a Node/Puppeteer script, a Go binary or any other command. Crawlab stores the code, starts it as an OS subprocess on a node, captures its logs and collects the records it reports back. Scheduling is done with cron expressions. Nothing in the repository fetches a web page, parses HTML or calls a model.

Where it fits in an AI scraping stack: it is the job-control layer. Your spider (perhaps a Crawl4AI or Firecrawl client, or a Playwright script that sends Markdown to an LLM) is the payload. Crawlab gives that payload a UI, cron schedules, per-node concurrency limits, live logs and a results table. Anything about fetching, stealth or extraction must live in the spider.

The pinned commit (October 2024) is the v0.6 rewrite. The code has *V2 services next to legacy ones, and some features are only half moved over. The sections below note where that matters.

Architecture

flowchart LR
  UI["Web UI (crawlab-ui package)"] -->|"/api via nginx"| API["Gin REST API"]
  API --> ADM["Spider admin service"]
  SCH["Cron schedule service"] --> ADM
  ADM --> Q["Mongo task queue"]
  API --> DB["MongoDB"]
  W["Worker task handler"] -->|"gRPC Fetch"| GS["Master gRPC server"]
  GS --> Q
  W --> R["Task runner"]
  R -->|"HTTP /sync"| API
  R --> P["Spider subprocess"]
  P -->|"gRPC stream: data, logs"| GS
  GS --> DB
  M["Master monitor"] -->|"ping every 15 s"| W

Component	Path	Role
Entry point	core/cmd/server.go, core/apps/	crawlab server starts the API, gRPC server and node services
REST API	core/controllers/router_v2.go	Gin routes for spiders, tasks, schedules, nodes, results, export, sync
Spider admin	core/spider/admin/service_v2.go	Turns “run spider” into a task and a queue item
Task scheduler	core/task/scheduler/service_v2.go	Enqueue, cancel, status recovery, 30-day cleanup
Task handler	core/task/handler/service_v2.go	Per-node loop that fetches tasks and runs them
Task runner	core/task/handler/runner_v2.go	Syncs files, builds the command, sets env, starts and watches the process
gRPC task server	core/grpc/server/task_server_v2.go	Dequeues tasks; receives data and log streams
Master service	core/node/service/master_service_v2.go	Monitors workers, marks them offline
Result writer	core/task/stats/service_v2.go	Writes reported records to the spider’s collection
Schedules	core/schedule/service_v2.go	robfig/cron entries that call the spider admin
Frontend	frontend/, nginx/crawlab.conf	Loads the prebuilt crawlab-ui npm package; nginx serves it on 8080

How a request flows

Take a click on “Run” for a spider:

1. API. PostSpiderRun binds SpiderRunOptions (mode, node ids, cmd, param, priority) and calls the spider admin service (spider_v2.go).
2. Task creation. scheduleTasks builds one TaskV2 and fills empty fields from the spider’s defaults. If the mode named any nodes, it pins the task to the first one (service_v2.go). Enqueue inserts the task, a TaskQueueItemV2 with the priority and node id, and an empty stats row (scheduler/service_v2.go).
3. Fetch. On every node, ServiceV2.Fetch ticks. It skips inactive or disabled nodes and nodes already at MaxRunners (default 8). Otherwise it calls the master’s gRPC Fetch (handler/service_v2.go). The master first looks for a queue item assigned to that node, then for an unassigned one. It sorts by priority and id, sets the task’s node and deletes the queue item (task_server_v2.go, L242-L263).
4. Sync code. On a worker, RunnerV2.syncFiles asks the master’s /sync/:id/scan for a file list with hashes, deletes local files that are gone, and downloads changed files with 10 parallel requests (runner_v2.go).
5. Build and start. configureCmd joins the command and param and passes them to sys_exec.BuildCmd. That function splits the string on single spaces, with no shell and no quoting (runner_v2.go, sys_exec_linux.go). configureEnv adds CRAWLAB_TASK_ID, the gRPC address and auth key, and every global environment variable from the database (runner_v2.go). Run starts the process, pipes stdout and stderr to the log stream, and waits on a signal channel for finish, cancel, error or “lost” (runner_v2.go).
6. Report data. The spider uses the separate Crawlab SDK (save_item, or a Scrapy pipeline) to stream records over gRPC with the task id from the environment. Subscribe sends INSERT_DATA to handleInsertData and INSERT_LOGS to the log driver (task_server_v2.go, L212-L232). InsertData writes the batch into the spider’s Mongo collection and updates the result count (stats/service_v2.go).

Key components

Task queue

The queue is the task_queue_items Mongo collection, polled by every node through the master. This design is simple and needs no broker. Two details matter at scale. The dequeue is a find followed by a delete. It runs inside RunTransactionWithContext, but the model-service calls do not use the session context, so the transaction does not protect them. Two nodes polling at the same moment can claim the same item. And polling is the only dispatch: a task waits up to one fetch interval before a node picks it up.

Run modes

The UI offers “all nodes”, “random” and “selected nodes”. In the V2 admin service, all-nodes and selected-nodes both produce one task, assigned to the first node in the list. The legacy service created one task per node; that fan-out was not ported (legacy service.go). If you need the same job on every worker, schedule one task per node.

Node monitoring

The master runs monitor every 15 s. For each worker it subscribes, pings over gRPC and updates the free runner count. A worker that fails either check is marked offline (master_service_v2.go). The runner also has a process health check, but it returns at once when cmd.ProcessState is nil, and that is always the case while the process runs. So it never reports a lost process (runner_v2.go).

Results

Records go to the spider’s col_name collection in MongoDB. Only Pro editions (edition: global.edition.pro) route them to another database through a registry. The core/ds folder has MySQL, PostgreSQL, Elasticsearch, Kafka and other drivers, but no package imports it. Watch for one bug in the stats service. On a cache miss, getDatabaseServiceItem returns the nil item it looked up, not the entry it just cached. The first InsertData call for each task then dereferences nil (stats/service_v2.go). Results can be exported as CSV or JSON from the API.

Schedules

schedule/service_v2.go holds a robfig/cron instance. Each entry loads the schedule and the spider, merges their mode, nodes, cmd and param, and calls the same Schedule path as the Run button (schedule/service_v2.go).

Extending it

• Any language. A spider is a folder plus a cmd string. Install the runtime on the node image. Global environment variables reach every task, which is a good place for API keys and proxy URLs.
• Crawlab SDK. Spiders report records with save_item (Python) or CrawlabPipeline (Scrapy). Without the SDK, the platform still captures logs but sees no data.
• Git-backed spiders. SpiderV2.GitId and GitRootPath sync code from a repository (vcs/ module).
• REST API. Every UI action is a Gin route under /api, so CI or an agent can create spiders, start runs and read results.

Running it

• Use Docker Compose: one crawlabteam/crawlab master (CRAWLAB_NODE_MASTER=Y), any number of workers pointing at it through CRAWLAB_GRPC_ADDRESS, and MongoDB. The UI is on port 8080. nginx forwards /api/ to the Go server on 8000. gRPC uses 9666.
• The images are built from separate backend, frontend and plugin images. This repository’s frontend/ folder only bootstraps the crawlab-ui 0.6.3 npm package (main.ts).
• Change the defaults. The gRPC auth key falls back to the constant Crawlab2021!. The /sync/:id/scan and /sync/:id/download routes are in the anonymous route group, and they join the path query onto the workspace directory without checking that it stays inside it (router_v2.go, sync_v2.go). Keep the API on a private network.

Strengths and caveats

• Strength: framework-neutral. It runs anything with a command line, so it can schedule AI scrapers written in any stack without code changes.
• Strength: operations UI. Live logs, task history, cron schedules, per-node runner caps and a data browser come ready-made.
• Strength: few moving parts. Only MongoDB and the Crawlab binary. Workers sync code from the master automatically.
• Caveat: no scraping features. It has no fetching, proxies, stealth, robots.txt, URL dedup, extraction or LLM support. Crawl-level concerns stay inside each spider.
• Caveat: V2 migration gaps. Run modes do not fan out, the health check is dead code, the ds drivers are unused, and the stats cache has a nil bug. The two-version code makes it hard to tell what is live.
• Caveat: fragile command handling. Splitting on spaces breaks quoted arguments. Use a wrapper script for anything complex.
• Caveat: security defaults. A hard-coded gRPC key and unauthenticated file-sync routes mean it must not face the internet.
• Caveat: activity. The pinned commit is from October 2024, and the 0.6.0 changelog entry still says “TBC”.

Sources: code at 0485310, deepwiki-open wiki (11 pages), OpenDeepWiki wiki (20 pages), verified Q&A.