adbar/trafilatura

Overview

Trafilatura is a Python library and CLI that downloads web pages over plain HTTP and extracts the main text, comments and metadata with deterministic rules. It does not use a browser, a model, or an LLM. Inside it parses HTML with lxml, converts it to a small XML vocabulary (p, head, list, quote, code, table, graphic, ref), runs a cascade of extractors, and serialises the result as TXT, Markdown, CSV, JSON, HTML, XML or XML-TEI.

It is the quiet workhorse of this category. Corpus builders, RAG ingestion jobs and research crawlers use it because it is fast, needs no GPU or API key, and copes with a long tail of templates through accumulated heuristics (the pinned version is 2.3.0). The cost of the rule-based design is that it only sees what the server sends: JavaScript-rendered pages, PDFs and paywalled content are out of scope.

Beyond extraction, it ships its own discovery layer: sitemap and feed parsers, a polite focused crawler built on courlan’s UrlStore, and a CLI that downloads many URLs in parallel with per-domain back-off. In this category it is a pre-processor for LLM pipelines, not an AI scraper itself.

Architecture

flowchart TD
  IN["URL, file or HTML string"] --> DL["downloads.fetch_url (urllib3 or pycurl)"]
  DL --> LOAD["utils.load_html (lxml)"]
  IN --> LOAD
  LOAD --> META["metadata.extract_metadata"]
  LOAD --> SEQ["core.trafilatura_sequence"]
  SEQ --> CLEAN["htmlprocessing: tree_cleaning + convert_tags"]
  CLEAN --> MAIN["main_extractor.extract_content"]
  MAIN --> CMP["external.compare_extraction (readability fork, jusText)"]
  CMP --> BASE["baseline rescue"]
  BASE --> ESC["recall escalation"]
  ESC --> OUT["determine_returnstring: txt, md, json, csv, html, xml, tei"]
  META --> OUT
  DISC["sitemaps, feeds, spider"] --> DL

Component	Path	Role
Public API	trafilatura/__init__.py, core.py	fetch_url, extract, extract_with_metadata, bare_extraction, the extraction cascade and output dispatch
Options	trafilatura/settings.py, settings.cfg	Extractor options object, Document result, config-file defaults
Cleaning and conversion	trafilatura/htmlprocessing.py	Tag removal, link-density pruning, HTML to internal XML
Main extractor	trafilatura/main_extractor.py	Content-area XPaths, per-element handlers, wild-text recovery, comments
XPath rules	trafilatura/xpaths.py	BODY_XPATH, discard lists for sidebars, teasers, comments and ads
Fallbacks	trafilatura/external.py, readability_lxml.py, baseline.py	Vendored readability fork, jusText, and a last-resort baseline
Metadata	trafilatura/metadata.py, json_metadata.py	Meta tags, JSON-LD, title, author, date via htmldate
Downloads	trafilatura/downloads.py	Pooled HTTP with SSRF guard, retries, size cap, optional SOCKS proxy
Discovery	trafilatura/spider.py, sitemaps.py, feeds.py	Focused crawler, sitemap and feed URL discovery
CLI	trafilatura/cli.py, cli_utils.py	Batch processing, parallel downloads, crawl and explore modes

How a request flows

Take extract(fetch_url("https://example.org/post"), with_metadata=True, output_format="markdown"):

1. Download. fetch_response picks pycurl when it is installed and urllib3 otherwise. On an SSL error it retries without certificate checks unless INSECURE_SSL_FALLBACK is off (downloads.py). The urllib3 path streams the body in 128 KiB chunks capped at MAX_FILE_SIZE, with retry and redirect limits from the config (downloads.py). fetch_url returns decoded HTML only for a 200 of acceptable length (downloads.py).
2. Options and parse. bare_extraction packs the keyword arguments into an Extractor, parses the HTML, optionally checks the <html lang> attribute, and runs extract_metadata (core.py). Metadata comes from meta tags, then JSON-LD, then title and author heuristics, with dates from htmldate.find_date (metadata.py).
3. Prepare. trafilatura_sequence prunes appended articles, share widgets and (unless comments are wanted) comment sections, then tree_cleaning deletes unwanted tags and convert_tags maps HTML to the internal vocabulary (core.py, htmlprocessing.py).
4. Main extractor. _extract tries each BODY_XPATH expression in order, from itemprop='articleBody'-style matches through <article> and story/content classes to main. For each match it prunes discard XPaths and link-heavy blocks, converts children through handle_textelem, and stops at the first subtree with real content (main_extractor.py, xpaths.py). If the result is short, recover_wild_text scavenges loose paragraphs.
5. Compare. Unless fast=True, compare_extraction runs the vendored readability on the raw tree and switches to it when the own result is empty, much shorter, or structurally poor; jusText is then tried for unclean or short outputs (external.py).
6. Rescue and escalate. If text is still under MIN_EXTRACTED_SIZE (250 chars), baseline tries JSON-LD bodies, <article> blocks, paragraphs, then the whole body. In balanced mode, a result under 3,000 characters that also covers under 20% of the page text triggers a recall-mode retry plus a jusText candidate (core.py).
7. Filter and serialise. Size, duplicate and language checks can discard the document (the function returns None). determine_returnstring then writes Markdown with a YAML front-matter header built from the metadata (core.py).

Key components

The extraction cascade

The design idea is “each stage engages only if the previous one under-delivered” (core.py). The own extractor is precise but template-bound; readability is good on classic articles; jusText classifies paragraphs by density and stopwords and reaches content buried in generic divs; baseline is a blunt safety net. favor_precision and favor_recall shift each stage’s thresholds rather than choosing a different algorithm. There is special handling for forum threads (detected by schema.org DiscussionForumPosting), where comment containers are kept as content.

Cleaning and the internal XML

tree_cleaning removes a fixed tag list, keeps or drops tables and images depending on options, and in recall mode undoes the deletion if it would remove every paragraph (htmlprocessing.py). The XML intermediate is what makes the output formats cheap: Markdown, HTML and TEI are just different walks over the same tree.

Downloads and politeness

The default user agent is trafilatura/<version> with a project URL; USER_AGENTS and COOKIE in the config override it, picking one agent at random per request (downloads.py). By default a _SafePoolManager rejects private, loopback and link-local addresses on every hop. If http_proxy is set, a urllib3 SOCKSProxyManager is used instead, and that path skips the SSRF pool (downloads.py). Defaults: 30 s timeout, 20 MB cap, 2 redirects, 5 s between requests to one domain (settings.cfg).

Focused crawler

focused_crawler(homepage) reads robots.txt, keeps links inside the start URL’s path prefix, puts navigation pages at the front of the queue, sleeps for the robots crawl delay or SLEEP_TIME between fetches, and stops after max_seen_urls (default 10) pages (spider.py, L308-L351). It returns two URL lists, to visit and known; it does not extract text. The frontier is a module-level URL_STORE, so state persists across calls in one process.

CLI

trafilatura accepts a URL, an input file, a directory or stdin, plus --feed, --sitemap, --crawl, --explore and --probe. URL lists go through buffered_downloads, a thread pool fed by UrlStore.get_download_urls with per-domain back-off (downloads.py). --archived retries failed URLs through the Wayback Machine (cli_utils.py).

Extending it

• Tune without code. prune_xpath removes site-specific noise before extraction; --config-file or a ConfigParser changes size thresholds, timeouts, user agents and SSRF behaviour.
• Get structure, not text. bare_extraction returns a Document with the lxml body tree, so you can walk headings, lists and tables yourself before chunking for a RAG index.
• Custom rules. MANUALLY_CLEANED and the XPath lists in xpaths.py are module-level and can be mutated at runtime; the code explicitly honours a user who removes "form" from the cleaning list.
• JavaScript pages. Render with a browser of your choice, then pass the HTML string to extract. The library accepts strings, bytes or lxml trees.

Running it

• Install. pip install trafilatura pulls only lxml, urllib3, courlan, htmldate, justext, charset_normalizer and certifi. trafilatura[all] adds pycurl, SOCKS support, py3langid language detection, Brotli/zstd and faster charset detection.
• Python. extract(fetch_url(url)) is the one-liner; extract_with_metadata returns a Document with text and metadata fields (init.py).
• CLI. trafilatura -u URL --output-format markdown --with-metadata, or trafilatura -i urls.txt -o out/ --parallel 8 for batches.

Strengths and caveats

• Strength: fast, local and cheap. No browser, no model and no network calls beyond the fetch, so it runs anywhere Python and lxml run.
• Strength: layered fallbacks. Four extraction strategies, gated by measured text length, give good recall on odd templates without hurting typical articles.
• Strength: rich output. Metadata, comments, tables, formatting, links and images are optional, and TEI and YAML-headed Markdown are first-class.
• Caveat: static HTML only. There is no JavaScript execution and no wait strategy; client-rendered sites return little or nothing.
• Caveat: heuristics can misfire. Content-area detection relies on class and id patterns. A page that matches the wrong container yields confident but partial text, and the result silently becomes None when filters reject it.
• Caveat: transparent by default. The bot-identifying user agent, robots-aware crawler and lack of stealth are good citizenship, but mean protected sites will block it. The default insecure-SSL retry is convenient but worth turning off for sensitive pipelines.
• Caveat: crawler scope. The spider is a small single-site link discoverer with in-memory state, not a distributed crawler.

Sources: code at 6c1977a, deepwiki-open wiki (12 pages), verified Q&A.