mixmark-io/turndown

Overview

Turndown is a small JavaScript library that converts HTML to Markdown. It is not a scraper and has no AI features: it does not fetch pages, run JavaScript, crawl, or call a model. You give it an HTML string or a DOM node, and it returns a Markdown string. The whole engine is about 1,000 lines in src/, with one runtime dependency, @mixmark-io/domino, which parses HTML in Node. In the browser it uses the native DOM parser.

Where it fits in an AI scraping stack: it is the conversion step between “I have the HTML” and “I have text an LLM can read”. Many JavaScript crawlers and agent tools use it for this. A typical pipeline is: fetch or render the page (Playwright, a stealth driver, a crawler), cut out the main content (a readability library or your own selectors), run Turndown, then chunk the result and send it to a model. Turndown does only the third step. It converts everything you pass in, including navigation, footers, and the text of <script> and <style> tags, unless you tell it not to.

The project is mature (version 7.2.4 at the pinned commit from September 2026). It is maintained mainly for correct CommonMark output and safe escaping, not for scraping features.

Architecture

flowchart LR
  IN["HTML string or DOM node"] --> RN["RootNode"]
  RN --> PARSE["HTMLParser: DOMParser or domino"]
  RN --> CW["collapseWhitespace"]
  CW --> PROC["process(): walk child nodes"]
  PROC --> NODE["Node(): isBlock, isCode, isBlank, flanking whitespace"]
  NODE --> RULES["Rules.forNode()"]
  RULES --> REP["rule.replacement(content, node, options)"]
  REP --> JOIN["join() newline handling"]
  JOIN --> POST["postProcess(): rule.append, trim"]
  POST --> OUT["Markdown string"]

Component	Path	Role
Service	src/turndown.js	TurndownService: options, turndown(), use, addRule, keep, remove, escape; the recursive process walk
Rules	src/rules.js	Holds added, CommonMark, keep and remove rules; picks one per node
CommonMark rules	src/commonmark-rules.js	Paragraphs, headings, lists, code blocks, links, images, emphasis and so on
Parser	src/html-parser.js	Native DOMParser in browsers, domino.createDocument in Node
Root node	src/root-node.js	Wraps input in <x-turndown>, parses it, collapses whitespace
Node helpers	src/node.js	Marks block, code and blank nodes; computes leading/trailing whitespace
Utilities	src/utilities.js	Block/void element lists, Markdown escaping regexes
Whitespace	src/collapse-whitespace.js	Collapses HTML whitespace the way a browser renders it, except inside <pre>

How a request flows

Take new TurndownService({ headingStyle: 'atx' }).turndown(html):

1. Options. The constructor merges your options over the defaults: setext headings, * * * rules, * bullets, indented code blocks, _ and ** delimiters, inline links. It then builds a Rules object from COMMONMARK_RULES (turndown.js).
2. Validate. turndown() accepts a string, an element, a document or a fragment, and throws TypeError for anything else (turndown.js, L221-L230).
3. Parse. RootNode wraps a string in <x-turndown id="turndown-root">, parses it, and takes that element as the root. A DOM node is deep-cloned, so your document is not changed. Then collapseWhitespace runs on the tree (root-node.js). In Node the parser is domino.createDocument. In a browser it is the native DOMParser, or document.implementation.createHTMLDocument as a fallback (html-parser.js).
4. Walk. process reduces over child nodes. Text nodes are escaped with escape(), except inside code. Element nodes go to replacementForNode. That function picks a rule, converts the children first (depth-first), trims the content when the node has flanking whitespace, and calls rule.replacement(content, node, options) (turndown.js).
5. Join. join merges each replacement into the output and keeps at most two newlines between blocks (turndown.js).
6. Finish. postProcess calls every rule’s append hook (reference-style links use it to add the link list at the end) and trims leading and trailing whitespace (turndown.js).

Key components

Rule selection

Rules.forNode applies a fixed order. A blank node always gets the blank rule. After that come the rules in array (added rules are put first, then the CommonMark ones), then keep filters, then remove filters, and finally the default rule (rules.js). A filter can be a tag name, an array of tag names, or a function (node, options) => boolean (rules.js). One consequence: remove('a') does nothing, because the inlineLink rule matches first. To override a built-in element, use addRule.

Built-in rules

commonmark-rules.js has 13 rules: paragraph, line break, headings (setext for h1–h2 unless atx), blockquote, lists, list items (with start for <ol>), indented and fenced code blocks (the fence grows when the code contains backticks, and the language-* class sets the info string), horizontal rule, inline and reference links, emphasis, strong, inline code and images (commonmark-rules.js). Tables, strikethrough and task lists are not included. They come from the separate turndown-plugin-gfm package.

The default rule

An element with no matching rule keeps its converted children, with blank lines around it if it is a block element (turndown.js). For scraped pages this is the most important behaviour. Because the default rules keep the text of any element they do not match, <script> and <style> bodies end up as plain text, a <table> flattens to one paragraph per cell, and <nav> links are kept. Plan to remove(['script', 'style', 'noscript']) and either add the GFM table plugin or keep('table').

Escaping

escapeMarkdown runs 13 regexes over every text node. It backslash-escapes \ * _ \ [ ]everywhere and- + = # > ~~~ 1.at the start of a line ([utilities.js](https://github.com/mixmark-io/turndown/blob/aa84dfa3e2361edea8c43acbfc2b7a9363494bfd/src/utilities.js#L82-L102)). This keeps the output correct when rendered, but it adds many backslashes to prose with underscores or asterisks. For LLM input, which is never rendered, you can overrideTurndownService.prototype.escape` with a lighter function.

Extending it

• addRule(key, { filter, replacement }). Adds a rule ahead of the built-ins. The replacement function gets the converted content, the DOM node and the options. An append hook can add text at the end of the document.
• keep(filter) / remove(filter). keep outputs a node as raw HTML. remove drops it with its children. Both apply only to nodes no other rule matched.
• use(plugin). A plugin is a function that receives the service and calls the methods above. turndown-plugin-gfm is the usual one.
• Options. blankReplacement, keepReplacement and defaultReplacement change the three special rules. preformattedCode keeps whitespace inside <code>.

Running it

• npm install turndown. Node 18 or newer, according to engines. The npm package has CJS, ES and UMD builds, plus browser builds that leave out domino through the browser field in package.json. An IIFE file in dist/ works with a <script> tag.
• No services, network or configuration are needed. Conversion is synchronous and runs in the calling thread. Wrap it in a worker if you convert large pages on a busy server.
• Security: in Node, domino does not run scripts or load external resources. In a browser, the native parser decides. SECURITY.md advises sanitizing untrusted HTML first, because the parser’s side effects happen before Turndown runs.

Strengths and caveats

• Strength: small and predictable. One dependency, synchronous, deterministic output, and a short rule list that is easy to read and override.
• Strength: correct Markdown. Fence lengths adapt to the code, link destinations with spaces or brackets are escaped, and whitespace handling copies browser rendering.
• Strength: works everywhere. The same API runs in Node, browsers, extensions and edge runtimes.
• Caveat: no content extraction. It converts the whole input, including boilerplate and script text. You need a readability step or selectors before it.
• Caveat: CommonMark only by default. Tables are flattened unless you add a plugin. This matters a lot for data-heavy pages fed to an LLM.
• Caveat: aggressive escaping. The backslashes cost tokens and can confuse a model. Override escape if the output is only for machines.
• Caveat: shared state in reference links. The referenceLink rule stores pending references on the shared rule object and clears them in append (commonmark-rules.js). This is safe in synchronous use, but it ties together all instances that use linkStyle: 'referenced'.

Sources: code at aa84dfa, OpenDeepWiki wiki (7 pages), verified Q&A.