mishushakov/llm-scraper

Overview

llm-scraper is a thin TypeScript layer between a Playwright Page and the Vercel AI SDK. You open and navigate the page yourself, describe the data you want as an AI SDK Output (usually Output.object({ schema }) with a Zod schema), and call run, stream or generate. The library turns the page into one model input (cleaned HTML, raw HTML, Markdown, Readability text, a screenshot, or your own string) and makes one generateText or streamText call.

The whole library is four source files and about 330 lines. It has no fetching, waiting, crawling, retry, chunking or anti-bot logic. Those are left to Playwright and to the caller. Its runtime dependencies are only ai, @ai-sdk/provider and turndown. Playwright and Zod are dev dependencies, because the library only uses Playwright’s Page type and never launches a browser (package.json).

It suits developers who already drive a browser from Node and want typed extraction from arbitrary pages without writing selectors. The generate mode is the cost-saving option: the model writes a JavaScript extractor once, and you run it with page.evaluate on later pages.

Architecture

flowchart LR
  C["Caller: Playwright page + Output schema"] --> L["LLMScraper (index.ts)"]
  L --> P["preprocess (preprocess.ts)"]
  P --> CL["cleanup (in-page DOM strip)"]
  P --> TD["Turndown markdown"]
  P --> RD["Readability via CDN import"]
  P --> SS["page.screenshot base64"]
  L --> M["models.ts"]
  M --> GT["generateText (run)"]
  M --> ST["streamText (stream)"]
  M --> GC["generateText code prompt (generate)"]
  GT --> D["data, url"]
  ST --> S["partialOutputStream, url"]
  GC --> K["code, url"]

Component	Path	Role
LLMScraper	src/index.ts	Public class: run, stream, generate, all wrapping preprocess and one model call
Preprocessor	src/preprocess.ts	Turns the live page into {url, content, format} in one of six formats
DOM cleanup	src/cleanup.ts	Runs inside the page. Removes noisy elements and attributes before HTML capture
Model calls	src/models.ts	Builds messages and system prompts, calls the AI SDK, strips code fences
Examples	examples/*.ts	HN extraction, streaming, codegen, Ollama, tool use
Tests	tests/	Live Vitest runs against Hacker News and example.com with gpt-4o-mini

How a request flows

For scraper.run(page, Output.object({ schema }), { format: 'markdown' }):

1. LLMScraper.run calls preprocess(page, options), then generateAISDKCompletions with the model you passed to the constructor (index.ts#L24-L56).
2. preprocess reads page.url() and picks the format, html by default (preprocess.ts#L25-L83):
   • raw_html: page.content().
   • markdown: page.innerHTML('body') converted with new Turndown().turndown(...) in Node.
   • text: inside the page, dynamically import('https://cdn.skypack.dev/@mozilla/readability'), run Readability(document).parse(), and return Page Title: ... plus textContent.
   • html: page.evaluate(cleanup), then page.content().
   • image: page.screenshot({ fullPage }) as base64.
   • custom: await options.formatFunction(page). It throws if the function is missing.
3. prepareAISDKPage wraps the content as a single text part, or an image part for image (models.ts#L25-L35).
4. generateAISDKCompletions builds messages = [user: page content, ...options.messages], uses the default system prompt “You are a sophisticated web scraper. Extract the contents of the webpage” unless you override system, and spreads the remaining options into generateText({ model, output, system, messages, ...rest }) (models.ts#L12-L63).
5. It returns { data: result.output, url }. Schema enforcement and parsing come from the AI SDK’s Output handling. The library itself never calls schema.parse.

stream is the same, except that it calls streamText and returns { stream: partialOutputStream, url } for for await consumption (models.ts#L65-L91).

Key components

Preprocessing and cleanup

cleanup is a function serialised into the page by page.evaluate. It walks document.querySelectorAll('*'), removes 30 tag types (script, style, svg, img, form, input, button, header, footer, nav, aside, head and others), and strips attributes whose names start with style, src, alt, title, role, aria-, tabindex, on or data- (cleanup.ts#L1-L60). The goal is fewer tokens, not boilerplate detection. class, id and href survive. The cleanup mutates the live page: after an html-mode call, the same Page has no scripts, forms, nav or images. This matters if you keep using the page, or if you generate code in html mode and then evaluate it on the stripped DOM. The bundled codegen example uses raw_html.

text mode loads Readability from a public CDN at run time inside the page. It needs outbound network access from the browser, and a page Content Security Policy can block it.

Model integration

The library accepts any AI SDK LanguageModel, so providers are whatever AI SDK adapters you install. The examples use @ai-sdk/openai and ollama-ai-provider-v2, and the README also shows Anthropic, Google and Groq. ScraperLLMOptions is CallSettings plus system and extra messages, so temperature, max tokens and retries go to the SDK unchanged (index.ts#L11-L22). The whole preprocessed page goes into one message. Nothing counts tokens, chunks or truncates, so a long page in raw_html can exceed the context window.

Code generation

generateAISDKCode awaits output.responseFormat to get the JSON Schema that the AI SDK derived from your Zod schema. It then sends Website / Schema / Content as one user message under a system prompt that asks for a comment-free IIFE, and strips Markdown fences from the reply with stripMarkdownBackticks (models.ts#L93-L129). It returns { code, url } and does not run the code. The caller runs page.evaluate(code) and should validate the result, as examples/codegen.ts does with schema.parse. ScraperGenerateOptions limits format to html | raw_html at the type level.

Extending it

• Custom input: format: 'custom' with a formatFunction(page) lets you send just one region, an accessibility snapshot, or anything else you build.
• Prompting: override system or append messages (for example few-shot hints) through options.
• Providers and settings: any AI SDK model and any CallSettings.
• Agent tool: examples/toolUse.ts wraps scraper.run in an AI SDK tool(). Note that the example ignores the url argument the model passes in and always opens Hacker News.
• Stealth, auth, waiting: configure Playwright before you hand over the Page (contexts, cookies, waitForSelector, a stealth plugin). The library touches none of this.

Running it

npm i zod playwright llm-scraper plus an AI SDK provider package such as @ai-sdk/openai, and that provider’s API key in the environment. The package is ESM ("type": "module") and builds with tsc. npm test runs Vitest tests that launch Chromium, hit news.ycombinator.com and example.com, and call gpt-4o-mini, so they need network access and an OpenAI key (tests/index.ts). No server or database is involved.

Strengths and caveats

• Strength: very small surface. Three methods, one model call each, about 330 lines. Easy to audit or vendor.
• Strength: typed output and multimodal input. AI SDK Output gives schema-shaped data. image mode lets vision models read layouts that HTML hides.
• Strength: codegen escape hatch. generate trades per-page LLM cost for a reusable extractor you can review.
• Caveat: no fetching or rendering logic. Readiness, navigation, retries and blocking are all the caller’s job.
• Caveat: one-shot context. No chunking or token budgeting. Cost and context limits scale with page size and format choice.
• Caveat: side effects and external fetches. html mode rewrites the live DOM, and text mode pulls code from cdn.skypack.dev at run time.
• Caveat: generated code is untrusted. The generate output is run in the page as-is. Nothing sandboxes or checks it beyond what you add.

Sources: code at 2b43d99, deepwiki-open wiki (11 pages), OpenDeepWiki wiki (9 pages), verified Q&A.