raznem/parsera

Overview

Parsera is a small Python library for “describe the fields, get JSON back” scraping. You give it a URL and a dict such as {"Title": "News title", "Points": "Number of points"}. It opens the page in headless Firefox through Playwright, grabs the rendered HTML (iframes included), turns it into Markdown with markdownify, and asks a language model to return a list of records with those keys. There are no selectors, no XPath and no learned extraction rules. Every run re-reads the page with the model.

The whole package is about 1,100 lines of code. The interesting decision is where the LLM lives. A bare Parsera() does not call OpenAI at all. It sends the page to the vendor’s hosted endpoint, https://api.parsera.org/v1/parse, authenticated with PARSERA_API_KEY (parsera.py, api_extractor.py). Pass any LangChain chat model and extraction runs locally instead, through a chunking extractor with its own prompts. So the open-source library is both a client for the commercial Parsera service and a self-contained LLM scraper.

It suits one-off or low-volume jobs where the page layout changes often and per-page LLM cost is acceptable: a listing page, a product table, a news front page. It is a single-URL tool. There is no crawler, queue, link following or retry logic.

Architecture

flowchart LR
  U["Your code / CLI"] --> P["Parsera.run / arun"]
  P --> L["PageLoader"]
  L --> FF["Headless Firefox (Playwright)"]
  L --> HTML["HTML + iframe HTML"]
  HTML --> X{"Extractor"}
  X -->|"no model"| API["APIExtractor -> api.parsera.org"]
  X -->|"model"| CH["ChunksTabularExtractor"]
  X -->|"model, typed=True"| ST["StructuredExtractor"]
  CH --> MD["markdownify -> chunks"]
  ST --> MD
  MD --> LLM["LangChain chat model"]
  LLM --> J["JSON records"]
  API --> J

Component	Path	Role
Facade	parsera/parsera.py	Parsera picks an extractor, owns a PageLoader, exposes run (sync) and arun (async)
Page loader	parsera/page.py	Firefox launch, context with proxy and cookies, playwright-stealth, scrolling, iframe capture
API extractor	parsera/engine/api_extractor.py	Extractor base class and the hosted-API client
Local extractors	parsera/engine/simple_extractor.py	LocalExtractor plus TabularExtractor, ListExtractor, ItemExtractor (prompt variants)
Chunking	parsera/engine/chunks_extractor.py	Token-based splitting, sequential append prompts, final merge call
Typed output	parsera/engine/structured_extractor.py	Builds a Pydantic schema from field types and uses with_structured_output
Model wrappers	parsera/engine/model.py	GPT4oMiniModel, AzureModel, HuggingFaceModel singletons
CLI	parsera/main.py	python -m parsera.main URL --scheme/--file --scrolls --output

How a request flows

Take Parsera(model=llm).run(url, elements):

1. Choose the extractor. The constructor accepts either a model or an extractor, never both. No model means APIExtractor. A model means ChunksTabularExtractor, or StructuredExtractor when typed=True (parsera.py).
2. Open a session once. _run creates the browser session on the first call only: PageLoader.create_session launches Firefox headless, opens a context with the optional proxy and cookies, and, with stealth=True (the default), rebuilds the context and applies stealth_async. An initial_script callback runs here, which is where login flows go (parsera.py, page.py).
3. Fetch. fetch_page calls page.goto, waits for networkidle (a timeout is silently ignored), runs the per-call playwright_script, then either scrolls or reads the full HTML (page.py).
4. Collect HTML. get_full_html concatenates the main document’s outerHTML with every non-detached iframe’s HTML, fetched concurrently with asyncio.gather (page.py).
5. Convert and split. The extractor runs MarkdownConverter().convert(html), then splits the Markdown with LangChain’s RecursiveCharacterTextSplitter, measuring length in o200k_base tokens. The default chunk is 100,000 tokens with a one-third overlap (chunks_extractor.py).
6. Extract. One chunk means one call: a few-shot system prompt asking for a list of flat records, plus the Markdown and the requested fields, parsed with JsonOutputParser (chunks_extractor.py).
7. Multi-chunk path. For several chunks, the loop runs sequentially. Each call gets the tail of the previous chunk’s records and is asked to fix truncated rows and continue the sequence. A final LLM call merges all per-chunk lists and removes duplicates (chunks_extractor.py). A page split into N chunks costs N + 1 calls.
8. Return. The parsed JSON (normally list[dict]) is returned as is. There is no schema validation on this path.

Key components

PageLoader

PageLoader holds one Playwright instance, one browser, one context and one page, and reuses them across calls on the same Parsera object. The browser is always Firefox. Stealth mode reads navigator.userAgent, replaces HeadlessChrome/ with Chrome/, reopens the context with that UA, and runs playwright-stealth with its UA, plugins and vendor patches turned off (page.py). The UA rewrite targets Chrome’s headless marker, so it does nothing on Firefox. The remaining playwright-stealth scripts are the actual evasion layer. There is no CAPTCHA handling and no rate limiting.

Infinite scroll

scroll_page installs a MutationObserver that records the outerHTML of every removed element. It then scrolls to the bottom up to scrolls_limit times, sleeping 2 s each time and stopping when the height stops growing. It returns every captured snapshot plus the removed nodes, concatenated (page.py). This handles virtualised lists that drop off-screen rows. The cost is heavy duplication, because each snapshot repeats the whole page, and the merge step is left to the LLM. Iframes are not captured on this path.

Extractors

All local extractors share LocalExtractor.run: convert to Markdown, fill a template, call model.ainvoke, parse JSON (simple_extractor.py). The subclasses differ only in system prompt: TabularExtractor (list of rows), ListExtractor (dict of lists) and ItemExtractor (one dict). These single-call extractors never chunk. ChunksTabularExtractor is the only one Parsera builds for you.

Typed extraction

With typed=True, each field is {"description": ..., "type": "string"|"integer"|"number"|"bool"|"list"|"object"|"any"}. StructuredExtractor.create_schema builds a Pydantic record model where every field is optional, wraps it in {reasoning, data: [record]}, and binds it with model.with_structured_output (structured_extractor.py). The reasoning field is a built-in chain-of-thought slot that costs output tokens and is then dropped. An all-null result is turned into [].

Hosted API path

APIExtractor.run posts the raw HTML (not Markdown), the prompt, the attributes as {name, description} pairs and mode: "standard" to the Parsera API with a 180 s timeout. A detail key in the response becomes a Python warning, not an exception (api_extractor.py). It uses blocking requests inside an async method, so arun calls on this path block the event loop.

Extending it

• Any LangChain model. Pass model= with an OpenAI, Azure, Ollama or other BaseChatModel. HuggingFaceModel wraps a local transformers pipeline and runs it in a thread executor (model.py). Note that the bundled wrappers are @singletons, so the first constructor arguments win for the whole process.
• Custom extractor. Subclass Extractor (one async run(content, attributes, prompt)) or a LocalExtractor with your own system_prompt, and pass it as extractor=. ChunksTabularExtractor also accepts chunk_size, a token_counter and a custom MarkdownConverter.
• Playwright hooks. initial_script runs once per session, and playwright_script runs after every navigation. Both receive and return a Page, so clicks, logins and waits go there.
• Direct loader use. PageLoader(browser=...) accepts your own Playwright browser, for example a headed one with slow_mo for debugging, as the bundled scrolling example does.

Running it

• pip install parsera and playwright install (Firefox is the browser that is used). Python 3.10 or newer.
• Hosted mode: set PARSERA_API_KEY and call Parsera().run(url=..., elements=...).
• Local mode: pass model=, and set whatever key that provider needs.
• CLI: python -m parsera.main URL --scheme '{"title":"h1"}' --scrolls 5 --output out.json. The CLI always builds GPT4oMiniModel, so it needs OPENAI_API_KEY, not a Parsera key (main.py).
• Docker: the image installs Poetry and Playwright Firefox and uses the CLI as its entrypoint. docker-compose.yaml mounts a scheme.json and an output/ folder.

Strengths and caveats

• Strength: tiny and readable. The whole data path fits in five files. It is easy to audit, fork, or lift the chunk-and-merge idea from.
• Strength: chunking that respects rows. Overlapping chunks, “continue this sequence” prompts and a merge call are a reasonable answer to tables longer than a context window.
• Strength: lazy-load capture. The removed-node observer recovers rows that virtualised lists delete while scrolling, which many scrapers silently miss.
• Caveat: the default is a paid cloud call. Parsera() with no model sends the whole page HTML to a third-party API. You must pass model= to keep data local.
• Caveat: full LLM cost on every page. Nothing is cached or turned into selectors. A large page that splits into N chunks costs N + 1 sequential calls, and the merge call re-sends all extracted data.
• Caveat: weak anti-bot. Firefox plus a Chrome-oriented UA patch plus playwright-stealth is basic. There are no CAPTCHA, retry or backoff features.
• Caveat: session reuse is fragile. run() calls asyncio.run each time, while the Playwright session from the first call is kept on the loader. Repeated sync calls on one instance can hit a session bound to a closed event loop. Use arun for multi-page jobs, or a fresh instance per URL.
• Caveat: hygiene. The committed run.py sets a literal PARSERA_API_KEY value, and the test suite needs live Azure OpenAI credentials and network access.

Sources: code at 865d777, deepwiki-open wiki (10 pages), OpenDeepWiki wiki (9 pages), verified Q&A.