alirezamika/autoscraper

Overview

AutoScraper is a small Python library (one class, about 725 lines, plus a 59-line utils.py) that writes scraping rules for you. You give it a page and a few values you can see on that page, for example a product title and a price. It finds the elements that hold those values, records the path from the document root down to each one, and stores those paths as rules. Later you apply the rules to another page with the same layout and get back the matching values.

Despite the category it sits in, there is no machine learning and no LLM here. “Learning” means a DOM walk plus string comparison, with optional fuzzy matching from difflib.SequenceMatcher. Pages are fetched with a plain requests.get, so JavaScript-rendered content is invisible unless you pass in rendered HTML yourself. There is no crawler, CLI or service: it is a library for people who want a quick extractor for a list or detail page without writing CSS selectors.

The interesting part is the rule format. A rule is a JSON-serialisable dict: a list of (tag, {class, style}, sibling_index) steps plus a few flags. This makes rules easy to save, diff, prune and share.

Architecture

flowchart LR
  U["Caller: url or html + wanted_list"] --> S["_get_soup"]
  S --> F["_fetch_html (requests.get)"]
  S --> BS["BeautifulSoup lxml tree"]
  BS --> GC["_get_children / _child_has_text"]
  GC --> BST["_build_stack"]
  BST --> SL["stack_list (rules)"]
  SL --> J["save / load JSON"]
  SL --> SIM["_get_result_with_stack (similar)"]
  SL --> EX["_get_result_with_stack_index_based (exact)"]
  SIM --> CR["_clean_result"]
  EX --> CR
  CR --> R["list or dict of strings"]

Component	Path	Role
AutoScraper class	autoscraper/auto_scraper.py	Public API: build, get_result_similar, get_result_exact, get_result, rule management, save/load
Fetch and parse	auto_scraper.py _fetch_html, _get_soup	One requests.get, encoding fix-up, NFKD normalisation, BeautifulSoup(html, "lxml")
Matcher	auto_scraper.py _child_has_text, _get_children	Finds elements whose text, direct text or attribute equals a wanted value
Rule builder	auto_scraper.py _build_stack	Walks from a matched element up to the root and records the path
Rule replay	auto_scraper.py _get_result_with_stack, _get_result_with_stack_index_based	Re-walks a path on a new tree, broadly or by exact position
Utilities	autoscraper/utils.py	text_match, FuzzyText, normalize, ResultItem, order-preserving dedupe

How a request flows

Learning rules with build():

1. build(url=..., wanted_list=[...]) checks that some target was given, then calls _get_soup, which either parses the html you passed or calls _fetch_html. That method does one requests.get with a hard-coded Chrome 84 User-Agent, a Host header taken from the URL, and any request_args you supplied; it fixes a wrongly declared ISO-8859-1 encoding (auto_scraper.py#L95-L121).
2. A plain wanted_list becomes wanted_dict = {"": wanted_list}, so aliased and unaliased targets share one loop. Unless update=True, existing rules are dropped (auto_scraper.py#L226-L258).
3. For each wanted value, _get_children scans every element in reverse document order. _child_has_text accepts an element if its full text matches (but not when its parent has the same text, so the outermost element with exactly that text wins), if its own non-recursive text matches, or if an attribute value matches. For href/src it also tries the URL resolved against the page URL (auto_scraper.py#L135-L175).
4. _build_stack climbs from the match to the root. At each level it stores the parent tag, its class and style only, and the index of the child among siblings with the same tag and attributes. The rule gets a SHA-256 hash and a stack_id of rule_ plus 8 hex chars (auto_scraper.py#L260-L297).
5. Each new rule is replayed at once on the same tree with the “similar” strategy, and those values are what build() returns. Rules are deduplicated by hash with unique_stack_list.

Applying rules with get_result_similar() / get_result_exact():

1. Both go through _get_result_by_func, which parses the page once, optionally tags every element with a child_index (needed for keep_order and group_by_alias), and runs the chosen replay function for each rule (auto_scraper.py#L406-L469).
2. Similar replay keeps every matching child at every level (findAll(tag, attrs, recursive=False)), which is how one example item generalises to all rows of a list. Only at the leaf does it pick by stored index, unless contain_sibling_leaves=True (auto_scraper.py#L330-L370).
3. Exact replay follows the stored sibling index at every level and returns at most one value per rule (auto_scraper.py#L372-L404).
4. _clean_result flattens, sorts by DOM position if asked, and removes duplicates. Or it returns a dict keyed by stack_id (grouped=True) or by alias (group_by_alias=True).

Key components

Matching

text_match in utils.py#L35-L40 is the only comparison primitive. A compiled regex is tested with fullmatch. A ratio of 1.0 (the default) means exact string equality. Anything lower uses SequenceMatcher.ratio(). That is text_fuzz_ratio at build time. At replay time, attr_fuzz_ratio < 1.0 wraps the stored class/style values in FuzzyText, whose search() method BeautifulSoup calls as a matcher (utils.py#L52-L59). Fuzzy matching is O(elements x targets) SequenceMatcher calls, so it gets slow on large pages.

Rule format

_get_valid_attrs keeps only class and style, and fills in empty strings when they are missing (auto_scraper.py#L123-L133). id, data-* and other attributes never appear in a rule. Besides the path, a rule stores wanted_attr (which attribute to read, or None for text), is_full_url, is_non_rec_text, alias, and the page url when the value was a resolved link.

Rule management and persistence

save() writes {"stack_list": [...]} as JSON. load() also accepts the older bare-list format (auto_scraper.py#L53-L93). remove_rules, keep_rules and set_rule_aliases edit the list by stack_id. generate_python_code() still exists but only prints a deprecation notice (auto_scraper.py#L673-L725). The usual workflow: build, inspect get_result_similar(grouped=True) to see which rule produced which values, keep_rules the good ones, then save.

Combined call

get_result() parses once and returns a (similar, exact) tuple. It does not take keep_order or contain_sibling_leaves (auto_scraper.py#L613-L671).

Extending it

There are no plugin hooks. The practical extension points are:

• Bring your own fetcher. Every entry point accepts html=, and get_result_similar/get_result_exact also accept a pre-built soup=. Render with Playwright or fetch through your own proxy layer, then pass the HTML in.
• request_args goes straight to requests.get, so proxies, cookies, timeouts and extra headers work. Your headers are merged over the default UA.
• Subclassing. _get_valid_attrs and request_headers are class-level. Overriding them changes which attributes rules depend on, or the default UA.
• Rules as data. The JSON rule file can be edited or generated by other tools.

Running it

pip install autoscraper (or from git). Its dependencies are requests, bs4 and lxml, Python 3.6+ (setup.py#L1-L30). No services, keys or browsers are needed. Tests use pytest under tests/unit and tests/integration.

Strengths and caveats

• Strength: tiny and dependency-light. The whole algorithm fits in one file you can read in an hour, and rules are plain JSON.
• Strength: no per-page cost. After build, extraction is pure tree traversal. It is fast and deterministic, with no API calls.
• Strength: list generalisation. One example row often yields every row, because similar-mode replay fans out at every non-leaf level.
• Caveat: structural brittleness. Rules depend on exact tag paths, class/style values and sibling positions from the root down. A wrapper div added to the layout breaks every rule. attr_fuzz_ratio only helps with class-name drift.
• Caveat: noisy rule sets. Every element that matches a wanted value produces a rule. Common strings, such as a price that appears twice, create extra rules you must prune by hand.
• Caveat: static HTML only. No JS rendering, retries, rate limiting or rotation. The built-in UA is a 2020 Chrome string.
• Caveat: single page per call. No link following or crawling; you loop over URLs yourself.

Sources: code at 68a8181, deepwiki-open wiki (16 pages), OpenDeepWiki wiki (6 pages), verified Q&A.