What Is Web Scraping? From an HTML Page to JSON

Correction (2026-09-29)

Fetching a page does not grant access to its underlying database, and a scraping service cannot promise universal access or freedom from blocks. This refresh removes those earlier claims and adds a tested HTML-to-JSON example with explicit validation failures.

Web scraping is extracting selected information from web pages into data you can use elsewhere. Instead of copying each product name and price by hand, a program can read the page, find those fields and save them as records.

The important boundary is what the page actually provides. A scraper processes the content it receives; it does not gain access to a website's private database. Below, an owned sample catalog becomes JSON, with a check that catches a missing price before incomplete data reaches the output.

What can you use web scraping for?

As a simple example, imagine comparing the displayed names and prices in a product catalog. You need one record per product, rather than the surrounding navigation and layout. Another possible task is collecting the titles and dates from public document listings.

The workflow is: choose a source, retrieve its content, extract the fields you need, check the result, then save it for analysis. JSON and CSV are possible output formats; the choice depends on what will consume the records. This tutorial uses JSON.

Scraping, crawling, APIs and rendering

These are related steps, with different jobs:

Term	Job	In this example
Fetching	Retrieve content from a source	We start from a local HTML file, so no website request is needed
Parsing and extraction	Interpret the HTML and select fields	Beautiful Soup finds product cards and reads their text
Crawling	Discover and fetch pages to visit	No links are followed
Data API	Expose data through a programmatic interface	A suitable source API could replace HTML parsing
Rendering	Let a browser build the page, including changes made by JavaScript	Not needed for the static catalog

Beautiful Soup supplies the parsing and selection tools. A browser's Document Object Model (DOM) represents a document so code can inspect or change it. That is why HTML received initially and the document after JavaScript runs can differ.

An API is an interface for programs. A website's documented data API and a scraping service's API are different choices: the former supplies the source's data interface; the latter handles retrieval on your behalf. If a suitable source API or export already supplies your required fields, start there. See MDN's API introduction for the broader concept.

For a fuller comparison, see web scraping versus web crawling.

Try it: HTML product cards to JSON

Get the files from the example directory, then open a terminal in that directory. The example was tested with Python 3.14.6 and Beautiful Soup 4.15.0. Its requirements file pins the dependencies.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
./run.sh

The activation command is for Bash/Zsh. Installing dependencies may need internet access; the exercise itself runs offline and requires no API key.

All products are synthetic teaching data, not actual offers. This is the first card in fixtures/catalog.html:

<article class="product" data-id="p1">
  <h2>Notebook</h2>
  <span class="price">USD 4.50</span>
</article>

The card's data-id becomes id, its h2 text becomes title, and its .price text becomes price. Prices remain text, including the currency label; the example does not guess numeric formats or currencies.

Here is the complete extract.py from the packet:

"""Extract synthetic product cards from an owned, local HTML file."""
import argparse
import json
import sys
from pathlib import Path

from bs4 import BeautifulSoup


def extract_products(html):
    soup = BeautifulSoup(html, "html.parser")
    cards = soup.select("article.product")
    if not cards:
        raise ValueError("no product cards found")
    records = []
    seen_ids = set()
    for card in cards:
        product_id = (card.get("data-id") or "").strip()
        if not product_id:
            raise ValueError("product card: missing id")
        if product_id in seen_ids:
            raise ValueError(f"{product_id}: duplicate id")
        record = {"id": product_id}
        for field, selector in (("title", "h2"), ("price", ".price")):
            element = card.select_one(selector)
            value = element.get_text(" ", strip=True) if element else ""
            if not value:
                raise ValueError(f"{product_id}: missing {field}")
            record[field] = value
        seen_ids.add(product_id)
        records.append(record)
    return records


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("html", type=Path)
    args = parser.parse_args()
    try:
        records = extract_products(args.html.read_text(encoding="utf-8"))
    except (OSError, UnicodeError, ValueError) as exc:
        print(f"ERROR: {exc}", file=sys.stderr)
        return 1
    print(json.dumps(records, ensure_ascii=False, indent=2))
    return 0


if __name__ == "__main__":
    sys.exit(main())

article.product selects each product card. Within that card, select_one() finds the title and price. Validation happens before json.dumps() serializes the records: a missing required value stops the whole extraction instead of quietly writing a partial catalog.

Run the extractor on its own:

python extract.py fixtures/catalog.html

Captured output:

[
  {
    "id": "p1",
    "title": "Notebook",
    "price": "USD 4.50"
  },
  {
    "id": "p2",
    "title": "Café & tea mug",
    "price": "USD 12.00"
  },
  {
    "id": "p3",
    "title": "Desk lamp",
    "price": "USD 25.00"
  }
]

The second title shows both a Unicode character and an HTML entity: Café & tea mug becomes Café & tea mug. The packet compares this output against a separately authored fixtures/expected-records.json, rather than generating its own expected answer from the extractor.

Use the Beautiful Soup cheat sheet when you need more selectors or text-handling examples.

Catch a missing field before saving the data

The second fixture contains a complete first product followed by a product with no price. Run:

python extract.py fixtures/missing-price.html

Captured error:

ERROR: p2: missing price

The command exits with code 1 and writes no JSON to stdout. That matters: successfully reading a file does not mean every required field was extracted.

Local check	Observed result
Complete catalog	3/3 expected records and 9/9 field values match the independent expected output
Missing price	Exit 1; no partial JSON
Card created only inside JavaScript	Exit 1: no product cards found

The last fixture puts its product markup inside a script. Beautiful Soup parses the supplied HTML; it does not run that script to create a product card. A JavaScript-dependent page needs a different retrieval approach before the same kind of extraction can work.

These results describe the supplied fixtures. They are not an accuracy benchmark across websites. Different page layouts need different selectors; multiple prices or pagination also need rules this example does not implement.

When would you need ScrapingAnt?

You do not need ScrapingAnt for this exercise, or for a suitable data API or export you can already access directly.

For a later JavaScript-dependent task, with browser=true, ScrapingAnt renders the target page with JavaScript and returns its HTML. A request with JavaScript rendering through a datacenter proxy costs 10 API credits. Keep return_page_source unset or false for this rendered-content mode; see the headless browser documentation.

That capability is documented, not tested in this packet. The local run made no target HTTP requests or paid API calls. Whatever retrieves the HTML, your extraction still needs to check that the required records and fields are present.

For that next step, see scraping a dynamic website with Python.

Examples tested on 2026-09-29 with Python 3.14.6, beautifulsoup4 4.15.0 and soupsieve 2.9.2. Code: tested evidence packet.

This article was drafted with AI assistance from a tested evidence packet. Oleg Kulyk is responsible for the published article and corrections.