269 posts tagged with "web scraping"

Choose the smallest representation that preserves the information your extraction task needs. Plain text can be sufficient for visible values. Markdown is a useful candidate when headings, tables and link destinations matter. Keep HTML when the answer depends on DOM attributes or script data that conversion removes.

We tested the same six-field extraction contract on three saved inputs, then on 15 hosted MCP returns. HTML produced correct records in each tested case. Markdown worked for the purchase-link case but returned empty arrays for the table. Local text worked for the table; hosted raw-mode text did not. These small, fixed-model results make the practical rule sharper: preserve the fields, then validate complete records before comparing cost.

Choose a scraping method by the records it delivers. A fast response containing a loading placeholder cannot satisfy a price-monitoring job. A browser may add unnecessary work when the required table is already in the HTML.

The recorded benchmark compared Python Requests, direct Playwright and both ScrapingAnt modes against two controlled, already published fixtures. All four extracted the static catalog correctly. Only the browser arms passed the delayed JavaScript diagnostic. For the catalog, ScrapingAnt raw delivered the same validated records for 90% fewer API credits than ScrapingAnt rendered: 0.3333 credits per accepted record observation, versus 3.3333 credits with rendering. Direct Requests and Playwright had lower observed latency; their total operating cost was not measured.

Apify's monthly Starter plan is $19 with $19 of prepaid usage. That does not mean every Actor costs $19, or that its event price always has compute and proxy charges added on top. The chosen Actor's charging model determines which line items consume that allowance.

This guide starts with the current prices, then provides an offline charge-reconciliation worksheet. It helps you add known charges once and expose missing inputs. All seven example ledgers are synthetic: their Actor IDs, versions, consumption and accepted-record counts are invented test inputs. A separate worked example uses observed Actor runs on owned synthetic pages. Neither set is a customer invoice. ScrapingAnt publishes this article and sells a competing retrieval API.

Firecrawl's basic scrape costs one credit per page. Your useful unit is the result your application can accept: a complete record, rendered content, or a table with the correct cell associations. Budgeting from successful HTTP responses alone misses that difference.

This guide answers the plan-price question first, then models a recurring URL list and explains a 200-credit controlled study. The study used free credits; it measured credit consumption, not cash expenditure. ScrapingAnt publishes this article and sells a competing retrieval API. There is no ScrapingAnt head-to-head test here.

Use page.locator("#catalog > .product") to find repeated elements in Python Playwright. Narrow to an intended record before reading a single value, or call all() to iterate the matches after the list has finished loading. To build records in the browser, use evaluate_all() with a JavaScript projection.

Finding a match is only part of extraction. In the examples below, .first returned a sponsored product, while an early count() found a loading placeholder. Both operations worked; both selected the wrong data. The runnable example waits for the catalog's ready state and checks every SKU, title, currency, price and link against independently specified records.

Use page.$() to get the first matching element and page.$$() to get all matching elements. When you want data rather than handles, page.$eval() runs a callback on the first match, and page.$$eval() passes all matches to a callback that can return JSON records.

The selector still needs to identify the right records, and the fields must be populated before you read them. This guide demonstrates both problems with a local catalogue: a promotional card that matches an overly broad selector, and product cards that exist before their titles and prices arrive.

You ask Claude Code to fetch a website and summarize it. Instead of useful content, you get back an empty <div> and a bunch of script tags. If you've used Claude Code for web scraping for more than a week, you've hit this wall. Here's why it happens and how to fix it in 30 seconds.

A job-posting refresh needs more than a list of titles. It needs stable posting IDs, source URLs, validated fields, and a way to distinguish a complete collection from a failed second page. Otherwise, a transient failure can turn into a false disappearance alert.

This walkthrough collects one employer listing that you are authorized to access. The runnable pagination and refresh example uses a fictional, owned careers site. Separate evidence includes an official Greenhouse JSON alternative and one accepted raw managed retrieval of a selected published Greenhouse posting. ScrapingAnt is an optional HTML transport; parsing and validation remain explicit Python code.

A scraper returning HTTP 200 can still produce an empty page, malformed data or the wrong price. Before connecting it to a downstream dataset, test what it does when retrieval, extraction, validation and storage fail separately.

The Python example below runs against an owned, local catalog. It checks retry limits, timeouts, record validation and duplicate delivery. Its most useful result is a failure of the data checks: three deliberately wrong prices passed the schema and reached storage.

Teams that depend on SERP data for competitive intelligence, content research, or data extraction increasingly look beyond Google because HTML pages are volatile, highly personalized, and protected by advanced anti-bot systems—issues that raise cost, legal risk, and maintenance burden for scrapers. The 2025 landscape favors an API-first approach with alternative search engines that return stable, structured JSON (or XML) and clear terms, making pipelines more reliable and compliant for SEO analytics and web data extraction.

Among general-purpose options, Microsoft’s Bing remains the most practical choice for production pipelines due to its mature multi-vertical Web, Image, Video, and News endpoints, robust localization, and predictable quotas via the Azure-hosted Bing Web Search API (Bing Web Search API). For teams that value an independent index with strong privacy posture, the Brave Search API provides web, images, and news in well-structured JSON and plan-based quotas.

Privacy-first and lightweight use cases sometimes start with DuckDuckGo. While it does not expose a full web search API, its Instant Answer (IA) API can power specific knowledge lookups, and its minimalist HTML endpoint is simple to parse at modest volumes—always within policy and with conservative rate limits (DuckDuckGo Instant Answer API, DuckDuckGo parameters). When you need a controllable gateway that aggregates multiple engines into a single JSON format, self-hosted SearXNG is a strong option; just remember that you—not SearXNG—are responsible for complying with each backend’s terms (SearXNG docs).