73 posts tagged with "python"

Imagine if you could describe the data you need in simple English, and AI takes care of the entire extraction and processing, whether from websites or local documents like PDFs, JSON, Markdown, and more. Even better, what if AI could summarize the data into an audio file or find the most relevant Google search results for your query—all at no cost or for just a few cents? This powerful functionality is provided by ScrapeGraphAI, an open-source AI-based Python scraper!

To set a cookie in Selenium Python, first navigate to its target origin, then call driver.add_cookie() with a name/value dictionary. Navigate or refresh afterward so the next request uses it. To reuse a session, save the relevant cookies, restore them on the same origin in the new browser, and check the resulting data.

That last step matters. In our controlled catalog, every extraction returned HTTP 200 and four product rows. Restoring only the session cookie still selected member prices—but in USD when the intended dataset was EUR. A successful navigation, an authenticated-looking page and the expected row count all missed the error.

This guide shows the cookie operations, a complete runnable save/restore example, and the measured failures behind that distinction. The catalog and accounts are synthetic; the results demonstrate mechanisms, not production-site success rates.

Use Selenium Python's execute_script to access localStorage in the current document. Navigate to the intended origin first, pass keys and values as arguments, and include return when you need a value back in Python. If the application reads a preference at startup, set it before entering that application page or trigger its documented refresh afterward.

The last step matters for extraction. In our fixture, writing eu after the table loaded changed localStorage but left all four prices in USD. A wait that checked only the stored key would have accepted the wrong dataset. The examples below check storage, the application's request and the resulting records.

To change localStorage on an already loaded origin, use page.evaluate(). If the application reads that value during startup, install an origin-guarded context.add_init_script() before navigation, or create the context with the required storage_state.

That timing matters when scraping. In our catalog fixture, setting the region to eu after the table had loaded changed the stored value but left four US-priced rows on screen. A reload produced the intended EUR records. A successful storage write alone did not validate the data.

Puppeteer is a powerful Node.js library that provides a high-level API for controlling browsers through the DevTools Protocol. It is commonly used for testing, web scraping, and automating repetitive browser tasks. However, Puppeteer's default settings can trigger bot detection systems, especially in headless mode.

Which Playwright stealth library should you use? In this comparison, Patchright returned BrowserScan's Normal verdict when headed, while its headless user agent was flagged. Node's playwright-extra with the stealth plugin and the Firefox-based Camoufox returned Normal in both modes. Python playwright-stealth had no failed Sannysoft checks but still received BrowserScan's Navigator flag.

That difference is the useful result: changing a browser property, passing a diagnostic page and successfully retrieving your target are separate tests. Start with a reproducible control and keep the browser version, operating system and launch mode attached to every observation.

To set cookies with Playwright Python, call context.add_cookies() with a list of cookie dictionaries. Supply url, or a suitable domain and path, and set the cookies before navigating when the first request needs them. Use context.storage_state() when the workflow also depends on supported browser storage beyond cookies.

Restoring the cookie does not necessarily restore the dataset. In our catalog, cookies-only restoration kept member pricing but selected USD instead of EUR. A browser-associated API request made the same mistake when it omitted a region parameter that the page normally reads from local storage.

This guide shows the working state handoff and the failures around it. The fixture is self-authored and synthetic; it demonstrates specific mechanisms, not the probability that a real website will accept a restored login.

Anti-bot systems such as Cloudflare, DataDome, PerimeterX (HUMAN) and Akamai look at many signals at once: the IP address and its reputation, the TLS handshake, the browser's JavaScript environment, and how the visitor behaves. Open-source libraries attack different parts of that list. None of them covers all of it, and how well they work changes every time the anti-bot vendors update.

This page lists the libraries that are worth a look in 2026, with their maintenance status, and ends with the cases where a scraping API is less work.

In the rapidly evolving landscape of web technologies, web scraping has emerged as a crucial tool for data extraction and analysis. As of 2024, two programming languages, JavaScript and Python, stand out as popular choices for developers engaging in web scraping tasks. Each language offers unique strengths and capabilities, making the decision between them a significant consideration for developers at all levels.

In the ever-evolving landscape of web automation and testing, two tools have consistently stood out: Playwright and Puppeteer. As of 2024, both have matured significantly, offering robust features for developers and testers alike. Both tools, developed by teams at Microsoft and Google respectively, offer robust solutions for automating browser tasks, but they cater to slightly different needs and preferences.