What are the best web scraping tools?

Part 1 of this series discussed setting up and running local models with Ollama to extract data from complex local documents such as HTML and JSON. This part will focus on using API-based models for more efficient web scraping.

Imagine if you could describe the data you need in simple English, and AI takes care of the entire extraction and processing, whether from websites or local documents like PDFs, JSON, Markdown, and more. Even better, what if AI could summarize the data into an audio file or find the most relevant Google search results for your query—all at no cost or for just a few cents? This powerful functionality is provided by ScrapeGraphAI, an open-source AI-based Python scraper!

To set a cookie in Selenium Python, first navigate to its target origin, then call driver.add_cookie() with a name/value dictionary. Navigate or refresh afterward so the next request uses it. To reuse a session, save the relevant cookies, restore them on the same origin in the new browser, and check the resulting data.

That last step matters. In our controlled catalog, every extraction returned HTTP 200 and four product rows. Restoring only the session cookie still selected member prices—but in USD when the intended dataset was EUR. A successful navigation, an authenticated-looking page and the expected row count all missed the error.

This guide shows the cookie operations, a complete runnable save/restore example, and the measured failures behind that distinction. The catalog and accounts are synthetic; the results demonstrate mechanisms, not production-site success rates.

Use Selenium Python's execute_script to access localStorage in the current document. Navigate to the intended origin first, pass keys and values as arguments, and include return when you need a value back in Python. If the application reads a preference at startup, set it before entering that application page or trigger its documented refresh afterward.

The last step matters for extraction. In our fixture, writing eu after the table loaded changed localStorage but left all four prices in USD. A wait that checked only the stored key would have accepted the wrong dataset. The examples below check storage, the application's request and the resulting records.

To change localStorage on an already loaded origin, use page.evaluate(). If the application reads that value during startup, install an origin-guarded context.add_init_script() before navigation, or create the context with the required storage_state.

That timing matters when scraping. In our catalog fixture, setting the region to eu after the table had loaded changed the stored value but left four US-priced rows on screen. A reload produced the intended EUR records. A successful storage write alone did not validate the data.

Use page.evaluate() to read or change localStorage in an already loaded Puppeteer page. To supply a value before the application's scripts run, install an origin-guarded page.evaluateOnNewDocument() callback before navigation. Pass values as arguments; a function executing in the page cannot implicitly use your Node.js variables or helpers.

Then check the data you meant to extract. Our catalog returned four rows with a captured HTTP 200 status in every extraction observation, yet 15 of the 36 observations contained the deliberately selected wrong dataset. A successful write to localStorage was not enough to update the page.

In the realm of web automation and testing, Puppeteer has emerged as a powerful tool for developers and QA engineers. One crucial aspect of web interactions is the management of cookies, which play a vital role in maintaining user sessions, personalizing experiences, and handling authentication. This comprehensive guide delves into the intricacies of setting cookies in Puppeteer using JavaScript, exploring various methods and best practices to enhance your web automation projects.

Cookies are small pieces of data stored by websites on a user's browser, serving as a memory for web applications. In Puppeteer, manipulating these cookies programmatically allows for sophisticated automation scenarios, from maintaining login states to testing complex user flows. As web applications become increasingly complex, the ability to effectively manage cookies in automated environments has become a critical skill for developers.

This article will explore the fundamental methods for setting cookies in Puppeteer, including the versatile page.setCookie() function and the context-wide context.addCookies() method. We'll also delve into advanced techniques for cookie persistence, handling secure and HttpOnly cookies, and managing cookie expiration and deletion. Additionally, we'll cover best practices and advanced techniques that will elevate your cookie management skills, ensuring your Puppeteer scripts are robust, secure, and efficient.

By mastering these techniques, developers can create more reliable and sophisticated web automation solutions, capable of handling complex authentication flows, maintaining long-running sessions, and accurately simulating user interactions across various web applications. Whether you're building automated testing suites, web scrapers, or complex browser-based tools, understanding the nuances of cookie management in Puppeteer is essential for success in modern web development landscapes.

As we explore these topics, we'll provide detailed code samples and explanations, ensuring that both beginners and experienced developers can enhance their Puppeteer skills and create more powerful, efficient, and secure web automation solutions.

Looking for Playwright? Check out our guide on How to Set Cookies in Playwright.

Puppeteer is a powerful Node.js library that provides a high-level API for controlling browsers through the DevTools Protocol. It is commonly used for testing, web scraping, and automating repetitive browser tasks. However, Puppeteer's default settings can trigger bot detection systems, especially in headless mode.

Which Playwright stealth library should you use? In this comparison, Patchright returned BrowserScan's Normal verdict when headed, while its headless user agent was flagged. Node's playwright-extra with the stealth plugin and the Firefox-based Camoufox returned Normal in both modes. Python playwright-stealth had no failed Sannysoft checks but still received BrowserScan's Navigator flag.

That difference is the useful result: changing a browser property, passing a diagnostic page and successfully retrieving your target are separate tests. Start with a reproducible control and keep the browser version, operating system and launch mode attached to every observation.

To set cookies with Playwright Python, call context.add_cookies() with a list of cookie dictionaries. Supply url, or a suitable domain and path, and set the cookies before navigating when the first request needs them. Use context.storage_state() when the workflow also depends on supported browser storage beyond cookies.

Restoring the cookie does not necessarily restore the dataset. In our catalog, cookies-only restoration kept member pricing but selected USD instead of EUR. A browser-associated API request made the same mistake when it omitted a region parameter that the page normally reads from local storage.

This guide shows the working state handoff and the failures around it. The fixture is self-authored and synthetic; it demonstrates specific mechanisms, not the probability that a real website will accept a restored login.