Web Scraping Blog — Page 14

In Part 3, we focused on analyzing and cleaning the extracted data to address potential issues like missing values, inconsistencies, and outliers. To make it easier for future decision-making, we saved the cleaned data in various formats, such as CSV, databases, and S3 buckets.

In Part 4, we'll delve into strategies for bypassing common web scraping hurdles. We'll explore techniques such as using proxies, rotating user agents, and leveraging web scraping APIs to keep your scraping tasks running smoothly.

Without further ado, let’s get started!

Web scraping has become an essential technique for extracting data from websites, especially in an era where data-driven decision-making is paramount. Among the myriad of tools available for web scraping, Selenium stands out due to its ability to interact with web pages like a real user.

However, when it comes to accessing and manipulating network traffic, Selenium's capabilities are limited. This is where Selenium Wire comes into play, offering a powerful extension to the standard Selenium library.

This blog delves into various aspects of Selenium Wire, covering its installation, configuration, and features. It includes details on capturing and modifying HTTP requests, proxy configuration, and advanced request blocking techniques to enhance performance. Additionally, it delves into advanced techniques for request blocking, optimization of performance, and troubleshooting common issues.

In Part 2, we talked about creating a web scraper with Playwright to extract data from the Nike website, which has dynamically loaded content.

In Part 3, we will focus on carefully analyzing the extracted data and ensuring it's properly cleaned to deal with potential issues like missing values, inconsistencies, and outliers. The cleaned data will then be stored in different formats such as CSV, databases, and S3 buckets to make it easier for future decision-making.

Without further ado, let’s get started!

In Part 1, you learned about the basics of Playwright, environment setup, browser launching, and taking screenshots.

In Part 2, you’ll learn how to build a scraper from scratch. We'll cover how to locate and extract data, manage dynamically loaded content, utilize Playwright's network event feature, and improve the scraper's performance by blocking unnecessary resources.

Without further ado, let’s get started!

Introducing the 4-Part Series on Web Scraping with Playwright! This comprehensive series will delve into web scraping using Playwright, a powerful and versatile tool for automating browser interactions.

By the end of this series, you'll have a solid understanding of web scraping with Playwright. You'll be able to build robust scrapers that can handle dynamic content, efficiently store data, and navigate through anti-scraping mechanisms.

In Part 1, you'll learn about the basics of Playwright, why it's useful, how to set up the environment, how to launch the browser using Playwright, and how to take screenshots.

requests and httpx make the same request with almost the same line of code. The differences that matter are elsewhere: whether a connection is reused, how you run 200 requests at once, which protocol you speak, what happens by default on a redirect or a slow server, and what "retry" means. This page measures each of those on one server, then lists what breaks when you move a requests codebase to httpx 0.28.

The short version, from the tables below (Apple Silicon laptop, loopback, Python 3.12.11; the ratios are the claim, not the seconds):

Pick requests when the code is synchronous, the targets speak HTTP/1.1, and you want the larger ecosystem (plugins, mocks, examples). Pick httpx when you are inside asyncio (or an async framework), when you want HTTP/2, or when you want timeouts on by default. On a single reused connection the two are within a third of each other, and the one case where requests came out ahead was the loopback streaming test; nothing here says one is "faster".

pip install beautifulsoup4 lxml   # html5lib, requests and pandas only for the sections that use them

from bs4 import BeautifulSoup
with open("fixtures/page.html", encoding="utf-8") as f:
    PAGE = f.read()

soup = BeautifulSoup(PAGE, "html.parser")                 # from a str
print(soup.title.string, "|", soup.h1.get_text(" ", strip=True))

Ant Supply — catalogue | Products (3)

That is the whole idea: parse, find, read. The sheet below is the rest of it, every line run against the same fixture page (a small catalogue with a nav, three product cards, a table, a comment and a script).

Python has emerged as a dominant language due to its simplicity and versatility. One crucial aspect of web development and scraping is making HTTP requests, and Python offers a rich ecosystem of libraries tailored for this purpose.

This report delves into the best Python HTTP clients, exploring their unique features and use cases. From the ubiquitous Requests library, known for its simplicity and ease of use, to the modern and asynchronous HTTPX, which supports the latest protocols like HTTP/2 and WebSockets, there is a tool for every need. Additionally, libraries like aiohttp offer versatile async capabilities, making them ideal for real-time data scraping tasks.

For those requiring low-level control, urllib3 stands out with its robust and flexible features. On the other hand, Uplink provides a declarative approach to API interactions, while GRequests combines the simplicity of Requests with the power of Gevent's asynchronous capabilities. This report also highlights best practices for making HTTP requests and provides a comprehensive guide to efficient web scraping using HTTPX and ScrapingAnt. By understanding the strengths and weaknesses of each library, developers can make informed decisions and choose the best tool for their web scraping and development tasks.

You called requests.get() and got this:

requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1010)')))

The two-line way to make it go away:

import requests, urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
response = requests.get("https://localhost:8443/", verify=False)

And the one-line way to fix it instead of hiding it, when you have the server's certificate or your company's CA file:

response = requests.get("https://localhost:8443/", verify="certs/localhost.pem")

The rest of this page shows what each option does, with the real output, including the recipes you will find elsewhere that do nothing or crash in current requests.

The flag you are looking for, run against the package's self-signed test server (substitute your URL):

wget -nv -O- --no-check-certificate https://localhost:8443/index.html

WARNING: cannot verify localhost's certificate, issued by 'CN=localhost':
  Self-signed certificate encountered.
<!doctype html><title>self-signed fixture</title><p>served over HTTPS</p>
<time> URL:https://localhost:8443/index.html [74/74] -> "-" [1]
exit=0

The download proceeds and the warning stays. The one-line way to make the error go away without switching verification off, when you have the server's certificate or your CA file:

wget -nv -O- --ca-certificate=certs/localhost.pem https://localhost:8443/index.html

<!doctype html><title>self-signed fixture</title><p>served over HTTPS</p>
<time> URL:https://localhost:8443/index.html [74/74] -> "-" [1]
exit=0

The rest of this page shows what each option does with the real output, how to read the different error messages, how to make a setting permanent, and what does and does not work depending on how your wget was built. Prefer curl? See How to ignore SSL certificate errors in cURL; the same problem in Python is in Python Requests: ignore SSL certificate errors.