In the rapidly evolving landscape of web automation and testing, two open-source frameworks have emerged as leading tools: Playwright and Selenium. Both frameworks offer unique features and capabilities, making the choice between them a nuanced decision that depends on specific project requirements and team expertise.
In the ever-evolving landscape of web scraping, Python remains the language of choice for developers due to its simplicity, readability, and a robust ecosystem of libraries. Python offers a diverse array of HTTP clients that cater to various web scraping needs, from simple data extraction to complex, high-concurrency tasks.
This guide delves into the top Python HTTP clients, exploring their features, pros, cons, and providing code examples to get started.
In Part 3, we focused on analyzing and cleaning the extracted data to address potential issues like missing values, inconsistencies, and outliers. To make it easier for future decision-making, we saved the cleaned data in various formats, such as CSV, databases, and S3 buckets.
In Part 4, we'll delve into strategies for bypassing common web scraping hurdles. We'll explore techniques such as using proxies, rotating user agents, and leveraging web scraping APIs to keep your scraping tasks running smoothly.
Without further ado, let’s get started!
Web scraping has become an essential technique for extracting data from websites, especially in an era where data-driven decision-making is paramount. Among the myriad of tools available for web scraping, Selenium stands out due to its ability to interact with web pages like a real user.
However, when it comes to accessing and manipulating network traffic, Selenium's capabilities are limited. This is where Selenium Wire comes into play, offering a powerful extension to the standard Selenium library.
This blog delves into various aspects of Selenium Wire, covering its installation, configuration, and features. It includes details on capturing and modifying HTTP requests, proxy configuration, and advanced request blocking techniques to enhance performance. Additionally, it delves into advanced techniques for request blocking, optimization of performance, and troubleshooting common issues.
In Part 2, we talked about creating a web scraper with Playwright to extract data from the Nike website, which has dynamically loaded content.
In Part 3, we will focus on carefully analyzing the extracted data and ensuring it's properly cleaned to deal with potential issues like missing values, inconsistencies, and outliers. The cleaned data will then be stored in different formats such as CSV, databases, and S3 buckets to make it easier for future decision-making.
Without further ado, let’s get started!
Fixed an async function declaration and request.resource_type property access, awaited routing operations, and removed blanket XHR blocking that could prevent collecting product data. An unchanged scroll height is a stopping heuristic, not proof of complete extraction. This correction does not claim a fresh live Nike test.
In Part 1, you learned about the basics of Playwright, environment setup, browser launching, and taking screenshots.
In Part 2, you’ll learn how to build a scraper from scratch. We'll cover how to locate and extract data, manage dynamically loaded content, utilize Playwright's network event feature, and improve the scraper's performance by blocking unnecessary resources.
Without further ado, let’s get started!
Added a section that fetches the same page through the ScrapingAnt API, run once against the live API on 2026-10-26. The rest of the article was not re-tested in this update.
Introducing the 4-Part Series on Web Scraping with Playwright! This comprehensive series will delve into web scraping using Playwright, a powerful and versatile tool for automating browser interactions.
By the end of this series, you'll have a solid understanding of web scraping with Playwright. You'll be able to build robust scrapers that can handle dynamic content, efficiently store data, and navigate through anti-scraping mechanisms.
In Part 1, you'll learn about the basics of Playwright, why it's useful, how to set up the environment, how to launch the browser using Playwright, and how to take screenshots.
Every timing on this page now comes from one local server run by the evidence packet on requests 2.34.2 and httpx 0.28.1. The earlier version's benchmark table, which quoted third-party posts and unsourced figures, is gone. New: connection counting, threads vs asyncio, HTTP/2 on the same server, what "retries" means in each library, the errors a migration hits, and what the httpx 1.0 pre-release does to 0.28 code.
Updated 2026-10-23: added a section that sends the same GET through the ScrapingAnt API with both libraries, run once against books.toscrape.com.
requests and httpx make the same request with almost the same line of code. The differences that matter are elsewhere: whether a connection is reused, how you run 200 requests at once, which protocol you speak, what happens by default on a redirect or a slow server, and what "retry" means. This page measures each of those on one server, then lists what breaks when you move a requests codebase to httpx 0.28.
The short version, from the tables below (Apple Silicon laptop, loopback, Python 3.12.11; the ratios are the claim, not the seconds):
| Case | requests 2.34.2 | httpx 0.28.1 |
|---|---|---|
300 sequential GETs, one Session / Client (plain HTTP) | 0.150 s, 1 connection | 0.126 s, 1 connection |
300 sequential GETs, no Session / Client (plain HTTP) | 0.217 s, 300 connections | 3.090 s, 300 connections (an SSL context is built per call) |
| 200 GETs with a 50 ms server delay, 20 threads | 0.543 s (32 connections; 20 with pool_maxsize=20) | 0.545 s, 20 connections |
Same 200 GETs, AsyncClient + asyncio.gather, Semaphore(20) | not available | 0.558 s, 20 connections |
| Same 200 GETs over TLS, no semaphore (httpx caps in-flight work at 100), HTTP/1.1 vs HTTP/2 | HTTP/1.1 only | 0.476 s, 200 connections vs 0.171 s, 1 connection |
| 50 MiB download, streamed, peak RSS | 35 MiB | 44 MiB |
| Retries against a 503 | urllib3.Retry(status_forcelist=[503]): 4 requests sent | HTTPTransport(retries=3): 1 request (retries cover connection failures only) |
Pick requests when the code is synchronous, the targets speak HTTP/1.1, and you want the larger ecosystem (plugins, mocks, examples). Pick httpx when you are inside asyncio (or an async framework), when you want HTTP/2, or when you want timeouts on by default. On a single reused connection the two are within a third of each other, and the one case where requests came out ahead was the loopback streaming test; nothing here says one is "faster".
Rewritten as a tested cheat sheet. Every snippet below was run by the example package against one fixture page on beautifulsoup4 4.15.0, and the output is copied from that run. The 2024 version had no captured output (only inline # Output: comments), used the text= argument that has warned since 4.11, passed from_encoding to a str (ignored), and called pd.read_html with a string that pandas 3 now treats as a filename; all corrected. New: parsers compared on broken markup, a CPU-time table with min/median/max columns and a note on how much such timings move under load, the 4.13–4.15 deprecations captured with a type-checked example, and the errors you will actually see.
Updated 2026-10-23: added a section that fetches the HTML through the ScrapingAnt API before parsing, run once against books.toscrape.com.
pip install beautifulsoup4 lxml # html5lib, requests and pandas only for the sections that use them
from bs4 import BeautifulSoup
with open("fixtures/page.html", encoding="utf-8") as f:
PAGE = f.read()
soup = BeautifulSoup(PAGE, "html.parser") # from a str
print(soup.title.string, "|", soup.h1.get_text(" ", strip=True))
Ant Supply — catalogue | Products (3)
That is the whole idea: parse, find, read. The sheet below is the rest of it, every line run against the same fixture page (a small catalogue with a nav, three product cards, a table, a comment and a script).
Python has emerged as a dominant language due to its simplicity and versatility. One crucial aspect of web development and scraping is making HTTP requests, and Python offers a rich ecosystem of libraries tailored for this purpose.
This report delves into the best Python HTTP clients, exploring their unique features and use cases. From the ubiquitous Requests library, known for its simplicity and ease of use, to the modern and asynchronous HTTPX, which supports the latest protocols like HTTP/2 and WebSockets, there is a tool for every need. Additionally, libraries like aiohttp offer versatile async capabilities, making them ideal for real-time data scraping tasks.
For those requiring low-level control, urllib3 stands out with its robust and flexible features. On the other hand, Uplink provides a declarative approach to API interactions, while GRequests combines the simplicity of Requests with the power of Gevent's asynchronous capabilities. This report also highlights best practices for making HTTP requests and provides a comprehensive guide to efficient web scraping using HTTPX and ScrapingAnt. By understanding the strengths and weaknesses of each library, developers can make informed decisions and choose the best tool for their web scraping and development tasks.