259 posts tagged with "web scraping"

In Part 1, you learned about the basics of Playwright, environment setup, browser launching, and taking screenshots.

In Part 2, you’ll learn how to build a scraper from scratch. We'll cover how to locate and extract data, manage dynamically loaded content, utilize Playwright's network event feature, and improve the scraper's performance by blocking unnecessary resources.

Without further ado, let’s get started!

Introducing the 4-Part Series on Web Scraping with Playwright! This comprehensive series will delve into web scraping using Playwright, a powerful and versatile tool for automating browser interactions.

By the end of this series, you'll have a solid understanding of web scraping with Playwright. You'll be able to build robust scrapers that can handle dynamic content, efficiently store data, and navigate through anti-scraping mechanisms.

In Part 1, you'll learn about the basics of Playwright, why it's useful, how to set up the environment, how to launch the browser using Playwright, and how to take screenshots.

requests and httpx make the same request with almost the same line of code. The differences that matter are elsewhere: whether a connection is reused, how you run 200 requests at once, which protocol you speak, what happens by default on a redirect or a slow server, and what "retry" means. This page measures each of those on one server, then lists what breaks when you move a requests codebase to httpx 0.28.

The short version, from the tables below (Apple Silicon laptop, loopback, Python 3.12.11; the ratios are the claim, not the seconds):

Pick requests when the code is synchronous, the targets speak HTTP/1.1, and you want the larger ecosystem (plugins, mocks, examples). Pick httpx when you are inside asyncio (or an async framework), when you want HTTP/2, or when you want timeouts on by default. On a single reused connection the two are within a third of each other, and the one case where requests came out ahead was the loopback streaming test; nothing here says one is "faster".

pip install beautifulsoup4 lxml   # html5lib, requests and pandas only for the sections that use them

from bs4 import BeautifulSoup
with open("fixtures/page.html", encoding="utf-8") as f:
    PAGE = f.read()

soup = BeautifulSoup(PAGE, "html.parser")                 # from a str
print(soup.title.string, "|", soup.h1.get_text(" ", strip=True))

Ant Supply — catalogue | Products (3)

That is the whole idea: parse, find, read. The sheet below is the rest of it, every line run against the same fixture page (a small catalogue with a nav, three product cards, a table, a comment and a script).

Python has emerged as a dominant language due to its simplicity and versatility. One crucial aspect of web development and scraping is making HTTP requests, and Python offers a rich ecosystem of libraries tailored for this purpose.

This report delves into the best Python HTTP clients, exploring their unique features and use cases. From the ubiquitous Requests library, known for its simplicity and ease of use, to the modern and asynchronous HTTPX, which supports the latest protocols like HTTP/2 and WebSockets, there is a tool for every need. Additionally, libraries like aiohttp offer versatile async capabilities, making them ideal for real-time data scraping tasks.

For those requiring low-level control, urllib3 stands out with its robust and flexible features. On the other hand, Uplink provides a declarative approach to API interactions, while GRequests combines the simplicity of Requests with the power of Gevent's asynchronous capabilities. This report also highlights best practices for making HTTP requests and provides a comprehensive guide to efficient web scraping using HTTPX and ScrapingAnt. By understanding the strengths and weaknesses of each library, developers can make informed decisions and choose the best tool for their web scraping and development tasks.

You called requests.get() and got this:

requests.exceptions.SSLError: HTTPSConnectionPool(host='localhost', port=8443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1010)')))

The two-line way to make it go away:

import requests, urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
response = requests.get("https://localhost:8443/", verify=False)

And the one-line way to fix it instead of hiding it, when you have the server's certificate or your company's CA file:

response = requests.get("https://localhost:8443/", verify="certs/localhost.pem")

The rest of this page shows what each option does, with the real output, including the recipes you will find elsewhere that do nothing or crash in current requests.

The flag you are looking for, run against the package's self-signed test server (substitute your URL):

wget -nv -O- --no-check-certificate https://localhost:8443/index.html

WARNING: cannot verify localhost's certificate, issued by 'CN=localhost':
  Self-signed certificate encountered.
<!doctype html><title>self-signed fixture</title><p>served over HTTPS</p>
<time> URL:https://localhost:8443/index.html [74/74] -> "-" [1]
exit=0

The download proceeds and the warning stays. The one-line way to make the error go away without switching verification off, when you have the server's certificate or your CA file:

wget -nv -O- --ca-certificate=certs/localhost.pem https://localhost:8443/index.html

<!doctype html><title>self-signed fixture</title><p>served over HTTPS</p>
<time> URL:https://localhost:8443/index.html [74/74] -> "-" [1]
exit=0

The rest of this page shows what each option does with the real output, how to read the different error messages, how to make a setting permanent, and what does and does not work depending on how your wget was built. Prefer curl? See How to ignore SSL certificate errors in cURL; the same problem in Python is in Python Requests: ignore SSL certificate errors.

Web scraping has become an indispensable tool for acquiring data from websites, especially in the era of big data and data-driven decision-making. However, the complexity of scraping has increased with the advent of dynamic websites, which generate content on-the-fly using JavaScript and AJAX. Unlike static websites, which serve pre-built HTML pages, dynamic websites respond to user interactions and real-time data updates, making traditional scraping techniques ineffective.

To navigate this landscape, developers need to understand the intricacies of client-side and server-side rendering, the role of JavaScript frameworks such as React, Angular, and Vue.js, and the importance of AJAX for asynchronous data loading. This knowledge is crucial for choosing the right tools and techniques to effectively scrape dynamic websites. In this report, we delve into the methodologies for scraping dynamic websites using C++, exploring essential libraries like libcurl, Gumbo, and Boost, and providing a detailed, step-by-step guide to building robust web scrapers.

Dynamic websites have become increasingly prevalent due to their ability to deliver personalized and interactive content to users. Unlike static websites, which serve pre-built HTML pages, dynamic websites generate content on-the-fly based on user interactions, database queries, or real-time data. This dynamic nature is achieved through the use of server-side programming languages such as PHP, Ruby, and Python, as well as client-side JavaScript frameworks like React, Angular, and Vue.js.

Dynamic websites are characterized by asynchronous content loading, client-side rendering, real-time updates, personalized content, and complex DOM structures. These features enhance user experience but also introduce significant challenges for web scraping. Traditional scraping tools that rely on static HTML parsing often fall short when dealing with dynamic websites, necessitating the use of more sophisticated methods and tools.

To effectively scrape dynamic websites using C#, developers must employ specialized tools such as Selenium WebDriver and PuppeteerSharp, which can interact with web pages as if they were real users, executing JavaScript and waiting for content to load. These tools, along with proper wait mechanisms and dynamic element location strategies, enable the extraction of data from even the most complex and interactive web applications.

Web scraping has become an essential technique for data extraction, particularly with the rise of dynamic websites that deliver content through AJAX and JavaScript. Traditional methods of web scraping often fall short when dealing with these modern web architectures, necessitating more advanced approaches. Using the Go programming language for web scraping offers several advantages, including high performance, robust concurrency support, and a growing ecosystem of libraries specifically designed for this task.

Go, often referred to as Golang, is a statically typed, compiled language that excels in performance and efficiency. Its compilation to machine code results in faster execution times compared to interpreted languages like Python. This is particularly beneficial for large-scale web scraping projects where speed and resource utilization are critical. Additionally, Go's built-in support for concurrency through goroutines enables developers to scrape multiple web pages concurrently, making it highly scalable.

This report delves into the techniques and best practices for scraping dynamic websites using Go. It covers essential topics such as identifying and mimicking AJAX requests, utilizing headless browsers, and handling infinite scrolling. Furthermore, it provides insights into managing browser dependencies, optimizing performance, and adhering to ethical scraping practices. By the end of this report, you will have a comprehensive understanding of how to effectively scrape dynamic websites using Go, leveraging its unique features to build efficient and scalable web scraping solutions.