252 posts tagged with "data extraction"

pandas.read_html() finds every <table> in a page and returns a list of DataFrames. It takes a file path, a URL, or a file-like object (StringIO around HTML you already have; a raw string no longer works on pandas 3). The three-line version, on a file:

import pandas as pd
PAGE = "fixtures/tables.html"
tables = pd.read_html(PAGE)
print("file:", len(tables), "tables; shapes:", [df.shape for df in tables])
print(tables[0])

file: 12 tables; shapes: [(3, 3), (3, 4), (2, 5), (3, 3), (2, 3), (2, 3), (2, 2), (2, 2), (5, 2), (2, 2), (3, 2), (3, 3)]
    Name  Age       City
0    Ada   36     London
1  Grace   45  Arlington
2  Linus   28   Helsinki

The same call on HTML you already hold, and on a URL:

pd.read_html(StringIO(html)): 12
pd.read_html(open(PAGE, 'rb')): 12
pd.read_html(URL): 12

The rest of this page is what happens after that line: how to get the one table you want, why headers, numbers and links come out the way they do, which parser is running, why a Wikipedia URL returns 403, and which idioms from older tutorials (including the 2024 version of this page) raise on pandas 3.

To parse XML in C++, use a library to read the document, check the parse result, and then validate the fields your application requires. A successful parse alone does not tell you whether a record has a title, whether an ID is an integer, or whether the document is safe to process.

This guide starts with a complete pugixml file parser that extracts book IDs and titles. A smaller TinyXML-2 example shows how to parse a string. The library comparison explains when to move to Xerces-C++, libxml2, or libxml++ for namespaces, validation, or streaming. No scraping API is needed to parse XML you already have.

Web scraping has become an essential tool for data extraction from websites, enabling developers to gather information for various applications such as market research, competitive analysis, and content aggregation. Haskell, a statically-typed, functional programming language, offers a robust ecosystem for web scraping through its strong type system, concurrency capabilities, and extensive libraries. This guide aims to provide a comprehensive overview of web scraping with Haskell, covering everything from setting up the development environment to leveraging advanced techniques for efficient and reliable scraping.

HTML parsing is a fundamental process in web development and data extraction. It involves breaking down HTML documents into their constituent elements, allowing for easy manipulation and analysis of the structure and content. In the context of C++, HTML parsing can be particularly advantageous due to the language's high performance and low-level control. However, the process also presents challenges, such as handling nested elements, malformed HTML, and varying HTML versions.

This comprehensive guide aims to provide an in-depth exploration of HTML parsing in C++. It covers essential concepts such as tokenization, tree construction, and DOM (Document Object Model) representation, along with practical code examples. We will delve into various parsing techniques, discuss performance considerations, and highlight best practices for robust error handling. Furthermore, we will review some of the most popular HTML parsing libraries available for C++, including Gumbo Parser, libxml++, Boost.Beast, MyHTML, and TinyXML-2, to help developers choose the best tool for their specific needs.

The standard library does this without installing anything. Given a file books.xml with a <catalog> of <book> elements:

import xml.etree.ElementTree as ET

root = ET.parse("fixtures/books.xml").getroot()
for book in root.findall("book"):
    print(book.get("id"), book.find("title").text, book.findtext("price", default="n/a"))

bk101 XML Developer's Guide 44.95
bk102 Midnight Rain 5.95
bk103 Maeve Ascendant n/a

That covers most "read an XML file" tasks. The rest of this page is the things that go wrong next, each with the real output: elements you cannot find because of a namespace, a file too big for parse(), encoding and syntax errors, and when to reach for lxml, xmltodict or pandas instead.

In today's digital landscape, securing internet communications is paramount, and SSL/TLS certificates play a crucial role in this process. SSL (Secure Sockets Layer) and its successor TLS (Transport Layer Security) are cryptographic protocols designed to ensure data privacy, authentication, and trust between web servers and browsers. SSL/TLS certificates, issued by Certificate Authorities (CAs), authenticate a website's identity and enable encrypted connections. This authentication process is similar to issuing passports, wherein the CA verifies the entity's identity before issuing the certificate.

However, there are scenarios, especially during development and testing, where developers might need to bypass these SSL checks. This is where cURL, a command-line tool for transferring data using various protocols, comes into play. cURL provides options to handle SSL certificate validation, allowing developers to ignore SSL checks temporarily. While this practice can be invaluable in non-production environments, it also comes with significant security risks. Ignoring SSL certificate checks can expose systems to man-in-the-middle attacks, phishing, and data integrity compromises. Therefore, it's essential to understand both the methods and the implications of bypassing SSL checks with cURL.

To download a file with Selenium in Python, configure the browser's download directory before starting the session, click the link, and wait for the expected file to pass a content check before closing the browser. Waiting for the link to be clickable and waiting for the file to finish are separate steps.

Below are tested Chrome and Firefox configurations, including headless runs. The examples download a CSV, a delayed binary file and a PDF from a local fixture server. They verify the expected length and SHA-256 digest rather than trusting that a filename appeared.

In the realm of Linux-based environments, downloading files from the internet is a common task that can be accomplished using a variety of tools. Among these tools, Wget and cURL stand out as the most popular and widely used. Both tools offer robust capabilities for downloading files, but they cater to slightly different use cases and have unique strengths and weaknesses. Understanding these differences is crucial for selecting the right tool for specific tasks, whether you are downloading a single file, mirroring a website, or interacting with complex APIs.

Wget, short for 'World Wide Web get', is designed primarily for downloading files and mirroring websites. Its straightforward syntax and default behaviors make it user-friendly for quick, one-off downloads. For example, the command wget [URL] will download the file from the specified URL and save it to the current directory. Wget excels in tasks like recursive downloads and website mirroring, making it a preferred choice for archiving websites or downloading entire directories of files.

cURL, short for 'Client URL', is a versatile tool that supports a wide array of protocols beyond HTTP and HTTPS. It can be used for various network operations, including FTP, SCP, SFTP, and more. cURL requires additional options for saving files, such as curl -O [URL], but offers extensive customization options for HTTP headers, methods, and data. This makes cURL particularly useful for API interactions and complex web requests.

This comprehensive guide aims to provide a detailed comparison of Wget and cURL, covering their basic file download capabilities, protocol support, recursive download features, resume mechanisms, and advanced HTTP request handling. By the end of this guide, you will have a clear understanding of which tool is best suited for your specific needs.

Use driver.find_element(By.CSS_SELECTOR, selector) for the first matching element and driver.find_elements(By.CSS_SELECTOR, selector) for a list of all matches. If nothing matches, the singular method raises NoSuchElementException; the plural method returns an empty list.

For scraping, finding elements is only the first step. You also need the right container, populated fields and a way to recover when the page replaces nodes. The examples below extract a complete catalog and deliberately test selectors that return plausible but wrong results.

Puppeteer, a Node.js library developed by Google, offers a high-level API to control headless Chrome or Chromium browsers, making it an indispensable tool for web scraping, automated testing, and form submission automation. In today's digital landscape, automating form submissions is crucial for a variety of applications, ranging from data collection to user interaction testing. Puppeteer provides a robust solution for these tasks, allowing developers to programmatically interact with web pages as if they were using a regular browser. This guide delves into the setup and advanced techniques for using Puppeteer to automate form submissions, ensuring reliable and efficient automation processes. By following the outlined steps, users can install and configure Puppeteer, create basic scripts, handle dynamic form elements, manage complex inputs, and integrate with testing frameworks like Jest. Additionally, this guide explores effective strategies for bypassing CAPTCHAs and anti-bot measures, which are common obstacles in web automation.

Looking for a Playwright guide? Check out: How to submit a form with Playwright?