NEWScrapingAnt MCP for Claude Code, Cursor & Windsurf — try it free →
Skip to main content

How to Parse XML in Python: ElementTree, Namespaces, Large Files, lxml

· 20 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to Parse XML in Python: ElementTree, Namespaces, Large Files, lxml

Updated 2026-09-16

Rewritten as a tested reference. Every example was executed by the example package against small fixture files and one generated 23.8 MB file; the output blocks are copied from that run. The 2024 version's performance numbers were not measured and are replaced by a measured table; its xml.etree.cElementTree recommendation (a deprecated alias since Python 3.3; xml.etree.ElementTree already uses the C accelerator), its codecs.open encoding advice (wrong; the equivalent open(..., encoding="utf-8") is shown failing below) and a stray "Meta Description" section are gone; namespaces, large-file streaming, writing, xmltodict, pandas.read_xml and the current security picture are new.

The standard library does this without installing anything. Given a file books.xml with a <catalog> of <book> elements:

import xml.etree.ElementTree as ET

root = ET.parse("fixtures/books.xml").getroot()
for book in root.findall("book"):
print(book.get("id"), book.find("title").text, book.findtext("price", default="n/a"))
bk101 XML Developer's Guide 44.95
bk102 Midnight Rain 5.95
bk103 Maeve Ascendant n/a

That covers most "read an XML file" tasks. The rest of this page is the things that go wrong next, each with the real output: elements you cannot find because of a namespace, a file too big for parse(), encoding and syntax errors, and when to reach for lxml, xmltodict or pandas instead.

How to Use cURL With a Proxy: -x, Auth, Env Variables, no_proxy

· 15 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to Use cURL With a Proxy: -x, Auth, Env Variables, no_proxy

Updated 2026-09-15

Rewritten as a tested reference. Every command below was run against local proxies started by the example package (an HTTP proxy with Basic authentication, a SOCKS5 proxy with authentication, an open HTTP proxy) and the output is copied from that run. The previous version repeated its configuration section three times, used placeholder hosts throughout, and contained a mangled example address; all of that is gone.

The short answer:

curl -x http://proxy.example.com:8080 https://example.com/ # through an HTTP proxy
curl -x http://proxy.example.com:8080 --proxy-user user:pass https://example.com/ # with authentication
curl -x socks5h://user:pass@proxy.example.com:1080 https://example.com/ # SOCKS5, proxy resolves names

-x (long form --proxy) takes [scheme://][user:pass@]host[:port]; without a scheme curl assumes an HTTP proxy, and without a port it assumes 1080. The rest of this page shows what actually happens on the wire for each variant, how to set a proxy through the environment or .curlrc, how to switch it off for some hosts, and how the common failures look. Transcripts marked -v are trimmed to the proxy-related lines (a real -v prints a lot more), and non-local IP addresses are masked as <ip>. Tested with curl 8.7.1.

How to Ignore SSL Certificate With cURL

· 21 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to Ignore SSL Certificate With cURL

In today's digital landscape, securing internet communications is paramount, and SSL/TLS certificates play a crucial role in this process. SSL (Secure Sockets Layer) and its successor TLS (Transport Layer Security) are cryptographic protocols designed to ensure data privacy, authentication, and trust between web servers and browsers. SSL/TLS certificates, issued by Certificate Authorities (CAs), authenticate a website's identity and enable encrypted connections. This authentication process is similar to issuing passports, wherein the CA verifies the entity's identity before issuing the certificate.

However, there are scenarios, especially during development and testing, where developers might need to bypass these SSL checks. This is where cURL, a command-line tool for transferring data using various protocols, comes into play. cURL provides options to handle SSL certificate validation, allowing developers to ignore SSL checks temporarily. While this practice can be invaluable in non-production environments, it also comes with significant security risks. Ignoring SSL certificate checks can expose systems to man-in-the-middle attacks, phishing, and data integrity compromises. Therefore, it's essential to understand both the methods and the implications of bypassing SSL checks with cURL.

How to download a file with Selenium in Python

· 14 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to download a file with Selenium in Python

Selenium has emerged as a powerful tool for automating browser interactions using Python. One common task that developers often need to automate is the downloading of files from the web. Ensuring seamless and automated file downloads across different browsers and operating systems can be challenging. This comprehensive guide aims to address these challenges by providing detailed instructions on how to configure Selenium for file downloads in various browsers, including Google Chrome, Mozilla Firefox, Microsoft Edge, and Safari. Furthermore, it explores best practices and alternative methods to enhance the robustness and efficiency of the file download process. By following the guidelines and code samples provided here, developers can create reliable and cross-platform compatible automation scripts that handle file downloads effortlessly.

This guide is a part of the series on web scraping and file downloading with different web drivers and programming languages. Check out the other articles in the series:

Wget vs cURL for Downloading Files in Linux

· 13 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

Wget vs cURL for Downloading Files in Linux

In the realm of Linux-based environments, downloading files from the internet is a common task that can be accomplished using a variety of tools. Among these tools, Wget and cURL stand out as the most popular and widely used. Both tools offer robust capabilities for downloading files, but they cater to slightly different use cases and have unique strengths and weaknesses. Understanding these differences is crucial for selecting the right tool for specific tasks, whether you are downloading a single file, mirroring a website, or interacting with complex APIs.

Wget, short for 'World Wide Web get', is designed primarily for downloading files and mirroring websites. Its straightforward syntax and default behaviors make it user-friendly for quick, one-off downloads. For example, the command wget [URL] will download the file from the specified URL and save it to the current directory. Wget excels in tasks like recursive downloads and website mirroring, making it a preferred choice for archiving websites or downloading entire directories of files.

cURL, short for 'Client URL', is a versatile tool that supports a wide array of protocols beyond HTTP and HTTPS. It can be used for various network operations, including FTP, SCP, SFTP, and more. cURL requires additional options for saving files, such as curl -O [URL], but offers extensive customization options for HTTP headers, methods, and data. This makes cURL particularly useful for API interactions and complex web requests.

This comprehensive guide aims to provide a detailed comparison of Wget and cURL, covering their basic file download capabilities, protocol support, recursive download features, resume mechanisms, and advanced HTTP request handling. By the end of this guide, you will have a clear understanding of which tool is best suited for your specific needs.

How to Find Elements With Selenium in Python

· 17 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to Find Elements With Selenium in Python

Understanding how to find elements with Selenium in Python is essential for anyone engaged in web automation and testing. Selenium, a powerful open-source tool, allows developers and testers to simulate user interactions with web applications, automating the testing process and ensuring that web applications function as expected (Selenium). One of the most crucial aspects of using Selenium effectively is mastering the various locator strategies available in Selenium Python. These strategies are pivotal for identifying and interacting with web elements, which are integral to executing automated test scripts successfully.

There are multiple strategies available for locating elements in Selenium Python, each with its own strengths and specific use cases. Commonly used methods include locating elements by ID, name, XPath, CSS Selector, class name, tag name, and link text. Each method has its own set of advantages and potential pitfalls. For instance, locating elements by ID is highly reliable due to the uniqueness of ID attributes on a webpage, whereas using XPath can be more flexible but potentially less efficient and more brittle.

To ensure reliability and maintainability of Selenium test scripts, it is important to prioritize unique and stable locators, avoid brittle locators, implement robust waiting strategies, and utilize design patterns such as the Page Object Model (POM). Additionally, understanding and addressing common challenges like handling dynamic content, dealing with stale elements, and navigating iframes and Shadow DOMs can significantly enhance the effectiveness of Selenium-based tests (Selenium documentation).

This guide delves into the detailed locator strategies, best practices, and common challenges associated with finding elements using Selenium Python. With code samples and thorough explanations, it aims to provide a comprehensive understanding of this critical aspect of web automation.

How to submit a form with Puppeteer?

· 13 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to submit a form with Puppeteer?

Puppeteer, a Node.js library developed by Google, offers a high-level API to control headless Chrome or Chromium browsers, making it an indispensable tool for web scraping, automated testing, and form submission automation. In today's digital landscape, automating form submissions is crucial for a variety of applications, ranging from data collection to user interaction testing. Puppeteer provides a robust solution for these tasks, allowing developers to programmatically interact with web pages as if they were using a regular browser. This guide delves into the setup and advanced techniques for using Puppeteer to automate form submissions, ensuring reliable and efficient automation processes. By following the outlined steps, users can install and configure Puppeteer, create basic scripts, handle dynamic form elements, manage complex inputs, and integrate with testing frameworks like Jest. Additionally, this guide explores effective strategies for bypassing CAPTCHAs and anti-bot measures, which are common obstacles in web automation.

Looking for a Playwright guide? Check out: How to submit a form with Playwright?

How to download images with cURL?

· 13 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to download images with cURL?

The ability to efficiently download images from the internet is not just a convenience but a necessity for developers, system administrators, and many other professionals. cURL, a robust command-line tool, provides a versatile and powerful solution for this task. Whether you are looking to perform basic image downloads, handle complex redirects, or manage multiple simultaneous transfers, cURL has the capabilities to meet your needs. This comprehensive guide delves into both fundamental and advanced image downloading techniques using cURL, offering insights into handling redirects, managing authentication, optimizing large image transfers, and ensuring secure file storage. By mastering these techniques, users can significantly enhance their image retrieval processes, making them faster, more secure, and more efficient. The following sections will provide detailed explanations, code samples, and best practices drawn from authoritative sources to help you leverage cURL to its fullest potential.

This article is a part of the series on image downloading with different programming languages. Check out the other articles in the series:

How to download images with Java?

· 16 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to download images with Java?

In the current digital age, the ability to download and process images efficiently is an essential skill for Java developers. Whether it's for a simple application or a complex system, understanding the various methods available for image downloading can significantly enhance performance and functionality. This comprehensive guide explores five key methods for downloading images in Java, utilizing built-in libraries, third-party libraries, and advanced techniques (Oracle Java Documentation). Each method is detailed with step-by-step explanations and code samples, making it suitable for both beginners and experienced developers. Additionally, we delve into performance optimization, reliability, memory management, security considerations, and the best libraries for efficient image downloading. By understanding these concepts, developers can create robust and efficient image downloading solutions tailored to their specific needs.

This article is a part of the series on image downloading with different programming languages. Check out the other articles in the series:

How to download images with C#?

· 19 min read
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to download images with C#?

Downloading images programmatically in C# is a fundamental task in various applications, ranging from web scraping to automated testing. This comprehensive guide delves into different methods to achieve this, including the use of HttpClient, WebClient, and ImageSharp. Each method is explored with detailed code examples and best practices to ensure efficient and reliable image downloading.

The HttpClient class is a modern, feature-rich way to handle HTTP requests and responses, making it a popular choice for downloading images. Its flexibility and performance advantages are well-documented (Microsoft Docs). On the other hand, WebClient, although considered legacy, still finds use in older codebases due to its simplicity (Stack Overflow). For advanced image processing, the ImageSharp library offers robust capabilities beyond simple downloading, making it ideal for applications requiring image manipulation (Code Maze).

This guide also covers critical aspects such as asynchronous downloads, error handling, and memory management, ensuring that developers can create robust systems for downloading images in C#. By following these best practices, you can optimize performance and reliability, addressing common challenges encountered in real-world applications.

This article is a part of the series on image downloading with different programming languages. Check out the other articles in the series: