NEWScrapingAnt MCP for Claude Code, Cursor & Windsurf — try it free →
Skip to main content

5 posts tagged with "bash"

View All Tags

How to Ignore SSL Certificate Errors With Wget (and Trust One Instead)

· 15 min read
Oleg Kulyk
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to Ignore SSL Certificate Errors With Wget (and Trust One Instead)

Updated 2026-09-16

Rewritten as a tested reference. Every command below was run by the example package against three local HTTPS servers (a self-signed certificate, an expired one, and one issued by a local internal CA) and the output is copied from that run: wget 1.25.0 built with OpenSSL on macOS, plus the same core cases on Debian's GnuTLS build of wget 1.21.3 in a container. The 2024 version presented SSL_CERT_FILE as a wget feature; measured, it works on the OpenSSL build and is ignored on the GnuTLS build. New: the error messages and their causes on both builds, .wgetrc, --ca-directory, --pinnedpubkey, the internal-CA case, exit codes. Gone: a stray "Meta Description" section, unsourced breach statistics, and a wrapper script.

The flag you are looking for, run against the package's self-signed test server (substitute your URL):

wget -nv -O- --no-check-certificate https://localhost:8443/index.html
WARNING: cannot verify localhost's certificate, issued by 'CN=localhost':
  Self-signed certificate encountered.
<!doctype html><title>self-signed fixture</title><p>served over HTTPS</p>
<time> URL:https://localhost:8443/index.html [74/74] -> "-" [1]
exit=0

The download proceeds and the warning stays. The one-line way to make the error go away without switching verification off, when you have the server's certificate or your CA file:

wget -nv -O- --ca-certificate=certs/localhost.pem https://localhost:8443/index.html
<!doctype html><title>self-signed fixture</title><p>served over HTTPS</p>
<time> URL:https://localhost:8443/index.html [74/74] -> "-" [1]
exit=0

The rest of this page shows what each option does with the real output, how to read the different error messages, how to make a setting permanent, and what does and does not work depending on how your wget was built. Prefer curl? See How to ignore SSL certificate errors in cURL; the same problem in Python is in Python Requests: ignore SSL certificate errors.

How to Use cURL With a Proxy: -x, Auth, Env Variables, no_proxy

· 15 min read
Oleg Kulyk
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to Use cURL With a Proxy: -x, Auth, Env Variables, no_proxy

Updated 2026-09-15

Rewritten as a tested reference. Every command below was run against local proxies started by the example package (an HTTP proxy with Basic authentication, a SOCKS5 proxy with authentication, an open HTTP proxy) and the output is copied from that run. The previous version repeated its configuration section three times, used placeholder hosts throughout, and contained a mangled example address; all of that is gone.

The short answer:

curl -x http://proxy.example.com:8080 https://example.com/            # through an HTTP proxy
curl -x http://proxy.example.com:8080 --proxy-user user:pass https://example.com/   # with authentication
curl -x socks5h://user:pass@proxy.example.com:1080 https://example.com/            # SOCKS5, proxy resolves names

-x (long form --proxy) takes [scheme://][user:pass@]host[:port]; without a scheme curl assumes an HTTP proxy, and without a port it assumes 1080. The rest of this page shows what actually happens on the wire for each variant, how to set a proxy through the environment or .curlrc, how to switch it off for some hosts, and how the common failures look. Transcripts marked -v are trimmed to the proxy-related lines (a real -v prints a lot more), and non-local IP addresses are masked as <ip>. Tested with curl 8.7.1.

How to Ignore SSL Certificate With cURL

· 21 min read
Oleg Kulyk
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to Ignore SSL Certificate With cURL

In today's digital landscape, securing internet communications is paramount, and SSL/TLS certificates play a crucial role in this process. SSL (Secure Sockets Layer) and its successor TLS (Transport Layer Security) are cryptographic protocols designed to ensure data privacy, authentication, and trust between web servers and browsers. SSL/TLS certificates, issued by Certificate Authorities (CAs), authenticate a website's identity and enable encrypted connections. This authentication process is similar to issuing passports, wherein the CA verifies the entity's identity before issuing the certificate.

However, there are scenarios, especially during development and testing, where developers might need to bypass these SSL checks. This is where cURL, a command-line tool for transferring data using various protocols, comes into play. cURL provides options to handle SSL certificate validation, allowing developers to ignore SSL checks temporarily. While this practice can be invaluable in non-production environments, it also comes with significant security risks. Ignoring SSL certificate checks can expose systems to man-in-the-middle attacks, phishing, and data integrity compromises. Therefore, it's essential to understand both the methods and the implications of bypassing SSL checks with cURL.

Wget vs cURL for Downloading Files in Linux

· 13 min read
Oleg Kulyk
Oleg Kulyk
Co-Founder @ ScrapingAnt

Wget vs cURL for Downloading Files in Linux

In the realm of Linux-based environments, downloading files from the internet is a common task that can be accomplished using a variety of tools. Among these tools, Wget and cURL stand out as the most popular and widely used. Both tools offer robust capabilities for downloading files, but they cater to slightly different use cases and have unique strengths and weaknesses. Understanding these differences is crucial for selecting the right tool for specific tasks, whether you are downloading a single file, mirroring a website, or interacting with complex APIs.

Wget, short for 'World Wide Web get', is designed primarily for downloading files and mirroring websites. Its straightforward syntax and default behaviors make it user-friendly for quick, one-off downloads. For example, the command wget [URL] will download the file from the specified URL and save it to the current directory. Wget excels in tasks like recursive downloads and website mirroring, making it a preferred choice for archiving websites or downloading entire directories of files.

cURL, short for 'Client URL', is a versatile tool that supports a wide array of protocols beyond HTTP and HTTPS. It can be used for various network operations, including FTP, SCP, SFTP, and more. cURL requires additional options for saving files, such as curl -O [URL], but offers extensive customization options for HTTP headers, methods, and data. This makes cURL particularly useful for API interactions and complex web requests.

This comprehensive guide aims to provide a detailed comparison of Wget and cURL, covering their basic file download capabilities, protocol support, recursive download features, resume mechanisms, and advanced HTTP request handling. By the end of this guide, you will have a clear understanding of which tool is best suited for your specific needs.

How to download images with cURL?

· 13 min read
Oleg Kulyk
Oleg Kulyk
Co-Founder @ ScrapingAnt

How to download images with cURL?

The ability to efficiently download images from the internet is not just a convenience but a necessity for developers, system administrators, and many other professionals. cURL, a robust command-line tool, provides a versatile and powerful solution for this task. Whether you are looking to perform basic image downloads, handle complex redirects, or manage multiple simultaneous transfers, cURL has the capabilities to meet your needs. This comprehensive guide delves into both fundamental and advanced image downloading techniques using cURL, offering insights into handling redirects, managing authentication, optimizing large image transfers, and ensuring secure file storage. By mastering these techniques, users can significantly enhance their image retrieval processes, making them faster, more secure, and more efficient. The following sections will provide detailed explanations, code samples, and best practices drawn from authoritative sources to help you leverage cURL to its fullest potential.

This article is a part of the series on image downloading with different programming languages. Check out the other articles in the series: