Wget supports various protocols such as HTTP, HTTPS, and FTP, making it an indispensable tool for developers, system administrators, and data analysts alike. Its simplicity, combined with extensive customization options, allows users to automate downloads, manage bandwidth, handle authentication, and even perform recursive website mirroring with ease.
Whether you're downloading a single file or scraping an entire website, understanding the fundamental syntax and advanced features of Wget can significantly streamline your workflow. For instance, Wget's ability to handle multiple URLs simultaneously or sequentially through brace expansions simplifies batch downloads, saving valuable time and effort. Additionally, its robust options for managing download behavior, such as setting timeouts and retries, ensure reliability even under unstable network conditions.
Fundamental Syntax and Command Structure
The basic syntax of the wget command follows a simple and consistent structure, making it straightforward for users to quickly grasp and apply. The general syntax is as follows:
wget [options] [URL]
[options]: Optional flags that modify the behavior of wget.[URL]: The web address of the file or resource to be downloaded.
For instance, downloading a file from a given URL would look like this:
wget https://example.com/file.zip
This command downloads the file named "file.zip" from the specified URL directly to the current working directory.
Handling Multiple URLs and URL Patterns
While previous sections have discussed single file downloads, this section specifically addresses downloading multiple files simultaneously or using URL patterns. wget allows users to specify multiple URLs directly in the command line or via brace expansions to download multiple files at once.
For example, to download multiple files explicitly listed in the command line:
wget https://example.com/file1.zip https://example.com/file2.zip https://example.com/file3.zip
Alternatively, wget supports brace expansions to simplify sequential downloads:
wget https://example.com/images/{1..10}.jpg
This command downloads images named from "1.jpg" through "10.jpg" from the given URL. This functionality is particularly useful when dealing with sequentially numbered files.
Customizing Download Output and File Naming
While earlier sections have covered basic file downloads, this subsection specifically explores how to customize the naming and output of downloaded files. By default, wget saves files under their original names, but users can explicitly specify the filename or redirect output to stdout.
To save a downloaded file under a different name, use the -O or --output-document option:
wget https://example.com/file.zip -O custom_name.zip
This command downloads the file from the specified URL and saves it locally as "custom_name.zip" instead of the original filename.
Alternatively, wget can redirect the downloaded content directly to stdout, which can be useful for piping content into other commands or scripts:
wget -q -O - https://example.com/data.json
The above command quietly (-q) fetches the content from the URL and outputs it directly to the terminal without saving it to a file.
Managing Download Behavior with Timeouts and Retries
This subsection specifically addresses managing wget's behavior in scenarios involving unstable network connections or slow servers, a topic not previously covered. wget provides options to control connection timeouts, retries, and related behaviors to ensure robust downloads.
To set a timeout for the entire download process, use the -T or --timeout option followed by the number of seconds:
wget --timeout=30 https://example.com/largefile.iso
This command sets a timeout of 30 seconds, meaning wget will abort the download if no data is received within this period.
Additionally, wget allows specifying the number of retries in case of download failures or interruptions using the -t or --tries option:
wget --tries=5 https://example.com/file.zip
In this example, wget will attempt to download the file up to 5 times before giving up. To handle connection refusals explicitly, the --retry-connrefused option can be added:
wget --tries=10 --retry-connrefused https://example.com/file.zip
This command instructs wget to retry even if the connection is explicitly refused by the server.
Background Execution and Logging of Downloads
Previous sections have not covered wget's capability to execute downloads in the background and manage logging. This subsection specifically addresses these features, which are essential for users who need to initiate lengthy downloads without maintaining an active terminal session.
To run wget in the background, use the -b or --background option:
wget -b https://example.com/largefile.iso
When executed, wget detaches from the terminal and continues downloading in the background, producing a log file (wget-log) in the current directory. Users can monitor the download progress by inspecting this log file:
tail -f wget-log
This command displays real-time updates from the log file, allowing users to monitor download progress without interrupting the background process.
Recursive Downloads and Depth Control
While previous sections have touched upon website mirroring, this subsection specifically addresses the control of recursion depth, a topic not yet discussed. Recursive downloads allow wget to fetch multiple linked pages or resources from a website, and users can precisely control the recursion depth to limit the scope of downloads.
To initiate a recursive download, use the -r or --recursive option:
wget -r https://example.com/docs/
By default, wget recursively retrieves linked pages up to a depth of 5. To explicitly control recursion depth, use the -l or --level option:
wget -r -l 2 https://example.com/docs/
This command limits recursion to two levels deep, downloading only the initial page and pages directly linked from it. To allow unlimited recursion depth, users can specify inf or 0:
wget -r -l inf https://example.com/docs/
This command recursively downloads all linked pages without depth limitation (CRNX).
Controlling Bandwidth Usage and Download Speed
This subsection specifically addresses bandwidth management, a topic not previously covered. wget provides useful options to limit download speed, which is particularly beneficial when network resources are limited or shared.
To limit the download speed, use the --limit-rate option followed by the desired speed in bytes, kilobytes (k), megabytes (m), or gigabytes (g):
wget --limit-rate=500k https://example.com/file.zip
This command restricts wget's download speed to 500 kilobytes per second, helping to conserve bandwidth for other applications or users (CommandMasters).
Downloading Files Using Authentication
While previous sections have briefly mentioned authentication, this subsection specifically explores how to handle HTTP Basic Authentication and FTP authentication in wget commands.
For HTTP Basic Authentication, users can specify credentials directly in the command:
wget --user=username --password=password https://example.com/protected/file.zip
For FTP downloads requiring authentication, wget similarly allows specifying credentials:
wget --ftp-user=username --ftp-password=password ftp://example.com/file.tar.gz
These commands enable wget to access and download files from protected resources requiring authentication.
Using Input Files for Batch Downloads
This subsection specifically addresses batch downloads using input files, a topic not previously discussed. wget can read URLs from a file, making it efficient for downloading multiple resources without manually specifying each URL.
To download URLs listed in a file, use the -i or --input-file option:
wget -i urls.txt
Here, wget reads URLs line-by-line from the file "urls.txt" and downloads each resource sequentially. This approach significantly simplifies batch download tasks, especially when dealing with numerous URLs.
Wget command options summary
Startup:
-V, --version display the version of Wget and exit.
-h, --help print this help.
-b, --background go to background after startup.
-e, --execute=COMMAND execute a `.wgetrc'-style command.
Logging and input file:
-o, --output-file=FILE log messages to FILE.
-a, --append-output=FILE append messages to FILE.
-d, --debug print lots of debugging information.
-q, --quiet quiet (no output).
-v, --verbose be verbose (this is the default).
-nv, --no-verbose turn off verboseness, without being quiet.
--report-speed=TYPE Output bandwidth as TYPE. TYPE can be bits.
-i, --input-file=FILE download URLs found in local or external FILE.
-F, --force-html treat input file as HTML.
-B, --base=URL resolves HTML input-file links (-i -F)
relative to URL.
--config=FILE Specify config file to use.
Download:
-t, --tries=NUMBER set number of retries to NUMBER (0 unlimits).
--retry-connrefused retry even if connection is refused.
-O, --output-document=FILE write documents to FILE.
-nc, --no-clobber skip downloads that would download to
existing files (overwriting them).
-c, --continue resume getting a partially-downloaded file.
--progress=TYPE select progress gauge type.
-N, --timestamping don't re-retrieve files unless newer than
local.
--no-use-server-timestamps don't set the local file's timestamp by
the one on the server.
-S, --server-response print server response.
--spider don't download anything.
-T, --timeout=SECONDS set all timeout values to SECONDS.
--dns-timeout=SECS set the DNS lookup timeout to SECS.
--connect-timeout=SECS set the connect timeout to SECS.
--read-timeout=SECS set the read timeout to SECS.
-w, --wait=SECONDS wait SECONDS between retrievals.
--waitretry=SECONDS wait 1..SECONDS between retries of a retrieval.
--random-wait wait from 0.5*WAIT...1.5*WAIT secs between retrievals.
--no-proxy explicitly turn off proxy.
-Q, --quota=NUMBER set retrieval quota to NUMBER.
--bind-address=ADDRESS bind to ADDRESS (hostname or IP) on local host.
--limit-rate=RATE limit download rate to RATE.
--no-dns-cache disable caching DNS lookups.
--restrict-file-names=OS restrict chars in file names to ones OS allows.
--ignore-case ignore case when matching files/directories.
-4, --inet4-only connect only to IPv4 addresses.
-6, --inet6-only connect only to IPv6 addresses.
--prefer-family=FAMILY connect first to addresses of specified family,
one of IPv6, IPv4, or none.
--user=USER set both ftp and http user to USER.
--password=PASS set both ftp and http password to PASS.
--ask-password prompt for passwords.
--no-iri turn off IRI support.
--local-encoding=ENC use ENC as the local encoding for IRIs.
--remote-encoding=ENC use ENC as the default remote encoding.
--unlink remove file before clobber.
Directories:
-nd, --no-directories don't create directories.
-x, --force-directories force creation of directories.
-nH, --no-host-directories don't create host directories.
--protocol-directories use protocol name in directories.
-P, --directory-prefix=PREFIX save files to PREFIX/...
--cut-dirs=NUMBER ignore NUMBER remote directory components.
HTTP options:
--http-user=USER set http user to USER.
--http-password=PASS set http password to PASS.
--no-cache disallow server-cached data.
--default-page=NAME Change the default page name (normally
this is `index.html'.).
-E, --adjust-extension save HTML/CSS documents with proper extensions.
--ignore-length ignore `Content-Length' header field.
--header=STRING insert STRING among the headers.
--max-redirect maximum redirections allowed per page.
--proxy-user=USER set USER as proxy username.
--proxy-password=PASS set PASS as proxy password.
--referer=URL include `Referer: URL' header in HTTP request.
--save-headers save the HTTP headers to file.
-U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION.
--no-http-keep-alive disable HTTP keep-alive (persistent connections).
--no-cookies don't use cookies.
--load-cookies=FILE load cookies from FILE before session.
--save-cookies=FILE save cookies to FILE after session.
--keep-session-cookies load and save session (non-permanent) cookies.
--post-data=STRING use the POST method; send STRING as the data.
--post-file=FILE use the POST method; send contents of FILE.
--content-disposition honor the Content-Disposition header when
choosing local file names (EXPERIMENTAL).
--content-on-error output the received content on server errors.
--auth-no-challenge send Basic HTTP authentication information
without first waiting for the server's
challenge.
HTTPS (SSL/TLS) options:
--secure-protocol=PR choose secure protocol, one of auto, SSLv2,
SSLv3, and TLSv1.
--no-check-certificate don't validate the server's certificate.
--certificate=FILE client certificate file.
--certificate-type=TYPE client certificate type, PEM or DER.
--private-key=FILE private key file.
--private-key-type=TYPE private key type, PEM or DER.
--ca-certificate=FILE file with the bundle of CA's.
--ca-directory=DIR directory where hash list of CA's is stored.
--random-file=FILE file with random data for seeding the SSL PRNG.
--egd-file=FILE file naming the EGD socket with random data.
FTP options:
--ftp-user=USER set ftp user to USER.
--ftp-password=PASS set ftp password to PASS.
--no-remove-listing don't remove `.listing' files.
--no-glob turn off FTP file name globbing.
--no-passive-ftp disable the "passive" transfer mode.
--preserve-permissions preserve remote file permissions.
--retr-symlinks when recursing, get linked-to files (not dir).
WARC options:
--warc-file=FILENAME save request/response data to a .warc.gz file.
--warc-header=STRING insert STRING into the warcinfo record.
--warc-max-size=NUMBER set maximum size of WARC files to NUMBER.
--warc-cdx write CDX index files.
--warc-dedup=FILENAME do not store records listed in this CDX file.
--no-warc-compression do not compress WARC files with GZIP.
--no-warc-digests do not calculate SHA1 digests.
--no-warc-keep-log do not store the log file in a WARC record.
--warc-tempdir=DIRECTORY location for temporary files created by the
WARC writer.
Recursive download:
-r, --recursive specify recursive download.
-l, --level=NUMBER maximum recursion depth (inf or 0 for infinite).
--delete-after delete files locally after downloading them.
-k, --convert-links make links in downloaded HTML or CSS point to
local files.
-K, --backup-converted before converting file X, back up as X.orig.
-m, --mirror shortcut for -N -r -l inf --no-remove-listing.
-p, --page-requisites get all images, etc. needed to display HTML page.
--strict-comments turn on strict (SGML) handling of HTML comments.
Recursive accept/reject:
-A, --accept=LIST comma-separated list of accepted extensions.
-R, --reject=LIST comma-separated list of rejected extensions.
--accept-regex=REGEX regex matching accepted URLs.
--reject-regex=REGEX regex matching rejected URLs.
--regex-type=TYPE regex type (posix).
-D, --domains=LIST comma-separated list of accepted domains.
--exclude-domains=LIST comma-separated list of rejected domains.
--follow-ftp follow FTP links from HTML documents.
--follow-tags=LIST comma-separated list of followed HTML tags.
--ignore-tags=LIST comma-separated list of ignored HTML tags.
-H, --span-hosts go to foreign hosts when recursive.
-L, --relative follow relative links only.
-I, --include-directories=LIST list of allowed directories.
--trust-server-names use the name specified by the redirection
url last component.
-X, --exclude-directories=LIST list of excluded directories.
-np, --no-parent don't ascend to the parent directory.
Summary and Best Practices for Using Wget in Web Scraping
Mastering Wget is crucial for anyone involved in web scraping, data extraction, or automated file retrieval tasks. Its extensive range of features — from basic file downloads to advanced recursive scraping and bandwidth management — makes it a versatile and powerful tool. By leveraging Wget's capabilities, users can efficiently handle multiple URLs, customize download behaviors, and manage authentication seamlessly, significantly enhancing productivity and reliability in data-driven workflows.
Moreover, Wget's ability to execute downloads in the background and log progress allows users to initiate lengthy downloads without maintaining active terminal sessions, further improving workflow efficiency. Its robust handling of network instabilities through configurable retries and timeouts ensures that downloads are resilient and reliable.
In conclusion, becoming proficient with Wget not only simplifies complex download tasks but also empowers users to automate and optimize their data extraction processes effectively. Whether you're a beginner or an experienced professional, incorporating Wget into your toolkit will undoubtedly streamline your web scraping and data extraction projects.