1 \input texinfo @c -*-texinfo-*-
7 @settitle GNU Wget @value{VERSION} Manual
8 @c Disable the monstrous rectangles beside overfull hbox-es.
10 @c Use `odd' to print double-sided.
15 @c Remove this if you don't use A4 paper.
19 @c Title for man page. The weird way texi2pod.pl is written requires
20 @c the preceding @set.
22 @c man title Wget The non-interactive network downloader.
24 @dircategory Network Applications
26 * Wget: (wget). The non-interactive network downloader.
30 This file documents the the GNU Wget utility for downloading network
33 @c man begin COPYRIGHT
34 Copyright @copyright{} 1996--2005 Free Software Foundation, Inc.
36 Permission is granted to make and distribute verbatim copies of
37 this manual provided the copyright notice and this permission notice
38 are preserved on all copies.
41 Permission is granted to process this file through TeX and print the
42 results, provided the printed document carries a copying permission
43 notice identical to this one except for the removal of this paragraph
44 (this paragraph not being relevant to the printed manual).
46 Permission is granted to copy, distribute and/or modify this document
47 under the terms of the GNU Free Documentation License, Version 1.2 or
48 any later version published by the Free Software Foundation; with the
49 Invariant Sections being ``GNU General Public License'' and ``GNU Free
50 Documentation License'', with no Front-Cover Texts, and with no
51 Back-Cover Texts. A copy of the license is included in the section
52 entitled ``GNU Free Documentation License''.
57 @title GNU Wget @value{VERSION}
58 @subtitle The non-interactive download utility
59 @subtitle Updated for Wget @value{VERSION}, @value{UPDATED}
60 @author by Hrvoje Nik@v{s}i@'{c} and the developers
64 Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
67 GNU Info entry for @file{wget}.
72 @vskip 0pt plus 1filll
73 Copyright @copyright{} 1996--2005, Free Software Foundation, Inc.
75 Permission is granted to copy, distribute and/or modify this document
76 under the terms of the GNU Free Documentation License, Version 1.2 or
77 any later version published by the Free Software Foundation; with the
78 Invariant Sections being ``GNU General Public License'' and ``GNU Free
79 Documentation License'', with no Front-Cover Texts, and with no
80 Back-Cover Texts. A copy of the license is included in the section
81 entitled ``GNU Free Documentation License''.
86 @top Wget @value{VERSION}
88 This manual documents version @value{VERSION} of GNU Wget, the freely
89 available utility for network downloads.
91 Copyright @copyright{} 1996--2005 Free Software Foundation, Inc.
94 * Overview:: Features of Wget.
95 * Invoking:: Wget command-line arguments.
96 * Recursive Download:: Downloading interlinked pages.
97 * Following Links:: The available methods of chasing links.
98 * Time-Stamping:: Mirroring according to time-stamps.
99 * Startup File:: Wget's initialization file.
100 * Examples:: Examples of usage.
101 * Various:: The stuff that doesn't fit anywhere else.
102 * Appendices:: Some useful references.
103 * Copying:: You may give out copies of Wget and of this manual.
104 * Concept Index:: Topics covered by this manual.
113 @c man begin DESCRIPTION
114 GNU Wget is a free utility for non-interactive download of files from
115 the Web. It supports @sc{http}, @sc{https}, and @sc{ftp} protocols, as
116 well as retrieval through @sc{http} proxies.
119 This chapter is a partial overview of Wget's features.
123 @c man begin DESCRIPTION
124 Wget is non-interactive, meaning that it can work in the background,
125 while the user is not logged on. This allows you to start a retrieval
126 and disconnect from the system, letting Wget finish the work. By
127 contrast, most of the Web browsers require constant user's presence,
128 which can be a great hindrance when transferring a lot of data.
134 @c man begin DESCRIPTION
138 @c man begin DESCRIPTION
139 Wget can follow links in @sc{html} and @sc{xhtml} pages and create local
140 versions of remote web sites, fully recreating the directory structure of
141 the original site. This is sometimes referred to as ``recursive
142 downloading.'' While doing that, Wget respects the Robot Exclusion
143 Standard (@file{/robots.txt}). Wget can be instructed to convert the
144 links in downloaded @sc{html} files to the local files for offline
150 File name wildcard matching and recursive mirroring of directories are
151 available when retrieving via @sc{ftp}. Wget can read the time-stamp
152 information given by both @sc{http} and @sc{ftp} servers, and store it
153 locally. Thus Wget can see if the remote file has changed since last
154 retrieval, and automatically retrieve the new version if it has. This
155 makes Wget suitable for mirroring of @sc{ftp} sites, as well as home
161 @c man begin DESCRIPTION
165 @c man begin DESCRIPTION
166 Wget has been designed for robustness over slow or unstable network
167 connections; if a download fails due to a network problem, it will
168 keep retrying until the whole file has been retrieved. If the server
169 supports regetting, it will instruct the server to continue the
170 download from where it left off.
175 Wget supports proxy servers, which can lighten the network load, speed
176 up retrieval and provide access behind firewalls. However, if you are
177 behind a firewall that requires that you use a socks style gateway,
178 you can get the socks library and build Wget with support for socks.
179 Wget uses the passive @sc{ftp} downloading by default, active @sc{ftp}
184 Wget supports IP version 6, the next generation of IP. IPv6 is
185 autodetected at compile-time, and can be disabled at either build or
186 run time. Binaries built with IPv6 support work well in both
187 IPv4-only and dual family environments.
191 Built-in features offer mechanisms to tune which links you wish to follow
192 (@pxref{Following Links}).
196 The progress of individual downloads is traced using a progress gauge.
197 Interactive downloads are tracked using a ``thermometer''-style gauge,
198 whereas non-interactive ones are traced with dots, each dot
199 representing a fixed amount of data received (1KB by default). Either
200 gauge can be customized to your preferences.
204 Most of the features are fully configurable, either through command line
205 options, or via the initialization file @file{.wgetrc} (@pxref{Startup
206 File}). Wget allows you to define @dfn{global} startup files
207 (@file{/usr/local/etc/wgetrc} by default) for site settings.
212 @item /usr/local/etc/wgetrc
213 Default location of the @dfn{global} startup file.
223 Finally, GNU Wget is free software. This means that everyone may use
224 it, redistribute it and/or modify it under the terms of the GNU General
225 Public License, as published by the Free Software Foundation
236 By default, Wget is very simple to invoke. The basic syntax is:
239 @c man begin SYNOPSIS
240 wget [@var{option}]@dots{} [@var{URL}]@dots{}
244 Wget will simply download all the @sc{url}s specified on the command
245 line. @var{URL} is a @dfn{Uniform Resource Locator}, as defined below.
247 However, you may wish to change some of the default parameters of
248 Wget. You can do it two ways: permanently, adding the appropriate
249 command to @file{.wgetrc} (@pxref{Startup File}), or specifying it on
255 * Basic Startup Options::
256 * Logging and Input File Options::
258 * Directory Options::
260 * HTTPS (SSL/TLS) Options::
262 * Recursive Retrieval Options::
263 * Recursive Accept/Reject Options::
271 @dfn{URL} is an acronym for Uniform Resource Locator. A uniform
272 resource locator is a compact string representation for a resource
273 available via the Internet. Wget recognizes the @sc{url} syntax as per
274 @sc{rfc1738}. This is the most widely used form (square brackets denote
278 http://host[:port]/directory/file
279 ftp://host[:port]/directory/file
282 You can also encode your username and password within a @sc{url}:
285 ftp://user:password@@host/path
286 http://user:password@@host/path
289 Either @var{user} or @var{password}, or both, may be left out. If you
290 leave out either the @sc{http} username or password, no authentication
291 will be sent. If you leave out the @sc{ftp} username, @samp{anonymous}
292 will be used. If you leave out the @sc{ftp} password, your email
293 address will be supplied as a default password.@footnote{If you have a
294 @file{.netrc} file in your home directory, password will also be
297 @strong{Important Note}: if you specify a password-containing @sc{url}
298 on the command line, the username and password will be plainly visible
299 to all users on the system, by way of @code{ps}. On multi-user systems,
300 this is a big security risk. To work around it, use @code{wget -i -}
301 and feed the @sc{url}s to Wget's standard input, each on a separate
302 line, terminated by @kbd{C-d}.
304 You can encode unsafe characters in a @sc{url} as @samp{%xy}, @code{xy}
305 being the hexadecimal representation of the character's @sc{ascii}
306 value. Some common unsafe characters include @samp{%} (quoted as
307 @samp{%25}), @samp{:} (quoted as @samp{%3A}), and @samp{@@} (quoted as
308 @samp{%40}). Refer to @sc{rfc1738} for a comprehensive list of unsafe
311 Wget also supports the @code{type} feature for @sc{ftp} @sc{url}s. By
312 default, @sc{ftp} documents are retrieved in the binary mode (type
313 @samp{i}), which means that they are downloaded unchanged. Another
314 useful mode is the @samp{a} (@dfn{ASCII}) mode, which converts the line
315 delimiters between the different operating systems, and is thus useful
316 for text files. Here is an example:
319 ftp://host/directory/file;type=a
322 Two alternative variants of @sc{url} specification are also supported,
323 because of historical (hysterical?) reasons and their widespreaded use.
325 @sc{ftp}-only syntax (supported by @code{NcFTP}):
330 @sc{http}-only syntax (introduced by @code{Netscape}):
335 These two alternative forms are deprecated, and may cease being
336 supported in the future.
338 If you do not understand the difference between these notations, or do
339 not know which one to use, just use the plain ordinary format you use
340 with your favorite browser, like @code{Lynx} or @code{Netscape}.
345 @section Option Syntax
346 @cindex option syntax
347 @cindex syntax of options
349 Since Wget uses GNU getopt to process command-line arguments, every
350 option has a long form along with the short one. Long options are
351 more convenient to remember, but take time to type. You may freely
352 mix different option styles, or specify options after the command-line
353 arguments. Thus you may write:
356 wget -r --tries=10 http://fly.srk.fer.hr/ -o log
359 The space between the option accepting an argument and the argument may
360 be omitted. Instead @samp{-o log} you can write @samp{-olog}.
362 You may put several options that do not require arguments together,
369 This is a complete equivalent of:
372 wget -d -r -c @var{URL}
375 Since the options can be specified after the arguments, you may
376 terminate them with @samp{--}. So the following will try to download
377 @sc{url} @samp{-x}, reporting failure to @file{log}:
383 The options that accept comma-separated lists all respect the convention
384 that specifying an empty list clears its value. This can be useful to
385 clear the @file{.wgetrc} settings. For instance, if your @file{.wgetrc}
386 sets @code{exclude_directories} to @file{/cgi-bin}, the following
387 example will first reset it, and then set it to exclude @file{/~nobody}
388 and @file{/~somebody}. You can also clear the lists in @file{.wgetrc}
389 (@pxref{Wgetrc Syntax}).
392 wget -X '' -X /~nobody,/~somebody
395 Most options that do not accept arguments are @dfn{boolean} options,
396 so named because their state can be captured with a yes-or-no
397 (``boolean'') variable. For example, @samp{--follow-ftp} tells Wget
398 to follow FTP links from HTML files and, on the other hand,
399 @samp{--no-glob} tells it not to perform file globbing on FTP URLs. A
400 boolean option is either @dfn{affirmative} or @dfn{negative}
401 (beginning with @samp{--no}). All such options share several
404 Unless stated otherwise, it is assumed that the default behavior is
405 the opposite of what the option accomplishes. For example, the
406 documented existence of @samp{--follow-ftp} assumes that the default
407 is to @emph{not} follow FTP links from HTML pages.
409 Affirmative options can be negated by prepending the @samp{--no-} to
410 the option name; negative options can be negated by omitting the
411 @samp{--no-} prefix. This might seem superfluous---if the default for
412 an affirmative option is to not do something, then why provide a way
413 to explicitly turn it off? But the startup file may in fact change
414 the default. For instance, using @code{follow_ftp = off} in
415 @file{.wgetrc} makes Wget @emph{not} follow FTP links by default, and
416 using @samp{--no-follow-ftp} is the only way to restore the factory
417 default from the command line.
419 @node Basic Startup Options
420 @section Basic Startup Options
425 Display the version of Wget.
429 Print a help message describing all of Wget's command-line options.
433 Go to background immediately after startup. If no output file is
434 specified via the @samp{-o}, output is redirected to @file{wget-log}.
436 @cindex execute wgetrc command
437 @item -e @var{command}
438 @itemx --execute @var{command}
439 Execute @var{command} as if it were a part of @file{.wgetrc}
440 (@pxref{Startup File}). A command thus invoked will be executed
441 @emph{after} the commands in @file{.wgetrc}, thus taking precedence over
442 them. If you need to specify more than one wgetrc command, use multiple
443 instances of @samp{-e}.
447 @node Logging and Input File Options
448 @section Logging and Input File Options
453 @item -o @var{logfile}
454 @itemx --output-file=@var{logfile}
455 Log all messages to @var{logfile}. The messages are normally reported
458 @cindex append to log
459 @item -a @var{logfile}
460 @itemx --append-output=@var{logfile}
461 Append to @var{logfile}. This is the same as @samp{-o}, only it appends
462 to @var{logfile} instead of overwriting the old log file. If
463 @var{logfile} does not exist, a new file is created.
468 Turn on debug output, meaning various information important to the
469 developers of Wget if it does not work properly. Your system
470 administrator may have chosen to compile Wget without debug support, in
471 which case @samp{-d} will not work. Please note that compiling with
472 debug support is always safe---Wget compiled with the debug support will
473 @emph{not} print any debug info unless requested with @samp{-d}.
474 @xref{Reporting Bugs}, for more information on how to use @samp{-d} for
480 Turn off Wget's output.
485 Turn on verbose output, with all the available data. The default output
490 Turn off verbose without being completely quiet (use @samp{-q} for
491 that), which means that error messages and basic information still get
496 @itemx --input-file=@var{file}
497 Read @sc{url}s from @var{file}. If @samp{-} is specified as
498 @var{file}, @sc{url}s are read from the standard input. (Use
499 @samp{./-} to read from a file literally named @samp{-}.)
501 If this function is used, no @sc{url}s need be present on the command
502 line. If there are @sc{url}s both on the command line and in an input
503 file, those on the command lines will be the first ones to be
504 retrieved. The @var{file} need not be an @sc{html} document (but no
505 harm if it is)---it is enough if the @sc{url}s are just listed
508 However, if you specify @samp{--force-html}, the document will be
509 regarded as @samp{html}. In that case you may have problems with
510 relative links, which you can solve either by adding @code{<base
511 href="@var{url}">} to the documents or by specifying
512 @samp{--base=@var{url}} on the command line.
517 When input is read from a file, force it to be treated as an @sc{html}
518 file. This enables you to retrieve relative links from existing
519 @sc{html} files on your local disk, by adding @code{<base
520 href="@var{url}">} to @sc{html}, or using the @samp{--base} command-line
523 @cindex base for relative links in input file
525 @itemx --base=@var{URL}
526 When used in conjunction with @samp{-F}, prepends @var{URL} to relative
527 links in the file specified by @samp{-i}.
530 @node Download Options
531 @section Download Options
535 @cindex client IP address
536 @cindex IP address, client
537 @item --bind-address=@var{ADDRESS}
538 When making client TCP/IP connections, bind to @var{ADDRESS} on
539 the local machine. @var{ADDRESS} may be specified as a hostname or IP
540 address. This option can be useful if your machine is bound to multiple
545 @cindex number of retries
546 @item -t @var{number}
547 @itemx --tries=@var{number}
548 Set number of retries to @var{number}. Specify 0 or @samp{inf} for
549 infinite retrying. The default is to retry 20 times, with the exception
550 of fatal errors like ``connection refused'' or ``not found'' (404),
551 which are not retried.
554 @itemx --output-document=@var{file}
555 The documents will not be written to the appropriate files, but all
556 will be concatenated together and written to @var{file}. If @samp{-}
557 is used as @var{file}, documents will be printed to standard output,
558 disabling link conversion. (Use @samp{./-} to print to a file
559 literally named @samp{-}.)
561 Note that a combination with @samp{-k} is only well-defined for
562 downloading a single document.
564 @cindex clobbering, file
565 @cindex downloading multiple times
569 If a file is downloaded more than once in the same directory, Wget's
570 behavior depends on a few options, including @samp{-nc}. In certain
571 cases, the local file will be @dfn{clobbered}, or overwritten, upon
572 repeated download. In other cases it will be preserved.
574 When running Wget without @samp{-N}, @samp{-nc}, or @samp{-r},
575 downloading the same file in the same directory will result in the
576 original copy of @var{file} being preserved and the second copy being
577 named @samp{@var{file}.1}. If that file is downloaded yet again, the
578 third copy will be named @samp{@var{file}.2}, and so on. When
579 @samp{-nc} is specified, this behavior is suppressed, and Wget will
580 refuse to download newer copies of @samp{@var{file}}. Therefore,
581 ``@code{no-clobber}'' is actually a misnomer in this mode---it's not
582 clobbering that's prevented (as the numeric suffixes were already
583 preventing clobbering), but rather the multiple version saving that's
586 When running Wget with @samp{-r}, but without @samp{-N} or @samp{-nc},
587 re-downloading a file will result in the new copy simply overwriting the
588 old. Adding @samp{-nc} will prevent this behavior, instead causing the
589 original version to be preserved and any newer copies on the server to
592 When running Wget with @samp{-N}, with or without @samp{-r}, the
593 decision as to whether or not to download a newer copy of a file depends
594 on the local and remote timestamp and size of the file
595 (@pxref{Time-Stamping}). @samp{-nc} may not be specified at the same
598 Note that when @samp{-nc} is specified, files with the suffixes
599 @samp{.html} or @samp{.htm} will be loaded from the local disk and
600 parsed as if they had been retrieved from the Web.
602 @cindex continue retrieval
603 @cindex incomplete downloads
604 @cindex resume download
607 Continue getting a partially-downloaded file. This is useful when you
608 want to finish up a download started by a previous instance of Wget, or
609 by another program. For instance:
612 wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
615 If there is a file named @file{ls-lR.Z} in the current directory, Wget
616 will assume that it is the first portion of the remote file, and will
617 ask the server to continue the retrieval from an offset equal to the
618 length of the local file.
620 Note that you don't need to specify this option if you just want the
621 current invocation of Wget to retry downloading a file should the
622 connection be lost midway through. This is the default behavior.
623 @samp{-c} only affects resumption of downloads started @emph{prior} to
624 this invocation of Wget, and whose local files are still sitting around.
626 Without @samp{-c}, the previous example would just download the remote
627 file to @file{ls-lR.Z.1}, leaving the truncated @file{ls-lR.Z} file
630 Beginning with Wget 1.7, if you use @samp{-c} on a non-empty file, and
631 it turns out that the server does not support continued downloading,
632 Wget will refuse to start the download from scratch, which would
633 effectively ruin existing contents. If you really want the download to
634 start from scratch, remove the file.
636 Also beginning with Wget 1.7, if you use @samp{-c} on a file which is of
637 equal size as the one on the server, Wget will refuse to download the
638 file and print an explanatory message. The same happens when the file
639 is smaller on the server than locally (presumably because it was changed
640 on the server since your last download attempt)---because ``continuing''
641 is not meaningful, no download occurs.
643 On the other side of the coin, while using @samp{-c}, any file that's
644 bigger on the server than locally will be considered an incomplete
645 download and only @code{(length(remote) - length(local))} bytes will be
646 downloaded and tacked onto the end of the local file. This behavior can
647 be desirable in certain cases---for instance, you can use @samp{wget -c}
648 to download just the new portion that's been appended to a data
649 collection or log file.
651 However, if the file is bigger on the server because it's been
652 @emph{changed}, as opposed to just @emph{appended} to, you'll end up
653 with a garbled file. Wget has no way of verifying that the local file
654 is really a valid prefix of the remote file. You need to be especially
655 careful of this when using @samp{-c} in conjunction with @samp{-r},
656 since every file will be considered as an "incomplete download" candidate.
658 Another instance where you'll get a garbled file if you try to use
659 @samp{-c} is if you have a lame @sc{http} proxy that inserts a
660 ``transfer interrupted'' string into the local file. In the future a
661 ``rollback'' option may be added to deal with this case.
663 Note that @samp{-c} only works with @sc{ftp} servers and with @sc{http}
664 servers that support the @code{Range} header.
666 @cindex progress indicator
668 @item --progress=@var{type}
669 Select the type of the progress indicator you wish to use. Legal
670 indicators are ``dot'' and ``bar''.
672 The ``bar'' indicator is used by default. It draws an @sc{ascii} progress
673 bar graphics (a.k.a ``thermometer'' display) indicating the status of
674 retrieval. If the output is not a TTY, the ``dot'' bar will be used by
677 Use @samp{--progress=dot} to switch to the ``dot'' display. It traces
678 the retrieval by printing dots on the screen, each dot representing a
679 fixed amount of downloaded data.
681 When using the dotted retrieval, you may also set the @dfn{style} by
682 specifying the type as @samp{dot:@var{style}}. Different styles assign
683 different meaning to one dot. With the @code{default} style each dot
684 represents 1K, there are ten dots in a cluster and 50 dots in a line.
685 The @code{binary} style has a more ``computer''-like orientation---8K
686 dots, 16-dots clusters and 48 dots per line (which makes for 384K
687 lines). The @code{mega} style is suitable for downloading very large
688 files---each dot represents 64K retrieved, there are eight dots in a
689 cluster, and 48 dots on each line (so each line contains 3M).
691 Note that you can set the default style using the @code{progress}
692 command in @file{.wgetrc}. That setting may be overridden from the
693 command line. The exception is that, when the output is not a TTY, the
694 ``dot'' progress will be favored over ``bar''. To force the bar output,
695 use @samp{--progress=bar:force}.
698 @itemx --timestamping
699 Turn on time-stamping. @xref{Time-Stamping}, for details.
701 @cindex server response, print
703 @itemx --server-response
704 Print the headers sent by @sc{http} servers and responses sent by
707 @cindex Wget as spider
710 When invoked with this option, Wget will behave as a Web @dfn{spider},
711 which means that it will not download the pages, just check that they
712 are there. For example, you can use Wget to check your bookmarks:
715 wget --spider --force-html -i bookmarks.html
718 This feature needs much more work for Wget to get close to the
719 functionality of real web spiders.
723 @itemx --timeout=@var{seconds}
724 Set the network timeout to @var{seconds} seconds. This is equivalent
725 to specifying @samp{--dns-timeout}, @samp{--connect-timeout}, and
726 @samp{--read-timeout}, all at the same time.
728 When interacting with the network, Wget can check for timeout and
729 abort the operation if it takes too long. This prevents anomalies
730 like hanging reads and infinite connects. The only timeout enabled by
731 default is a 900-second read timeout. Setting a timeout to 0 disables
732 it altogether. Unless you know what you are doing, it is best not to
733 change the default timeout settings.
735 All timeout-related options accept decimal values, as well as
736 subsecond values. For example, @samp{0.1} seconds is a legal (though
737 unwise) choice of timeout. Subsecond timeouts are useful for checking
738 server response times or for testing network latency.
742 @item --dns-timeout=@var{seconds}
743 Set the DNS lookup timeout to @var{seconds} seconds. DNS lookups that
744 don't complete within the specified time will fail. By default, there
745 is no timeout on DNS lookups, other than that implemented by system
748 @cindex connect timeout
749 @cindex timeout, connect
750 @item --connect-timeout=@var{seconds}
751 Set the connect timeout to @var{seconds} seconds. TCP connections that
752 take longer to establish will be aborted. By default, there is no
753 connect timeout, other than that implemented by system libraries.
756 @cindex timeout, read
757 @item --read-timeout=@var{seconds}
758 Set the read (and write) timeout to @var{seconds} seconds. The
759 ``time'' of this timeout refers @dfn{idle time}: if, at any point in
760 the download, no data is received for more than the specified number
761 of seconds, reading fails and the download is restarted. This option
762 does not directly affect the duration of the entire download.
764 Of course, the remote server may choose to terminate the connection
765 sooner than this option requires. The default read timeout is 900
768 @cindex bandwidth, limit
770 @cindex limit bandwidth
771 @item --limit-rate=@var{amount}
772 Limit the download speed to @var{amount} bytes per second. Amount may
773 be expressed in bytes, kilobytes with the @samp{k} suffix, or megabytes
774 with the @samp{m} suffix. For example, @samp{--limit-rate=20k} will
775 limit the retrieval rate to 20KB/s. This is useful when, for whatever
776 reason, you don't want Wget to consume the entire available bandwidth.
778 This option allows the use of decimal numbers, usually in conjunction
779 with power suffixes; for example, @samp{--limit-rate=2.5k} is a legal
782 Note that Wget implements the limiting by sleeping the appropriate
783 amount of time after a network read that took less time than specified
784 by the rate. Eventually this strategy causes the TCP transfer to slow
785 down to approximately the specified rate. However, it may take some
786 time for this balance to be achieved, so don't be surprised if limiting
787 the rate doesn't work well with very small files.
791 @item -w @var{seconds}
792 @itemx --wait=@var{seconds}
793 Wait the specified number of seconds between the retrievals. Use of
794 this option is recommended, as it lightens the server load by making the
795 requests less frequent. Instead of in seconds, the time can be
796 specified in minutes using the @code{m} suffix, in hours using @code{h}
797 suffix, or in days using @code{d} suffix.
799 Specifying a large value for this option is useful if the network or the
800 destination host is down, so that Wget can wait long enough to
801 reasonably expect the network error to be fixed before the retry.
803 @cindex retries, waiting between
804 @cindex waiting between retries
805 @item --waitretry=@var{seconds}
806 If you don't want Wget to wait between @emph{every} retrieval, but only
807 between retries of failed downloads, you can use this option. Wget will
808 use @dfn{linear backoff}, waiting 1 second after the first failure on a
809 given file, then waiting 2 seconds after the second failure on that
810 file, up to the maximum number of @var{seconds} you specify. Therefore,
811 a value of 10 will actually make Wget wait up to (1 + 2 + ... + 10) = 55
814 Note that this option is turned on by default in the global
820 Some web sites may perform log analysis to identify retrieval programs
821 such as Wget by looking for statistically significant similarities in
822 the time between requests. This option causes the time between requests
823 to vary between 0 and 2 * @var{wait} seconds, where @var{wait} was
824 specified using the @samp{--wait} option, in order to mask Wget's
825 presence from such analysis.
827 A recent article in a publication devoted to development on a popular
828 consumer platform provided code to perform this analysis on the fly.
829 Its author suggested blocking at the class C address level to ensure
830 automated retrieval programs were blocked despite changing DHCP-supplied
833 The @samp{--random-wait} option was inspired by this ill-advised
834 recommendation to block many unrelated users from a web site due to the
839 Don't use proxies, even if the appropriate @code{*_proxy} environment
842 For more information about the use of proxies with Wget, @xref{Proxies}.
846 @itemx --quota=@var{quota}
847 Specify download quota for automatic retrievals. The value can be
848 specified in bytes (default), kilobytes (with @samp{k} suffix), or
849 megabytes (with @samp{m} suffix).
851 Note that quota will never affect downloading a single file. So if you
852 specify @samp{wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz}, all of the
853 @file{ls-lR.gz} will be downloaded. The same goes even when several
854 @sc{url}s are specified on the command-line. However, quota is
855 respected when retrieving either recursively, or from an input file.
856 Thus you may safely type @samp{wget -Q2m -i sites}---download will be
857 aborted when the quota is exceeded.
859 Setting quota to 0 or to @samp{inf} unlimits the download quota.
862 @cindex caching of DNS lookups
864 Turn off caching of DNS lookups. Normally, Wget remembers the IP
865 addresses it looked up from DNS so it doesn't have to repeatedly
866 contact the DNS server for the same (typically small) set of hosts it
867 retrieves from. This cache exists in memory only; a new Wget run will
870 However, it has been reported that in some situations it is not
871 desirable to cache host names, even for the duration of a
872 short-running application like Wget. With this option Wget issues a
873 new DNS lookup (more precisely, a new call to @code{gethostbyname} or
874 @code{getaddrinfo}) each time it makes a new connection. Please note
875 that this option will @emph{not} affect caching that might be
876 performed by the resolving library or by an external caching layer,
879 If you don't understand exactly what this option does, you probably
882 @cindex file names, restrict
883 @cindex Windows file names
884 @item --restrict-file-names=@var{mode}
885 Change which characters found in remote URLs may show up in local file
886 names generated from those URLs. Characters that are @dfn{restricted}
887 by this option are escaped, i.e. replaced with @samp{%HH}, where
888 @samp{HH} is the hexadecimal number that corresponds to the restricted
891 By default, Wget escapes the characters that are not valid as part of
892 file names on your operating system, as well as control characters that
893 are typically unprintable. This option is useful for changing these
894 defaults, either because you are downloading to a non-native partition,
895 or because you want to disable escaping of the control characters.
897 When mode is set to ``unix'', Wget escapes the character @samp{/} and
898 the control characters in the ranges 0--31 and 128--159. This is the
899 default on Unix-like OS'es.
901 When mode is set to ``windows'', Wget escapes the characters @samp{\},
902 @samp{|}, @samp{/}, @samp{:}, @samp{?}, @samp{"}, @samp{*}, @samp{<},
903 @samp{>}, and the control characters in the ranges 0--31 and 128--159.
904 In addition to this, Wget in Windows mode uses @samp{+} instead of
905 @samp{:} to separate host and port in local file names, and uses
906 @samp{@@} instead of @samp{?} to separate the query portion of the file
907 name from the rest. Therefore, a URL that would be saved as
908 @samp{www.xemacs.org:4300/search.pl?input=blah} in Unix mode would be
909 saved as @samp{www.xemacs.org+4300/search.pl@@input=blah} in Windows
910 mode. This mode is the default on Windows.
912 If you append @samp{,nocontrol} to the mode, as in
913 @samp{unix,nocontrol}, escaping of the control characters is also
914 switched off. You can use @samp{--restrict-file-names=nocontrol} to
915 turn off escaping of control characters without affecting the choice of
916 the OS to use as file name restriction mode.
923 Force connecting to IPv4 or IPv6 addresses. With @samp{--inet4-only}
924 or @samp{-4}, Wget will only connect to IPv4 hosts, ignoring AAAA
925 records in DNS, and refusing to connect to IPv6 addresses specified in
926 URLs. Conversely, with @samp{--inet6-only} or @samp{-6}, Wget will
927 only connect to IPv6 hosts and ignore A records and IPv4 addresses.
929 Neither options should be needed normally. By default, an IPv6-aware
930 Wget will use the address family specified by the host's DNS record.
931 If the DNS specifies both an A record and an AAAA record, Wget will
932 try them in sequence until it finds one it can connect to.
934 These options can be used to deliberately force the use of IPv4 or
935 IPv6 address families on dual family systems, usually to aid debugging
936 or to deal with broken network configuration. Only one of
937 @samp{--inet6-only} and @samp{--inet4-only} may be specified in the
938 same command. Neither option is available in Wget compiled without
941 @item --prefer-family=IPv4/IPv6/none
942 When given a choice of several addresses, connect to the addresses
943 with specified address family first. IPv4 addresses are preferred by
946 This avoids spurious errors and connect attempts when accessing hosts
947 that resolve to both IPv6 and IPv4 addresses from IPv4 networks. For
948 example, @samp{www.kame.net} resolves to
949 @samp{2001:200:0:8002:203:47ff:fea5:3085} and to
950 @samp{203.178.141.194}. When the preferred family is @code{IPv4}, the
951 IPv4 address is used first; when the preferred family is @code{IPv6},
952 the IPv6 address is used first; if the specified value is @code{none},
953 the address order returned by DNS is used without change.
955 Unlike @samp{-4} and @samp{-6}, this option doesn't inhibit access to
956 any address family, it only changes the @emph{order} in which the
957 addresses are accessed. Also note that the reordering performed by
958 this option is @dfn{stable}---it doesn't affect order of addresses of
959 the same family. That is, the relative order of all IPv4 addresses
960 and of all IPv6 addresses remains intact in all cases.
962 @item --retry-connrefused
963 Consider ``connection refused'' a transient error and try again.
964 Normally Wget gives up on a URL when it is unable to connect to the
965 site because failure to connect is taken as a sign that the server is
966 not running at all and that retries would not help. This option is
967 for mirroring unreliable sites whose servers tend to disappear for
968 short periods of time.
972 @cindex authentication
973 @item --user=@var{user}
974 @itemx --password=@var{password}
975 Specify the username @var{user} and password @var{password} for both
976 @sc{ftp} and @sc{http} file retrieval. These parameters can be overridden
977 using the @samp{--ftp-user} and @samp{--ftp-password} options for
978 @sc{ftp} connections and the @samp{--http-user} and @samp{--http-password}
979 options for @sc{http} connections.
982 @node Directory Options
983 @section Directory Options
987 @itemx --no-directories
988 Do not create a hierarchy of directories when retrieving recursively.
989 With this option turned on, all files will get saved to the current
990 directory, without clobbering (if a name shows up more than once, the
991 filenames will get extensions @samp{.n}).
994 @itemx --force-directories
995 The opposite of @samp{-nd}---create a hierarchy of directories, even if
996 one would not have been created otherwise. E.g. @samp{wget -x
997 http://fly.srk.fer.hr/robots.txt} will save the downloaded file to
998 @file{fly.srk.fer.hr/robots.txt}.
1001 @itemx --no-host-directories
1002 Disable generation of host-prefixed directories. By default, invoking
1003 Wget with @samp{-r http://fly.srk.fer.hr/} will create a structure of
1004 directories beginning with @file{fly.srk.fer.hr/}. This option disables
1007 @item --protocol-directories
1008 Use the protocol name as a directory component of local file names. For
1009 example, with this option, @samp{wget -r http://@var{host}} will save to
1010 @samp{http/@var{host}/...} rather than just to @samp{@var{host}/...}.
1012 @cindex cut directories
1013 @item --cut-dirs=@var{number}
1014 Ignore @var{number} directory components. This is useful for getting a
1015 fine-grained control over the directory where recursive retrieval will
1018 Take, for example, the directory at
1019 @samp{ftp://ftp.xemacs.org/pub/xemacs/}. If you retrieve it with
1020 @samp{-r}, it will be saved locally under
1021 @file{ftp.xemacs.org/pub/xemacs/}. While the @samp{-nH} option can
1022 remove the @file{ftp.xemacs.org/} part, you are still stuck with
1023 @file{pub/xemacs}. This is where @samp{--cut-dirs} comes in handy; it
1024 makes Wget not ``see'' @var{number} remote directory components. Here
1025 are several examples of how @samp{--cut-dirs} option works.
1029 No options -> ftp.xemacs.org/pub/xemacs/
1031 -nH --cut-dirs=1 -> xemacs/
1032 -nH --cut-dirs=2 -> .
1034 --cut-dirs=1 -> ftp.xemacs.org/xemacs/
1039 If you just want to get rid of the directory structure, this option is
1040 similar to a combination of @samp{-nd} and @samp{-P}. However, unlike
1041 @samp{-nd}, @samp{--cut-dirs} does not lose with subdirectories---for
1042 instance, with @samp{-nH --cut-dirs=1}, a @file{beta/} subdirectory will
1043 be placed to @file{xemacs/beta}, as one would expect.
1045 @cindex directory prefix
1046 @item -P @var{prefix}
1047 @itemx --directory-prefix=@var{prefix}
1048 Set directory prefix to @var{prefix}. The @dfn{directory prefix} is the
1049 directory where all other files and subdirectories will be saved to,
1050 i.e. the top of the retrieval tree. The default is @samp{.} (the
1055 @section HTTP Options
1058 @cindex .html extension
1060 @itemx --html-extension
1061 If a file of type @samp{application/xhtml+xml} or @samp{text/html} is
1062 downloaded and the URL does not end with the regexp
1063 @samp{\.[Hh][Tt][Mm][Ll]?}, this option will cause the suffix @samp{.html}
1064 to be appended to the local filename. This is useful, for instance, when
1065 you're mirroring a remote site that uses @samp{.asp} pages, but you want
1066 the mirrored pages to be viewable on your stock Apache server. Another
1067 good use for this is when you're downloading CGI-generated materials. A URL
1068 like @samp{http://site.com/article.cgi?25} will be saved as
1069 @file{article.cgi?25.html}.
1071 Note that filenames changed in this way will be re-downloaded every time
1072 you re-mirror a site, because Wget can't tell that the local
1073 @file{@var{X}.html} file corresponds to remote URL @samp{@var{X}} (since
1074 it doesn't yet know that the URL produces output of type
1075 @samp{text/html} or @samp{application/xhtml+xml}. To prevent this
1076 re-downloading, you must use @samp{-k} and @samp{-K} so that the original
1077 version of the file will be saved as @file{@var{X}.orig} (@pxref{Recursive
1078 Retrieval Options}).
1081 @cindex http password
1082 @cindex authentication
1083 @item --http-user=@var{user}
1084 @itemx --http-password=@var{password}
1085 Specify the username @var{user} and password @var{password} on an
1086 @sc{http} server. According to the type of the challenge, Wget will
1087 encode them using either the @code{basic} (insecure) or the
1088 @code{digest} authentication scheme.
1090 Another way to specify username and password is in the @sc{url} itself
1091 (@pxref{URL Format}). Either method reveals your password to anyone who
1092 bothers to run @code{ps}. To prevent the passwords from being seen,
1093 store them in @file{.wgetrc} or @file{.netrc}, and make sure to protect
1094 those files from other users with @code{chmod}. If the passwords are
1095 really important, do not leave them lying in those files either---edit
1096 the files and delete them after Wget has started the download.
1099 For more information about security issues with Wget, @xref{Security
1106 Disable server-side cache. In this case, Wget will send the remote
1107 server an appropriate directive (@samp{Pragma: no-cache}) to get the
1108 file from the remote service, rather than returning the cached version.
1109 This is especially useful for retrieving and flushing out-of-date
1110 documents on proxy servers.
1112 Caching is allowed by default.
1116 Disable the use of cookies. Cookies are a mechanism for maintaining
1117 server-side state. The server sends the client a cookie using the
1118 @code{Set-Cookie} header, and the client responds with the same cookie
1119 upon further requests. Since cookies allow the server owners to keep
1120 track of visitors and for sites to exchange this information, some
1121 consider them a breach of privacy. The default is to use cookies;
1122 however, @emph{storing} cookies is not on by default.
1124 @cindex loading cookies
1125 @cindex cookies, loading
1126 @item --load-cookies @var{file}
1127 Load cookies from @var{file} before the first HTTP retrieval.
1128 @var{file} is a textual file in the format originally used by Netscape's
1129 @file{cookies.txt} file.
1131 You will typically use this option when mirroring sites that require
1132 that you be logged in to access some or all of their content. The login
1133 process typically works by the web server issuing an @sc{http} cookie
1134 upon receiving and verifying your credentials. The cookie is then
1135 resent by the browser when accessing that part of the site, and so
1136 proves your identity.
1138 Mirroring such a site requires Wget to send the same cookies your
1139 browser sends when communicating with the site. This is achieved by
1140 @samp{--load-cookies}---simply point Wget to the location of the
1141 @file{cookies.txt} file, and it will send the same cookies your browser
1142 would send in the same situation. Different browsers keep textual
1143 cookie files in different locations:
1147 The cookies are in @file{~/.netscape/cookies.txt}.
1149 @item Mozilla and Netscape 6.x.
1150 Mozilla's cookie file is also named @file{cookies.txt}, located
1151 somewhere under @file{~/.mozilla}, in the directory of your profile.
1152 The full path usually ends up looking somewhat like
1153 @file{~/.mozilla/default/@var{some-weird-string}/cookies.txt}.
1155 @item Internet Explorer.
1156 You can produce a cookie file Wget can use by using the File menu,
1157 Import and Export, Export Cookies. This has been tested with Internet
1158 Explorer 5; it is not guaranteed to work with earlier versions.
1160 @item Other browsers.
1161 If you are using a different browser to create your cookies,
1162 @samp{--load-cookies} will only work if you can locate or produce a
1163 cookie file in the Netscape format that Wget expects.
1166 If you cannot use @samp{--load-cookies}, there might still be an
1167 alternative. If your browser supports a ``cookie manager'', you can use
1168 it to view the cookies used when accessing the site you're mirroring.
1169 Write down the name and value of the cookie, and manually instruct Wget
1170 to send those cookies, bypassing the ``official'' cookie support:
1173 wget --no-cookies --header "Cookie: @var{name}=@var{value}"
1176 @cindex saving cookies
1177 @cindex cookies, saving
1178 @item --save-cookies @var{file}
1179 Save cookies to @var{file} before exiting. This will not save cookies
1180 that have expired or that have no expiry time (so-called ``session
1181 cookies''), but also see @samp{--keep-session-cookies}.
1183 @cindex cookies, session
1184 @cindex session cookies
1185 @item --keep-session-cookies
1186 When specified, causes @samp{--save-cookies} to also save session
1187 cookies. Session cookies are normally not saved because they are
1188 meant to be kept in memory and forgotten when you exit the browser.
1189 Saving them is useful on sites that require you to log in or to visit
1190 the home page before you can access some pages. With this option,
1191 multiple Wget runs are considered a single browser session as far as
1192 the site is concerned.
1194 Since the cookie file format does not normally carry session cookies,
1195 Wget marks them with an expiry timestamp of 0. Wget's
1196 @samp{--load-cookies} recognizes those as session cookies, but it might
1197 confuse other browsers. Also note that cookies so loaded will be
1198 treated as other session cookies, which means that if you want
1199 @samp{--save-cookies} to preserve them again, you must use
1200 @samp{--keep-session-cookies} again.
1202 @cindex Content-Length, ignore
1203 @cindex ignore length
1204 @item --ignore-length
1205 Unfortunately, some @sc{http} servers (@sc{cgi} programs, to be more
1206 precise) send out bogus @code{Content-Length} headers, which makes Wget
1207 go wild, as it thinks not all the document was retrieved. You can spot
1208 this syndrome if Wget retries getting the same document again and again,
1209 each time claiming that the (otherwise normal) connection has closed on
1212 With this option, Wget will ignore the @code{Content-Length} header---as
1213 if it never existed.
1216 @item --header=@var{header-line}
1217 Send @var{header-line} along with the rest of the headers in each
1218 @sc{http} request. The supplied header is sent as-is, which means it
1219 must contain name and value separated by colon, and must not contain
1222 You may define more than one additional header by specifying
1223 @samp{--header} more than once.
1227 wget --header='Accept-Charset: iso-8859-2' \
1228 --header='Accept-Language: hr' \
1229 http://fly.srk.fer.hr/
1233 Specification of an empty string as the header value will clear all
1234 previous user-defined headers.
1236 As of Wget 1.10, this option can be used to override headers otherwise
1237 generated automatically. This example instructs Wget to connect to
1238 localhost, but to specify @samp{foo.bar} in the @code{Host} header:
1241 wget --header="Host: foo.bar" http://localhost/
1244 In versions of Wget prior to 1.10 such use of @samp{--header} caused
1245 sending of duplicate headers.
1248 @cindex proxy password
1249 @cindex proxy authentication
1250 @item --proxy-user=@var{user}
1251 @itemx --proxy-password=@var{password}
1252 Specify the username @var{user} and password @var{password} for
1253 authentication on a proxy server. Wget will encode them using the
1254 @code{basic} authentication scheme.
1256 Security considerations similar to those with @samp{--http-password}
1257 pertain here as well.
1259 @cindex http referer
1260 @cindex referer, http
1261 @item --referer=@var{url}
1262 Include `Referer: @var{url}' header in HTTP request. Useful for
1263 retrieving documents with server-side processing that assume they are
1264 always being retrieved by interactive web browsers and only come out
1265 properly when Referer is set to one of the pages that point to them.
1267 @cindex server response, save
1268 @item --save-headers
1269 Save the headers sent by the @sc{http} server to the file, preceding the
1270 actual contents, with an empty line as the separator.
1273 @item -U @var{agent-string}
1274 @itemx --user-agent=@var{agent-string}
1275 Identify as @var{agent-string} to the @sc{http} server.
1277 The @sc{http} protocol allows the clients to identify themselves using a
1278 @code{User-Agent} header field. This enables distinguishing the
1279 @sc{www} software, usually for statistical purposes or for tracing of
1280 protocol violations. Wget normally identifies as
1281 @samp{Wget/@var{version}}, @var{version} being the current version
1284 However, some sites have been known to impose the policy of tailoring
1285 the output according to the @code{User-Agent}-supplied information.
1286 While this is not such a bad idea in theory, it has been abused by
1287 servers denying information to clients other than (historically)
1288 Netscape or, more frequently, Microsoft Internet Explorer. This
1289 option allows you to change the @code{User-Agent} line issued by Wget.
1290 Use of this option is discouraged, unless you really know what you are
1293 Specifying empty user agent with @samp{--user-agent=""} instructs Wget
1294 not to send the @code{User-Agent} header in @sc{http} requests.
1297 @item --post-data=@var{string}
1298 @itemx --post-file=@var{file}
1299 Use POST as the method for all HTTP requests and send the specified data
1300 in the request body. @code{--post-data} sends @var{string} as data,
1301 whereas @code{--post-file} sends the contents of @var{file}. Other than
1302 that, they work in exactly the same way.
1304 Please be aware that Wget needs to know the size of the POST data in
1305 advance. Therefore the argument to @code{--post-file} must be a regular
1306 file; specifying a FIFO or something like @file{/dev/stdin} won't work.
1307 It's not quite clear how to work around this limitation inherent in
1308 HTTP/1.0. Although HTTP/1.1 introduces @dfn{chunked} transfer that
1309 doesn't require knowing the request length in advance, a client can't
1310 use chunked unless it knows it's talking to an HTTP/1.1 server. And it
1311 can't know that until it receives a response, which in turn requires the
1312 request to have been completed -- a chicken-and-egg problem.
1314 Note: if Wget is redirected after the POST request is completed, it
1315 will not send the POST data to the redirected URL. This is because
1316 URLs that process POST often respond with a redirection to a regular
1317 page, which does not desire or accept POST. It is not completely
1318 clear that this behavior is optimal; if it doesn't work out, it might
1319 be changed in the future.
1321 This example shows how to log to a server using POST and then proceed to
1322 download the desired pages, presumably only accessible to authorized
1327 # @r{Log in to the server. This can be done only once.}
1328 wget --save-cookies cookies.txt \
1329 --post-data 'user=foo&password=bar' \
1330 http://server.com/auth.php
1332 # @r{Now grab the page or pages we care about.}
1333 wget --load-cookies cookies.txt \
1334 -p http://server.com/interesting/article.php
1338 If the server is using session cookies to track user authentication,
1339 the above will not work because @samp{--save-cookies} will not save
1340 them (and neither will browsers) and the @file{cookies.txt} file will
1341 be empty. In that case use @samp{--keep-session-cookies} along with
1342 @samp{--save-cookies} to force saving of session cookies.
1345 @node HTTPS (SSL/TLS) Options
1346 @section HTTPS (SSL/TLS) Options
1349 To support encrypted HTTP (HTTPS) downloads, Wget must be compiled
1350 with an external SSL library, currently OpenSSL. If Wget is compiled
1351 without SSL support, none of these options are available.
1354 @cindex SSL protocol, choose
1355 @item --secure-protocol=@var{protocol}
1356 Choose the secure protocol to be used. Legal values are @samp{auto},
1357 @samp{SSLv2}, @samp{SSLv3}, and @samp{TLSv1}. If @samp{auto} is used,
1358 the SSL library is given the liberty of choosing the appropriate
1359 protocol automatically, which is achieved by sending an SSLv2 greeting
1360 and announcing support for SSLv3 and TLSv1. This is the default.
1362 Specifying @samp{SSLv2}, @samp{SSLv3}, or @samp{TLSv1} forces the use
1363 of the corresponding protocol. This is useful when talking to old and
1364 buggy SSL server implementations that make it hard for OpenSSL to
1365 choose the correct protocol version. Fortunately, such servers are
1368 @cindex SSL certificate, check
1369 @item --no-check-certificate
1370 Don't check the server certificate against the available certificate
1371 authorities. Also don't require the URL host name to match the common
1372 name presented by the certificate.
1374 As of Wget 1.10, the default is to verify the server's certificate
1375 against the recognized certificate authorities, breaking the SSL
1376 handshake and aborting the download if the verification fails.
1377 Although this provides more secure downloads, it does break
1378 interoperability with some sites that worked with previous Wget
1379 versions, particularly those using self-signed, expired, or otherwise
1380 invalid certificates. This option forces an ``insecure'' mode of
1381 operation that turns the certificate verification errors into warnings
1382 and allows you to proceed.
1384 If you encounter ``certificate verification'' errors or ones saying
1385 that ``common name doesn't match requested host name'', you can use
1386 this option to bypass the verification and proceed with the download.
1387 @emph{Only use this option if you are otherwise convinced of the
1388 site's authenticity, or if you really don't care about the validity of
1389 its certificate.} It is almost always a bad idea not to check the
1390 certificates when transmitting confidential or important data.
1392 @cindex SSL certificate
1393 @item --certificate=@var{file}
1394 Use the client certificate stored in @var{file}. This is needed for
1395 servers that are configured to require certificates from the clients
1396 that connect to them. Normally a certificate is not required and this
1399 @cindex SSL certificate type, specify
1400 @item --certificate-type=@var{type}
1401 Specify the type of the client certificate. Legal values are
1402 @samp{PEM} (assumed by default) and @samp{DER}, also known as
1405 @item --private-key=@var{file}
1406 Read the private key from @var{file}. This allows you to provide the
1407 private key in a file separate from the certificate.
1409 @item --private-key-type=@var{type}
1410 Specify the type of the private key. Accepted values are @samp{PEM}
1411 (the default) and @samp{DER}.
1413 @item --ca-certificate=@var{file}
1414 Use @var{file} as the file with the bundle of certificate authorities
1415 (``CA'') to verify the peers. The certificates must be in PEM format.
1417 Without this option Wget looks for CA certificates at the
1418 system-specified locations, chosen at OpenSSL installation time.
1420 @cindex SSL certificate authority
1421 @item --ca-directory=@var{directory}
1422 Specifies directory containing CA certificates in PEM format. Each
1423 file contains one CA certificate, and the file name is based on a hash
1424 value derived from the certificate. This is achieved by processing a
1425 certificate directory with the @code{c_rehash} utility supplied with
1426 OpenSSL. Using @samp{--ca-directory} is more efficient than
1427 @samp{--ca-certificate} when many certificates are installed because
1428 it allows Wget to fetch certificates on demand.
1430 Without this option Wget looks for CA certificates at the
1431 system-specified locations, chosen at OpenSSL installation time.
1433 @cindex entropy, specifying source of
1434 @cindex randomness, specifying source of
1435 @item --random-file=@var{file}
1436 Use @var{file} as the source of random data for seeding the
1437 pseudo-random number generator on systems without @file{/dev/random}.
1439 On such systems the SSL library needs an external source of randomness
1440 to initialize. Randomness may be provided by EGD (see
1441 @samp{--egd-file} below) or read from an external source specified by
1442 the user. If this option is not specified, Wget looks for random data
1443 in @code{$RANDFILE} or, if that is unset, in @file{$HOME/.rnd}. If
1444 none of those are available, it is likely that SSL encryption will not
1447 If you're getting the ``Could not seed OpenSSL PRNG; disabling SSL.''
1448 error, you should provide random data using some of the methods
1452 @item --egd-file=@var{file}
1453 Use @var{file} as the EGD socket. EGD stands for @dfn{Entropy
1454 Gathering Daemon}, a user-space program that collects data from
1455 various unpredictable system sources and makes it available to other
1456 programs that might need it. Encryption software, such as the SSL
1457 library, needs sources of non-repeating randomness to seed the random
1458 number generator used to produce cryptographically strong keys.
1460 OpenSSL allows the user to specify his own source of entropy using the
1461 @code{RAND_FILE} environment variable. If this variable is unset, or
1462 if the specified file does not produce enough randomness, OpenSSL will
1463 read random data from EGD socket specified using this option.
1465 If this option is not specified (and the equivalent startup command is
1466 not used), EGD is never contacted. EGD is not needed on modern Unix
1467 systems that support @file{/dev/random}.
1471 @section FTP Options
1475 @cindex ftp password
1476 @cindex ftp authentication
1477 @item --ftp-user=@var{user}
1478 @itemx --ftp-password=@var{password}
1479 Specify the username @var{user} and password @var{password} on an
1480 @sc{ftp} server. Without this, or the corresponding startup option,
1481 the password defaults to @samp{-wget@@}, normally used for anonymous
1484 Another way to specify username and password is in the @sc{url} itself
1485 (@pxref{URL Format}). Either method reveals your password to anyone who
1486 bothers to run @code{ps}. To prevent the passwords from being seen,
1487 store them in @file{.wgetrc} or @file{.netrc}, and make sure to protect
1488 those files from other users with @code{chmod}. If the passwords are
1489 really important, do not leave them lying in those files either---edit
1490 the files and delete them after Wget has started the download.
1493 For more information about security issues with Wget, @xref{Security
1497 @cindex .listing files, removing
1498 @item --no-remove-listing
1499 Don't remove the temporary @file{.listing} files generated by @sc{ftp}
1500 retrievals. Normally, these files contain the raw directory listings
1501 received from @sc{ftp} servers. Not removing them can be useful for
1502 debugging purposes, or when you want to be able to easily check on the
1503 contents of remote server directories (e.g. to verify that a mirror
1504 you're running is complete).
1506 Note that even though Wget writes to a known filename for this file,
1507 this is not a security hole in the scenario of a user making
1508 @file{.listing} a symbolic link to @file{/etc/passwd} or something and
1509 asking @code{root} to run Wget in his or her directory. Depending on
1510 the options used, either Wget will refuse to write to @file{.listing},
1511 making the globbing/recursion/time-stamping operation fail, or the
1512 symbolic link will be deleted and replaced with the actual
1513 @file{.listing} file, or the listing will be written to a
1514 @file{.listing.@var{number}} file.
1516 Even though this situation isn't a problem, though, @code{root} should
1517 never run Wget in a non-trusted user's directory. A user could do
1518 something as simple as linking @file{index.html} to @file{/etc/passwd}
1519 and asking @code{root} to run Wget with @samp{-N} or @samp{-r} so the file
1520 will be overwritten.
1522 @cindex globbing, toggle
1524 Turn off @sc{ftp} globbing. Globbing refers to the use of shell-like
1525 special characters (@dfn{wildcards}), like @samp{*}, @samp{?}, @samp{[}
1526 and @samp{]} to retrieve more than one file from the same directory at
1530 wget ftp://gnjilux.srk.fer.hr/*.msg
1533 By default, globbing will be turned on if the @sc{url} contains a
1534 globbing character. This option may be used to turn globbing on or off
1537 You may have to quote the @sc{url} to protect it from being expanded by
1538 your shell. Globbing makes Wget look for a directory listing, which is
1539 system-specific. This is why it currently works only with Unix @sc{ftp}
1540 servers (and the ones emulating Unix @code{ls} output).
1543 @item --no-passive-ftp
1544 Disable the use of the @dfn{passive} FTP transfer mode. Passive FTP
1545 mandates that the client connect to the server to establish the data
1546 connection rather than the other way around.
1548 If the machine is connected to the Internet directly, both passive and
1549 active FTP should work equally well. Behind most firewall and NAT
1550 configurations passive FTP has a better chance of working. However,
1551 in some rare firewall configurations, active FTP actually works when
1552 passive FTP doesn't. If you suspect this to be the case, use this
1553 option, or set @code{passive_ftp=off} in your init file.
1555 @cindex symbolic links, retrieving
1556 @item --retr-symlinks
1557 Usually, when retrieving @sc{ftp} directories recursively and a symbolic
1558 link is encountered, the linked-to file is not downloaded. Instead, a
1559 matching symbolic link is created on the local filesystem. The
1560 pointed-to file will not be downloaded unless this recursive retrieval
1561 would have encountered it separately and downloaded it anyway.
1563 When @samp{--retr-symlinks} is specified, however, symbolic links are
1564 traversed and the pointed-to files are retrieved. At this time, this
1565 option does not cause Wget to traverse symlinks to directories and
1566 recurse through them, but in the future it should be enhanced to do
1569 Note that when retrieving a file (not a directory) because it was
1570 specified on the command-line, rather than because it was recursed to,
1571 this option has no effect. Symbolic links are always traversed in this
1574 @cindex Keep-Alive, turning off
1575 @cindex Persistent Connections, disabling
1576 @item --no-http-keep-alive
1577 Turn off the ``keep-alive'' feature for HTTP downloads. Normally, Wget
1578 asks the server to keep the connection open so that, when you download
1579 more than one document from the same server, they get transferred over
1580 the same TCP connection. This saves time and at the same time reduces
1581 the load on the server.
1583 This option is useful when, for some reason, persistent (keep-alive)
1584 connections don't work for you, for example due to a server bug or due
1585 to the inability of server-side scripts to cope with the connections.
1588 @node Recursive Retrieval Options
1589 @section Recursive Retrieval Options
1594 Turn on recursive retrieving. @xref{Recursive Download}, for more
1597 @item -l @var{depth}
1598 @itemx --level=@var{depth}
1599 Specify recursion maximum depth level @var{depth} (@pxref{Recursive
1600 Download}). The default maximum depth is 5.
1602 @cindex proxy filling
1603 @cindex delete after retrieval
1604 @cindex filling proxy cache
1605 @item --delete-after
1606 This option tells Wget to delete every single file it downloads,
1607 @emph{after} having done so. It is useful for pre-fetching popular
1608 pages through a proxy, e.g.:
1611 wget -r -nd --delete-after http://whatever.com/~popular/page/
1614 The @samp{-r} option is to retrieve recursively, and @samp{-nd} to not
1617 Note that @samp{--delete-after} deletes files on the local machine. It
1618 does not issue the @samp{DELE} command to remote FTP sites, for
1619 instance. Also note that when @samp{--delete-after} is specified,
1620 @samp{--convert-links} is ignored, so @samp{.orig} files are simply not
1621 created in the first place.
1623 @cindex conversion of links
1624 @cindex link conversion
1626 @itemx --convert-links
1627 After the download is complete, convert the links in the document to
1628 make them suitable for local viewing. This affects not only the visible
1629 hyperlinks, but any part of the document that links to external content,
1630 such as embedded images, links to style sheets, hyperlinks to non-@sc{html}
1633 Each link will be changed in one of the two ways:
1637 The links to files that have been downloaded by Wget will be changed to
1638 refer to the file they point to as a relative link.
1640 Example: if the downloaded file @file{/foo/doc.html} links to
1641 @file{/bar/img.gif}, also downloaded, then the link in @file{doc.html}
1642 will be modified to point to @samp{../bar/img.gif}. This kind of
1643 transformation works reliably for arbitrary combinations of directories.
1646 The links to files that have not been downloaded by Wget will be changed
1647 to include host name and absolute path of the location they point to.
1649 Example: if the downloaded file @file{/foo/doc.html} links to
1650 @file{/bar/img.gif} (or to @file{../bar/img.gif}), then the link in
1651 @file{doc.html} will be modified to point to
1652 @file{http://@var{hostname}/bar/img.gif}.
1655 Because of this, local browsing works reliably: if a linked file was
1656 downloaded, the link will refer to its local name; if it was not
1657 downloaded, the link will refer to its full Internet address rather than
1658 presenting a broken link. The fact that the former links are converted
1659 to relative links ensures that you can move the downloaded hierarchy to
1662 Note that only at the end of the download can Wget know which links have
1663 been downloaded. Because of that, the work done by @samp{-k} will be
1664 performed at the end of all the downloads.
1666 @cindex backing up converted files
1668 @itemx --backup-converted
1669 When converting a file, back up the original version with a @samp{.orig}
1670 suffix. Affects the behavior of @samp{-N} (@pxref{HTTP Time-Stamping
1675 Turn on options suitable for mirroring. This option turns on recursion
1676 and time-stamping, sets infinite recursion depth and keeps @sc{ftp}
1677 directory listings. It is currently equivalent to
1678 @samp{-r -N -l inf --no-remove-listing}.
1680 @cindex page requisites
1681 @cindex required images, downloading
1683 @itemx --page-requisites
1684 This option causes Wget to download all the files that are necessary to
1685 properly display a given @sc{html} page. This includes such things as
1686 inlined images, sounds, and referenced stylesheets.
1688 Ordinarily, when downloading a single @sc{html} page, any requisite documents
1689 that may be needed to display it properly are not downloaded. Using
1690 @samp{-r} together with @samp{-l} can help, but since Wget does not
1691 ordinarily distinguish between external and inlined documents, one is
1692 generally left with ``leaf documents'' that are missing their
1695 For instance, say document @file{1.html} contains an @code{<IMG>} tag
1696 referencing @file{1.gif} and an @code{<A>} tag pointing to external
1697 document @file{2.html}. Say that @file{2.html} is similar but that its
1698 image is @file{2.gif} and it links to @file{3.html}. Say this
1699 continues up to some arbitrarily high number.
1701 If one executes the command:
1704 wget -r -l 2 http://@var{site}/1.html
1707 then @file{1.html}, @file{1.gif}, @file{2.html}, @file{2.gif}, and
1708 @file{3.html} will be downloaded. As you can see, @file{3.html} is
1709 without its requisite @file{3.gif} because Wget is simply counting the
1710 number of hops (up to 2) away from @file{1.html} in order to determine
1711 where to stop the recursion. However, with this command:
1714 wget -r -l 2 -p http://@var{site}/1.html
1717 all the above files @emph{and} @file{3.html}'s requisite @file{3.gif}
1718 will be downloaded. Similarly,
1721 wget -r -l 1 -p http://@var{site}/1.html
1724 will cause @file{1.html}, @file{1.gif}, @file{2.html}, and @file{2.gif}
1725 to be downloaded. One might think that:
1728 wget -r -l 0 -p http://@var{site}/1.html
1731 would download just @file{1.html} and @file{1.gif}, but unfortunately
1732 this is not the case, because @samp{-l 0} is equivalent to
1733 @samp{-l inf}---that is, infinite recursion. To download a single @sc{html}
1734 page (or a handful of them, all specified on the command-line or in a
1735 @samp{-i} @sc{url} input file) and its (or their) requisites, simply leave off
1736 @samp{-r} and @samp{-l}:
1739 wget -p http://@var{site}/1.html
1742 Note that Wget will behave as if @samp{-r} had been specified, but only
1743 that single page and its requisites will be downloaded. Links from that
1744 page to external documents will not be followed. Actually, to download
1745 a single page and all its requisites (even if they exist on separate
1746 websites), and make sure the lot displays properly locally, this author
1747 likes to use a few options in addition to @samp{-p}:
1750 wget -E -H -k -K -p http://@var{site}/@var{document}
1753 To finish off this topic, it's worth knowing that Wget's idea of an
1754 external document link is any URL specified in an @code{<A>} tag, an
1755 @code{<AREA>} tag, or a @code{<LINK>} tag other than @code{<LINK
1758 @cindex @sc{html} comments
1759 @cindex comments, @sc{html}
1760 @item --strict-comments
1761 Turn on strict parsing of @sc{html} comments. The default is to terminate
1762 comments at the first occurrence of @samp{-->}.
1764 According to specifications, @sc{html} comments are expressed as @sc{sgml}
1765 @dfn{declarations}. Declaration is special markup that begins with
1766 @samp{<!} and ends with @samp{>}, such as @samp{<!DOCTYPE ...>}, that
1767 may contain comments between a pair of @samp{--} delimiters. @sc{html}
1768 comments are ``empty declarations'', @sc{sgml} declarations without any
1769 non-comment text. Therefore, @samp{<!--foo-->} is a valid comment, and
1770 so is @samp{<!--one-- --two-->}, but @samp{<!--1--2-->} is not.
1772 On the other hand, most @sc{html} writers don't perceive comments as anything
1773 other than text delimited with @samp{<!--} and @samp{-->}, which is not
1774 quite the same. For example, something like @samp{<!------------>}
1775 works as a valid comment as long as the number of dashes is a multiple
1776 of four (!). If not, the comment technically lasts until the next
1777 @samp{--}, which may be at the other end of the document. Because of
1778 this, many popular browsers completely ignore the specification and
1779 implement what users have come to expect: comments delimited with
1780 @samp{<!--} and @samp{-->}.
1782 Until version 1.9, Wget interpreted comments strictly, which resulted in
1783 missing links in many web pages that displayed fine in browsers, but had
1784 the misfortune of containing non-compliant comments. Beginning with
1785 version 1.9, Wget has joined the ranks of clients that implements
1786 ``naive'' comments, terminating each comment at the first occurrence of
1789 If, for whatever reason, you want strict comment parsing, use this
1790 option to turn it on.
1793 @node Recursive Accept/Reject Options
1794 @section Recursive Accept/Reject Options
1797 @item -A @var{acclist} --accept @var{acclist}
1798 @itemx -R @var{rejlist} --reject @var{rejlist}
1799 Specify comma-separated lists of file name suffixes or patterns to
1800 accept or reject (@pxref{Types of Files} for more details).
1802 @item -D @var{domain-list}
1803 @itemx --domains=@var{domain-list}
1804 Set domains to be followed. @var{domain-list} is a comma-separated list
1805 of domains. Note that it does @emph{not} turn on @samp{-H}.
1807 @item --exclude-domains @var{domain-list}
1808 Specify the domains that are @emph{not} to be followed.
1809 (@pxref{Spanning Hosts}).
1811 @cindex follow FTP links
1813 Follow @sc{ftp} links from @sc{html} documents. Without this option,
1814 Wget will ignore all the @sc{ftp} links.
1816 @cindex tag-based recursive pruning
1817 @item --follow-tags=@var{list}
1818 Wget has an internal table of @sc{html} tag / attribute pairs that it
1819 considers when looking for linked documents during a recursive
1820 retrieval. If a user wants only a subset of those tags to be
1821 considered, however, he or she should be specify such tags in a
1822 comma-separated @var{list} with this option.
1824 @item --ignore-tags=@var{list}
1825 This is the opposite of the @samp{--follow-tags} option. To skip
1826 certain @sc{html} tags when recursively looking for documents to download,
1827 specify them in a comma-separated @var{list}.
1829 In the past, this option was the best bet for downloading a single page
1830 and its requisites, using a command-line like:
1833 wget --ignore-tags=a,area -H -k -K -r http://@var{site}/@var{document}
1836 However, the author of this option came across a page with tags like
1837 @code{<LINK REL="home" HREF="/">} and came to the realization that
1838 specifying tags to ignore was not enough. One can't just tell Wget to
1839 ignore @code{<LINK>}, because then stylesheets will not be downloaded.
1840 Now the best bet for downloading a single page and its requisites is the
1841 dedicated @samp{--page-requisites} option.
1845 Enable spanning across hosts when doing recursive retrieving
1846 (@pxref{Spanning Hosts}).
1850 Follow relative links only. Useful for retrieving a specific home page
1851 without any distractions, not even those from the same hosts
1852 (@pxref{Relative Links}).
1855 @itemx --include-directories=@var{list}
1856 Specify a comma-separated list of directories you wish to follow when
1857 downloading (@pxref{Directory-Based Limits} for more details.) Elements
1858 of @var{list} may contain wildcards.
1861 @itemx --exclude-directories=@var{list}
1862 Specify a comma-separated list of directories you wish to exclude from
1863 download (@pxref{Directory-Based Limits} for more details.) Elements of
1864 @var{list} may contain wildcards.
1868 Do not ever ascend to the parent directory when retrieving recursively.
1869 This is a useful option, since it guarantees that only the files
1870 @emph{below} a certain hierarchy will be downloaded.
1871 @xref{Directory-Based Limits}, for more details.
1876 @node Recursive Download
1877 @chapter Recursive Download
1880 @cindex recursive download
1882 GNU Wget is capable of traversing parts of the Web (or a single
1883 @sc{http} or @sc{ftp} server), following links and directory structure.
1884 We refer to this as to @dfn{recursive retrieval}, or @dfn{recursion}.
1886 With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} from
1887 the given @sc{url}, documents, retrieving the files the @sc{html}
1888 document was referring to, through markup like @code{href}, or
1889 @code{src}. If the freshly downloaded file is also of type
1890 @code{text/html} or @code{application/xhtml+xml}, it will be parsed and
1893 Recursive retrieval of @sc{http} and @sc{html} content is
1894 @dfn{breadth-first}. This means that Wget first downloads the requested
1895 @sc{html} document, then the documents linked from that document, then the
1896 documents linked by them, and so on. In other words, Wget first
1897 downloads the documents at depth 1, then those at depth 2, and so on
1898 until the specified maximum depth.
1900 The maximum @dfn{depth} to which the retrieval may descend is specified
1901 with the @samp{-l} option. The default maximum depth is five layers.
1903 When retrieving an @sc{ftp} @sc{url} recursively, Wget will retrieve all
1904 the data from the given directory tree (including the subdirectories up
1905 to the specified depth) on the remote server, creating its mirror image
1906 locally. @sc{ftp} retrieval is also limited by the @code{depth}
1907 parameter. Unlike @sc{http} recursion, @sc{ftp} recursion is performed
1910 By default, Wget will create a local directory tree, corresponding to
1911 the one found on the remote server.
1913 Recursive retrieving can find a number of applications, the most
1914 important of which is mirroring. It is also useful for @sc{www}
1915 presentations, and any other opportunities where slow network
1916 connections should be bypassed by storing the files locally.
1918 You should be warned that recursive downloads can overload the remote
1919 servers. Because of that, many administrators frown upon them and may
1920 ban access from your site if they detect very fast downloads of big
1921 amounts of content. When downloading from Internet servers, consider
1922 using the @samp{-w} option to introduce a delay between accesses to the
1923 server. The download will take a while longer, but the server
1924 administrator will not be alarmed by your rudeness.
1926 Of course, recursive download may cause problems on your machine. If
1927 left to run unchecked, it can easily fill up the disk. If downloading
1928 from local network, it can also take bandwidth on the system, as well as
1929 consume memory and CPU.
1931 Try to specify the criteria that match the kind of download you are
1932 trying to achieve. If you want to download only one page, use
1933 @samp{--page-requisites} without any additional recursion. If you want
1934 to download things under one directory, use @samp{-np} to avoid
1935 downloading things from other directories. If you want to download all
1936 the files from one directory, use @samp{-l 1} to make sure the recursion
1937 depth never exceeds one. @xref{Following Links}, for more information
1940 Recursive retrieval should be used with care. Don't say you were not
1943 @node Following Links
1944 @chapter Following Links
1946 @cindex following links
1948 When retrieving recursively, one does not wish to retrieve loads of
1949 unnecessary data. Most of the time the users bear in mind exactly what
1950 they want to download, and want Wget to follow only specific links.
1952 For example, if you wish to download the music archive from
1953 @samp{fly.srk.fer.hr}, you will not want to download all the home pages
1954 that happen to be referenced by an obscure part of the archive.
1956 Wget possesses several mechanisms that allows you to fine-tune which
1957 links it will follow.
1960 * Spanning Hosts:: (Un)limiting retrieval based on host name.
1961 * Types of Files:: Getting only certain files.
1962 * Directory-Based Limits:: Getting only certain directories.
1963 * Relative Links:: Follow relative links only.
1964 * FTP Links:: Following FTP links.
1967 @node Spanning Hosts
1968 @section Spanning Hosts
1969 @cindex spanning hosts
1970 @cindex hosts, spanning
1972 Wget's recursive retrieval normally refuses to visit hosts different
1973 than the one you specified on the command line. This is a reasonable
1974 default; without it, every retrieval would have the potential to turn
1975 your Wget into a small version of google.
1977 However, visiting different hosts, or @dfn{host spanning,} is sometimes
1978 a useful option. Maybe the images are served from a different server.
1979 Maybe you're mirroring a site that consists of pages interlinked between
1980 three servers. Maybe the server has two equivalent names, and the @sc{html}
1981 pages refer to both interchangeably.
1984 @item Span to any host---@samp{-H}
1986 The @samp{-H} option turns on host spanning, thus allowing Wget's
1987 recursive run to visit any host referenced by a link. Unless sufficient
1988 recursion-limiting criteria are applied depth, these foreign hosts will
1989 typically link to yet more hosts, and so on until Wget ends up sucking
1990 up much more data than you have intended.
1992 @item Limit spanning to certain domains---@samp{-D}
1994 The @samp{-D} option allows you to specify the domains that will be
1995 followed, thus limiting the recursion only to the hosts that belong to
1996 these domains. Obviously, this makes sense only in conjunction with
1997 @samp{-H}. A typical example would be downloading the contents of
1998 @samp{www.server.com}, but allowing downloads from
1999 @samp{images.server.com}, etc.:
2002 wget -rH -Dserver.com http://www.server.com/
2005 You can specify more than one address by separating them with a comma,
2006 e.g. @samp{-Ddomain1.com,domain2.com}.
2008 @item Keep download off certain domains---@samp{--exclude-domains}
2010 If there are domains you want to exclude specifically, you can do it
2011 with @samp{--exclude-domains}, which accepts the same type of arguments
2012 of @samp{-D}, but will @emph{exclude} all the listed domains. For
2013 example, if you want to download all the hosts from @samp{foo.edu}
2014 domain, with the exception of @samp{sunsite.foo.edu}, you can do it like
2018 wget -rH -Dfoo.edu --exclude-domains sunsite.foo.edu \
2024 @node Types of Files
2025 @section Types of Files
2026 @cindex types of files
2028 When downloading material from the web, you will often want to restrict
2029 the retrieval to only certain file types. For example, if you are
2030 interested in downloading @sc{gif}s, you will not be overjoyed to get
2031 loads of PostScript documents, and vice versa.
2033 Wget offers two options to deal with this problem. Each option
2034 description lists a short name, a long name, and the equivalent command
2037 @cindex accept wildcards
2038 @cindex accept suffixes
2039 @cindex wildcards, accept
2040 @cindex suffixes, accept
2042 @item -A @var{acclist}
2043 @itemx --accept @var{acclist}
2044 @itemx accept = @var{acclist}
2045 The argument to @samp{--accept} option is a list of file suffixes or
2046 patterns that Wget will download during recursive retrieval. A suffix
2047 is the ending part of a file, and consists of ``normal'' letters,
2048 e.g. @samp{gif} or @samp{.jpg}. A matching pattern contains shell-like
2049 wildcards, e.g. @samp{books*} or @samp{zelazny*196[0-9]*}.
2051 So, specifying @samp{wget -A gif,jpg} will make Wget download only the
2052 files ending with @samp{gif} or @samp{jpg}, i.e. @sc{gif}s and
2053 @sc{jpeg}s. On the other hand, @samp{wget -A "zelazny*196[0-9]*"} will
2054 download only files beginning with @samp{zelazny} and containing numbers
2055 from 1960 to 1969 anywhere within. Look up the manual of your shell for
2056 a description of how pattern matching works.
2058 Of course, any number of suffixes and patterns can be combined into a
2059 comma-separated list, and given as an argument to @samp{-A}.
2061 @cindex reject wildcards
2062 @cindex reject suffixes
2063 @cindex wildcards, reject
2064 @cindex suffixes, reject
2065 @item -R @var{rejlist}
2066 @itemx --reject @var{rejlist}
2067 @itemx reject = @var{rejlist}
2068 The @samp{--reject} option works the same way as @samp{--accept}, only
2069 its logic is the reverse; Wget will download all files @emph{except} the
2070 ones matching the suffixes (or patterns) in the list.
2072 So, if you want to download a whole page except for the cumbersome
2073 @sc{mpeg}s and @sc{.au} files, you can use @samp{wget -R mpg,mpeg,au}.
2074 Analogously, to download all files except the ones beginning with
2075 @samp{bjork}, use @samp{wget -R "bjork*"}. The quotes are to prevent
2076 expansion by the shell.
2079 The @samp{-A} and @samp{-R} options may be combined to achieve even
2080 better fine-tuning of which files to retrieve. E.g. @samp{wget -A
2081 "*zelazny*" -R .ps} will download all the files having @samp{zelazny} as
2082 a part of their name, but @emph{not} the PostScript files.
2084 Note that these two options do not affect the downloading of @sc{html}
2085 files; Wget must load all the @sc{html}s to know where to go at
2086 all---recursive retrieval would make no sense otherwise.
2088 @node Directory-Based Limits
2089 @section Directory-Based Limits
2091 @cindex directory limits
2093 Regardless of other link-following facilities, it is often useful to
2094 place the restriction of what files to retrieve based on the directories
2095 those files are placed in. There can be many reasons for this---the
2096 home pages may be organized in a reasonable directory structure; or some
2097 directories may contain useless information, e.g. @file{/cgi-bin} or
2098 @file{/dev} directories.
2100 Wget offers three different options to deal with this requirement. Each
2101 option description lists a short name, a long name, and the equivalent
2102 command in @file{.wgetrc}.
2104 @cindex directories, include
2105 @cindex include directories
2106 @cindex accept directories
2109 @itemx --include @var{list}
2110 @itemx include_directories = @var{list}
2111 @samp{-I} option accepts a comma-separated list of directories included
2112 in the retrieval. Any other directories will simply be ignored. The
2113 directories are absolute paths.
2115 So, if you wish to download from @samp{http://host/people/bozo/}
2116 following only links to bozo's colleagues in the @file{/people}
2117 directory and the bogus scripts in @file{/cgi-bin}, you can specify:
2120 wget -I /people,/cgi-bin http://host/people/bozo/
2123 @cindex directories, exclude
2124 @cindex exclude directories
2125 @cindex reject directories
2127 @itemx --exclude @var{list}
2128 @itemx exclude_directories = @var{list}
2129 @samp{-X} option is exactly the reverse of @samp{-I}---this is a list of
2130 directories @emph{excluded} from the download. E.g. if you do not want
2131 Wget to download things from @file{/cgi-bin} directory, specify @samp{-X
2132 /cgi-bin} on the command line.
2134 The same as with @samp{-A}/@samp{-R}, these two options can be combined
2135 to get a better fine-tuning of downloading subdirectories. E.g. if you
2136 want to load all the files from @file{/pub} hierarchy except for
2137 @file{/pub/worthless}, specify @samp{-I/pub -X/pub/worthless}.
2142 @itemx no_parent = on
2143 The simplest, and often very useful way of limiting directories is
2144 disallowing retrieval of the links that refer to the hierarchy
2145 @dfn{above} than the beginning directory, i.e. disallowing ascent to the
2146 parent directory/directories.
2148 The @samp{--no-parent} option (short @samp{-np}) is useful in this case.
2149 Using it guarantees that you will never leave the existing hierarchy.
2150 Supposing you issue Wget with:
2153 wget -r --no-parent http://somehost/~luzer/my-archive/
2156 You may rest assured that none of the references to
2157 @file{/~his-girls-homepage/} or @file{/~luzer/all-my-mpegs/} will be
2158 followed. Only the archive you are interested in will be downloaded.
2159 Essentially, @samp{--no-parent} is similar to
2160 @samp{-I/~luzer/my-archive}, only it handles redirections in a more
2161 intelligent fashion.
2164 @node Relative Links
2165 @section Relative Links
2166 @cindex relative links
2168 When @samp{-L} is turned on, only the relative links are ever followed.
2169 Relative links are here defined those that do not refer to the web
2170 server root. For example, these links are relative:
2174 <a href="foo/bar.gif">
2175 <a href="../foo/bar.gif">
2178 These links are not relative:
2182 <a href="/foo/bar.gif">
2183 <a href="http://www.server.com/foo/bar.gif">
2186 Using this option guarantees that recursive retrieval will not span
2187 hosts, even without @samp{-H}. In simple cases it also allows downloads
2188 to ``just work'' without having to convert links.
2190 This option is probably not very useful and might be removed in a future
2194 @section Following FTP Links
2195 @cindex following ftp links
2197 The rules for @sc{ftp} are somewhat specific, as it is necessary for
2198 them to be. @sc{ftp} links in @sc{html} documents are often included
2199 for purposes of reference, and it is often inconvenient to download them
2202 To have @sc{ftp} links followed from @sc{html} documents, you need to
2203 specify the @samp{--follow-ftp} option. Having done that, @sc{ftp}
2204 links will span hosts regardless of @samp{-H} setting. This is logical,
2205 as @sc{ftp} links rarely point to the same host where the @sc{http}
2206 server resides. For similar reasons, the @samp{-L} options has no
2207 effect on such downloads. On the other hand, domain acceptance
2208 (@samp{-D}) and suffix rules (@samp{-A} and @samp{-R}) apply normally.
2210 Also note that followed links to @sc{ftp} directories will not be
2211 retrieved recursively further.
2214 @chapter Time-Stamping
2215 @cindex time-stamping
2216 @cindex timestamping
2217 @cindex updating the archives
2218 @cindex incremental updating
2220 One of the most important aspects of mirroring information from the
2221 Internet is updating your archives.
2223 Downloading the whole archive again and again, just to replace a few
2224 changed files is expensive, both in terms of wasted bandwidth and money,
2225 and the time to do the update. This is why all the mirroring tools
2226 offer the option of incremental updating.
2228 Such an updating mechanism means that the remote server is scanned in
2229 search of @dfn{new} files. Only those new files will be downloaded in
2230 the place of the old ones.
2232 A file is considered new if one of these two conditions are met:
2236 A file of that name does not already exist locally.
2239 A file of that name does exist, but the remote file was modified more
2240 recently than the local file.
2243 To implement this, the program needs to be aware of the time of last
2244 modification of both local and remote files. We call this information the
2245 @dfn{time-stamp} of a file.
2247 The time-stamping in GNU Wget is turned on using @samp{--timestamping}
2248 (@samp{-N}) option, or through @code{timestamping = on} directive in
2249 @file{.wgetrc}. With this option, for each file it intends to download,
2250 Wget will check whether a local file of the same name exists. If it
2251 does, and the remote file is older, Wget will not download it.
2253 If the local file does not exist, or the sizes of the files do not
2254 match, Wget will download the remote file no matter what the time-stamps
2258 * Time-Stamping Usage::
2259 * HTTP Time-Stamping Internals::
2260 * FTP Time-Stamping Internals::
2263 @node Time-Stamping Usage
2264 @section Time-Stamping Usage
2265 @cindex time-stamping usage
2266 @cindex usage, time-stamping
2268 The usage of time-stamping is simple. Say you would like to download a
2269 file so that it keeps its date of modification.
2272 wget -S http://www.gnu.ai.mit.edu/
2275 A simple @code{ls -l} shows that the time stamp on the local file equals
2276 the state of the @code{Last-Modified} header, as returned by the server.
2277 As you can see, the time-stamping info is preserved locally, even
2278 without @samp{-N} (at least for @sc{http}).
2280 Several days later, you would like Wget to check if the remote file has
2281 changed, and download it if it has.
2284 wget -N http://www.gnu.ai.mit.edu/
2287 Wget will ask the server for the last-modified date. If the local file
2288 has the same timestamp as the server, or a newer one, the remote file
2289 will not be re-fetched. However, if the remote file is more recent,
2290 Wget will proceed to fetch it.
2292 The same goes for @sc{ftp}. For example:
2295 wget "ftp://ftp.ifi.uio.no/pub/emacs/gnus/*"
2298 (The quotes around that URL are to prevent the shell from trying to
2299 interpret the @samp{*}.)
2301 After download, a local directory listing will show that the timestamps
2302 match those on the remote server. Reissuing the command with @samp{-N}
2303 will make Wget re-fetch @emph{only} the files that have been modified
2304 since the last download.
2306 If you wished to mirror the GNU archive every week, you would use a
2307 command like the following, weekly:
2310 wget --timestamping -r ftp://ftp.gnu.org/pub/gnu/
2313 Note that time-stamping will only work for files for which the server
2314 gives a timestamp. For @sc{http}, this depends on getting a
2315 @code{Last-Modified} header. For @sc{ftp}, this depends on getting a
2316 directory listing with dates in a format that Wget can parse
2317 (@pxref{FTP Time-Stamping Internals}).
2319 @node HTTP Time-Stamping Internals
2320 @section HTTP Time-Stamping Internals
2321 @cindex http time-stamping
2323 Time-stamping in @sc{http} is implemented by checking of the
2324 @code{Last-Modified} header. If you wish to retrieve the file
2325 @file{foo.html} through @sc{http}, Wget will check whether
2326 @file{foo.html} exists locally. If it doesn't, @file{foo.html} will be
2327 retrieved unconditionally.
2329 If the file does exist locally, Wget will first check its local
2330 time-stamp (similar to the way @code{ls -l} checks it), and then send a
2331 @code{HEAD} request to the remote server, demanding the information on
2334 The @code{Last-Modified} header is examined to find which file was
2335 modified more recently (which makes it ``newer''). If the remote file
2336 is newer, it will be downloaded; if it is older, Wget will give
2337 up.@footnote{As an additional check, Wget will look at the
2338 @code{Content-Length} header, and compare the sizes; if they are not the
2339 same, the remote file will be downloaded no matter what the time-stamp
2342 When @samp{--backup-converted} (@samp{-K}) is specified in conjunction
2343 with @samp{-N}, server file @samp{@var{X}} is compared to local file
2344 @samp{@var{X}.orig}, if extant, rather than being compared to local file
2345 @samp{@var{X}}, which will always differ if it's been converted by
2346 @samp{--convert-links} (@samp{-k}).
2348 Arguably, @sc{http} time-stamping should be implemented using the
2349 @code{If-Modified-Since} request.
2351 @node FTP Time-Stamping Internals
2352 @section FTP Time-Stamping Internals
2353 @cindex ftp time-stamping
2355 In theory, @sc{ftp} time-stamping works much the same as @sc{http}, only
2356 @sc{ftp} has no headers---time-stamps must be ferreted out of directory
2359 If an @sc{ftp} download is recursive or uses globbing, Wget will use the
2360 @sc{ftp} @code{LIST} command to get a file listing for the directory
2361 containing the desired file(s). It will try to analyze the listing,
2362 treating it like Unix @code{ls -l} output, extracting the time-stamps.
2363 The rest is exactly the same as for @sc{http}. Note that when
2364 retrieving individual files from an @sc{ftp} server without using
2365 globbing or recursion, listing files will not be downloaded (and thus
2366 files will not be time-stamped) unless @samp{-N} is specified.
2368 Assumption that every directory listing is a Unix-style listing may
2369 sound extremely constraining, but in practice it is not, as many
2370 non-Unix @sc{ftp} servers use the Unixoid listing format because most
2371 (all?) of the clients understand it. Bear in mind that @sc{rfc959}
2372 defines no standard way to get a file list, let alone the time-stamps.
2373 We can only hope that a future standard will define this.
2375 Another non-standard solution includes the use of @code{MDTM} command
2376 that is supported by some @sc{ftp} servers (including the popular
2377 @code{wu-ftpd}), which returns the exact time of the specified file.
2378 Wget may support this command in the future.
2381 @chapter Startup File
2382 @cindex startup file
2388 Once you know how to change default settings of Wget through command
2389 line arguments, you may wish to make some of those settings permanent.
2390 You can do that in a convenient way by creating the Wget startup
2391 file---@file{.wgetrc}.
2393 Besides @file{.wgetrc} is the ``main'' initialization file, it is
2394 convenient to have a special facility for storing passwords. Thus Wget
2395 reads and interprets the contents of @file{$HOME/.netrc}, if it finds
2396 it. You can find @file{.netrc} format in your system manuals.
2398 Wget reads @file{.wgetrc} upon startup, recognizing a limited set of
2402 * Wgetrc Location:: Location of various wgetrc files.
2403 * Wgetrc Syntax:: Syntax of wgetrc.
2404 * Wgetrc Commands:: List of available commands.
2405 * Sample Wgetrc:: A wgetrc example.
2408 @node Wgetrc Location
2409 @section Wgetrc Location
2410 @cindex wgetrc location
2411 @cindex location of wgetrc
2413 When initializing, Wget will look for a @dfn{global} startup file,
2414 @file{/usr/local/etc/wgetrc} by default (or some prefix other than
2415 @file{/usr/local}, if Wget was not installed there) and read commands
2416 from there, if it exists.
2418 Then it will look for the user's file. If the environmental variable
2419 @code{WGETRC} is set, Wget will try to load that file. Failing that, no
2420 further attempts will be made.
2422 If @code{WGETRC} is not set, Wget will try to load @file{$HOME/.wgetrc}.
2424 The fact that user's settings are loaded after the system-wide ones
2425 means that in case of collision user's wgetrc @emph{overrides} the
2426 system-wide wgetrc (in @file{/usr/local/etc/wgetrc} by default).
2427 Fascist admins, away!
2430 @section Wgetrc Syntax
2431 @cindex wgetrc syntax
2432 @cindex syntax of wgetrc
2434 The syntax of a wgetrc command is simple:
2440 The @dfn{variable} will also be called @dfn{command}. Valid
2441 @dfn{values} are different for different commands.
2443 The commands are case-insensitive and underscore-insensitive. Thus
2444 @samp{DIr__PrefiX} is the same as @samp{dirprefix}. Empty lines, lines
2445 beginning with @samp{#} and lines containing white-space only are
2448 Commands that expect a comma-separated list will clear the list on an
2449 empty command. So, if you wish to reset the rejection list specified in
2450 global @file{wgetrc}, you can do it with:
2456 @node Wgetrc Commands
2457 @section Wgetrc Commands
2458 @cindex wgetrc commands
2460 The complete set of commands is listed below. Legal values are listed
2461 after the @samp{=}. Simple Boolean values can be set or unset using
2462 @samp{on} and @samp{off} or @samp{1} and @samp{0}.
2464 Some commands take pseudo-arbitrary values. @var{address} values can be
2465 hostnames or dotted-quad IP addresses. @var{n} can be any positive
2466 integer, or @samp{inf} for infinity, where appropriate. @var{string}
2467 values can be any non-empty string.
2469 Most of these commands have direct command-line equivalents. Also, any
2470 wgetrc command can be specified on the command line using the
2471 @samp{--execute} switch (@pxref{Basic Startup Options}.)
2474 @item accept/reject = @var{string}
2475 Same as @samp{-A}/@samp{-R} (@pxref{Types of Files}).
2477 @item add_hostdir = on/off
2478 Enable/disable host-prefixed file names. @samp{-nH} disables it.
2480 @item continue = on/off
2481 If set to on, force continuation of preexistent partially retrieved
2482 files. See @samp{-c} before setting it.
2484 @item background = on/off
2485 Enable/disable going to background---the same as @samp{-b} (which
2488 @item backup_converted = on/off
2489 Enable/disable saving pre-converted files with the suffix
2490 @samp{.orig}---the same as @samp{-K} (which enables it).
2492 @c @item backups = @var{number}
2493 @c #### Document me!
2495 @item base = @var{string}
2496 Consider relative @sc{url}s in @sc{url} input files forced to be
2497 interpreted as @sc{html} as being relative to @var{string}---the same as
2498 @samp{--base=@var{string}}.
2500 @item bind_address = @var{address}
2501 Bind to @var{address}, like the @samp{--bind-address=@var{address}}.
2503 @item ca_certificate = @var{file}
2504 Set the certificate authority bundle file to @var{file}. The same
2505 as @samp{--ca-certificate=@var{file}}.
2507 @item ca_directory = @var{directory}
2508 Set the directory used for certificate authorities. The same as
2509 @samp{--ca-directory=@var{directory}}.
2511 @item cache = on/off
2512 When set to off, disallow server-caching. See the @samp{--no-cache}
2515 @item certificate = @var{file}
2516 Set the client certificate file name to @var{file}. The same as
2517 @samp{--certificate=@var{file}}.
2519 @item certificate_type = @var{string}
2520 Specify the type of the client certificate, legal values being
2521 @samp{PEM} (the default) and @samp{DER} (aka ASN1). The same as
2522 @samp{--certificate-type=@var{string}}.
2524 @item check_certificate = on/off
2525 If this is set to off, the server certificate is not checked against
2526 the specified client authorities. The default is ``on''. The same as
2527 @samp{--check-certificate}.
2529 @item convert_links = on/off
2530 Convert non-relative links locally. The same as @samp{-k}.
2532 @item cookies = on/off
2533 When set to off, disallow cookies. See the @samp{--cookies} option.
2535 @item connect_timeout = @var{n}
2536 Set the connect timeout---the same as @samp{--connect-timeout}.
2538 @item cut_dirs = @var{n}
2539 Ignore @var{n} remote directory components. Equivalent to
2540 @samp{--cut-dirs=@var{n}}.
2542 @item debug = on/off
2543 Debug mode, same as @samp{-d}.
2545 @item delete_after = on/off
2546 Delete after download---the same as @samp{--delete-after}.
2548 @item dir_prefix = @var{string}
2549 Top of directory tree---the same as @samp{-P @var{string}}.
2551 @item dirstruct = on/off
2552 Turning dirstruct on or off---the same as @samp{-x} or @samp{-nd},
2555 @item dns_cache = on/off
2556 Turn DNS caching on/off. Since DNS caching is on by default, this
2557 option is normally used to turn it off and is equivalent to
2558 @samp{--no-dns-cache}.
2560 @item dns_timeout = @var{n}
2561 Set the DNS timeout---the same as @samp{--dns-timeout}.
2563 @item domains = @var{string}
2564 Same as @samp{-D} (@pxref{Spanning Hosts}).
2566 @item dot_bytes = @var{n}
2567 Specify the number of bytes ``contained'' in a dot, as seen throughout
2568 the retrieval (1024 by default). You can postfix the value with
2569 @samp{k} or @samp{m}, representing kilobytes and megabytes,
2570 respectively. With dot settings you can tailor the dot retrieval to
2571 suit your needs, or you can use the predefined @dfn{styles}
2572 (@pxref{Download Options}).
2574 @item dots_in_line = @var{n}
2575 Specify the number of dots that will be printed in each line throughout
2576 the retrieval (50 by default).
2578 @item dot_spacing = @var{n}
2579 Specify the number of dots in a single cluster (10 by default).
2581 @item egd_file = @var{file}
2582 Use @var{string} as the EGD socket file name. The same as
2583 @samp{--egd-file=@var{file}}.
2585 @item exclude_directories = @var{string}
2586 Specify a comma-separated list of directories you wish to exclude from
2587 download---the same as @samp{-X @var{string}} (@pxref{Directory-Based
2590 @item exclude_domains = @var{string}
2591 Same as @samp{--exclude-domains=@var{string}} (@pxref{Spanning
2594 @item follow_ftp = on/off
2595 Follow @sc{ftp} links from @sc{html} documents---the same as
2596 @samp{--follow-ftp}.
2598 @item follow_tags = @var{string}
2599 Only follow certain @sc{html} tags when doing a recursive retrieval,
2600 just like @samp{--follow-tags=@var{string}}.
2602 @item force_html = on/off
2603 If set to on, force the input filename to be regarded as an @sc{html}
2604 document---the same as @samp{-F}.
2606 @item ftp_password = @var{string}
2607 Set your @sc{ftp} password to @var{string}. Without this setting, the
2608 password defaults to @samp{-wget@@}, which is a useful default for
2609 anonymous @sc{ftp} access.
2611 This command used to be named @code{passwd} prior to Wget 1.10.
2613 @item ftp_proxy = @var{string}
2614 Use @var{string} as @sc{ftp} proxy, instead of the one specified in
2617 @item ftp_user = @var{string}
2618 Set @sc{ftp} user to @var{string}.
2620 This command used to be named @code{login} prior to Wget 1.10.
2623 Turn globbing on/off---the same as @samp{--glob} and @samp{--no-glob}.
2625 @item header = @var{string}
2626 Define a header for HTTP doewnloads, like using
2627 @samp{--header=@var{string}}.
2629 @item html_extension = on/off
2630 Add a @samp{.html} extension to @samp{text/html} or
2631 @samp{application/xhtml+xml} files without it, like @samp{-E}.
2633 @item http_keep_alive = on/off
2634 Turn the keep-alive feature on or off (defaults to on). Turning it
2635 off is equivalent to @samp{--no-http-keep-alive}.
2637 @item http_password = @var{string}
2638 Set @sc{http} password, equivalent to
2639 @samp{--http-password=@var{string}}.
2641 @item http_proxy = @var{string}
2642 Use @var{string} as @sc{http} proxy, instead of the one specified in
2645 @item http_user = @var{string}
2646 Set @sc{http} user to @var{string}, equivalent to
2647 @samp{--http-user=@var{string}}.
2649 @item ignore_length = on/off
2650 When set to on, ignore @code{Content-Length} header; the same as
2651 @samp{--ignore-length}.
2653 @item ignore_tags = @var{string}
2654 Ignore certain @sc{html} tags when doing a recursive retrieval, like
2655 @samp{--ignore-tags=@var{string}}.
2657 @item include_directories = @var{string}
2658 Specify a comma-separated list of directories you wish to follow when
2659 downloading---the same as @samp{-I @var{string}}.
2661 @item inet4_only = on/off
2662 Force connecting to IPv4 addresses, off by default. You can put this
2663 in the global init file to disable Wget's attempts to resolve and
2664 connect to IPv6 hosts. Available only if Wget was compiled with IPv6
2665 support. The same as @samp{--inet4-only} or @samp{-4}.
2667 @item inet6_only = on/off
2668 Force connecting to IPv6 addresses, off by default. Available only if
2669 Wget was compiled with IPv6 support. The same as @samp{--inet6-only}
2672 @item input = @var{file}
2673 Read the @sc{url}s from @var{string}, like @samp{-i @var{file}}.
2675 @item kill_longer = on/off
2676 Consider data longer than specified in content-length header as invalid
2677 (and retry getting it). The default behavior is to save as much data
2678 as there is, provided there is more than or equal to the value in
2679 @code{Content-Length}.
2681 @item limit_rate = @var{rate}
2682 Limit the download speed to no more than @var{rate} bytes per second.
2683 The same as @samp{--limit-rate=@var{rate}}.
2685 @item load_cookies = @var{file}
2686 Load cookies from @var{file}. See @samp{--load-cookies @var{file}}.
2688 @item logfile = @var{file}
2689 Set logfile to @var{file}, the same as @samp{-o @var{file}}.
2691 @item mirror = on/off
2692 Turn mirroring on/off. The same as @samp{-m}.
2694 @item netrc = on/off
2695 Turn reading netrc on or off.
2697 @item noclobber = on/off
2700 @item no_parent = on/off
2701 Disallow retrieving outside the directory hierarchy, like
2702 @samp{--no-parent} (@pxref{Directory-Based Limits}).
2704 @item no_proxy = @var{string}
2705 Use @var{string} as the comma-separated list of domains to avoid in
2706 proxy loading, instead of the one specified in environment.
2708 @item output_document = @var{file}
2709 Set the output filename---the same as @samp{-O @var{file}}.
2711 @item page_requisites = on/off
2712 Download all ancillary documents necessary for a single @sc{html} page to
2713 display properly---the same as @samp{-p}.
2715 @item passive_ftp = on/off
2716 Change setting of passive @sc{ftp}, equivalent to the
2717 @samp{--passive-ftp} option.
2719 @itemx password = @var{string}
2720 Specify password @var{string} for both @sc{ftp} and @sc{http} file retrieval.
2721 This command can be overridden using the @samp{ftp_password} and
2722 @samp{http_password} command for @sc{ftp} and @sc{http} respectively.
2724 @item post_data = @var{string}
2725 Use POST as the method for all HTTP requests and send @var{string} in
2726 the request body. The same as @samp{--post-data=@var{string}}.
2728 @item post_file = @var{file}
2729 Use POST as the method for all HTTP requests and send the contents of
2730 @var{file} in the request body. The same as
2731 @samp{--post-file=@var{file}}.
2733 @item prefer_family = IPv4/IPv6/none
2734 When given a choice of several addresses, connect to the addresses
2735 with specified address family first. IPv4 addresses are preferred by
2736 default. The same as @samp{--prefer-family}, which see for a detailed
2737 discussion of why this is useful.
2739 @item private_key = @var{file}
2740 Set the private key file to @var{file}. The same as
2741 @samp{--private-key=@var{file}}.
2743 @item private_key_type = @var{string}
2744 Specify the type of the private key, legal values being @samp{PEM}
2745 (the default) and @samp{DER} (aka ASN1). The same as
2746 @samp{--private-type=@var{string}}.
2748 @item progress = @var{string}
2749 Set the type of the progress indicator. Legal types are @samp{dot}
2750 and @samp{bar}. Equivalent to @samp{--progress=@var{string}}.
2752 @item protocol_directories = on/off
2753 When set, use the protocol name as a directory component of local file
2754 names. The same as @samp{--protocol-directories}.
2756 @item proxy_user = @var{string}
2757 Set proxy authentication user name to @var{string}, like
2758 @samp{--proxy-user=@var{string}}.
2760 @item proxy_password = @var{string}
2761 Set proxy authentication password to @var{string}, like
2762 @samp{--proxy-password=@var{string}}.
2764 @item quiet = on/off
2765 Quiet mode---the same as @samp{-q}.
2767 @item quota = @var{quota}
2768 Specify the download quota, which is useful to put in the global
2769 @file{wgetrc}. When download quota is specified, Wget will stop
2770 retrieving after the download sum has become greater than quota. The
2771 quota can be specified in bytes (default), kbytes @samp{k} appended) or
2772 mbytes (@samp{m} appended). Thus @samp{quota = 5m} will set the quota
2773 to 5 megabytes. Note that the user's startup file overrides system
2776 @item random_file = @var{file}
2777 Use @var{file} as a source of randomness on systems lacking
2780 @item read_timeout = @var{n}
2781 Set the read (and write) timeout---the same as
2782 @samp{--read-timeout=@var{n}}.
2784 @item reclevel = @var{n}
2785 Recursion level (depth)---the same as @samp{-l @var{n}}.
2787 @item recursive = on/off
2788 Recursive on/off---the same as @samp{-r}.
2790 @item referer = @var{string}
2791 Set HTTP @samp{Referer:} header just like
2792 @samp{--referer=@var{string}}. (Note it was the folks who wrote the
2793 @sc{http} spec who got the spelling of ``referrer'' wrong.)
2795 @item relative_only = on/off
2796 Follow only relative links---the same as @samp{-L} (@pxref{Relative
2799 @item remove_listing = on/off
2800 If set to on, remove @sc{ftp} listings downloaded by Wget. Setting it
2801 to off is the same as @samp{--no-remove-listing}.
2803 @item restrict_file_names = unix/windows
2804 Restrict the file names generated by Wget from URLs. See
2805 @samp{--restrict-file-names} for a more detailed description.
2807 @item retr_symlinks = on/off
2808 When set to on, retrieve symbolic links as if they were plain files; the
2809 same as @samp{--retr-symlinks}.
2811 @item retry_connrefused = on/off
2812 When set to on, consider ``connection refused'' a transient
2813 error---the same as @samp{--retry-connrefused}.
2815 @item robots = on/off
2816 Specify whether the norobots convention is respected by Wget, ``on'' by
2817 default. This switch controls both the @file{/robots.txt} and the
2818 @samp{nofollow} aspect of the spec. @xref{Robot Exclusion}, for more
2819 details about this. Be sure you know what you are doing before turning
2822 @item save_cookies = @var{file}
2823 Save cookies to @var{file}. The same as @samp{--save-cookies
2826 @item secure_protocol = @var{string}
2827 Choose the secure protocol to be used. Legal values are @samp{auto}
2828 (the default), @samp{SSLv2}, @samp{SSLv3}, and @samp{TLSv1}. The same
2829 as @samp{--secure-protocol=@var{string}}.
2831 @item server_response = on/off
2832 Choose whether or not to print the @sc{http} and @sc{ftp} server
2833 responses---the same as @samp{-S}.
2835 @item span_hosts = on/off
2838 @item strict_comments = on/off
2839 Same as @samp{--strict-comments}.
2841 @item timeout = @var{n}
2842 Set all applicable timeout values to @var{n}, the same as @samp{-T
2845 @item timestamping = on/off
2846 Turn timestamping on/off. The same as @samp{-N} (@pxref{Time-Stamping}).
2848 @item tries = @var{n}
2849 Set number of retries per @sc{url}---the same as @samp{-t @var{n}}.
2851 @item use_proxy = on/off
2852 When set to off, don't use proxy even when proxy-related environment
2853 variables are set. In that case it is the same as using
2856 @item user = @var{string}
2857 Specify username @var{string} for both @sc{ftp} and @sc{http} file retrieval.
2858 This command can be overridden using the @samp{ftp_user} and
2859 @samp{http_user} command for @sc{ftp} and @sc{http} respectively.
2861 @item verbose = on/off
2862 Turn verbose on/off---the same as @samp{-v}/@samp{-nv}.
2864 @item wait = @var{n}
2865 Wait @var{n} seconds between retrievals---the same as @samp{-w
2868 @item waitretry = @var{n}
2869 Wait up to @var{n} seconds between retries of failed retrievals
2870 only---the same as @samp{--waitretry=@var{n}}. Note that this is
2871 turned on by default in the global @file{wgetrc}.
2873 @item randomwait = on/off
2874 Turn random between-request wait times on or off. The same as
2875 @samp{--random-wait}.
2879 @section Sample Wgetrc
2880 @cindex sample wgetrc
2882 This is the sample initialization file, as given in the distribution.
2883 It is divided in two section---one for global usage (suitable for global
2884 startup file), and one for local usage (suitable for
2885 @file{$HOME/.wgetrc}). Be careful about the things you change.
2887 Note that almost all the lines are commented out. For a command to have
2888 any effect, you must remove the @samp{#} character at the beginning of
2892 @include sample.wgetrc.munged_for_texi_inclusion
2899 @c man begin EXAMPLES
2900 The examples are divided into three sections loosely based on their
2904 * Simple Usage:: Simple, basic usage of the program.
2905 * Advanced Usage:: Advanced tips.
2906 * Very Advanced Usage:: The hairy stuff.
2910 @section Simple Usage
2914 Say you want to download a @sc{url}. Just type:
2917 wget http://fly.srk.fer.hr/
2921 But what will happen if the connection is slow, and the file is lengthy?
2922 The connection will probably fail before the whole file is retrieved,
2923 more than once. In this case, Wget will try getting the file until it
2924 either gets the whole of it, or exceeds the default number of retries
2925 (this being 20). It is easy to change the number of tries to 45, to
2926 insure that the whole file will arrive safely:
2929 wget --tries=45 http://fly.srk.fer.hr/jpg/flyweb.jpg
2933 Now let's leave Wget to work in the background, and write its progress
2934 to log file @file{log}. It is tiring to type @samp{--tries}, so we
2935 shall use @samp{-t}.
2938 wget -t 45 -o log http://fly.srk.fer.hr/jpg/flyweb.jpg &
2941 The ampersand at the end of the line makes sure that Wget works in the
2942 background. To unlimit the number of retries, use @samp{-t inf}.
2945 The usage of @sc{ftp} is as simple. Wget will take care of login and
2949 wget ftp://gnjilux.srk.fer.hr/welcome.msg
2953 If you specify a directory, Wget will retrieve the directory listing,
2954 parse it and convert it to @sc{html}. Try:
2957 wget ftp://ftp.gnu.org/pub/gnu/
2962 @node Advanced Usage
2963 @section Advanced Usage
2967 You have a file that contains the URLs you want to download? Use the
2974 If you specify @samp{-} as file name, the @sc{url}s will be read from
2978 Create a five levels deep mirror image of the GNU web site, with the
2979 same directory structure the original has, with only one try per
2980 document, saving the log of the activities to @file{gnulog}:
2983 wget -r http://www.gnu.org/ -o gnulog
2987 The same as the above, but convert the links in the @sc{html} files to
2988 point to local files, so you can view the documents off-line:
2991 wget --convert-links -r http://www.gnu.org/ -o gnulog
2995 Retrieve only one @sc{html} page, but make sure that all the elements needed
2996 for the page to be displayed, such as inline images and external style
2997 sheets, are also downloaded. Also make sure the downloaded page
2998 references the downloaded links.
3001 wget -p --convert-links http://www.server.com/dir/page.html
3004 The @sc{html} page will be saved to @file{www.server.com/dir/page.html}, and
3005 the images, stylesheets, etc., somewhere under @file{www.server.com/},
3006 depending on where they were on the remote server.
3009 The same as the above, but without the @file{www.server.com/} directory.
3010 In fact, I don't want to have all those random server directories
3011 anyway---just save @emph{all} those files under a @file{download/}
3012 subdirectory of the current directory.
3015 wget -p --convert-links -nH -nd -Pdownload \
3016 http://www.server.com/dir/page.html
3020 Retrieve the index.html of @samp{www.lycos.com}, showing the original
3024 wget -S http://www.lycos.com/
3028 Save the server headers with the file, perhaps for post-processing.
3031 wget --save-headers http://www.lycos.com/
3036 Retrieve the first two levels of @samp{wuarchive.wustl.edu}, saving them
3040 wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/
3044 You want to download all the @sc{gif}s from a directory on an @sc{http}
3045 server. You tried @samp{wget http://www.server.com/dir/*.gif}, but that
3046 didn't work because @sc{http} retrieval does not support globbing. In
3050 wget -r -l1 --no-parent -A.gif http://www.server.com/dir/
3053 More verbose, but the effect is the same. @samp{-r -l1} means to
3054 retrieve recursively (@pxref{Recursive Download}), with maximum depth
3055 of 1. @samp{--no-parent} means that references to the parent directory
3056 are ignored (@pxref{Directory-Based Limits}), and @samp{-A.gif} means to
3057 download only the @sc{gif} files. @samp{-A "*.gif"} would have worked
3061 Suppose you were in the middle of downloading, when Wget was
3062 interrupted. Now you do not want to clobber the files already present.
3066 wget -nc -r http://www.gnu.org/
3070 If you want to encode your own username and password to @sc{http} or
3071 @sc{ftp}, use the appropriate @sc{url} syntax (@pxref{URL Format}).
3074 wget ftp://hniksic:mypassword@@unix.server.com/.emacs
3077 Note, however, that this usage is not advisable on multi-user systems
3078 because it reveals your password to anyone who looks at the output of
3081 @cindex redirecting output
3083 You would like the output documents to go to standard output instead of
3087 wget -O - http://jagor.srce.hr/ http://www.srce.hr/
3090 You can also combine the two options and make pipelines to retrieve the
3091 documents from remote hotlists:
3094 wget -O - http://cool.list.com/ | wget --force-html -i -
3098 @node Very Advanced Usage
3099 @section Very Advanced Usage
3104 If you wish Wget to keep a mirror of a page (or @sc{ftp}
3105 subdirectories), use @samp{--mirror} (@samp{-m}), which is the shorthand
3106 for @samp{-r -l inf -N}. You can put Wget in the crontab file asking it
3107 to recheck a site each Sunday:
3111 0 0 * * 0 wget --mirror http://www.gnu.org/ -o /home/me/weeklog
3115 In addition to the above, you want the links to be converted for local
3116 viewing. But, after having read this manual, you know that link
3117 conversion doesn't play well with timestamping, so you also want Wget to
3118 back up the original @sc{html} files before the conversion. Wget invocation
3119 would look like this:
3122 wget --mirror --convert-links --backup-converted \
3123 http://www.gnu.org/ -o /home/me/weeklog
3127 But you've also noticed that local viewing doesn't work all that well
3128 when @sc{html} files are saved under extensions other than @samp{.html},
3129 perhaps because they were served as @file{index.cgi}. So you'd like
3130 Wget to rename all the files served with content-type @samp{text/html}
3131 or @samp{application/xhtml+xml} to @file{@var{name}.html}.
3134 wget --mirror --convert-links --backup-converted \
3135 --html-extension -o /home/me/weeklog \
3139 Or, with less typing:
3142 wget -m -k -K -E http://www.gnu.org/ -o /home/me/weeklog
3151 This chapter contains all the stuff that could not fit anywhere else.
3154 * Proxies:: Support for proxy servers
3155 * Distribution:: Getting the latest version.
3156 * Mailing List:: Wget mailing list for announcements and discussion.
3157 * Reporting Bugs:: How and where to report bugs.
3158 * Portability:: The systems Wget works on.
3159 * Signals:: Signal-handling performed by Wget.
3166 @dfn{Proxies} are special-purpose @sc{http} servers designed to transfer
3167 data from remote servers to local clients. One typical use of proxies
3168 is lightening network load for users behind a slow connection. This is
3169 achieved by channeling all @sc{http} and @sc{ftp} requests through the
3170 proxy which caches the transferred data. When a cached resource is
3171 requested again, proxy will return the data from cache. Another use for
3172 proxies is for companies that separate (for security reasons) their
3173 internal networks from the rest of Internet. In order to obtain
3174 information from the Web, their users connect and retrieve remote data
3175 using an authorized proxy.
3177 Wget supports proxies for both @sc{http} and @sc{ftp} retrievals. The
3178 standard way to specify proxy location, which Wget recognizes, is using
3179 the following environment variables:
3183 This variable should contain the @sc{url} of the proxy for @sc{http}
3187 This variable should contain the @sc{url} of the proxy for @sc{ftp}
3188 connections. It is quite common that @sc{http_proxy} and @sc{ftp_proxy}
3189 are set to the same @sc{url}.
3192 This variable should contain a comma-separated list of domain extensions
3193 proxy should @emph{not} be used for. For instance, if the value of
3194 @code{no_proxy} is @samp{.mit.edu}, proxy will not be used to retrieve
3198 In addition to the environment variables, proxy location and settings
3199 may be specified from within Wget itself.
3203 @itemx proxy = on/off
3204 This option and the corresponding command may be used to suppress the
3205 use of proxy, even if the appropriate environment variables are set.
3207 @item http_proxy = @var{URL}
3208 @itemx ftp_proxy = @var{URL}
3209 @itemx no_proxy = @var{string}
3210 These startup file variables allow you to override the proxy settings
3211 specified by the environment.
3214 Some proxy servers require authorization to enable you to use them. The
3215 authorization consists of @dfn{username} and @dfn{password}, which must
3216 be sent by Wget. As with @sc{http} authorization, several
3217 authentication schemes exist. For proxy authorization only the
3218 @code{Basic} authentication scheme is currently implemented.
3220 You may specify your username and password either through the proxy
3221 @sc{url} or through the command-line options. Assuming that the
3222 company's proxy is located at @samp{proxy.company.com} at port 8001, a
3223 proxy @sc{url} location containing authorization data might look like
3227 http://hniksic:mypassword@@proxy.company.com:8001/
3230 Alternatively, you may use the @samp{proxy-user} and
3231 @samp{proxy-password} options, and the equivalent @file{.wgetrc}
3232 settings @code{proxy_user} and @code{proxy_password} to set the proxy
3233 username and password.
3236 @section Distribution
3237 @cindex latest version
3239 Like all GNU utilities, the latest version of Wget can be found at the
3240 master GNU archive site ftp.gnu.org, and its mirrors. For example,
3241 Wget @value{VERSION} can be found at
3242 @url{ftp://ftp.gnu.org/pub/gnu/wget/wget-@value{VERSION}.tar.gz}
3245 @section Mailing List
3246 @cindex mailing list
3249 There are several Wget-related mailing lists, all hosted by
3250 SunSITE.dk. The general discussion list is at
3251 @email{wget@@sunsite.dk}. It is the preferred place for bug reports
3252 and suggestions, as well as for discussion of development. You are
3253 invited to subscribe.
3255 To subscribe, simply send mail to @email{wget-subscribe@@sunsite.dk}
3256 and follow the instructions. Unsubscribe by mailing to
3257 @email{wget-unsubscribe@@sunsite.dk}. The mailing list is archived at
3258 @url{http://www.mail-archive.com/wget%40sunsite.dk/} and at
3259 @url{http://news.gmane.org/gmane.comp.web.wget.general}.
3261 The second mailing list is at @email{wget-patches@@sunsite.dk}, and is
3262 used to submit patches for review by Wget developers. A ``patch'' is
3263 a textual representation of change to source code, readable by both
3264 humans and programs. The file @file{PATCHES} that comes with Wget
3265 covers the creation and submitting of patches in detail. Please don't
3266 send general suggestions or bug reports to @samp{wget-patches}; use it
3267 only for patch submissions.
3269 To subscribe, simply send mail to @email{wget-subscribe@@sunsite.dk}
3270 and follow the instructions. Unsubscribe by mailing to
3271 @email{wget-unsubscribe@@sunsite.dk}. The mailing list is archived at
3272 @url{http://news.gmane.org/gmane.comp.web.wget.patches}.
3274 Finally, there is a read-only list at @email{wget-cvs@@sunsite.dk}
3275 that tracks commits to the Wget CVS repository. To subscribe to that
3276 list, send mail to @email{wget-cvs-subscribe@@sunsite.dk}. The list
3279 @node Reporting Bugs
3280 @section Reporting Bugs
3282 @cindex reporting bugs
3286 You are welcome to send bug reports about GNU Wget to
3287 @email{bug-wget@@gnu.org}.
3289 Before actually submitting a bug report, please try to follow a few
3294 Please try to ascertain that the behavior you see really is a bug. If
3295 Wget crashes, it's a bug. If Wget does not behave as documented,
3296 it's a bug. If things work strange, but you are not sure about the way
3297 they are supposed to work, it might well be a bug.
3300 Try to repeat the bug in as simple circumstances as possible. E.g. if
3301 Wget crashes while downloading @samp{wget -rl0 -kKE -t5 -Y0
3302 http://yoyodyne.com -o /tmp/log}, you should try to see if the crash is
3303 repeatable, and if will occur with a simpler set of options. You might
3304 even try to start the download at the page where the crash occurred to
3305 see if that page somehow triggered the crash.
3307 Also, while I will probably be interested to know the contents of your
3308 @file{.wgetrc} file, just dumping it into the debug message is probably
3309 a bad idea. Instead, you should first try to see if the bug repeats
3310 with @file{.wgetrc} moved out of the way. Only if it turns out that
3311 @file{.wgetrc} settings affect the bug, mail me the relevant parts of
3315 Please start Wget with @samp{-d} option and send us the resulting
3316 output (or relevant parts thereof). If Wget was compiled without
3317 debug support, recompile it---it is @emph{much} easier to trace bugs
3318 with debug support on.
3320 Note: please make sure to remove any potentially sensitive information
3321 from the debug log before sending it to the bug address. The
3322 @code{-d} won't go out of its way to collect sensitive information,
3323 but the log @emph{will} contain a fairly complete transcript of Wget's
3324 communication with the server, which may include passwords and pieces
3325 of downloaded data. Since the bug address is publically archived, you
3326 may assume that all bug reports are visible to the public.
3329 If Wget has crashed, try to run it in a debugger, e.g. @code{gdb `which
3330 wget` core} and type @code{where} to get the backtrace. This may not
3331 work if the system administrator has disabled core files, but it is
3337 @section Portability
3339 @cindex operating systems
3341 Like all GNU software, Wget works on the GNU system. However, since it
3342 uses GNU Autoconf for building and configuring, and mostly avoids using
3343 ``special'' features of any particular Unix, it should compile (and
3344 work) on all common Unix flavors.
3346 Various Wget versions have been compiled and tested under many kinds
3347 of Unix systems, including GNU/Linux, Solaris, SunOS 4.x, OSF (aka
3348 Digital Unix or Tru64), Ultrix, *BSD, IRIX, AIX, and others. Some of
3349 those systems are no longer in widespread use and may not be able to
3350 support recent versions of Wget. If Wget fails to compile on your
3351 system, we would like to know about it.
3353 Thanks to kind contributors, this version of Wget compiles and works
3354 on 32-bit Microsoft Windows platforms. It has been compiled
3355 successfully using MS Visual C++ 6.0, Watcom, Borland C, and GCC
3356 compilers. Naturally, it is crippled of some features available on
3357 Unix, but it should work as a substitute for people stuck with
3358 Windows. Note that Windows-specific portions of Wget are not
3359 guaranteed to be supported in the future, although this has been the
3360 case in practice for many years now. All questions and problems in
3361 Windows usage should be reported to Wget mailing list at
3362 @email{wget@@sunsite.dk} where the volunteers who maintain the
3363 Windows-related features might look at them.
3367 @cindex signal handling
3370 Since the purpose of Wget is background work, it catches the hangup
3371 signal (@code{SIGHUP}) and ignores it. If the output was on standard
3372 output, it will be redirected to a file named @file{wget-log}.
3373 Otherwise, @code{SIGHUP} is ignored. This is convenient when you wish
3374 to redirect the output of Wget after having started it.
3377 $ wget http://www.gnus.org/dist/gnus.tar.gz &
3380 SIGHUP received, redirecting output to `wget-log'.
3383 Other than that, Wget will not try to interfere with signals in any way.
3384 @kbd{C-c}, @code{kill -TERM} and @code{kill -KILL} should kill it alike.
3389 This chapter contains some references I consider useful.
3392 * Robot Exclusion:: Wget's support for RES.
3393 * Security Considerations:: Security with Wget.
3394 * Contributors:: People who helped.
3397 @node Robot Exclusion
3398 @section Robot Exclusion
3399 @cindex robot exclusion
3401 @cindex server maintenance
3403 It is extremely easy to make Wget wander aimlessly around a web site,
3404 sucking all the available data in progress. @samp{wget -r @var{site}},
3405 and you're set. Great? Not for the server admin.
3407 As long as Wget is only retrieving static pages, and doing it at a
3408 reasonable rate (see the @samp{--wait} option), there's not much of a
3409 problem. The trouble is that Wget can't tell the difference between the
3410 smallest static page and the most demanding CGI. A site I know has a
3411 section handled by a CGI Perl script that converts Info files to @sc{html} on
3412 the fly. The script is slow, but works well enough for human users
3413 viewing an occasional Info file. However, when someone's recursive Wget
3414 download stumbles upon the index page that links to all the Info files
3415 through the script, the system is brought to its knees without providing
3416 anything useful to the user (This task of converting Info files could be
3417 done locally and access to Info documentation for all installed GNU
3418 software on a system is available from the @code{info} command).
3420 To avoid this kind of accident, as well as to preserve privacy for
3421 documents that need to be protected from well-behaved robots, the
3422 concept of @dfn{robot exclusion} was invented. The idea is that
3423 the server administrators and document authors can specify which
3424 portions of the site they wish to protect from robots and those
3425 they will permit access.
3427 The most popular mechanism, and the @i{de facto} standard supported by
3428 all the major robots, is the ``Robots Exclusion Standard'' (RES) written
3429 by Martijn Koster et al. in 1994. It specifies the format of a text
3430 file containing directives that instruct the robots which URL paths to
3431 avoid. To be found by the robots, the specifications must be placed in
3432 @file{/robots.txt} in the server root, which the robots are expected to
3435 Although Wget is not a web robot in the strictest sense of the word, it
3436 can downloads large parts of the site without the user's intervention to
3437 download an individual page. Because of that, Wget honors RES when
3438 downloading recursively. For instance, when you issue:
3441 wget -r http://www.server.com/
3444 First the index of @samp{www.server.com} will be downloaded. If Wget
3445 finds that it wants to download more documents from that server, it will
3446 request @samp{http://www.server.com/robots.txt} and, if found, use it
3447 for further downloads. @file{robots.txt} is loaded only once per each
3450 Until version 1.8, Wget supported the first version of the standard,
3451 written by Martijn Koster in 1994 and available at
3452 @url{http://www.robotstxt.org/wc/norobots.html}. As of version 1.8,
3453 Wget has supported the additional directives specified in the internet
3454 draft @samp{<draft-koster-robots-00.txt>} titled ``A Method for Web
3455 Robots Control''. The draft, which has as far as I know never made to
3456 an @sc{rfc}, is available at
3457 @url{http://www.robotstxt.org/wc/norobots-rfc.txt}.
3459 This manual no longer includes the text of the Robot Exclusion Standard.
3461 The second, less known mechanism, enables the author of an individual
3462 document to specify whether they want the links from the file to be
3463 followed by a robot. This is achieved using the @code{META} tag, like
3467 <meta name="robots" content="nofollow">
3470 This is explained in some detail at
3471 @url{http://www.robotstxt.org/wc/meta-user.html}. Wget supports this
3472 method of robot exclusion in addition to the usual @file{/robots.txt}
3475 If you know what you are doing and really really wish to turn off the
3476 robot exclusion, set the @code{robots} variable to @samp{off} in your
3477 @file{.wgetrc}. You can achieve the same effect from the command line
3478 using the @code{-e} switch, e.g. @samp{wget -e robots=off @var{url}...}.
3480 @node Security Considerations
3481 @section Security Considerations
3484 When using Wget, you must be aware that it sends unencrypted passwords
3485 through the network, which may present a security problem. Here are the
3486 main issues, and some solutions.
3490 The passwords on the command line are visible using @code{ps}. The best
3491 way around it is to use @code{wget -i -} and feed the @sc{url}s to
3492 Wget's standard input, each on a separate line, terminated by @kbd{C-d}.
3493 Another workaround is to use @file{.netrc} to store passwords; however,
3494 storing unencrypted passwords is also considered a security risk.
3497 Using the insecure @dfn{basic} authentication scheme, unencrypted
3498 passwords are transmitted through the network routers and gateways.
3501 The @sc{ftp} passwords are also in no way encrypted. There is no good
3502 solution for this at the moment.
3505 Although the ``normal'' output of Wget tries to hide the passwords,
3506 debugging logs show them, in all forms. This problem is avoided by
3507 being careful when you send debug logs (yes, even when you send them to
3512 @section Contributors
3513 @cindex contributors
3516 GNU Wget was written by Hrvoje Nik@v{s}i@'{c} @email{hniksic@@xemacs.org}.
3519 GNU Wget was written by Hrvoje Niksic @email{hniksic@@xemacs.org}.
3521 However, its development could never have gone as far as it has, were it
3522 not for the help of many people, either with bug reports, feature
3523 proposals, patches, or letters saying ``Thanks!''.
3525 Special thanks goes to the following people (no particular order):
3529 Karsten Thygesen---donated system resources such as the mailing list,
3530 web space, and @sc{ftp} space, along with a lot of time to make these
3534 Shawn McHorse---bug reports and patches.
3537 Kaveh R. Ghazi---on-the-fly @code{ansi2knr}-ization. Lots of
3541 Gordon Matzigkeit---@file{.netrc} support.
3545 Zlatko @v{C}alu@v{s}i@'{c}, Tomislav Vujec and Dra@v{z}en
3546 Ka@v{c}ar---feature suggestions and ``philosophical'' discussions.
3549 Zlatko Calusic, Tomislav Vujec and Drazen Kacar---feature suggestions
3550 and ``philosophical'' discussions.
3554 Darko Budor---initial port to Windows.
3557 Antonio Rosella---help and suggestions, plus the Italian translation.
3561 Tomislav Petrovi@'{c}, Mario Miko@v{c}evi@'{c}---many bug reports and
3565 Tomislav Petrovic, Mario Mikocevic---many bug reports and suggestions.
3570 Fran@,{c}ois Pinard---many thorough bug reports and discussions.
3573 Francois Pinard---many thorough bug reports and discussions.
3577 Karl Eichwalder---lots of help with internationalization and other
3581 Junio Hamano---donated support for Opie and @sc{http} @code{Digest}
3585 The people who provided donations for development, including Brian
3589 The following people have provided patches, bug/build reports, useful
3590 suggestions, beta testing services, fan mail and all the other things
3591 that make maintenance so much fun:
3611 Kristijan @v{C}onka@v{s},
3620 Bertrand Demiddelaer,
3633 Aleksandar Erkalovi@'{c},
3636 Aleksandar Erkalovic,
3657 Erik Magnus Hulthen,
3676 Goran Kezunovi@'{c},
3689 $\Sigma\acute{\iota}\mu o\varsigma\;
3690 \Xi\varepsilon\nu\iota\tau\acute{\epsilon}\lambda\lambda\eta\varsigma$
3691 (Simos KSenitellis),
3700 Nicol@'{a}s Lichtmeier,
3706 Alexander V. Lukyanov,
3741 @c Texinfo doesn't grok @'{@i}, so we have to use TeX itself.
3743 Juan Jos\'{e} Rodr\'{\i}guez,
3746 Juan Jose Rodriguez,
3763 Szakacsits Szabolcs,
3777 Douglas E. Wegscheid,
3788 Apologies to all who I accidentally left out, and many thanks to all the
3789 subscribers of the Wget mailing list.
3796 @cindex free software
3798 GNU Wget is licensed under the GNU General Public License (GNU GPL),
3799 which makes it @dfn{free software}. Please note that ``free'' in ``free
3800 software'' refers to liberty, not price. As some people like to point
3801 out, it's the ``free'' of ``free speech'', not the ``free'' of ``free
3804 The exact and legally binding distribution terms are spelled out below.
3805 The GPL guarantees that you have the right (freedom) to run and change
3806 GNU Wget and distribute it to others, and even---if you want---charge
3807 money for doing any of those things. With these rights comes the
3808 obligation to distribute the source code along with the software and to
3809 grant your recipients the same rights and impose the same restrictions.
3811 This licensing model is also known as @dfn{open source} because it,
3812 among other things, makes sure that all recipients will receive the
3813 source code along with the program, and be able to improve it. The GNU
3814 project prefers the term ``free software'' for reasons outlined at
3815 @url{http://www.gnu.org/philosophy/free-software-for-freedom.html}.
3817 The exact license terms are defined by this paragraph and the GNU
3818 General Public License it refers to:
3821 GNU Wget is free software; you can redistribute it and/or modify it
3822 under the terms of the GNU General Public License as published by the
3823 Free Software Foundation; either version 2 of the License, or (at your
3824 option) any later version.
3826 GNU Wget is distributed in the hope that it will be useful, but WITHOUT
3827 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
3828 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
3831 A copy of the GNU General Public License is included as part of this
3832 manual; if you did not receive it, write to the Free Software
3833 Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
3836 In addition to this, this manual is free in the same sense:
3839 Permission is granted to copy, distribute and/or modify this document
3840 under the terms of the GNU Free Documentation License, Version 1.2 or
3841 any later version published by the Free Software Foundation; with the
3842 Invariant Sections being ``GNU General Public License'' and ``GNU Free
3843 Documentation License'', with no Front-Cover Texts, and with no
3844 Back-Cover Texts. A copy of the license is included in the section
3845 entitled ``GNU Free Documentation License''.
3848 @c #### Maybe we should wrap these licenses in ifinfo? Stallman says
3849 @c that the GFDL needs to be present in the manual, and to me it would
3850 @c suck to include the license for the manual and not the license for
3853 The full texts of the GNU General Public License and of the GNU Free
3854 Documentation License are available below.
3857 * GNU General Public License::
3858 * GNU Free Documentation License::
3861 @node GNU General Public License
3862 @section GNU General Public License
3863 @center Version 2, June 1991
3866 Copyright @copyright{} 1989, 1991 Free Software Foundation, Inc.
3867 675 Mass Ave, Cambridge, MA 02139, USA
3869 Everyone is permitted to copy and distribute verbatim copies
3870 of this license document, but changing it is not allowed.
3873 @unnumberedsec Preamble
3875 The licenses for most software are designed to take away your
3876 freedom to share and change it. By contrast, the GNU General Public
3877 License is intended to guarantee your freedom to share and change free
3878 software---to make sure the software is free for all its users. This
3879 General Public License applies to most of the Free Software
3880 Foundation's software and to any other program whose authors commit to
3881 using it. (Some other Free Software Foundation software is covered by
3882 the GNU Library General Public License instead.) You can apply it to
3885 When we speak of free software, we are referring to freedom, not
3886 price. Our General Public Licenses are designed to make sure that you
3887 have the freedom to distribute copies of free software (and charge for
3888 this service if you wish), that you receive source code or can get it
3889 if you want it, that you can change the software or use pieces of it
3890 in new free programs; and that you know you can do these things.
3892 To protect your rights, we need to make restrictions that forbid
3893 anyone to deny you these rights or to ask you to surrender the rights.
3894 These restrictions translate to certain responsibilities for you if you
3895 distribute copies of the software, or if you modify it.
3897 For example, if you distribute copies of such a program, whether
3898 gratis or for a fee, you must give the recipients all the rights that
3899 you have. You must make sure that they, too, receive or can get the
3900 source code. And you must show them these terms so they know their
3903 We protect your rights with two steps: (1) copyright the software, and
3904 (2) offer you this license which gives you legal permission to copy,
3905 distribute and/or modify the software.
3907 Also, for each author's protection and ours, we want to make certain
3908 that everyone understands that there is no warranty for this free
3909 software. If the software is modified by someone else and passed on, we
3910 want its recipients to know that what they have is not the original, so
3911 that any problems introduced by others will not reflect on the original
3912 authors' reputations.
3914 Finally, any free program is threatened constantly by software
3915 patents. We wish to avoid the danger that redistributors of a free
3916 program will individually obtain patent licenses, in effect making the
3917 program proprietary. To prevent this, we have made it clear that any
3918 patent must be licensed for everyone's free use or not licensed at all.
3920 The precise terms and conditions for copying, distribution and
3921 modification follow.
3924 @unnumberedsec TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
3927 @center TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
3932 This License applies to any program or other work which contains
3933 a notice placed by the copyright holder saying it may be distributed
3934 under the terms of this General Public License. The ``Program'', below,
3935 refers to any such program or work, and a ``work based on the Program''
3936 means either the Program or any derivative work under copyright law:
3937 that is to say, a work containing the Program or a portion of it,
3938 either verbatim or with modifications and/or translated into another
3939 language. (Hereinafter, translation is included without limitation in
3940 the term ``modification''.) Each licensee is addressed as ``you''.
3942 Activities other than copying, distribution and modification are not
3943 covered by this License; they are outside its scope. The act of
3944 running the Program is not restricted, and the output from the Program
3945 is covered only if its contents constitute a work based on the
3946 Program (independent of having been made by running the Program).
3947 Whether that is true depends on what the Program does.
3950 You may copy and distribute verbatim copies of the Program's
3951 source code as you receive it, in any medium, provided that you
3952 conspicuously and appropriately publish on each copy an appropriate
3953 copyright notice and disclaimer of warranty; keep intact all the
3954 notices that refer to this License and to the absence of any warranty;
3955 and give any other recipients of the Program a copy of this License
3956 along with the Program.
3958 You may charge a fee for the physical act of transferring a copy, and
3959 you may at your option offer warranty protection in exchange for a fee.
3962 You may modify your copy or copies of the Program or any portion
3963 of it, thus forming a work based on the Program, and copy and
3964 distribute such modifications or work under the terms of Section 1
3965 above, provided that you also meet all of these conditions:
3969 You must cause the modified files to carry prominent notices
3970 stating that you changed the files and the date of any change.
3973 You must cause any work that you distribute or publish, that in
3974 whole or in part contains or is derived from the Program or any
3975 part thereof, to be licensed as a whole at no charge to all third
3976 parties under the terms of this License.
3979 If the modified program normally reads commands interactively
3980 when run, you must cause it, when started running for such
3981 interactive use in the most ordinary way, to print or display an
3982 announcement including an appropriate copyright notice and a
3983 notice that there is no warranty (or else, saying that you provide
3984 a warranty) and that users may redistribute the program under
3985 these conditions, and telling the user how to view a copy of this
3986 License. (Exception: if the Program itself is interactive but
3987 does not normally print such an announcement, your work based on
3988 the Program is not required to print an announcement.)
3991 These requirements apply to the modified work as a whole. If
3992 identifiable sections of that work are not derived from the Program,
3993 and can be reasonably considered independent and separate works in
3994 themselves, then this License, and its terms, do not apply to those
3995 sections when you distribute them as separate works. But when you
3996 distribute the same sections as part of a whole which is a work based
3997 on the Program, the distribution of the whole must be on the terms of
3998 this License, whose permissions for other licensees extend to the
3999 entire whole, and thus to each and every part regardless of who wrote it.
4001 Thus, it is not the intent of this section to claim rights or contest
4002 your rights to work written entirely by you; rather, the intent is to
4003 exercise the right to control the distribution of derivative or
4004 collective works based on the Program.
4006 In addition, mere aggregation of another work not based on the Program
4007 with the Program (or with a work based on the Program) on a volume of
4008 a storage or distribution medium does not bring the other work under
4009 the scope of this License.
4012 You may copy and distribute the Program (or a work based on it,
4013 under Section 2) in object code or executable form under the terms of
4014 Sections 1 and 2 above provided that you also do one of the following:
4018 Accompany it with the complete corresponding machine-readable
4019 source code, which must be distributed under the terms of Sections
4020 1 and 2 above on a medium customarily used for software interchange; or,
4023 Accompany it with a written offer, valid for at least three
4024 years, to give any third party, for a charge no more than your
4025 cost of physically performing source distribution, a complete
4026 machine-readable copy of the corresponding source code, to be
4027 distributed under the terms of Sections 1 and 2 above on a medium
4028 customarily used for software interchange; or,
4031 Accompany it with the information you received as to the offer
4032 to distribute corresponding source code. (This alternative is
4033 allowed only for noncommercial distribution and only if you
4034 received the program in object code or executable form with such
4035 an offer, in accord with Subsection b above.)
4038 The source code for a work means the preferred form of the work for
4039 making modifications to it. For an executable work, complete source
4040 code means all the source code for all modules it contains, plus any
4041 associated interface definition files, plus the scripts used to
4042 control compilation and installation of the executable. However, as a
4043 special exception, the source code distributed need not include
4044 anything that is normally distributed (in either source or binary
4045 form) with the major components (compiler, kernel, and so on) of the
4046 operating system on which the executable runs, unless that component
4047 itself accompanies the executable.
4049 If distribution of executable or object code is made by offering
4050 access to copy from a designated place, then offering equivalent
4051 access to copy the source code from the same place counts as
4052 distribution of the source code, even though third parties are not
4053 compelled to copy the source along with the object code.
4056 You may not copy, modify, sublicense, or distribute the Program
4057 except as expressly provided under this License. Any attempt
4058 otherwise to copy, modify, sublicense or distribute the Program is
4059 void, and will automatically terminate your rights under this License.
4060 However, parties who have received copies, or rights, from you under
4061 this License will not have their licenses terminated so long as such
4062 parties remain in full compliance.
4065 You are not required to accept this License, since you have not
4066 signed it. However, nothing else grants you permission to modify or
4067 distribute the Program or its derivative works. These actions are
4068 prohibited by law if you do not accept this License. Therefore, by
4069 modifying or distributing the Program (or any work based on the
4070 Program), you indicate your acceptance of this License to do so, and
4071 all its terms and conditions for copying, distributing or modifying
4072 the Program or works based on it.
4075 Each time you redistribute the Program (or any work based on the
4076 Program), the recipient automatically receives a license from the
4077 original licensor to copy, distribute or modify the Program subject to
4078 these terms and conditions. You may not impose any further
4079 restrictions on the recipients' exercise of the rights granted herein.
4080 You are not responsible for enforcing compliance by third parties to
4084 If, as a consequence of a court judgment or allegation of patent
4085 infringement or for any other reason (not limited to patent issues),
4086 conditions are imposed on you (whether by court order, agreement or
4087 otherwise) that contradict the conditions of this License, they do not
4088 excuse you from the conditions of this License. If you cannot
4089 distribute so as to satisfy simultaneously your obligations under this
4090 License and any other pertinent obligations, then as a consequence you
4091 may not distribute the Program at all. For example, if a patent
4092 license would not permit royalty-free redistribution of the Program by
4093 all those who receive copies directly or indirectly through you, then
4094 the only way you could satisfy both it and this License would be to
4095 refrain entirely from distribution of the Program.
4097 If any portion of this section is held invalid or unenforceable under
4098 any particular circumstance, the balance of the section is intended to
4099 apply and the section as a whole is intended to apply in other
4102 It is not the purpose of this section to induce you to infringe any
4103 patents or other property right claims or to contest validity of any
4104 such claims; this section has the sole purpose of protecting the
4105 integrity of the free software distribution system, which is
4106 implemented by public license practices. Many people have made
4107 generous contributions to the wide range of software distributed
4108 through that system in reliance on consistent application of that
4109 system; it is up to the author/donor to decide if he or she is willing
4110 to distribute software through any other system and a licensee cannot
4113 This section is intended to make thoroughly clear what is believed to
4114 be a consequence of the rest of this License.
4117 If the distribution and/or use of the Program is restricted in
4118 certain countries either by patents or by copyrighted interfaces, the
4119 original copyright holder who places the Program under this License
4120 may add an explicit geographical distribution limitation excluding
4121 those countries, so that distribution is permitted only in or among
4122 countries not thus excluded. In such case, this License incorporates
4123 the limitation as if written in the body of this License.
4126 The Free Software Foundation may publish revised and/or new versions
4127 of the General Public License from time to time. Such new versions will
4128 be similar in spirit to the present version, but may differ in detail to
4129 address new problems or concerns.
4131 Each version is given a distinguishing version number. If the Program
4132 specifies a version number of this License which applies to it and ``any
4133 later version'', you have the option of following the terms and conditions
4134 either of that version or of any later version published by the Free
4135 Software Foundation. If the Program does not specify a version number of
4136 this License, you may choose any version ever published by the Free Software
4140 If you wish to incorporate parts of the Program into other free
4141 programs whose distribution conditions are different, write to the author
4142 to ask for permission. For software which is copyrighted by the Free
4143 Software Foundation, write to the Free Software Foundation; we sometimes
4144 make exceptions for this. Our decision will be guided by the two goals
4145 of preserving the free status of all derivatives of our free software and
4146 of promoting the sharing and reuse of software generally.
4149 @heading NO WARRANTY
4157 BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
4158 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
4159 OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
4160 PROVIDE THE PROGRAM ``AS IS'' WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
4161 OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
4162 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
4163 TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
4164 PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
4165 REPAIR OR CORRECTION.
4168 IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
4169 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
4170 REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
4171 INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
4172 OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
4173 TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
4174 YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
4175 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
4176 POSSIBILITY OF SUCH DAMAGES.
4180 @heading END OF TERMS AND CONDITIONS
4183 @center END OF TERMS AND CONDITIONS
4187 @unnumberedsec How to Apply These Terms to Your New Programs
4189 If you develop a new program, and you want it to be of the greatest
4190 possible use to the public, the best way to achieve this is to make it
4191 free software which everyone can redistribute and change under these terms.
4193 To do so, attach the following notices to the program. It is safest
4194 to attach them to the start of each source file to most effectively
4195 convey the exclusion of warranty; and each file should have at least
4196 the ``copyright'' line and a pointer to where the full notice is found.
4199 @var{one line to give the program's name and an idea of what it does.}
4200 Copyright (C) 20@var{yy} @var{name of author}
4202 This program is free software; you can redistribute it and/or
4203 modify it under the terms of the GNU General Public License
4204 as published by the Free Software Foundation; either version 2
4205 of the License, or (at your option) any later version.
4207 This program is distributed in the hope that it will be useful,
4208 but WITHOUT ANY WARRANTY; without even the implied warranty of
4209 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4210 GNU General Public License for more details.
4212 You should have received a copy of the GNU General Public License
4213 along with this program; if not, write to the Free Software
4214 Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
4217 Also add information on how to contact you by electronic and paper mail.
4219 If the program is interactive, make it output a short notice like this
4220 when it starts in an interactive mode:
4223 Gnomovision version 69, Copyright (C) 20@var{yy} @var{name of author}
4224 Gnomovision comes with ABSOLUTELY NO WARRANTY; for details
4225 type `show w'. This is free software, and you are welcome
4226 to redistribute it under certain conditions; type `show c'
4230 The hypothetical commands @samp{show w} and @samp{show c} should show
4231 the appropriate parts of the General Public License. Of course, the
4232 commands you use may be called something other than @samp{show w} and
4233 @samp{show c}; they could even be mouse-clicks or menu items---whatever
4236 You should also get your employer (if you work as a programmer) or your
4237 school, if any, to sign a ``copyright disclaimer'' for the program, if
4238 necessary. Here is a sample; alter the names:
4242 Yoyodyne, Inc., hereby disclaims all copyright
4243 interest in the program `Gnomovision'
4244 (which makes passes at compilers) written
4247 @var{signature of Ty Coon}, 1 April 1989
4248 Ty Coon, President of Vice
4252 This General Public License does not permit incorporating your program into
4253 proprietary programs. If your program is a subroutine library, you may
4254 consider it more useful to permit linking proprietary applications with the
4255 library. If this is what you want to do, use the GNU Library General
4256 Public License instead of this License.
4261 @unnumbered Concept Index