@set Wget Wget
@c man title Wget The non-interactive network downloader.
-@dircategory Network Applications
+@dircategory Network applications
@direntry
-* Wget: (wget). The non-interactive network downloader.
+* Wget: (wget). Non-interactive network downloader.
@end direntry
@copying
@c man begin COPYRIGHT
Copyright @copyright{} 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003,
-2004, 2005, 2006, 2007, 2008, 2009, 2010 Free Software Foundation, Inc.
+2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Free Software Foundation,
+Inc.
@iftex
Permission is granted to make and distribute verbatim copies of
(this paragraph not being relevant to the printed manual).
@end ignore
Permission is granted to copy, distribute and/or modify this document
-under the terms of the GNU Free Documentation License, Version 1.2 or
+under the terms of the GNU Free Documentation License, Version 1.3 or
any later version published by the Free Software Foundation; with no
-Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
-copy of the license is included in the section entitled ``GNU Free
-Documentation License''.
+Invariant Sections, with no Front-Cover Texts, and with no Back-Cover
+Texts. A copy of the license is included in the section entitled
+``GNU Free Documentation License''.
@c man end
@end copying
@ignore
@c man begin AUTHOR
Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
-Currently maintained by Micah Cowan <micah@cowan.name>.
@c man end
@c man begin SEEALSO
This is @strong{not} the complete manual for GNU Wget.
that), which means that error messages and basic information still get
printed.
+@item --report-speed=@var{type}
+Output bandwidth as @var{type}. The only accepted value is @samp{bits}.
+
@cindex input-file
@item -i @var{file}
@itemx --input-file=@var{file}
@cindex retries
@cindex tries
-@cindex number of retries
+@cindex number of tries
@item -t @var{number}
@itemx --tries=@var{number}
-Set number of retries to @var{number}. Specify 0 or @samp{inf} for
+Set number of tries to @var{number}. Specify 0 or @samp{inf} for
infinite retrying. The default is to retry 20 times, with the exception
of fatal errors like ``connection refused'' or ``not found'' (404),
which are not retried.
@samp{.html} or @samp{.htm} will be loaded from the local disk and
parsed as if they had been retrieved from the Web.
+@cindex backing up files
+@item --backups=@var{backups}
+Before (over)writing a file, back up an existing file by adding a
+@samp{.1} suffix (@samp{_1} on VMS) to the file name. Such backup
+files are rotated to @samp{.2}, @samp{.3}, and so on, up to
+@var{backups} (and lost beyond that).
+
@cindex continue retrieval
@cindex incomplete downloads
@cindex resume download
Note that @samp{-c} only works with @sc{ftp} servers and with @sc{http}
servers that support the @code{Range} header.
+@cindex offset
+@cindex continue retrieval
+@cindex incomplete downloads
+@cindex resume download
+@cindex start position
+@item --start-pos=@var{OFFSET}
+Start downloading at zero-based position @var{OFFSET}. Offset may be expressed
+in bytes, kilobytes with the `k' suffix, or megabytes with the `m' suffix, etc.
+
+@samp{--start-pos} has higher precedence over @samp{--continue}. When
+@samp{--start-pos} and @samp{--continue} are both specified, wget will emit a
+warning then proceed as if @samp{--continue} was absent.
+
+Server support for continued download is required, otherwise @samp{--start-pos}
+cannot help. See @samp{-c} for details.
+
@cindex progress indicator
@cindex dot style
@item --progress=@var{type}
the retrieval by printing dots on the screen, each dot representing a
fixed amount of downloaded data.
-When using the dotted retrieval, you may also set the @dfn{style} by
+The progress @var{type} can also take one or more parameters. The parameters
+vary based on the @var{type} selected. Parameters to @var{type} are passed by
+appending them to the type sperated by a colon (:) like this:
+@samp{--progress=@var{type}:@var{parameter1}:@var{parameter2}}.
+
+When using the dotted retrieval, you may set the @dfn{style} by
specifying the type as @samp{dot:@var{style}}. Different styles assign
different meaning to one dot. With the @code{default} style each dot
represents 1K, there are ten dots in a cluster and 50 dots in a line.
The @code{binary} style has a more ``computer''-like orientation---8K
dots, 16-dots clusters and 48 dots per line (which makes for 384K
-lines). The @code{mega} style is suitable for downloading very large
+lines). The @code{mega} style is suitable for downloading large
files---each dot represents 64K retrieved, there are eight dots in a
cluster, and 48 dots on each line (so each line contains 3M).
+If @code{mega} is not enough then you can use the @code{giga}
+style---each dot represents 1M retrieved, there are eight dots in a
+cluster, and 32 dots on each line (so each line contains 32M).
+
+With @samp{--progress=bar}, there are currently two possible parameters,
+@var{force} and @var{noscroll}.
+
+When the output is not a TTY, the progress bar always falls back to ``dot'',
+even if @samp{--progress=bar} was passed to Wget during invokation. This
+behaviour can be overridden and the ``bar'' output forced by using the ``force''
+parameter as @samp{--progress=bar:force}.
+
+By default, the @samp{bar} style progress bar scroll the name of the file from
+left to right for the file being downloaded if the filename exceeds the maximum
+length allotted for its display. In certain cases, such as with
+@samp{--progress=bar:force}, one may not want the scrolling filename in the
+progress bar. By passing the ``noscroll'' parameter, Wget can be forced to
+display as much of the filename as possible without scrolling through it.
Note that you can set the default style using the @code{progress}
command in @file{.wgetrc}. That setting may be overridden from the
-command line. The exception is that, when the output is not a TTY, the
-``dot'' progress will be favored over ``bar''. To force the bar output,
-use @samp{--progress=bar:force}.
+command line. For example, to force the bar output without scrolling,
+use @samp{--progress=bar:force:noscroll}.
+
+@item --show-progress
+Force wget to display the progress bar in any verbosity.
+
+By default, wget only displays the progress bar in verbose mode. One may
+however want wget to display the progress bar on screen in conjunction with
+any other verbosity modes like @samp{--no-verbose} or @samp{--quiet}. This
+is often a desired a property when invoking wget to download several small/large
+files. In such a case, wget could simply be invoked with this parameter to get
+a much cleaner output on the screen.
@item -N
@itemx --timestamping
@item --no-use-server-timestamps
Don't set the local file's timestamp by the one on the server.
-By default, when a file is downloaded, it's timestamps are set to
+By default, when a file is downloaded, its timestamps are set to
match those from the remote file. This allows the use of
@samp{--timestamping} on subsequent invocations of wget. However, it
is sometimes useful to base the local file's timestamp on when it was
actions of one.
@cindex proxy
-@itemx --no-proxy
+@item --no-proxy
Don't use proxies, even if the appropriate @code{*_proxy} environment
variable is defined.
whose encoding does not match the one used locally.
@cindex IPv6
-@itemx -4
+@item -4
@itemx --inet4-only
@itemx -6
@itemx --inet6-only
Other than that, they work in exactly the same way. In particular,
they @emph{both} expect content of the form @code{key1=value1&key2=value2},
with percent-encoding for special characters; the only difference is
-that one expects its content as a command-line paramter and the other
+that one expects its content as a command-line parameter and the other
accepts its content from a file. In particular, @samp{--post-file} is
@emph{not} for transmitting files as form attachments: those must
appear as @code{key=value} data (with appropriate percent-coding) just
@code{application/x-www-form-urlencoded}. Only one of
@samp{--post-data} and @samp{--post-file} should be specified.
+Please note that wget does not require the content to be of the form
+@code{key1=value1&key2=value2}, and neither does it test for it. Wget will
+simply transmit whatever data is provided to it. Most servers however expect
+the POST data to be in the above format when processing HTML Forms.
+
Please be aware that Wget needs to know the size of the POST data in
advance. Therefore the argument to @code{--post-file} must be a regular
file; specifying a FIFO or something like @file{/dev/stdin} won't work.
can't know that until it receives a response, which in turn requires the
request to have been completed -- a chicken-and-egg problem.
-Note: if Wget is redirected after the POST request is completed, it
-will not send the POST data to the redirected URL. This is because
-URLs that process POST often respond with a redirection to a regular
-page, which does not desire or accept POST. It is not completely
-clear that this behavior is optimal; if it doesn't work out, it might
-be changed in the future.
+Note: As of version 1.15 if Wget is redirected after the POST request is
+completed, its behaviour will depend on the response code returned by the
+server. In case of a 301 Moved Permanently, 302 Moved Temporarily or
+307 Temporary Redirect, Wget will, in accordance with RFC2616, continue
+to send a POST request.
+In case a server wants the client to change the Request method upon
+redirection, it should send a 303 See Other response code.
-This example shows how to log to a server using POST and then proceed to
+This example shows how to log in to a server using POST and then proceed to
download the desired pages, presumably only accessible to authorized
users:
be empty. In that case use @samp{--keep-session-cookies} along with
@samp{--save-cookies} to force saving of session cookies.
+@cindex Other HTTP Methods
+@item --method=@var{HTTP-Method}
+For the purpose of RESTful scripting, Wget allows sending of other HTTP Methods
+without the need to explicitly set them using @samp{--header=Header-Line}.
+Wget will use whatever string is passed to it after @samp{--method} as the HTTP
+Method to the server.
+
+@item --body-data=@var{Data-String}
+@itemx --body-file=@var{Data-File}
+Must be set when additional data needs to be sent to the server along with the
+Method specified using @samp{--method}. @samp{--body-data} sends @var{string} as
+data, whereas @samp{--body-file} sends the contents of @var{file}. Other than that,
+they work in exactly the same way.
+
+Currently, @samp{--body-file} is @emph{not} for transmitting files as a whole.
+Wget does not currently support @code{multipart/form-data} for transmitting data;
+only @code{application/x-www-form-urlencoded}. In the future, this may be changed
+so that wget sends the @samp{--body-file} as a complete file instead of sending its
+contents to the server. Please be aware that Wget needs to know the contents of
+BODY Data in advance, and hence the argument to @samp{--body-file} should be a
+regular file. See @samp{--post-file} for a more detailed explanation.
+Only one of @samp{--body-data} and @samp{--body-file} should be specified.
+
+If Wget is redirected after the request is completed, Wget will
+suspend the current method and send a GET request till the redirection
+is completed. This is true for all redirection response codes except
+307 Temporary Redirect which is used to explicitly specify that the
+request method should @emph{not} change. Another exception is when
+the method is set to @code{POST}, in which case the redirection rules
+specified under @samp{--post-data} are followed.
+
@cindex Content-Disposition
@item --content-disposition
@code{Content-Disposition} headers to describe what the name of a
downloaded file should be.
+@cindex Content On Error
+@item --content-on-error
+
+If this is set to on, wget will not skip the content when the server responds
+with a http status code that indicates error.
+
@cindex Trust server names
@item --trust-server-names
@cindex SSL protocol, choose
@item --secure-protocol=@var{protocol}
Choose the secure protocol to be used. Legal values are @samp{auto},
-@samp{SSLv2}, @samp{SSLv3}, and @samp{TLSv1}. If @samp{auto} is used,
-the SSL library is given the liberty of choosing the appropriate
+@samp{SSLv2}, @samp{SSLv3}, @samp{TLSv1} and @samp{PFS}. If @samp{auto}
+is used, the SSL library is given the liberty of choosing the appropriate
protocol automatically, which is achieved by sending an SSLv2 greeting
and announcing support for SSLv3 and TLSv1. This is the default.
Specifying @samp{SSLv2}, @samp{SSLv3}, or @samp{TLSv1} forces the use
of the corresponding protocol. This is useful when talking to old and
-buggy SSL server implementations that make it hard for OpenSSL to
-choose the correct protocol version. Fortunately, such servers are
-quite rare.
+buggy SSL server implementations that make it hard for the underlying
+SSL library to choose the correct protocol version. Fortunately, such
+servers are quite rare.
+
+Specifying @samp{PFS} enforces the use of the so-called Perfect Forward
+Security cipher suites. In short, PFS adds security by creating a one-time
+key for each SSL connection. It has a bit more CPU impact on client and server.
+We use known to be secure ciphers (e.g. no MD4) and the TLS protocol.
+
+@item --https-only
+When in recursive mode, only HTTPS links are followed.
@cindex SSL certificate, check
@item --no-check-certificate
systems that support @file{/dev/random}.
@end table
+@cindex WARC
+@table @samp
+@item --warc-file=@var{file}
+Use @var{file} as the destination WARC file.
+
+@item --warc-header=@var{string}
+Use @var{string} into as the warcinfo record.
+
+@item --warc-max-size=@var{size}
+Set the maximum size of the WARC files to @var{size}.
+
+@item --warc-cdx
+Write CDX index files.
+
+@item --warc-dedup=@var{file}
+Do not store records listed in this CDX file.
+
+@item --no-warc-compression
+Do not compress WARC files with GZIP.
+
+@item --no-warc-digests
+Do not calculate SHA1 digests.
+
+@item --no-warc-keep-log
+Do not store the log file in a WARC record.
+
+@item --warc-tempdir=@var{dir}
+Specify the location for temporary files created by the WARC writer.
+@end table
+
@node FTP Options, Recursive Retrieval Options, HTTPS (SSL/TLS) Options, Invoking
@section FTP Options
passive FTP doesn't. If you suspect this to be the case, use this
option, or set @code{passive_ftp=off} in your init file.
+@cindex file permissions
+@item --preserve-permissions
+Preserve remote file permissions instead of permissions set by umask.
+
@cindex symbolic links, retrieving
@item --retr-symlinks
Usually, when retrieving @sc{ftp} directories recursively and a symbolic
@item -r
@itemx --recursive
Turn on recursive retrieving. @xref{Recursive Download}, for more
-details.
+details. The default maximum depth is 5.
@item -l @var{depth}
@itemx --level=@var{depth}
Specify recursion maximum depth level @var{depth} (@pxref{Recursive
-Download}). The default maximum depth is 5.
+Download}).
@cindex proxy filling
@cindex delete after retrieval
any of the wildcard characters, @samp{*}, @samp{?}, @samp{[} or
@samp{]}, appear in an element of @var{acclist} or @var{rejlist},
it will be treated as a pattern, rather than a suffix.
+In this case, you have to enclose the pattern into quotes to prevent
+your shell from expanding it, like in @samp{-A "*.mp3"} or @samp{-A '*.mp3'}.
+
+@item --accept-regex @var{urlregex}
+@itemx --reject-regex @var{urlregex}
+Specify a regular expression to accept or reject the complete URL.
+
+@item --regex-type @var{regextype}
+Specify the regular expression type. Possible types are @samp{posix} or
+@samp{pcre}. Note that to be able to use @samp{pcre} type, wget has to be
+compiled with libpcre support.
@item -D @var{domain-list}
@itemx --domains=@var{domain-list}
Ignore case when matching files and directories. This influences the
behavior of -R, -A, -I, and -X options, as well as globbing
implemented when downloading from FTP sites. For example, with this
-option, @samp{-A *.txt} will match @samp{file1.txt}, but also
+option, @samp{-A "*.txt"} will match @samp{file1.txt}, but also
@samp{file2.TXT}, @samp{file3.TxT}, and so on.
+The quotes in the example are to prevent the shell from expanding the
+pattern.
@item -H
@itemx --span-hosts
@item -A @var{acclist}
@itemx --accept @var{acclist}
@itemx accept = @var{acclist}
+@itemx --accept-regex @var{urlregex}
+@itemx accept-regex = @var{urlregex}
The argument to @samp{--accept} option is a list of file suffixes or
patterns that Wget will download during recursive retrieval. A suffix
is the ending part of a file, and consists of ``normal'' letters,
Of course, any number of suffixes and patterns can be combined into a
comma-separated list, and given as an argument to @samp{-A}.
+The argument to @samp{--accept-regex} option is a regular expression which
+is matched against the complete URL.
+
@cindex reject wildcards
@cindex reject suffixes
@cindex wildcards, reject
@item -R @var{rejlist}
@itemx --reject @var{rejlist}
@itemx reject = @var{rejlist}
+@itemx --reject-regex @var{urlregex}
+@itemx reject-regex = @var{urlregex}
The @samp{--reject} option works the same way as @samp{--accept}, only
its logic is the reverse; Wget will download all files @emph{except} the
ones matching the suffixes (or patterns) in the list.
expansion by the shell.
@end table
+The argument to @samp{--accept-regex} option is a regular expression which
+is matched against the complete URL.
+
@noindent
The @samp{-A} and @samp{-R} options may be combined to achieve even
better fine-tuning of which files to retrieve. E.g. @samp{wget -A
Enable/disable saving pre-converted files with the suffix
@samp{.orig}---the same as @samp{-K} (which enables it).
-@c @item backups = @var{number}
-@c #### Document me!
-@c
+@item backups = @var{number}
+Use up to @var{number} backups for a file. Backups are rotated by
+adding an incremental counter that starts at @samp{1}. The default is
+@samp{0}.
+
@item base = @var{string}
Consider relative @sc{url}s in input files (specified via the
@samp{input} command or the @samp{--input-file}/@samp{-i} option,
Change setting of passive @sc{ftp}, equivalent to the
@samp{--passive-ftp} option.
-@itemx password = @var{string}
+@item password = @var{string}
Specify password @var{string} for both @sc{ftp} and @sc{http} file retrieval.
This command can be overridden using the @samp{ftp_password} and
@samp{http_password} command for @sc{ftp} and @sc{http} respectively.
Choose whether or not to print the @sc{http} and @sc{ftp} server
responses---the same as @samp{-S}.
+@item show_all_dns_entries = on/off
+When a DNS name is resolved, show all the IP addresses, not just the first
+three.
+
@item span_hosts = on/off
Same as @samp{-H}.
information from the Web, their users connect and retrieve remote data
using an authorized proxy.
+@c man begin ENVIRONMENT
Wget supports proxies for both @sc{http} and @sc{ftp} retrievals. The
standard way to specify proxy location, which Wget recognizes, is using
the following environment variables:
-@table @code
+@table @env
@item http_proxy
@itemx https_proxy
-If set, the @code{http_proxy} and @code{https_proxy} variables should
+If set, the @env{http_proxy} and @env{https_proxy} variables should
contain the @sc{url}s of the proxies for @sc{http} and @sc{https}
connections respectively.
@item ftp_proxy
This variable should contain the @sc{url} of the proxy for @sc{ftp}
-connections. It is quite common that @code{http_proxy} and
-@code{ftp_proxy} are set to the same @sc{url}.
+connections. It is quite common that @env{http_proxy} and
+@env{ftp_proxy} are set to the same @sc{url}.
@item no_proxy
This variable should contain a comma-separated list of domain extensions
proxy should @emph{not} be used for. For instance, if the value of
-@code{no_proxy} is @samp{.mit.edu}, proxy will not be used to retrieve
+@env{no_proxy} is @samp{.mit.edu}, proxy will not be used to retrieve
documents from MIT.
@end table
+@c man end
In addition to the environment variables, proxy location and settings
may be specified from within Wget itself.
@table @samp
-@itemx --no-proxy
+@item --no-proxy
@itemx proxy = on/off
This option and the corresponding command may be used to suppress the
use of proxy, even if the appropriate environment variables are set.
GNU Wget was written by Hrvoje Nik@v{s}i@'{c} @email{hniksic@@xemacs.org},
@end iftex
@ifnottex
-GNU Wget was written by Hrvoje Niksic @email{hniksic@@xemacs.org},
+GNU Wget was written by Hrvoje Niksic @email{hniksic@@xemacs.org}.
@end ifnottex
-and it is currently maintained by Micah Cowan @email{micah@@cowan.name}.
However, the development of Wget could never have gone as far as it has, were
it not for the help of many people, either with bug reports, feature proposals,
@appendix Copying this manual
@menu
-* GNU Free Documentation License:: Licnse for copying this manual.
+* GNU Free Documentation License:: License for copying this manual.
@end menu
@node GNU Free Documentation License, , Copying this manual, Copying this manual
projects / wget / blobdiff