@c %**start of header
@setfilename wget.info
@include version.texi
-@set UPDATED Jul 2006
+@set UPDATED Jun 2008
@settitle GNU Wget @value{VERSION} Manual
@c Disable the monstrous rectangles beside overfull hbox-es.
@finalout
* Wget: (wget). The non-interactive network downloader.
@end direntry
-@ifnottex
-This file documents the the GNU Wget utility for downloading network
+@copying
+This file documents the GNU Wget utility for downloading network
data.
@c man begin COPYRIGHT
-Copyright @copyright{} 1996--2006 Free Software Foundation, Inc.
+Copyright @copyright{} 1996, 1997, 1998, 1999, 2000, 2001, 2002,
+2003, 2004, 2005, 2006, 2007, 2008 Free Software Foundation, Inc.
+@iftex
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.
+@end iftex
@ignore
Permission is granted to process this file through TeX and print the
copy of the license is included in the section entitled ``GNU Free
Documentation License''.
@c man end
-@end ifnottex
+@end copying
@titlepage
@title GNU Wget @value{VERSION}
@page
@vskip 0pt plus 1filll
-Copyright @copyright{} 1996--2006, Free Software Foundation, Inc.
-
-Permission is granted to copy, distribute and/or modify this document
-under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no
-Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
-copy of the license is included in the section entitled ``GNU Free
-Documentation License''.
+@insertcopying
@end titlepage
+@contents
+
@ifnottex
@node Top
@top Wget @value{VERSION}
-This manual documents version @value{VERSION} of GNU Wget, the freely
-available utility for network downloads.
-
-Copyright @copyright{} 1996--2006 Free Software Foundation, Inc.
+@insertcopying
+@end ifnottex
@menu
* Overview:: Features of Wget.
* Copying this manual:: You may give out copies of Wget and of this manual.
* Concept Index:: Topics covered by this manual.
@end menu
-@end ifnottex
@node Overview
@chapter Overview
@c man end
@end ignore
@c man begin DESCRIPTION
-Wget can follow links in @sc{html} and @sc{xhtml} pages and create local
-versions of remote web sites, fully recreating the directory structure of
-the original site. This is sometimes referred to as ``recursive
-downloading.'' While doing that, Wget respects the Robot Exclusion
-Standard (@file{/robots.txt}). Wget can be instructed to convert the
-links in downloaded @sc{html} files to the local files for offline
-viewing.
+Wget can follow links in @sc{html}, @sc{xhtml}, and @sc{css} pages, to
+create local versions of remote web sites, fully recreating the
+directory structure of the original site. This is sometimes referred to
+as ``recursive downloading.'' While doing that, Wget respects the Robot
+Exclusion Standard (@file{/robots.txt}). Wget can be instructed to
+convert the links in downloaded files to point at the local files, for
+offline viewing.
@c man end
@item
@item
Wget supports proxy servers, which can lighten the network load, speed
-up retrieval and provide access behind firewalls. However, if you are
-behind a firewall that requires that you use a socks style gateway,
-you can get the socks library and build Wget with support for socks.
-Wget uses the passive @sc{ftp} downloading by default, active @sc{ftp}
-being an option.
+up retrieval and provide access behind firewalls. Wget uses the passive
+@sc{ftp} downloading by default, active @sc{ftp} being an option.
@item
Wget supports IP version 6, the next generation of IP. IPv6 is
@cindex input-file
@item -i @var{file}
@itemx --input-file=@var{file}
-Read @sc{url}s from @var{file}. If @samp{-} is specified as
-@var{file}, @sc{url}s are read from the standard input. (Use
-@samp{./-} to read from a file literally named @samp{-}.)
+Read @sc{url}s from a local or external @var{file}. If @samp{-} is
+specified as @var{file}, @sc{url}s are read from the standard input.
+(Use @samp{./-} to read from a file literally named @samp{-}.)
If this function is used, no @sc{url}s need be present on the command
line. If there are @sc{url}s both on the command line and in an input
href="@var{url}">} to the documents or by specifying
@samp{--base=@var{url}} on the command line.
+If the @var{file} is an external one, the document will be automatically
+treated as @samp{html} if the Content-Type matches @samp{text/html}.
+Furthermore, the @var{file}'s location will be implicitly used as base
+href if none was specified.
+
@cindex force html
@item -F
@itemx --force-html
@samp{wget -O - http://foo > file}; @file{file} will be truncated
immediately, and @emph{all} downloaded content will be written there.
+For this reason, @samp{-N} (for timestamp-checking) is not supported
+in combination with @samp{-O}: since @var{file} is always newly
+created, it will always have a very new timestamp. A warning will be
+issued if this combination is used.
+
+Similarly, using @samp{-r} or @samp{-p} with @samp{-O} may not work as
+you expect: Wget won't just download the first file to @var{file} and
+then download the rest to their normal names: @emph{all} downloaded
+content will be placed in @var{file}. This was disabled in version
+1.11, but has been reinstated (with a warning) in 1.11.2, as there are
+some cases where this behavior can actually have some use.
+
Note that a combination with @samp{-k} is only permitted when
-downloading a single document, and combination with any of @samp{-r},
-@samp{-p}, or @samp{-N} is not allowed.
+downloading a single document, as in that case it will just convert
+all relative URIs to external ones; @samp{-k} makes no sense for
+multiple URIs when they're all being downloaded to a single file.
@cindex clobbering, file
@cindex downloading multiple times
same time. Neither option is available in Wget compiled without IPv6
support.
-@item --prefer-family=IPv4/IPv6/none
+@item --prefer-family=none/IPv4/IPv6
When given a choice of several addresses, connect to the addresses
-with specified address family first. IPv4 addresses are preferred by
-default.
+with specified address family first. The address order returned by
+DNS is used without change by default.
This avoids spurious errors and connect attempts when accessing hosts
that resolve to both IPv6 and IPv4 addresses from IPv4 networks. For
using the @samp{--ftp-user} and @samp{--ftp-password} options for
@sc{ftp} connections and the @samp{--http-user} and @samp{--http-password}
options for @sc{http} connections.
+
+@item --ask-password
+Prompt for a password for each connection established. Cannot be specified
+when @samp{--password} is being used, because they are mutually exclusive.
@end table
@node Directory Options
@section Directory Options
-@table @samp
+@table @samp
@item -nd
@itemx --no-directories
Do not create a hierarchy of directories when retrieving recursively.
version of the file will be saved as @file{@var{X}.orig} (@pxref{Recursive
Retrieval Options}).
+As of version 1.12, Wget will also ensure that any downloaded files of
+type @samp{text/css} end in the suffix @samp{.css}. Obviously, this
+makes the name @samp{--html-extension} misleading; a better name is
+expected to be offered as an alternative in the near future.
+
@cindex http user
@cindex http password
@cindex authentication
them (and neither will browsers) and the @file{cookies.txt} file will
be empty. In that case use @samp{--keep-session-cookies} along with
@samp{--save-cookies} to force saving of session cookies.
+
+@cindex Content-Disposition
+@item --content-disposition
+
+If this is set to on, experimental (not fully-functional) support for
+@code{Content-Disposition} headers is enabled. This can currently result in
+extra round-trips to the server for a @code{HEAD} request, and is known
+to suffer from a few bugs, which is why it is not currently enabled by default.
+
+This option is useful for some file-downloading CGI programs that use
+@code{Content-Disposition} headers to describe what the name of a
+downloaded file should be.
+
+@cindex authentication
+@item --auth-no-challenge
+
+If this option is given, Wget will send Basic HTTP authentication
+information (plaintext username and password) for all requests, just
+like Wget 1.10.2 and prior did by default.
+
+Use of this option is not recommended, and is intended only to support
+some few obscure servers, which never send HTTP authentication
+challenges, but accept unsolicited auth info, say, in addition to
+form-based authentication.
+
@end table
@node HTTPS (SSL/TLS) Options
@sc{http} or @sc{ftp} server), following links and directory structure.
We refer to this as to @dfn{recursive retrieval}, or @dfn{recursion}.
-With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} from
-the given @sc{url}, documents, retrieving the files the @sc{html}
-document was referring to, through markup like @code{href}, or
-@code{src}. If the freshly downloaded file is also of type
-@code{text/html} or @code{application/xhtml+xml}, it will be parsed and
-followed further.
+With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} or
+@sc{css} from the given @sc{url}, retrieving the files the document
+refers to, through markup like @code{href} or @code{src}, or @sc{css}
+@sc{uri} values specified using the @samp{url()} functional notation.
+If the freshly downloaded file is also of type @code{text/html},
+@code{application/xhtml+xml}, or @code{text/css}, it will be parsed
+and followed further.
-Recursive retrieval of @sc{http} and @sc{html} content is
+Recursive retrieval of @sc{http} and @sc{html}/@sc{css} content is
@dfn{breadth-first}. This means that Wget first downloads the requested
-@sc{html} document, then the documents linked from that document, then the
+document, then the documents linked from that document, then the
documents linked by them, and so on. In other words, Wget first
downloads the documents at depth 1, then those at depth 2, and so on
until the specified maximum depth.
expansion by the shell.
@end table
+@noindent
The @samp{-A} and @samp{-R} options may be combined to achieve even
better fine-tuning of which files to retrieve. E.g. @samp{wget -A
"*zelazny*" -R .ps} will download all the files having @samp{zelazny} as
a part of their name, but @emph{not} the PostScript files.
Note that these two options do not affect the downloading of @sc{html}
-files; Wget must load all the @sc{html}s to know where to go at
-all---recursive retrieval would make no sense otherwise.
+files (as determined by a @samp{.htm} or @samp{.html} filename
+prefix). This behavior may not be desirable for all users, and may be
+changed for future versions of Wget.
+
+Note, too, that query strings (strings at the end of a URL beginning
+with a question mark (@samp{?}) are not included as part of the
+filename for accept/reject rules, even though these will actually
+contribute to the name chosen for the local file. It is expected that
+a future version of Wget will provide an option to allow matching
+against query strings.
+
+Finally, it's worth noting that the accept/reject lists are matched
+@emph{twice} against downloaded files: once against the URL's filename
+portion, to determine if the file should be downloaded in the first
+place; then, after it has been accepted and successfully downloaded,
+the local file's name is also checked against the accept/reject lists
+to see if it should be removed. The rationale was that, since
+@samp{.htm} and @samp{.html} files are always downloaded regardless of
+accept/reject rules, they should be removed @emph{after} being
+downloaded and scanned for links, if they did match the accept/reject
+lists. However, this can lead to unexpected results, since the local
+filenames can differ from the original URL filenames in the following
+ways, all of which can change whether an accept/reject rule matches:
+
+@itemize @bullet
+@item
+If the local file already exists and @samp{--no-directories} was
+specified, a numeric suffix will be appended to the original name.
+@item
+If @samp{--html-extension} was specified, the local filename will have
+@samp{.html} appended to it. If Wget is invoked with @samp{-E -A.php},
+a filename such as @samp{index.php} will match be accepted, but upon
+download will be named @samp{index.php.html}, which no longer matches,
+and so the file will be deleted.
+@item
+Query strings do not contribute to URL matching, but are included in
+local filenames, and so @emph{do} contribute to filename matching.
+@end itemize
+
+@noindent
+This behavior, too, is considered less-than-desirable, and may change
+in a future version of Wget.
@node Directory-Based Limits
@section Directory-Based Limits
Essentially, @samp{--no-parent} is similar to
@samp{-I/~luzer/my-archive}, only it handles redirections in a more
intelligent fashion.
+
+@strong{Note} that, for HTTP (and HTTPS), the trailing slash is very
+important to @samp{--no-parent}. HTTP has no concept of a ``directory''---Wget
+relies on you to indicate what's a directory and what isn't. In
+@samp{http://foo/bar/}, Wget will consider @samp{bar} to be a
+directory, while in @samp{http://foo/bar} (no trailing slash),
+@samp{bar} will be considered a filename (so @samp{--no-parent} would be
+meaningless, as its parent is @samp{/}).
@end table
@node Relative Links
Set the connect timeout---the same as @samp{--connect-timeout}.
@item content_disposition = on/off
-If this is set to on, experimental (not fully-functional) support for
-@samp{Content-Disposition} headers is enabled. This can currently result in
-extra round-trips to the server for a @samp{HEAD} request, which is why
-it is not currently enabled by default.
-
-This option is useful for some file-downloading CGI programs that use
-@samp{Content-Disposition} headers to describe what the name of a
-downloaded file should be.
+Turn on recognition of the (non-standard) @samp{Content-Disposition}
+HTTP header---if set to @samp{on}, the same as @samp{--content-disposition}.
@item continue = on/off
If set to on, force continuation of preexistent partially retrieved
Turn globbing on/off---the same as @samp{--glob} and @samp{--no-glob}.
@item header = @var{string}
-Define a header for HTTP doewnloads, like using
+Define a header for HTTP downloads, like using
@samp{--header=@var{string}}.
@item html_extension = on/off
Add a @samp{.html} extension to @samp{text/html} or
-@samp{application/xhtml+xml} files without it, like @samp{-E}.
+@samp{application/xhtml+xml} files without it, or a @samp{.css}
+extension to @samp{text/css} files without it, like @samp{-E}.
@item http_keep_alive = on/off
Turn the keep-alive feature on or off (defaults to on). Turning it
@var{file} in the request body. The same as
@samp{--post-file=@var{file}}.
-@item prefer_family = IPv4/IPv6/none
+@item prefer_family = none/IPv4/IPv6
When given a choice of several addresses, connect to the addresses
-with specified address family first. IPv4 addresses are preferred by
-default. The same as @samp{--prefer-family}, which see for a detailed
-discussion of why this is useful.
+with specified address family first. The address order returned by
+DNS is used without change by default. The same as @samp{--prefer-family},
+which see for a detailed discussion of why this is useful.
@item private_key = @var{file}
Set the private key file to @var{file}. The same as
@end example
@item
-The same as the above, but convert the links in the @sc{html} files to
+The same as the above, but convert the links in the downloaded files to
point to local files, so you can view the documents off-line:
@example
* Distribution:: Getting the latest version.
* Web Site:: GNU Wget's presence on the World Wide Web.
* Mailing List:: Wget mailing list for announcements and discussion.
+* Internet Relay Chat:: Wget's presence on IRC.
* Reporting Bugs:: How and where to report bugs.
* Portability:: The systems Wget works on.
* Signals:: Signal-handling performed by Wget.
Another mailing list is at @email{wget-patches@@sunsite.dk}, and is
used to submit patches for review by Wget developers. A ``patch'' is
a textual representation of change to source code, readable by both
-humans and programs. The file @file{PATCHES} that comes with Wget
+humans and programs. The
+@url{http://wget.addictivecode.org/PatchGuidelines} page
covers the creation and submitting of patches in detail. Please don't
send general suggestions or bug reports to @samp{wget-patches}; use it
only for patch submissions.
@url{http://news.gmane.org/gmane.comp.web.wget.patches}.
Finally, there is the @email{wget-notify@@addictivecode.org} mailing
-list. This is a non-discussion list that receives commit notifications
-from the source repository, and also bug report-change notifications.
-This is the highest-traffic list for Wget, and is recommended only for
-people who are seriously interested in ongoing Wget development.
-Subscription is through the @code{mailman} interface at
+list. This is a non-discussion list that receives bug report-change
+notifications from the bug-tracker. Unlike for the other mailing lists,
+subscription is through the @code{mailman} interface at
@url{http://addictivecode.org/mailman/listinfo/wget-notify}.
+@node Internet Relay Chat
+@section Internet Relay Chat
+@cindex Internet Relay Chat
+@cindex IRC
+@cindex #wget
+
+In addition to the mailinglists, we also have a support channel set up
+via IRC at @code{irc.freenode.org}, @code{#wget}. Come check it out!
+
@node Reporting Bugs
@section Reporting Bugs
@cindex bugs
@item
Try to repeat the bug in as simple circumstances as possible. E.g. if
-Wget crashes while downloading @samp{wget -rl0 -kKE -t5 -Y0
+Wget crashes while downloading @samp{wget -rl0 -kKE -t5 --no-proxy
http://yoyodyne.com -o /tmp/log}, you should try to see if the crash is
repeatable, and if will occur with a simpler set of options. You might
even try to start the download at the page where the crash occurred to
authentication.
@item
-Mauro Tortonesi---Improved IPv6 support, adding support for dual
+Mauro Tortonesi---improved IPv6 support, adding support for dual
family systems. Refactored and enhanced FTP IPv6 code. Maintained GNU
Wget from 2004--2007.
@item
-Christopher G.@: Lewis---Maintenance of the Windows version of GNU WGet.
+Christopher G.@: Lewis---maintenance of the Windows version of GNU WGet.
@item
-Gisle Vanem---Many helpful patches and improvements, especially for
+Gisle Vanem---many helpful patches and improvements, especially for
Windows and MS-DOS support.
+@item
+Ralf Wildenhues---contributed patches to convert Wget to use Automake as
+part of its build process, and various bugfixes.
+
+@item
+Steven Schubiger---Many helpful patches, bugfixes and improvements.
+Notably, conversion of Wget to use the Gnulib quotes and quoteargs
+modules, and the addition of password prompts at the console, via the
+Gnulib getpasswd-gnu module.
+
+@item
+Ted Mielczarek---donated support for CSS.
+
@item
People who provided donations for development---including Brian Gough.
@end itemize
Daniel Bodea,
Mark Boyns,
John Burden,
+Julien Buty,
Wanderlei Cavassin,
Gilles Cedoc,
Tim Charron,
Ahmon Dancy,
Andrew Davison,
Bertrand Demiddelaer,
+Alexander Dergachev,
Andrew Deryabin,
Ulrich Drepper,
Marc Duponcheel,
Aleksandar Erkalovic,
@end ifnottex
Andy Eskilsson,
+@iftex
+Jo@~{a}o Ferreira,
+@end iftex
+@ifnottex
+Joao Ferreira,
+@end ifnottex
Christian Fraenkel,
David Fritz,
+Mike Frysinger,
Charles C.@: Fu,
FUJISHIMA Satsuki,
Masashi Fujita,
Marcel Gerrits,
Lemble Gregory,
Hans Grobler,
+Alain Guibert,
Mathieu Guillaume,
Aaron Hawley,
Jochen Hein,
Karl Heuer,
+Madhusudan Hosaagrahara,
HIROSE Masaaki,
Ulf Harnhammar,
Gregor Hoffleit,
Aurelien Marchand,
Matthew J.@: Mellon,
Jordan Mendelson,
+Ted Mielczarek,
Lin Zhe Min,
Jan Minar,
Tim Mooney,
Simon Munton,
Charlie Negyesi,
R.@: K.@: Owen,
+Jim Paris,
+Kenny Parnell,
Leonid Petrov,
Simone Piunno,
Andrew Pollock,
Heinz Salzmann,
Robert Schmidt,
Nicolas Schodet,
+Benno Schulenberg,
Andreas Schwab,
Steven M.@: Schweda,
Chris Seawood,
+Pranab Shenoy,
Dennis Smit,
Toomas Soome,
Tage Stabell-Kulo,
Mauro Tortonesi,
Dave Turner,
Gisle Vanem,
+Rabin Vincent,
Russell Vincent,
@iftex
@v{Z}eljko Vrba,
projects / wget / blobdiff