@c %**start of header
@setfilename wget.info
@include version.texi
-@set UPDATED Jul 2006
+@set UPDATED Mar 2008
@settitle GNU Wget @value{VERSION} Manual
@c Disable the monstrous rectangles beside overfull hbox-es.
@finalout
* Wget: (wget). The non-interactive network downloader.
@end direntry
-@ifnottex
-This file documents the the GNU Wget utility for downloading network
+@copying
+This file documents the GNU Wget utility for downloading network
data.
@c man begin COPYRIGHT
-Copyright @copyright{} 1996--2006 Free Software Foundation, Inc.
+Copyright @copyright{} 1996, 1997, 1998, 1999, 2000, 2001, 2002,
+2003, 2004, 2005, 2006, 2007, 2008 Free Software Foundation, Inc.
+@iftex
Permission is granted to make and distribute verbatim copies of
this manual provided the copyright notice and this permission notice
are preserved on all copies.
+@end iftex
@ignore
Permission is granted to process this file through TeX and print the
copy of the license is included in the section entitled ``GNU Free
Documentation License''.
@c man end
-@end ifnottex
+@end copying
@titlepage
@title GNU Wget @value{VERSION}
@page
@vskip 0pt plus 1filll
-Copyright @copyright{} 1996--2006, Free Software Foundation, Inc.
-
-Permission is granted to copy, distribute and/or modify this document
-under the terms of the GNU Free Documentation License, Version 1.2 or
-any later version published by the Free Software Foundation; with no
-Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A
-copy of the license is included in the section entitled ``GNU Free
-Documentation License''.
+@insertcopying
@end titlepage
+@contents
+
@ifnottex
@node Top
@top Wget @value{VERSION}
-This manual documents version @value{VERSION} of GNU Wget, the freely
-available utility for network downloads.
-
-Copyright @copyright{} 1996--2006 Free Software Foundation, Inc.
+@insertcopying
+@end ifnottex
@menu
* Overview:: Features of Wget.
* Copying this manual:: You may give out copies of Wget and of this manual.
* Concept Index:: Topics covered by this manual.
@end menu
-@end ifnottex
@node Overview
@chapter Overview
@c man end
@end ignore
@c man begin DESCRIPTION
-Wget can follow links in @sc{html} and @sc{xhtml} pages and create local
-versions of remote web sites, fully recreating the directory structure of
-the original site. This is sometimes referred to as ``recursive
-downloading.'' While doing that, Wget respects the Robot Exclusion
-Standard (@file{/robots.txt}). Wget can be instructed to convert the
-links in downloaded @sc{html} files to the local files for offline
-viewing.
+Wget can follow links in @sc{html}, @sc{xhtml}, and @sc{css} pages, to
+create local versions of remote web sites, fully recreating the
+directory structure of the original site. This is sometimes referred to
+as ``recursive downloading.'' While doing that, Wget respects the Robot
+Exclusion Standard (@file{/robots.txt}). Wget can be instructed to
+convert the links in downloaded files to point at the local files, for
+offline viewing.
@c man end
@item
@item
Wget supports proxy servers, which can lighten the network load, speed
-up retrieval and provide access behind firewalls. However, if you are
-behind a firewall that requires that you use a socks style gateway,
-you can get the socks library and build Wget with support for socks.
-Wget uses the passive @sc{ftp} downloading by default, active @sc{ftp}
-being an option.
+up retrieval and provide access behind firewalls. Wget uses the passive
+@sc{ftp} downloading by default, active @sc{ftp} being an option.
@item
Wget supports IP version 6, the next generation of IP. IPv6 is
version of the file will be saved as @file{@var{X}.orig} (@pxref{Recursive
Retrieval Options}).
+As of version 1.12, Wget will also ensure that any downloaded files of
+type @samp{text/css} end in the suffix @samp{.css}. Obviously, this
+makes the name @samp{--html-extension} misleading; a better name is
+expected to be offered as an alternative in the near future.
+
@cindex http user
@cindex http password
@cindex authentication
them (and neither will browsers) and the @file{cookies.txt} file will
be empty. In that case use @samp{--keep-session-cookies} along with
@samp{--save-cookies} to force saving of session cookies.
+
+@cindex Content-Disposition
+@item --content-disposition
+
+If this is set to on, experimental (not fully-functional) support for
+@code{Content-Disposition} headers is enabled. This can currently result in
+extra round-trips to the server for a @code{HEAD} request, and is known
+to suffer from a few bugs, which is why it is not currently enabled by default.
+
+This option is useful for some file-downloading CGI programs that use
+@code{Content-Disposition} headers to describe what the name of a
+downloaded file should be.
+
+@cindex authentication
+@item --auth-no-challenge
+
+If this option is given, Wget will send Basic HTTP authentication
+information (plaintext username and password) for all requests, just
+like Wget 1.10.2 and prior did by default.
+
+Use of this option is not recommended, and is intended only to support
+some few obscure servers, which never send HTTP authentication
+challenges, but accept unsolicited auth info, say, in addition to
+form-based authentication.
+
@end table
@node HTTPS (SSL/TLS) Options
@sc{http} or @sc{ftp} server), following links and directory structure.
We refer to this as to @dfn{recursive retrieval}, or @dfn{recursion}.
-With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} from
-the given @sc{url}, documents, retrieving the files the @sc{html}
-document was referring to, through markup like @code{href}, or
-@code{src}. If the freshly downloaded file is also of type
-@code{text/html} or @code{application/xhtml+xml}, it will be parsed and
-followed further.
+With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} or
+@sc{css} from the given @sc{url}, retrieving the files the document
+refers to, through markup like @code{href} or @code{src}, or @sc{css}
+@sc{uri} values specified using the @samp{url()} functional notation.
+If the freshly downloaded file is also of type @code{text/html},
+@code{application/xhtml+xml}, or @code{text/css}, it will be parsed
+and followed further.
-Recursive retrieval of @sc{http} and @sc{html} content is
+Recursive retrieval of @sc{http} and @sc{html}/@sc{css} content is
@dfn{breadth-first}. This means that Wget first downloads the requested
-@sc{html} document, then the documents linked from that document, then the
+document, then the documents linked from that document, then the
documents linked by them, and so on. In other words, Wget first
downloads the documents at depth 1, then those at depth 2, and so on
until the specified maximum depth.
expansion by the shell.
@end table
+@noindent
The @samp{-A} and @samp{-R} options may be combined to achieve even
better fine-tuning of which files to retrieve. E.g. @samp{wget -A
"*zelazny*" -R .ps} will download all the files having @samp{zelazny} as
a part of their name, but @emph{not} the PostScript files.
Note that these two options do not affect the downloading of @sc{html}
-files; Wget must load all the @sc{html}s to know where to go at
-all---recursive retrieval would make no sense otherwise.
+files (as determined by a @samp{.htm} or @samp{.html} filename
+prefix). This behavior may not be desirable for all users, and may be
+changed for future versions of Wget.
+
+Note, too, that query strings (strings at the end of a URL beginning
+with a question mark (@samp{?}) are not included as part of the
+filename for accept/reject rules, even though these will actually
+contribute to the name chosen for the local file. It is expected that
+a future version of Wget will provide an option to allow matching
+against query strings.
+
+Finally, it's worth noting that the accept/reject lists are matched
+@emph{twice} against downloaded files: once against the URL's filename
+portion, to determine if the file should be downloaded in the first
+place; then, after it has been accepted and successfully downloaded,
+the local file's name is also checked against the accept/reject lists
+to see if it should be removed. The rationale was that, since
+@samp{.htm} and @samp{.html} files are always downloaded regardless of
+accept/reject rules, they should be removed @emph{after} being
+downloaded and scanned for links, if they did match the accept/reject
+lists. However, this can lead to unexpected results, since the local
+filenames can differ from the original URL filenames in the following
+ways, all of which can change whether an accept/reject rule matches:
+
+@itemize @bullet
+@item
+If the local file already exists and @samp{--no-directories} was
+specified, a numeric suffix will be appended to the original name.
+@item
+If @samp{--html-extension} was specified, the local filename will have
+@samp{.html} appended to it. If Wget is invoked with @samp{-E -A.php},
+a filename such as @samp{index.php} will match be accepted, but upon
+download will be named @samp{index.php.html}, which no longer matches,
+and so the file will be deleted.
+@item
+Query strings do not contribute to URL matching, but are included in
+local filenames, and so @emph{do} contribute to filename matching.
+@end itemize
+
+@noindent
+This behavior, too, is considered less-than-desirable, and may change
+in a future version of Wget.
@node Directory-Based Limits
@section Directory-Based Limits
Essentially, @samp{--no-parent} is similar to
@samp{-I/~luzer/my-archive}, only it handles redirections in a more
intelligent fashion.
+
+@strong{Note} that, for HTTP (and HTTPS), the trailing slash is very
+important to @samp{--no-parent}. HTTP has no concept of a ``directory''---Wget
+relies on you to indicate what's a directory and what isn't. In
+@samp{http://foo/bar/}, Wget will consider @samp{bar} to be a
+directory, while in @samp{http://foo/bar} (no trailing slash),
+@samp{bar} will be considered a filename (so @samp{--no-parent} would be
+meaningless, as its parent is @samp{/}).
@end table
@node Relative Links
Set the connect timeout---the same as @samp{--connect-timeout}.
@item content_disposition = on/off
-If this is set to on, experimental (not fully-functional) support for
-@samp{Content-Disposition} headers is enabled. This can currently result in
-extra round-trips to the server for a @samp{HEAD} request, which is why
-it is not currently enabled by default.
-
-This option is useful for some file-downloading CGI programs that use
-@samp{Content-Disposition} headers to describe what the name of a
-downloaded file should be.
+Turn on recognition of the (non-standard) @samp{Content-Disposition}
+HTTP header---if set to @samp{on}, the same as @samp{--content-disposition}.
@item continue = on/off
If set to on, force continuation of preexistent partially retrieved
Turn globbing on/off---the same as @samp{--glob} and @samp{--no-glob}.
@item header = @var{string}
-Define a header for HTTP doewnloads, like using
+Define a header for HTTP downloads, like using
@samp{--header=@var{string}}.
@item html_extension = on/off
Add a @samp{.html} extension to @samp{text/html} or
-@samp{application/xhtml+xml} files without it, like @samp{-E}.
+@samp{application/xhtml+xml} files without it, or a @samp{.css}
+extension to @samp{text/css} files without it, like @samp{-E}.
@item http_keep_alive = on/off
Turn the keep-alive feature on or off (defaults to on). Turning it
@end example
@item
-The same as the above, but convert the links in the @sc{html} files to
+The same as the above, but convert the links in the downloaded files to
point to local files, so you can view the documents off-line:
@example
* Distribution:: Getting the latest version.
* Web Site:: GNU Wget's presence on the World Wide Web.
* Mailing List:: Wget mailing list for announcements and discussion.
+* Internet Relay Chat:: Wget's presence on IRC.
* Reporting Bugs:: How and where to report bugs.
* Portability:: The systems Wget works on.
* Signals:: Signal-handling performed by Wget.
Another mailing list is at @email{wget-patches@@sunsite.dk}, and is
used to submit patches for review by Wget developers. A ``patch'' is
a textual representation of change to source code, readable by both
-humans and programs. The file @file{PATCHES} that comes with Wget
+humans and programs. The
+@url{http://wget.addictivecode.org/PatchGuidelines} page
covers the creation and submitting of patches in detail. Please don't
send general suggestions or bug reports to @samp{wget-patches}; use it
only for patch submissions.
Subscription is through the @code{mailman} interface at
@url{http://addictivecode.org/mailman/listinfo/wget-notify}.
+@node Internet Relay Chat
+@section Internet Relay Chat
+@cindex Internet Relay Chat
+@cindex IRC
+@cindex #wget
+
+While, at the time of this writing, there is very low activity, we do
+have a support channel set up via IRC at @code{irc.freenode.org},
+@code{#wget}. Come check it out!
+
@node Reporting Bugs
@section Reporting Bugs
@cindex bugs
@item
Try to repeat the bug in as simple circumstances as possible. E.g. if
-Wget crashes while downloading @samp{wget -rl0 -kKE -t5 -Y0
+Wget crashes while downloading @samp{wget -rl0 -kKE -t5 --no-proxy
http://yoyodyne.com -o /tmp/log}, you should try to see if the crash is
repeatable, and if will occur with a simpler set of options. You might
even try to start the download at the page where the crash occurred to
authentication.
@item
-Mauro Tortonesi---Improved IPv6 support, adding support for dual
+Mauro Tortonesi---improved IPv6 support, adding support for dual
family systems. Refactored and enhanced FTP IPv6 code. Maintained GNU
Wget from 2004--2007.
@item
-Christopher G.@: Lewis---Maintenance of the Windows version of GNU WGet.
+Christopher G.@: Lewis---maintenance of the Windows version of GNU WGet.
@item
-Gisle Vanem---Many helpful patches and improvements, especially for
+Gisle Vanem---many helpful patches and improvements, especially for
Windows and MS-DOS support.
@item
-Ralf Wildenhues---Contributed patches to convert Wget to use Automake as
+Ralf Wildenhues---contributed patches to convert Wget to use Automake as
part of its build process, and various bugfixes.
+@item
+Ted Mielczarek---donated support for CSS.
+
@item
People who provided donations for development---including Brian Gough.
@end itemize
Daniel Bodea,
Mark Boyns,
John Burden,
+Julien Buty,
Wanderlei Cavassin,
Gilles Cedoc,
Tim Charron,
Ahmon Dancy,
Andrew Davison,
Bertrand Demiddelaer,
+Alexander Dergachev,
Andrew Deryabin,
Ulrich Drepper,
Marc Duponcheel,
Aurelien Marchand,
Matthew J.@: Mellon,
Jordan Mendelson,
+Ted Mielczarek,
Lin Zhe Min,
Jan Minar,
Tim Mooney,
Mauro Tortonesi,
Dave Turner,
Gisle Vanem,
+Rabin Vincent,
Russell Vincent,
@iftex
@v{Z}eljko Vrba,
projects / wget / blobdiff