likes to use a few options in addition to @samp{-p}:
@example
-wget -E -H -k -K -nh -p http://@var{site}/@var{document}
+wget -E -H -k -K -p http://@var{site}/@var{document}
@end example
In one case you'll need to add a couple more options. If @var{document}
@item -D @var{domain-list}
@itemx --domains=@var{domain-list}
-Set domains to be accepted and @sc{dns} looked-up, where
-@var{domain-list} is a comma-separated list. Note that it does
-@emph{not} turn on @samp{-H}. This option speeds things up, even if
-only one host is spanned (@pxref{Domain Acceptance}).
+Set domains to be followed. @var{domain-list} is a comma-separated list
+of domains. Note that it does @emph{not} turn on @samp{-H}.
@item --exclude-domains @var{domain-list}
-Exclude the domains given in a comma-separated @var{domain-list} from
-@sc{dns}-lookup (@pxref{Domain Acceptance}).
+Specify the domains that are @emph{not} to be followed.
+(@pxref{Spanning Hosts}).
@cindex follow FTP links
@item --follow-ftp
single page and its requisites, using a commandline like:
@example
-wget -Ga,area -H -k -K -nh -r http://@var{site}/@var{document}
+wget -Ga,area -H -k -K -r http://@var{site}/@var{document}
@end example
However, the author of this option came across a page with tags like
@item -H
@itemx --span-hosts
-Enable spanning across hosts when doing recursive retrieving (@pxref{All
-Hosts}).
+Enable spanning across hosts when doing recursive retrieving
+(@pxref{Spanning Hosts}).
@item -L
@itemx --relative
download (@pxref{Directory-Based Limits} for more details.) Elements of
@var{list} may contain wildcards.
-@item -nh
-@itemx --no-host-lookup
-Disable the time-consuming @sc{dns} lookup of almost all hosts
-(@pxref{Host Checking}).
-
@item -np
@item --no-parent
Do not ever ascend to the parent directory when retrieving recursively.
@cindex recursive retrieval
GNU Wget is capable of traversing parts of the Web (or a single
-@sc{http} or @sc{ftp} server), depth-first following links and directory
-structure. This is called @dfn{recursive} retrieving, or
-@dfn{recursion}.
+@sc{http} or @sc{ftp} server), following links and directory structure.
+We refer to this as to @dfn{recursive retrieving}, or @dfn{recursion}.
With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} from
the given @sc{url}, documents, retrieving the files the @sc{html}
@code{src}. If the freshly downloaded file is also of type
@code{text/html}, it will be parsed and followed further.
+Recursive retrieval of @sc{http} and @sc{html} content is
+@dfn{breadth-first}. This means that Wget first downloads the requested
+HTML document, then the documents linked from that document, then the
+documents linked by them, and so on. In other words, Wget first
+downloads the documents at depth 1, then those at depth 2, and so on
+until the specified maximum depth.
+
The maximum @dfn{depth} to which the retrieval may descend is specified
-with the @samp{-l} option (the default maximum depth is five layers).
-@xref{Recursive Retrieval}.
+with the @samp{-l} option. The default maximum depth is five layers.
When retrieving an @sc{ftp} @sc{url} recursively, Wget will retrieve all
the data from the given directory tree (including the subdirectories up
to the specified depth) on the remote server, creating its mirror image
locally. @sc{ftp} retrieval is also limited by the @code{depth}
-parameter.
+parameter. Unlike @sc{http} recursion, @sc{ftp} recursion is performed
+depth-first.
By default, Wget will create a local directory tree, corresponding to
the one found on the remote server.
presentations, and any other opportunities where slow network
connections should be bypassed by storing the files locally.
-You should be warned that invoking recursion may cause grave overloading
-on your system, because of the fast exchange of data through the
-network; all of this may hamper other users' work. The same stands for
-the foreign server you are mirroring---the more requests it gets in a
-rows, the greater is its load.
-
-Careless retrieving can also fill your file system uncontrollably, which
-can grind the machine to a halt.
-
-The load can be minimized by lowering the maximum recursion level
-(@samp{-l}) and/or by lowering the number of retries (@samp{-t}). You
-may also consider using the @samp{-w} option to slow down your requests
-to the remote servers, as well as the numerous options to narrow the
-number of followed links (@pxref{Following Links}).
-
-Recursive retrieval is a good thing when used properly. Please take all
-precautions not to wreak havoc through carelessness.
+You should be warned that recursive downloads can overload the remote
+servers. Because of that, many administrators frown upon them and may
+ban access from your site if they detect very fast downloads of big
+amounts of content. When downloading from Internet servers, consider
+using the @samp{-w} option to introduce a delay between accesses to the
+server. The download will take a while longer, but the server
+administrator will not be alarmed by your rudeness.
+
+Of course, recursive download may cause problems on your machine. If
+left to run unchecked, it can easily fill up the disk. If downloading
+from local network, it can also take bandwidth on the system, as well as
+consume memory and CPU.
+
+Try to specify the criteria that match the kind of download you are
+trying to achieve. If you want to download only one page, use
+@samp{--page-requisites} without any additional recursion. If you want
+to download things under one directory, use @samp{-np} to avoid
+downloading things from other directories. If you want to download all
+the files from one directory, use @samp{-l 1} to make sure the recursion
+depth never exceeds one. @xref{Following Links}, for more information
+about this.
+
+Recursive retrieval should be used with care. Don't say you were not
+warned.
@node Following Links, Time-Stamping, Recursive Retrieval, Top
@chapter Following Links
links it will follow.
@menu
-* Relative Links:: Follow relative links only.
-* Host Checking:: Follow links on the same host.
-* Domain Acceptance:: Check on a list of domains.
-* All Hosts:: No host restrictions.
+* Spanning Hosts:: (Un)limiting retrieval based on host name.
* Types of Files:: Getting only certain files.
* Directory-Based Limits:: Getting only certain directories.
+* Relative Links:: Follow relative links only.
* FTP Links:: Following FTP links.
@end menu
-@node Relative Links, Host Checking, Following Links, Following Links
-@section Relative Links
-@cindex relative links
+@node Spanning Hosts, Types of Files, Following Links, Following Links
+@section Spanning Hosts
+@cindex spanning hosts
+@cindex hosts, spanning
-When only relative links are followed (option @samp{-L}), recursive
-retrieving will never span hosts. No time-expensive @sc{dns}-lookups
-will be performed, and the process will be very fast, with the minimum
-strain of the network. This will suit your needs often, especially when
-mirroring the output of various @code{x2html} converters, since they
-generally output relative links.
-
-@node Host Checking, Domain Acceptance, Relative Links, Following Links
-@section Host Checking
-@cindex DNS lookup
-@cindex host lookup
-@cindex host checking
-
-The drawback of following the relative links solely is that humans often
-tend to mix them with absolute links to the very same host, and the very
-same page. In this mode (which is the default mode for following links)
-all @sc{url}s that refer to the same host will be retrieved.
-
-The problem with this option are the aliases of the hosts and domains.
-Thus there is no way for Wget to know that @samp{regoc.srce.hr} and
-@samp{www.srce.hr} are the same host, or that @samp{fly.srk.fer.hr} is
-the same as @samp{fly.cc.fer.hr}. Whenever an absolute link is
-encountered, the host is @sc{dns}-looked-up with @code{gethostbyname} to
-check whether we are maybe dealing with the same hosts. Although the
-results of @code{gethostbyname} are cached, it is still a great
-slowdown, e.g. when dealing with large indices of home pages on different
-hosts (because each of the hosts must be @sc{dns}-resolved to see
-whether it just @emph{might} be an alias of the starting host).
-
-To avoid the overhead you may use @samp{-nh}, which will turn off
-@sc{dns}-resolving and make Wget compare hosts literally. This will
-make things run much faster, but also much less reliable
-(e.g. @samp{www.srce.hr} and @samp{regoc.srce.hr} will be flagged as
-different hosts).
-
-Note that modern @sc{http} servers allow one IP address to host several
-@dfn{virtual servers}, each having its own directory hierarchy. Such
-``servers'' are distinguished by their hostnames (all of which point to
-the same IP address); for this to work, a client must send a @code{Host}
-header, which is what Wget does. However, in that case Wget @emph{must
-not} try to divine a host's ``real'' address, nor try to use the same
-hostname for each access, i.e. @samp{-nh} must be turned on.
-
-In other words, the @samp{-nh} option must be used to enable the
-retrieval from virtual servers distinguished by their hostnames. As the
-number of such server setups grow, the behavior of @samp{-nh} may become
-the default in the future.
-
-@node Domain Acceptance, All Hosts, Host Checking, Following Links
-@section Domain Acceptance
-
-With the @samp{-D} option you may specify the domains that will be
-followed. The hosts the domain of which is not in this list will not be
-@sc{dns}-resolved. Thus you can specify @samp{-Dmit.edu} just to make
-sure that @strong{nothing outside of @sc{mit} gets looked up}. This is
-very important and useful. It also means that @samp{-D} does @emph{not}
-imply @samp{-H} (span all hosts), which must be specified explicitly.
-Feel free to use this options since it will speed things up, with almost
-all the reliability of checking for all hosts. Thus you could invoke
+Wget's recursive retrieval normally refuses to visit hosts different
+than the one you specified on the command line. This is a reasonable
+default; without it, every retrieval would have the potential to turn
+your Wget into a small version of google.
-@example
-wget -r -D.hr http://fly.srk.fer.hr/
-@end example
+However, visiting different hosts, or @dfn{host spanning,} is sometimes
+a useful option. Maybe the images are served from a different server.
+Maybe you're mirroring a site that consists of pages interlinked between
+three servers. Maybe the server has two equivalent names, and the HTML
+pages refer to both interchangeably.
+
+@table @asis
+@item Span to any host---@samp{-H}
+
+The @samp{-H} option turns on host spanning, thus allowing Wget's
+recursive run to visit any host referenced by a link. Unless sufficient
+recursion-limiting criteria are applied depth, these foreign hosts will
+typically link to yet more hosts, and so on until Wget ends up sucking
+up much more data than you have intended.
-to make sure that only the hosts in @samp{.hr} domain get
-@sc{dns}-looked-up for being equal to @samp{fly.srk.fer.hr}. So
-@samp{fly.cc.fer.hr} will be checked (only once!) and found equal, but
-@samp{www.gnu.ai.mit.edu} will not even be checked.
+@item Limit spanning to certain domains---@samp{-D}
-Of course, domain acceptance can be used to limit the retrieval to
-particular domains with spanning of hosts in them, but then you must
-specify @samp{-H} explicitly. E.g.:
+The @samp{-D} option allows you to specify the domains that will be
+followed, thus limiting the recursion only to the hosts that belong to
+these domains. Obviously, this makes sense only in conjunction with
+@samp{-H}. A typical example would be downloading the contents of
+@samp{www.server.com}, but allowing downloads from
+@samp{images.server.com}, etc.:
@example
-wget -r -H -Dmit.edu,stanford.edu http://www.mit.edu/
+wget -rH -Dserver.com http://www.server.com/
@end example
-will start with @samp{http://www.mit.edu/}, following links across
-@sc{mit} and Stanford.
+You can specify more than one address by separating them with a comma,
+e.g. @samp{-Ddomain1.com,domain2.com}.
+
+@item Keep download off certain domains---@samp{--exclude-domains}
If there are domains you want to exclude specifically, you can do it
with @samp{--exclude-domains}, which accepts the same type of arguments
this:
@example
-wget -rH -Dfoo.edu --exclude-domains sunsite.foo.edu http://www.foo.edu/
+wget -rH -Dfoo.edu --exclude-domains sunsite.foo.edu \
+ http://www.foo.edu/
@end example
-@node All Hosts, Types of Files, Domain Acceptance, Following Links
-@section All Hosts
-@cindex all hosts
-@cindex span hosts
-
-When @samp{-H} is specified without @samp{-D}, all hosts are freely
-spanned. There are no restrictions whatsoever as to what part of the
-net Wget will go to fetch documents, other than maximum retrieval depth.
-If a page references @samp{www.yahoo.com}, so be it. Such an option is
-rarely useful for itself.
+@end table
-@node Types of Files, Directory-Based Limits, All Hosts, Following Links
+@node Types of Files, Directory-Based Limits, Spanning Hosts, Following Links
@section Types of Files
@cindex types of files
files; Wget must load all the @sc{html}s to know where to go at
all---recursive retrieval would make no sense otherwise.
-@node Directory-Based Limits, FTP Links, Types of Files, Following Links
+@node Directory-Based Limits, Relative Links, Types of Files, Following Links
@section Directory-Based Limits
@cindex directories
@cindex directory limits
intelligent fashion.
@end table
-@node FTP Links, , Directory-Based Limits, Following Links
+@node Relative Links, FTP Links, Directory-Based Limits, Following Links
+@section Relative Links
+@cindex relative links
+
+When @samp{-L} is turned on, only the relative links are ever followed.
+Relative links are here defined those that do not refer to the web
+server root. For example, these links are relative:
+
+@example
+<a href="foo.gif">
+<a href="foo/bar.gif">
+<a href="../foo/bar.gif">
+@end example
+
+These links are not relative:
+
+@example
+<a href="/foo.gif">
+<a href="/foo/bar.gif">
+<a href="http://www.server.com/foo/bar.gif">
+@end example
+
+Using this option guarantees that recursive retrieval will not span
+hosts, even without @samp{-H}. In simple cases it also allows downloads
+to ``just work'' without having to convert links.
+
+This option is probably not very useful and might be removed in a future
+release.
+
+@node FTP Links, , Relative Links, Following Links
@section Following FTP Links
@cindex following ftp links
respectively.
@item domains = @var{string}
-Same as @samp{-D} (@pxref{Domain Acceptance}).
+Same as @samp{-D} (@pxref{Spanning Hosts}).
@item dot_bytes = @var{n}
Specify the number of bytes ``contained'' in a dot, as seen throughout
download---the same as @samp{-X} (@pxref{Directory-Based Limits}).
@item exclude_domains = @var{string}
-Same as @samp{--exclude-domains} (@pxref{Domain Acceptance}).
+Same as @samp{--exclude-domains} (@pxref{Spanning Hosts}).
@item follow_ftp = on/off
Follow @sc{ftp} links from @sc{html} documents---the same as
responses---the same as @samp{-S}.
@item simple_host_check = on/off
-Same as @samp{-nh} (@pxref{Host Checking}).
+Same as @samp{-nh} (@pxref{Spanning Hosts}).
@item span_hosts = on/off
Same as @samp{-H}.
wget --mirror -A.html http://www.w3.org/
@end example
-@item
-But what about mirroring the hosts networkologically close to you? It
-seems so awfully slow because of all that @sc{dns} resolving. Just use
-@samp{-D} (@pxref{Domain Acceptance}).
-
-@example
-wget -rN -Dsrce.hr http://www.srce.hr/
-@end example
-
-Now Wget will correctly find out that @samp{regoc.srce.hr} is the same
-as @samp{www.srce.hr}, but will not even take into consideration the
-link to @samp{www.mit.edu}.
-
@item
You have a presentation and would like the dumb absolute links to be
converted to relative? Use @samp{-k}:
and you're set. Great? Not for the server admin.
While Wget is retrieving static pages, there's not much of a problem.
-But for Wget, there is no real difference between the smallest static
-page and the hardest, most demanding CGI or dynamic page. For instance,
-a site I know has a section handled by an, uh, bitchin' CGI script that
-converts all the Info files to HTML. The script can and does bring the
-machine to its knees without providing anything useful to the
-downloader.
+But for Wget, there is no real difference between a static page and the
+most demanding CGI. For instance, a site I know has a section handled
+by an, uh, @dfn{bitchin'} CGI script that converts all the Info files to
+HTML. The script can and does bring the machine to its knees without
+providing anything useful to the downloader.
For such and similar cases various robot exclusion schemes have been
devised as a means for the server administrators and document authors to
protect chosen portions of their sites from the wandering of robots.
-The more popular mechanism is the @dfn{Robots Exclusion Standard}
-written by Martijn Koster et al. in 1994. It is specified by placing a
-file named @file{/robots.txt} in the server root, which the robots are
-supposed to download and parse. Wget supports this specification.
+The more popular mechanism is the @dfn{Robots Exclusion Standard}, or
+@sc{res}, written by Martijn Koster et al. in 1994. It specifies the
+format of a text file containing directives that instruct the robots
+which URL paths to avoid. To be found by the robots, the specifications
+must be placed in @file{/robots.txt} in the server root, which the
+robots are supposed to download and parse.
-Norobots support is turned on only when retrieving recursively, and
-@emph{never} for the first page. Thus, you may issue:
+Wget supports @sc{res} when downloading recursively. So, when you
+issue:
@example
-wget -r http://fly.srk.fer.hr/
+wget -r http://www.server.com/
@end example
-First the index of fly.srk.fer.hr will be downloaded. If Wget finds
-anything worth downloading on the same host, only @emph{then} will it
-load the robots, and decide whether or not to load the links after all.
-@file{/robots.txt} is loaded only once per host.
-
-Note that the exlusion standard discussed here has undergone some
-revisions. However, but Wget supports only the first version of
-@sc{res}, the one written by Martijn Koster in 1994, available at
-@url{http://info.webcrawler.com/mak/projects/robots/norobots.html}. A
-later version exists in the form of an internet draft
-<draft-koster-robots-00.txt> titled ``A Method for Web Robots Control'',
-which expired on June 4, 1997. I am not aware if it ever made to an
-@sc{rfc}. The text of the draft is available at
+First the index of @samp{www.server.com} will be downloaded. If Wget
+finds that it wants to download more documents from that server, it will
+request @samp{http://www.server.com/robots.txt} and, if found, use it
+for further downloads. @file{robots.txt} is loaded only once per each
+server.
+
+Until version 1.8, Wget supported the first version of the standard,
+written by Martijn Koster in 1994 and available at
+@url{http://info.webcrawler.com/mak/projects/robots/norobots.html}. As
+of version 1.8, Wget has supported the additional directives specified
+in the internet draft @samp{<draft-koster-robots-00.txt>} titled ``A
+Method for Web Robots Control''. The draft, which has as far as I know
+never made to an @sc{rfc}, is available at
@url{http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html}.
-Wget does not yet support the new directives specified by this draft,
-but we plan to add them.
-This manual no longer includes the text of the old standard.
+This manual no longer includes the text of the Robot Exclusion Standard.
The second, less known mechanism, enables the author of an individual
document to specify whether they want the links from the file to be
authentication.
@item
-Brian Gough---a generous donation.
+The people who provided donations for development, including Brian
+Gough.
@end itemize
The following people have provided patches, bug/build reports, useful
suggestions, beta testing services, fan mail and all the other things
that make maintenance so much fun:
+Ian Abbott
Tim Adam,
Adrian Aichner,
Martin Baehr,
Dieter Baron,
-Roger Beeman and the Gurus at Cisco,
+Roger Beeman,
Dan Berger,
+T. Bharath,
Paul Bludov,
+Daniel Bodea,
Mark Boyns,
John Burden,
Wanderlei Cavassin,
@ifinfo
Damir Dzeko,
@end ifinfo
+Alan Eldridge,
@iftex
Aleksandar Erkalovi@'{c},
@end iftex
Masashi Fujita,
Howard Gayle,
Marcel Gerrits,
+Lemble Gregory,
Hans Grobler,
Mathieu Guillaume,
Dan Harkless,
-Heiko Herold,
+Herold Heiko,
+Jochen Hein,
Karl Heuer,
HIROSE Masaaki,
Gregor Hoffleit,
Heinz Salzmann,
Robert Schmidt,
Andreas Schwab,
+Chris Seawood,
Toomas Soome,
Tage Stabell-Kulo,
Sven Sternberger,
Szakacsits Szabolcs,
Mike Thomas,
Philipp Thomas,
+Dave Turner,
Russell Vincent,
Charles G Waldman,
Douglas E. Wegscheid,
projects / wget / commitdiff