1 \input texinfo @c -*-texinfo-*-
7 @settitle GNU Wget @value{VERSION} Manual
8 @c Disable the monstrous rectangles beside overfull hbox-es.
10 @c Use `odd' to print double-sided.
15 @c Remove this if you don't use A4 paper.
19 @c Title for man page. The weird way texi2pod.pl is written requires
20 @c the preceding @set.
22 @c man title Wget The non-interactive network downloader.
24 @dircategory Network Applications
26 * Wget: (wget). The non-interactive network downloader.
30 This file documents the the GNU Wget utility for downloading network
33 @c man begin COPYRIGHT
34 Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2002, 2003 Free
35 Software Foundation, Inc.
37 Permission is granted to make and distribute verbatim copies of
38 this manual provided the copyright notice and this permission notice
39 are preserved on all copies.
42 Permission is granted to process this file through TeX and print the
43 results, provided the printed document carries a copying permission
44 notice identical to this one except for the removal of this paragraph
45 (this paragraph not being relevant to the printed manual).
47 Permission is granted to copy, distribute and/or modify this document
48 under the terms of the GNU Free Documentation License, Version 1.1 or
49 any later version published by the Free Software Foundation; with the
50 Invariant Sections being ``GNU General Public License'' and ``GNU Free
51 Documentation License'', with no Front-Cover Texts, and with no
52 Back-Cover Texts. A copy of the license is included in the section
53 entitled ``GNU Free Documentation License''.
58 @title GNU Wget @value{VERSION}
59 @subtitle The non-interactive download utility
60 @subtitle Updated for Wget @value{VERSION}, @value{UPDATED}
61 @author by Hrvoje Nik@v{s}i@'{c} and the developers
65 Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
68 GNU Info entry for @file{wget}.
73 @vskip 0pt plus 1filll
74 Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2003 Free Software
77 Permission is granted to copy, distribute and/or modify this document
78 under the terms of the GNU Free Documentation License, Version 1.1 or
79 any later version published by the Free Software Foundation; with the
80 Invariant Sections being ``GNU General Public License'' and ``GNU Free
81 Documentation License'', with no Front-Cover Texts, and with no
82 Back-Cover Texts. A copy of the license is included in the section
83 entitled ``GNU Free Documentation License''.
88 @top Wget @value{VERSION}
90 This manual documents version @value{VERSION} of GNU Wget, the freely
91 available utility for network downloads.
93 Copyright @copyright{} 1996, 1997, 1998, 2000, 2001, 2003 Free Software
97 * Overview:: Features of Wget.
98 * Invoking:: Wget command-line arguments.
99 * Recursive Download:: Downloading interlinked pages.
100 * Following Links:: The available methods of chasing links.
101 * Time-Stamping:: Mirroring according to time-stamps.
102 * Startup File:: Wget's initialization file.
103 * Examples:: Examples of usage.
104 * Various:: The stuff that doesn't fit anywhere else.
105 * Appendices:: Some useful references.
106 * Copying:: You may give out copies of Wget and of this manual.
107 * Concept Index:: Topics covered by this manual.
116 @c man begin DESCRIPTION
117 GNU Wget is a free utility for non-interactive download of files from
118 the Web. It supports @sc{http}, @sc{https}, and @sc{ftp} protocols, as
119 well as retrieval through @sc{http} proxies.
122 This chapter is a partial overview of Wget's features.
126 @c man begin DESCRIPTION
127 Wget is non-interactive, meaning that it can work in the background,
128 while the user is not logged on. This allows you to start a retrieval
129 and disconnect from the system, letting Wget finish the work. By
130 contrast, most of the Web browsers require constant user's presence,
131 which can be a great hindrance when transferring a lot of data.
137 @c man begin DESCRIPTION
141 @c man begin DESCRIPTION
142 Wget can follow links in @sc{html} and @sc{xhtml} pages and create local
143 versions of remote web sites, fully recreating the directory structure of
144 the original site. This is sometimes referred to as ``recursive
145 downloading.'' While doing that, Wget respects the Robot Exclusion
146 Standard (@file{/robots.txt}). Wget can be instructed to convert the
147 links in downloaded @sc{html} files to the local files for offline
153 File name wildcard matching and recursive mirroring of directories are
154 available when retrieving via @sc{ftp}. Wget can read the time-stamp
155 information given by both @sc{http} and @sc{ftp} servers, and store it
156 locally. Thus Wget can see if the remote file has changed since last
157 retrieval, and automatically retrieve the new version if it has. This
158 makes Wget suitable for mirroring of @sc{ftp} sites, as well as home
164 @c man begin DESCRIPTION
168 @c man begin DESCRIPTION
169 Wget has been designed for robustness over slow or unstable network
170 connections; if a download fails due to a network problem, it will
171 keep retrying until the whole file has been retrieved. If the server
172 supports regetting, it will instruct the server to continue the
173 download from where it left off.
178 Wget supports proxy servers, which can lighten the network load, speed
179 up retrieval and provide access behind firewalls. However, if you are
180 behind a firewall that requires that you use a socks style gateway, you
181 can get the socks library and build Wget with support for socks. Wget
182 also supports the passive @sc{ftp} downloading as an option.
186 Built-in features offer mechanisms to tune which links you wish to follow
187 (@pxref{Following Links}).
191 The retrieval is conveniently traced with printing dots, each dot
192 representing a fixed amount of data received (1KB by default). These
193 representations can be customized to your preferences.
197 Most of the features are fully configurable, either through command line
198 options, or via the initialization file @file{.wgetrc} (@pxref{Startup
199 File}). Wget allows you to define @dfn{global} startup files
200 (@file{/usr/local/etc/wgetrc} by default) for site settings.
205 @item /usr/local/etc/wgetrc
206 Default location of the @dfn{global} startup file.
216 Finally, GNU Wget is free software. This means that everyone may use
217 it, redistribute it and/or modify it under the terms of the GNU General
218 Public License, as published by the Free Software Foundation
229 By default, Wget is very simple to invoke. The basic syntax is:
232 @c man begin SYNOPSIS
233 wget [@var{option}]@dots{} [@var{URL}]@dots{}
237 Wget will simply download all the @sc{url}s specified on the command
238 line. @var{URL} is a @dfn{Uniform Resource Locator}, as defined below.
240 However, you may wish to change some of the default parameters of
241 Wget. You can do it two ways: permanently, adding the appropriate
242 command to @file{.wgetrc} (@pxref{Startup File}), or specifying it on
248 * Basic Startup Options::
249 * Logging and Input File Options::
251 * Directory Options::
254 * Recursive Retrieval Options::
255 * Recursive Accept/Reject Options::
263 @dfn{URL} is an acronym for Uniform Resource Locator. A uniform
264 resource locator is a compact string representation for a resource
265 available via the Internet. Wget recognizes the @sc{url} syntax as per
266 @sc{rfc1738}. This is the most widely used form (square brackets denote
270 http://host[:port]/directory/file
271 ftp://host[:port]/directory/file
274 You can also encode your username and password within a @sc{url}:
277 ftp://user:password@@host/path
278 http://user:password@@host/path
281 Either @var{user} or @var{password}, or both, may be left out. If you
282 leave out either the @sc{http} username or password, no authentication
283 will be sent. If you leave out the @sc{ftp} username, @samp{anonymous}
284 will be used. If you leave out the @sc{ftp} password, your email
285 address will be supplied as a default password.@footnote{If you have a
286 @file{.netrc} file in your home directory, password will also be
289 @strong{Important Note}: if you specify a password-containing @sc{url}
290 on the command line, the username and password will be plainly visible
291 to all users on the system, by way of @code{ps}. On multi-user systems,
292 this is a big security risk. To work around it, use @code{wget -i -}
293 and feed the @sc{url}s to Wget's standard input, each on a separate
294 line, terminated by @kbd{C-d}.
296 You can encode unsafe characters in a @sc{url} as @samp{%xy}, @code{xy}
297 being the hexadecimal representation of the character's @sc{ascii}
298 value. Some common unsafe characters include @samp{%} (quoted as
299 @samp{%25}), @samp{:} (quoted as @samp{%3A}), and @samp{@@} (quoted as
300 @samp{%40}). Refer to @sc{rfc1738} for a comprehensive list of unsafe
303 Wget also supports the @code{type} feature for @sc{ftp} @sc{url}s. By
304 default, @sc{ftp} documents are retrieved in the binary mode (type
305 @samp{i}), which means that they are downloaded unchanged. Another
306 useful mode is the @samp{a} (@dfn{ASCII}) mode, which converts the line
307 delimiters between the different operating systems, and is thus useful
308 for text files. Here is an example:
311 ftp://host/directory/file;type=a
314 Two alternative variants of @sc{url} specification are also supported,
315 because of historical (hysterical?) reasons and their widespreaded use.
317 @sc{ftp}-only syntax (supported by @code{NcFTP}):
322 @sc{http}-only syntax (introduced by @code{Netscape}):
327 These two alternative forms are deprecated, and may cease being
328 supported in the future.
330 If you do not understand the difference between these notations, or do
331 not know which one to use, just use the plain ordinary format you use
332 with your favorite browser, like @code{Lynx} or @code{Netscape}.
335 @section Option Syntax
336 @cindex option syntax
337 @cindex syntax of options
339 Since Wget uses GNU getopts to process its arguments, every option has a
340 short form and a long form. Long options are more convenient to
341 remember, but take time to type. You may freely mix different option
342 styles, or specify options after the command-line arguments. Thus you
346 wget -r --tries=10 http://fly.srk.fer.hr/ -o log
349 The space between the option accepting an argument and the argument may
350 be omitted. Instead @samp{-o log} you can write @samp{-olog}.
352 You may put several options that do not require arguments together,
359 This is a complete equivalent of:
362 wget -d -r -c @var{URL}
365 Since the options can be specified after the arguments, you may
366 terminate them with @samp{--}. So the following will try to download
367 @sc{url} @samp{-x}, reporting failure to @file{log}:
373 The options that accept comma-separated lists all respect the convention
374 that specifying an empty list clears its value. This can be useful to
375 clear the @file{.wgetrc} settings. For instance, if your @file{.wgetrc}
376 sets @code{exclude_directories} to @file{/cgi-bin}, the following
377 example will first reset it, and then set it to exclude @file{/~nobody}
378 and @file{/~somebody}. You can also clear the lists in @file{.wgetrc}
379 (@pxref{Wgetrc Syntax}).
382 wget -X '' -X /~nobody,/~somebody
387 @node Basic Startup Options
388 @section Basic Startup Options
393 Display the version of Wget.
397 Print a help message describing all of Wget's command-line options.
401 Go to background immediately after startup. If no output file is
402 specified via the @samp{-o}, output is redirected to @file{wget-log}.
404 @cindex execute wgetrc command
405 @item -e @var{command}
406 @itemx --execute @var{command}
407 Execute @var{command} as if it were a part of @file{.wgetrc}
408 (@pxref{Startup File}). A command thus invoked will be executed
409 @emph{after} the commands in @file{.wgetrc}, thus taking precedence over
413 @node Logging and Input File Options
414 @section Logging and Input File Options
419 @item -o @var{logfile}
420 @itemx --output-file=@var{logfile}
421 Log all messages to @var{logfile}. The messages are normally reported
424 @cindex append to log
425 @item -a @var{logfile}
426 @itemx --append-output=@var{logfile}
427 Append to @var{logfile}. This is the same as @samp{-o}, only it appends
428 to @var{logfile} instead of overwriting the old log file. If
429 @var{logfile} does not exist, a new file is created.
434 Turn on debug output, meaning various information important to the
435 developers of Wget if it does not work properly. Your system
436 administrator may have chosen to compile Wget without debug support, in
437 which case @samp{-d} will not work. Please note that compiling with
438 debug support is always safe---Wget compiled with the debug support will
439 @emph{not} print any debug info unless requested with @samp{-d}.
440 @xref{Reporting Bugs}, for more information on how to use @samp{-d} for
446 Turn off Wget's output.
451 Turn on verbose output, with all the available data. The default output
456 Non-verbose output---turn off verbose without being completely quiet
457 (use @samp{-q} for that), which means that error messages and basic
458 information still get printed.
462 @itemx --input-file=@var{file}
463 Read @sc{url}s from @var{file}, in which case no @sc{url}s need to be on
464 the command line. If there are @sc{url}s both on the command line and
465 in an input file, those on the command lines will be the first ones to
466 be retrieved. The @var{file} need not be an @sc{html} document (but no
467 harm if it is)---it is enough if the @sc{url}s are just listed
470 However, if you specify @samp{--force-html}, the document will be
471 regarded as @samp{html}. In that case you may have problems with
472 relative links, which you can solve either by adding @code{<base
473 href="@var{url}">} to the documents or by specifying
474 @samp{--base=@var{url}} on the command line.
479 When input is read from a file, force it to be treated as an @sc{html}
480 file. This enables you to retrieve relative links from existing
481 @sc{html} files on your local disk, by adding @code{<base
482 href="@var{url}">} to @sc{html}, or using the @samp{--base} command-line
485 @cindex base for relative links in input file
487 @itemx --base=@var{URL}
488 When used in conjunction with @samp{-F}, prepends @var{URL} to relative
489 links in the file specified by @samp{-i}.
492 @node Download Options
493 @section Download Options
496 @cindex bind() address
497 @cindex client IP address
498 @cindex IP address, client
499 @item --bind-address=@var{ADDRESS}
500 When making client TCP/IP connections, @code{bind()} to @var{ADDRESS} on
501 the local machine. @var{ADDRESS} may be specified as a hostname or IP
502 address. This option can be useful if your machine is bound to multiple
507 @cindex number of retries
508 @item -t @var{number}
509 @itemx --tries=@var{number}
510 Set number of retries to @var{number}. Specify 0 or @samp{inf} for
511 infinite retrying. The default is to retry 20 times, with the exception
512 of fatal errors like ``connection refused'' or ``not found'' (404),
513 which are not retried.
516 @itemx --output-document=@var{file}
517 The documents will not be written to the appropriate files, but all will
518 be concatenated together and written to @var{file}. If @var{file}
519 already exists, it will be overwritten. If the @var{file} is @samp{-},
520 the documents will be written to standard output. Including this option
521 automatically sets the number of tries to 1.
523 @cindex clobbering, file
524 @cindex downloading multiple times
528 If a file is downloaded more than once in the same directory, Wget's
529 behavior depends on a few options, including @samp{-nc}. In certain
530 cases, the local file will be @dfn{clobbered}, or overwritten, upon
531 repeated download. In other cases it will be preserved.
533 When running Wget without @samp{-N}, @samp{-nc}, or @samp{-r},
534 downloading the same file in the same directory will result in the
535 original copy of @var{file} being preserved and the second copy being
536 named @samp{@var{file}.1}. If that file is downloaded yet again, the
537 third copy will be named @samp{@var{file}.2}, and so on. When
538 @samp{-nc} is specified, this behavior is suppressed, and Wget will
539 refuse to download newer copies of @samp{@var{file}}. Therefore,
540 ``@code{no-clobber}'' is actually a misnomer in this mode---it's not
541 clobbering that's prevented (as the numeric suffixes were already
542 preventing clobbering), but rather the multiple version saving that's
545 When running Wget with @samp{-r}, but without @samp{-N} or @samp{-nc},
546 re-downloading a file will result in the new copy simply overwriting the
547 old. Adding @samp{-nc} will prevent this behavior, instead causing the
548 original version to be preserved and any newer copies on the server to
551 When running Wget with @samp{-N}, with or without @samp{-r}, the
552 decision as to whether or not to download a newer copy of a file depends
553 on the local and remote timestamp and size of the file
554 (@pxref{Time-Stamping}). @samp{-nc} may not be specified at the same
557 Note that when @samp{-nc} is specified, files with the suffixes
558 @samp{.html} or @samp{.htm} will be loaded from the local disk and
559 parsed as if they had been retrieved from the Web.
561 @cindex continue retrieval
562 @cindex incomplete downloads
563 @cindex resume download
566 Continue getting a partially-downloaded file. This is useful when you
567 want to finish up a download started by a previous instance of Wget, or
568 by another program. For instance:
571 wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
574 If there is a file named @file{ls-lR.Z} in the current directory, Wget
575 will assume that it is the first portion of the remote file, and will
576 ask the server to continue the retrieval from an offset equal to the
577 length of the local file.
579 Note that you don't need to specify this option if you just want the
580 current invocation of Wget to retry downloading a file should the
581 connection be lost midway through. This is the default behavior.
582 @samp{-c} only affects resumption of downloads started @emph{prior} to
583 this invocation of Wget, and whose local files are still sitting around.
585 Without @samp{-c}, the previous example would just download the remote
586 file to @file{ls-lR.Z.1}, leaving the truncated @file{ls-lR.Z} file
589 Beginning with Wget 1.7, if you use @samp{-c} on a non-empty file, and
590 it turns out that the server does not support continued downloading,
591 Wget will refuse to start the download from scratch, which would
592 effectively ruin existing contents. If you really want the download to
593 start from scratch, remove the file.
595 Also beginning with Wget 1.7, if you use @samp{-c} on a file which is of
596 equal size as the one on the server, Wget will refuse to download the
597 file and print an explanatory message. The same happens when the file
598 is smaller on the server than locally (presumably because it was changed
599 on the server since your last download attempt)---because ``continuing''
600 is not meaningful, no download occurs.
602 On the other side of the coin, while using @samp{-c}, any file that's
603 bigger on the server than locally will be considered an incomplete
604 download and only @code{(length(remote) - length(local))} bytes will be
605 downloaded and tacked onto the end of the local file. This behavior can
606 be desirable in certain cases---for instance, you can use @samp{wget -c}
607 to download just the new portion that's been appended to a data
608 collection or log file.
610 However, if the file is bigger on the server because it's been
611 @emph{changed}, as opposed to just @emph{appended} to, you'll end up
612 with a garbled file. Wget has no way of verifying that the local file
613 is really a valid prefix of the remote file. You need to be especially
614 careful of this when using @samp{-c} in conjunction with @samp{-r},
615 since every file will be considered as an "incomplete download" candidate.
617 Another instance where you'll get a garbled file if you try to use
618 @samp{-c} is if you have a lame @sc{http} proxy that inserts a
619 ``transfer interrupted'' string into the local file. In the future a
620 ``rollback'' option may be added to deal with this case.
622 Note that @samp{-c} only works with @sc{ftp} servers and with @sc{http}
623 servers that support the @code{Range} header.
625 @cindex progress indicator
627 @item --progress=@var{type}
628 Select the type of the progress indicator you wish to use. Legal
629 indicators are ``dot'' and ``bar''.
631 The ``bar'' indicator is used by default. It draws an @sc{ascii} progress
632 bar graphics (a.k.a ``thermometer'' display) indicating the status of
633 retrieval. If the output is not a TTY, the ``dot'' bar will be used by
636 Use @samp{--progress=dot} to switch to the ``dot'' display. It traces
637 the retrieval by printing dots on the screen, each dot representing a
638 fixed amount of downloaded data.
640 When using the dotted retrieval, you may also set the @dfn{style} by
641 specifying the type as @samp{dot:@var{style}}. Different styles assign
642 different meaning to one dot. With the @code{default} style each dot
643 represents 1K, there are ten dots in a cluster and 50 dots in a line.
644 The @code{binary} style has a more ``computer''-like orientation---8K
645 dots, 16-dots clusters and 48 dots per line (which makes for 384K
646 lines). The @code{mega} style is suitable for downloading very large
647 files---each dot represents 64K retrieved, there are eight dots in a
648 cluster, and 48 dots on each line (so each line contains 3M).
650 Note that you can set the default style using the @code{progress}
651 command in @file{.wgetrc}. That setting may be overridden from the
652 command line. The exception is that, when the output is not a TTY, the
653 ``dot'' progress will be favored over ``bar''. To force the bar output,
654 use @samp{--progress=bar:force}.
657 @itemx --timestamping
658 Turn on time-stamping. @xref{Time-Stamping}, for details.
660 @cindex server response, print
662 @itemx --server-response
663 Print the headers sent by @sc{http} servers and responses sent by
666 @cindex Wget as spider
669 When invoked with this option, Wget will behave as a Web @dfn{spider},
670 which means that it will not download the pages, just check that they
671 are there. For example, you can use Wget to check your bookmarks:
674 wget --spider --force-html -i bookmarks.html
677 This feature needs much more work for Wget to get close to the
678 functionality of real web spiders.
682 @itemx --timeout=@var{seconds}
683 Set the network timeout to @var{seconds} seconds. This is equivalent
684 to specifying @samp{--dns-timeout}, @samp{--connect-timeout}, and
685 @samp{--read-timeout}, all at the same time.
687 Whenever Wget connects to or reads from a remote host, it checks for a
688 timeout and aborts the operation if the time expires. This prevents
689 anomalous occurrences such as hanging reads or infinite connects. The
690 only timeout enabled by default is a 900-second timeout for reading.
691 Setting timeout to 0 disables checking for timeouts.
693 Unless you know what you are doing, it is best not to set any of the
694 timeout-related options.
698 @item --dns-timeout=@var{seconds}
699 Set the DNS lookup timeout to @var{seconds} seconds. DNS lookups that
700 don't complete within the specified time will fail. By default, there
701 is no timeout on DNS lookups, other than that implemented by system
704 @cindex connect timeout
705 @cindex timeout, connect
706 @item --connect-timeout=@var{seconds}
707 Set the connect timeout to @var{seconds} seconds. TCP connections that
708 take longer to establish will be aborted. By default, there is no
709 connect timeout, other than that implemented by system libraries.
712 @cindex timeout, read
713 @item --read-timeout=@var{seconds}
714 Set the read (and write) timeout to @var{seconds} seconds. Reads that
715 take longer will fail. The default value for read timeout is 900
718 @cindex bandwidth, limit
720 @cindex limit bandwidth
721 @item --limit-rate=@var{amount}
722 Limit the download speed to @var{amount} bytes per second. Amount may
723 be expressed in bytes, kilobytes with the @samp{k} suffix, or megabytes
724 with the @samp{m} suffix. For example, @samp{--limit-rate=20k} will
725 limit the retrieval rate to 20KB/s. This kind of thing is useful when,
726 for whatever reason, you don't want Wget to consume the entire available
729 Note that Wget implements the limiting by sleeping the appropriate
730 amount of time after a network read that took less time than specified
731 by the rate. Eventually this strategy causes the TCP transfer to slow
732 down to approximately the specified rate. However, it may take some
733 time for this balance to be achieved, so don't be surprised if limiting
734 the rate doesn't work well with very small files.
738 @item -w @var{seconds}
739 @itemx --wait=@var{seconds}
740 Wait the specified number of seconds between the retrievals. Use of
741 this option is recommended, as it lightens the server load by making the
742 requests less frequent. Instead of in seconds, the time can be
743 specified in minutes using the @code{m} suffix, in hours using @code{h}
744 suffix, or in days using @code{d} suffix.
746 Specifying a large value for this option is useful if the network or the
747 destination host is down, so that Wget can wait long enough to
748 reasonably expect the network error to be fixed before the retry.
750 @cindex retries, waiting between
751 @cindex waiting between retries
752 @item --waitretry=@var{seconds}
753 If you don't want Wget to wait between @emph{every} retrieval, but only
754 between retries of failed downloads, you can use this option. Wget will
755 use @dfn{linear backoff}, waiting 1 second after the first failure on a
756 given file, then waiting 2 seconds after the second failure on that
757 file, up to the maximum number of @var{seconds} you specify. Therefore,
758 a value of 10 will actually make Wget wait up to (1 + 2 + ... + 10) = 55
761 Note that this option is turned on by default in the global
767 Some web sites may perform log analysis to identify retrieval programs
768 such as Wget by looking for statistically significant similarities in
769 the time between requests. This option causes the time between requests
770 to vary between 0 and 2 * @var{wait} seconds, where @var{wait} was
771 specified using the @samp{--wait} option, in order to mask Wget's
772 presence from such analysis.
774 A recent article in a publication devoted to development on a popular
775 consumer platform provided code to perform this analysis on the fly.
776 Its author suggested blocking at the class C address level to ensure
777 automated retrieval programs were blocked despite changing DHCP-supplied
780 The @samp{--random-wait} option was inspired by this ill-advised
781 recommendation to block many unrelated users from a web site due to the
786 @itemx --proxy=on/off
787 Turn proxy support on or off. The proxy is on by default if the
788 appropriate environment variable is defined.
790 For more information about the use of proxies with Wget, @xref{Proxies}.
794 @itemx --quota=@var{quota}
795 Specify download quota for automatic retrievals. The value can be
796 specified in bytes (default), kilobytes (with @samp{k} suffix), or
797 megabytes (with @samp{m} suffix).
799 Note that quota will never affect downloading a single file. So if you
800 specify @samp{wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz}, all of the
801 @file{ls-lR.gz} will be downloaded. The same goes even when several
802 @sc{url}s are specified on the command-line. However, quota is
803 respected when retrieving either recursively, or from an input file.
804 Thus you may safely type @samp{wget -Q2m -i sites}---download will be
805 aborted when the quota is exceeded.
807 Setting quota to 0 or to @samp{inf} unlimits the download quota.
810 @cindex caching of DNS lookups
811 @item --dns-cache=off
812 Turn off caching of DNS lookups. Normally, Wget remembers the addresses
813 it looked up from DNS so it doesn't have to repeatedly contact the DNS
814 server for the same (typically small) set of addresses it retrieves
815 from. This cache exists in memory only; a new Wget run will contact DNS
818 However, in some cases it is not desirable to cache host names, even for
819 the duration of a short-running application like Wget. For example,
820 some HTTP servers are hosted on machines with dynamically allocated IP
821 addresses that change from time to time. Their DNS entries are updated
822 along with each change. When Wget's download from such a host gets
823 interrupted by IP address change, Wget retries the download, but (due to
824 DNS caching) it contacts the old address. With the DNS cache turned
825 off, Wget will repeat the DNS lookup for every connect and will thus get
826 the correct dynamic address every time---at the cost of additional DNS
827 lookups where they're probably not needed.
829 If you don't understand the above description, you probably won't need
832 @cindex file names, restrict
833 @cindex Windows file names
834 @item --restrict-file-names=@var{mode}
835 Change which characters found in remote URLs may show up in local file
836 names generated from those URLs. Characters that are @dfn{restricted}
837 by this option are escaped, i.e. replaced with @samp{%HH}, where
838 @samp{HH} is the hexadecimal number that corresponds to the restricted
841 By default, Wget escapes the characters that are not valid as part of
842 file names on your operating system, as well as control characters that
843 are typically unprintable. This option is useful for changing these
844 defaults, either because you are downloading to a non-native partition,
845 or because you want to disable escaping of the control characters.
847 When mode is set to ``unix'', Wget escapes the character @samp{/} and
848 the control characters in the ranges 0--31 and 128--159. This is the
849 default on Unix-like OS'es.
851 When mode is set to ``windows'', Wget escapes the characters @samp{\},
852 @samp{|}, @samp{/}, @samp{:}, @samp{?}, @samp{"}, @samp{*}, @samp{<},
853 @samp{>}, and the control characters in the ranges 0--31 and 128--159.
854 In addition to this, Wget in Windows mode uses @samp{+} instead of
855 @samp{:} to separate host and port in local file names, and uses
856 @samp{@@} instead of @samp{?} to separate the query portion of the file
857 name from the rest. Therefore, a URL that would be saved as
858 @samp{www.xemacs.org:4300/search.pl?input=blah} in Unix mode would be
859 saved as @samp{www.xemacs.org+4300/search.pl@@input=blah} in Windows
860 mode. This mode is the default on Windows.
862 If you append @samp{,nocontrol} to the mode, as in
863 @samp{unix,nocontrol}, escaping of the control characters is also
864 switched off. You can use @samp{--restrict-file-names=nocontrol} to
865 turn off escaping of control characters without affecting the choice of
866 the OS to use as file name restriction mode.
869 @node Directory Options
870 @section Directory Options
874 @itemx --no-directories
875 Do not create a hierarchy of directories when retrieving recursively.
876 With this option turned on, all files will get saved to the current
877 directory, without clobbering (if a name shows up more than once, the
878 filenames will get extensions @samp{.n}).
881 @itemx --force-directories
882 The opposite of @samp{-nd}---create a hierarchy of directories, even if
883 one would not have been created otherwise. E.g. @samp{wget -x
884 http://fly.srk.fer.hr/robots.txt} will save the downloaded file to
885 @file{fly.srk.fer.hr/robots.txt}.
888 @itemx --no-host-directories
889 Disable generation of host-prefixed directories. By default, invoking
890 Wget with @samp{-r http://fly.srk.fer.hr/} will create a structure of
891 directories beginning with @file{fly.srk.fer.hr/}. This option disables
894 @cindex cut directories
895 @item --cut-dirs=@var{number}
896 Ignore @var{number} directory components. This is useful for getting a
897 fine-grained control over the directory where recursive retrieval will
900 Take, for example, the directory at
901 @samp{ftp://ftp.xemacs.org/pub/xemacs/}. If you retrieve it with
902 @samp{-r}, it will be saved locally under
903 @file{ftp.xemacs.org/pub/xemacs/}. While the @samp{-nH} option can
904 remove the @file{ftp.xemacs.org/} part, you are still stuck with
905 @file{pub/xemacs}. This is where @samp{--cut-dirs} comes in handy; it
906 makes Wget not ``see'' @var{number} remote directory components. Here
907 are several examples of how @samp{--cut-dirs} option works.
911 No options -> ftp.xemacs.org/pub/xemacs/
913 -nH --cut-dirs=1 -> xemacs/
914 -nH --cut-dirs=2 -> .
916 --cut-dirs=1 -> ftp.xemacs.org/xemacs/
921 If you just want to get rid of the directory structure, this option is
922 similar to a combination of @samp{-nd} and @samp{-P}. However, unlike
923 @samp{-nd}, @samp{--cut-dirs} does not lose with subdirectories---for
924 instance, with @samp{-nH --cut-dirs=1}, a @file{beta/} subdirectory will
925 be placed to @file{xemacs/beta}, as one would expect.
927 @cindex directory prefix
928 @item -P @var{prefix}
929 @itemx --directory-prefix=@var{prefix}
930 Set directory prefix to @var{prefix}. The @dfn{directory prefix} is the
931 directory where all other files and subdirectories will be saved to,
932 i.e. the top of the retrieval tree. The default is @samp{.} (the
937 @section HTTP Options
940 @cindex .html extension
942 @itemx --html-extension
943 If a file of type @samp{application/xhtml+xml} or @samp{text/html} is
944 downloaded and the URL does not end with the regexp
945 @samp{\.[Hh][Tt][Mm][Ll]?}, this option will cause the suffix @samp{.html}
946 to be appended to the local filename. This is useful, for instance, when
947 you're mirroring a remote site that uses @samp{.asp} pages, but you want
948 the mirrored pages to be viewable on your stock Apache server. Another
949 good use for this is when you're downloading CGI-generated materials. A URL
950 like @samp{http://site.com/article.cgi?25} will be saved as
951 @file{article.cgi?25.html}.
953 Note that filenames changed in this way will be re-downloaded every time
954 you re-mirror a site, because Wget can't tell that the local
955 @file{@var{X}.html} file corresponds to remote URL @samp{@var{X}} (since
956 it doesn't yet know that the URL produces output of type
957 @samp{text/html} or @samp{application/xhtml+xml}. To prevent this
958 re-downloading, you must use @samp{-k} and @samp{-K} so that the original
959 version of the file will be saved as @file{@var{X}.orig} (@pxref{Recursive
963 @cindex http password
964 @cindex authentication
965 @item --http-user=@var{user}
966 @itemx --http-passwd=@var{password}
967 Specify the username @var{user} and password @var{password} on an
968 @sc{http} server. According to the type of the challenge, Wget will
969 encode them using either the @code{basic} (insecure) or the
970 @code{digest} authentication scheme.
972 Another way to specify username and password is in the @sc{url} itself
973 (@pxref{URL Format}). Either method reveals your password to anyone who
974 bothers to run @code{ps}. To prevent the passwords from being seen,
975 store them in @file{.wgetrc} or @file{.netrc}, and make sure to protect
976 those files from other users with @code{chmod}. If the passwords are
977 really important, do not leave them lying in those files either---edit
978 the files and delete them after Wget has started the download.
980 For more information about security issues with Wget, @xref{Security
986 Disable server-side cache. In this case, Wget will send the remote
987 server an appropriate directive (@samp{Pragma: no-cache}) to get the
988 file from the remote service, rather than returning the cached version.
989 This is especially useful for retrieving and flushing out-of-date
990 documents on proxy servers.
992 Caching is allowed by default.
996 Disable the use of cookies. Cookies are a mechanism for maintaining
997 server-side state. The server sends the client a cookie using the
998 @code{Set-Cookie} header, and the client responds with the same cookie
999 upon further requests. Since cookies allow the server owners to keep
1000 track of visitors and for sites to exchange this information, some
1001 consider them a breach of privacy. The default is to use cookies;
1002 however, @emph{storing} cookies is not on by default.
1004 @cindex loading cookies
1005 @cindex cookies, loading
1006 @item --load-cookies @var{file}
1007 Load cookies from @var{file} before the first HTTP retrieval.
1008 @var{file} is a textual file in the format originally used by Netscape's
1009 @file{cookies.txt} file.
1011 You will typically use this option when mirroring sites that require
1012 that you be logged in to access some or all of their content. The login
1013 process typically works by the web server issuing an @sc{http} cookie
1014 upon receiving and verifying your credentials. The cookie is then
1015 resent by the browser when accessing that part of the site, and so
1016 proves your identity.
1018 Mirroring such a site requires Wget to send the same cookies your
1019 browser sends when communicating with the site. This is achieved by
1020 @samp{--load-cookies}---simply point Wget to the location of the
1021 @file{cookies.txt} file, and it will send the same cookies your browser
1022 would send in the same situation. Different browsers keep textual
1023 cookie files in different locations:
1027 The cookies are in @file{~/.netscape/cookies.txt}.
1029 @item Mozilla and Netscape 6.x.
1030 Mozilla's cookie file is also named @file{cookies.txt}, located
1031 somewhere under @file{~/.mozilla}, in the directory of your profile.
1032 The full path usually ends up looking somewhat like
1033 @file{~/.mozilla/default/@var{some-weird-string}/cookies.txt}.
1035 @item Internet Explorer.
1036 You can produce a cookie file Wget can use by using the File menu,
1037 Import and Export, Export Cookies. This has been tested with Internet
1038 Explorer 5; it is not guaranteed to work with earlier versions.
1040 @item Other browsers.
1041 If you are using a different browser to create your cookies,
1042 @samp{--load-cookies} will only work if you can locate or produce a
1043 cookie file in the Netscape format that Wget expects.
1046 If you cannot use @samp{--load-cookies}, there might still be an
1047 alternative. If your browser supports a ``cookie manager'', you can use
1048 it to view the cookies used when accessing the site you're mirroring.
1049 Write down the name and value of the cookie, and manually instruct Wget
1050 to send those cookies, bypassing the ``official'' cookie support:
1053 wget --cookies=off --header "Cookie: @var{name}=@var{value}"
1056 @cindex saving cookies
1057 @cindex cookies, saving
1058 @item --save-cookies @var{file}
1059 Save cookies to @var{file} before exiting. This will not save cookies
1060 that have expired or that have no expiry time (so-called ``session
1061 cookies''), but also see @samp{--keep-session-cookies}.
1063 @cindex cookies, session
1064 @cindex session cookies
1065 @item --keep-session-cookies
1067 When specified, causes @samp{--save-cookies} to also save session
1068 cookies. Session cookies are normally not save because they are
1069 supposed to be forgotten when you exit the browser. Saving them is
1070 useful on sites that require you to log in or to visit the home page
1071 before you can access some pages. With this option, multiple Wget runs
1072 are considered a single browser session as far as the site is concerned.
1074 Since the cookie file format does not normally carry session cookies,
1075 Wget marks them with an expiry timestamp of 0. Wget's
1076 @samp{--load-cookies} recognizes those as session cookies, but it might
1077 confuse other browsers. Also note that cookies so loaded will be
1078 treated as other session cookies, which means that if you want
1079 @samp{--save-cookies} to preserve them again, you must use
1080 @samp{--keep-session-cookies} again.
1082 @cindex Content-Length, ignore
1083 @cindex ignore length
1084 @item --ignore-length
1085 Unfortunately, some @sc{http} servers (@sc{cgi} programs, to be more
1086 precise) send out bogus @code{Content-Length} headers, which makes Wget
1087 go wild, as it thinks not all the document was retrieved. You can spot
1088 this syndrome if Wget retries getting the same document again and again,
1089 each time claiming that the (otherwise normal) connection has closed on
1092 With this option, Wget will ignore the @code{Content-Length} header---as
1093 if it never existed.
1096 @item --header=@var{additional-header}
1097 Define an @var{additional-header} to be passed to the @sc{http} servers.
1098 Headers must contain a @samp{:} preceded by one or more non-blank
1099 characters, and must not contain newlines.
1101 You may define more than one additional header by specifying
1102 @samp{--header} more than once.
1106 wget --header='Accept-Charset: iso-8859-2' \
1107 --header='Accept-Language: hr' \
1108 http://fly.srk.fer.hr/
1112 Specification of an empty string as the header value will clear all
1113 previous user-defined headers.
1116 @cindex proxy password
1117 @cindex proxy authentication
1118 @item --proxy-user=@var{user}
1119 @itemx --proxy-passwd=@var{password}
1120 Specify the username @var{user} and password @var{password} for
1121 authentication on a proxy server. Wget will encode them using the
1122 @code{basic} authentication scheme.
1124 Security considerations similar to those with @samp{--http-passwd}
1125 pertain here as well.
1127 @cindex http referer
1128 @cindex referer, http
1129 @item --referer=@var{url}
1130 Include `Referer: @var{url}' header in HTTP request. Useful for
1131 retrieving documents with server-side processing that assume they are
1132 always being retrieved by interactive web browsers and only come out
1133 properly when Referer is set to one of the pages that point to them.
1135 @cindex server response, save
1136 @item --save-headers
1137 Save the headers sent by the @sc{http} server to the file, preceding the
1138 actual contents, with an empty line as the separator.
1141 @item -U @var{agent-string}
1142 @itemx --user-agent=@var{agent-string}
1143 Identify as @var{agent-string} to the @sc{http} server.
1145 The @sc{http} protocol allows the clients to identify themselves using a
1146 @code{User-Agent} header field. This enables distinguishing the
1147 @sc{www} software, usually for statistical purposes or for tracing of
1148 protocol violations. Wget normally identifies as
1149 @samp{Wget/@var{version}}, @var{version} being the current version
1152 However, some sites have been known to impose the policy of tailoring
1153 the output according to the @code{User-Agent}-supplied information.
1154 While conceptually this is not such a bad idea, it has been abused by
1155 servers denying information to clients other than @code{Mozilla} or
1156 Microsoft @code{Internet Explorer}. This option allows you to change
1157 the @code{User-Agent} line issued by Wget. Use of this option is
1158 discouraged, unless you really know what you are doing.
1161 @item --post-data=@var{string}
1162 @itemx --post-file=@var{file}
1163 Use POST as the method for all HTTP requests and send the specified data
1164 in the request body. @code{--post-data} sends @var{string} as data,
1165 whereas @code{--post-file} sends the contents of @var{file}. Other than
1166 that, they work in exactly the same way.
1168 Please be aware that Wget needs to know the size of the POST data in
1169 advance. Therefore the argument to @code{--post-file} must be a regular
1170 file; specifying a FIFO or something like @file{/dev/stdin} won't work.
1171 It's not quite clear how to work around this limitation inherent in
1172 HTTP/1.0. Although HTTP/1.1 introduces @dfn{chunked} transfer that
1173 doesn't require knowing the request length in advance, a client can't
1174 use chunked unless it knows it's talking to an HTTP/1.1 server. And it
1175 can't know that until it receives a response, which in turn requires the
1176 request to have been completed -- a chicken-and-egg problem.
1178 Note: if Wget is redirected after the POST request is completed, it will
1179 not send the POST data to the redirected URL. This is because URLs that
1180 process POST often respond with a redirection to a regular page
1181 (although that's technically disallowed), which does not desire or
1182 accept POST. It is not yet clear that this behavior is optimal; if it
1183 doesn't work out, it will be changed.
1185 This example shows how to log to a server using POST and then proceed to
1186 download the desired pages, presumably only accessible to authorized
1191 # @r{Log in to the server. This can be done only once.}
1192 wget --save-cookies cookies.txt \
1193 --post-data 'user=foo&password=bar' \
1194 http://server.com/auth.php
1196 # @r{Now grab the page or pages we care about.}
1197 wget --load-cookies cookies.txt \
1198 -p http://server.com/interesting/article.php
1204 @section FTP Options
1207 @cindex .listing files, removing
1209 @itemx --dont-remove-listing
1210 Don't remove the temporary @file{.listing} files generated by @sc{ftp}
1211 retrievals. Normally, these files contain the raw directory listings
1212 received from @sc{ftp} servers. Not removing them can be useful for
1213 debugging purposes, or when you want to be able to easily check on the
1214 contents of remote server directories (e.g. to verify that a mirror
1215 you're running is complete).
1217 Note that even though Wget writes to a known filename for this file,
1218 this is not a security hole in the scenario of a user making
1219 @file{.listing} a symbolic link to @file{/etc/passwd} or something and
1220 asking @code{root} to run Wget in his or her directory. Depending on
1221 the options used, either Wget will refuse to write to @file{.listing},
1222 making the globbing/recursion/time-stamping operation fail, or the
1223 symbolic link will be deleted and replaced with the actual
1224 @file{.listing} file, or the listing will be written to a
1225 @file{.listing.@var{number}} file.
1227 Even though this situation isn't a problem, though, @code{root} should
1228 never run Wget in a non-trusted user's directory. A user could do
1229 something as simple as linking @file{index.html} to @file{/etc/passwd}
1230 and asking @code{root} to run Wget with @samp{-N} or @samp{-r} so the file
1231 will be overwritten.
1233 @cindex globbing, toggle
1235 Turn off @sc{ftp} globbing. Globbing refers to the use of shell-like
1236 special characters (@dfn{wildcards}), like @samp{*}, @samp{?}, @samp{[}
1237 and @samp{]} to retrieve more than one file from the same directory at
1241 wget ftp://gnjilux.srk.fer.hr/*.msg
1244 By default, globbing will be turned on if the @sc{url} contains a
1245 globbing character. This option may be used to turn globbing on or off
1248 You may have to quote the @sc{url} to protect it from being expanded by
1249 your shell. Globbing makes Wget look for a directory listing, which is
1250 system-specific. This is why it currently works only with Unix @sc{ftp}
1251 servers (and the ones emulating Unix @code{ls} output).
1255 Use the @dfn{passive} @sc{ftp} retrieval scheme, in which the client
1256 initiates the data connection. This is sometimes required for @sc{ftp}
1257 to work behind firewalls.
1259 @cindex symbolic links, retrieving
1260 @item --retr-symlinks
1261 Usually, when retrieving @sc{ftp} directories recursively and a symbolic
1262 link is encountered, the linked-to file is not downloaded. Instead, a
1263 matching symbolic link is created on the local filesystem. The
1264 pointed-to file will not be downloaded unless this recursive retrieval
1265 would have encountered it separately and downloaded it anyway.
1267 When @samp{--retr-symlinks} is specified, however, symbolic links are
1268 traversed and the pointed-to files are retrieved. At this time, this
1269 option does not cause Wget to traverse symlinks to directories and
1270 recurse through them, but in the future it should be enhanced to do
1273 Note that when retrieving a file (not a directory) because it was
1274 specified on the command-line, rather than because it was recursed to,
1275 this option has no effect. Symbolic links are always traversed in this
1279 @node Recursive Retrieval Options
1280 @section Recursive Retrieval Options
1285 Turn on recursive retrieving. @xref{Recursive Download}, for more
1288 @item -l @var{depth}
1289 @itemx --level=@var{depth}
1290 Specify recursion maximum depth level @var{depth} (@pxref{Recursive
1291 Download}). The default maximum depth is 5.
1293 @cindex proxy filling
1294 @cindex delete after retrieval
1295 @cindex filling proxy cache
1296 @item --delete-after
1297 This option tells Wget to delete every single file it downloads,
1298 @emph{after} having done so. It is useful for pre-fetching popular
1299 pages through a proxy, e.g.:
1302 wget -r -nd --delete-after http://whatever.com/~popular/page/
1305 The @samp{-r} option is to retrieve recursively, and @samp{-nd} to not
1308 Note that @samp{--delete-after} deletes files on the local machine. It
1309 does not issue the @samp{DELE} command to remote FTP sites, for
1310 instance. Also note that when @samp{--delete-after} is specified,
1311 @samp{--convert-links} is ignored, so @samp{.orig} files are simply not
1312 created in the first place.
1314 @cindex conversion of links
1315 @cindex link conversion
1317 @itemx --convert-links
1318 After the download is complete, convert the links in the document to
1319 make them suitable for local viewing. This affects not only the visible
1320 hyperlinks, but any part of the document that links to external content,
1321 such as embedded images, links to style sheets, hyperlinks to non-@sc{html}
1324 Each link will be changed in one of the two ways:
1328 The links to files that have been downloaded by Wget will be changed to
1329 refer to the file they point to as a relative link.
1331 Example: if the downloaded file @file{/foo/doc.html} links to
1332 @file{/bar/img.gif}, also downloaded, then the link in @file{doc.html}
1333 will be modified to point to @samp{../bar/img.gif}. This kind of
1334 transformation works reliably for arbitrary combinations of directories.
1337 The links to files that have not been downloaded by Wget will be changed
1338 to include host name and absolute path of the location they point to.
1340 Example: if the downloaded file @file{/foo/doc.html} links to
1341 @file{/bar/img.gif} (or to @file{../bar/img.gif}), then the link in
1342 @file{doc.html} will be modified to point to
1343 @file{http://@var{hostname}/bar/img.gif}.
1346 Because of this, local browsing works reliably: if a linked file was
1347 downloaded, the link will refer to its local name; if it was not
1348 downloaded, the link will refer to its full Internet address rather than
1349 presenting a broken link. The fact that the former links are converted
1350 to relative links ensures that you can move the downloaded hierarchy to
1353 Note that only at the end of the download can Wget know which links have
1354 been downloaded. Because of that, the work done by @samp{-k} will be
1355 performed at the end of all the downloads.
1357 @cindex backing up converted files
1359 @itemx --backup-converted
1360 When converting a file, back up the original version with a @samp{.orig}
1361 suffix. Affects the behavior of @samp{-N} (@pxref{HTTP Time-Stamping
1366 Turn on options suitable for mirroring. This option turns on recursion
1367 and time-stamping, sets infinite recursion depth and keeps @sc{ftp}
1368 directory listings. It is currently equivalent to
1369 @samp{-r -N -l inf -nr}.
1371 @cindex page requisites
1372 @cindex required images, downloading
1374 @itemx --page-requisites
1375 This option causes Wget to download all the files that are necessary to
1376 properly display a given @sc{html} page. This includes such things as
1377 inlined images, sounds, and referenced stylesheets.
1379 Ordinarily, when downloading a single @sc{html} page, any requisite documents
1380 that may be needed to display it properly are not downloaded. Using
1381 @samp{-r} together with @samp{-l} can help, but since Wget does not
1382 ordinarily distinguish between external and inlined documents, one is
1383 generally left with ``leaf documents'' that are missing their
1386 For instance, say document @file{1.html} contains an @code{<IMG>} tag
1387 referencing @file{1.gif} and an @code{<A>} tag pointing to external
1388 document @file{2.html}. Say that @file{2.html} is similar but that its
1389 image is @file{2.gif} and it links to @file{3.html}. Say this
1390 continues up to some arbitrarily high number.
1392 If one executes the command:
1395 wget -r -l 2 http://@var{site}/1.html
1398 then @file{1.html}, @file{1.gif}, @file{2.html}, @file{2.gif}, and
1399 @file{3.html} will be downloaded. As you can see, @file{3.html} is
1400 without its requisite @file{3.gif} because Wget is simply counting the
1401 number of hops (up to 2) away from @file{1.html} in order to determine
1402 where to stop the recursion. However, with this command:
1405 wget -r -l 2 -p http://@var{site}/1.html
1408 all the above files @emph{and} @file{3.html}'s requisite @file{3.gif}
1409 will be downloaded. Similarly,
1412 wget -r -l 1 -p http://@var{site}/1.html
1415 will cause @file{1.html}, @file{1.gif}, @file{2.html}, and @file{2.gif}
1416 to be downloaded. One might think that:
1419 wget -r -l 0 -p http://@var{site}/1.html
1422 would download just @file{1.html} and @file{1.gif}, but unfortunately
1423 this is not the case, because @samp{-l 0} is equivalent to
1424 @samp{-l inf}---that is, infinite recursion. To download a single @sc{html}
1425 page (or a handful of them, all specified on the command-line or in a
1426 @samp{-i} @sc{url} input file) and its (or their) requisites, simply leave off
1427 @samp{-r} and @samp{-l}:
1430 wget -p http://@var{site}/1.html
1433 Note that Wget will behave as if @samp{-r} had been specified, but only
1434 that single page and its requisites will be downloaded. Links from that
1435 page to external documents will not be followed. Actually, to download
1436 a single page and all its requisites (even if they exist on separate
1437 websites), and make sure the lot displays properly locally, this author
1438 likes to use a few options in addition to @samp{-p}:
1441 wget -E -H -k -K -p http://@var{site}/@var{document}
1444 To finish off this topic, it's worth knowing that Wget's idea of an
1445 external document link is any URL specified in an @code{<A>} tag, an
1446 @code{<AREA>} tag, or a @code{<LINK>} tag other than @code{<LINK
1449 @cindex @sc{html} comments
1450 @cindex comments, @sc{html}
1451 @item --strict-comments
1452 Turn on strict parsing of @sc{html} comments. The default is to terminate
1453 comments at the first occurrence of @samp{-->}.
1455 According to specifications, @sc{html} comments are expressed as @sc{sgml}
1456 @dfn{declarations}. Declaration is special markup that begins with
1457 @samp{<!} and ends with @samp{>}, such as @samp{<!DOCTYPE ...>}, that
1458 may contain comments between a pair of @samp{--} delimiters. @sc{html}
1459 comments are ``empty declarations'', @sc{sgml} declarations without any
1460 non-comment text. Therefore, @samp{<!--foo-->} is a valid comment, and
1461 so is @samp{<!--one-- --two-->}, but @samp{<!--1--2-->} is not.
1463 On the other hand, most @sc{html} writers don't perceive comments as anything
1464 other than text delimited with @samp{<!--} and @samp{-->}, which is not
1465 quite the same. For example, something like @samp{<!------------>}
1466 works as a valid comment as long as the number of dashes is a multiple
1467 of four (!). If not, the comment technically lasts until the next
1468 @samp{--}, which may be at the other end of the document. Because of
1469 this, many popular browsers completely ignore the specification and
1470 implement what users have come to expect: comments delimited with
1471 @samp{<!--} and @samp{-->}.
1473 Until version 1.9, Wget interpreted comments strictly, which resulted in
1474 missing links in many web pages that displayed fine in browsers, but had
1475 the misfortune of containing non-compliant comments. Beginning with
1476 version 1.9, Wget has joined the ranks of clients that implements
1477 ``naive'' comments, terminating each comment at the first occurrence of
1480 If, for whatever reason, you want strict comment parsing, use this
1481 option to turn it on.
1484 @node Recursive Accept/Reject Options
1485 @section Recursive Accept/Reject Options
1488 @item -A @var{acclist} --accept @var{acclist}
1489 @itemx -R @var{rejlist} --reject @var{rejlist}
1490 Specify comma-separated lists of file name suffixes or patterns to
1491 accept or reject (@pxref{Types of Files} for more details).
1493 @item -D @var{domain-list}
1494 @itemx --domains=@var{domain-list}
1495 Set domains to be followed. @var{domain-list} is a comma-separated list
1496 of domains. Note that it does @emph{not} turn on @samp{-H}.
1498 @item --exclude-domains @var{domain-list}
1499 Specify the domains that are @emph{not} to be followed.
1500 (@pxref{Spanning Hosts}).
1502 @cindex follow FTP links
1504 Follow @sc{ftp} links from @sc{html} documents. Without this option,
1505 Wget will ignore all the @sc{ftp} links.
1507 @cindex tag-based recursive pruning
1508 @item --follow-tags=@var{list}
1509 Wget has an internal table of @sc{html} tag / attribute pairs that it
1510 considers when looking for linked documents during a recursive
1511 retrieval. If a user wants only a subset of those tags to be
1512 considered, however, he or she should be specify such tags in a
1513 comma-separated @var{list} with this option.
1515 @item --ignore-tags=@var{list}
1516 This is the opposite of the @samp{--follow-tags} option. To skip
1517 certain @sc{html} tags when recursively looking for documents to download,
1518 specify them in a comma-separated @var{list}.
1520 In the past, this option was the best bet for downloading a single page
1521 and its requisites, using a command-line like:
1524 wget --ignore-tags=a,area -H -k -K -r http://@var{site}/@var{document}
1527 However, the author of this option came across a page with tags like
1528 @code{<LINK REL="home" HREF="/">} and came to the realization that
1529 specifying tags to ignore was not enough. One can't just tell Wget to
1530 ignore @code{<LINK>}, because then stylesheets will not be downloaded.
1531 Now the best bet for downloading a single page and its requisites is the
1532 dedicated @samp{--page-requisites} option.
1536 Enable spanning across hosts when doing recursive retrieving
1537 (@pxref{Spanning Hosts}).
1541 Follow relative links only. Useful for retrieving a specific home page
1542 without any distractions, not even those from the same hosts
1543 (@pxref{Relative Links}).
1546 @itemx --include-directories=@var{list}
1547 Specify a comma-separated list of directories you wish to follow when
1548 downloading (@pxref{Directory-Based Limits} for more details.) Elements
1549 of @var{list} may contain wildcards.
1552 @itemx --exclude-directories=@var{list}
1553 Specify a comma-separated list of directories you wish to exclude from
1554 download (@pxref{Directory-Based Limits} for more details.) Elements of
1555 @var{list} may contain wildcards.
1559 Do not ever ascend to the parent directory when retrieving recursively.
1560 This is a useful option, since it guarantees that only the files
1561 @emph{below} a certain hierarchy will be downloaded.
1562 @xref{Directory-Based Limits}, for more details.
1567 @node Recursive Download
1568 @chapter Recursive Download
1571 @cindex recursive download
1573 GNU Wget is capable of traversing parts of the Web (or a single
1574 @sc{http} or @sc{ftp} server), following links and directory structure.
1575 We refer to this as to @dfn{recursive retrieval}, or @dfn{recursion}.
1577 With @sc{http} @sc{url}s, Wget retrieves and parses the @sc{html} from
1578 the given @sc{url}, documents, retrieving the files the @sc{html}
1579 document was referring to, through markup like @code{href}, or
1580 @code{src}. If the freshly downloaded file is also of type
1581 @code{text/html} or @code{application/xhtml+xml}, it will be parsed and
1584 Recursive retrieval of @sc{http} and @sc{html} content is
1585 @dfn{breadth-first}. This means that Wget first downloads the requested
1586 @sc{html} document, then the documents linked from that document, then the
1587 documents linked by them, and so on. In other words, Wget first
1588 downloads the documents at depth 1, then those at depth 2, and so on
1589 until the specified maximum depth.
1591 The maximum @dfn{depth} to which the retrieval may descend is specified
1592 with the @samp{-l} option. The default maximum depth is five layers.
1594 When retrieving an @sc{ftp} @sc{url} recursively, Wget will retrieve all
1595 the data from the given directory tree (including the subdirectories up
1596 to the specified depth) on the remote server, creating its mirror image
1597 locally. @sc{ftp} retrieval is also limited by the @code{depth}
1598 parameter. Unlike @sc{http} recursion, @sc{ftp} recursion is performed
1601 By default, Wget will create a local directory tree, corresponding to
1602 the one found on the remote server.
1604 Recursive retrieving can find a number of applications, the most
1605 important of which is mirroring. It is also useful for @sc{www}
1606 presentations, and any other opportunities where slow network
1607 connections should be bypassed by storing the files locally.
1609 You should be warned that recursive downloads can overload the remote
1610 servers. Because of that, many administrators frown upon them and may
1611 ban access from your site if they detect very fast downloads of big
1612 amounts of content. When downloading from Internet servers, consider
1613 using the @samp{-w} option to introduce a delay between accesses to the
1614 server. The download will take a while longer, but the server
1615 administrator will not be alarmed by your rudeness.
1617 Of course, recursive download may cause problems on your machine. If
1618 left to run unchecked, it can easily fill up the disk. If downloading
1619 from local network, it can also take bandwidth on the system, as well as
1620 consume memory and CPU.
1622 Try to specify the criteria that match the kind of download you are
1623 trying to achieve. If you want to download only one page, use
1624 @samp{--page-requisites} without any additional recursion. If you want
1625 to download things under one directory, use @samp{-np} to avoid
1626 downloading things from other directories. If you want to download all
1627 the files from one directory, use @samp{-l 1} to make sure the recursion
1628 depth never exceeds one. @xref{Following Links}, for more information
1631 Recursive retrieval should be used with care. Don't say you were not
1634 @node Following Links
1635 @chapter Following Links
1637 @cindex following links
1639 When retrieving recursively, one does not wish to retrieve loads of
1640 unnecessary data. Most of the time the users bear in mind exactly what
1641 they want to download, and want Wget to follow only specific links.
1643 For example, if you wish to download the music archive from
1644 @samp{fly.srk.fer.hr}, you will not want to download all the home pages
1645 that happen to be referenced by an obscure part of the archive.
1647 Wget possesses several mechanisms that allows you to fine-tune which
1648 links it will follow.
1651 * Spanning Hosts:: (Un)limiting retrieval based on host name.
1652 * Types of Files:: Getting only certain files.
1653 * Directory-Based Limits:: Getting only certain directories.
1654 * Relative Links:: Follow relative links only.
1655 * FTP Links:: Following FTP links.
1658 @node Spanning Hosts
1659 @section Spanning Hosts
1660 @cindex spanning hosts
1661 @cindex hosts, spanning
1663 Wget's recursive retrieval normally refuses to visit hosts different
1664 than the one you specified on the command line. This is a reasonable
1665 default; without it, every retrieval would have the potential to turn
1666 your Wget into a small version of google.
1668 However, visiting different hosts, or @dfn{host spanning,} is sometimes
1669 a useful option. Maybe the images are served from a different server.
1670 Maybe you're mirroring a site that consists of pages interlinked between
1671 three servers. Maybe the server has two equivalent names, and the @sc{html}
1672 pages refer to both interchangeably.
1675 @item Span to any host---@samp{-H}
1677 The @samp{-H} option turns on host spanning, thus allowing Wget's
1678 recursive run to visit any host referenced by a link. Unless sufficient
1679 recursion-limiting criteria are applied depth, these foreign hosts will
1680 typically link to yet more hosts, and so on until Wget ends up sucking
1681 up much more data than you have intended.
1683 @item Limit spanning to certain domains---@samp{-D}
1685 The @samp{-D} option allows you to specify the domains that will be
1686 followed, thus limiting the recursion only to the hosts that belong to
1687 these domains. Obviously, this makes sense only in conjunction with
1688 @samp{-H}. A typical example would be downloading the contents of
1689 @samp{www.server.com}, but allowing downloads from
1690 @samp{images.server.com}, etc.:
1693 wget -rH -Dserver.com http://www.server.com/
1696 You can specify more than one address by separating them with a comma,
1697 e.g. @samp{-Ddomain1.com,domain2.com}.
1699 @item Keep download off certain domains---@samp{--exclude-domains}
1701 If there are domains you want to exclude specifically, you can do it
1702 with @samp{--exclude-domains}, which accepts the same type of arguments
1703 of @samp{-D}, but will @emph{exclude} all the listed domains. For
1704 example, if you want to download all the hosts from @samp{foo.edu}
1705 domain, with the exception of @samp{sunsite.foo.edu}, you can do it like
1709 wget -rH -Dfoo.edu --exclude-domains sunsite.foo.edu \
1715 @node Types of Files
1716 @section Types of Files
1717 @cindex types of files
1719 When downloading material from the web, you will often want to restrict
1720 the retrieval to only certain file types. For example, if you are
1721 interested in downloading @sc{gif}s, you will not be overjoyed to get
1722 loads of PostScript documents, and vice versa.
1724 Wget offers two options to deal with this problem. Each option
1725 description lists a short name, a long name, and the equivalent command
1728 @cindex accept wildcards
1729 @cindex accept suffixes
1730 @cindex wildcards, accept
1731 @cindex suffixes, accept
1733 @item -A @var{acclist}
1734 @itemx --accept @var{acclist}
1735 @itemx accept = @var{acclist}
1736 The argument to @samp{--accept} option is a list of file suffixes or
1737 patterns that Wget will download during recursive retrieval. A suffix
1738 is the ending part of a file, and consists of ``normal'' letters,
1739 e.g. @samp{gif} or @samp{.jpg}. A matching pattern contains shell-like
1740 wildcards, e.g. @samp{books*} or @samp{zelazny*196[0-9]*}.
1742 So, specifying @samp{wget -A gif,jpg} will make Wget download only the
1743 files ending with @samp{gif} or @samp{jpg}, i.e. @sc{gif}s and
1744 @sc{jpeg}s. On the other hand, @samp{wget -A "zelazny*196[0-9]*"} will
1745 download only files beginning with @samp{zelazny} and containing numbers
1746 from 1960 to 1969 anywhere within. Look up the manual of your shell for
1747 a description of how pattern matching works.
1749 Of course, any number of suffixes and patterns can be combined into a
1750 comma-separated list, and given as an argument to @samp{-A}.
1752 @cindex reject wildcards
1753 @cindex reject suffixes
1754 @cindex wildcards, reject
1755 @cindex suffixes, reject
1756 @item -R @var{rejlist}
1757 @itemx --reject @var{rejlist}
1758 @itemx reject = @var{rejlist}
1759 The @samp{--reject} option works the same way as @samp{--accept}, only
1760 its logic is the reverse; Wget will download all files @emph{except} the
1761 ones matching the suffixes (or patterns) in the list.
1763 So, if you want to download a whole page except for the cumbersome
1764 @sc{mpeg}s and @sc{.au} files, you can use @samp{wget -R mpg,mpeg,au}.
1765 Analogously, to download all files except the ones beginning with
1766 @samp{bjork}, use @samp{wget -R "bjork*"}. The quotes are to prevent
1767 expansion by the shell.
1770 The @samp{-A} and @samp{-R} options may be combined to achieve even
1771 better fine-tuning of which files to retrieve. E.g. @samp{wget -A
1772 "*zelazny*" -R .ps} will download all the files having @samp{zelazny} as
1773 a part of their name, but @emph{not} the PostScript files.
1775 Note that these two options do not affect the downloading of @sc{html}
1776 files; Wget must load all the @sc{html}s to know where to go at
1777 all---recursive retrieval would make no sense otherwise.
1779 @node Directory-Based Limits
1780 @section Directory-Based Limits
1782 @cindex directory limits
1784 Regardless of other link-following facilities, it is often useful to
1785 place the restriction of what files to retrieve based on the directories
1786 those files are placed in. There can be many reasons for this---the
1787 home pages may be organized in a reasonable directory structure; or some
1788 directories may contain useless information, e.g. @file{/cgi-bin} or
1789 @file{/dev} directories.
1791 Wget offers three different options to deal with this requirement. Each
1792 option description lists a short name, a long name, and the equivalent
1793 command in @file{.wgetrc}.
1795 @cindex directories, include
1796 @cindex include directories
1797 @cindex accept directories
1800 @itemx --include @var{list}
1801 @itemx include_directories = @var{list}
1802 @samp{-I} option accepts a comma-separated list of directories included
1803 in the retrieval. Any other directories will simply be ignored. The
1804 directories are absolute paths.
1806 So, if you wish to download from @samp{http://host/people/bozo/}
1807 following only links to bozo's colleagues in the @file{/people}
1808 directory and the bogus scripts in @file{/cgi-bin}, you can specify:
1811 wget -I /people,/cgi-bin http://host/people/bozo/
1814 @cindex directories, exclude
1815 @cindex exclude directories
1816 @cindex reject directories
1818 @itemx --exclude @var{list}
1819 @itemx exclude_directories = @var{list}
1820 @samp{-X} option is exactly the reverse of @samp{-I}---this is a list of
1821 directories @emph{excluded} from the download. E.g. if you do not want
1822 Wget to download things from @file{/cgi-bin} directory, specify @samp{-X
1823 /cgi-bin} on the command line.
1825 The same as with @samp{-A}/@samp{-R}, these two options can be combined
1826 to get a better fine-tuning of downloading subdirectories. E.g. if you
1827 want to load all the files from @file{/pub} hierarchy except for
1828 @file{/pub/worthless}, specify @samp{-I/pub -X/pub/worthless}.
1833 @itemx no_parent = on
1834 The simplest, and often very useful way of limiting directories is
1835 disallowing retrieval of the links that refer to the hierarchy
1836 @dfn{above} than the beginning directory, i.e. disallowing ascent to the
1837 parent directory/directories.
1839 The @samp{--no-parent} option (short @samp{-np}) is useful in this case.
1840 Using it guarantees that you will never leave the existing hierarchy.
1841 Supposing you issue Wget with:
1844 wget -r --no-parent http://somehost/~luzer/my-archive/
1847 You may rest assured that none of the references to
1848 @file{/~his-girls-homepage/} or @file{/~luzer/all-my-mpegs/} will be
1849 followed. Only the archive you are interested in will be downloaded.
1850 Essentially, @samp{--no-parent} is similar to
1851 @samp{-I/~luzer/my-archive}, only it handles redirections in a more
1852 intelligent fashion.
1855 @node Relative Links
1856 @section Relative Links
1857 @cindex relative links
1859 When @samp{-L} is turned on, only the relative links are ever followed.
1860 Relative links are here defined those that do not refer to the web
1861 server root. For example, these links are relative:
1865 <a href="foo/bar.gif">
1866 <a href="../foo/bar.gif">
1869 These links are not relative:
1873 <a href="/foo/bar.gif">
1874 <a href="http://www.server.com/foo/bar.gif">
1877 Using this option guarantees that recursive retrieval will not span
1878 hosts, even without @samp{-H}. In simple cases it also allows downloads
1879 to ``just work'' without having to convert links.
1881 This option is probably not very useful and might be removed in a future
1885 @section Following FTP Links
1886 @cindex following ftp links
1888 The rules for @sc{ftp} are somewhat specific, as it is necessary for
1889 them to be. @sc{ftp} links in @sc{html} documents are often included
1890 for purposes of reference, and it is often inconvenient to download them
1893 To have @sc{ftp} links followed from @sc{html} documents, you need to
1894 specify the @samp{--follow-ftp} option. Having done that, @sc{ftp}
1895 links will span hosts regardless of @samp{-H} setting. This is logical,
1896 as @sc{ftp} links rarely point to the same host where the @sc{http}
1897 server resides. For similar reasons, the @samp{-L} options has no
1898 effect on such downloads. On the other hand, domain acceptance
1899 (@samp{-D}) and suffix rules (@samp{-A} and @samp{-R}) apply normally.
1901 Also note that followed links to @sc{ftp} directories will not be
1902 retrieved recursively further.
1905 @chapter Time-Stamping
1906 @cindex time-stamping
1907 @cindex timestamping
1908 @cindex updating the archives
1909 @cindex incremental updating
1911 One of the most important aspects of mirroring information from the
1912 Internet is updating your archives.
1914 Downloading the whole archive again and again, just to replace a few
1915 changed files is expensive, both in terms of wasted bandwidth and money,
1916 and the time to do the update. This is why all the mirroring tools
1917 offer the option of incremental updating.
1919 Such an updating mechanism means that the remote server is scanned in
1920 search of @dfn{new} files. Only those new files will be downloaded in
1921 the place of the old ones.
1923 A file is considered new if one of these two conditions are met:
1927 A file of that name does not already exist locally.
1930 A file of that name does exist, but the remote file was modified more
1931 recently than the local file.
1934 To implement this, the program needs to be aware of the time of last
1935 modification of both local and remote files. We call this information the
1936 @dfn{time-stamp} of a file.
1938 The time-stamping in GNU Wget is turned on using @samp{--timestamping}
1939 (@samp{-N}) option, or through @code{timestamping = on} directive in
1940 @file{.wgetrc}. With this option, for each file it intends to download,
1941 Wget will check whether a local file of the same name exists. If it
1942 does, and the remote file is older, Wget will not download it.
1944 If the local file does not exist, or the sizes of the files do not
1945 match, Wget will download the remote file no matter what the time-stamps
1949 * Time-Stamping Usage::
1950 * HTTP Time-Stamping Internals::
1951 * FTP Time-Stamping Internals::
1954 @node Time-Stamping Usage
1955 @section Time-Stamping Usage
1956 @cindex time-stamping usage
1957 @cindex usage, time-stamping
1959 The usage of time-stamping is simple. Say you would like to download a
1960 file so that it keeps its date of modification.
1963 wget -S http://www.gnu.ai.mit.edu/
1966 A simple @code{ls -l} shows that the time stamp on the local file equals
1967 the state of the @code{Last-Modified} header, as returned by the server.
1968 As you can see, the time-stamping info is preserved locally, even
1969 without @samp{-N} (at least for @sc{http}).
1971 Several days later, you would like Wget to check if the remote file has
1972 changed, and download it if it has.
1975 wget -N http://www.gnu.ai.mit.edu/
1978 Wget will ask the server for the last-modified date. If the local file
1979 has the same timestamp as the server, or a newer one, the remote file
1980 will not be re-fetched. However, if the remote file is more recent,
1981 Wget will proceed to fetch it.
1983 The same goes for @sc{ftp}. For example:
1986 wget "ftp://ftp.ifi.uio.no/pub/emacs/gnus/*"
1989 (The quotes around that URL are to prevent the shell from trying to
1990 interpret the @samp{*}.)
1992 After download, a local directory listing will show that the timestamps
1993 match those on the remote server. Reissuing the command with @samp{-N}
1994 will make Wget re-fetch @emph{only} the files that have been modified
1995 since the last download.
1997 If you wished to mirror the GNU archive every week, you would use a
1998 command like the following, weekly:
2001 wget --timestamping -r ftp://ftp.gnu.org/pub/gnu/
2004 Note that time-stamping will only work for files for which the server
2005 gives a timestamp. For @sc{http}, this depends on getting a
2006 @code{Last-Modified} header. For @sc{ftp}, this depends on getting a
2007 directory listing with dates in a format that Wget can parse
2008 (@pxref{FTP Time-Stamping Internals}).
2010 @node HTTP Time-Stamping Internals
2011 @section HTTP Time-Stamping Internals
2012 @cindex http time-stamping
2014 Time-stamping in @sc{http} is implemented by checking of the
2015 @code{Last-Modified} header. If you wish to retrieve the file
2016 @file{foo.html} through @sc{http}, Wget will check whether
2017 @file{foo.html} exists locally. If it doesn't, @file{foo.html} will be
2018 retrieved unconditionally.
2020 If the file does exist locally, Wget will first check its local
2021 time-stamp (similar to the way @code{ls -l} checks it), and then send a
2022 @code{HEAD} request to the remote server, demanding the information on
2025 The @code{Last-Modified} header is examined to find which file was
2026 modified more recently (which makes it ``newer''). If the remote file
2027 is newer, it will be downloaded; if it is older, Wget will give
2028 up.@footnote{As an additional check, Wget will look at the
2029 @code{Content-Length} header, and compare the sizes; if they are not the
2030 same, the remote file will be downloaded no matter what the time-stamp
2033 When @samp{--backup-converted} (@samp{-K}) is specified in conjunction
2034 with @samp{-N}, server file @samp{@var{X}} is compared to local file
2035 @samp{@var{X}.orig}, if extant, rather than being compared to local file
2036 @samp{@var{X}}, which will always differ if it's been converted by
2037 @samp{--convert-links} (@samp{-k}).
2039 Arguably, @sc{http} time-stamping should be implemented using the
2040 @code{If-Modified-Since} request.
2042 @node FTP Time-Stamping Internals
2043 @section FTP Time-Stamping Internals
2044 @cindex ftp time-stamping
2046 In theory, @sc{ftp} time-stamping works much the same as @sc{http}, only
2047 @sc{ftp} has no headers---time-stamps must be ferreted out of directory
2050 If an @sc{ftp} download is recursive or uses globbing, Wget will use the
2051 @sc{ftp} @code{LIST} command to get a file listing for the directory
2052 containing the desired file(s). It will try to analyze the listing,
2053 treating it like Unix @code{ls -l} output, extracting the time-stamps.
2054 The rest is exactly the same as for @sc{http}. Note that when
2055 retrieving individual files from an @sc{ftp} server without using
2056 globbing or recursion, listing files will not be downloaded (and thus
2057 files will not be time-stamped) unless @samp{-N} is specified.
2059 Assumption that every directory listing is a Unix-style listing may
2060 sound extremely constraining, but in practice it is not, as many
2061 non-Unix @sc{ftp} servers use the Unixoid listing format because most
2062 (all?) of the clients understand it. Bear in mind that @sc{rfc959}
2063 defines no standard way to get a file list, let alone the time-stamps.
2064 We can only hope that a future standard will define this.
2066 Another non-standard solution includes the use of @code{MDTM} command
2067 that is supported by some @sc{ftp} servers (including the popular
2068 @code{wu-ftpd}), which returns the exact time of the specified file.
2069 Wget may support this command in the future.
2072 @chapter Startup File
2073 @cindex startup file
2079 Once you know how to change default settings of Wget through command
2080 line arguments, you may wish to make some of those settings permanent.
2081 You can do that in a convenient way by creating the Wget startup
2082 file---@file{.wgetrc}.
2084 Besides @file{.wgetrc} is the ``main'' initialization file, it is
2085 convenient to have a special facility for storing passwords. Thus Wget
2086 reads and interprets the contents of @file{$HOME/.netrc}, if it finds
2087 it. You can find @file{.netrc} format in your system manuals.
2089 Wget reads @file{.wgetrc} upon startup, recognizing a limited set of
2093 * Wgetrc Location:: Location of various wgetrc files.
2094 * Wgetrc Syntax:: Syntax of wgetrc.
2095 * Wgetrc Commands:: List of available commands.
2096 * Sample Wgetrc:: A wgetrc example.
2099 @node Wgetrc Location
2100 @section Wgetrc Location
2101 @cindex wgetrc location
2102 @cindex location of wgetrc
2104 When initializing, Wget will look for a @dfn{global} startup file,
2105 @file{/usr/local/etc/wgetrc} by default (or some prefix other than
2106 @file{/usr/local}, if Wget was not installed there) and read commands
2107 from there, if it exists.
2109 Then it will look for the user's file. If the environmental variable
2110 @code{WGETRC} is set, Wget will try to load that file. Failing that, no
2111 further attempts will be made.
2113 If @code{WGETRC} is not set, Wget will try to load @file{$HOME/.wgetrc}.
2115 The fact that user's settings are loaded after the system-wide ones
2116 means that in case of collision user's wgetrc @emph{overrides} the
2117 system-wide wgetrc (in @file{/usr/local/etc/wgetrc} by default).
2118 Fascist admins, away!
2121 @section Wgetrc Syntax
2122 @cindex wgetrc syntax
2123 @cindex syntax of wgetrc
2125 The syntax of a wgetrc command is simple:
2131 The @dfn{variable} will also be called @dfn{command}. Valid
2132 @dfn{values} are different for different commands.
2134 The commands are case-insensitive and underscore-insensitive. Thus
2135 @samp{DIr__PrefiX} is the same as @samp{dirprefix}. Empty lines, lines
2136 beginning with @samp{#} and lines containing white-space only are
2139 Commands that expect a comma-separated list will clear the list on an
2140 empty command. So, if you wish to reset the rejection list specified in
2141 global @file{wgetrc}, you can do it with:
2147 @node Wgetrc Commands
2148 @section Wgetrc Commands
2149 @cindex wgetrc commands
2151 The complete set of commands is listed below. Legal values are listed
2152 after the @samp{=}. Simple Boolean values can be set or unset using
2153 @samp{on} and @samp{off} or @samp{1} and @samp{0}. A fancier kind of
2154 Boolean allowed in some cases is the @dfn{lockable Boolean}, which may
2155 be set to @samp{on}, @samp{off}, @samp{always}, or @samp{never}. If an
2156 option is set to @samp{always} or @samp{never}, that value will be
2157 locked in for the duration of the Wget invocation---command-line options
2160 Some commands take pseudo-arbitrary values. @var{address} values can be
2161 hostnames or dotted-quad IP addresses. @var{n} can be any positive
2162 integer, or @samp{inf} for infinity, where appropriate. @var{string}
2163 values can be any non-empty string.
2165 Most of these commands have command-line equivalents (@pxref{Invoking}),
2166 though some of the more obscure or rarely used ones do not.
2169 @item accept/reject = @var{string}
2170 Same as @samp{-A}/@samp{-R} (@pxref{Types of Files}).
2172 @item add_hostdir = on/off
2173 Enable/disable host-prefixed file names. @samp{-nH} disables it.
2175 @item continue = on/off
2176 If set to on, force continuation of preexistent partially retrieved
2177 files. See @samp{-c} before setting it.
2179 @item background = on/off
2180 Enable/disable going to background---the same as @samp{-b} (which
2183 @item backup_converted = on/off
2184 Enable/disable saving pre-converted files with the suffix
2185 @samp{.orig}---the same as @samp{-K} (which enables it).
2187 @c @item backups = @var{number}
2188 @c #### Document me!
2190 @item base = @var{string}
2191 Consider relative @sc{url}s in @sc{url} input files forced to be
2192 interpreted as @sc{html} as being relative to @var{string}---the same as
2195 @item bind_address = @var{address}
2196 Bind to @var{address}, like the @samp{--bind-address} option.
2198 @item cache = on/off
2199 When set to off, disallow server-caching. See the @samp{--no-cache}
2202 @item convert_links = on/off
2203 Convert non-relative links locally. The same as @samp{-k}.
2205 @item cookies = on/off
2206 When set to off, disallow cookies. See the @samp{--cookies} option.
2208 @item load_cookies = @var{file}
2209 Load cookies from @var{file}. See @samp{--load-cookies}.
2211 @item save_cookies = @var{file}
2212 Save cookies to @var{file}. See @samp{--save-cookies}.
2214 @item connect_timeout = @var{n}
2215 Set the connect timeout---the same as @samp{--connect-timeout}.
2217 @item cut_dirs = @var{n}
2218 Ignore @var{n} remote directory components.
2220 @item debug = on/off
2221 Debug mode, same as @samp{-d}.
2223 @item delete_after = on/off
2224 Delete after download---the same as @samp{--delete-after}.
2226 @item dir_prefix = @var{string}
2227 Top of directory tree---the same as @samp{-P}.
2229 @item dirstruct = on/off
2230 Turning dirstruct on or off---the same as @samp{-x} or @samp{-nd},
2233 @item dns_cache = on/off
2234 Turn DNS caching on/off. Since DNS caching is on by default, this
2235 option is normally used to turn it off. Same as @samp{--dns-cache}.
2237 @item dns_timeout = @var{n}
2238 Set the DNS timeout---the same as @samp{--dns-timeout}.
2240 @item domains = @var{string}
2241 Same as @samp{-D} (@pxref{Spanning Hosts}).
2243 @item dot_bytes = @var{n}
2244 Specify the number of bytes ``contained'' in a dot, as seen throughout
2245 the retrieval (1024 by default). You can postfix the value with
2246 @samp{k} or @samp{m}, representing kilobytes and megabytes,
2247 respectively. With dot settings you can tailor the dot retrieval to
2248 suit your needs, or you can use the predefined @dfn{styles}
2249 (@pxref{Download Options}).
2251 @item dots_in_line = @var{n}
2252 Specify the number of dots that will be printed in each line throughout
2253 the retrieval (50 by default).
2255 @item dot_spacing = @var{n}
2256 Specify the number of dots in a single cluster (10 by default).
2258 @item exclude_directories = @var{string}
2259 Specify a comma-separated list of directories you wish to exclude from
2260 download---the same as @samp{-X} (@pxref{Directory-Based Limits}).
2262 @item exclude_domains = @var{string}
2263 Same as @samp{--exclude-domains} (@pxref{Spanning Hosts}).
2265 @item follow_ftp = on/off
2266 Follow @sc{ftp} links from @sc{html} documents---the same as
2267 @samp{--follow-ftp}.
2269 @item follow_tags = @var{string}
2270 Only follow certain @sc{html} tags when doing a recursive retrieval, just like
2271 @samp{--follow-tags}.
2273 @item force_html = on/off
2274 If set to on, force the input filename to be regarded as an @sc{html}
2275 document---the same as @samp{-F}.
2277 @item ftp_proxy = @var{string}
2278 Use @var{string} as @sc{ftp} proxy, instead of the one specified in
2282 Turn globbing on/off---the same as @samp{--glob} and @samp{--no-glob}.
2284 @item header = @var{string}
2285 Define an additional header, like @samp{--header}.
2287 @item html_extension = on/off
2288 Add a @samp{.html} extension to @samp{text/html} or
2289 @samp{application/xhtml+xml} files without it, like
2292 @item http_passwd = @var{string}
2293 Set @sc{http} password.
2295 @item http_proxy = @var{string}
2296 Use @var{string} as @sc{http} proxy, instead of the one specified in
2299 @item http_user = @var{string}
2300 Set @sc{http} user to @var{string}.
2302 @item ignore_length = on/off
2303 When set to on, ignore @code{Content-Length} header; the same as
2304 @samp{--ignore-length}.
2306 @item ignore_tags = @var{string}
2307 Ignore certain @sc{html} tags when doing a recursive retrieval, just like
2308 @samp{--ignore-tags}.
2310 @item include_directories = @var{string}
2311 Specify a comma-separated list of directories you wish to follow when
2312 downloading---the same as @samp{-I}.
2314 @item input = @var{string}
2315 Read the @sc{url}s from @var{string}, like @samp{-i}.
2317 @item kill_longer = on/off
2318 Consider data longer than specified in content-length header as invalid
2319 (and retry getting it). The default behavior is to save as much data
2320 as there is, provided there is more than or equal to the value in
2321 @code{Content-Length}.
2323 @item limit_rate = @var{rate}
2324 Limit the download speed to no more than @var{rate} bytes per second.
2325 The same as @samp{--limit-rate}.
2327 @item logfile = @var{string}
2328 Set logfile---the same as @samp{-o}.
2330 @item login = @var{string}
2331 Your user name on the remote machine, for @sc{ftp}. Defaults to
2334 @item mirror = on/off
2335 Turn mirroring on/off. The same as @samp{-m}.
2337 @item netrc = on/off
2338 Turn reading netrc on or off.
2340 @item noclobber = on/off
2343 @item no_parent = on/off
2344 Disallow retrieving outside the directory hierarchy, like
2345 @samp{--no-parent} (@pxref{Directory-Based Limits}).
2347 @item no_proxy = @var{string}
2348 Use @var{string} as the comma-separated list of domains to avoid in
2349 proxy loading, instead of the one specified in environment.
2351 @item output_document = @var{string}
2352 Set the output filename---the same as @samp{-O}.
2354 @item page_requisites = on/off
2355 Download all ancillary documents necessary for a single @sc{html} page to
2356 display properly---the same as @samp{-p}.
2358 @item passive_ftp = on/off/always/never
2359 Set passive @sc{ftp}---the same as @samp{--passive-ftp}. Some scripts
2360 and @samp{.pm} (Perl module) files download files using @samp{wget
2361 --passive-ftp}. If your firewall does not allow this, you can set
2362 @samp{passive_ftp = never} to override the command-line.
2364 @item passwd = @var{string}
2365 Set your @sc{ftp} password to @var{password}. Without this setting, the
2366 password defaults to @samp{username@@hostname.domainname}.
2368 @item post_data = @var{string}
2369 Use POST as the method for all HTTP requests and send @var{string} in
2370 the request body. The same as @samp{--post-data}.
2372 @item post_file = @var{file}
2373 Use POST as the method for all HTTP requests and send the contents of
2374 @var{file} in the request body. The same as @samp{--post-file}.
2376 @item progress = @var{string}
2377 Set the type of the progress indicator. Legal types are ``dot'' and
2380 @item proxy_user = @var{string}
2381 Set proxy authentication user name to @var{string}, like @samp{--proxy-user}.
2383 @item proxy_passwd = @var{string}
2384 Set proxy authentication password to @var{string}, like @samp{--proxy-passwd}.
2386 @item referer = @var{string}
2387 Set HTTP @samp{Referer:} header just like @samp{--referer}. (Note it
2388 was the folks who wrote the @sc{http} spec who got the spelling of
2389 ``referrer'' wrong.)
2391 @item quiet = on/off
2392 Quiet mode---the same as @samp{-q}.
2394 @item quota = @var{quota}
2395 Specify the download quota, which is useful to put in the global
2396 @file{wgetrc}. When download quota is specified, Wget will stop
2397 retrieving after the download sum has become greater than quota. The
2398 quota can be specified in bytes (default), kbytes @samp{k} appended) or
2399 mbytes (@samp{m} appended). Thus @samp{quota = 5m} will set the quota
2400 to 5 mbytes. Note that the user's startup file overrides system
2403 @item read_timeout = @var{n}
2404 Set the read (and write) timeout---the same as @samp{--read-timeout}.
2406 @item reclevel = @var{n}
2407 Recursion level---the same as @samp{-l}.
2409 @item recursive = on/off
2410 Recursive on/off---the same as @samp{-r}.
2412 @item relative_only = on/off
2413 Follow only relative links---the same as @samp{-L} (@pxref{Relative
2416 @item remove_listing = on/off
2417 If set to on, remove @sc{ftp} listings downloaded by Wget. Setting it
2418 to off is the same as @samp{-nr}.
2420 @item restrict_file_names = unix/windows
2421 Restrict the file names generated by Wget from URLs. See
2422 @samp{--restrict-file-names} for a more detailed description.
2424 @item retr_symlinks = on/off
2425 When set to on, retrieve symbolic links as if they were plain files; the
2426 same as @samp{--retr-symlinks}.
2428 @item robots = on/off
2429 Specify whether the norobots convention is respected by Wget, ``on'' by
2430 default. This switch controls both the @file{/robots.txt} and the
2431 @samp{nofollow} aspect of the spec. @xref{Robot Exclusion}, for more
2432 details about this. Be sure you know what you are doing before turning
2435 @item server_response = on/off
2436 Choose whether or not to print the @sc{http} and @sc{ftp} server
2437 responses---the same as @samp{-S}.
2439 @item span_hosts = on/off
2442 @item strict_comments = on/off
2443 Same as @samp{--strict-comments}.
2445 @item timeout = @var{n}
2446 Set timeout value---the same as @samp{-T}.
2448 @item timestamping = on/off
2449 Turn timestamping on/off. The same as @samp{-N} (@pxref{Time-Stamping}).
2451 @item tries = @var{n}
2452 Set number of retries per @sc{url}---the same as @samp{-t}.
2454 @item use_proxy = on/off
2455 Turn proxy support on/off. The same as @samp{-Y}.
2457 @item verbose = on/off
2458 Turn verbose on/off---the same as @samp{-v}/@samp{-nv}.
2460 @item wait = @var{n}
2461 Wait @var{n} seconds between retrievals---the same as @samp{-w}.
2463 @item waitretry = @var{n}
2464 Wait up to @var{n} seconds between retries of failed retrievals
2465 only---the same as @samp{--waitretry}. Note that this is turned on by
2466 default in the global @file{wgetrc}.
2468 @item randomwait = on/off
2469 Turn random between-request wait times on or off. The same as
2470 @samp{--random-wait}.
2474 @section Sample Wgetrc
2475 @cindex sample wgetrc
2477 This is the sample initialization file, as given in the distribution.
2478 It is divided in two section---one for global usage (suitable for global
2479 startup file), and one for local usage (suitable for
2480 @file{$HOME/.wgetrc}). Be careful about the things you change.
2482 Note that almost all the lines are commented out. For a command to have
2483 any effect, you must remove the @samp{#} character at the beginning of
2487 @include sample.wgetrc.munged_for_texi_inclusion
2494 @c man begin EXAMPLES
2495 The examples are divided into three sections loosely based on their
2499 * Simple Usage:: Simple, basic usage of the program.
2500 * Advanced Usage:: Advanced tips.
2501 * Very Advanced Usage:: The hairy stuff.
2505 @section Simple Usage
2509 Say you want to download a @sc{url}. Just type:
2512 wget http://fly.srk.fer.hr/
2516 But what will happen if the connection is slow, and the file is lengthy?
2517 The connection will probably fail before the whole file is retrieved,
2518 more than once. In this case, Wget will try getting the file until it
2519 either gets the whole of it, or exceeds the default number of retries
2520 (this being 20). It is easy to change the number of tries to 45, to
2521 insure that the whole file will arrive safely:
2524 wget --tries=45 http://fly.srk.fer.hr/jpg/flyweb.jpg
2528 Now let's leave Wget to work in the background, and write its progress
2529 to log file @file{log}. It is tiring to type @samp{--tries}, so we
2530 shall use @samp{-t}.
2533 wget -t 45 -o log http://fly.srk.fer.hr/jpg/flyweb.jpg &
2536 The ampersand at the end of the line makes sure that Wget works in the
2537 background. To unlimit the number of retries, use @samp{-t inf}.
2540 The usage of @sc{ftp} is as simple. Wget will take care of login and
2544 wget ftp://gnjilux.srk.fer.hr/welcome.msg
2548 If you specify a directory, Wget will retrieve the directory listing,
2549 parse it and convert it to @sc{html}. Try:
2552 wget ftp://ftp.gnu.org/pub/gnu/
2557 @node Advanced Usage
2558 @section Advanced Usage
2562 You have a file that contains the URLs you want to download? Use the
2569 If you specify @samp{-} as file name, the @sc{url}s will be read from
2573 Create a five levels deep mirror image of the GNU web site, with the
2574 same directory structure the original has, with only one try per
2575 document, saving the log of the activities to @file{gnulog}:
2578 wget -r http://www.gnu.org/ -o gnulog
2582 The same as the above, but convert the links in the @sc{html} files to
2583 point to local files, so you can view the documents off-line:
2586 wget --convert-links -r http://www.gnu.org/ -o gnulog
2590 Retrieve only one @sc{html} page, but make sure that all the elements needed
2591 for the page to be displayed, such as inline images and external style
2592 sheets, are also downloaded. Also make sure the downloaded page
2593 references the downloaded links.
2596 wget -p --convert-links http://www.server.com/dir/page.html
2599 The @sc{html} page will be saved to @file{www.server.com/dir/page.html}, and
2600 the images, stylesheets, etc., somewhere under @file{www.server.com/},
2601 depending on where they were on the remote server.
2604 The same as the above, but without the @file{www.server.com/} directory.
2605 In fact, I don't want to have all those random server directories
2606 anyway---just save @emph{all} those files under a @file{download/}
2607 subdirectory of the current directory.
2610 wget -p --convert-links -nH -nd -Pdownload \
2611 http://www.server.com/dir/page.html
2615 Retrieve the index.html of @samp{www.lycos.com}, showing the original
2619 wget -S http://www.lycos.com/
2623 Save the server headers with the file, perhaps for post-processing.
2626 wget -s http://www.lycos.com/
2631 Retrieve the first two levels of @samp{wuarchive.wustl.edu}, saving them
2635 wget -r -l2 -P/tmp ftp://wuarchive.wustl.edu/
2639 You want to download all the @sc{gif}s from a directory on an @sc{http}
2640 server. You tried @samp{wget http://www.server.com/dir/*.gif}, but that
2641 didn't work because @sc{http} retrieval does not support globbing. In
2645 wget -r -l1 --no-parent -A.gif http://www.server.com/dir/
2648 More verbose, but the effect is the same. @samp{-r -l1} means to
2649 retrieve recursively (@pxref{Recursive Download}), with maximum depth
2650 of 1. @samp{--no-parent} means that references to the parent directory
2651 are ignored (@pxref{Directory-Based Limits}), and @samp{-A.gif} means to
2652 download only the @sc{gif} files. @samp{-A "*.gif"} would have worked
2656 Suppose you were in the middle of downloading, when Wget was
2657 interrupted. Now you do not want to clobber the files already present.
2661 wget -nc -r http://www.gnu.org/
2665 If you want to encode your own username and password to @sc{http} or
2666 @sc{ftp}, use the appropriate @sc{url} syntax (@pxref{URL Format}).
2669 wget ftp://hniksic:mypassword@@unix.server.com/.emacs
2672 Note, however, that this usage is not advisable on multi-user systems
2673 because it reveals your password to anyone who looks at the output of
2676 @cindex redirecting output
2678 You would like the output documents to go to standard output instead of
2682 wget -O - http://jagor.srce.hr/ http://www.srce.hr/
2685 You can also combine the two options and make pipelines to retrieve the
2686 documents from remote hotlists:
2689 wget -O - http://cool.list.com/ | wget --force-html -i -
2693 @node Very Advanced Usage
2694 @section Very Advanced Usage
2699 If you wish Wget to keep a mirror of a page (or @sc{ftp}
2700 subdirectories), use @samp{--mirror} (@samp{-m}), which is the shorthand
2701 for @samp{-r -l inf -N}. You can put Wget in the crontab file asking it
2702 to recheck a site each Sunday:
2706 0 0 * * 0 wget --mirror http://www.gnu.org/ -o /home/me/weeklog
2710 In addition to the above, you want the links to be converted for local
2711 viewing. But, after having read this manual, you know that link
2712 conversion doesn't play well with timestamping, so you also want Wget to
2713 back up the original @sc{html} files before the conversion. Wget invocation
2714 would look like this:
2717 wget --mirror --convert-links --backup-converted \
2718 http://www.gnu.org/ -o /home/me/weeklog
2722 But you've also noticed that local viewing doesn't work all that well
2723 when @sc{html} files are saved under extensions other than @samp{.html},
2724 perhaps because they were served as @file{index.cgi}. So you'd like
2725 Wget to rename all the files served with content-type @samp{text/html}
2726 or @samp{application/xhtml+xml} to @file{@var{name}.html}.
2729 wget --mirror --convert-links --backup-converted \
2730 --html-extension -o /home/me/weeklog \
2734 Or, with less typing:
2737 wget -m -k -K -E http://www.gnu.org/ -o /home/me/weeklog
2746 This chapter contains all the stuff that could not fit anywhere else.
2749 * Proxies:: Support for proxy servers
2750 * Distribution:: Getting the latest version.
2751 * Mailing List:: Wget mailing list for announcements and discussion.
2752 * Reporting Bugs:: How and where to report bugs.
2753 * Portability:: The systems Wget works on.
2754 * Signals:: Signal-handling performed by Wget.
2761 @dfn{Proxies} are special-purpose @sc{http} servers designed to transfer
2762 data from remote servers to local clients. One typical use of proxies
2763 is lightening network load for users behind a slow connection. This is
2764 achieved by channeling all @sc{http} and @sc{ftp} requests through the
2765 proxy which caches the transferred data. When a cached resource is
2766 requested again, proxy will return the data from cache. Another use for
2767 proxies is for companies that separate (for security reasons) their
2768 internal networks from the rest of Internet. In order to obtain
2769 information from the Web, their users connect and retrieve remote data
2770 using an authorized proxy.
2772 Wget supports proxies for both @sc{http} and @sc{ftp} retrievals. The
2773 standard way to specify proxy location, which Wget recognizes, is using
2774 the following environment variables:
2778 This variable should contain the @sc{url} of the proxy for @sc{http}
2782 This variable should contain the @sc{url} of the proxy for @sc{ftp}
2783 connections. It is quite common that @sc{http_proxy} and @sc{ftp_proxy}
2784 are set to the same @sc{url}.
2787 This variable should contain a comma-separated list of domain extensions
2788 proxy should @emph{not} be used for. For instance, if the value of
2789 @code{no_proxy} is @samp{.mit.edu}, proxy will not be used to retrieve
2793 In addition to the environment variables, proxy location and settings
2794 may be specified from within Wget itself.
2798 @itemx --proxy=on/off
2799 @itemx proxy = on/off
2800 This option may be used to turn the proxy support on or off. Proxy
2801 support is on by default, provided that the appropriate environment
2804 @item http_proxy = @var{URL}
2805 @itemx ftp_proxy = @var{URL}
2806 @itemx no_proxy = @var{string}
2807 These startup file variables allow you to override the proxy settings
2808 specified by the environment.
2811 Some proxy servers require authorization to enable you to use them. The
2812 authorization consists of @dfn{username} and @dfn{password}, which must
2813 be sent by Wget. As with @sc{http} authorization, several
2814 authentication schemes exist. For proxy authorization only the
2815 @code{Basic} authentication scheme is currently implemented.
2817 You may specify your username and password either through the proxy
2818 @sc{url} or through the command-line options. Assuming that the
2819 company's proxy is located at @samp{proxy.company.com} at port 8001, a
2820 proxy @sc{url} location containing authorization data might look like
2824 http://hniksic:mypassword@@proxy.company.com:8001/
2827 Alternatively, you may use the @samp{proxy-user} and
2828 @samp{proxy-password} options, and the equivalent @file{.wgetrc}
2829 settings @code{proxy_user} and @code{proxy_passwd} to set the proxy
2830 username and password.
2833 @section Distribution
2834 @cindex latest version
2836 Like all GNU utilities, the latest version of Wget can be found at the
2837 master GNU archive site ftp.gnu.org, and its mirrors. For example,
2838 Wget @value{VERSION} can be found at
2839 @url{ftp://ftp.gnu.org/pub/gnu/wget/wget-@value{VERSION}.tar.gz}
2842 @section Mailing List
2843 @cindex mailing list
2846 Wget has its own mailing list at @email{wget@@sunsite.dk}, thanks
2847 to Karsten Thygesen. The mailing list is for discussion of Wget
2848 features and web, reporting Wget bugs (those that you think may be of
2849 interest to the public) and mailing announcements. You are welcome to
2850 subscribe. The more people on the list, the better!
2852 To subscribe, simply send mail to @email{wget-subscribe@@sunsite.dk}.
2853 Unsubscribe by mailing to @email{wget-unsubscribe@@sunsite.dk}.
2855 The mailing list is archived at @url{http://fly.srk.fer.hr/archive/wget}.
2856 Alternative archive is available at
2857 @url{http://www.mail-archive.com/wget%40sunsite.auc.dk/}.
2859 @node Reporting Bugs
2860 @section Reporting Bugs
2862 @cindex reporting bugs
2866 You are welcome to send bug reports about GNU Wget to
2867 @email{bug-wget@@gnu.org}.
2869 Before actually submitting a bug report, please try to follow a few
2874 Please try to ascertain that the behavior you see really is a bug. If
2875 Wget crashes, it's a bug. If Wget does not behave as documented,
2876 it's a bug. If things work strange, but you are not sure about the way
2877 they are supposed to work, it might well be a bug.
2880 Try to repeat the bug in as simple circumstances as possible. E.g. if
2881 Wget crashes while downloading @samp{wget -rl0 -kKE -t5 -Y0
2882 http://yoyodyne.com -o /tmp/log}, you should try to see if the crash is
2883 repeatable, and if will occur with a simpler set of options. You might
2884 even try to start the download at the page where the crash occurred to
2885 see if that page somehow triggered the crash.
2887 Also, while I will probably be interested to know the contents of your
2888 @file{.wgetrc} file, just dumping it into the debug message is probably
2889 a bad idea. Instead, you should first try to see if the bug repeats
2890 with @file{.wgetrc} moved out of the way. Only if it turns out that
2891 @file{.wgetrc} settings affect the bug, mail me the relevant parts of
2895 Please start Wget with @samp{-d} option and send the log (or the
2896 relevant parts of it). If Wget was compiled without debug support,
2897 recompile it. It is @emph{much} easier to trace bugs with debug support
2901 If Wget has crashed, try to run it in a debugger, e.g. @code{gdb `which
2902 wget` core} and type @code{where} to get the backtrace.
2907 @section Portability
2909 @cindex operating systems
2911 Since Wget uses GNU Autoconf for building and configuring, and avoids
2912 using ``special'' ultra--mega--cool features of any particular Unix, it
2913 should compile (and work) on all common Unix flavors.
2915 Various Wget versions have been compiled and tested under many kinds of
2916 Unix systems, including Solaris, Linux, SunOS, OSF (aka Digital Unix),
2917 Ultrix, *BSD, IRIX, and others; refer to the file @file{MACHINES} in the
2918 distribution directory for a comprehensive list. If you compile it on
2919 an architecture not listed there, please let me know so I can update it.
2921 Wget should also compile on the other Unix systems, not listed in
2922 @file{MACHINES}. If it doesn't, please let me know.
2924 Thanks to kind contributors, this version of Wget compiles and works on
2925 Microsoft Windows 95 and Windows NT platforms. It has been compiled
2926 successfully using MS Visual C++ 4.0, Watcom, and Borland C compilers,
2927 with Winsock as networking software. Naturally, it is crippled of some
2928 features available on Unix, but it should work as a substitute for
2929 people stuck with Windows. Note that the Windows port is
2930 @strong{neither tested nor maintained} by me---all questions and
2931 problems should be reported to Wget mailing list at
2932 @email{wget@@sunsite.dk} where the maintainers will look at them.
2936 @cindex signal handling
2939 Since the purpose of Wget is background work, it catches the hangup
2940 signal (@code{SIGHUP}) and ignores it. If the output was on standard
2941 output, it will be redirected to a file named @file{wget-log}.
2942 Otherwise, @code{SIGHUP} is ignored. This is convenient when you wish
2943 to redirect the output of Wget after having started it.
2946 $ wget http://www.ifi.uio.no/~larsi/gnus.tar.gz &
2947 $ kill -HUP %% # Redirect the output to wget-log
2950 Other than that, Wget will not try to interfere with signals in any way.
2951 @kbd{C-c}, @code{kill -TERM} and @code{kill -KILL} should kill it alike.
2956 This chapter contains some references I consider useful.
2959 * Robot Exclusion:: Wget's support for RES.
2960 * Security Considerations:: Security with Wget.
2961 * Contributors:: People who helped.
2964 @node Robot Exclusion
2965 @section Robot Exclusion
2966 @cindex robot exclusion
2968 @cindex server maintenance
2970 It is extremely easy to make Wget wander aimlessly around a web site,
2971 sucking all the available data in progress. @samp{wget -r @var{site}},
2972 and you're set. Great? Not for the server admin.
2974 As long as Wget is only retrieving static pages, and doing it at a
2975 reasonable rate (see the @samp{--wait} option), there's not much of a
2976 problem. The trouble is that Wget can't tell the difference between the
2977 smallest static page and the most demanding CGI. A site I know has a
2978 section handled by a CGI Perl script that converts Info files to @sc{html} on
2979 the fly. The script is slow, but works well enough for human users
2980 viewing an occasional Info file. However, when someone's recursive Wget
2981 download stumbles upon the index page that links to all the Info files
2982 through the script, the system is brought to its knees without providing
2983 anything useful to the user (This task of converting Info files could be
2984 done locally and access to Info documentation for all installed GNU
2985 software on a system is available from the @code{info} command).
2987 To avoid this kind of accident, as well as to preserve privacy for
2988 documents that need to be protected from well-behaved robots, the
2989 concept of @dfn{robot exclusion} was invented. The idea is that
2990 the server administrators and document authors can specify which
2991 portions of the site they wish to protect from robots and those
2992 they will permit access.
2994 The most popular mechanism, and the @i{de facto} standard supported by
2995 all the major robots, is the ``Robots Exclusion Standard'' (RES) written
2996 by Martijn Koster et al. in 1994. It specifies the format of a text
2997 file containing directives that instruct the robots which URL paths to
2998 avoid. To be found by the robots, the specifications must be placed in
2999 @file{/robots.txt} in the server root, which the robots are expected to
3002 Although Wget is not a web robot in the strictest sense of the word, it
3003 can downloads large parts of the site without the user's intervention to
3004 download an individual page. Because of that, Wget honors RES when
3005 downloading recursively. For instance, when you issue:
3008 wget -r http://www.server.com/
3011 First the index of @samp{www.server.com} will be downloaded. If Wget
3012 finds that it wants to download more documents from that server, it will
3013 request @samp{http://www.server.com/robots.txt} and, if found, use it
3014 for further downloads. @file{robots.txt} is loaded only once per each
3017 Until version 1.8, Wget supported the first version of the standard,
3018 written by Martijn Koster in 1994 and available at
3019 @url{http://www.robotstxt.org/wc/norobots.html}. As of version 1.8,
3020 Wget has supported the additional directives specified in the internet
3021 draft @samp{<draft-koster-robots-00.txt>} titled ``A Method for Web
3022 Robots Control''. The draft, which has as far as I know never made to
3023 an @sc{rfc}, is available at
3024 @url{http://www.robotstxt.org/wc/norobots-rfc.txt}.
3026 This manual no longer includes the text of the Robot Exclusion Standard.
3028 The second, less known mechanism, enables the author of an individual
3029 document to specify whether they want the links from the file to be
3030 followed by a robot. This is achieved using the @code{META} tag, like
3034 <meta name="robots" content="nofollow">
3037 This is explained in some detail at
3038 @url{http://www.robotstxt.org/wc/meta-user.html}. Wget supports this
3039 method of robot exclusion in addition to the usual @file{/robots.txt}
3042 If you know what you are doing and really really wish to turn off the
3043 robot exclusion, set the @code{robots} variable to @samp{off} in your
3044 @file{.wgetrc}. You can achieve the same effect from the command line
3045 using the @code{-e} switch, e.g. @samp{wget -e robots=off @var{url}...}.
3047 @node Security Considerations
3048 @section Security Considerations
3051 When using Wget, you must be aware that it sends unencrypted passwords
3052 through the network, which may present a security problem. Here are the
3053 main issues, and some solutions.
3056 @item The passwords on the command line are visible using @code{ps}.
3057 The best way around it is to use @code{wget -i -} and feed the @sc{url}s
3058 to Wget's standard input, each on a separate line, terminated by
3059 @kbd{C-d}. Another workaround is to use @file{.netrc} to store
3060 passwords; however, storing unencrypted passwords is also considered a
3064 Using the insecure @dfn{basic} authentication scheme, unencrypted
3065 passwords are transmitted through the network routers and gateways.
3068 The @sc{ftp} passwords are also in no way encrypted. There is no good
3069 solution for this at the moment.
3072 Although the ``normal'' output of Wget tries to hide the passwords,
3073 debugging logs show them, in all forms. This problem is avoided by
3074 being careful when you send debug logs (yes, even when you send them to
3079 @section Contributors
3080 @cindex contributors
3083 GNU Wget was written by Hrvoje Nik@v{s}i@'{c} @email{hniksic@@xemacs.org}.
3086 GNU Wget was written by Hrvoje Niksic @email{hniksic@@xemacs.org}.
3088 However, its development could never have gone as far as it has, were it
3089 not for the help of many people, either with bug reports, feature
3090 proposals, patches, or letters saying ``Thanks!''.
3092 Special thanks goes to the following people (no particular order):
3096 Karsten Thygesen---donated system resources such as the mailing list,
3097 web space, and @sc{ftp} space, along with a lot of time to make these
3101 Shawn McHorse---bug reports and patches.
3104 Kaveh R. Ghazi---on-the-fly @code{ansi2knr}-ization. Lots of
3108 Gordon Matzigkeit---@file{.netrc} support.
3112 Zlatko @v{C}alu@v{s}i@'{c}, Tomislav Vujec and Dra@v{z}en
3113 Ka@v{c}ar---feature suggestions and ``philosophical'' discussions.
3116 Zlatko Calusic, Tomislav Vujec and Drazen Kacar---feature suggestions
3117 and ``philosophical'' discussions.
3121 Darko Budor---initial port to Windows.
3124 Antonio Rosella---help and suggestions, plus the Italian translation.
3128 Tomislav Petrovi@'{c}, Mario Miko@v{c}evi@'{c}---many bug reports and
3132 Tomislav Petrovic, Mario Mikocevic---many bug reports and suggestions.
3137 Fran@,{c}ois Pinard---many thorough bug reports and discussions.
3140 Francois Pinard---many thorough bug reports and discussions.
3144 Karl Eichwalder---lots of help with internationalization and other
3148 Junio Hamano---donated support for Opie and @sc{http} @code{Digest}
3152 The people who provided donations for development, including Brian
3156 The following people have provided patches, bug/build reports, useful
3157 suggestions, beta testing services, fan mail and all the other things
3158 that make maintenance so much fun:
3177 Kristijan @v{C}onka@v{s},
3196 Aleksandar Erkalovi@'{c},
3199 Aleksandar Erkalovic,
3216 Erik Magnus Hulthen,
3234 Goran Kezunovi@'{c},
3245 $\Sigma\acute{\iota}\mu o\varsigma\;
3246 \Xi\varepsilon\nu\iota\tau\acute{\epsilon}\lambda\lambda\eta\varsigma$
3247 (Simos KSenitellis),
3255 Nicol@'{a}s Lichtmeier,
3261 Alexander V. Lukyanov,
3289 @c Texinfo doesn't grok @'{@i}, so we have to use TeX itself.
3291 Juan Jos\'{e} Rodr\'{\i}gues,
3294 Juan Jose Rodrigues,
3307 Szakacsits Szabolcs,
3315 Douglas E. Wegscheid,
3325 Apologies to all who I accidentally left out, and many thanks to all the
3326 subscribers of the Wget mailing list.
3333 @cindex free software
3335 GNU Wget is licensed under the GNU GPL, which makes it @dfn{free
3338 Please note that ``free'' in ``free software'' refers to liberty, not
3339 price. As some GNU project advocates like to point out, think of ``free
3340 speech'' rather than ``free beer''. The exact and legally binding
3341 distribution terms are spelled out below; in short, you have the right
3342 (freedom) to run and change Wget and distribute it to other people, and
3343 even---if you want---charge money for doing either. The important
3344 restriction is that you have to grant your recipients the same rights
3345 and impose the same restrictions.
3347 This method of licensing software is also known as @dfn{open source}
3348 because, among other things, it makes sure that all recipients will
3349 receive the source code along with the program, and be able to improve
3350 it. The GNU project prefers the term ``free software'' for reasons
3352 @url{http://www.gnu.org/philosophy/free-software-for-freedom.html}.
3354 The exact license terms are defined by this paragraph and the GNU
3355 General Public License it refers to:
3358 GNU Wget is free software; you can redistribute it and/or modify it
3359 under the terms of the GNU General Public License as published by the
3360 Free Software Foundation; either version 2 of the License, or (at your
3361 option) any later version.
3363 GNU Wget is distributed in the hope that it will be useful, but WITHOUT
3364 ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
3365 FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
3368 A copy of the GNU General Public License is included as part of this
3369 manual; if you did not receive it, write to the Free Software
3370 Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
3373 In addition to this, this manual is free in the same sense:
3376 Permission is granted to copy, distribute and/or modify this document
3377 under the terms of the GNU Free Documentation License, Version 1.1 or
3378 any later version published by the Free Software Foundation; with the
3379 Invariant Sections being ``GNU General Public License'' and ``GNU Free
3380 Documentation License'', with no Front-Cover Texts, and with no
3381 Back-Cover Texts. A copy of the license is included in the section
3382 entitled ``GNU Free Documentation License''.
3385 @c #### Maybe we should wrap these licenses in ifinfo? Stallman says
3386 @c that the GFDL needs to be present in the manual, and to me it would
3387 @c suck to include the license for the manual and not the license for
3390 The full texts of the GNU General Public License and of the GNU Free
3391 Documentation License are available below.
3394 * GNU General Public License::
3395 * GNU Free Documentation License::
3398 @node GNU General Public License
3399 @section GNU General Public License
3400 @center Version 2, June 1991
3403 Copyright @copyright{} 1989, 1991 Free Software Foundation, Inc.
3404 675 Mass Ave, Cambridge, MA 02139, USA
3406 Everyone is permitted to copy and distribute verbatim copies
3407 of this license document, but changing it is not allowed.
3410 @unnumberedsec Preamble
3412 The licenses for most software are designed to take away your
3413 freedom to share and change it. By contrast, the GNU General Public
3414 License is intended to guarantee your freedom to share and change free
3415 software---to make sure the software is free for all its users. This
3416 General Public License applies to most of the Free Software
3417 Foundation's software and to any other program whose authors commit to
3418 using it. (Some other Free Software Foundation software is covered by
3419 the GNU Library General Public License instead.) You can apply it to
3422 When we speak of free software, we are referring to freedom, not
3423 price. Our General Public Licenses are designed to make sure that you
3424 have the freedom to distribute copies of free software (and charge for
3425 this service if you wish), that you receive source code or can get it
3426 if you want it, that you can change the software or use pieces of it
3427 in new free programs; and that you know you can do these things.
3429 To protect your rights, we need to make restrictions that forbid
3430 anyone to deny you these rights or to ask you to surrender the rights.
3431 These restrictions translate to certain responsibilities for you if you
3432 distribute copies of the software, or if you modify it.
3434 For example, if you distribute copies of such a program, whether
3435 gratis or for a fee, you must give the recipients all the rights that
3436 you have. You must make sure that they, too, receive or can get the
3437 source code. And you must show them these terms so they know their
3440 We protect your rights with two steps: (1) copyright the software, and
3441 (2) offer you this license which gives you legal permission to copy,
3442 distribute and/or modify the software.
3444 Also, for each author's protection and ours, we want to make certain
3445 that everyone understands that there is no warranty for this free
3446 software. If the software is modified by someone else and passed on, we
3447 want its recipients to know that what they have is not the original, so
3448 that any problems introduced by others will not reflect on the original
3449 authors' reputations.
3451 Finally, any free program is threatened constantly by software
3452 patents. We wish to avoid the danger that redistributors of a free
3453 program will individually obtain patent licenses, in effect making the
3454 program proprietary. To prevent this, we have made it clear that any
3455 patent must be licensed for everyone's free use or not licensed at all.
3457 The precise terms and conditions for copying, distribution and
3458 modification follow.
3461 @unnumberedsec TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
3464 @center TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
3469 This License applies to any program or other work which contains
3470 a notice placed by the copyright holder saying it may be distributed
3471 under the terms of this General Public License. The ``Program'', below,
3472 refers to any such program or work, and a ``work based on the Program''
3473 means either the Program or any derivative work under copyright law:
3474 that is to say, a work containing the Program or a portion of it,
3475 either verbatim or with modifications and/or translated into another
3476 language. (Hereinafter, translation is included without limitation in
3477 the term ``modification''.) Each licensee is addressed as ``you''.
3479 Activities other than copying, distribution and modification are not
3480 covered by this License; they are outside its scope. The act of
3481 running the Program is not restricted, and the output from the Program
3482 is covered only if its contents constitute a work based on the
3483 Program (independent of having been made by running the Program).
3484 Whether that is true depends on what the Program does.
3487 You may copy and distribute verbatim copies of the Program's
3488 source code as you receive it, in any medium, provided that you
3489 conspicuously and appropriately publish on each copy an appropriate
3490 copyright notice and disclaimer of warranty; keep intact all the
3491 notices that refer to this License and to the absence of any warranty;
3492 and give any other recipients of the Program a copy of this License
3493 along with the Program.
3495 You may charge a fee for the physical act of transferring a copy, and
3496 you may at your option offer warranty protection in exchange for a fee.
3499 You may modify your copy or copies of the Program or any portion
3500 of it, thus forming a work based on the Program, and copy and
3501 distribute such modifications or work under the terms of Section 1
3502 above, provided that you also meet all of these conditions:
3506 You must cause the modified files to carry prominent notices
3507 stating that you changed the files and the date of any change.
3510 You must cause any work that you distribute or publish, that in
3511 whole or in part contains or is derived from the Program or any
3512 part thereof, to be licensed as a whole at no charge to all third
3513 parties under the terms of this License.
3516 If the modified program normally reads commands interactively
3517 when run, you must cause it, when started running for such
3518 interactive use in the most ordinary way, to print or display an
3519 announcement including an appropriate copyright notice and a
3520 notice that there is no warranty (or else, saying that you provide
3521 a warranty) and that users may redistribute the program under
3522 these conditions, and telling the user how to view a copy of this
3523 License. (Exception: if the Program itself is interactive but
3524 does not normally print such an announcement, your work based on
3525 the Program is not required to print an announcement.)
3528 These requirements apply to the modified work as a whole. If
3529 identifiable sections of that work are not derived from the Program,
3530 and can be reasonably considered independent and separate works in
3531 themselves, then this License, and its terms, do not apply to those
3532 sections when you distribute them as separate works. But when you
3533 distribute the same sections as part of a whole which is a work based
3534 on the Program, the distribution of the whole must be on the terms of
3535 this License, whose permissions for other licensees extend to the
3536 entire whole, and thus to each and every part regardless of who wrote it.
3538 Thus, it is not the intent of this section to claim rights or contest
3539 your rights to work written entirely by you; rather, the intent is to
3540 exercise the right to control the distribution of derivative or
3541 collective works based on the Program.
3543 In addition, mere aggregation of another work not based on the Program
3544 with the Program (or with a work based on the Program) on a volume of
3545 a storage or distribution medium does not bring the other work under
3546 the scope of this License.
3549 You may copy and distribute the Program (or a work based on it,
3550 under Section 2) in object code or executable form under the terms of
3551 Sections 1 and 2 above provided that you also do one of the following:
3555 Accompany it with the complete corresponding machine-readable
3556 source code, which must be distributed under the terms of Sections
3557 1 and 2 above on a medium customarily used for software interchange; or,
3560 Accompany it with a written offer, valid for at least three
3561 years, to give any third party, for a charge no more than your
3562 cost of physically performing source distribution, a complete
3563 machine-readable copy of the corresponding source code, to be
3564 distributed under the terms of Sections 1 and 2 above on a medium
3565 customarily used for software interchange; or,
3568 Accompany it with the information you received as to the offer
3569 to distribute corresponding source code. (This alternative is
3570 allowed only for noncommercial distribution and only if you
3571 received the program in object code or executable form with such
3572 an offer, in accord with Subsection b above.)
3575 The source code for a work means the preferred form of the work for
3576 making modifications to it. For an executable work, complete source
3577 code means all the source code for all modules it contains, plus any
3578 associated interface definition files, plus the scripts used to
3579 control compilation and installation of the executable. However, as a
3580 special exception, the source code distributed need not include
3581 anything that is normally distributed (in either source or binary
3582 form) with the major components (compiler, kernel, and so on) of the
3583 operating system on which the executable runs, unless that component
3584 itself accompanies the executable.
3586 If distribution of executable or object code is made by offering
3587 access to copy from a designated place, then offering equivalent
3588 access to copy the source code from the same place counts as
3589 distribution of the source code, even though third parties are not
3590 compelled to copy the source along with the object code.
3593 You may not copy, modify, sublicense, or distribute the Program
3594 except as expressly provided under this License. Any attempt
3595 otherwise to copy, modify, sublicense or distribute the Program is
3596 void, and will automatically terminate your rights under this License.
3597 However, parties who have received copies, or rights, from you under
3598 this License will not have their licenses terminated so long as such
3599 parties remain in full compliance.
3602 You are not required to accept this License, since you have not
3603 signed it. However, nothing else grants you permission to modify or
3604 distribute the Program or its derivative works. These actions are
3605 prohibited by law if you do not accept this License. Therefore, by
3606 modifying or distributing the Program (or any work based on the
3607 Program), you indicate your acceptance of this License to do so, and
3608 all its terms and conditions for copying, distributing or modifying
3609 the Program or works based on it.
3612 Each time you redistribute the Program (or any work based on the
3613 Program), the recipient automatically receives a license from the
3614 original licensor to copy, distribute or modify the Program subject to
3615 these terms and conditions. You may not impose any further
3616 restrictions on the recipients' exercise of the rights granted herein.
3617 You are not responsible for enforcing compliance by third parties to
3621 If, as a consequence of a court judgment or allegation of patent
3622 infringement or for any other reason (not limited to patent issues),
3623 conditions are imposed on you (whether by court order, agreement or
3624 otherwise) that contradict the conditions of this License, they do not
3625 excuse you from the conditions of this License. If you cannot
3626 distribute so as to satisfy simultaneously your obligations under this
3627 License and any other pertinent obligations, then as a consequence you
3628 may not distribute the Program at all. For example, if a patent
3629 license would not permit royalty-free redistribution of the Program by
3630 all those who receive copies directly or indirectly through you, then
3631 the only way you could satisfy both it and this License would be to
3632 refrain entirely from distribution of the Program.
3634 If any portion of this section is held invalid or unenforceable under
3635 any particular circumstance, the balance of the section is intended to
3636 apply and the section as a whole is intended to apply in other
3639 It is not the purpose of this section to induce you to infringe any
3640 patents or other property right claims or to contest validity of any
3641 such claims; this section has the sole purpose of protecting the
3642 integrity of the free software distribution system, which is
3643 implemented by public license practices. Many people have made
3644 generous contributions to the wide range of software distributed
3645 through that system in reliance on consistent application of that
3646 system; it is up to the author/donor to decide if he or she is willing
3647 to distribute software through any other system and a licensee cannot
3650 This section is intended to make thoroughly clear what is believed to
3651 be a consequence of the rest of this License.
3654 If the distribution and/or use of the Program is restricted in
3655 certain countries either by patents or by copyrighted interfaces, the
3656 original copyright holder who places the Program under this License
3657 may add an explicit geographical distribution limitation excluding
3658 those countries, so that distribution is permitted only in or among
3659 countries not thus excluded. In such case, this License incorporates
3660 the limitation as if written in the body of this License.
3663 The Free Software Foundation may publish revised and/or new versions
3664 of the General Public License from time to time. Such new versions will
3665 be similar in spirit to the present version, but may differ in detail to
3666 address new problems or concerns.
3668 Each version is given a distinguishing version number. If the Program
3669 specifies a version number of this License which applies to it and ``any
3670 later version'', you have the option of following the terms and conditions
3671 either of that version or of any later version published by the Free
3672 Software Foundation. If the Program does not specify a version number of
3673 this License, you may choose any version ever published by the Free Software
3677 If you wish to incorporate parts of the Program into other free
3678 programs whose distribution conditions are different, write to the author
3679 to ask for permission. For software which is copyrighted by the Free
3680 Software Foundation, write to the Free Software Foundation; we sometimes
3681 make exceptions for this. Our decision will be guided by the two goals
3682 of preserving the free status of all derivatives of our free software and
3683 of promoting the sharing and reuse of software generally.
3686 @heading NO WARRANTY
3694 BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
3695 FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
3696 OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
3697 PROVIDE THE PROGRAM ``AS IS'' WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
3698 OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
3699 MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
3700 TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
3701 PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
3702 REPAIR OR CORRECTION.
3705 IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
3706 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
3707 REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
3708 INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
3709 OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
3710 TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
3711 YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
3712 PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
3713 POSSIBILITY OF SUCH DAMAGES.
3717 @heading END OF TERMS AND CONDITIONS
3720 @center END OF TERMS AND CONDITIONS
3724 @unnumberedsec How to Apply These Terms to Your New Programs
3726 If you develop a new program, and you want it to be of the greatest
3727 possible use to the public, the best way to achieve this is to make it
3728 free software which everyone can redistribute and change under these terms.
3730 To do so, attach the following notices to the program. It is safest
3731 to attach them to the start of each source file to most effectively
3732 convey the exclusion of warranty; and each file should have at least
3733 the ``copyright'' line and a pointer to where the full notice is found.
3736 @var{one line to give the program's name and an idea of what it does.}
3737 Copyright (C) 19@var{yy} @var{name of author}
3739 This program is free software; you can redistribute it and/or
3740 modify it under the terms of the GNU General Public License
3741 as published by the Free Software Foundation; either version 2
3742 of the License, or (at your option) any later version.
3744 This program is distributed in the hope that it will be useful,
3745 but WITHOUT ANY WARRANTY; without even the implied warranty of
3746 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3747 GNU General Public License for more details.
3749 You should have received a copy of the GNU General Public License
3750 along with this program; if not, write to the Free Software
3751 Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
3754 Also add information on how to contact you by electronic and paper mail.
3756 If the program is interactive, make it output a short notice like this
3757 when it starts in an interactive mode:
3760 Gnomovision version 69, Copyright (C) 19@var{yy} @var{name of author}
3761 Gnomovision comes with ABSOLUTELY NO WARRANTY; for details
3762 type `show w'. This is free software, and you are welcome
3763 to redistribute it under certain conditions; type `show c'
3767 The hypothetical commands @samp{show w} and @samp{show c} should show
3768 the appropriate parts of the General Public License. Of course, the
3769 commands you use may be called something other than @samp{show w} and
3770 @samp{show c}; they could even be mouse-clicks or menu items---whatever
3773 You should also get your employer (if you work as a programmer) or your
3774 school, if any, to sign a ``copyright disclaimer'' for the program, if
3775 necessary. Here is a sample; alter the names:
3779 Yoyodyne, Inc., hereby disclaims all copyright
3780 interest in the program `Gnomovision'
3781 (which makes passes at compilers) written
3784 @var{signature of Ty Coon}, 1 April 1989
3785 Ty Coon, President of Vice
3789 This General Public License does not permit incorporating your program into
3790 proprietary programs. If your program is a subroutine library, you may
3791 consider it more useful to permit linking proprietary applications with the
3792 library. If this is what you want to do, use the GNU Library General
3793 Public License instead of this License.
3795 @node GNU Free Documentation License
3796 @section GNU Free Documentation License
3797 @center Version 1.1, March 2000
3800 Copyright (C) 2000 Free Software Foundation, Inc.
3801 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
3803 Everyone is permitted to copy and distribute verbatim copies
3804 of this license document, but changing it is not allowed.
3811 The purpose of this License is to make a manual, textbook, or other
3812 written document ``free'' in the sense of freedom: to assure everyone
3813 the effective freedom to copy and redistribute it, with or without
3814 modifying it, either commercially or noncommercially. Secondarily,
3815 this License preserves for the author and publisher a way to get
3816 credit for their work, while not being considered responsible for
3817 modifications made by others.
3819 This License is a kind of ``copyleft'', which means that derivative
3820 works of the document must themselves be free in the same sense. It
3821 complements the GNU General Public License, which is a copyleft
3822 license designed for free software.
3824 We have designed this License in order to use it for manuals for free
3825 software, because free software needs free documentation: a free
3826 program should come with manuals providing the same freedoms that the
3827 software does. But this License is not limited to software manuals;
3828 it can be used for any textual work, regardless of subject matter or
3829 whether it is published as a printed book. We recommend this License
3830 principally for works whose purpose is instruction or reference.
3834 APPLICABILITY AND DEFINITIONS
3836 This License applies to any manual or other work that contains a
3837 notice placed by the copyright holder saying it can be distributed
3838 under the terms of this License. The ``Document'', below, refers to any
3839 such manual or work. Any member of the public is a licensee, and is
3840 addressed as ``you''.
3842 A ``Modified Version'' of the Document means any work containing the
3843 Document or a portion of it, either copied verbatim, or with
3844 modifications and/or translated into another language.
3846 A ``Secondary Section'' is a named appendix or a front-matter section of
3847 the Document that deals exclusively with the relationship of the
3848 publishers or authors of the Document to the Document's overall subject
3849 (or to related matters) and contains nothing that could fall directly
3850 within that overall subject. (For example, if the Document is in part a
3851 textbook of mathematics, a Secondary Section may not explain any
3852 mathematics.) The relationship could be a matter of historical
3853 connection with the subject or with related matters, or of legal,
3854 commercial, philosophical, ethical or political position regarding
3857 The ``Invariant Sections'' are certain Secondary Sections whose titles
3858 are designated, as being those of Invariant Sections, in the notice
3859 that says that the Document is released under this License.
3861 The ``Cover Texts'' are certain short passages of text that are listed,
3862 as Front-Cover Texts or Back-Cover Texts, in the notice that says that
3863 the Document is released under this License.
3865 A ``Transparent'' copy of the Document means a machine-readable copy,
3866 represented in a format whose specification is available to the
3867 general public, whose contents can be viewed and edited directly and
3868 straightforwardly with generic text editors or (for images composed of
3869 pixels) generic paint programs or (for drawings) some widely available
3870 drawing editor, and that is suitable for input to text formatters or
3871 for automatic translation to a variety of formats suitable for input
3872 to text formatters. A copy made in an otherwise Transparent file
3873 format whose markup has been designed to thwart or discourage
3874 subsequent modification by readers is not Transparent. A copy that is
3875 not ``Transparent'' is called ``Opaque''.
3877 Examples of suitable formats for Transparent copies include plain
3878 @sc{ascii} without markup, Texinfo input format, LaTeX input format, @sc{sgml}
3879 or @sc{xml} using a publicly available @sc{dtd}, and standard-conforming simple
3880 @sc{html} designed for human modification. Opaque formats include
3881 PostScript, @sc{pdf}, proprietary formats that can be read and edited only
3882 by proprietary word processors, @sc{sgml} or @sc{xml} for which the @sc{dtd} and/or
3883 processing tools are not generally available, and the
3884 machine-generated @sc{html} produced by some word processors for output
3887 The ``Title Page'' means, for a printed book, the title page itself,
3888 plus such following pages as are needed to hold, legibly, the material
3889 this License requires to appear in the title page. For works in
3890 formats which do not have any title page as such, ``Title Page'' means
3891 the text near the most prominent appearance of the work's title,
3892 preceding the beginning of the body of the text.
3897 You may copy and distribute the Document in any medium, either
3898 commercially or noncommercially, provided that this License, the
3899 copyright notices, and the license notice saying this License applies
3900 to the Document are reproduced in all copies, and that you add no other
3901 conditions whatsoever to those of this License. You may not use
3902 technical measures to obstruct or control the reading or further
3903 copying of the copies you make or distribute. However, you may accept
3904 compensation in exchange for copies. If you distribute a large enough
3905 number of copies you must also follow the conditions in section 3.
3907 You may also lend copies, under the same conditions stated above, and
3908 you may publicly display copies.
3913 If you publish printed copies of the Document numbering more than 100,
3914 and the Document's license notice requires Cover Texts, you must enclose
3915 the copies in covers that carry, clearly and legibly, all these Cover
3916 Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
3917 the back cover. Both covers must also clearly and legibly identify
3918 you as the publisher of these copies. The front cover must present
3919 the full title with all words of the title equally prominent and
3920 visible. You may add other material on the covers in addition.
3921 Copying with changes limited to the covers, as long as they preserve
3922 the title of the Document and satisfy these conditions, can be treated
3923 as verbatim copying in other respects.
3925 If the required texts for either cover are too voluminous to fit
3926 legibly, you should put the first ones listed (as many as fit
3927 reasonably) on the actual cover, and continue the rest onto adjacent
3930 If you publish or distribute Opaque copies of the Document numbering
3931 more than 100, you must either include a machine-readable Transparent
3932 copy along with each Opaque copy, or state in or with each Opaque copy
3933 a publicly-accessible computer-network location containing a complete
3934 Transparent copy of the Document, free of added material, which the
3935 general network-using public has access to download anonymously at no
3936 charge using public-standard network protocols. If you use the latter
3937 option, you must take reasonably prudent steps, when you begin
3938 distribution of Opaque copies in quantity, to ensure that this
3939 Transparent copy will remain thus accessible at the stated location
3940 until at least one year after the last time you distribute an Opaque
3941 copy (directly or through your agents or retailers) of that edition to
3944 It is requested, but not required, that you contact the authors of the
3945 Document well before redistributing any large number of copies, to give
3946 them a chance to provide you with an updated version of the Document.
3951 You may copy and distribute a Modified Version of the Document under
3952 the conditions of sections 2 and 3 above, provided that you release
3953 the Modified Version under precisely this License, with the Modified
3954 Version filling the role of the Document, thus licensing distribution
3955 and modification of the Modified Version to whoever possesses a copy
3956 of it. In addition, you must do these things in the Modified Version:
3958 A. Use in the Title Page (and on the covers, if any) a title distinct
3959 from that of the Document, and from those of previous versions
3960 (which should, if there were any, be listed in the History section
3961 of the Document). You may use the same title as a previous version
3962 if the original publisher of that version gives permission.@*
3963 B. List on the Title Page, as authors, one or more persons or entities
3964 responsible for authorship of the modifications in the Modified
3965 Version, together with at least five of the principal authors of the
3966 Document (all of its principal authors, if it has less than five).@*
3967 C. State on the Title page the name of the publisher of the
3968 Modified Version, as the publisher.@*
3969 D. Preserve all the copyright notices of the Document.@*
3970 E. Add an appropriate copyright notice for your modifications
3971 adjacent to the other copyright notices.@*
3972 F. Include, immediately after the copyright notices, a license notice
3973 giving the public permission to use the Modified Version under the
3974 terms of this License, in the form shown in the Addendum below.@*
3975 G. Preserve in that license notice the full lists of Invariant Sections
3976 and required Cover Texts given in the Document's license notice.@*
3977 H. Include an unaltered copy of this License.@*
3978 I. Preserve the section entitled ``History'', and its title, and add to
3979 it an item stating at least the title, year, new authors, and
3980 publisher of the Modified Version as given on the Title Page. If
3981 there is no section entitled ``History'' in the Document, create one
3982 stating the title, year, authors, and publisher of the Document as
3983 given on its Title Page, then add an item describing the Modified
3984 Version as stated in the previous sentence.@*
3985 J. Preserve the network location, if any, given in the Document for
3986 public access to a Transparent copy of the Document, and likewise
3987 the network locations given in the Document for previous versions
3988 it was based on. These may be placed in the ``History'' section.
3989 You may omit a network location for a work that was published at
3990 least four years before the Document itself, or if the original
3991 publisher of the version it refers to gives permission.@*
3992 K. In any section entitled ``Acknowledgements'' or ``Dedications'',
3993 preserve the section's title, and preserve in the section all the
3994 substance and tone of each of the contributor acknowledgements
3995 and/or dedications given therein.@*
3996 L. Preserve all the Invariant Sections of the Document,
3997 unaltered in their text and in their titles. Section numbers
3998 or the equivalent are not considered part of the section titles.@*
3999 M. Delete any section entitled ``Endorsements''. Such a section
4000 may not be included in the Modified Version.@*
4001 N. Do not retitle any existing section as ``Endorsements''
4002 or to conflict in title with any Invariant Section.@*
4004 If the Modified Version includes new front-matter sections or
4005 appendices that qualify as Secondary Sections and contain no material
4006 copied from the Document, you may at your option designate some or all
4007 of these sections as invariant. To do this, add their titles to the
4008 list of Invariant Sections in the Modified Version's license notice.
4009 These titles must be distinct from any other section titles.
4011 You may add a section entitled ``Endorsements'', provided it contains
4012 nothing but endorsements of your Modified Version by various
4013 parties--for example, statements of peer review or that the text has
4014 been approved by an organization as the authoritative definition of a
4017 You may add a passage of up to five words as a Front-Cover Text, and a
4018 passage of up to 25 words as a Back-Cover Text, to the end of the list
4019 of Cover Texts in the Modified Version. Only one passage of
4020 Front-Cover Text and one of Back-Cover Text may be added by (or
4021 through arrangements made by) any one entity. If the Document already
4022 includes a cover text for the same cover, previously added by you or
4023 by arrangement made by the same entity you are acting on behalf of,
4024 you may not add another; but you may replace the old one, on explicit
4025 permission from the previous publisher that added the old one.
4027 The author(s) and publisher(s) of the Document do not by this License
4028 give permission to use their names for publicity for or to assert or
4029 imply endorsement of any Modified Version.
4034 You may combine the Document with other documents released under this
4035 License, under the terms defined in section 4 above for modified
4036 versions, provided that you include in the combination all of the
4037 Invariant Sections of all of the original documents, unmodified, and
4038 list them all as Invariant Sections of your combined work in its
4041 The combined work need only contain one copy of this License, and
4042 multiple identical Invariant Sections may be replaced with a single
4043 copy. If there are multiple Invariant Sections with the same name but
4044 different contents, make the title of each such section unique by
4045 adding at the end of it, in parentheses, the name of the original
4046 author or publisher of that section if known, or else a unique number.
4047 Make the same adjustment to the section titles in the list of
4048 Invariant Sections in the license notice of the combined work.
4050 In the combination, you must combine any sections entitled ``History''
4051 in the various original documents, forming one section entitled
4052 ``History''; likewise combine any sections entitled ``Acknowledgements'',
4053 and any sections entitled ``Dedications''. You must delete all sections
4054 entitled ``Endorsements.''
4057 COLLECTIONS OF DOCUMENTS
4059 You may make a collection consisting of the Document and other documents
4060 released under this License, and replace the individual copies of this
4061 License in the various documents with a single copy that is included in
4062 the collection, provided that you follow the rules of this License for
4063 verbatim copying of each of the documents in all other respects.
4065 You may extract a single document from such a collection, and distribute
4066 it individually under this License, provided you insert a copy of this
4067 License into the extracted document, and follow this License in all
4068 other respects regarding verbatim copying of that document.
4071 AGGREGATION WITH INDEPENDENT WORKS
4073 A compilation of the Document or its derivatives with other separate
4074 and independent documents or works, in or on a volume of a storage or
4075 distribution medium, does not as a whole count as a Modified Version
4076 of the Document, provided no compilation copyright is claimed for the
4077 compilation. Such a compilation is called an ``aggregate'', and this
4078 License does not apply to the other self-contained works thus compiled
4079 with the Document, on account of their being thus compiled, if they
4080 are not themselves derivative works of the Document.
4082 If the Cover Text requirement of section 3 is applicable to these
4083 copies of the Document, then if the Document is less than one quarter
4084 of the entire aggregate, the Document's Cover Texts may be placed on
4085 covers that surround only the Document within the aggregate.
4086 Otherwise they must appear on covers around the whole aggregate.
4091 Translation is considered a kind of modification, so you may
4092 distribute translations of the Document under the terms of section 4.
4093 Replacing Invariant Sections with translations requires special
4094 permission from their copyright holders, but you may include
4095 translations of some or all Invariant Sections in addition to the
4096 original versions of these Invariant Sections. You may include a
4097 translation of this License provided that you also include the
4098 original English version of this License. In case of a disagreement
4099 between the translation and the original English version of this
4100 License, the original English version will prevail.
4105 You may not copy, modify, sublicense, or distribute the Document except
4106 as expressly provided for under this License. Any other attempt to
4107 copy, modify, sublicense or distribute the Document is void, and will
4108 automatically terminate your rights under this License. However,
4109 parties who have received copies, or rights, from you under this
4110 License will not have their licenses terminated so long as such
4111 parties remain in full compliance.
4114 FUTURE REVISIONS OF THIS LICENSE
4116 The Free Software Foundation may publish new, revised versions
4117 of the GNU Free Documentation License from time to time. Such new
4118 versions will be similar in spirit to the present version, but may
4119 differ in detail to address new problems or concerns. See
4120 http://www.gnu.org/copyleft/.
4122 Each version of the License is given a distinguishing version number.
4123 If the Document specifies that a particular numbered version of this
4124 License ``or any later version'' applies to it, you have the option of
4125 following the terms and conditions either of that specified version or
4126 of any later version that has been published (not as a draft) by the
4127 Free Software Foundation. If the Document does not specify a version
4128 number of this License, you may choose any version ever published (not
4129 as a draft) by the Free Software Foundation.
4133 @unnumberedsec ADDENDUM: How to use this License for your documents
4135 To use this License in a document you have written, include a copy of
4136 the License in the document and put the following copyright and
4137 license notices just after the title page:
4142 Copyright (C) @var{year} @var{your name}.
4143 Permission is granted to copy, distribute and/or modify this document
4144 under the terms of the GNU Free Documentation License, Version 1.1
4145 or any later version published by the Free Software Foundation;
4146 with the Invariant Sections being @var{list their titles}, with the
4147 Front-Cover Texts being @var{list}, and with the Back-Cover Texts being @var{list}.
4148 A copy of the license is included in the section entitled ``GNU
4149 Free Documentation License''.
4152 If you have no Invariant Sections, write ``with no Invariant Sections''
4153 instead of saying which ones are invariant. If you have no
4154 Front-Cover Texts, write ``no Front-Cover Texts'' instead of
4155 ``Front-Cover Texts being @var{list}''; likewise for Back-Cover Texts.
4157 If your document contains nontrivial examples of program code, we
4158 recommend releasing these examples in parallel under your choice of
4159 free software license, such as the GNU General Public License,
4160 to permit their use in free software.
4164 @unnumbered Concept Index