1 This is Info file wget.info, produced by Makeinfo version 1.68 from the
2 input file ./wget.texi.
4 INFO-DIR-SECTION Net Utilities
5 INFO-DIR-SECTION World Wide Web
7 * Wget: (wget). The non-interactive network downloader.
10 This file documents the the GNU Wget utility for downloading network
13 Copyright (C) 1996, 1997, 1998, 2000 Free Software Foundation, Inc.
15 Permission is granted to make and distribute verbatim copies of this
16 manual provided the copyright notice and this permission notice are
17 preserved on all copies.
19 Permission is granted to copy and distribute modified versions of
20 this manual under the conditions for verbatim copying, provided also
21 that the sections entitled "Copying" and "GNU General Public License"
22 are included exactly as in the original, and provided that the entire
23 resulting derived work is distributed under the terms of a permission
24 notice identical to this one.
27 File: wget.info, Node: Top, Next: Overview, Prev: (dir), Up: (dir)
32 This manual documents version 1.5.3+dev of GNU Wget, the freely
33 available utility for network download.
35 Copyright (C) 1996, 1997, 1998 Free Software Foundation, Inc.
39 * Overview:: Features of Wget.
40 * Invoking:: Wget command-line arguments.
41 * Recursive Retrieval:: Description of recursive retrieval.
42 * Following Links:: The available methods of chasing links.
43 * Time-Stamping:: Mirroring according to time-stamps.
44 * Startup File:: Wget's initialization file.
45 * Examples:: Examples of usage.
46 * Various:: The stuff that doesn't fit anywhere else.
47 * Appendices:: Some useful references.
48 * Copying:: You may give out copies of Wget.
49 * Concept Index:: Topics covered by this manual.
52 File: wget.info, Node: Overview, Next: Invoking, Prev: Top, Up: Top
57 GNU Wget is a freely available network utility to retrieve files from
58 the World Wide Web, using HTTP (Hyper Text Transfer Protocol) and FTP
59 (File Transfer Protocol), the two most widely used Internet protocols.
60 It has many useful features to make downloading easier, some of them
63 * Wget is non-interactive, meaning that it can work in the
64 background, while the user is not logged on. This allows you to
65 start a retrieval and disconnect from the system, letting Wget
66 finish the work. By contrast, most of the Web browsers require
67 constant user's presence, which can be a great hindrance when
68 transferring a lot of data.
70 * Wget is capable of descending recursively through the structure of
71 HTML documents and FTP directory trees, making a local copy of the
72 directory hierarchy similar to the one on the remote server. This
73 feature can be used to mirror archives and home pages, or traverse
74 the web in search of data, like a WWW robot (*Note Robots::). In
75 that spirit, Wget understands the `norobots' convention.
77 * File name wildcard matching and recursive mirroring of directories
78 are available when retrieving via FTP. Wget can read the
79 time-stamp information given by both HTTP and FTP servers, and
80 store it locally. Thus Wget can see if the remote file has
81 changed since last retrieval, and automatically retrieve the new
82 version if it has. This makes Wget suitable for mirroring of FTP
83 sites, as well as home pages.
85 * Wget works exceedingly well on slow or unstable connections,
86 retrying the document until it is fully retrieved, or until a
87 user-specified retry count is surpassed. It will try to resume the
88 download from the point of interruption, using `REST' with FTP and
89 `Range' with HTTP servers that support them.
91 * By default, Wget supports proxy servers, which can lighten the
92 network load, speed up retrieval and provide access behind
93 firewalls. However, if you are behind a firewall that requires
94 that you use a socks style gateway, you can get the socks library
95 and build wget with support for socks. Wget also supports the
96 passive FTP downloading as an option.
98 * Builtin features offer mechanisms to tune which links you wish to
99 follow (*Note Following Links::).
101 * The retrieval is conveniently traced with printing dots, each dot
102 representing a fixed amount of data received (1KB by default).
103 These representations can be customized to your preferences.
105 * Most of the features are fully configurable, either through
106 command line options, or via the initialization file `.wgetrc'
107 (*Note Startup File::). Wget allows you to define "global"
108 startup files (`/usr/local/etc/wgetrc' by default) for site
111 * Finally, GNU Wget is free software. This means that everyone may
112 use it, redistribute it and/or modify it under the terms of the
113 GNU General Public License, as published by the Free Software
114 Foundation (*Note Copying::).
117 File: wget.info, Node: Invoking, Next: Recursive Retrieval, Prev: Overview, Up: Top
122 By default, Wget is very simple to invoke. The basic syntax is:
124 wget [OPTION]... [URL]...
126 Wget will simply download all the URLs specified on the command
127 line. URL is a "Uniform Resource Locator", as defined below.
129 However, you may wish to change some of the default parameters of
130 Wget. You can do it two ways: permanently, adding the appropriate
131 command to `.wgetrc' (*Note Startup File::), or specifying it on the
138 * Basic Startup Options::
139 * Logging and Input File Options::
141 * Directory Options::
144 * Recursive Retrieval Options::
145 * Recursive Accept/Reject Options::
148 File: wget.info, Node: URL Format, Next: Option Syntax, Prev: Invoking, Up: Invoking
153 "URL" is an acronym for Uniform Resource Locator. A uniform
154 resource locator is a compact string representation for a resource
155 available via the Internet. Wget recognizes the URL syntax as per
156 RFC1738. This is the most widely used form (square brackets denote
159 http://host[:port]/directory/file
160 ftp://host[:port]/directory/file
162 You can also encode your username and password within a URL:
164 ftp://user:password@host/path
165 http://user:password@host/path
167 Either USER or PASSWORD, or both, may be left out. If you leave out
168 either the HTTP username or password, no authentication will be sent.
169 If you leave out the FTP username, `anonymous' will be used. If you
170 leave out the FTP password, your email address will be supplied as a
173 You can encode unsafe characters in a URL as `%xy', `xy' being the
174 hexadecimal representation of the character's ASCII value. Some common
175 unsafe characters include `%' (quoted as `%25'), `:' (quoted as `%3A'),
176 and `@' (quoted as `%40'). Refer to RFC1738 for a comprehensive list
177 of unsafe characters.
179 Wget also supports the `type' feature for FTP URLs. By default, FTP
180 documents are retrieved in the binary mode (type `i'), which means that
181 they are downloaded unchanged. Another useful mode is the `a'
182 ("ASCII") mode, which converts the line delimiters between the
183 different operating systems, and is thus useful for text files. Here
186 ftp://host/directory/file;type=a
188 Two alternative variants of URL specification are also supported,
189 because of historical (hysterical?) reasons and their widespreaded use.
191 FTP-only syntax (supported by `NcFTP'):
194 HTTP-only syntax (introduced by `Netscape'):
197 These two alternative forms are deprecated, and may cease being
198 supported in the future.
200 If you do not understand the difference between these notations, or
201 do not know which one to use, just use the plain ordinary format you use
202 with your favorite browser, like `Lynx' or `Netscape'.
204 ---------- Footnotes ----------
206 (1) If you have a `.netrc' file in your home directory, password
207 will also be searched for there.
210 File: wget.info, Node: Option Syntax, Next: Basic Startup Options, Prev: URL Format, Up: Invoking
215 Since Wget uses GNU getopts to process its arguments, every option
216 has a short form and a long form. Long options are more convenient to
217 remember, but take time to type. You may freely mix different option
218 styles, or specify options after the command-line arguments. Thus you
221 wget -r --tries=10 http://fly.cc.fer.hr/ -o log
223 The space between the option accepting an argument and the argument
224 may be omitted. Instead `-o log' you can write `-olog'.
226 You may put several options that do not require arguments together,
231 This is a complete equivalent of:
235 Since the options can be specified after the arguments, you may
236 terminate them with `--'. So the following will try to download URL
237 `-x', reporting failure to `log':
241 The options that accept comma-separated lists all respect the
242 convention that specifying an empty list clears its value. This can be
243 useful to clear the `.wgetrc' settings. For instance, if your `.wgetrc'
244 sets `exclude_directories' to `/cgi-bin', the following example will
245 first reset it, and then set it to exclude `/~nobody' and `/~somebody'.
246 You can also clear the lists in `.wgetrc' (*Note Wgetrc Syntax::).
248 wget -X '' -X /~nobody,/~somebody
251 File: wget.info, Node: Basic Startup Options, Next: Logging and Input File Options, Prev: Option Syntax, Up: Invoking
253 Basic Startup Options
254 =====================
258 Display the version of Wget.
262 Print a help message describing all of Wget's command-line options.
266 Go to background immediately after startup. If no output file is
267 specified via the `-o', output is redirected to `wget-log'.
271 Execute COMMAND as if it were a part of `.wgetrc' (*Note Startup
272 File::). A command thus invoked will be executed *after* the
273 commands in `.wgetrc', thus taking precedence over them.
276 File: wget.info, Node: Logging and Input File Options, Next: Download Options, Prev: Basic Startup Options, Up: Invoking
278 Logging and Input File Options
279 ==============================
282 `--output-file=LOGFILE'
283 Log all messages to LOGFILE. The messages are normally reported
287 `--append-output=LOGFILE'
288 Append to LOGFILE. This is the same as `-o', only it appends to
289 LOGFILE instead of overwriting the old log file. If LOGFILE does
290 not exist, a new file is created.
294 Turn on debug output, meaning various information important to the
295 developers of Wget if it does not work properly. Your system
296 administrator may have chosen to compile Wget without debug
297 support, in which case `-d' will not work. Please note that
298 compiling with debug support is always safe--Wget compiled with
299 the debug support will *not* print any debug info unless requested
300 with `-d'. *Note Reporting Bugs:: for more information on how to
301 use `-d' for sending bug reports.
305 Turn off Wget's output.
309 Turn on verbose output, with all the available data. The default
314 Non-verbose output--turn off verbose without being completely quiet
315 (use `-q' for that), which means that error messages and basic
316 information still get printed.
320 Read URLs from FILE, in which case no URLs need to be on the
321 command line. If there are URLs both on the command line and in
322 an input file, those on the command lines will be the first ones to
323 be retrieved. The FILE need not be an HTML document (but no harm
324 if it is)--it is enough if the URLs are just listed sequentially.
326 However, if you specify `--force-html', the document will be
327 regarded as `html'. In that case you may have problems with
328 relative links, which you can solve either by adding `<base
329 href="URL">' to the documents or by specifying `--base=URL' on the
334 When input is read from a file, force it to be treated as an HTML
335 file. This enables you to retrieve relative links from existing
336 HTML files on your local disk, by adding `<base href="URL">' to
337 HTML, or using the `--base' command-line option.
341 When used in conjunction with `-F', prepends URL to relative links
342 in the file specified by `-i'.
345 File: wget.info, Node: Download Options, Next: Directory Options, Prev: Logging and Input File Options, Up: Invoking
350 `--bind-address=ADDRESS'
351 When making client TCP/IP connections, `bind()' to ADDRESS on the
352 local machine. ADDRESS may be specified as a hostname or IP
353 address. This option can be useful if your machine is bound to
358 Set number of retries to NUMBER. Specify 0 or `inf' for infinite
362 `--output-document=FILE'
363 The documents will not be written to the appropriate files, but
364 all will be concatenated together and written to FILE. If FILE
365 already exists, it will be overwritten. If the FILE is `-', the
366 documents will be written to standard output. Including this
367 option automatically sets the number of tries to 1.
371 If a file is downloaded more than once in the same directory,
372 wget's behavior depends on a few options, including `-nc'. In
373 certain cases, the local file will be "clobbered", or overwritten,
374 upon repeated download. In other cases it will be preserved.
376 When running wget without `-N', `-nc', or `-r', downloading the
377 same file in the same directory will result in the original copy
378 of `FILE' being preserved and the second copy being named
379 `FILE.1'. If that file is downloaded yet again, the third copy
380 will be named `FILE.2', and so on. When `-nc' is specified, this
381 behavior is suppressed, and wget will refuse to download newer
382 copies of `FILE'. Therefore, "no-clobber" is actually a misnomer
383 in this mode - it's not clobbering that's prevented (as the
384 numeric suffixes were already preventing clobbering), but rather
385 the multiple version saving that's prevented.
387 When running wget with `-r', but without `-N' or `-nc',
388 re-downloading a file will result in the new copy simply
389 overwriting the old. Adding `-nc' will prevent this behavior,
390 instead causing the original version to be preserved and any newer
391 copies on the server to be ignored.
393 When running wget with `-N', with or without `-r', the decision as
394 to whether or not to download a newer copy of a file depends on
395 the local and remote timestamp and size of the file (*Note
396 Time-Stamping::). `-nc' may not be specified at the same time as
399 Note that when `-nc' is specified, files with the suffixes `.html'
400 or (yuck) `.htm' will be loaded from the local disk and parsed as
401 if they had been retrieved from the Web.
405 Continue getting an existing file. This is useful when you want to
406 finish up the download started by another program, or a previous
407 instance of Wget. Thus you can write:
409 wget -c ftp://sunsite.doc.ic.ac.uk/ls-lR.Z
411 If there is a file name `ls-lR.Z' in the current directory, Wget
412 will assume that it is the first portion of the remote file, and
413 will require the server to continue the retrieval from an offset
414 equal to the length of the local file.
416 Note that you need not specify this option if all you want is Wget
417 to continue retrieving where it left off when the connection is
418 lost--Wget does this by default. You need this option only when
419 you want to continue retrieval of a file already halfway
420 retrieved, saved by another FTP client, or left by Wget being
423 Without `-c', the previous example would just begin to download the
424 remote file to `ls-lR.Z.1'. The `-c' option is also applicable
425 for HTTP servers that support the `Range' header.
427 Note that if you use `-c' on a file that's already downloaded
428 completely, `FILE' will not be changed, nor will a second `FILE.1'
432 Set the retrieval style to STYLE. Wget traces the retrieval of
433 each document by printing dots on the screen, each dot
434 representing a fixed amount of retrieved data. Any number of dots
435 may be separated in a "cluster", to make counting easier. This
436 option allows you to choose one of the pre-defined styles,
437 determining the number of bytes represented by a dot, the number
438 of dots in a cluster, and the number of dots on the line.
440 With the `default' style each dot represents 1K, there are ten dots
441 in a cluster and 50 dots in a line. The `binary' style has a more
442 "computer"-like orientation--8K dots, 16-dots clusters and 48 dots
443 per line (which makes for 384K lines). The `mega' style is
444 suitable for downloading very large files--each dot represents 64K
445 retrieved, there are eight dots in a cluster, and 48 dots on each
446 line (so each line contains 3M). The `micro' style is exactly the
447 reverse; it is suitable for downloading small files, with 128-byte
448 dots, 8 dots per cluster, and 48 dots (6K) per line.
452 Turn on time-stamping. *Note Time-Stamping:: for details.
456 Print the headers sent by HTTP servers and responses sent by FTP
460 When invoked with this option, Wget will behave as a Web "spider",
461 which means that it will not download the pages, just check that
462 they are there. You can use it to check your bookmarks, e.g. with:
464 wget --spider --force-html -i bookmarks.html
466 This feature needs much more work for Wget to get close to the
467 functionality of real WWW spiders.
471 Set the read timeout to SECONDS seconds. Whenever a network read
472 is issued, the file descriptor is checked for a timeout, which
473 could otherwise leave a pending connection (uninterrupted read).
474 The default timeout is 900 seconds (fifteen minutes). Setting
475 timeout to 0 will disable checking for timeouts.
477 Please do not lower the default timeout value with this option
478 unless you know what you are doing.
482 Wait the specified number of seconds between the retrievals. Use
483 of this option is recommended, as it lightens the server load by
484 making the requests less frequent. Instead of in seconds, the
485 time can be specified in minutes using the `m' suffix, in hours
486 using `h' suffix, or in days using `d' suffix.
488 Specifying a large value for this option is useful if the network
489 or the destination host is down, so that Wget can wait long enough
490 to reasonably expect the network error to be fixed before the
493 `--waitretry=SECONDS'
494 If you don't want Wget to wait between *every* retrieval, but only
495 between retries of failed downloads, you can use this option.
496 Wget will use "linear backoff", waiting 1 second after the first
497 failure on a given file, then waiting 2 seconds after the second
498 failure on that file, up to the maximum number of SECONDS you
499 specify. Therefore, a value of 10 will actually make Wget wait up
500 to (1 + 2 + ... + 10) = 55 seconds per file.
502 Note that this option is turned on by default in the global
507 Turn proxy support on or off. The proxy is on by default if the
508 appropriate environmental variable is defined.
512 Specify download quota for automatic retrievals. The value can be
513 specified in bytes (default), kilobytes (with `k' suffix), or
514 megabytes (with `m' suffix).
516 Note that quota will never affect downloading a single file. So
517 if you specify `wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz',
518 all of the `ls-lR.gz' will be downloaded. The same goes even when
519 several URLs are specified on the command-line. However, quota is
520 respected when retrieving either recursively, or from an input
521 file. Thus you may safely type `wget -Q2m -i sites'--download
522 will be aborted when the quota is exceeded.
524 Setting quota to 0 or to `inf' unlimits the download quota.
527 File: wget.info, Node: Directory Options, Next: HTTP Options, Prev: Download Options, Up: Invoking
534 Do not create a hierarchy of directories when retrieving
535 recursively. With this option turned on, all files will get saved
536 to the current directory, without clobbering (if a name shows up
537 more than once, the filenames will get extensions `.n').
540 `--force-directories'
541 The opposite of `-nd'--create a hierarchy of directories, even if
542 one would not have been created otherwise. E.g. `wget -x
543 http://fly.cc.fer.hr/robots.txt' will save the downloaded file to
544 `fly.cc.fer.hr/robots.txt'.
547 `--no-host-directories'
548 Disable generation of host-prefixed directories. By default,
549 invoking Wget with `-r http://fly.cc.fer.hr/' will create a
550 structure of directories beginning with `fly.cc.fer.hr/'. This
551 option disables such behavior.
554 Ignore NUMBER directory components. This is useful for getting a
555 fine-grained control over the directory where recursive retrieval
558 Take, for example, the directory at
559 `ftp://ftp.xemacs.org/pub/xemacs/'. If you retrieve it with `-r',
560 it will be saved locally under `ftp.xemacs.org/pub/xemacs/'.
561 While the `-nH' option can remove the `ftp.xemacs.org/' part, you
562 are still stuck with `pub/xemacs'. This is where `--cut-dirs'
563 comes in handy; it makes Wget not "see" NUMBER remote directory
564 components. Here are several examples of how `--cut-dirs' option
567 No options -> ftp.xemacs.org/pub/xemacs/
569 -nH --cut-dirs=1 -> xemacs/
570 -nH --cut-dirs=2 -> .
572 --cut-dirs=1 -> ftp.xemacs.org/xemacs/
575 If you just want to get rid of the directory structure, this
576 option is similar to a combination of `-nd' and `-P'. However,
577 unlike `-nd', `--cut-dirs' does not lose with subdirectories--for
578 instance, with `-nH --cut-dirs=1', a `beta/' subdirectory will be
579 placed to `xemacs/beta', as one would expect.
582 `--directory-prefix=PREFIX'
583 Set directory prefix to PREFIX. The "directory prefix" is the
584 directory where all other files and subdirectories will be saved
585 to, i.e. the top of the retrieval tree. The default is `.' (the
589 File: wget.info, Node: HTTP Options, Next: FTP Options, Prev: Directory Options, Up: Invoking
596 If a file of type `text/html' is downloaded and the URL does not
597 end with the regexp "\.[Hh][Tt][Mm][Ll]?", this option will cause
598 the suffix `.html' to be appended to the local filename. This is
599 useful, for instance, when you're mirroring a remote site that uses
600 `.asp' pages, but you want the mirrored pages to be viewable on
601 your stock Apache server. Another good use for this is when you're
602 downloading the output of CGIs. A URL like
603 `http://site.com/article.cgi?25' will be saved as
604 `article.cgi?25.html'.
606 Note that filenames changed in this way will be re-downloaded
607 every time you re-mirror a site, because wget can't tell that the
608 local `X.html' file corresponds to remote URL `X' (since it
609 doesn't yet know that the URL produces output of type `text/html'.
610 To prevent this re-downloading, you must use `-k' and `-K' so
611 that the original version of the file will be saved as `X.orig'
612 (*Note Recursive Retrieval Options::).
615 `--http-passwd=PASSWORD'
616 Specify the username USER and password PASSWORD on an HTTP server.
617 According to the type of the challenge, Wget will encode them
618 using either the `basic' (insecure) or the `digest' authentication
621 Another way to specify username and password is in the URL itself
622 (*Note URL Format::). For more information about security issues
623 with Wget, *Note Security Considerations::.
627 When set to off, disable server-side cache. In this case, Wget
628 will send the remote server an appropriate directive (`Pragma:
629 no-cache') to get the file from the remote service, rather than
630 returning the cached version. This is especially useful for
631 retrieving and flushing out-of-date documents on proxy servers.
633 Caching is allowed by default.
636 Unfortunately, some HTTP servers (CGI programs, to be more
637 precise) send out bogus `Content-Length' headers, which makes Wget
638 go wild, as it thinks not all the document was retrieved. You can
639 spot this syndrome if Wget retries getting the same document again
640 and again, each time claiming that the (otherwise normal)
641 connection has closed on the very same byte.
643 With this option, Wget will ignore the `Content-Length' header--as
646 `--header=ADDITIONAL-HEADER'
647 Define an ADDITIONAL-HEADER to be passed to the HTTP servers.
648 Headers must contain a `:' preceded by one or more non-blank
649 characters, and must not contain newlines.
651 You may define more than one additional header by specifying
652 `--header' more than once.
654 wget --header='Accept-Charset: iso-8859-2' \
655 --header='Accept-Language: hr' \
656 http://fly.cc.fer.hr/
658 Specification of an empty string as the header value will clear all
659 previous user-defined headers.
662 `--proxy-passwd=PASSWORD'
663 Specify the username USER and password PASSWORD for authentication
664 on a proxy server. Wget will encode them using the `basic'
665 authentication scheme.
668 Include `Referer: URL' header in HTTP request. Useful for
669 retrieving documents with server-side processing that assume they
670 are always being retrieved by interactive web browsers and only
671 come out properly when Referer is set to one of the pages that
676 Save the headers sent by the HTTP server to the file, preceding the
677 actual contents, with an empty line as the separator.
680 `--user-agent=AGENT-STRING'
681 Identify as AGENT-STRING to the HTTP server.
683 The HTTP protocol allows the clients to identify themselves using a
684 `User-Agent' header field. This enables distinguishing the WWW
685 software, usually for statistical purposes or for tracing of
686 protocol violations. Wget normally identifies as `Wget/VERSION',
687 VERSION being the current version number of Wget.
689 However, some sites have been known to impose the policy of
690 tailoring the output according to the `User-Agent'-supplied
691 information. While conceptually this is not such a bad idea, it
692 has been abused by servers denying information to clients other
693 than `Mozilla' or Microsoft `Internet Explorer'. This option
694 allows you to change the `User-Agent' line issued by Wget. Use of
695 this option is discouraged, unless you really know what you are
699 File: wget.info, Node: FTP Options, Next: Recursive Retrieval Options, Prev: HTTP Options, Up: Invoking
705 Usually, when retrieving FTP directories recursively and a symbolic
706 link is encountered, the linked-to file is not downloaded.
707 Instead, a matching symbolic link is created on the local
708 filesystem. The pointed-to file will not be downloaded unless
709 this recursive retrieval would have encountered it separately and
710 downloaded it anyway.
712 When `--retr-symlinks' is specified, however, symbolic links are
713 traversed and the pointed-to files are retrieved. At this time,
714 this option does not cause wget to traverse symlinks to
715 directories and recurse through them, but in the future it should
716 be enhanced to do this.
718 Note that when retrieving a file (not a directory) because it was
719 specified on the commandline, rather than because it was recursed
720 to, this option has no effect. Symbolic links are always
721 traversed in this case.
725 Turn FTP globbing on or off. Globbing means you may use the
726 shell-like special characters ("wildcards"), like `*', `?', `['
727 and `]' to retrieve more than one file from the same directory at
730 wget ftp://gnjilux.cc.fer.hr/*.msg
732 By default, globbing will be turned on if the URL contains a
733 globbing character. This option may be used to turn globbing on
736 You may have to quote the URL to protect it from being expanded by
737 your shell. Globbing makes Wget look for a directory listing,
738 which is system-specific. This is why it currently works only
739 with Unix FTP servers (and the ones emulating Unix `ls' output).
742 Use the "passive" FTP retrieval scheme, in which the client
743 initiates the data connection. This is sometimes required for FTP
744 to work behind firewalls.
747 File: wget.info, Node: Recursive Retrieval Options, Next: Recursive Accept/Reject Options, Prev: FTP Options, Up: Invoking
749 Recursive Retrieval Options
750 ===========================
754 Turn on recursive retrieving. *Note Recursive Retrieval:: for more
759 Specify recursion maximum depth level DEPTH (*Note Recursive
760 Retrieval::). The default maximum depth is 5.
763 This option tells Wget to delete every single file it downloads,
764 *after* having done so. It is useful for pre-fetching popular
765 pages through a proxy, e.g.:
767 wget -r -nd --delete-after http://whatever.com/~popular/page/
769 The `-r' option is to retrieve recursively, and `-nd' to not
772 Note that `--delete-after' deletes files on the local machine. It
773 does not issue the `DELE' command to remote FTP sites, for
774 instance. Also note that when `--delete-after' is specified,
775 `--convert-links' is ignored, so `.orig' files are simply not
776 created in the first place.
780 Convert the non-relative links to relative ones locally. Only the
781 references to the documents actually downloaded will be converted;
782 the rest will be left unchanged.
784 Note that only at the end of the download can Wget know which
785 links have been downloaded. Because of that, much of the work
786 done by `-k' will be performed at the end of the downloads.
790 When converting a file, back up the original version with a `.orig'
791 suffix. Affects the behavior of `-N' (*Note HTTP Time-Stamping
796 Turn on options suitable for mirroring. This option turns on
797 recursion and time-stamping, sets infinite recursion depth and
798 keeps FTP directory listings. It is currently equivalent to `-r
802 `--dont-remove-listing'
803 Don't remove the temporary `.listing' files generated by FTP
804 retrievals. Normally, these files contain the raw directory
805 listings received from FTP servers. Not removing them can be
806 useful to access the full remote file list when running a mirror,
807 or for debugging purposes.
811 This option causes wget to download all the files that are
812 necessary to properly display a given HTML page. This includes
813 such things as inlined images, sounds, and referenced stylesheets.
815 Ordinarily, when downloading a single HTML page, any requisite
816 documents that may be needed to display it properly are not
817 downloaded. Using `-r' together with `-l' can help, but since
818 wget does not ordinarily distinguish between external and inlined
819 documents, one is generally left with "leaf documents" that are
820 missing their requisites.
822 For instance, say document `1.html' contains an `<IMG>' tag
823 referencing `1.gif' and an `<A>' tag pointing to external document
824 `2.html'. Say that `2.html' is the same but that its image is
825 `2.gif' and it links to `3.html'. Say this continues up to some
826 arbitrarily high number.
828 If one executes the command:
830 wget -r -l 2 http://SITE/1.html
832 then `1.html', `1.gif', `2.html', `2.gif', and `3.html' will be
833 downloaded. As you can see, `3.html' is without its requisite
834 `3.gif' because wget is simply counting the number of hops (up to
835 2) away from `1.html' in order to determine where to stop the
836 recursion. However, with this command:
838 wget -r -l 2 -p http://SITE/1.html
840 all the above files *and* `3.html''s requisite `3.gif' will be
841 downloaded. Similarly,
843 wget -r -l 1 -p http://SITE/1.html
845 will cause `1.html', `1.gif', `2.html', and `2.gif' to be
846 downloaded. One might think that:
848 wget -r -l 0 -p http://SITE/1.html
850 would download just `1.html' and `1.gif', but unfortunately this
851 is not the case, because `-l 0' is equivalent to `-l inf' - that
852 is, infinite recursion. To download a single HTML page (or a
853 handful of them, all specified on the commandline or in a `-i' URL
854 input file) and its requisites, simply leave off `-p' and `-l':
856 wget -p http://SITE/1.html
858 Note that wget will behave as if `-r' had been specified, but only
859 that single page and its requisites will be downloaded. Links
860 from that page to external documents will not be followed.
861 Actually, to download a single page and all its requisites (even
862 if they exist on separate websites), and make sure the lot
863 displays properly locally, this author likes to use a few options
866 wget -E -H -k -K -nh -p http://SITE/DOCUMENT
868 To finish off this topic, it's worth knowing that wget's idea of an
869 external document link is any URL specified in an `<A>' tag, an
870 `<AREA>' tag, or a `<LINK>' tag other than `<LINK
874 File: wget.info, Node: Recursive Accept/Reject Options, Prev: Recursive Retrieval Options, Up: Invoking
876 Recursive Accept/Reject Options
877 ===============================
879 `-A ACCLIST --accept ACCLIST'
880 `-R REJLIST --reject REJLIST'
881 Specify comma-separated lists of file name suffixes or patterns to
882 accept or reject (*Note Types of Files:: for more details).
885 `--domains=DOMAIN-LIST'
886 Set domains to be accepted and DNS looked-up, where DOMAIN-LIST is
887 a comma-separated list. Note that it does *not* turn on `-H'.
888 This option speeds things up, even if only one host is spanned
889 (*Note Domain Acceptance::).
891 `--exclude-domains DOMAIN-LIST'
892 Exclude the domains given in a comma-separated DOMAIN-LIST from
893 DNS-lookup (*Note Domain Acceptance::).
896 Follow FTP links from HTML documents. Without this option, Wget
897 will ignore all the FTP links.
900 Wget has an internal table of HTML tag / attribute pairs that it
901 considers when looking for linked documents during a recursive
902 retrieval. If a user wants only a subset of those tags to be
903 considered, however, he or she should be specify such tags in a
904 comma-separated LIST with this option.
908 This is the opposite of the `--follow-tags' option. To skip
909 certain HTML tags when recursively looking for documents to
910 download, specify them in a comma-separated LIST.
912 In the past, the `-G' option was the best bet for downloading a
913 single page and its requisites, using a commandline like:
915 wget -Ga,area -H -k -K -nh -r http://SITE/DOCUMENT
917 However, the author of this option came across a page with tags
918 like `<LINK REL="home" HREF="/">' and came to the realization that
919 `-G' was not enough. One can't just tell wget to ignore `<LINK>',
920 because then stylesheets will not be downloaded. Now the best bet
921 for downloading a single page and its requisites is the dedicated
922 `--page-requisites' option.
926 Enable spanning across hosts when doing recursive retrieving
931 Follow relative links only. Useful for retrieving a specific home
932 page without any distractions, not even those from the same hosts
933 (*Note Relative Links::).
936 `--include-directories=LIST'
937 Specify a comma-separated list of directories you wish to follow
938 when downloading (*Note Directory-Based Limits:: for more
939 details.) Elements of LIST may contain wildcards.
942 `--exclude-directories=LIST'
943 Specify a comma-separated list of directories you wish to exclude
944 from download (*Note Directory-Based Limits:: for more details.)
945 Elements of LIST may contain wildcards.
949 Disable the time-consuming DNS lookup of almost all hosts (*Note
955 Do not ever ascend to the parent directory when retrieving
956 recursively. This is a useful option, since it guarantees that
957 only the files *below* a certain hierarchy will be downloaded.
958 *Note Directory-Based Limits:: for more details.
961 File: wget.info, Node: Recursive Retrieval, Next: Following Links, Prev: Invoking, Up: Top
966 GNU Wget is capable of traversing parts of the Web (or a single HTTP
967 or FTP server), depth-first following links and directory structure.
968 This is called "recursive" retrieving, or "recursion".
970 With HTTP URLs, Wget retrieves and parses the HTML from the given
971 URL, documents, retrieving the files the HTML document was referring
972 to, through markups like `href', or `src'. If the freshly downloaded
973 file is also of type `text/html', it will be parsed and followed
976 The maximum "depth" to which the retrieval may descend is specified
977 with the `-l' option (the default maximum depth is five layers). *Note
978 Recursive Retrieval::.
980 When retrieving an FTP URL recursively, Wget will retrieve all the
981 data from the given directory tree (including the subdirectories up to
982 the specified depth) on the remote server, creating its mirror image
983 locally. FTP retrieval is also limited by the `depth' parameter.
985 By default, Wget will create a local directory tree, corresponding to
986 the one found on the remote server.
988 Recursive retrieving can find a number of applications, the most
989 important of which is mirroring. It is also useful for WWW
990 presentations, and any other opportunities where slow network
991 connections should be bypassed by storing the files locally.
993 You should be warned that invoking recursion may cause grave
994 overloading on your system, because of the fast exchange of data
995 through the network; all of this may hamper other users' work. The
996 same stands for the foreign server you are mirroring--the more requests
997 it gets in a rows, the greater is its load.
999 Careless retrieving can also fill your file system uncontrollably,
1000 which can grind the machine to a halt.
1002 The load can be minimized by lowering the maximum recursion level
1003 (`-l') and/or by lowering the number of retries (`-t'). You may also
1004 consider using the `-w' option to slow down your requests to the remote
1005 servers, as well as the numerous options to narrow the number of
1006 followed links (*Note Following Links::).
1008 Recursive retrieval is a good thing when used properly. Please take
1009 all precautions not to wreak havoc through carelessness.
1012 File: wget.info, Node: Following Links, Next: Time-Stamping, Prev: Recursive Retrieval, Up: Top
1017 When retrieving recursively, one does not wish to retrieve loads of
1018 unnecessary data. Most of the time the users bear in mind exactly what
1019 they want to download, and want Wget to follow only specific links.
1021 For example, if you wish to download the music archive from
1022 `fly.cc.fer.hr', you will not want to download all the home pages that
1023 happen to be referenced by an obscure part of the archive.
1025 Wget possesses several mechanisms that allows you to fine-tune which
1026 links it will follow.
1030 * Relative Links:: Follow relative links only.
1031 * Host Checking:: Follow links on the same host.
1032 * Domain Acceptance:: Check on a list of domains.
1033 * All Hosts:: No host restrictions.
1034 * Types of Files:: Getting only certain files.
1035 * Directory-Based Limits:: Getting only certain directories.
1036 * FTP Links:: Following FTP links.
1039 File: wget.info, Node: Relative Links, Next: Host Checking, Prev: Following Links, Up: Following Links
1044 When only relative links are followed (option `-L'), recursive
1045 retrieving will never span hosts. No time-expensive DNS-lookups will
1046 be performed, and the process will be very fast, with the minimum
1047 strain of the network. This will suit your needs often, especially when
1048 mirroring the output of various `x2html' converters, since they
1049 generally output relative links.
1052 File: wget.info, Node: Host Checking, Next: Domain Acceptance, Prev: Relative Links, Up: Following Links
1057 The drawback of following the relative links solely is that humans
1058 often tend to mix them with absolute links to the very same host, and
1059 the very same page. In this mode (which is the default mode for
1060 following links) all URLs that refer to the same host will be retrieved.
1062 The problem with this option are the aliases of the hosts and
1063 domains. Thus there is no way for Wget to know that `regoc.srce.hr' and
1064 `www.srce.hr' are the same host, or that `fly.cc.fer.hr' is the same as
1065 `fly.cc.etf.hr'. Whenever an absolute link is encountered, the host is
1066 DNS-looked-up with `gethostbyname' to check whether we are maybe
1067 dealing with the same hosts. Although the results of `gethostbyname'
1068 are cached, it is still a great slowdown, e.g. when dealing with large
1069 indices of home pages on different hosts (because each of the hosts
1070 must be DNS-resolved to see whether it just *might* be an alias of the
1073 To avoid the overhead you may use `-nh', which will turn off
1074 DNS-resolving and make Wget compare hosts literally. This will make
1075 things run much faster, but also much less reliable (e.g. `www.srce.hr'
1076 and `regoc.srce.hr' will be flagged as different hosts).
1078 Note that modern HTTP servers allow one IP address to host several
1079 "virtual servers", each having its own directory hierarchy. Such
1080 "servers" are distinguished by their hostnames (all of which point to
1081 the same IP address); for this to work, a client must send a `Host'
1082 header, which is what Wget does. However, in that case Wget *must not*
1083 try to divine a host's "real" address, nor try to use the same hostname
1084 for each access, i.e. `-nh' must be turned on.
1086 In other words, the `-nh' option must be used to enable the
1087 retrieval from virtual servers distinguished by their hostnames. As the
1088 number of such server setups grow, the behavior of `-nh' may become the
1089 default in the future.
1092 File: wget.info, Node: Domain Acceptance, Next: All Hosts, Prev: Host Checking, Up: Following Links
1097 With the `-D' option you may specify the domains that will be
1098 followed. The hosts the domain of which is not in this list will not be
1099 DNS-resolved. Thus you can specify `-Dmit.edu' just to make sure that
1100 *nothing outside of MIT gets looked up*. This is very important and
1101 useful. It also means that `-D' does *not* imply `-H' (span all
1102 hosts), which must be specified explicitly. Feel free to use this
1103 options since it will speed things up, with almost all the reliability
1104 of checking for all hosts. Thus you could invoke
1106 wget -r -D.hr http://fly.cc.fer.hr/
1108 to make sure that only the hosts in `.hr' domain get DNS-looked-up
1109 for being equal to `fly.cc.fer.hr'. So `fly.cc.etf.hr' will be checked
1110 (only once!) and found equal, but `www.gnu.ai.mit.edu' will not even be
1113 Of course, domain acceptance can be used to limit the retrieval to
1114 particular domains with spanning of hosts in them, but then you must
1115 specify `-H' explicitly. E.g.:
1117 wget -r -H -Dmit.edu,stanford.edu http://www.mit.edu/
1119 will start with `http://www.mit.edu/', following links across MIT
1122 If there are domains you want to exclude specifically, you can do it
1123 with `--exclude-domains', which accepts the same type of arguments of
1124 `-D', but will *exclude* all the listed domains. For example, if you
1125 want to download all the hosts from `foo.edu' domain, with the
1126 exception of `sunsite.foo.edu', you can do it like this:
1128 wget -rH -Dfoo.edu --exclude-domains sunsite.foo.edu http://www.foo.edu/
1131 File: wget.info, Node: All Hosts, Next: Types of Files, Prev: Domain Acceptance, Up: Following Links
1136 When `-H' is specified without `-D', all hosts are freely spanned.
1137 There are no restrictions whatsoever as to what part of the net Wget
1138 will go to fetch documents, other than maximum retrieval depth. If a
1139 page references `www.yahoo.com', so be it. Such an option is rarely
1143 File: wget.info, Node: Types of Files, Next: Directory-Based Limits, Prev: All Hosts, Up: Following Links
1148 When downloading material from the web, you will often want to
1149 restrict the retrieval to only certain file types. For example, if you
1150 are interested in downloading GIFs, you will not be overjoyed to get
1151 loads of PostScript documents, and vice versa.
1153 Wget offers two options to deal with this problem. Each option
1154 description lists a short name, a long name, and the equivalent command
1160 The argument to `--accept' option is a list of file suffixes or
1161 patterns that Wget will download during recursive retrieval. A
1162 suffix is the ending part of a file, and consists of "normal"
1163 letters, e.g. `gif' or `.jpg'. A matching pattern contains
1164 shell-like wildcards, e.g. `books*' or `zelazny*196[0-9]*'.
1166 So, specifying `wget -A gif,jpg' will make Wget download only the
1167 files ending with `gif' or `jpg', i.e. GIFs and JPEGs. On the
1168 other hand, `wget -A "zelazny*196[0-9]*"' will download only files
1169 beginning with `zelazny' and containing numbers from 1960 to 1969
1170 anywhere within. Look up the manual of your shell for a
1171 description of how pattern matching works.
1173 Of course, any number of suffixes and patterns can be combined
1174 into a comma-separated list, and given as an argument to `-A'.
1179 The `--reject' option works the same way as `--accept', only its
1180 logic is the reverse; Wget will download all files *except* the
1181 ones matching the suffixes (or patterns) in the list.
1183 So, if you want to download a whole page except for the cumbersome
1184 MPEGs and .AU files, you can use `wget -R mpg,mpeg,au'.
1185 Analogously, to download all files except the ones beginning with
1186 `bjork', use `wget -R "bjork*"'. The quotes are to prevent
1187 expansion by the shell.
1189 The `-A' and `-R' options may be combined to achieve even better
1190 fine-tuning of which files to retrieve. E.g. `wget -A "*zelazny*" -R
1191 .ps' will download all the files having `zelazny' as a part of their
1192 name, but *not* the PostScript files.
1194 Note that these two options do not affect the downloading of HTML
1195 files; Wget must load all the HTMLs to know where to go at
1196 all--recursive retrieval would make no sense otherwise.
projects / wget / blob