X-Git-Url: http://sjero.net/git/?p=wget;a=blobdiff_plain;f=doc%2Fwget.texi;h=cced7edda118e268b6f290ae272380179044dbce;hp=cd379e97b807a6e2faf07f3c8e1dc9f2ea213933;hb=42c78fdd71c311cf96210b709ec0a18ef45ef87f;hpb=e24e81725943d7bc54e1b0b902806c816b983785 diff --git a/doc/wget.texi b/doc/wget.texi index cd379e97..cced7edd 100644 --- a/doc/wget.texi +++ b/doc/wget.texi @@ -479,6 +479,9 @@ Turn off verbose without being completely quiet (use @samp{-q} for that), which means that error messages and basic information still get printed. +@item --report-speed=@var{type} +Output bandwidth as @var{type}. The only accepted value is @samp{bits}. + @cindex input-file @item -i @var{file} @itemx --input-file=@var{file} @@ -547,10 +550,10 @@ IPs. @cindex retries @cindex tries -@cindex number of retries +@cindex number of tries @item -t @var{number} @itemx --tries=@var{number} -Set number of retries to @var{number}. Specify 0 or @samp{inf} for +Set number of tries to @var{number}. Specify 0 or @samp{inf} for infinite retrying. The default is to retry 20 times, with the exception of fatal errors like ``connection refused'' or ``not found'' (404), which are not retried. @@ -627,6 +630,13 @@ Note that when @samp{-nc} is specified, files with the suffixes @samp{.html} or @samp{.htm} will be loaded from the local disk and parsed as if they had been retrieved from the Web. +@cindex backing up files +@item --backups=@var{backups} +Before (over)writing a file, back up an existing file by adding a +@samp{.1} suffix (@samp{_1} on VMS) to the file name. Such backup +files are rotated to @samp{.2}, @samp{.3}, and so on, up to +@var{backups} (and lost beyond that). + @cindex continue retrieval @cindex incomplete downloads @cindex resume download @@ -712,9 +722,12 @@ different meaning to one dot. With the @code{default} style each dot represents 1K, there are ten dots in a cluster and 50 dots in a line. The @code{binary} style has a more ``computer''-like orientation---8K dots, 16-dots clusters and 48 dots per line (which makes for 384K -lines). The @code{mega} style is suitable for downloading very large +lines). The @code{mega} style is suitable for downloading large files---each dot represents 64K retrieved, there are eight dots in a cluster, and 48 dots on each line (so each line contains 3M). +If @code{mega} is not enough then you can use the @code{giga} +style---each dot represents 1M retrieved, there are eight dots in a +cluster, and 32 dots on each line (so each line contains 32M). Note that you can set the default style using the @code{progress} command in @file{.wgetrc}. That setting may be overridden from the @@ -729,7 +742,7 @@ Turn on time-stamping. @xref{Time-Stamping}, for details. @item --no-use-server-timestamps Don't set the local file's timestamp by the one on the server. -By default, when a file is downloaded, it's timestamps are set to +By default, when a file is downloaded, its timestamps are set to match those from the remote file. This allows the use of @samp{--timestamping} on subsequent invocations of wget. However, it is sometimes useful to base the local file's timestamp on when it was @@ -872,7 +885,7 @@ recommendation to block many unrelated users from a web site due to the actions of one. @cindex proxy -@itemx --no-proxy +@item --no-proxy Don't use proxies, even if the appropriate @code{*_proxy} environment variable is defined. @@ -973,7 +986,7 @@ are outside the range of @sc{ascii} characters (that is, greater than whose encoding does not match the one used locally. @cindex IPv6 -@itemx -4 +@item -4 @itemx --inet4-only @itemx -6 @itemx --inet6-only @@ -1454,6 +1467,11 @@ like everything else. Wget does not currently support @code{application/x-www-form-urlencoded}. Only one of @samp{--post-data} and @samp{--post-file} should be specified. +Please note that wget does not require the content to be of the form +@code{key1=value1&key2=value2}, and neither does it test for it. Wget will +simply transmit whatever data is provided to it. Most servers however expect +the POST data to be in the above format when processing HTML Forms. + Please be aware that Wget needs to know the size of the POST data in advance. Therefore the argument to @code{--post-file} must be a regular file; specifying a FIFO or something like @file{/dev/stdin} won't work. @@ -1464,14 +1482,15 @@ use chunked unless it knows it's talking to an HTTP/1.1 server. And it can't know that until it receives a response, which in turn requires the request to have been completed -- a chicken-and-egg problem. -Note: if Wget is redirected after the POST request is completed, it -will not send the POST data to the redirected URL. This is because -URLs that process POST often respond with a redirection to a regular -page, which does not desire or accept POST. It is not completely -clear that this behavior is optimal; if it doesn't work out, it might -be changed in the future. +Note: As of version 1.15 if Wget is redirected after the POST request is +completed, its behaviour will depend on the response code returned by the +server. In case of a 301 Moved Permanently, 302 Moved Temporarily or +307 Temporary Redirect, Wget will, in accordance with RFC2616, continue +to send a POST request. +In case a server wants the client to change the Request method upon +redirection, it should send a 303 See Other response code. -This example shows how to log to a server using POST and then proceed to +This example shows how to log in to a server using POST and then proceed to download the desired pages, presumably only accessible to authorized users: @@ -1494,6 +1513,37 @@ them (and neither will browsers) and the @file{cookies.txt} file will be empty. In that case use @samp{--keep-session-cookies} along with @samp{--save-cookies} to force saving of session cookies. +@cindex Other HTTP Methods +@item --method=@var{HTTP-Method} +For the purpose of RESTful scripting, Wget allows sending of other HTTP Methods +without the need to explicitly set them using @samp{--header=Header-Line}. +Wget will use whatever string is passed to it after @samp{--method} as the HTTP +Method to the server. + +@item --body-data=@var{Data-String} +@itemx --body-file=@var{Data-File} +Must be set when additional data needs to be sent to the server along with the +Method specified using @samp{--method}. @samp{--body-data} sends @var{string} as +data, whereas @samp{--body-file} sends the contents of @var{file}. Other than that, +they work in exactly the same way. + +Currently, @samp{--body-file} is @emph{not} for transmitting files as a whole. +Wget does not currently support @code{multipart/form-data} for transmitting data; +only @code{application/x-www-form-urlencoded}. In the future, this may be changed +so that wget sends the @samp{--body-file} as a complete file instead of sending its +contents to the server. Please be aware that Wget needs to know the contents of +BODY Data in advance, and hence the argument to @samp{--body-file} should be a +regular file. See @samp{--post-file} for a more detailed explanation. +Only one of @samp{--body-data} and @samp{--body-file} should be specified. + +If Wget is redirected after the request is completed, Wget will +suspend the current method and send a GET request till the redirection +is completed. This is true for all redirection response codes except +307 Temporary Redirect which is used to explicitly specify that the +request method should @emph{not} change. Another exception is when +the method is set to @code{POST}, in which case the redirection rules +specified under @samp{--post-data} are followed. + @cindex Content-Disposition @item --content-disposition @@ -1556,6 +1606,9 @@ buggy SSL server implementations that make it hard for OpenSSL to choose the correct protocol version. Fortunately, such servers are quite rare. +@item --https-only +When in recursive mode, only HTTPS links are followed. + @cindex SSL certificate, check @item --no-check-certificate Don't check the server certificate against the available certificate @@ -1658,6 +1711,36 @@ not used), EGD is never contacted. EGD is not needed on modern Unix systems that support @file{/dev/random}. @end table +@cindex WARC +@table @samp +@item --warc-file=@var{file} +Use @var{file} as the destination WARC file. + +@item --warc-header=@var{string} +Use @var{string} into as the warcinfo record. + +@item --warc-max-size=@var{size} +Set the maximum size of the WARC files to @var{size}. + +@item --warc-cdx +Write CDX index files. + +@item --warc-dedup=@var{file} +Do not store records listed in this CDX file. + +@item --no-warc-compression +Do not compress WARC files with GZIP. + +@item --no-warc-digests +Do not calculate SHA1 digests. + +@item --no-warc-keep-log +Do not store the log file in a WARC record. + +@item --warc-tempdir=@var{dir} +Specify the location for temporary files created by the WARC writer. +@end table + @node FTP Options, Recursive Retrieval Options, HTTPS (SSL/TLS) Options, Invoking @section FTP Options @@ -1743,6 +1826,10 @@ in some rare firewall configurations, active FTP actually works when passive FTP doesn't. If you suspect this to be the case, use this option, or set @code{passive_ftp=off} in your init file. +@cindex file permissions +@item --preserve-permissions +Preserve remote file permissions instead of permissions set by umask. + @cindex symbolic links, retrieving @item --retr-symlinks Usually, when retrieving @sc{ftp} directories recursively and a symbolic @@ -1980,6 +2067,15 @@ any of the wildcard characters, @samp{*}, @samp{?}, @samp{[} or @samp{]}, appear in an element of @var{acclist} or @var{rejlist}, it will be treated as a pattern, rather than a suffix. +@item --accept-regex @var{urlregex} +@itemx --reject-regex @var{urlregex} +Specify a regular expression to accept or reject the complete URL. + +@item --regex-type @var{regextype} +Specify the regular expression type. Possible types are @samp{posix} or +@samp{pcre}. Note that to be able to use @samp{pcre} type, wget has to be +compiled with libpcre support. + @item -D @var{domain-list} @itemx --domains=@var{domain-list} Set domains to be followed. @var{domain-list} is a comma-separated list @@ -2796,9 +2892,11 @@ enables it). Enable/disable saving pre-converted files with the suffix @samp{.orig}---the same as @samp{-K} (which enables it). -@c @item backups = @var{number} -@c #### Document me! -@c +@item backups = @var{number} +Use up to @var{number} backups for a file. Backups are rotated by +adding an incremental counter that starts at @samp{1}. The default is +@samp{0}. + @item base = @var{string} Consider relative @sc{url}s in input files (specified via the @samp{input} command or the @samp{--input-file}/@samp{-i} option, @@ -3060,7 +3158,7 @@ display properly---the same as @samp{-p}. Change setting of passive @sc{ftp}, equivalent to the @samp{--passive-ftp} option. -@itemx password = @var{string} +@item password = @var{string} Specify password @var{string} for both @sc{ftp} and @sc{http} file retrieval. This command can be overridden using the @samp{ftp_password} and @samp{http_password} command for @sc{ftp} and @sc{http} respectively. @@ -3542,34 +3640,36 @@ internal networks from the rest of Internet. In order to obtain information from the Web, their users connect and retrieve remote data using an authorized proxy. +@c man begin ENVIRONMENT Wget supports proxies for both @sc{http} and @sc{ftp} retrievals. The standard way to specify proxy location, which Wget recognizes, is using the following environment variables: -@table @code +@table @env @item http_proxy @itemx https_proxy -If set, the @code{http_proxy} and @code{https_proxy} variables should +If set, the @env{http_proxy} and @env{https_proxy} variables should contain the @sc{url}s of the proxies for @sc{http} and @sc{https} connections respectively. @item ftp_proxy This variable should contain the @sc{url} of the proxy for @sc{ftp} -connections. It is quite common that @code{http_proxy} and -@code{ftp_proxy} are set to the same @sc{url}. +connections. It is quite common that @env{http_proxy} and +@env{ftp_proxy} are set to the same @sc{url}. @item no_proxy This variable should contain a comma-separated list of domain extensions proxy should @emph{not} be used for. For instance, if the value of -@code{no_proxy} is @samp{.mit.edu}, proxy will not be used to retrieve +@env{no_proxy} is @samp{.mit.edu}, proxy will not be used to retrieve documents from MIT. @end table +@c man end In addition to the environment variables, proxy location and settings may be specified from within Wget itself. @table @samp -@itemx --no-proxy +@item --no-proxy @itemx proxy = on/off This option and the corresponding command may be used to suppress the use of proxy, even if the appropriate environment variables are set.