From: Micah Cowan Date: Thu, 13 Sep 2007 07:36:49 +0000 (-0700) Subject: Merge with SVN r2378 X-Git-Tag: v1.13~545 X-Git-Url: http://sjero.net/git/?p=wget;a=commitdiff_plain;h=661c3788c04052ca98270296a7c8d022cb267423;hp=d6c1be263d5b2687a066c5ada79d5c013f8680d0 Merge with SVN r2378 --- diff --git a/.hgignore b/.hgignore index 5243e1c7..48573d7b 100644 --- a/.hgignore +++ b/.hgignore @@ -15,10 +15,25 @@ src/*.o src/config.h src/config.h.in src/wget +src/tags +src/TAGS doc/wget.1 doc/wget.pod doc/wget.info doc/sample.wgetrc.munged_for_texi_inclusion +doc/wget.aux +doc/wget.cp +doc/wget.cps +doc/wget.dvi +doc/wget.fn +doc/wget.ky +doc/wget.log +doc/wget.pg +doc/wget.toc +doc/wget.tp +doc/wget.vr +doc/wget_a4.ps +doc/wget_us.ps po/Makefile.in po/POTFILES po/*.gmo diff --git a/TODO b/TODO new file mode 100644 index 00000000..4fc5a6c2 --- /dev/null +++ b/TODO @@ -0,0 +1,157 @@ + -*- outline -*- + +This is the to-do list for GNU Wget. There is no timetable of when we +plan to implement these features -- this is just a list of features +we'd like to see in Wget, as well as a list of problems that need +fixing. Patches to implement these items are likely to be accepted, +especially if they follow the coding convention outlined in PATCHES +and if they patch the documentation as well. + +The items are not listed in any particular order (except that +recently-added items may tend towards the top). Not all of these +represent user-visible changes. + +* Change the file name generation logic so that redirects can't dictate + file names (but redirects should still be followed). By default, file + names should be generated only from the URL the user provided. However, + with an appropriate flag, Wget will allow the remote server to specify + the file name, either through redirection (as is always the case now) + or via the increasingly popular header `Content-Disposition: XXX; + filename="FILE"'. + + The file name should be generated and displayed *after* processing + the server's response, not before, as it is done now. This will + allow trivial implementation of -nc, of O_EXCL when opening the + file, --html-extension will stop being a horrible hack, and so on. + +* -O should be respected, with no exceptions. It should work in + conjunction with -N and -k. (This is hard to achieve in the current + code base.) Ancillary files, such as directory listings and such, + should be downloaded either directly to memory, or to /tmp. + +* Implement digest and NTLM authorization for proxies. This is harder + than it seems because it requires some rethinking of the HTTP code. + +* Rethink the interaction between recur.c (the recursive download code) + and HTTP/FTP code. Ideally, the downloading code should have a way + to retrieve a file and, optionally, to specify a list of URLs for + continuing the "recursive" download. FTP code will surely benefit + from such a restructuring because its current incarnation is way too + smart for its own good. + +* Both HTTP and FTP connections should be first-class objects that can + be reused after a download is done. Currently information about both + is kept implicitly on the stack, and forgotten after each download. + +* Restructure the FTP code to remove massive amounts of code duplication + and repetition. Remove all the "intelligence" and make it work as + outlined in the previous bullet. + +* Add support for SFTP. Teach Wget about newer features of more + recent FTP servers in general, such as receiving reliable checksums + and timestamps. This can be used to implement really robust + downloads. + +* Wget shouldn't delete rejected files that were not downloaded, but + just found on disk because of `-nc'. For example, `wget -r -nc + -A.gif URL' should allow the user to get all the GIFs without + removing any of the existing HTML files. + +* Be careful not to lose username/password information given for the + URL on the command line. For example, + wget -r http://username:password@server/path/ should send that + username and password to all content under /path/ (this is apparently + what browsers do). + +* Don't send credentials using "Basic" authorization before the server + has a chance to tell us that it supports Digest or NTLM! + +* Add a --range parameter allowing you to explicitly specify a range + of bytes to get from a file over HTTP (FTP only supports ranges + ending at the end of the file, though forcibly disconnecting from + the server at the desired endpoint would work). For example, + --range=n-m would specify inclusive range (a la the Range header), + and --range=n:m would specify exclusive range (a la Python's + slices). -c should work with --range by assuming the range is + partially downloaded on disk, and contuing from there (effectively + requesting a smaller range). + +* If multiple FTP URLs are specified that are on the same host, Wget should + re-use the connection rather than opening a new one for each file. + This should be easy provided the above restructuring of FTP code that + would include the FTP connection becoming a first-class objects. + +* Try to devise a scheme so that, when password is unknown, Wget asks + the user for one. This is harder than it seems because the password + may be requested by some page encountered long after the user has + left Wget to run. + +* If -c used with -N, check to make sure a file hasn't changed on the server + before "continuing" to download it (preventing a bogus hybrid file). + +* Generalize --html-extension to something like --mime-extensions and + have consult mime.types for the preferred extension. Non-HTML files + with filenames changed this way would be re-downloaded each time + despite -N unless .orig files were saved for them. (#### Why? The + HEAD request we use to implement -N would still be able to construct + the correct file name based on the declared Content-Type.) + + Since .orig would contain the same data as non-.orig, the latter + could be just a link to the former. Another possibility would be to + implement a per-directory database called something like + .wget_url_mapping containing URLs and their corresponding filenames. + +* When spanning hosts, there's no way to say that you are only + interested in files in a certain directory on _one_ of the hosts (-I + and -X apply to all). Perhaps -I and -X should take an optional + "hostname:" before the directory? + +* --retr-symlinks should cause wget to traverse links to directories too. + +* Make wget return non-zero status in more situations, like incorrect HTTP auth. + Create and document different exit statuses for different errors. + +* Make -K compare X.orig to X and move the former on top of the latter if + they're the same, rather than leaving identical .orig files laying around. + +* Make `-k' check for files that were downloaded in the past and convert links + to them in newly-downloaded documents. + +* Devise a way for options to have effect on a per-URL basis. This is very + natural for some options, such as --post-data. It could be implemented + simply by having more than one struct options. + +* Add option to clobber existing file names (no `.N' suffixes). + +* Add option to only list wildcard matches without doing the download. The same + could be generalized to support something like apt's --print-uri. + +* Handle MIME types correctly. There should be an option to (not) + retrieve files based on MIME types, e.g. `--accept-types=image/*'. + This would work for FTP by translating file extensions to MIME types + using mime.types. + +* Allow time-stamping by arbitrary date. For example, + wget --if-modified-after DATE URL. + +* Make quota apply to single files, preferrably so that the download of an + oversized file is not attempted at all. + +* When updating an existing mirror, download to temporary files (such as .in*) + and rename the file after the download is done. + +* Add an option to delete or move no-longer-existent files when mirroring. + +* Implement uploading (--upload=FILE URL?) in FTP and HTTP. A beginning of + this is available in the form of --post-file, but it should be expanded to + be really useful. + +* Make HTTP timestamping use If-Modified-Since facility. + +* Add more protocols (such as news or possibly some of the streaming + protocols), implementing them in a modular fashion. + +* Add a "rollback" option to have continued retrieval throw away a + configurable number of bytes at the end of a file before resuming + download. Apparently, some stupid proxies insert a "transfer + interrupted" string we need to get rid of.