3 Wget 1.11 for VMS (1.11a)
4 =========================
6 ------------------------------------------------------------------------
11 No claims of any sort are made herein. The main Wget developers seem
12 indifferent to this effort. Any suggestions for improving the code in
13 the VMS-specific sections are welcome, but low expectations are more
14 realistic than high ones.
16 ------------------------------------------------------------------------
21 This note accompanies a quick VMS port of Wget version 1.10.2, a GNU
22 free network utility to retrieve files from the World Wide Web using
23 HTTP and FTP. The most obvious features seem to work, but many have not
26 Built and tested (slightly) in these environments:
28 OpenVMS Alpha V7.3-2, HP C V7.3-009, TCPIP V5.4 - ECO 6
30 OpenVMS IA64 V8.3, HP C V7.3-018, TCPIP V5.6.
32 OpenVMS VAX V7.3, Compaq C V6.4-005, TCPIP V5.1.
34 ------------------------------------------------------------------------
39 Version 1.11a (PRELIMINARY) 2008-01-30
40 ---------------------------------------
42 - General code update to Wget 1.11.
44 - Changed to avoid trying "LIST -a" for a VMS FTP server.
46 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
48 Version 1.10.2c 2006-11-16
49 ----------------------------
51 - Changed the code which determines the FTP server type to avoid an
52 ACCVIO if no text was returned in response to a "SYST" inquiry.
54 - Fixed a command-line parsing problem with the "--ftp-stmlf" option.
56 - Fixed an unclosed comment (again?) in FTP-LS.C which could have caused
57 problems with ODS5 extended file names.
59 - Changed VMS-WGET.COM to assume that nothing special is now required
60 for use with the MultiNet IP package. For obsolete VMS and/or
61 MultiNet versions where the old MultiNet-specific changes are still
62 needed, specify "MULTINET" on the VMS-WGET.COM command line.
64 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
66 Version 1.10.2b 2004-12-15
67 ----------------------------
69 - Changed the "-V" version report to include some host type information.
71 - Changed the I/O scheme on VMS for progress dots to avoid spurious
72 newline characters when output is directed to a file.
74 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
76 Version 1.10.2a 2004-10-13
77 ----------------------------
79 - General code update to Wget 1.10.2.
81 - Many legitimate FTP CWD operations on a VMS FTP server failed. The
82 typical symptom was "No such directory `perfectly/vaild/directory'."
84 - FTP CWD operations on a VMS FTP server now end with a "CWD []". For
85 an FTP URL like "ftp://host/a/b/c/d.e" (and contrary to the RFC), Wget
86 uses a UNIX-like directory specification in its CWD command, here,
87 "CWD a/b/c". This was observed to switch a TCPware FTP server
88 (V5.6-2) into its UNIX emulation mode. Because Wget checks the system
89 type only once at the beginning of a session, the associated change in
90 directory listing format confused the program. The typical symptom
91 was "No such file `-rwx---r-x'." The added "CWD []" command resets
92 the TCPware server to VMS mode, and should be harmless on other VMS
95 - VMS-WGET.COM now finds MMK when it's on DCL$PATH as well as when a
96 foreign command symbol "MMK" is defined.
98 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
100 Version 1.10.1a 2004-09-09
101 ---------------------------
103 - General code update to Wget 1.10.1.
105 - VMS-specific files are now in the new [.VMS] directory.
107 - Changes to VMS-WGET.COM:
108 - New command-line option, LARGE, enables large-file support on
110 - VMS-WGET.COM may now be run from any default directory, not only
113 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
115 Version 1.9.1e 2005-09-09
116 --------------------------
118 - The default User-Agent string on VMS now includes the host
119 architecture and VMS version. For example, "Wget/1.9.1e (VMS Alpha
122 - Fixed a problem on VMS with an ODS5 destination file system when a
123 URL-derived file name contained an invalid character ("?" or "*"). A
124 typical complaint might look like:
125 Cannot write to `www.foraddift.non/aspx/index.asp?sid=463e'
126 (file specification syntax error).
127 Now, "?" is converted to "!", and "*" to "#".
129 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
131 Version 1.9.1d 2005-06-21
132 --------------------------
134 - Replaced the remaining instances where ".orig" was appended to a file
135 name. Now (for VMS), "_orig" is used everywhere (except in the
138 - Disabled (for VMS) the code which appended ".nnn" (decimal "nnn") to
139 an output file name to avoid overwriting an existing file. The file
140 name is now used unchanged, and VMS file versioning is expected to
141 prevent overwriting. (The multi-dot file names made this way caused
142 I/O errors on ODS2 file systems.)
144 - Changed to do translation of CR-LF line endings to local line endings
145 for ASCII FTP transfers.
147 - Changed (on VMS) to use fixed-512 format for binary FTP files. Added
148 the new "--ftp-stmlf" option ("ftpstmlf on" command) to let the user
149 restore the old behavior, where Stream_LF was used for both ASCII and
152 - Added (on VMS) a new open callback function which senses (where
153 supported) the process RMS_DEFAULT values for file extend quantity
154 (deq), multi-block count (mbc), and multi-buffer count (mbf), and sets
155 the FAB/RAB parameters accordingly. The default deq is now much
156 larger than before (16384, was none), and the default mbc is now 127
157 (was none), speeding creation of a large output file. Explicitly set
158 RMS_DEFAULT values override built-in defaults.
160 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
162 Version 1.9.1c 2004-12-15
163 --------------------------
165 - Added https (SSL) support, changing DESCRIP.MMS and VMS-WGET.COM, and
166 adding WGET_SSL.OPT and WGET_SSL_HP.OPT. Limited testing was done on
167 Alpha, using HP SSL V1.1-B and OpenSSL version 0.9.7e. The build
168 procedure automatically enables SSL support if it can find one of the
169 SSL header files. The details are explained in comments in
170 VMS-WGET.COM, including its new command-line options, HPSSL and NOSSL.
172 - Changed CONNECT.C to fix some problems with MultiNet TCP/IP software.
174 - A build problem related to header files (netdb.h) when compiled
175 with MultiNet. Typical symptom:
177 %CC-E-REDEFSTRUCT, In this declaration, the struct "hostent" is
178 redefined. at line number 178 in module NETDB of text library
179 SYS$COMMON:[SYSLIB]DECC$RTLDEF.TLB;1
181 - A run-time problem with an FTP URL. Typical symptom:
185 - Changed FTP.C to fix various problems with time-stamping ("-N",
186 "--timestamping") when using an FTP URL. Some of the misbehavior was
187 specific to a VMS FTP server, but not all. The typical result was a
188 file with the modification date (and, in some cases, the creation
189 date) left at the date of the download, rather than set to match the
190 date on the server. Some misleading diagnostic messages also have
193 - Changed FTP-LS.C to fix a typographical error (unclosed comment).
195 - Changed NETRC.C to fix a potential build problem on non-VMS systems.
196 Adding some VMS-specific code introduced some improper statement
197 ordering, which may generate complaints (or failure) with some C
200 - Changed VMS.C to fix a problem with the utime() function (supplied
201 with Wget) when the user specified a non-UNIX file name with the Wget
202 "-O" option. Typical symptom:
204 utime(home_sms:[sms.wget]t.h): error 0
206 As the message might suggest, the file date-time was not being set
207 correctly. With this fix, utime() should work with either a UNIX-like
208 or VMS-like file specification, and it should also set errno properly
209 when it fails, enabling a more informative error message.
211 - New MMS/MMK description files and associated command procedures enable
212 automatic source dependency generation. Changed: DESCRIP.MMS and
213 VMS-WGET.COM. New: DESCRIP_DEPS.MMS, DESCRIP_MKDEPS.MMS,
214 DESCRIP_SRC.MMS, COLLECT_DEPS.COM, and MOD_DEP.COM.
216 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
218 Version 1.9.1b 2004-09-28
219 --------------------------
221 - Changed CONFIG.H_VMS to fix a build problem with C RTL before VMS
222 V7.3-1 with C Runtime ECO V3.0 on Alpha:
223 %LINK-W-NUDFSYMS, 2 undefined symbols:
224 %LINK-I-UDFSYM, DECC$GXSNPRINTF
225 %LINK-I-UDFSYM, DECC$GXVSNPRINTF
230 - Fixed a build problem with C RTL before V7.2:
231 %CC-E-UNDECLARED, In this statement, "DVI$C_ACP_F11V5" is not
232 declared. at line number 262 in file
233 disk:[xxx.WGET-1_9_1A_VMS.SRC]VMS.C;1
235 - Fixed a build problem with C RTL before VMS V7.3 with C Runtime ECO
237 %CC-I-IMPLICITFUNC, In this statement, the identifier
238 "decc$feature_get_index" is implicitly declared as a function.
239 at line number 155 in file disk:[xxx.WGET-1_9_1A_VMS.SRC]VMS.C;1
241 %CC-I-IMPLICITFUNC, In this statement, the identifier
242 "decc$feature_get_value" is implicitly declared as a function.
243 at line number 159 in file disk:[xxx.WGET-1_9_1A_VMS.SRC]VMS.C;1
245 %CC-I-IMPLICITFUNC, In this statement, the identifier
246 "decc$feature_set_value" is implicitly declared as a function.
247 at line number 169 in file disk:[xxx.WGET-1_9_1A_VMS.SRC]VMS.C;1
251 %LINK-W-NUDFSYMS, 3 undefined symbols:
252 %LINK-I-UDFSYM, DECC$FEATURE_GET_INDEX
253 %LINK-I-UDFSYM, DECC$FEATURE_GET_VALUE
254 %LINK-I-UDFSYM, DECC$FEATURE_SET_VALUE
257 - Fixed a build problem with Multinet involving use of both the
258 Multinet-specific and generic netdb.h header files:
260 %CC-E-REDEFSTRUCT, In this declaration, the struct "hostent" is
261 redefined. at line number 178 in module NETDB of text library
262 SYS$COMMON:[SYSLIB]DECC$RTLDEF.TLB;1
264 - Fixed a typographical error (unclosed comment) which kept the C RTL
265 feature DECC$EFS_CHARSET from being enabled as intended. This
266 caused a run-time problem on non-VAX systems with some ODS5
267 extended file specifications with multiple dots in file names,
268 typically when "-m" ("--mirror"), "-O" ("--output-document"), or
269 "-r" ("--recursive") was used:
271 hostname/test/a.B.c: file specification syntax error
273 test/a.B.c: non-translatable vms error code: 0x186D4
274 rms-f-syn, file specification syntax error
276 - Fixed a spurious diagnostic message when doing HTTP and
277 complex FTP transfers:
279 utime(). sys$parse() = 00010001.
281 - Changed FTP.C to fix a variety of problems encountered when using a
282 VMS FTP server. These problems (not seen in Wget 1.5.3h) were caused
283 mostly by inappropriate code added (in the main Wget development
284 stream) to "help" deal with a VMS FTP server. Removing this code and
285 reforming the procedure used for getting to relative directories seems
286 to have fixed the problems. As before, a single-slash URL
287 ("ftp://host/one/two/name.type") will be treated as specifying a
288 relative directory ("one/two"), while a double-slash URL
289 ("ftp://host//one/two/name.type") will be treated as specifying an
290 absolute directory ("/one/two"). As usual, the VMS FTP server will
291 attempt to puzzle out whether "/one/two" should really be interpreted
292 as "one:[two]", but it's better equipped to do it than Wget is.
294 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
296 Version 1.9.1a 2004-04-28
297 --------------------------
299 - Changes to VMS-WGET.COM since Wget 1.5.3h:
301 - A separate LIST command-line option now controls compiler listing
302 files. The DBG option no longer implies compiler listing files.
303 The LIST option also generates a link map with cross-reference.
305 - The DBG option may actually work now.
307 - Added support for IA64 architecture.
309 - Changes to DESCRIP.MMS since Wget 1.5.3h:
311 - Compiler listing files (.LIS) and link maps (.MAP) are now put into
312 the system-specific directory with the corresponding object files
313 (.OBJ), namely, [.SRC.ALPHA], [.SRC.IA64], or [.SRC.VAX].
314 Previously, they were put into the source directory ([.SRC])
317 - CLEAN target now deletes any .LIS and .MAP files.
319 - New HELP target creates a Wget help library.
321 - The OS_TYPE macro is now defined, including the VMS version.
323 - No more complaints like "%MMS-W-MBREDEFILL, Illegal attempt to
324 redefine macro 'CFLAGS'" on old versions of MMS.
326 - A separate LIST macro now controls compiler listing files. The
327 DBG macro no longer implies compiler listing files.
329 - New support for IA64 architecture.
331 - Built-in initialization of some DECC$* run-time options on non-VAX
332 systems. Currently this behavior is compiled in, so any changes must
333 be made in the VMS.C source file. These changes have no effect on VAX
334 (or older Alpha) systems where the OS and C run-time support is
337 - Command-line case preservation. Wget for VMS enables the following
338 DECC$* run-time option:
339 DECC$ARGV_PARSE_STYLE
340 With extended command parsing enabled ("SET PROCESS
341 /PARSE_STYLE = EXTENDED"), command-line case is preserved. This
342 obviates quoting case-sensitive option switches, URLs, and other
345 - Extended file names on ODS5 file systems. Wget for VMS enables the
346 following DECC$* run-time options:
347 DECC$EFS_CASE_PRESERVE
349 Thus, if an ODS5 file system is used as the destination for
350 downloaded files, multiple dots will be permitted and case will be
351 preserved in file names. If an ODS2 file system is used, the same
352 character substitutions will be made as in Wget 1.5.3h, namely,
353 "." -> "_" (where needed), "~" -> "-", and "@" -> "$".
355 ------------------------------------------------------------------------
357 Changes to Source Files
358 -----------------------
360 Files new for VMS (all in [.VMS]):
362 COLLECT_DEPS.COM Dependency collector for DESCRIP_MKDEPS.MMS.
364 config.h_vms VMS-specific CONFIG.H.
366 decc_ver.c Diagnostic program to provide build environment
369 DESCRIP.MMS Main MMK or MMS description ("make") file.
371 DESCRIP_CONFIG.MMS CONFIG.H description file.
373 DESCRIP_DEPS.MMS Source dependency description file.
375 DESCRIP_MKDEPS.MMS Source dependency generator description file.
377 DESCRIP_SRC.MMS Source list description file.
379 VMS-WGET.COM DCL procedure to build the executable from the
380 source, using MMK or MMS (and DESCRIP*.MMS).
382 vms.c Various VMS-specific code: DEC C run-time
383 initialization, ODS2-ODS5 file system
384 detection, ODS2 file name adjustment, file open
385 callback function for better RMS parameters,
386 "utime()" replacement function, and for VMS
387 before V7.0, "getpwuid()" and "localtime_r()".
389 vms.h VMS-specific header file. For VMS C RTL before
390 V7.3, replacement for <utime.h>. For VMS C RTL
391 before V7.0, replacement for <pwd.h>.
392 Declarations of replacement data
393 types/structures and function prototypes for
396 vms_ip.h IP software type-specific "#include"
397 directives. Used by host.c.
399 vms_name_fix.sh UNIX shell script to restore file names after
400 storage on an ODS2 file system.
402 VMS_NOTES.TXT These notes.
404 WGET.HLP Simple help file.
406 WGET.OPT LINK options file.
408 WGET_MULTINET.OPT LINK options file supplement for Multinet.
410 WGET_SSL.OPT LINK options file supplement for OpenSSL.
412 WGET_SSL_HP.OPT LINK options file supplement for HP SSL.
415 Files modified for VMS (all in [.SRC]):
417 connect.c Added code for VMS header files.
419 convert.c Changed to use "_orig" instead of ".orig" on VMS
420 as the file name suffix used for the "-K"
421 ("--backup-converted") option.
423 ftp.c Added code for VMS header files.
424 Changed to eliminate a spurious -debug message
425 ("Unrecognized permissions for <directory>").
426 Changed to eliminate spurious complaints when
427 recursive downloads hit an empty directory.
428 Changed to fix various problems dealing with a
430 Changed to fix various problems with FTP and
431 time-stamping ("-N").
432 Changed to distinguish between ASCII and binary
433 FTP transfers for proper local line endings.
434 Added (for VMS) VMS-specific file open options.
436 ftp.h Added ftp_dele() function. (See ftp-basic.c.)
437 Added an argument to ftp_list() for the FTP
438 server type. (See ftp-basic.c.)
440 ftp-basic.c Added ftp_dele() function to accomodate a remote
441 delete-after-download option.
442 Changed ftp_list() to avoid trying "LIST -a" for
445 ftp-ls.c Changed to open the ".listing" file in text
447 Changed VMS FTP DIR-parsing code to tolerate
448 more variability (leading blank lines).
449 Added support for the environment variable
450 "WGET_TIMEZONE_DIFFERENTIAL" for file time
451 adjustment (as in Wget 1.5.3h for VMS).
453 host.c Added code for VMS header files.
455 host.h Added code for VMS header files.
457 http.c Added code for VMS header files.
458 Changed to use an OS-specific file name suffix,
459 "_orig" for VMS, rather than a hard-coded
461 Added (for VMS) VMS-specific file open options.
462 Added (for VMS) host type and OS version info to
463 the default User-Agent string.
464 Changed "ssl_connect()" to "ssl_connect_wget()"
465 to avoid conflict with "SSL_connect()".
466 Added a type cast to remove a
467 %CC-I-QUESTCOMPARE1 compiler complaint.
469 init.c Changed (on VMS) to use "SYS$LOGIN:.wgetrc" as
470 the name of the user-specific initialization
471 file (if "WGETRC" is not defined), and
472 "SYSTEM_WGETRC" (expected to be a logical name)
473 as the system-wide initialization file.
474 Changed to open ".wgetrc" in text mode, not
476 Added (for VMS) "--ftp-stmlf" option ("ftpstmlf
477 on" command to force Stream_LF format (instead
478 of fixed-512) for binary FTP output files.
480 log.c Changed the I/O scheme on VMS for progress dots
481 to avoid spurious newline characters when output
482 is directed to a file.
484 main.c Changed the program identification debug message
485 ("built on OS_TYPE" instead of "on OS_TYPE"),
486 and added host information to the "-V" version
488 Changed to set ODS5 flag if the download
489 destination is on an ODS5 file system.
490 Added (for VMS) "--ftp-stmlf" option ("ftpstmlf
491 on" command to force Stream_LF format (instead
492 of fixed-512) for binary FTP output files.
493 Adjusted the help message for -K to say "_orig"
494 instead of ".orig". (See convert.c.)
496 netrc.c Changed to use "SYS$LOGIN:.netrc" as the name of
497 the user name and password file on VMS.
499 openssl.c Changed "ssl_connect()" to "ssl_connect_wget()"
500 to avoid conflict with "SSL_connect()".
502 options.h Added (for VMS) "--ftp-stmlf" option ("ftpstmlf
503 on" command to force Stream_LF format (instead
504 of fixed-512) for binary FTP output files.
506 retr.c Changed to adjust CR-LF line endings in ASCII
507 FTP data to proper local line endings.
508 Changed to disable (for VMS) a useless fflush()
509 which was slowing file I/O.
511 retr.h Changed to accomodate changes for FTP to
512 distinguish between ASCII and binary FTP
513 transfers for proper local ASCII line endings.
515 ssl.h Changed "ssl_connect()" to "ssl_connect_wget()"
516 to avoid conflict with "SSL_connect()".
518 url.c Added code for VMS header files.
519 Added code to replace unacceptable dots (".")
520 (and other invalid characters) with underscores
521 ("_"), tildes ("~") with hyphens ("-"), and at
522 signs ("@") with dollar signs ("$"),
523 respectively, in URL-derived file specifications
524 on ODS2 file systems.
526 utils.c Added VMS-specific code to work around the lack
528 Changed to assist ODS2 name reversion.
529 Added (for VMS) VMS-specific file open options.
530 Changed to use a macro, UNIQ_SEP, instead of a
531 hard-coded ".", as the separator character when
532 adding a decimal "nnn" suffix to a file name to
533 make it unique. Not defining UNIQ_SEP, as on
534 VMS, now disables adding the suffix. (See
537 utils.h Added a declaration of the character properties
538 look-up table for use in ftp-ls.c.
540 version.h Adapted from version.c. Changed the version
541 string from "1.11" to "1.11a".
543 wget.h Added a definition of an OS-specific file name
544 suffix, ORIG_SFX ("_orig", for VMS), to be used
545 instead of a hard-coded ".orig". (See
547 Added a definition of an OS-specific separator
548 character, UNIQ_SEP, for the decimal "nnn"
549 version number appended to a file name to make
550 it unique. (See utils.c.)
551 Fixed a defective typedef (affecting small-file
554 ------------------------------------------------------------------------
559 http://www.gnu.org/directory/wget.html
560 http://ftp.gnu.org/gnu/wget/
561 ftp://ftp.gnu.org/gnu/wget/
563 ------------------------------------------------------------------------
568 Extract the files from the distribution kit. The kit includes
569 objects and executables for Alpha, IA64, and VAX, in the corresponding
570 host-specific subdirectories. These builds were done with options
571 (explained below) "DASHD", "LARGE" (except VAX), and "NOSSL".
573 The command procedure VMS-WGET.COM attempts to determine the CPU
574 type, the IP package, and SSL availability. It defines the appropriate
575 macros for MMK or MMS, and then builds the appropriate executable from
578 SET DEFAULT [.WGET-1_10_2C_VMS.VMS] ! For convenience.
579 @ VMS-WGET CLEAN ! Deletes all object, list, map, and executable
580 ! files for the current CPU type.
581 @ VMS-WGET ! Copies CONFIG.H_VMS to CONFIG.H, compiles the
582 ! source files, and links the executable,
584 ! Add "DASHD" to this command line to enable the
585 ! Wget "-d" command-line option.
586 ! Add "DBG" to this command line to build a
588 ! Add "LARGE" to enable large-file support on
590 ! Add "LIST" if compiler listing and link map
592 ! Add "MMS" to use MMS instead of MMK when both
594 ! Add "NOSSL" to inhibit use of any SSL package.
595 ! Add "HPSSL" to use the HP SSL package when
596 ! OpenSSL (the default choice) is also available.
597 @ VMS-WGET HELP ! Creates a help library. (Optional. See
600 Set the symbol for the foreign command:
602 WGET :== $ actual_device:[actual.directory]WGET.EXE
604 A very basic VMS HELP file is supplied in [.DOC]WGET.HLP. To be
605 used, it must be added to a HELP library, either an existing library or
606 one just for Wget. As mentioned above, a new Wget help library may be
607 created using the command, "@ VMS-WGET HELP".
609 When a new HELP library is created, a logical name "HLP$LIBRARY_n"
610 may be defined to allow the HELP utility to find it. See "HELP HELP
611 /USERLIBRARY" for details.
613 Try to read the original documentation to learn how to use Wget. The
614 principal differences in this VMS adaptation are:
616 "-b" ("background") has no effect. (It uses "fork()".)
618 Excess dots are replaced by underscores in URL-derived file
619 specifications on ODS2 file systems. For example, on a UNIX system, the
620 URL "ftp://ftp.anywhere.org/x.y.z/a.b.c" could produce the local file
621 "ftp.anywhere.org/x.y.z/a.b.c". With this VMS adaptation, the local
622 ODS2 file would be "[.FTP_ANYWHERE_ORG.X_Y_Z]A_B.C". On an ODS5 file
623 system, the file would be [.ftp^.anywhere^.org.x^.y^.z]a^.b.c".
625 ------------------------------------------------------------------------
627 The accompanying source kit may suffer from storage on an ODS2 file
628 system, which does not preserve case or allow multiple dots in a file
629 name. Building this kit should work on VMS, but it may be expected to
630 fail on other systems. To use this kit on a non-VMS system, the files
631 may need to be renamed to restore their original mixed-case and/or
632 multi-dot names. The UNIX shell script "vms_name_fix.sh" (or something
633 like it) should do the job.
635 ------------------------------------------------------------------------
637 Steven M. Schweda (+1) 651-699-9818 (voice)
638 382 South Warwick Street sms@antinode.org
639 Saint Paul MN 55105-2547