In our environment we use zip232 and unzip60 to archive zos/files in binary format on unix systems. if we have a need to restore files from this archive, we need a function to replace the target file or create a new file which differs in the first qualifiers. Because unzip60 doesn't has a function to restore the dataset with the same name, as it was created in the archive, we heve written a new function called makezosdsn with following commands:
command $LETDSN$ (let the orgin dsn from archive) means the datasetname from the arcive is also used for the MVS dataset name. Only backslash are changed to dot. sample: //ZIP EXEC PGM=UNZIP, // PARM='/-a -o dd:archive -d ''$LETDSN$'' '
command $MHQDSN$ (modify High Level Qualifiers) This command overwrites the High Level Qualifiers the rest of dataset qualifiers is used as in command $LETDSN$ sample: //UNZIP EXEC PGM=UNZIP, // PARM='&PARM01 -d ''$MHQDSN$.MYUSERID.UNZIPO'' in this example the first two qualifiers are replaced with the qualifiers "MYUSERID.UNZIPO'. if someone has a need to use it, i have attached the source of vmmvs.c The new function makezosdsn is called on line 308 and 509. The function itself is inserted on line 523. Attention: currently it works only in binary (-B). if using it with ascii (-a) the preallocated output DCB is changed to LRECL=1028. I'm currently finding out the reason and after it is solved; I'll put the fixed one on this thread. don't hestitate if you have questions to this function and drop me a note best regards Josef
LRECL=1028 typically means the logical record length (LRECL) was defaulted, plus the file is VB format so you need 4 more bytes for the RDW. Are you trying to use the z/OS USS port to manipulate MVS files? If so, you are illustrating why a proper z/OS MVS port is very different and absolutely required.
As mentioned in the other thread, the MVS-specific encoding for VB files is fundamentally broken.
Record data is are encoded without providing a record length or line termination.
Our proposal is a simple "VLEN" encoding with an eyecatcher at the beginning of the file data, and a 4-byte host order record length prefix before each record's data
This is a limitation of the original ZIP/UNZIP implementation, and is documented in the program source.
in our z/OS environment Infozip zip/unzip utilities are used only with z/OS MVS port. It's the fastest and easyest way to zip/unzip MVS files. Using unzip6.0 without my new function "makezosdsn" works fine on all record formats (variable, fixed) if the target dataset is preallocated. Without the new function "makezosdsn" we have found following problems on unzip of MVS-files: - if the dataset name in the archive is almost 44 characters long, we cannot append a qualifier with [-d exdir] option, because MVS-datasetname restrictions does'nt allow longer dataset names. to solve this problem, we need a solution, which allows to replace some qualifiers. - sometimes in dataset name in archive conains characters, which are in MVS datasetnames restricted. replace some qualifiers, would also help to solve this problem. - in some cases we need the dataset name from archive unchanged. E.G if we zip a file in binary on z/OS and need to recover the old corrupted file. unzip exchanges the last two qualifiers. to solve this, we need a option, which tells unzip let the datasetname unchanged from archive. So, if you can implement this requirements, our customers would be very glad.
Fits, The mapping of the last two sequential MVS dataset qualifiers, or last qualifier + member name to DOS-style names in ZIP does need to be fully reversable in UNZIP.
When generating the MVS file name from the encoded name, UNZIP does need to have a reliable mapping to the characters allowed by the MVS dataset syntax. This means 44 characters, max of 8 chars per qualifier, first char of qualifier A-Z, $, # or @, subsequent chars of qualifier A-Z, 0-9, $, # or @. PDS/PDSE member names are limited to 8 chars (with the same rules as the qualifier, but allow a syntax extension for GDG dataset relative generations).
The logic should be similar to that used by FTP, where each qualifier is treated as a directory. FTP normally treats the current user ID as the default directory, and I would suggest that unzip should at least have an option for this (or map a "/~" prefix to the current user ID). The -d exdir option should result in a prefix (not suffix) being added to the archive member name. There should be an option to ignore the original directory name from ZIP. There does indeed need to be code to recognize the case where the resulting name is too long, and options to allow the directory/member names to be mapped. There should not be the case that due to name encoding or character set (UTF8) you have member names that you can't retrieve.
Right now, we are restricted in how we use unzip in production. It needs a lot of work for MVS. We either restrict the ZIP files to having a single archive member, or only allow multiple members by extract one-by-one using -X. We would like to have a side file (just like zip) with regular expressions and the destination name (or DD) mapping.
To restrict the contents of the zip archive from controlling the file produced by unzip, we force output to stdout, and have a DD:SYSPRINT defined for stdout. It's a kludge, and we would prefer to use options and side files instead so we can actauly see unzip messages in case of an error.
Thankfully we run zip far more often than we run unzip in production.
as long we don't support record length on z/OS RECFM=VB/V and Option -B zip code should be changed to deny zip of files in this case. currently zip accepts files with RECFM=VB/V and Option -B and creates archive with return code=0. if we try to unzip this archive, we get a corrupted target file, because unzip don't have a record length to build the correct output file. in our zip31b version I've put following code fldata(stream,z->name,&fdata);
.... code put after Line 311
/* don't permit RECFM=VB/V and Option -B because zip currently */ /* doesn't store record length field on archive */ if (fdata.__recfmV & bflag) { printf(" FLDATA: RECFM=VB/V and Option -B not supported "); printf("file : %s not processed! ",z->name); z->name = " "; return ZE_NONE; }
... and befor Line 312
/*put the system ID */ in our environment this works fine, and it would be helful, if you can put it in the current zip source code, as long as the variabel record length problem isn't solved
As mentioned in the other thread, the MVS-specific encoding for VB files is fundamentally broken.
Record data is are encoded without providing a record length or line termination.
Our proposal is a simple "VLEN" encoding with an eyecatcher at the beginning of the file data, and a 4-byte host order record length prefix before each record's data
This is a limitation of the original ZIP/UNZIP implementation, and is documented in the program source.
Al, you are right, the best solution for supporting dynamic z/OS file name handling, would be to have a side file (just like zip). This side file should contain: 1.) specify a archive file full qualified or generic. 2.) specify the target z/OS filename full qualified or generic. 3.) specfiy the file attributes of the target z/os file 4.) specify the size attributes of z/OS file. full qualified example: BAT/DC/H/CASHFL.D091231 BAT.DC.H.CASHFL.D091231 RECFM=FB,LRECL=366,BLKSIZE=0 + CYL,PRI=300,SEC=300,BLKSIZ=0
more complex generic example: BAT/*/H/CASHFL.* BAT.*.T.CASHFL./*100625 RECFM=FB,LRECL=366,BLKSIZE=0 + CYL,PRI=300,SEC=300,BLKSIZ=0 if you can implement it in this manner, preallocation isn't more required. If a preallocated file is available and side file doesn't contain allocation attributs, we can use it again. I 'll look forward you can implement your very good idea in one of the next releases. Josef
Fits, The mapping of the last two sequential MVS dataset qualifiers, or last qualifier + member name to DOS-style names in ZIP does need to be fully reversable in UNZIP.
When generating the MVS file name from the encoded name, UNZIP does need to have a reliable mapping to the characters allowed by the MVS dataset syntax. This means 44 characters, max of 8 chars per qualifier, first char of qualifier A-Z, $, # or @, subsequent chars of qualifier A-Z, 0-9, $, # or @. PDS/PDSE member names are limited to 8 chars (with the same rules as the qualifier, but allow a syntax extension for GDG dataset relative generations).
The logic should be similar to that used by FTP, where each qualifier is treated as a directory. FTP normally treats the current user ID as the default directory, and I would suggest that unzip should at least have an option for this (or map a "/~" prefix to the current user ID). The -d exdir option should result in a prefix (not suffix) being added to the archive member name. There should be an option to ignore the original directory name from ZIP. There does indeed need to be code to recognize the case where the resulting name is too long, and options to allow the directory/member names to be mapped. There should not be the case that due to name encoding or character set (UTF8) you have member names that you can't retrieve.
Right now, we are restricted in how we use unzip in production. It needs a lot of work for MVS. We either restrict the ZIP files to having a single archive member, or only allow multiple members by extract one-by-one using -X. We would like to have a side file (just like zip) with regular expressions and the destination name (or DD) mapping.
To restrict the contents of the zip archive from controlling the file produced by unzip, we force output to stdout, and have a DD:SYSPRINT defined for stdout. It's a kludge, and we would prefer to use options and side files instead so we can actauly see unzip messages in case of an error.
Thankfully we run zip far more often than we run unzip in production.
Josef, Ed has requested that I hold off until he releases the ZIP 3.1c beta before trying to integrate my changes.
He's also suggested that I start separate topics for each discussion point (under ZIP or UNZIP, as appropriate), so they don't all get jammed together. To reply very briefly to each of your points: Re: Put out "%s not processed!" message when MVS-format RECFM=Vxxx ZIP 3.1b is the latest beta, and is being replaced by 3.1c... hopefully today. While you could ask Ed to put that message in 3.1c at the last minute, the best bet is to for me to work towards a 3.1d beta that works properly on z/OS MVS (without negatively affecting z/OS USS and CMS functionality). It makes sense to take the time to make that path work in 3.1d, rather than making it a fatal error. Re: ZIP side file for controlling name mapping, etc I'm not taling about an MVS-specific file with CLIST-style name continuation. I'm talking about something that adds function to ZIP in the general case, so that name mapping works in general. I was thinking about something along the lines of: infilename_pattern as(outfilename_pattern) Example 1: Save Universal Command ucopy utility stdout and stderr as ZIP members Given file 1 (allocated to DD:F1) containing the previous JCL job step's stdout, and file 2 (DD:F2) with the corresponding stderr: Side file would contain something like: DD:F1 as(ucopy.stdout) DD:F2 as(ucopy.stderr) If the file was empty, you would get a zero-length file for the corresponding zip member.
Example 2: Endevor source mapping
Our Endevor system stores the latest source in PDS libraries with the source type in the 2nd-last qualifier. I would like them to be stored as "member.filetype". It would be trivial using DDs: DD:CSRC(*) as(*.C) Using file names would be more difficult, something like this with -r active:
SL.DPT.PRJ.SRC.C.SOUT as(*.C) Hmmm... need to get far fancier with how to specify the pattern for the "encode as" name so that can relate to the input name pattern.
Re: Attributes in ZIP side file
It is trivial to check z/OS MVS (and USS) file attributes using fldata(). I don't see how adding that information in the ZIP control file would be required, or helpful. ZIP should be passing along the actual file information, and perhaps UNZIP can provide the kind of controls you are talking about (as options and perhaps within the UNZIP side file). By the way, while going over our changes to ZIP 2.32, I also noted that ZIP only captures the PDS member name, and totally ignores the ISPF-format information like last modification date/time that may be present. When I redo this for ZIP 3.1d, I need to add support for both the short and long (new in z/OS 1.11) forms. For sequential MVS files, there is only a date stamp in the dataset catalog entry (DSCB), but no time stamp. INot sure how to easily grab that in pure C - may need to make some dynalloc calls. By the way, the main ZIP/UNZIP developers indicated that they don't mind bringing in relevant ideas from other compression utilities. I think we need to check out what products like "PKZIP/SecureZIP (tm) for z/OS" do in these areas and discuss whether/how they might be applicable to ZIP and UNZIP. Lots of stuff to do... need to take care and not code ourselves into corners.
Al, currently zip232 to 31b stores stores dcb attributes like recfm, lrecl into zip archive. But the the size attributes like CYL,TRKS,BLKS, PRIMARY=,SECONDARY= quantity are not stored. Let assume we have stored this information also, we don't need preallocate the target dataset in the unzip job. if we try to unzip a file, which wasn't created on a z/OS plattform, we don't have this information on zip archive and need a way to set this information in the masking rules. E.G. our customers don't use preallocated datasets in ftp jobs, they use site and lsite commands to allocate new datasets. So they would it also prefer it in the same manner in zip/unzip utilties. regards, Josef <Re: Attributes in ZIP side file It is trivial to check z/OS MVS (and USS) file attributes using fldata(). I don't see how adding that information in the ZIP control file would be required, or helpful. >>
Josef, Ed has requested that I hold off until he releases the ZIP 3.1c beta before trying to integrate my changes.
He's also suggested that I start separate topics for each discussion point (under ZIP or UNZIP, as appropriate), so they don't all get jammed together. To reply very briefly to each of your points: Re: Put out "%s not processed!" message when MVS-format RECFM=Vxxx ZIP 3.1b is the latest beta, and is being replaced by 3.1c... hopefully today. While you could ask Ed to put that message in 3.1c at the last minute, the best bet is to for me to work towards a 3.1d beta that works properly on z/OS MVS (without negatively affecting z/OS USS and CMS functionality). It makes sense to take the time to make that path work in 3.1d, rather than making it a fatal error. Re: ZIP side file for controlling name mapping, etc I'm not taling about an MVS-specific file with CLIST-style name continuation. I'm talking about something that adds function to ZIP in the general case, so that name mapping works in general. I was thinking about something along the lines of: infilename_pattern as(outfilename_pattern) Example 1: Save Universal Command ucopy utility stdout and stderr as ZIP members Given file 1 (allocated to DD:F1) containing the previous JCL job step's stdout, and file 2 (DD:F2) with the corresponding stderr: Side file would contain something like: DD:F1 as(ucopy.stdout) DD:F2 as(ucopy.stderr) If the file was empty, you would get a zero-length file for the corresponding zip member.
Example 2: Endevor source mapping
Our Endevor system stores the latest source in PDS libraries with the source type in the 2nd-last qualifier. I would like them to be stored as "member.filetype". It would be trivial using DDs: DD:CSRC(*) as(*.C) Using file names would be more difficult, something like this with -r active:
SL.DPT.PRJ.SRC.C.SOUT as(*.C) Hmmm... need to get far fancier with how to specify the pattern for the "encode as" name so that can relate to the input name pattern.
Re: Attributes in ZIP side file
It is trivial to check z/OS MVS (and USS) file attributes using fldata(). I don't see how adding that information in the ZIP control file would be required, or helpful. ZIP should be passing along the actual file information, and perhaps UNZIP can provide the kind of controls you are talking about (as options and perhaps within the UNZIP side file). By the way, while going over our changes to ZIP 2.32, I also noted that ZIP only captures the PDS member name, and totally ignores the ISPF-format information like last modification date/time that may be present. When I redo this for ZIP 3.1d, I need to add support for both the short and long (new in z/OS 1.11) forms. For sequential MVS files, there is only a date stamp in the dataset catalog entry (DSCB), but no time stamp. INot sure how to easily grab that in pure C - may need to make some dynalloc calls. By the way, the main ZIP/UNZIP developers indicated that they don't mind bringing in relevant ideas from other compression utilities. I think we need to check out what products like "PKZIP/SecureZIP (tm) for z/OS" do in these areas and discuss whether/how they might be applicable to ZIP and UNZIP. Lots of stuff to do... need to take care and not code ourselves into corners.
One thing to remember is that the current ZIP/UNZIP code barely tolerates zOS MVS, as an afterthought to z/OS UNIX support.
They do not use any of the MVS-specific services easily accessible from C, especially Dynamic allocation. I have some familliarity with those from my primary job, and I'm exploring more as I expand my primary project's capabilities. Making zip and unzip do reasonable things, with or without new keywords is going to be an ongoing process over multiple releases and time. The more we can do this while creating general facilities that are also usable on other ports, the better.
As far as space allocation in unzip, one can calculate the approximate space required in each new destination dataset. When output is directed to PDS/PDSE datasets there would need to be some look-ahead to ensure that all members can be accomodated. Remember that one can grow zip archives with multiple zip invocations, and if one drops directory mapping in zip or remaps the destination in unzip the output dataset sizes would become quite different than those at input. There certainly does need to be a way to specify some system-specific keywords, but this needs to be kept sane.
We have to take care to use z/OS facilities in a way that map well for zip/unzip. I don't see a need for CYLS/TRACKS/BLOCKS and messing around with fragile DASD geometry calculations. Dynamic allocation also supports allocation in KB, MB, and GB units, and those are easy to use.
We need a syntax to have unzip create new PDS/PDSE libaries (equivalent to zip's -r), but I would likely default to generating PDSEs rather than PDSs as this removes the directory size calculation.
I do intend to support extracting the ISPF last update member timestamps during zip (both usual and new large z/OS 1.11 formats), but we would need more attributes to also store creation timestamps and TSO user ID. Both are useful historical information, and for those folks using zip in an archival role would be very worthwhile for a tiny bit of additional complexity. I need to get into the code more, but should be able to use existing common UNIX-type zip attributes for most or all of this. Al
Al, my experience on UNZIP60 was B37 Abends, if a target dataset was'nt preallocated. So if the newest version of UNZIP does correct calulate the space from the archive file, you are right, there is no need to tell the utility the correct space requirment. Do you think this problem is in the current UNZIP60F version solved ? Josef
Josef, That code does not yet exist, either in the current unzip or my version. You've identified the need, and I certainly agree it is a reasonable requirement.
Ed has requested that we start some new topics to discuss the MVS port. I'm on vacation this week, but will set some up next week.
The priorities as I see them are to first focus on getting the changes I have into the latest source base, and then fixing known problems (there are still some situations where grow doesn't work). Next would be getting the attributes working so VB is reliable, and then on to new stuff where new code and options are required.
As I'm currently doing more with UnZip lately, I'm starting to read the UnZip forum threads more. Before now I have tended to focus on the Zip threads and the Zip code. So I have some catching up to do on the UnZip issues. That said ...
It's possible to store extra information about archive members in the extra fields block. If you find there's some meta information that you need stored by Zip and restored by UnZip, we can set up a new extra field that Zip can use to store the information. UnZip can then look at this extra field stored with the file in the archive and pull the information.
The AppNote defines the following related extra field: -MVS Extra Field (0x0065):
The following is the layout of the MVS "extra" block. Note: Some fields are stored in Big Endian format. All text is in EBCDIC format unless otherwise specified.
Value Size Description ----- ---- ----------- (MVS) 0x0065 2 bytes Tag for this "extra" block type TSize 2 bytes Size for the following data block ID 4 bytes EBCDIC "Z390" 0xE9F3F9F0 or "T4MV" for TargetFour
(var) TSize-4 Attribute data (see APPENDIX B)
If there's extra information you need to store, I recommend starting by creating a list of the information you need. Probably it will be cleanest to design a new extra field rather than work with the above old extra field. Don't worry about the implementation details, just the data you need stored, assuming you need to store data about the entry. We can work the details.
We have been planning to add a generic renaming capability to either Zip or UnZip or both. The various possibilities are daunting so this has been on the back burner for some time as we think about how to specify renaming patterns for various entries from possibly different ports. This capability should work for all ports.
No doubt we know things that you all don't that might be helpful to you. When you run into something sticky let us know and we might have something that could move things forward faster. That said, you all have the hardware and knowledge about z/OS and MVS and need to help us there. We, on the other hand, have some history and some idea of how the code is supposed to work. Keeping the details going back and forth sounds like a very good idea as you all develop some updates.
Would appreciate references to on-line documentation relevant to specific design decisions as development proceeds. That way we can get at least a little knowledgeable about what is being implemented.
I'll try to start following these UnZip threads more. Thanks for your help!
I'm on vacation this week. As I've previously mentioned after I get back I'm going to follow your suggestion to create create new topics for each of the separate areas of zip/unzip related to z/OS USS/MVS and in some cases CMS. It sounds like we need to do the same for some of the discussions that are currently spread out through various forums. Perhaps we need a separate area to discuss common new function, as these are not zip/unzip/zipnote specific?
Starting next week I will be focussing about a quarter of my work time into the immediate effort of integrating my code with the latest zip and unzip, so the z/OS MVS users can have minimal function with the new 64-bit archive support equivalent to that I have in our port of the older releases. Using the z/OS USS port can only provide limited support for the MVS files, because the UNIX-style file library functions simply have minimal support for MVS files.
We've already agreed on how I should use the VMS-style attribute and per-line record prefixes as a first pass to complete the existing support for variable-format files. That is using the MVS extension that you mentioned, simply completing what is currenly there as-is for now makes sense.
The next priority for the MVS port is to provide checking and some simple mapping in unzip so that the output file names conform to the MVS dataset and member name rules. A simple truncation-style first pass mapping can get the job done for now, and we can architect common file selection and output name mapping support down the road.
There is a lot that can be done to seriously enhance capabilities and performance on z/OS for both USS and MVS. Two high priority areas: - Building with xplink and finally 64-bit to use the existing power of the z/Series processors (z/OS is now at core running 64-bit) to eliminate long long emulation in 32-bit code. - Adding a simple option to the file open for input files (indicating file data will not be updated) results in file reads being accz/OS USSellerated by a factor of more than two.
I'm very interested in being part of the discussion for architecting new character translation support as well as input and output file mapping. These are very high profile on the z/OS (both MVS and USS) and CMS ports. I would request that we take care not to get too far into the actual implementation before I can get the basics up to speed for the mainframe ports.
Logged
