IBM Mainframe Forum

Mainframe Technical Support Forums

Skip to content


Reformat JES job log records



IBM's flagship sort product DFSORT for sorting, merging, copying, data manipulation and reporting. Includes ICETOOL and ICEGENER
Post a reply

Reformat JES job log records

Postby alexm » Wed Oct 13, 2010 8:31 pm

Good morning,
May I please ask if it's possible to do the following using DFSORT/ICETOOL - I've tried for hours now but didn't get a solution till now.

We have the output records from a JES2 started task (i.e. a DB2 subsystem). It contains the messages that the subsystem wanted to write to the JES job log. Such messages can be one line messages, or they can be split into multiple records, by this adding a sequence number as the last word in a record, and the same sequence number as the first word in the continued record. This is how it might look like:

Select all
****** ***************************** Top of Data ******************************
=COLS> ----+----1----+----2----+----3----+----4----+----5----+----6----+----7
000001 1                     J E S 2  J O B  L O G  --  S Y S T E M  T X A 1   
000002 0                                                                       
000003  05.13.42 S0000120 ---- WEDNESDAY, 29 SEP 2010 ----                     
000004  05.13.42 S0000120  IEF695I START DSN1MSTR WITH JOBNAME DSN1MSTR       
000005  05.13.50 S0000120  S DSN1DBM1                                         
000006  05.46.06 S0000120  DSNT501I  +DSN1 DSNB1LDA RESOURCE UNAVAILABLE  235 
000007     235                        CORRELATION-ID=010.TLPLKNC3             
000008     235                        CONNECTION-ID=DSN1                       
000009     235                        LUW-ID=*                                 
000010     235                        REASON 00C200EA                         
000011     235                        TYPE 00000200                           
000012     235                        NAME A125XMDE.TDTCOD2                   
000013  18.03.57 S0000120  DSNT500I  +DSN1 DSNXIDPM RESOURCE UNAVAILABLE  004 
000014     004                        REASON 00E70010                         
000015     004                        TYPE 00000801                           
000016     004                        NAME SAPCL.DB2CLDB.18D0E54106A5C1DF     
000017  00.00.00 S0000120 ---- THURSDAY,  30 SEP 2010 ----                     
000018  05.05.22 S0000120  DSNJ138I  +DSN1 DSNJOFF1 OFFLOAD COMPLETE FOR  182 
000019     182             LOG TRUNCATION FROM AN ARCHIVE LOG COMMAND         
000020  05.05.22 S0000120  DSNJ139I  +DSN1 LOG OFFLOAD TASK ENDED             
000021  05.14.01 S0082560  DSNL512I  +DSN1 DSNLILNR TCP/IP  977               
000022     977             GETHOSTBYADDR(nn.nn.nn.nn7) FAILED WITH               
000023     977                        RETURN CODE=1 AND REASONCODE=00000000   
000024  23.14.01 S0082560  DSNL004I  +ED01 DDF START COMPLETE                 
****** **************************** Bottom of Data ****************************
(Please ignore the 'top of data' and the 'COLS>' record, the record sequence numbers, and the 'bottom of data' record).

What I do need is the following:
  • Search that message file for a specific string, e.g. the string 'REASON'. (in the sample above, you'll find it on lines 10, 14, and 23).
  • If that string has been found, extract the entire message.
  • If the message has continuation indicators (these three digit numbers), remove the numbers from the end of the first message and all continuation records and combine all these parts to one message line.
  • Remove all but one space between all words of a resulting message line
  • The dataset can contain messages for more than one weekday (see lines 3 and 17). These dates should be included in the output.
A sample of the generated output for lines 13-16 and 21-23 should look like:

29 SEP 2010 18.03.57 S0000120 DSNT500I +DSN1 DSNXIDPM RESOURCE UNAVAILABLE REASON 00E70010 TYPE 00000801 NAME SAPCL.DB2CLDB.18D0E54106A5C1DF
30 SEP 2010 05.14.01 S0082560 DSNL512I +DSN1 DSNLILNR TCP/IP GETHOSTBYADDR(nn.nn.nn.nn) FAILED WITH RETURN CODE=1 AND REASONCODE=00000000

Known specs:
  • Date messages can be identified by four dashes in record position 20, the date without leading weekday starts at pos 36.
  • A message starts with a time (hours) in pos 2-3 and may or may not have a continuation indication (digits)
  • A continued message shows two spaces in record pos 2-3, and then 3 digits
  • Input dataset is RECFM=VBA, LRECL=134, the max output record length should be (truncated to) 32760.

Some more background: Actually, I do have a REXX program doing exactly what I want. We do have an ISPF application where users can select various input file(s) and specify a search term (e.g. REASON). These files are then read one by one, and messages are massaged to long single line messages. All the resulting messages from all selected input files are then combined into one output dataset and made available for ISPF browse.
However, I have the problem that we get these 'machine storage exhausted' messages when processing large input datasets. To find a solution, I thought of SORT to exclude messages that are not relevant, and this is where my current odyssee started. When I recognized all these capabilities of sort, I've started reading tons of paperwork and tried and tried..., but none of my attempts has given me a solution and I'm completely lost now.
I know that my question is not simple - more like writing a application - but some of you specialists might want to give me a hint, or two...?

Thanks in advance!
Alex
User avatar
alexm
 
Posts: 35
Joined: Wed Oct 13, 2010 6:40 pm
Has thanked: 0 time
Been thanked: 0 time
Top

Re: Reformat JES job log records

Postby NicC » Wed Oct 13, 2010 10:37 pm

Your 'machine stoage exhausted' messages should not appear if you are holding only the minimum records necessary - i.e. reading one line at a time and write your created lines as soon as they are complete. If you are reading the file into a stem then that is your problem on large files although I have had a rexx program with 2 30k record stems in it with no problem (recsize about 200 bytes)
The problem I have is that people can explain things quickly but I can only comprehend slowly.
Regards
Nic
NicC
Global moderator
 
Posts: 3025
Joined: Sun Jul 04, 2010 12:13 am
Location: Pushing up the daisies (almost)
Has thanked: 4 times
Been thanked: 136 times
Top

Re: Reformat JES job log records

Postby Frank Yaeger » Wed Oct 13, 2010 10:51 pm

If I were doing this with DFSORT, I would start by using WHEN=GROUP to assign an id to each group of message records. Then I would create one file with records containing the id for each message group that has 'REASON' and another file with all of the message groups. I would then use JOINKEYS to select the relevant message groups for output. But the difficult part would be combining the pieces of the message from the different lines, given that we don't know the maximum number of lines a message could have. Assuming some maximum, you could use SPLICE or IFTHEN to combine the message lines. You might also need SQZ to format them correctly.

But this is a complex application, so it would take some doing to get it right.
Frank Yaeger - DFSORT Development Team (IBM) - yaeger@us.ibm.com
Specialties: JOINKEYS, FINDREP, WHEN=GROUP, ICETOOL, Symbols, Migration
=> DFSORT/MVS is on the Web at http://www.ibm.com/storage/dfsort
User avatar
Frank Yaeger
Global moderator
 
Posts: 1079
Joined: Sat Jun 09, 2007 8:44 pm
Has thanked: 0 time
Been thanked: 15 times
Top

Re: Reformat JES job log records

Postby alexm » Wed Oct 13, 2010 11:04 pm

Nic,
thank you for your fast response. My file contains 1,567,555 records, the file size is 188 CYLS (the reason for that file size is that the task is active for weeks). From other forums and other sources for information I've learned that using REXX EXECIO is not a good choice to read large files; in fact, I've read various statements that recommend SORT for performance reasons, and because remodelling of data would be possible by sort. And that's why I started investigating in sort. The question stays - is it possible that sort can do the job?
There's another thread (dfsort-icetool-icegener/topic2846.html ) with a similar request. I used this as a base, but I can't find out how to strip these continuation sequence numbers from my records.

Frank,
also thank you for your suggestions. I will need some time to understand your thougths and then try to play around with all these keywords you mentioned...
User avatar
alexm
 
Posts: 35
Joined: Wed Oct 13, 2010 6:40 pm
Has thanked: 0 time
Been thanked: 0 time
Top

Re: Reformat JES job log records

Postby alexm » Thu Nov 04, 2010 3:18 pm

Thanks for your help.
Various reasons have caused me to re-code my ISPF application to do single line processing of the JESMSGLG datasets (instead of passing the entire dataset to SORT and then process the resulting records only). One of the reasons for re-coding was that even SORT returned a S878 abend (region size too small - TSO foreground processing). However, I've tried, and I've created a sort job as shown below. It does not exactly what I required, but at least it would be possible to postprocess the sort's output records using REXX. The only problem that is left over now is to remove that message line continuation sequence number that appears at the end of a first message line.
Select all
****** ***************************** Top of Data ******************************
000001 //JOBNAME      JOB      ....                                           
000002 //*--------------------------------------------------------------------
000003 //* INPUT:      //JESMSGLG DATASET RECFM=FBA LRECL=134                 
000004 //* ASSUMPTION: A SINGLE MESSAGE CONSISTS OF AT MOST 20 RECORDS         
000005 //* SELECT:     ALL MESSAGES THAT CONTAIN THE STRING 'REASON'           
000006 //*--------------------------------------------------------------------
000007 //ST1          EXEC     PGM=SORT                                       
000008 //SYSOUT       DD       SYSOUT=*                                       
000009 //SORTIN       DD       DISP=SHR,DSN=&SYSUID..JESMSGLG.IN.DATA         
000010 //SORTOUT      DD       DISP=(NEW,CATLG,CATLG),                         
000011 //             DSN=&SYSUID..JESMSGLG.OUT.DATA,                         
000012 //             LRECL=5004,BLKSIZE=5008,RECFM=VB,                       
000013 //             SPACE=(TRK,(15,5))                                       
000014 //SYSIN        DD       *                                               
000015  SORT FIELDS=COPY                                                       
000016  INREC IFOUTLEN=5008,                                                   
000017    IFTHEN=(WHEN=INIT,BUILD=(5012:1,134)),                               
000018    IFTHEN=(WHEN=GROUP,BEGIN=(5013,1,CH,NE,C' '),                       
000019                       PUSH=(5001:ID=8,SEQ=3)),                         
000020    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,01),PUSH=(00001:5012,134)),   
000021    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,02),                         
000022                         END=(5009,3,ZD,EQ,01),PUSH=(00135:5032,114)),   
000023    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,03),                         
000024                         END=(5009,3,ZD,EQ,01),PUSH=(00269:5032,114)),   
000025    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,04),                         
000026                         END=(5009,3,ZD,EQ,01),PUSH=(00403:5032,114)),   
000027    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,05),                         
000028                         END=(5009,3,ZD,EQ,01),PUSH=(00537:5032,114)),   
000029    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,06),                         
000030                         END=(5009,3,ZD,EQ,01),PUSH=(00671:5032,114)),   
000031    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,07),                         
000032                         END=(5009,3,ZD,EQ,01),PUSH=(00805:5032,114)),   
000033    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,08),                         
000034                         END=(5009,3,ZD,EQ,01),PUSH=(00939:5032,114)),   
000035    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,09),                         
000036                         END=(5009,3,ZD,EQ,01),PUSH=(01073:5032,114)),   
000037    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,10),                         
000038                         END=(5009,3,ZD,EQ,01),PUSH=(01207:5032,114)),   
000039    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,11),                         
000040                         END=(5009,3,ZD,EQ,01),PUSH=(01341:5032,114)),   
000041    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,12),                         
000042                         END=(5009,3,ZD,EQ,01),PUSH=(01475:5032,114)),   
000043    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,13),                         
000044                         END=(5009,3,ZD,EQ,01),PUSH=(01609:5032,114)),   
000045    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,14),                         
000046                         END=(5009,3,ZD,EQ,01),PUSH=(01743:5032,114)),   
000047    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,15),                         
000048                         END=(5009,3,ZD,EQ,01),PUSH=(01877:5032,114)),   
000049    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,16),                         
000050                         END=(5009,3,ZD,EQ,01),PUSH=(02011:5032,114)),   
000051    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,17),                         
000052                         END=(5009,3,ZD,EQ,01),PUSH=(02145:5032,114)),   
000053    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,18),                         
000054                         END=(5009,3,ZD,EQ,01),PUSH=(02279:5032,114)),   
000055    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,19),                         
000056                         END=(5009,3,ZD,EQ,01),PUSH=(02413:5032,114)),   
000057    IFTHEN=(WHEN=GROUP,BEGIN=(5009,3,ZD,EQ,20),                         
000058                         END=(5009,3,ZD,EQ,01),PUSH=(02547:5032,114)),   
000059    IFTHEN=(WHEN=(5009,3,ZD,EQ,01),                                     
000060            OVERLAY=(135:4095X,771X))                                   
000061                                                                         
000062  OUTFIL REMOVECC,                                                       
000063    NODETAIL,                                                           
000064    BUILD=(1,5000),                                                     
000065    SECTIONS=(5001,8,                                                   
000066      TRAILER3=(00001,256,00257,256,00513,256,00769,256,                 
000067                01025,256,01281,256,01537,256,01793,256,                 
000068                02049,256,02305,256,02561,256,02817,256,                 
000069                03073,256,03329,256,03585,256,03841,256,                 
000070                04097,256,04353,256,04609,256,04865,135)),               
000071    INCLUDE=(1,5000,SS,EQ,C'REASON',OR,                                 
000072             19,6,SS,EQ,C' ---- '),                                     
000073    FTOV,VLTRIM=C' '                                                     
****** **************************** Bottom of Data ****************************

As I said, I've re-coded the application, and I do not need any further help on this.
Regards, Alex
User avatar
alexm
 
Posts: 35
Joined: Wed Oct 13, 2010 6:40 pm
Has thanked: 0 time
Been thanked: 0 time
Top
Post a reply

Return to DFSORT/ICETOOL/ICEGENER

 


  • Related topics
    Replies
    Views
    Last post
Forum Rules|FAQ|Subscriptions|Bookmarks