IBM Mainframe Forum

by **claudioz** » Tue Jan 08, 2013 4:26 pm

Hi all,
is there a way to filter/reformat input (e.g. syslog) with multiline messages as if they were single lines?

for example, IEC205I has variable length messages (I have some ones with two and others with a set of 5 lines); is there a way to interpret the "continuation code" ("126" and "791" in the example) that is added at the end of the first line and on the beginning of the others?
I found ways to manage multiline messages if they have a fixed number of lines of if the lines are grouped by an header and a trailer but still nothing in this case.
Thanks a lot!

Follows an example of my input:

Select all

J0014673 00000090 IEC205I OUTFIL01,GCCJ0101,S020,FILESEQ=1, COMPLETE VOLUME LIST, 674
674 00000090 DSN=IBN0P.GCCF0001.STORICO.DN.G1695V00,VOLS=123731,TOTALBLOCKS=1768
J0014668 00000290 -S010 P001 00 384 59 .10 .00 .1
128882 WBATCH 0 0 0 0
J0230314 00000090 IEC205I OUT3,FC165104,S011,FILESEQ=1, COMPLETE VOLUME LIST, 126
126 00000090 DSN=TMP.T3592.CRY.PR13.DS,VOLS=600919,
126 00000090 LISTED VOL(S) HAVE BEEN DATA ENCRYPTED,KL1CD:L,KL2CD:L,
126 00000090 KL1=MY.TKLM.TAPES,KL2=MY.TKLM.TAPES,
126 00000090 TOTALBLOCKS=1 *HAS
J0230445 00000090 IEC205I SYSCPY1,NSEJBK99,P010,FILESEQ=7, COMPLETE VOLUME LIST, 791
791 00000090 DSN=D2SBKP0P.BNSE0.USFFPR.D130107.RF38WG.DN,VOLS=127050,
791 00000090 TOTALBLOCKS=356 *H
J0014673 00000281 IEF404I GCCJ0101 - ENDED - TIME=06.15.23
J0014673 00000290 -GCCJ0101 ENDED. NAME-GEST.I-O TOTAL TCB CPU TIME=
.00 TOTAL ELAPSED TIME= .0

ps I just tried with "IFTHEN=(WHEN=GROUP ... RECORDS=5", then RESIZE ICETOOL command, as in "Create large records from small records" from "Smart DFSORT Tricks", but this is not the correct solution in my case. It may works, as I tested, but it is not the solution for this type of messages.

by **claudioz** » Tue Jan 08, 2013 5:38 pm

I am just adding a note: a partial work is done with the use of BEGIN END on the first column field of the syslog, that I was not looking at! It delimits the message.
OK, I just understand how to isolate relevant messages from the others.
Anyway, I am not understanding how to read one message as if it were "one line", i.e. with the usual column or field position of dfsort.

In the syslog I see that you can watch at 'M' for the start of a multiline message, and the 'E' for the end...

Select all

S 2752 WBATCH 0 0 0 0
M 0020000 MVSP0 2013007 12:16:23.21 J0230445 00000090 IEC205I SYSCPY1,NSEJBK99,P010,FILESEQ=7, COMPLETE VOLUME LIST, 791
D 791 00000090 DSN=D2SBKP0P.BNSE0.USFFPR.D130107.RF38WG.DN,VOLS=127050,
E 791 00000090 TOTALBLOCKS=356 *H
N 0020000 MVSP0 2013007 12:16:23.24 S0140163 00000281 IGD103I SMS ALLOCATED TO DDNAME SYS12164

by **dick scherrer** » Tue Jan 08, 2013 10:26 pm

Hello and welcome to the forum,

You will need to "stack" (read about PUSH) the records and then format and write the new output as you want it.

You will need some way to define a "group". Is the number of lines per group always the same?

It will help if you show the desired output from the input you posted - and additional samples as appropriate.

by **skolusu** » Tue Jan 08, 2013 11:04 pm

claudioz wrote:Hi all,
is there a way to filter/reformat input (e.g. syslog) with multiline messages as if they were single lines?

for example, IEC205I has variable length messages (I have some ones with two and others with a set of 5 lines); is there a way to interpret the "continuation code" ("126" and "791" in the example) that is added at the end of the first line and on the beginning of the others?
I found ways to manage multiline messages if they have a fixed number of lines of if the lines are grouped by an header and a trailer but still nothing in this case.
Thanks a lot!
ps I just tried with "IFTHEN=(WHEN=GROUP ... RECORDS=5", then RESIZE ICETOOL command, as in "Create large records from small records" from "Smart DFSORT Tricks", but this is not the correct solution in my case. It may works, as I tested, but it is not the solution for this type of messages.

claudioz,

It is quite easy. if you look at the syslog , you will notice that the continuation messages doesn't have a time stamp value and it is spaces. So use that value for Group to Push the string IEC205I. Use the following DFSORT JCL which will give you the desired results. I assumed your input is FBA and LRECL=133. The timestamp field starts at pos 27 for a length of 11 bytes and the IEC205I message starts at pos 58 for a length of 8 bytes.

Select all

//STEP0100 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN DD DISP=SHR,DSN=Your input FBA 133 byte SYSLOG dataset
//SORTOUT DD SYSOUT=*
//SYSIN DD *
OPTION COPY
INREC IFTHEN=(WHEN=GROUP,BEGIN=(27,11,CH,NE,C' '),PUSH=(134:58,8))
OUTFIL BUILD=(1,133),INCLUDE=(134,8,CH,EQ,C'IEC205I')
//*

by **claudioz** » Wed Jan 09, 2013 5:38 pm

Well... as in my previous post, I'll found how to group that records, even if the number of lines per group is not fixed, thanks to the mark in the first column.
I've just used WHEN=GROUP with begin/end limits, instead of the number of records
BEGIN=(1,1,CH,EQ,C'M',AND,59,7,CH,EQ,C'IEC205I') and END=(1,1,CH,EQ,C'E'),
and identifing interesting records with PUSH=(134:ID=8,142:SEQ=1))

At this point, I need to rewrite all multiline IEC205I messages as if they were single line messages (with a big LRECL), regardless of the number of rows they are composed of (ok, I will assume 5 as the maximum).

So, as an example, consider the following as a compacted syslog (a multi-line log):

Select all

----+----1----+----2----+----3----+----4----+----5
N 11:01:55.22 J0227540 I123421 FASDFSA
M 11:01:58.14 J0227280 IEC205I SYSUT2
E 929 DSN=TMP.FC1,TO
N 11:02:55.22 J0227542 I123421 FASDFSA
M 11:02:58.14 J0227290 IEC205I SYSUT2
D 929 DSN=TMP.FC2
D 929 LISTED VOL(S)
D 929 KL1=TKLM.TAPE0
E 929 TOTALBLOCKS=1
N 11:02:58.23 J0227290 IEF234E K 9207,
N 11:02:59.00 J0227291 IEF234E K 9205,

My goal is to rewrite multiline messages to produce a single-line-per-message log (keeping IEC205I messages only):

Select all

----+----1----+----2----+----3----+----4----+----5----+----6----+----7----+----8----+----9----+----0
M 11:01:58.14 J0227280 IEC205I SYSUT2 DSN=TMP.FC1,TO
M 11:02:58.14 J0227290 IEC205I SYSUT2 DSN=TMP.FC2 LISTED VOL(S) KL1=TKLM.TAPE0 TOTALBLOCKS=1

If the number of rows were fixed (i.e. alway 5) it will be easier, using a DFSORT RESIZE-TOLEN with the correct LRECL (multiple of the initial).
But in this case the PUSH or the SPLICE could be helpful?

Thanks for your suggestions.

by **skolusu** » Wed Jan 09, 2013 11:33 pm

claudioz,

You can use GROUP to push the records individually and then use sections to get 1 large record. Alternatively you can just strip the records and then use splice. I will show you the GROUP/SECTIONS option. I assumed your LRECL=133 and RECFM=FBA. I am showing you a way to club a max of 10 records.

Select all

//STEP0100 EXEC PGM=SORT
//SYSOUT DD SYSOUT=*
//SORTIN DD DISP=SHR,DSN=Your input FBA SYSLOG
//SORTOUT DD SYSOUT=*
//SYSIN DD *
OPTION COPY
INREC IFTHEN=(WHEN=INIT,OVERLAY=(150:1330X)),
IFTHEN=(WHEN=GROUP,BEGIN=(58,8,CH,EQ,C'IEC205I'),
END=(2,1,CH,EQ,C'E'),PUSH=(134:ID=8,SEQ=1)),

IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,1),
END=(2,1,CH,EQ,C'E'),PUSH=(0150:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,2),
END=(2,1,CH,EQ,C'E'),PUSH=(0283:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,3),
END=(2,1,CH,EQ,C'E'),PUSH=(0416:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,4),
END=(2,1,CH,EQ,C'E'),PUSH=(0549:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,5),
END=(2,1,CH,EQ,C'E'),PUSH=(0682:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,6),
END=(2,1,CH,EQ,C'E'),PUSH=(0815:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,7),
END=(2,1,CH,EQ,C'E'),PUSH=(0948:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,8),
END=(2,1,CH,EQ,C'E'),PUSH=(1081:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,1,ZD,EQ,9),
END=(2,1,CH,EQ,C'E'),PUSH=(1214:1,133)),
IFTHEN=(WHEN=(142,1,ZD,EQ,10),OVERLAY=(1347:1,133))

OUTFIL INCLUDE=(134,1,CH,GT,C' '),REMOVECC,NODETAIL,
SECTIONS=(134,8,TRAILER3=(150,1330))
//*

by **claudioz** » Thu Jan 10, 2013 9:41 pm

Hi skolusu,
Well done

the last code is what I was expecting.
Thanks for the work of collecting all the details and producing those statements!

by **skolusu** » Fri Jan 11, 2013 1:15 am

Claudioz,

Glad it worked for you. However I apologize for making a mistake. I showed you to handle 10 occurrences, however I only had 1 byte sequence number on the WHEN=GROUP. Sorry about that. The updated control cards are below.

Select all

//SYSIN DD *
OPTION COPY
INREC IFTHEN=(WHEN=INIT,OVERLAY=(150:1330X)),
IFTHEN=(WHEN=GROUP,BEGIN=(58,8,CH,EQ,C'IEC205I'),
END=(2,1,CH,EQ,C'E'),PUSH=(134:ID=8,SEQ=2)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,1),
END=(2,1,CH,EQ,C'E'),PUSH=(0150:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,2),
END=(2,1,CH,EQ,C'E'),PUSH=(0283:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,3),
END=(2,1,CH,EQ,C'E'),PUSH=(0416:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,4),
END=(2,1,CH,EQ,C'E'),PUSH=(0549:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,5),
END=(2,1,CH,EQ,C'E'),PUSH=(0682:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,6),
END=(2,1,CH,EQ,C'E'),PUSH=(0815:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,7),
END=(2,1,CH,EQ,C'E'),PUSH=(0948:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,8),
END=(2,1,CH,EQ,C'E'),PUSH=(1081:1,133)),
IFTHEN=(WHEN=GROUP,BEGIN=(142,2,ZD,EQ,9),
END=(2,1,CH,EQ,C'E'),PUSH=(1214:1,133)),
IFTHEN=(WHEN=(142,2,ZD,EQ,10),OVERLAY=(1347:1,133))

OUTFIL INCLUDE=(134,1,CH,GT,C' '),REMOVECC,NODETAIL,
SECTIONS=(134,8,TRAILER3=(150,1330))
//*

IBM Mainframe Forum

INCLUDE lines of variable multiline message IEC205I

INCLUDE lines of variable multiline message IEC205I

Re: INCLUDE lines of variable multiline message IEC205I

Re: INCLUDE lines of variable multiline message IEC205I

Re: INCLUDE lines of variable multiline message IEC205I

Re: INCLUDE lines of variable multiline message IEC205I

Re: INCLUDE lines of variable multiline message IEC205I

Re: INCLUDE lines of variable multiline message IEC205I

Re: INCLUDE lines of variable multiline message IEC205I