IBM Mainframe Forum

by **Aki88** » Sun Jun 28, 2015 2:14 am

Hello,

We have a batch COBOL program with internal sort; the job processes a fairly large amount of data almost everyday, test run file was a replica with 65% of actual record count; count standing at: 202429599 records. We were trying to benchmark the SORTWork area to be provided to DFSORT; SYSOUTs are given below for reference:

Select all

With 12 SORTWork datasets of:

//SORTWK01 DD DSN=&&TEMP,DSNTYPE=LARGE,
// DISP=(NEW,DELETE,DELETE),
// SPACE=(CYL,(1000,1000),RLSE)

SYSOUT -

1ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE118I 0 UNKNOWN FILE SIZE - FILSZ=EN MAY IMPROVE RESOURCE USAGE AND PERFORMANCE
ICE751I 0 C5-I12416 C6-K90026 C7-K94453 C8-K94453 E4-K58148 C9-BASE E5-K80744 E7-I12416
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 14:03 ON MON JUN 22, 2015 -
0 OPTION MOSIZE=10%,HIPRMAX=10%,DYNALLOC=(SYSDA,15),DSPSIZE=300 -HP13512
EXPMAX=10%,EXPOLD=10%,EXPRES=10% HP13512
ICE140I 0 END OF PARAMETERS FROM DFSPARM - SYSIN OR SORTCNTL/PARAMETER LIST CONTROL STATEMENTS FOLLOW
SORT FIELDS=(0001,0023,CH,A)
RECORD TYPE=F,LENGTH=(000111,,)
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE089I 1 TEST . , INPUT LRECL = 111, TYPE = F
ICE093I 0 MAIN STORAGE = (MAX,6291456,6291456)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (6234096,6234096)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=6291456,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERET=RC16 ,MSGDDN=SYSOUT
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=N ,ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=0
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=1406 ,DSPSIZE=300 ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=1407
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y
ICE750I 0 DC 0 TC 0 CS DSVVV KSZ 23 VSZ 23
ICE752I 0 FSZ=0 RE IGN=0 C AVG=112 0 WSP=0 E DYN=0 0
ICE247I 0 INTERMEDIATE MERGE ENTERED - PERFORMANCE MAY BE DEGRADED
(PGMSTAT) SCSMCU - SMCU - OPENED: SEQ INPUT 393 READS, 0 WRITES, 0 REWRITES, 0 DELETES
ICE751I 1 DE-K83743 D5-K91600 D3-K83080 E1-I16099 D6-K58148 D3-K83080 D7-K58148 E8-I12416
ICE091I 0 OUTPUT LRECL = 111, TYPE = F
ICE055I 0 INSERT 202429599, DELETE 202429599
ICE054I 0 RECORDS - IN: 0, OUT: 0
ICE134I 0 NUMBER OF BYTES SORTED: 22469685489
ICE253I 0 RECORDS SORTED - PROCESSED: 202429599, EXPECTED: 0
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 510000 , TRACKS USED: 507435
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 1407M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE052I 0 END OF DFSORT

Select all

Without SORTWork datasets, but additional DFSORT parameters coded:

1ICE060I 0 INVALID DFSPARM PARAMETER
ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE751I 0 C5-I12416 C6-K90026 C7-K94453 C8-K94453 E4-K58148 C9-BASE E5-K80744 E6-K94453 C4-K94453 E7-I12416
ICE143I 0 BLOCKSET SORT TECHNIQUE SELECTED
ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 0 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R12 - 22:17 ON SAT JUN 27, 2015 -
0 OPTION MOSIZE=10%,HIPRMAX=10%,DYNALLOC=(,6),FILSZ=E99999999 - HP13512
EXPMAX=10%,EXPOLD=10%,EXPRES=10% HP13512
$ $ $
ICE140I 0 END OF PARAMETERS FROM DFSPARM - SYSIN OR SORTCNTL/PARAMETER LIST CONTROL STATEMENTS FOLLOW
SORT FIELDS=(0001,0023,CH,A)
RECORD TYPE=F,LENGTH=(000111,,)
ICE193I 0 ICEAM2 INVOCATION ENVIRONMENT IN EFFECT - ICEAM2 ENVIRONMENT SELECTED
ICE089I 1 TEST . , INPUT LRECL = 111, TYPE = F
ICE093I 0 MAIN STORAGE = (MAX,40307196,40307196)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (40249836,40249836)
ICE127I 0 OPTIONS: OVFLO=RC0 ,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC16,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=256
ICE128I 0 OPTIONS: SIZE=40307196,MAXLIM=1048576,MINLIM=450560,EQUALS=N,LIST=Y,ERET=RC16 ,MSGDDN=SYSOUT
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=NO ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(SYSDA ,006),ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=4096,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=6291456,ARESALL=0,ARESINV=0,OVERRGN=16384,CINV=Y,CFW=Y,DSA=64
ICE132I 0 OPTIONS: VLSHRT=N,ZDPRINT=Y,IEXIT=N,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=1406 ,DSPSIZE=MAX ,ODMAXBF=0,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=1407
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE236I 0 OPTIONS: DYNAPCT=10 ,MOWRK=Y
ICE750I 0 DC 0 TC 0 CS DSVVV KSZ 23 VSZ 23
ICE752I 0 FSZ=99999999 RE IGN=0 C AVG=112 0 WSP=14546875 E DYN=252474 53216
(PGMSTAT) SCSMCU - SMCU - OPENED: SEQ INPUT 393 READS, 0 WRITES, 0 REWRITES, 0 DELETES
ICE751I 1 DE-K83743 D5-K91600 D3-K83080 D7-K58148 E8-I12416
ICE091I 0 OUTPUT LRECL = 111, TYPE = F
ICE055I 0 INSERT 202429599, DELETE 202429599
ICE054I 0 RECORDS - IN: 0, OUT: 0
ICE134I 0 NUMBER OF BYTES SORTED: 22469685489
ICE253I 0 RECORDS SORTED - PROCESSED: 202429599, EXPECTED: 99999999
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 438000 , TRACKS USED: 429585
ICE199I 0 MEMORY OBJECT USED AS MAIN STORAGE = 0M BYTES
ICE299I 0 MEMORY OBJECT USED AS WORK STORAGE = 1404M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 0K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE052I 0 END OF DFSORT

The JOB execution stats are as below:

Select all

JOB with SORTWork datasets:

JOBNAME STEPNAME PROCSTEP RC EXCP CPU SRB CLOCK
TEST STEP002 00 1683K 12.24 .38 78.06

Select all

JOB without SORTWork datasets and additional parameters:

JOBNAME STEPNAME PROCSTEP RC EXCP CPU SRB CLOCK
TEST STEP002 00 863K 12.25 .19 19.28

The query is, what caused the EXCP/SRB to be almost halved in either runs; we have tried multiple test runs, the results in all remaining the same.

Any pointers would be really helpful.

<Edit: JOB names have been modified in the stats>

Thanks.

by **BillyBoyo** » Sun Jun 28, 2015 2:13 pm

Look at the likely-looking messages.

Post the code relating to the SORT in the COBOL program.

by **Aki88** » Sun Jun 28, 2015 3:46 pm

Hello Billy,

My sincere apologies, the code is proprietry material, hence I am unable to share a snippet of the same. Though I'll try putting across the logic as bullet points:
1. The SD goes as:

Select all

000100 SD XXXX-SORT-FILE.
000200*

001600 01 XXXX-RECORD.
001700 03 XXXX-SORT-KEY1.
001800 05 FILLER PIC X(23).
001900 03 FILLER PIC X(88).
002000*

2. The SORT section trigger goes as below; prior to this there are control VSAM files which are opened, read, loaded into tables:

Select all

038700 SORT XXXX-SORT-FILE
038800 ON ASCENDING KEY XXXX-SORT-KEY1
038900 INPUT PROCEDURE IS 0500-OPEN-AND-READ-YYYY
039000 OUTPUT PROCEDURE IS 1000-RECAP-ZZZZ-PROCESSING.
039100*

3. In the OPEN and READ section, the input sequential file with below definition is read top-down and loaded to the XXXX-RECORD:

Select all

Organization . . . : PS
Record format . . . : FB
Record length . . . : 111
Block size . . . . : 27861
1st extent cylinders: 200
Secondary cylinders : 200
Data set name type : EXTENDED

052700 MOVE YYYY-RECORD TO XXXX-RECORD.
052800 RELEASE XXXX-RECORD.
052900 GO TO NNNN-READ-YYYY.
053000*

4. In the Output procedure, for each record read from the input file, further conditional checks are done, reporting tables are loaded, amount/interest buckets etc totals are done; and a chunk of accounting information is calculated.

5. Depeding on point-4 calculations, report breaks are performed.

6. Finally the report record is written out and the program goes to close the open files, if end of file has been reached for the driver file. There are a few VSAM files as well, which are read, these are control record files, which are loaded at the onset of the program itself, and are external to the SORT logic.

Hope this gives a little more clarity than before; though I agree that the explanation is murky, and I apologize for not being able to share the source code.

Though, what makes me wonder is how playing around with a few parameters would actually impact the EXCP, to a degree, where it is almost halved.

Thanks.

by **Magesh23586** » Sun Jun 28, 2015 4:49 pm

`Did you notice this message

Select all

ICE118I 0 UNKNOWN FILE SIZE - FILSZ=EN MAY IMPROVE RESOURCE USAGE AND PERFORMANCE

Which means DFSORT could not predict the input dataset size/records, If you provide an approximate number of records, DFSORT could consider it and allocate dynamic workspace optimally

I would suggest removing the Sort work space, and give OPTION FILSZ=240000000. considering a 20% increase of your current filesize

Select all

//DFSPARM DD *
OPTION FILSZ=E240000000
//*

you can use DFSPARM if you need to override the default, in your case you have (SYSDA, 15) already, So you just need to add OPTION FILSZ=240000000.

To know system default.

Submit following jcl.

Select all

//STEP0100 EXEC PGM=ICETOOL
//TOOLMSG DD SYSOUT=*
//DFSMSG DD SYSOUT=*
//OUT DD SYSOUT=*
//TOOLIN DD *
DEFAULTS LIST(OUT)
//*

Thanks
Magesh

by **BillyBoyo** » Sun Jun 28, 2015 7:08 pm

In addition, did you see this one? ICE247I 0 INTERMEDIATE MERGE ENTERED - PERFORMANCE MAY BE DEGRADED.

Look up the ICE247I for description of what is happening there.

Also look for things which didn't work in the second job.

Also look at compiler option FASTSRT. Look at using that, removing the input procedure, and then you don't have to tell DSFORT how many records, it will make its normal assumptions.

Have a look at the DFSORT manuals as they relate to performance and the COBOL manuals for SORT/MERGE.

by **Aki88** » Sun Jun 28, 2015 7:20 pm

Hello Billy,

Thank you for the pointers; indeed, I did look at the messages, was referring the tuning guide for further pointers.

I'll do another test run with FASTSRT in place; region defaults to NOFASTSRT.

Thank you.

by **Magesh23586** » Mon Jun 29, 2015 10:11 am

FASTSRT compiler option will not be enabled/used as long as you are using Input/Ouput procedure.

You need to change your logic to "USING/GIVING" in order to have FASTSRT.

regards.
Magesh

by **Aki88** » Mon Jun 29, 2015 10:46 am

Hello Magesh,

Indeed, and I have modified the code post Billy's suggestions for the test run.

Thanks.

IBM Mainframe Forum

SORTWork area changes v/s Job Execution Stats

SORTWork area changes v/s Job Execution Stats

Re: SORTWork area changes v/s Job Execution Stats

Re: SORTWork area changes v/s Job Execution Stats

Re: SORTWork area changes v/s Job Execution Stats

Re: SORTWork area changes v/s Job Execution Stats

Re: SORTWork area changes v/s Job Execution Stats

Re: SORTWork area changes v/s Job Execution Stats

Re: SORTWork area changes v/s Job Execution Stats