IBM Mainframe Forum

Hi I have a BIG file (more than 70 millions of records of 389 LRECL) that I'm sorting with the next code:




It work very well but the problem is that the sort take about 30 minutes to end.

Any Idea to optimize the sort in order to improve the efficiency? it is possible?

Here the dfsort log:

ICE250I 0 VISIT http://www.ibm.com/storage/dfsort FOR DFSORT PAPERS, EXAMPLES AND MORE
ICE000I 1 - CONTROL STATEMENTS FOR 5694-A01, Z/OS DFSORT V1R10 - 07:59 ON WED JUN 20, 2012 -
SORT FIELDS=(371,8,A,92,10,A,37,5,A,42,5,A,47,3,A),
FORMAT=BI,SIZE=E90000
INCLUDE COND=(50,2,SS,EQ,C' ,X , X')
OUTFIL FNAMES=SALIDA1,
INCLUDE=(31,1,CH,EQ,C'X'),BUILD=(1,389)
OUTFIL FNAMES=SALIDA2,
INCLUDE=(31,1,CH,NE,C'X'),BUILD=(1,389)
END
ICE201I H RECORD TYPE IS F - DATA STARTS IN POSITION 1
ICE858I 0 LA=1712768, DA=61769, AA=1650999, BA=4093, CP=0, TA=4141
ICE859I 0 LB=9756, DB=2044, AB=7712, BB=0, CP=3, RS=8, TB=31
ICE860I 0 F=YN, P=2, M=N, B=2048
ICE992I 0 RA 0 WR 0 TR 1
ICE751I 0 C5-K90025 C6-K90025 C7-K54603 C8-K62201 E4-K51707 C9-BASE E5-K62201 E6-K51707 C4-K62201 E7-K62201
ICE193I 0 ICEAM1 INVOCATION ENVIRONMENT IN EFFECT - ICEAM1 ENVIRONMENT SELECTED
ICE252I 1 PARMLIB OPTIONS WERE MERGED WITH INSTALLATION MODULE DEFAULTS
ICE088I 1 ZJCFPHOP.SORT005 . , INPUT LRECL = 389, BLKSIZE = 27619, TYPE = FB
ICE093I 0 MAIN STORAGE = (MAX,63250892,63222220)
ICE156I 0 MAIN STORAGE ABOVE 16MB = (63164860,63164860)
ICE127I 0 OPTIONS: OVFLO=RC16,PAD=RC0 ,TRUNC=RC0 ,SPANINC=RC4 ,VLSCMP=N,SZERO=Y,RESET=Y,VSAMEMT=Y,DYNSPC=4000
ICE128I 0 OPTIONS: SIZE=63250892,MAXLIM=2097152,MINLIM=524288,EQUALS=N,LIST=Y,ERET=ABEND,MSGDDN=SYSOUT
ICE129I 0 OPTIONS: VIO=N,RESDNT=ALL ,SMF=FULL ,WRKSEC=Y,OUTSEC=Y,VERIFY=N,CHALT=N,DYNALOC=(SYSALLDA,003),ABCODE=MSG
ICE130I 0 OPTIONS: RESALL=12288,RESINV=0,SVC=109 ,CHECK=Y,WRKREL=Y,OUTREL=Y,CKPT=N,COBEXIT=COB2
ICE131I 0 OPTIONS: TMAXLIM=20971520,ARESALL=0,ARESINV=0,OVERRGN=65536,CINV=Y,CFW=Y,DSA=256
ICE132I 0 OPTIONS: VLSHRT=Y,ZDPRINT=Y,IEXIT=Y,TEXIT=N,LISTX=N,EFS=NONE ,EXITCK=S,PARMDDN=DFSPARM ,FSZEST=N
ICE133I 0 OPTIONS: HIPRMAX=OPTIMAL,DSPSIZE=0 ,ODMAXBF=2097152,SOLRF=Y,VLLONG=N,VSAMIO=N,MOSIZE=0
ICE235I 0 OPTIONS: NULLOUT=RC0
ICE084I 0 BSAM ACCESS METHOD USED FOR SORTIN
ICE750I 0 DC 27368415147 TC 0 CS DSVVV KSZ 31 VSZ 31
ICE752I 0 FSZ=70355823 RC IGN=90000 E AVG=392 0 WSP=35821008 C DYN=419616 53216
ICE915I 0 MOFSZ=0,MOSZ=0,MOSYS=0(1),MOSTG=86833,MEML=17592186040320(3)
ICE916I 0 MOFR=0003,MOVR=VV
ICE996I 0 ESM=29147136,ESO=8744140,ESR=0,ESP=4096,ESS=16384,CES=29147136,HSZ=524288
ICE997I 0 HWSP=16833180,HMAX=3538944,HES=3538944,ASV=22229440,EQ=I9,HN=1
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=3538944,CMAX=524288,HU=99,BUN=53216,MD=M3,DU=90,DR=0,HN=1
ICE880I 0 QP=128 QA=1091 HI=4591 LI=4338 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=1
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=3014656,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=2
ICE880I 0 QP=128 QA=1237 HI=4337 LI=4192 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=2
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=2490368,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=3
ICE880I 0 QP=128 QA=2603 HI=4191 LI=2826 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=3
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=1966080,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=4
ICE880I 0 QP=128 QA=4899 HI=2825 LI=530 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=4
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=1441792,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=5
ICE880I 0 QP=128 QA=5026 HI=5053 LI=403 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=5
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=917504,CMAX=524288,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=6
ICE880I 0 QP=128 QA=5153 HI=5055 LI=276 MI=5428 TZ=524288 N1=524288 N2=524288 SZ=129 HN=6
ICE898I 0 OMAX=18414594,NMAX=22229440,ENQT=393216,CMAX=393216,HU=99,BUN=53216,MD=MH,DU=90,DR=0,HN=7

ICE880I 0 QP=96 QA=5426 HI=275 LI=3 MI=5428 TZ=393216 N1=393216 N2=393216 SZ=97 HN=7
ICE889I 0 CT=MAX , SB=3, L=0, D=0000, CCW=1MAM
ICE901I 0 W 02PP17 01PP17 03PP13
ICE902I 0 O I PP10
ICE897I 0 AVAL=14762821,BVAL=21747845,ENQT=0,ASV=21747845,HSZ=0,HM=M1,HN=10
ICE231I 0 STORAGE USED FOR OUTFIL : BELOW 16M = 31744, ABOVE 16M = 4240384
ICE855I 0 SALIDA1 : TX=N, R=J, L=J, B=D, BL=3, BR=9, DCT=32, VS=N, RU=X, SB=8
ICE210I 0 SALIDA1 : BSAM USED, LRECL = 389, BLKSIZE = 32676, TYPE = FB
ICE855I 0 SALIDA2 : TX=N, R=J, L=J, B=D, BL=3, BR=9, DCT=32, VS=N, RU=X, SB=8
ICE210I 0 SALIDA2 : BSAM USED, LRECL = 389, BLKSIZE = 32676, TYPE = FB
ICE751I 1 DE-K61785 D5-K62201 D3-BASE D7-BASE CB-K64631 E8-K61438
ICE900I 0 CON=1,MUV=1,VOL=30,ENU=0,SBK=0,SRC=0,VEM=0
ICE999I 0 PWK=2 PSP=419616 SWK=0 SSP=0 TWK=0 TSP=0 RWK=0 RSP=0 AWK=2 AWP=419616
ICE055I 0 INSERT 0, DELETE 1
ICE054I 0 RECORDS - IN: 70355823, OUT: 70355822
ICE227I 0 SALIDA1 : DELETED = 70238819, REPORT = 0, DATA = 117003
ICE228I 0 SALIDA1 : TOTAL IN = 70355822, TOTAL OUT = 117003
ICE227I 0 SALIDA2 : DELETED = 117003, REPORT = 0, DATA = 70238819
ICE228I 0 SALIDA2 : TOTAL IN = 70355822, TOTAL OUT = 70238819
ICE134I 0 NUMBER OF BYTES SORTED: 27368414758
ICE253I 0 RECORDS SORTED - PROCESSED: 70355822, EXPECTED: 70355823
ICE165I 0 TOTAL WORK DATA SET TRACKS ALLOCATED: 419640 , TRACKS USED: 253905
ICE199I 0 MEMORY OBJECT STORAGE USED = 0M BYTES
ICE180I 0 HIPERSPACE STORAGE USED = 14858260K BYTES
ICE188I 0 DATA SPACE STORAGE USED = 0K BYTES
ICE891I 1 63213368 WMAIN, 8848 CMAIN, MAX CALLOC, N SCN, A BA, 0 AZ, 1 BZ, NN QC, 0 CZ, 0 DZ, 0 PLE
ICE892I 1 389 RIN 27619 BLI 0 BLO 392 RUN 53216 BUN 2817 CPU 00 CVC
ICE893I 1 255 XIN 14 WIN 136392 GIN NDEY PFP42 BJ0 CMB00 CIX UPTH LMD VS RUX
ICE894I 0 178 STR 178 MOR 1028 IPB 0 OPB 16927 CYL O MN
ICE881I 0 EQ=I9 DX=0 D2=0 D3=127 D4=1 AS=0 SA=0 SB=0 SC=0 HN=1
ICE885I 0 DAT 00 DSR 0003 BINS 0 BSZ 0 RCP 0 AJC 0 RLC 0 DUNIT 0
ICE895I 0 296 MUNIT 42 SUNIT 31 OUNIT
ICE896I 0 34825 SET 6681620 DEXTOT 521242 BLK 70355823 CSZ 33028 WE
ICE804I 1 SORTWK03 EXCP COUNT: 1
ICE804I 1 SORTWK02 EXCP COUNT: 16526
ICE804I 1 SORTWK01 EXCP COUNT: 16501
ICE899I 0 HSR=2735,HSW=2885,HRE=40276,HWE=4046,HRP=523588,HWP=523588,HWM=524242,HNM=1
ICE052I 0 END OF DFSORT


Thanks.javivi

Posts: 30
Joined: Fri Jan 21, 2011 2:53 pm
Private message

Your last 3 sort fields are contiguous - why not make them into one (sort may do that already anyway - not sure about that) but 30 minutes isn't all that long.

Javivi,

Our performance expert(Dave Betten) suggests that you increase the ODMAXBF to 6M (currently you have it at 2M) and see if it improves your elapsed time. Add the following to your JCL

You have SIZE=E90000, you could check what that means with your installation options. If not used (so DFSORT able to reasonably estimate iput file size) you could remove it. If it is being used, it should be a much closer approximation to the filesize.

You have a BUILD on each of the OUTFILs. You have not extended/shortened the records, so I can't see that they are necessary. Perhaps doing it 70,000,0000 times is something noticeable? If not needed, remove.