IBM Mainframe Forum

I am curious about efficiencies that I can build into my Sort steps so speed them along faster. I have files to sort each day where the sork keys are simple, but there can be many parts of them. Sometimes one key will have 11 different parts. Example:

SORT FIELDS=(1,23,A,522,1,A,471,2,A,24,3,A,111,50,A,87,3,A,161,310,A,
90,6,A,106,4,A,494,3,A,35,10,A),FORMAT=CH


I have about 20+ Million records a night, LRECL=2171, RECFM = VB


I am curious if I should focus on moving the parts of the key out in position 552, 471, and 494 closer to the front of the records? They are still on the first 1/4 of the max record length.

Another option I can look at is combining the fields so they are contiguous and would require less individual parameters, but I don't know if having many parameters slows down SORT or not.

Advice is appreciated.

Another item I can look at is the ...161,310,A... parameter. The field may have 40 characters of the potential 310. Would cutting this down make it quicker?

Hello Farmer.

Please add the following JCL to the sort step in question and rerun the job:

This should produce some additional tuning messages in sysout. Feel free to forward the complete listing to me at alissa.margulies@syncsort.com as a text attachment and I would be happy to help you tune this application.

Regards,

I don't use Syncsort, and I do not have access to its documentation, so any advice I can give is very general, but ...

A reasonably accurate estimated record count (or an exact record count) will go a long way towards helping Syncsort run better, especially on a moderately large sort like yours. I don't know if Syncsort does this, but it might be able to deduce a record count from the DCB attributes and data set sizes if the data set is on disk, but for data sets with variable length records like yours its estimate could be way off.

Every unique field definition you specify adds a little bit of overhead for each record. If it is possible, consolidate the fields. This is something you're proposing anyway, but you might find the cost to do this is greater than any benefit you could could get in a Syncsort or DFSORT. The sort products (e.g., Syncsort or IBM's DFSORT) can almost certainly do I/O better than you or me!

steve-myers wrote:I don't use Syncsort, and I do not have access to its documentation, so any advice I can give is very general, but ...

A reasonably accurate estimated record count (or an exact record count) will go a long way towards helping Syncsort run better, especially on a moderately large sort like yours. I don't know if Syncsort does this, but it might be able to deduce a record count from the DCB attributes and data set sizes if the data set is on disk, but for data sets with variable length records like yours its estimate could be way off.

Every unique field definition you specify adds a little bit of overhead for each record. If it is possible, consolidate the fields. This is something you're proposing anyway, but you might find the cost to do this is greater than any benefit you could could get in a Syncsort or DFSORT. The sort products (e.g., Syncsort or IBM's DFSORT) can almost certainly do I/O better than you or me!

I believe Syncsort has a database behind it to keep track of items like this so it can make determinations on how much space it needs. In testing with my operations guys we simply outlawed any kind of buffereing or other tweaks in our SORTs because Syncsort is simply better than any of us. As you stated, I am looking to reduce the overhead to save a few minutes here or there and maybe revamping the entire key would be something I can look into. There is a 'key' on each record, but it doesn't mean anything as they system the key was built for disappeared 10 years ago. We could rebuild all of the key info up front and reduce the number of sort parameters. Sounds like a project for a slow day.

Thanks for chiming in!

The Mean Farmer wrote:... I am curious if I should focus on moving the parts of the key out in position 552, 471, and 494 closer to the front of the records? They are still on the first 1/4 of the max record length.

I very much doubt that moving the keys within the record will have any impact on sort performance. Perhaps Ms. Margulis can provide more insight into this.

The Mean Farmer wrote:...
SORT FIELDS=(1,23,A,522,1,A,471,2,A,24,3,A,111,50,A,87,3,A,161,310,A,
90,6,A,106,4,A,494,3,A,35,10,A),FORMAT=CH


I have about 20+ Million records a night, LRECL=2171, RECFM = VB

I just noticed this. With VB records, the start position includes the RDW, the 4 byte prefix of all V format logical records. So, FIELDS=(1,23,...) includes the RDW.

A couple of years ago I grumbled about this in another forum with Frank Yaeger, the DFSORT guru. He conceded at the time the people that built the OS/360 sort program, the great grandaddy of both Syncsort and DFSORT, screwed up, but it was far too late to correct this unwelcome "feature."

If the data can be easily reformatted when it is first being created, then I would say go ahead - you may see a slight CPU performance improvement in the sort step, and possibly a slight elapsed time improvement as well. However, if an additional step is required to do this, then its not worth the extra read/write of the data. It will also not improve performance to perform the aforementioned reformatting in an INREC statement within the sort step itself. I will review the listings sent to me offline to see if any additional recommendations can be made to improve the elapsed time performance in this step.

steve-myers wrote:I just noticed this. With VB records, the start position includes the RDW, the 4 byte prefix of all V format logical records. So, FIELDS=(1,23,...) includes the RDW.

Contrary to the original post, the actual job listing reflects that the data is FB, so we do not need to address this issue here. Thanks.

To recap for everyone else what I sent in the email to the OP, I noted that this step is sorting a 35GB file in about 10 minutes, with a transfer rate of about 110 MB/Sec. The sort is running in ELAP mode with increased VSCORET. This job appears to already be running optimally.