IBM Mainframe Forum

Hi

My requirement is to FTP a mainframe tape dataset to Linux server. The tape dataset has 400 million records. Is it possible to send the data in chunks, say 100,000 records, as one chunk and next 100,000 records as another chunk? The file at the receiving end will have Acct.Part.1, Acct.Part.2 etc as the file name for each of the chunks. This is to enable parallel processing. Also is there an option to specify a wait period as input. This is the time the process on mainframe waits before sending the next chunk.

Kindly let me know if my requirement is not clear.

Thanks
Prasanna G.

why not speak to both Your LOCAL and REMOTE Location network support people
we do not know about Your organisation setup, they do

four concurrent ftp transfers might put a severe strain on the communication <line> to the remote

You are asking 2 conceptually independent questions here.

• Can FTP, all by its little self, separate a file transfer into "groups"
  
  No, it can't.
• Can an FTP client process more than one file at a time.
  
  No, it can't. The client is basically a one process, one thread, function. A server, by definition, is a multiple thread function that can communicate with many FTP clients at a time.

There is no conceptual issue with running several FTP clients at a time all talking to a single FTP server.

I have tried, in the past, the equivalent of your idea, and it never seemed to work very well.

Breaking a large data transmission into "chunks" is not a bad idea, so that if there is a transmission problem with one "chunk" only the failing "chunk" needs to be rerun, not the entire transmission. In my experience, FTP on modern communication links rarely fails, so it's simply not worth the hassle.

In a past life I did customer support for a large ISV. From time to time our clients had to send us fairly large files. While we did have quite a few bad transmissions, it was almost always related to an error by the client. An example would be trying to send binary data as a text file, or sending a VB data set in binary. I don't recall any transfers aborted by a bad communication link. I was the local "expert" in trying to sort out these messes. About half the time, I was able to put humpty together. I could usually, for example, correctly deduce if a file represented SYS1.DUMPxx type data, or if it represented a file compacted by TRSMAIN or AMATERSE. My first two examples could not be corrected and had to be resent.

Hi Steve

Thanks for your inputs.

You are suggesting 100,000 records per transfer. That's 10 transfers per million records, and you have 400million records, so that is 4,000 transfers.

Mmm.... I suspect that would require automation.

What size are the records? Fixed- or variable-length? If those are an average of 80 bytes per records, that'd only be 8Mb per transfer, and I suspect "startup/breakdown" would outweigh any possible savings through "parallelisms".

Even doing a million records at a time, you'd end up with 400 processes which would need to be verified automatically.

Also bear in mind that you don't want to process the tape file multiple times. 4000 datasets is more than you can (probably) have as output (DDs) from one step.

How often are you transferring this data?

I'd "top'n'tail' the multiple files with a header and a trailer. Header to contain Logical file-name, environment-name, business/data-date, sequence number (if sequence of data is important). Trailer to include, at minimum, a record count.

Overall file header and trailer, same type of information on header, trailer to include number of individual files.

Process on the receiver then has to check all that when putting the data back together.

Without that, I'd not want to touch the received data with a barge pole.