IBM Mainframe Forum

I have some JCL that performs SFTP transfers and I'm having some trouble getting it to work as a scheduled task. My problem appears to be with the user account that's used to run the scheduler and various permissions issues. I was wondering if anyone had any experience scheduling SFTP and would be willing to share any case studies with the forum? One error that's really giving me fits is:

1IKJ56644I NO VALID TSO USERID, DEFAULT USER ATTRIBUTES USED

When I try to copy data from MVS to the USS filesystem and I'm curious as to whether an account has to have TSO access in order to execute ported tools like SFTP. My company has restricted the ID used from having a TSO account for business and security reasons. I can make the case to add access but I'd like to know with reasonable certainty that this would alleviate the issue before pursuing it.

FTP/SFTP issues are the same whatever the scheduler used,
using a scheduler(any of them) usually jobs are submitted using a default userid
with <powerful> enough authorities to have the jobs perform as needed

work with Your support in order to ...
provide the <scheduler> userid with the proper TSO segment

use the surrogate userid facility.

anyhow/anyway you will have to work with Your support
we might just wasting time giving suggestions that Your support might reject

My support is looking to me to determine what they need to do. It sounds like from your post that I would need TSO access for the account that runs the scheduler. I'll follow up on that and post the results here.

I was able to figure this out. Since I haven't seen it anywhere else on the web, here's a scrubbed version of what I did:

Overview
SFTP transfers are possible using the Unix System Services Ported Tools package for z/OS. The SFTP client can only be executed in the OMVS Unix shell or using EXEC PGM=BPXBATCH to call the client from JCL. Files and batch commands are referenced and manipulated in OMVS Unix filesystem (HFS). This limitation can be circumvented by using OCOPY to move datasets in and out of the OMVS Unix filesystem.

Preparing Automated Logins
In order to execute a SFTP transfer from batch, the login must be automated. To facilitate this, the SSH public key must be sent to the external site so it can be linked with the FTP account.

Creating an Outbound Data File
SFTP transfers send files from the OMVS Unix filesystem. Data can be generated in MVS and output to datasets. These datasets can then be converted to Unix files using the OCOPY command. For example, suppose a dataset named TEST.SFTP.DATA was created with intent to send to an external source. The data could be copied to the Unix filesystem using the following JCL:

//INMVS DD DSN=TEST.SFTP.DATA,
// DISP=SHR
//OUTHFS DD FILEDATA=TEXT,
// PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
// PATHMODE=SIRWXU,
//*****************************************************
//* OUTPUT FILE PATH ON OMVS
//*****************************************************
// PATH='/sftp/outbound.txt’
//SYSTSIN DD *
OCOPY IND(INMVS) OUTDD(OUTHFS)
//*
//SYSTSPRT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*

The above example copies TEST.SFTP.DATA to a Unix file named outbound.txt in directory /sftp. The directory and name are combined to form the PATH. Unix filesystem paths, directories and filenames are case-sensitive, which means outbound.txt, Outbound.txt and OUTBOUND.TXT are all unique files.

Importing an Inbound Data File
SFTP transfers retrieve files to the OMVS Unix filesystem. Data can be copied to MVS datasets from files located on OMVS. This process involves using the OCOPY command to convert the OMVS file to MVS-readable datasets. The data could be retrieved from the Unix filesystem using the following JCL:

//JS030 EXEC PGM=IKJEFT01
//INHFS DD PATH='/sftp/batchin/sftp.textget.txt',PATHOPTS=(ORDONLY)
//OUTMVS DD DSN=’TEST.SFTP.DATA’,DISP=SHR
//SYSTSIN DD *
OCOPY IND(INHFS) OUTDD(OUTMVS) TEXT CONVERT(YES) PATHOPTS(USE)
/*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *

The above example copies /sftp/batchin/sftp.textget.txt to a MVS dataset named TEST.SFTP.DATA. The directory in which the file is located and filename are combined to form the PATH. Unix filesystem paths, directories and filenames are case-sensitive, which means outbound.txt, Outbound.txt and OUTBOUND.TXT are all unique files.

Creating a Batch Command File
SFTP uses a batch command file to queue commands for non-interactive sessions. Batch command files are accessed from the OMVS Unix filesystem during SFTP execution. These files can be created as datasets in MVS and copied to the Unix filesystem. For example, a dataset with the following batch commands:

ascii
put '/sftp/outbound.txt’

Can be copied to the Unix filesystem using the following JCL:

//INMVS DD DSN=TEST.SFTP.CMD,
// DISP=SHR
//OUTHFS DD FILEDATA=TEXT,
// PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
// PATHMODE=SIRWXU,
//*****************************************************
//* BATCH CMD PATH ON OMVS
//*****************************************************
// PATH='/sftp/sftp.cmds'
//SYSTSIN DD *
OCOPY IND(INMVS) OUTDD(OUTHFS)
//*
//SYSTSPRT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*

The above example copies TEST.SFTP.CMD to a Unix file named sftp.cmds in directory /sftp. The directory and name are combined to form the PATH. Unix filesystem paths, directories and filenames are case-sensitive, which means sftp.cmds, SFTP.cmds and SFTP.CMDS are all unique files.

Executing a Transfer
A SFTP session is executed in JCL using the BPXBATCH program with the SFTP specific information listed as a PARM to the command. The PARM field must include the batch commands file, denoted with a –b flag, as well as the user ID being used for authentication and the target server. The following is a sample JCL transfer execution command:

//JS030 EXEC PGM=BPXBATCH,PARM='sh sftp -b /sftp/sftp.cmds user@servername.com'
//STDOUT DD PATH='/sftp/log/test.ftp.stdout',
// PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
// PATHMODE=SIRWXU
//STDERR DD PATH='/SFTP/log/test.ftp.stderr,
// PATHOPTS=(OWRONLY,OCREAT,OTRUNC),
// PATHMODE=SIRWXU
//SYSOUT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*

In the example, /sftp/sftp.cmds is called as the list of batch commands with the –b flag. This is proceeded by the user ID on the target server, user, and the target server name, servername.com. The @ between the user ID and the target server name let the SFTP client associate the ID with the target server and attempt authentication using that account instead of the account the batch job has been submitted under.

Importing Transfer Log Files
The STDOUT and STDERR options used in the transfer step represent the location the process will put logging output during command execution. STDOUT represents normal messages while STDERR represents error messages. These files can be accessed by system administrators for troubleshooting but have the possibility of being overwritten. For this reason it is recommended that both files are copied from OMVS and placed in MVS datasets. The following is a sample JCL copy command for the logging output:

//*****************************************************
//* Pull standard error over to a data set
//*****************************************************
//JS040 EXEC PGM=IKJEFT01
//INHFS DD PATH='/sftp/log/sftp.stderr',PATHOPTS=(ORDONLY)
//OUTMVS DD DSN='TEST.SFTP.STDERR',DISP=SHR
//* OCOPY IND(INHFS) OUTDD(OUTMVS) TEXT CONVERT(YES) PATHOPTS(USE)
//SYSTSIN DD *
oget '/sftp/log/sftp.stderr' 'TEST.SFTP.STDERR'
/*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *
//*****************************************************
//* Pull standard out over to a data set
//*****************************************************
//JS040 EXEC PGM=IKJEFT01
//INHFS DD PATH='/sftp/log/sftp.stdout',PATHOPTS=(ORDONLY)
//OUTMVS DD DSN='TEST.SFTP.STDOUT',DISP=SHR
//* OCOPY IND(INHFS) OUTDD(OUTMVS) TEXT CONVERT(YES) PATHOPTS(USE)
//SYSTSIN DD *
oget '/sftp/log/sftp.stdout' 'TEST.SFTP.STDOUT'
/*
//SYSTSPRT DD SYSOUT=*
//SYSTSIN DD *


OMVS Folder Structure and Use

/sftp – The base directory for SFTP file services. Location for batch command files due to inability to use line continuation on the batch execution command for transfer.

/sftp/log – Location for log storage. Choosing a unique prefix for logs based on the process creating them will prevent them from being overwritten by other processes.

/sftp/batchin – Location for ‘get’ operations that retrieve a file from a remote system and return it to our mainframe. It is the location that files should be copied from OMVS to MVS datasets.

/sftp/batchout – Location for ‘put’ operations that transmit a file to a remote system from out mainframe. It is the location that files should be copied from MVS datasets to OMVS.


Assigning Surrogate Authority for the Service Account

Account <service account> was created in RACF using these commands:
AU <service account> DFLTGRP(AUTOMATE) OWNER(SECURITY)
ALU <service account> NAME('SFTP APP')
PW USER(<service account>) NOINTERVAL
ALU <service account> DATA('REQUIRED TO EXECUTE OMVS COMMANDS AND ACCESS FILES STORED IN -
USS')

Surrogate access was permitted to <scheduler account ID> using these commands:
RDEF SURROGAT <service account>.SUBMIT UACC(NONE) OWNER(SYSPGMR) -----------> to define resource
PE <service account>.SUBMIT CL(SURROGAT) ID(<scheduler account ID) AC(READ) -----------------------> to permit access

An OMVS segment was defined for <service account> by using these commands:
ALU <service account> OMVS(UID(0000050500))
ALU <service account> OMVS(HOME('/sftp'))
ALU <service account> OMVS(PROGRAM('/bin/sh'))

The default group for the service account should be given a GID with this command:
ALG <default group> OMVS(GID(0000050500))

The commands for creating the TSO segment for <service account> are as follows:
ALU <service account> TSO(ACCTNUM('C5000') MAXSIZE(0010240) SIZE(0002560) +
PROC(ASPFDBII) UNIT(3390))

SSH requires use of keys to secure communications. These keys are created in OMVS by running the following commands:

ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ""
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ""

By default, SSH looks in the /.ssh directory for keyrings and authentication. For this reason, after generating the keys a new HFS file was mounted to this location and the keys generated from the preceding commands were moved to the default location.

Good to hear it is working - thank you for posting your resolution

d

A couple of gotchas to relay now that I've set up a couple of different jobs using sftp:

1. Your keyring has to have 600 permissions or the sftp executable will refuse to use the keyring.
2. I've had trouble getting the process to automatically add the external server's host key to the known_hosts file. To get around that I request their public key when I send them mine so I can add it manually.
3. We've had trouble with command continuation with omvs commands so we had to shorten our directory structure to ensure the entire command fits on one line.