Download Files from SFTP to Linux Box using Shell-bash Program

Often we need to download data files from SFTP location. Specifically if we work with data-warehouse type project, our source data may uploaded in SFTP location. In ETL process, we need to integrate that data to our data-warehouse. In that case we need to download those files to local machine and then we staged that data to our database for further processing.

To download files from SFTP location we need to follow 2 steps:

Step1: We have to think about security. From Linux machine you can connect with SFTP server with userid and password. When you connect with SFTP server with “sftp” command, it will create an interactive session. In that session you can use SFTP file copy command (“get” command). But if you want to copy files without providing password then what will we do? It will need when we go for download from bash script that will execute by Linux scheduled job. Without providing password, if we need to access SFTP location, we have to create public/private key pair. My previous blog post (first section) I wrote about that. You can check there:

Fetch File Names from SFTP Location using Shell (bash) Programming

Step2: It is better if we create a function in bash script. That function will wrap the SFTP commands which will copy files. The “function” look like:

1 function download_sftp_file(){
2    local sftp_host="${1}" #SFTP Host name
3    local sftp_port="${2}" #SFTP Port Number default is 22
4    local sftp_user="${3}" #SFTP user id
5    local sftp_identity="${4}" #private key location
6    local sftp_source_path="${5}" #source file path, which file need to download        
7    local sftp_target_path="${6}" # target path where file will be downloaded/copied
     
8 sftp -P $sftp_port -i $sftp_identity  $sftp_user@$sftp_host << ! >&2>/dev/null 
9    !mkdir -p $sftp_target_path 
10    get "$sftp_source_path" "$sftp_target_path"
11 !
12 }Code language: PHP (php)

The above “function” will accept HostName, Port Number, SFTP user name, private key path, source file path and target directory path (where file will be copied). After accepting those arguments, execute first SFTP command. 

1 sftp -P $sftp_port -i $sftp_identity  $sftp_user@$sftp_host << ! >&2>/dev/null 
2 !Code language: PHP (php)

It will create SFTP interaction session. Inside that session we will use sftp mkdir command

1 !mkdir -p $sftp_target_pathCode language: PHP (php)

That means we will create a directory structure with same of source location if target location have no such directory structure exists. Then execute:

1 get "$sftp_source_path" "$sftp_target_path"
Code language: PHP (php)

It will copy source file path to the target path. Here in target path we are not providing any file name. So source file name and after download, the target file name will be same. If we need different requirement then just see the documentation of “get” SFTP command.

Enjoy Linux shell (bash) scripting!!!