iXora Custom Software Development Blog

Read | Practice | Advance

Download Files from SFTP Location to Linux Box using Shell (bash) Programming

Posted by on in Blog
  • Font size: Larger Smaller
  • Hits: 1511
  • 0 Comments
  • Print

Often we need to download data files from SFTP location. Specifically if we work with data-warehouse type project, our source data may uploaded in SFTP location. In ETL process, we need to integrate that data to our data-warehouse. In that case we need to download those files to local machine and then we staged that data to our database for further processing.

 

b2ap3_thumbnail_sftpdownload.png

To download files from SFTP location we need to follow 2 steps:

Step1: We have to think about security. From Linux machine you can connect with SFTP server with userid and password. When you connect with SFTP server with “sftp” command, it will create an interactive session. In that session you can use SFTP file copy command (“get” command). But if you want to copy/download files without providing password then what will we do? It will need when we go for download from bash script that will execute by Linux scheduled job. Without providing password, if we need to access SFTP location, we have to create public/private key pair. My previous blog post (first section) I wrote about that. You can check there:

Fetch File Names from SFTP Location using Shell (bash) Programming

Step2: It is better if we create a function in bash script. That function will wrap the SFTP commands which will download files. The “function” look like:

function download_sftp_file(){
	local sftp_host="${1}" #SFTP Host name
	local sftp_port="${2}" #SFTP Port Number default is 22
	local sftp_user="${3}" #SFTP user id
	local sftp_identity="${4}" #private key location
	local sftp_source_path="${5}" #source file path, which file need to download		
	local sftp_target_path="${6}" # target path where file will be downloaded/copied
	
sftp -P $sftp_port -i $sftp_identity  $sftp_user@$sftp_host << ! >&2>/dev/null	
	!mkdir -p $sftp_target_path	
	get "$sftp_source_path" "$sftp_target_path"
!
}

The above “function” will accept HostName, Port Number, SFTP user name, private key path, source file path and target directory path (where file will be downloaded). After accepting those arguments, execute first SFTP command. 

sftp -P $sftp_port -i $sftp_identity  $sftp_user@$sftp_host << ! >&2>/dev/null	
!

It will create SFTP interaction session. Inside that session we will use sftp mkdir command

!mkdir -p $sftp_target_path

That means we will create a directory structure with same of source location if target location have no such directory structure exists. Then execute:

get "$sftp_source_path" "$sftp_target_path"

It will download source file path to the target path. Here in target path we are not providing any file name. So source file name and after download, the target file name will be same. If we need different requirement then just see the documentation of “get” SFTP command.

Enjoy Linux shell (bash) scripting!!!

Rate this blog entry:
1

Comments

  • No comments made yet. Be the first to submit a comment

Leave your comment

Guest
Guest Tuesday, 25 September 2018