iXora Custom Software Development Blog

Read | Practice | Advance

Fetch File Names & Download Files from FTP Location to Linux Box using Shell (bash) Programming

Posted by on in Blog
  • Font size: Larger Smaller
  • Hits: 810
  • 0 Comments
  • Print

Source data files may exists/uploaded in FTP location. We need to know file names of those and also need to download those files to local Linux box. Because we want to extract those files and stage in relational database for data-warehouse. It will be the part of ETL process.If we just want to get file names which are uploaded to FTP location, we have to create a bash function. Inside that function we will create an FTP interaction session.

function  get_ftp_file_name_list(){
	local host_name_with_port="${1}";
	local ftp_source_path="${2}"
	local ftp_user="${3}"
	local ftp_password="${4}"

   ftp -inv $host_name_with_port << ! 1>/dev/null
       user $ftp_user $ftp_password  
       cd "${ftp_source_path}"  
       ls -1 "all_file_names.txt"  
       close     
!
}
#Calling the function: get_ftp_file_name_list “ftp://test.com:21”  “datafiles/feeds/”  “testuser”  “***”

The function “get_ftp_file_name_list” expect 4 arguments:

SL#

Argument Name

Description

1

host_name_with_port

Host name with port number

2

ftp_source_path

FTP source file directory path

3

ftp_user

Valid FTP User Id

4

ftp_password

Valid FTP User Password

first "ftp" command create an interactive session:

ftp -inv $host_name_with_port << ! 1>/dev/null
!

 It uses 3 options “–inv”.

SL#

Argument Name

Description

1

-i

Turn off interactive prompting. 

2

-n

First time attempting for auto-login. 

3

-v

Enable verbose message.

user $ftp_user $ftp_password “/dev/null” means system message should not display or redirect to the null.

login to the ftp site with provided userid and password

cd "${ftp_source_path}"  

Change current directory to the provided directory.

ls -1 "all_file_names.txt"  

Fetch only file names from the current directory and result is logged in the all_file_names.txt file.

close

will be closed ftp interaction session.

Download Files: If we want to download files from FTP location Linux has a built in command “wget”. Using that we can download files from FTP location.  This command is actually non-interactive downloader from the web. Syntax:

wget -nd -nv -np -q -P "${dist_file_path}" --ftp-user="${ftp_user}" --ftp-password="${ftp_password}" 
--timeout 0 "${ftp_full_file_path}"

“wget” command expect few arguments:

SL#

Argument Name

Description

1

-nd

Turn off hierarchy of directory creation.

2

-nv

Turn off verbose log needed.

3

-q

After completion it will be quit.

4

-P

Directory path where all files will be stored.

5

--ftp-user

Name of the FTP user

6

--ftp-password

Password

7

--timeout

Maximum time the process will be wait for completion.

8

"${ftp_full_file_path}"

The FTP source path from where file will be downloaded.

 

SFTP location for File upload/store is more secured then FTP location. Now a days most of the clients store there data files to the SFTP location. I have written another blog for download files from SFTP location. URL:

Fetch File Names from SFTP Location using Shell (bash) Programming

Enjoy bash scripting!!!

Rate this blog entry:
0

Comments

  • No comments made yet. Be the first to submit a comment

Leave your comment

Guest
Guest Tuesday, 25 September 2018