Source data files may exists/uploaded in FTP location. We need to know file names of those and also need to download those files to local Linux box. Because we want to extract those files and stage in relational database for data-warehouse. It will be the part of ETL process. If we just want to get file names which are uploaded to FTP location, we have to create a bash function. Inside that function we will create an FTP interaction session.

1 function  get_ftp_file_name_list(){
2    local host_name_with_port="${1}";
3    local ftp_source_path="${2}"
4   local ftp_user="${3}"
5    local ftp_password="${4}"
6 
7   ftp -inv $host_name_with_port << ! 1>/dev/null
8       user $ftp_user $ftp_password  
9       cd "${ftp_source_path}" 
10       ls -1 "all_file_names.txt" 
11       close     
12 !
13 }
14 #Calling the function: get_ftp_file_name_list “ftp://test.com:21”  “datafiles/feeds/”  “testuser”  “***”
Code language: PHP (php)

The function “get_ftp_file_name_list” expect 4 arguments:

SL#	Argument Name	Description
1	host_name_with_port	Host name with port number
2	ftp_source_path	FTP source file directory path
3	ftp_user	Valid FTP User Id
4	ftp_password	Valid FTP User Password

first “ftp” command create an interactive session:

1 ftp -inv $host_name_with_port << ! 1>/dev/null
2 !Code language: JavaScript (javascript)

It uses 3 options “–inv”.

SL#	Argument Name	Description
1	-i	Turn off interactive prompting.
2	-n	First time attempting for auto-login.
3	-v	Enable verbose message.

user $ftp_user $ftp_password “/dev/null” means system message should not display or redirect to the null.

1 cd "${ftp_source_path}"Code language: JavaScript (javascript)

Change current directory to the provided directory.

1 ls -1 "all_file_names.txt"Code language: JavaScript (javascript)

Fetch only file names from the current directory and result is logged in the all_file_names.txt file.

1 close

will be closed ftp interaction session.

Download Files: If we want to download files from FTP location Linux has a built in command “wget”. Using that we can download files from FTP location. This command is actually non-interactive downloader from the web. Syntax:

1 wget -nd -nv -np -q -P "${dist_file_path}" --ftp-user="${ftp_user}" --ftp-password="${ftp_password}" <br>--timeout 0 "${ftp_full_file_path}"
Code language: JavaScript (javascript)

“wget” command expect few arguments:

SL#	Argument Name	Description
1	-nd	Turn off hierarchy of directory creation.
2	-nv	Turn off verbose log needed.
3	-q	After completion it will be quit.
4	-P	Directory path where all files will be stored.
5	–ftp-user	Name of the FTP user
6	–ftp-password	Password
7	–timeout	Maximum time the process will be wait for completion.
8	“${ftp_full_file_path}”	The FTP source path from where file will be downloaded.

SFTP location for File upload/store is more secured then FTP location. Now a days most of the clients store there data files to the SFTP location. I have written another blog for download files from SFTP location. URL:

Fetch File Names from SFTP Location using Shell (bash) Programming