Read | Practice | Advance
Often Linux shell script file is needed for Linux administrator for their regular jobs. Those shell scripts may contain few lines of statement. In that case, you can ignore many standards, useful tips. But we may need to write shell script for part of application process like data-warehouse ETL process. In that case our shell script may contain several lines of statement with many functionalities. In that case we have to think about code
Source data files may exists/uploaded in FTP location. We need to know file names of those and also need to download those files to local Linux box. Because we want to extract those files and stage in relational database for data-warehouse. It will be the part of ETL process.If we just want to get file names which are uploaded to FTP location, we have to create a bash function. Inside that function we will create an FTP interaction session.
Often we need to download data files from SFTP location. Specifically if we work with data-warehouse type project, our source data may uploaded in SFTP location. In ETL process, we need to integrate that data to our data-warehouse. In that case we need to download those files to local machine and then we staged that data to our database for further processing.