iXora Custom Software Development Blog

Read | Practice | Advance

Blog

Posted by on in Blog

Often Linux shell script file is needed for Linux administrator for their regular jobs. Those shell scripts may contain few lines of statement. In that case, you can ignore many standards, useful tips. But we may need to write shell script for part of application process like data-warehouse ETL process. In that case our shell script may contain several lines of statement with many functionalities. In that case we have to think about code

Hits: 716
Rate this blog entry:
0
Normally We can't upload files of  size > 4 MB. If we have to upload file of size maximum 4 GB then the web.config file must be contained the following codes <configuration> </system.web> <httpRuntime maxRequestLength="4124672" requestValidationMode="2.0" executionTimeout="36000" /> </system.web> <system.webServer> <security> <requestFiltering> <requestLimits maxAllowedContentLength="4223664128"></requestLimits> </requestFiltering> </security> </system.webServer> </configuration>  Notes:  maxRequestLength  in httpRuntime is KB (Kilo Bytes)  BUT  maxAllowedContentLength  in  requestLimits is B (Bytes) Reference: http://hrn2k1.blogspot.com/2012/06/tips-about-uploading-4-gb-file-on-web.html ...
Hits: 647
Rate this blog entry:
0

Source data files may exists/uploaded in FTP location. We need to know file names of those and also need to download those files to local Linux box. Because we want to extract those files and stage in relational database for data-warehouse. It will be the part of ETL process.If we just want to get file names which are uploaded to FTP location, we have to create a bash function. Inside that function we will create an FTP interaction session.

Hits: 677
Rate this blog entry:
0

Often we need to download data files from SFTP location. Specifically if we work with data-warehouse type project, our source data may uploaded in SFTP location. In ETL process, we need to integrate that data to our data-warehouse. In that case we need to download those files to local machine and then we staged that data to our database for further processing.

Hits: 1245
Rate this blog entry:
0
We may need to download files from SFTP location. I got a requirement where files need to download from SFTP location in Linux platform. Those are data files. File download process was  part of ETL process. Our target SFTP location contains previously processed and latest (unprocessed) both types of files. But we need to download only latest (unprocessed) files. Before download the unprocessed file we have to identify unprocessed files. Each data file contain date-stamp. Based on that date-stamp we can capable to identify those files. So we need to fetch all file names first then we have to go for...
Hits: 668
Rate this blog entry:
0