iXora Custom Software Development Blog

Read | Practice | Advance

Linux Shell (bash) Programming Guidelines

Posted by on in Blog
  • Font size: Larger Smaller
  • Hits: 2443
  • 0 Comments

Often Linux shell script file is needed for Linux administrator for their regular jobs. Those shell scripts may contain few lines of statement. In that case, you can ignore many standards, useful tips. But we may need to write shell script for part of application process like data-warehouse ETL process. In that case our shell script may contain several lines of statement with many functionalities. In that case we have to think about code

manageability, maintainability, reusability, extendibility etc. features. So any guideline will be help us to achieve that features. Here I am trying to explain with real-life experience.

Modularity:

If requirement demands to write much code, we should write modularize code. Modularization helps us to organize our code better. Every programming language has feature to write modularize code. In object oriented programming there is “Class” for creating object, JavaScript has constructor function to create object. So that environment we can use object to achieve modularization. But in bash script we only have function. So we can use this function for modularization. Before write function, just try to follow function writing guideline. For example: 

    1. Function should be independent as much as possible. It will help to test the function independently.
    2. Every function should do a particular task/job (single responsibility).
    3. Function name should be self-explanatory (what it does understand by its name).
    4. Write functions such a way that it can be re-usable.
    5. Limit variable scope. So create and use local variable as much as possible and try to avoid global variable.
    6. Top of the function declaration, write some useful comment so that it is clearly understand the purpose of the function.
    7. In big shell script file, create a co-ordinate main function. From that main function other functions may call. At the end of the file main         function will be called.
    8. All function variables declare top of the function and it will be one place. It will help to identify variable usage.
    9. Write set –0 (no unset) statement at the top of the script file. It will help you to stop using un-assigned variable. Actually it will notify you    if any variable mistakenly created.
#!/usr/bin/env bash
set -0
# addition 2 integer numbers and return its value
function add(){
   local first_value="${1}"
   local second_value="${2}"

   local sum_value=$(( $first_value+$second_value ))
   echo "$sum_value" # return value
}
# subtract send integer from first integer and return its value
function sub(){  
   local first_value="${1}"
   local second_value="${2}"
   
   local sub_value=$(( $first_value-$second_value ))
   echo "$sub_value" # return value
}
# main co-ordination function
function main(){
   local first_value="${1}"
   local second_value="${2}"
   local operation="${3}"

  if [ "$operation" == "add" ];then
     local result=$( add "$first_value" "$second_value" )
     echo "sum is: $result"
  elif [ "$operation" == "sub" ];then    
     local result=$( sub "$first_value" "$second_value" )
     echo "sub is: $result"
  else
     echo "sorry! no such operation has support yet."
  fi
}
# calling example: (assume that script file name is math_operation.sh)
# add function: math_operation.sh 20 10 "add" #it will print "sum is 30"  
# sub function: math_operation.sh 20 10 "sub" #it will print "sub is 10" 
main "${1}" "${2}" "${3}"

Exception Handling:

Modern programming languages has structured exception handling mechanism. Most of the cases they provide Try-Catch-Finally block. But bash script has no such feature. It provides a different way to handle error/exception. Let’s see what they are!

First of all if any error occurred in bash script, despite of error bash script is continuing its execution. That is little unusual. Suppose you write a script:

echo "started..."
cd linuxdir # assume linuxdir directory does not exists
ls

if “linuxdir” directory does not exists, script will show an error message like “cd: linuxdir: no such file or directory” but it will continue the execution. So next the ls command will display current working directories file/subdirectory list. It may create confusion. So if I want to stop execution if any error raised then I have to use:

set –e

in the top of the script. It will tell linux interpreter that stop execution when any error raised. Now the script will look like:

set -e
echo "started..."
cd linuxdir # assume linuxdir directory does not exists
ls

Still it is not completed. Bash has built in “$?” global variable. Based on that we can check any error raised or not. By default its value is 0. It means execution is completed without any errors. Non Zero means execution is completed with errors. Like:

echo "started..."
cd linuxdir
if [ "$?" -ne 0 ]; then
  echo "linuxdir directory not found. So execution is stopped"
  exit 1;
fi
ls
it is ok but when you use
set -e
“$?” will not work. Because when any error raised, script automatically stop execution and exit from the program. So if I need both feature then? Yes, bash has its solution too:
set -e
echo "started..."
set +e
cd linuxdir
if [ "$?" -ne 0 ]; then
  echo "linuxdir directory not found. so execution is stopped"
  exit 1;
fi
set -e
ls

We declare “set –e” in the top of the script. If any execution’s result we like to check then before that we use “set +e”. It will tell interpreter that not to exit from program if any error occurred. After that execution command/statement, write “set –e” again. That way you can achieve both feature.

Debugging & Logging:

Modern IDE has many debugging features like breakpoint, step-in/step-out, watch/quick-watch, stack trace, local/immediate window any many more. Unfortunately that type of IDE still not develop for bash script. Still most of the time we use built in text editor for that. But bash has a command “set –x”

set -x
echo "started..."
cd ..
ls

“set –x” will show every statement it executed and its result. Output like:

+ echo started...

started...

+ cd ..

+ ls

a.sh   b.sh   c.sh

So it is better to log the message to any file so that we can use that log file for analyze letter. To do so we can write function which will display output to the screen as well as to a file. That’s way we can use both features.bash has “echo” command. It will redirect output to the screen. It can also redirect output to a file. Many times we need to show any variable’s value or status of any command or some information by which we can know that this statement is executed. But if we print it on the screen the main problem is when we run that script we need to stay beside the screen. If we run the script with linux crown job scheduler then we did not see the echo result on the screen. Even if we deploy the script in production server and if another team maintain that server that time also we cannot see the result when needed.

function log_info(){
  local log_data="${1}"
  echo "$log_data"  
  echo "${1}" >> etl.log
}

function calling: log_info “started….”

It will display log data to the screen as well as append it to the “etl.log” file.

Reusable(utility/helper) Script Files:

When we start writing professional script files for various purpose, we see few functionalities write repeatedly. So we should stop that. We can follow “Don’t repeat yourself” programming principal. It says “if anything you write, same kind of things you should not write again, you should reuse your previous code”. In that case we can write common script files and from other files you can refer that files. The code example is as follows:

utility.sh file

# log provided data to etl.log file as well as it prints that to the console
function log_data(){
  local log_data="${1}"
  echo "$log_data"  
  echo "${1}" >> etl.log
}
#: return no of rows in a file including header part
#: ex: count_file_line "data.txt"
function count_file_line(){
  local file_full_name="${1}"
  local line_count=0
  local exists=$(is_file_exists "${file_full_name}")  

 if [ "${exists}" -eq 0 ];then
    echo 0
    exit 1
 fi
 line_count=$(wc -l < "${file_full_name}")
 echo ${line_count}
}
# From another script file, we can refer that utility script file.
#! /usr/bin/env bash
source ./utility.sh # should use proper path either relative or absolute
log_data “starting…”
ls

Defensive Coding: 

Bash scripting is interpreted scripting language. Errors occurred at run time. It is very difficult to identify those errors. So it will hamper our normal life. So we can take few precaution so that that type of error we can identify as quickly as possible. Few errors like:

File/Directory Existence check:

When we get any file/directory path, before use we must check its existence.

file="/home/user1/a.sh"
if [ -e "$file" ]; then 
  echo "file is exists"
else
  echo "file does not exists"
fi

Option

Description

-e

File existence check.

-d

File is a directory check.

-f

Regular file check.

-r

File has read permission.

-O

File ownership check for current user.

Variables Value Validate before Use:

If we receive any variable value from the user or file or other place first we much check its value is null/empty or not. Otherwise it may create run time error. The code example:

value=
if [ -z "$value" ]; then
  echo "value is empty"
fi
if [ ! -z "$value" ]; then
  echo "value is not empty"
fi

“-z” option will return true if the value is null/empty/space. It can also check array variable with same way. “! –z” option is just opposite.

Variable surrounded by double quote (“”)
file_name="/home/user1/Desktop/a b.csv"
cat "$file_name"
cat $file_name

if we are going to show file content of “a b.csv” file from “file_name” variable, if we use double quote with variable name ( “$file_name”) it will show the content of that file. But if we do not use double quote ($file_name) file content will not be displayed. Why because without using surrounded double quote (“”””) the bash take the value up-to the space character. So in that case without double quote

cat $file_name
will be translated to:
cat /home/user1/Desktop/a

As a result that “a” file does not exists error message. So it is good always use double quote when reading bash variable.

Array Usage:

Before use we have to learn array first. It has learning curve. Without proper learning array, it may be create very complicated problem that may identify and solve very costly. I have written a blog post about Array. Please visit:

All about array in linux shell bash script

Learn Linux Commands in Depth:

Linux has many commands. Bash programming is written based on that commands. Every single command have many switches/options. Before use there commands in our bash script, we have to know in detail. Otherwise we have to write unnecessary code or do tricks to solve problem that already solved by options. Few examples: 

echo: “echo” command is used for print anything in console. But we should know we can use it to write/append any content to a file. We can also use it to return any value from the function too.

For writing content if we write:

echo “I love my country” > country.txt

it will create a country.txt file (if not exists) or if exists it delete all its contents and add this to the first line. If we write:

echo “I love my country” >> country.txt

It will append this line to the country.txt file

function get_value(){
   echo 100
}
result=$( get_value )
echo "result is: $result"

The function “get_value” will return 100 when called.

rm: “rm” command is used for delete any file(s) from a path. “rmdir” is used for delete any empty directory. But if we want to delete any non empty directory then we should use “rm” command like:

rm –rf my_directory

If we want to delete any file like:

rm my_file.txt

and if my_file.txt does not exists in current path then bash will throw file not exists error. So if we does not like that error then we have to use

rm –f myfile.txt

If will stop raising error.

Regular expression: We should learn regular expression. It is little difficult. But once we learn the language it will help us the remaining life. Regular expression show just magic when text processing is needed. Bash has few tools to write and work with regular expression.

We can use it with directly with variables

input_date=$(echo ${input_date/[^0-9]*/})

It will replace all non-numeric characters with empty string.

We can use it with “grep” utility tool:

echo "$file_name" | grep -oP '[._ ][0-9][ -_.]+' > file_date.tmp   

it will pick date string from a provided file_name. It will start from either “.” Or  “_” or space and pick all numeric charecters and stop when found “_” or “ “ or “.” Character.

Interactive session: Bash script can create interactive session from its own session. Let me an example. We can write Linux shell script with python language too. If we want to use python function from my bash script what can I do? The easiest solution is we can create an interactive session from my bash script and within that interactive session we can execute python functions.

function send_email(){
python <<END
from common import *
reportType="$1"
SendReport(reportType)
END
echo "sent mail!!!"
}

Bash “send_email” function create a python interactive session. Inside that session, python function “SendReport” is executed. This "sendreport" function send email to clients. We can also create ftp/sftp interaction session with bash script. I write 2 separate blog posts about that:

How to Test:

Now a days unit/integration testing is very important for development context. The main reason behind that the rapid change of logic/rule. So code need to change. A programing world quote is “All code has guilty until test it”. So we easily understand the importance of testing. So many frameworks are developed to support that. Unfortunately still bash scripting have any testing framework. So we do it manually without any framework support. For example:

We write a bash script which contain a function. It just create a folder:

function create_dir(){
   local dir_name="${1}"
   mkdir "$dir_name"
}
create_dir "${1}"
Now we may create another script file which will make sure that this script is working properly or not.
#!/usr/bin/env bash
set -e
source ./a.sh
dir_name="tax_papers"
create_dir "$dir_name"

if [ -d "$dir_name" ];then
 echo "test passed! directory created successfully"
else
 echo "test failed! directory created successfully"
fi

Set Mindset: When we do shell (bash) programming we have to set our mindset to do so. Shell (bash) programming is difference then other programming language. It is actually command based programming, little similarities of Functional programming paradigm. If anything we want to do first we want to search which command Linux has provided for us. Identify best fitted command and make sure best use of that. Like framework libraries API of other platform. But command is very short in syntax and also using command is difference than API. So we have to remember that.

Enjoy shell programming!!!

Rate this blog entry:
0

Comments

  • No comments made yet. Be the first to submit a comment

Leave your comment

Guest
Guest Monday, 03 August 2020