Linux Shell (bash) Programming Guidelines

Often Linux shell script file is needed for Linux administrator for their regular jobs. Those shell scripts may contain few lines of statement. In that case, you can ignore many standards, useful tips. But we may need to write shell script for part of application process like data-warehouse ETL process. In that case our shell script may contain several lines of statement with many functionalities. In that case we have to think about code manageability, maintainability, reusability, extendibility etc. features. So any guideline will be help us to achieve that features. Here I am trying to explain with real-life experience.

Modularity:

If requirement demands to write much code, we should write modularize code. Modularization helps us to organize our code better. Every programming language has feature to write modularize code. In object oriented programming there is “Class” for creating object, JavaScript has constructor function to create object. So that environment we can use object to achieve modularization. But in bash script we only have function. So we can use this function for modularization. Before write function, just try to follow function writing guideline. For example: 

1. Function should be independent as much as possible. It will help to test the function independently.
2. Every function should do a particular task/job (single responsibility).
3. Function name should be self-explanatory (what it does understand by its name).
4. Write functions such a way that it can be re-usable.
5. Limit variable scope. So create and use local variable as much as possible and try to avoid global variable.
6. Top of the function declaration, write some useful comment so that it is clearly understand the purpose of the function.
7. In big shell script file, create a co-ordinate main function. From that main function other functions may call. At the end of the file main         function will be called.
8. All function variables declare top of the function and it will be one place. It will help to identify variable usage.
9. Write set –0 (no unset) statement at the top of the script file. It will help you to stop using un-assigned variable. Actually it will notify you    if any variable mistakenly created.

1 #!/usr/bin/env bash
2 set -0
3 # addition 2 integer numbers and return its value
4 function add(){
5   local first_value="${1}"
6   local second_value="${2}"
7 
8   local sum_value=$(( $first_value+$second_value ))
9   echo "$sum_value" # return value
10 }
11 # subtract send integer from first integer and return its value
12 function sub(){  
13   local first_value="${1}"
14   local second_value="${2}"
15    
16   local sub_value=$(( $first_value-$second_value ))
17   echo "$sub_value" # return value
18 }
19 # main co-ordination function
20 function main(){
21   local first_value="${1}"
22   local second_value="${2}"
23   local operation="${3}"
24
25  if [ "$operation" == "add" ];then
26     local result=$( add "$first_value" "$second_value" )
27     echo "sum is: $result"
28  elif [ "$operation" == "sub" ];then   
29     local result=$( sub "$first_value" "$second_value" )
30     echo "sub is: $result"
31  else
32     echo "sorry! no such operation has support yet."
33  fi
34 }
35 # calling example: (assume that script file name is math_operation.sh)
36 # add function: math_operation.sh 20 10 "add" #it will print "sum is 30"  
37 # sub function: math_operation.sh 20 10 "sub" #it will print "sub is 10" 
38 main "${1}" "${2}" "${3}"Code language: PHP (php)

Exception Handling:

Modern programming languages has structured exception handling mechanism. Most of the cases they provide Try-Catch-Finally block. But bash script has no such feature. It provides a different way to handle error/exception. Let’s see what they are!

First of all if any error occurred in bash script, despite of error bash script is continuing its execution. That is little unusual. Suppose you write a script:

1 echo "started..."
2 cd linuxdir # assume linuxdir directory does not exists
3 lsCode language: PHP (php)

if “linuxdir” directory does not exists, script will show an error message like “cd: linuxdir: no such file or directory” but it will continue the execution. So next the ls command will display current working directories file/subdirectory list. It may create confusion. So if I want to stop execution if any error raised then I have to use:

1 set –e
Code language: JavaScript (javascript)

in the top of the script. It will tell linux interpreter that stop execution when any error raised. Now the script will look like:

1 set -e
2 echo "started..."
3 cd linuxdir # assume linuxdir directory does not exists
4 lsCode language: PHP (php)

Still it is not completed. Bash has built in “$?” global variable. Based on that we can check any error raised or not. By default its value is 0. It means execution is completed without any errors. Non Zero means execution is completed with errors. Like:

1 echo "started..."
2 cd linuxdir
3 if [ "$?" -ne 0 ]; then
4  echo "linuxdir directory not found. So execution is stopped"
5  exit 1;
6 fi
7 ls
8 it is ok but when you use
9 set -e
10 “$?” will not work. Because when any error raised, script automatically stop execution and exit from the program. So if I need both feature then? Yes, bash has its solution too:
11 set -e
12 echo "started..."
13 set +e
14 cd linuxdir
15 if [ "$?" -ne 0 ]; then
16  echo "linuxdir directory not found. so execution is stopped"
17  exit 1;
18 fi
19 set -e
20 lsCode language: PHP (php)

We declare “set –e” in the top of the script. If any execution’s result we like to check then before that we use “set +e”. It will tell interpreter that not to exit from program if any error occurred. After that execution command/statement, write “set –e” again. That way you can achieve both feature.

Debugging & Logging:

Modern IDE has many debugging features like breakpoint, step-in/step-out, watch/quick-watch, stack trace, local/immediate window any many more. Unfortunately that type of IDE still not develop for bash script. Still most of the time we use built in text editor for that. But bash has a command “set –x”

1 set -x
2 echo "started..."
3 cd ..
4 lsCode language: PHP (php)

“set –x” will show every statement it executed and its result. Output like:

+ echo started…

started…

+ cd ..

+ ls

a.sh   b.sh   c.sh

So it is better to log the message to any file so that we can use that log file for analyze letter. To do so we can write function which will display output to the screen as well as to a file. That’s way we can use both features.bash has “echo” command. It will redirect output to the screen. It can also redirect output to a file. Many times we need to show any variable’s value or status of any command or some information by which we can know that this statement is executed. But if we print it on the screen the main problem is when we run that script we need to stay beside the screen. If we run the script with linux crown job scheduler then we did not see the echo result on the screen. Even if we deploy the script in production server and if another team maintain that server that time also we cannot see the result when needed.

1 function log_info(){
2 local log_data="${1}"
3  echo "$log_data" 
4  echo "${1}" >> etl.log
5 }Code language: PHP (php)

function calling: log_info “started….”

It will display log data to the screen as well as append it to the “etl.log” file.

Reusable(utility/helper) Script Files:

When we start writing professional script files for various purpose, we see few functionalities write repeatedly. So we should stop that. We can follow “Don’t repeat yourself” programming principal. It says “if anything you write, same kind of things you should not write again, you should reuse your previous code”. In that case we can write common script files and from other files you can refer that files. The code example is as follows:

utility.sh file

1 # log provided data to etl.log file as well as it prints that to the console
2 function log_data(){
3  local log_data="${1}"
4  echo "$log_data" 
5  echo "${1}" >> etl.log
6 }
7 #: return no of rows in a file including header part
8 #: ex: count_file_line "data.txt"
9 function count_file_line(){
10  local file_full_name="${1}"
11  local line_count=0
12  local exists=$(is_file_exists "${file_full_name}")  
13
14 if [ "${exists}" -eq 0 ];then
15    echo 0
16    exit 1
17  fi
18 line_count=$(wc -l < "${file_full_name}")
19 echo ${line_count}
20 }
21 # From another script file, we can refer that utility script file.
22 #! /usr/bin/env bash
23 source ./utility.sh # should use proper path either relative or absolute
24 log_data “starting…”
25 lsCode language: PHP (php)

Defensive Coding:

Bash scripting is interpreted scripting language. Errors occurred at run time. It is very difficult to identify those errors. So it will hamper our normal life. So we can take few precaution so that that type of error we can identify as quickly as possible. Few errors like:

File/Directory Existence check:

When we get any file/directory path, before use we must check its existence.

1 file="/home/user1/a.sh"
2 if [ -e "$file" ]; then
3  echo "file is exists"
4 else
5  echo "file does not exists"
6 fiCode language: PHP (php)

Variables Value Validate before Use:

If we receive any variable value from the user or file or other place first we much check its value is null/empty or not. Otherwise it may create run time error. The code example:

1 value=
2 if [ -z "$value" ]; then
3  echo "value is empty"
4 fi
5 if [ ! -z "$value" ]; then
6  echo "value is not empty"
7 fiCode language: PHP (php)

“-z” option will return true if the value is null/empty/space. It can also check array variable with same way. “! –z” option is just opposite.

Variable surrounded by double quote (“”)

1 file_name="/home/user1/Desktop/a b.csv"
2 cat "$file_name"
3 cat $file_nameCode language: PHP (php)

if we are going to show file content of “a b.csv” file from “file_name” variable, if we use double quote with variable name ( “$file_name”) it will show the content of that file. But if we do not use double quote ($file_name) file content will not be displayed. Why because without using surrounded double quote (“”””) the bash take the value up-to the space character. So in that case without double quote

1 cat $file_name
2 will be translated to:
3 cat /home/user1/Desktop/aCode language: PHP (php)

As a result that “a” file does not exists error message. So it is good always use double quote when reading bash variable.

Array Usage:

Before use we have to learn array first. It has learning curve. Without proper learning array, it may be create very complicated problem that may identify and solve very costly. I have written a blog post about Array. Please visit:

All about array in linux shell bash script

Learn Linux Commands in Depth:

Linux has many commands. Bash programming is written based on that commands. Every single command have many switches/options. Before use there commands in our bash script, we have to know in detail. Otherwise we have to write unnecessary code or do tricks to solve problem that already solved by options. Few examples: 

echo: “echo” command is used for print anything in console. But we should know we can use it to write/append any content to a file. We can also use it to return any value from the function too.

For writing content if we write:

1 echo “I love my country” > country.txtCode language: CSS (css)

it will create a country.txt file (if not exists) or if exists it delete all its contents and add this to the first line. If we write:

1 echo “I love my country” >> country.txtCode language: CSS (css)

It will append this line to the country.txt file

1 function get_value(){
2   echo 100
3 }
4 result=$( get_value )
5 echo "result is: $result"Code language: PHP (php)

The function “get_value” will return 100 when called.

rm: “rm” command is used for delete any file(s) from a path. “rmdir” is used for delete any empty directory. But if we want to delete any non empty directory then we should use “rm” command like:

1 rm –rf my_directory

If we want to delete any file like:

1 rm my_file.txtCode language: CSS (css)

and if my_file.txt does not exists in current path then bash will throw file not exists error. So if we does not like that error then we have to use

1 rm –f myfile.txtCode language: CSS (css)

If will stop raising error.

Regular expression: We should learn regular expression. It is little difficult. But once we learn the language it will help us the remaining life. Regular expression show just magic when text processing is needed. Bash has few tools to write and work with regular expression.

We can use it with directly with variables

1 input_date=$(echo ${input_date/[^0-9]*/})Code language: JavaScript (javascript)

It will replace all non-numeric characters with empty string.

We can use it with “grep” utility tool:

1 echo "$file_name" | grep -oP '[._ ][0-9][ -_.]+' > file_date.tmpCode language: PHP (php)

it will pick date string from a provided file_name. It will start from either “.” Or  “_” or space and pick all numeric charecters and stop when found “_” or “ “ or “.” Character.

Interactive session: Bash script can create interactive session from its own session. Let me an example. We can write Linux shell script with python language too. If we want to use python function from my bash script what can I do? The easiest solution is we can create an interactive session from my bash script and within that interactive session we can execute python functions.

1 function send_email(){
2 python <<END
3 from common import *
4 reportType="$1"
5 SendReport(reportType)
6 END
7 echo "sent mail!!!"
8 }Code language: JavaScript (javascript)

Bash “send_email” function create a python interactive session. Inside that session, python function “SendReport” is executed. This “sendreport” function send email to clients. We can also create ftp/sftp interaction session with bash script. I write 2 separate blog posts about that:

1. Fetch File Names & Download Files from FTP Location to Linux Box using Shell (bash) Programming
2. Fetch File Names from SFTP Location using Shell (bash) Programming

How to Test:

Now a days unit/integration testing is very important for development context. The main reason behind that the rapid change of logic/rule. So code need to change. A programing world quote is “All code has guilty until test it”. So we easily understand the importance of testing. So many frameworks are developed to support that. Unfortunately still bash scripting have any testing framework. So we do it manually without any framework support. For example:

We write a bash script which contain a function. It just create a folder:

1 function create_dir(){
2   local dir_name="${1}"
3   mkdir "$dir_name"
4 }
5 create_dir "${1}"
6 Now we may create another script file which will make sure that this script is working properly or not.
7 #!/usr/bin/env bash
8 set -e
9 source ./a.sh
10 dir_name="tax_papers"
11 create_dir "$dir_name"
12 
13 if [ -d "$dir_name" ];then
14 echo "test passed! directory created successfully"
15 else
16 echo "test failed! directory created successfully"
17 fiCode language: PHP (php)

Set Mindset: When we do shell (bash) programming we have to set our mindset to do so. Shell (bash) programming is difference then other programming language. It is actually command based programming, little similarities of Functional programming paradigm. If anything we want to do first we want to search which command Linux has provided for us. Identify best fitted command and make sure best use of that. Like framework libraries API of other platform. But command is very short in syntax and also using command is difference than API. So we have to remember that.

Enjoy shell programming!!!