Functional and flexible shell scripting tricks

by John Smith

Why shell scripting still matters

It‘s 2023, and with so many powerful high-level scripting languages like Python, Ruby, and JavaScript available, you may wonder if good old shell scripting still has a place. The answer is a resounding yes! Shell scripts continue to play a vital role in automating tasks, prototyping ideas, and gluing together other tools and commands.

Some key advantages of shell scripting:

• The shell and core utilities are present on virtually every Unix-like system. Your scripts will run almost anywhere.
• Shell scripts excel at interactively processing data and taking user input, making them ideal for command-line tools.
• By leveraging the many powerful tools and commands available in a Unix environment, you can accomplish a lot with relatively little code.

Let‘s dive into some practical techniques to level up your shell scripting skills. These tips will focus on bash/sh, but many of the concepts are applicable to other shells as well.

Modular scripting with functions

Just like any programming language, functions are a key building block for creating modular, reusable shell scripts. Defining a function is straightforward:

greet() {
  echo "Hello, $1!"
}

This defines a function named greet that takes a single argument and prints a greeting. To call it:

greet John  # prints "Hello, John!"

Here $1 refers to the first argument passed to the function. If your function takes multiple arguments, you can access them with $2, $3, etc. To capture all arguments as an array, use $@:

print_args() {
  for arg in "$@"; do
    echo "$arg"
  done
}

print_args one two three
# Prints:
# one
# two 
# three

A few key things to remember about shell functions:

• They must be defined in a script before they are called. Typically function definitions are grouped at the beginning of a script.
• Variables defined inside a function are in scope for the whole script. There is no concept of local function scope.
• The return statement is used to return an integer exit status, not a value. To "return" output, simply print to stdout and capture with command substitution.

add() {
  echo $(($1 + $2))  
}

result=$(add 2 3)
echo $result  # prints 5

Functions are often used to conditionally execute blocks of code based on their return status:

is_root() {
  [ "$EUID" -eq 0 ] 
}

if is_root; then
  echo "Running as root"
else
  echo "Not running as root"
fi

Here we define an is_root function that returns a 0 status (truthy) if the effective user ID is 0 (root), or 1 otherwise. We can then use the function directly as the condition in an if statement.

Multi-line output with heredocs

Many scripts need to print multi-line usage help text, ASCII art, or other formatted output. While you can simply use multiple echo statements, a more elegant solution is a here document or "heredoc":

cat <<EOF
Usage: $0 [OPTIONS]

A cool script that does stuff.

Options:
  -h    Print this help text  
  -v    Enable verbose output
EOF

The cat <<EOF syntax starts a heredoc block that continues until the EOF delimiter is seen. The text in the block is printed verbatim, with variable expansions performed. A few things to watch out for:

• The closing delimiter (EOF in this case) must appear on a line by itself with no leading whitespace.
• If the heredoc appears in an indented block like an if statement, the contents must have no leading indentation.
• You can use any identifier as the delimiter, but EOF or END is conventional. If the delimiter is quoted, no expansions are performed in the block.

if some_condition; then
  cat <<EOF
A multi-line message
with $(some_command)
EOF
fi

Heredocs are a clean and readable way to include longer text blocks or even embed other scripts or code within a shell script.

Processing options and flags

Most Unix commands and shell scripts support various options and flags to modify their behavior. A common pattern is to use getopts to process options supplied on the command line.

usage() {
  cat <<EOF
Usage: $0 [OPTIONS] FILE

Process FILE, optionally gzipping it.

Options:
  -g    Gzip the file
  -v    Enable verbose output
EOF
}

gzip_file=0
verbose=0

while getopts gv opt; do
  case "$opt" in
    g)
      gzip_file=1
      ;;
    v)  
      verbose=1
      ;;
    *)
      usage
      exit 1
      ;;
  esac
done

shift $((OPTIND - 1))

if [ $# -ne 1 ]; then
  usage
  exit 1
fi

file=$1

if [ $gzip_file -eq 1 ]; then 
  gzip "$file"
fi

This script showcases several common idioms:

• A usage function to print help text, usually called when invalid options are supplied or with -h
• Variables to track option state, either booleans or saving option arguments
• A while loop calling getopts with the option characters to parse
• A case statement to handle each option flag
• Shifting positional parameters to handle remaining non-option arguments
• Checking the number of arguments to ensure the script is called correctly

Some important getopts details:

• Options are specified as single characters in alphabetical order
• Adding a : after an option character indicates it takes an argument
• Option arguments are available in the $OPTARG variable in the loop
• The * case handles invalid options
• shift discards the parsed options, leaving only positional arguments

Proper handling of options and flags is key to writing flexible, usable, and maintainable shell scripts.

More shell scripting tricks

We‘ve covered some of the core techniques, but there are many more tricks to streamline your shell scripts:

• Parameter expansions like ${var:-default} and ${var//pattern/replace} to manipulate and substitute variables
• Arithmetic expansion with $((expression)) to perform mathematical operations
• Built-in regular expression matching with =~ in [[ ]] conditional expressions
• C-style for ((i=0; i<10; i++)) loops in addition to the shell‘s for arg in $list syntax
• Multiline if/then/elif/else conditionals
• Indexed and associative arrays
• Sourcing other script files to share functions and variables

Here‘s an example showing some of these in action:

# Default to user‘s home directory if DIR is not supplied
DIR=${1:-$HOME}

# Loop through files in DIR matching a regex
for file in "$DIR"/*.txt; do
  if [[ $file =~ .*important.* ]]; then
    echo "Found important file: $file"
  elif [[ -s $file ]]; then
    file_size=$(($(stat -c%s "$file") / 1024))
    echo "File is ${file_size}kb"
  else
    echo "Skipping $file"
    continue
  fi

  # Do stuff with file
done

Here we default DIR to $HOME if not supplied as an argument, use a C-style for loop to iterate over text files, perform regular expression matching on the filename, get the file size in kilobytes with arithmetic expansion, and skip some files with continue.

Conclusion

We‘ve explored several key techniques to write modular, robust, and expressive shell scripts. From functions to heredocs to option parsing to parameter expansion, these constructs provide a powerful toolkit for scripting.

The shell‘s conciseness, ubiquity, and integration with Unix tools continue to make it an essential skill to have in your toolbox. Mastering techniques like these will enable you to automate complex tasks, write sophisticated command-line utilities, and be more productive at the terminal.

So dive in and start experimenting with shell scripting! You‘ll be surprised at how much you can accomplish with a few lines of code and the power of the Unix shell.