Shell and Bash Roadmap

What is Bash? Core

A shell is the program that reads what you type, runs commands and prints the results. Bash, the Bourne Again Shell, is the most widely used shell on Linux servers and the default for most scripts you will meet in CI pipelines, Dockerfiles and deployment tools. It is both an interactive command line and a programming language.

CLI vs GUI: a graphical interface is easier to discover, but the command line is faster for repeated work, works over SSH on remote machines, and can be scripted. Text commands can be saved, reviewed, shared and run automatically, which is why every engineer needs basic shell fluency.

Practice. Open a terminal, run echo $SHELL and bash --version, and note which shell and version you have.

Popular Shells Recommended

Several shells exist, and knowing their differences prevents confusing bugs.

• sh: the POSIX standard shell. On Debian and Ubuntu it is dash, a small, fast shell without Bash extras such as arrays or [[ ]].
• Bash: the usual scripting choice on Linux.
• zsh: the default interactive shell on macOS, largely Bash-compatible with richer completion.
• fish: friendly interactive shell with its own, non-POSIX syntax.
• ksh and tcsh: older shells still found on some Unix systems.
• PowerShell: object-based shell from Microsoft, cross-platform; cmd is the legacy Windows command prompt.

Use whatever you like interactively, but write scripts for Bash with a Bash shebang, or for plain sh if they must run everywhere.

Practice. Run the same small script with bash and with dash and explain any error you see.

Setting Up Bash Core

On Linux, Bash is already installed. On macOS, the system Bash is very old because of licensing; install a current one with Homebrew (brew install bash) and keep zsh for interactive use if you prefer. On Windows, use WSL to get a real Linux environment, or Git Bash for lighter needs.

Interactive settings live in ~/.bashrc; login shells read ~/.bash_profile or ~/.profile, which usually sources ~/.bashrc. Put your aliases, prompt and PATH changes there and run source ~/.bashrc to reload.

shopt -s histappend
HISTSIZE=50000
export EDITOR=vim

Practice. Set up a ~/.bashrc with a larger history, your editor and one alias, and reload it.

Help Commands Core

The documentation is on the machine. Use help commands before searching online.

man grep          # manual page; / to search, q to quit
grep --help       # short usage summary
help cd           # help for Bash builtins
type ls           # alias, builtin, function or file
whatis tar
apropos compress  # search manual descriptions

Builtins such as cd, read and export are part of Bash itself, so help documents them rather than man. Manual pages are split into sections: 1 for commands, 5 for file formats, 8 for admin tools; man 5 crontab shows the file format rather than the command.

Practice. Use only man and help to find how to make grep print line numbers and how read sets a timeout.

Tab Completion, Repeat Commands and Bash Alias Core

Speed on the command line comes from typing less. Tab completion finishes command names, paths and, with the bash-completion package, options and Git branches; press Tab twice to see choices.

To repeat commands: the up arrow walks history, Ctrl+R searches it, !! repeats the last command, sudo !! reruns it as root, and !$ reuses the last argument. Ctrl+A and Ctrl+E jump to the start and end of the line.

alias ll='ls -lah'
alias gs='git status'
unalias gs

A Bash alias is a shortcut for a command. Aliases do not take arguments in the middle; use a function for that.

Practice. Add three aliases you would really use and practise Ctrl+R until you stop using the arrow keys to find old commands.

Text Editors Core

You will edit scripts and configuration in a terminal, often over SSH. Nano is the easiest: shortcuts are listed at the bottom, Ctrl+O saves and Ctrl+X exits.

Vi is on every Unix system, and Vim is its improved version. They are modal: normal mode for commands, insert mode for typing. Basic editor operations to learn first: i to insert, Esc to leave insert mode, :w to save, :q! to quit without saving, dd to delete a line, u to undo and /word to search. Emacs is another powerful editor with its own key bindings and a loyal following.

Learn enough Vim to edit safely on any server, even if you use a graphical editor day to day.

Practice. Edit a file in Vim: search for a word, delete two lines, undo once, save and quit.

Navigate Between Directories Core

Three commands cover navigation. pwd prints the current directory, ls lists its contents and cd changes directory.

pwd
ls -la            # long format, including hidden dot files
ls -lt            # newest first
cd /etc           # absolute path
cd ../logs        # relative: .. is the parent, . is here
cd ~ ; cd -       # home; previous directory
pushd /tmp; popd  # directory stack

Paths are case-sensitive. Names that start with a dot are hidden from plain ls. Quote paths with spaces: cd "My Files".

Practice. Move between five directories using absolute paths, relative paths, ~ and cd - only.

Creating and Removing Files Core

The basic file commands:

touch notes.md               # create empty file, or update its time
mkdir -p src/lib/utils       # mkdir with parents
cp file.txt backup.txt       # cp command: copy
cp -r project/ project-copy/ # copy a directory
mv old.txt new.txt           # mv command: rename
mv *.log logs/               # move into a directory
rmdir empty-dir              # rmdir: only removes empty dirs
rm file.txt                  # rm command: delete
rm -r build/                 # delete a directory tree

There is no recycle bin. Use rm -i while learning, run ls with the same wildcard before deleting, and never run rm -rf on a path built from a variable that might be empty.

Practice. Create a nested project structure in one command, copy it, rename files in the copy and delete it safely.

Viewing Files: cat, less and more, head and tail Core

cat prints whole files and joins several together. For long files use a pager: less and more both page through text, but less scrolls both ways and searches, so it has replaced more for most people.

cat a.txt b.txt > both.txt
less /var/log/syslog     # /pattern to search, G end, q quit
head -n 5 data.csv       # first lines
tail -n 20 app.log       # last lines
tail -f app.log          # follow new lines live

head and tail are the quickest way to peek at large files, and tail -f is the standard way to watch logs while you test something.

Practice. Use head to view a CSV header, tail -f to watch a log you append to in another terminal, and less to search it.

Find Core

find walks a directory tree and selects files by name, type, size, age, owner or permissions, then prints them or runs a command on each.

find . -name "*.py"
find /var/log -type f -size +100M
find . -type f -mtime -1          # changed in the last day
find . -name "*.tmp" -delete
find src -name "*.js" -exec grep -l "TODO" {} +
find . -type f -print0 | xargs -0 wc -l

Quote the pattern so the shell does not expand it first. -exec ... {} + passes many files per command, which is faster than one at a time. -print0 with xargs -0 handles names containing spaces or newlines safely.

Practice. Find all files over 50 MB in your home directory, then all Markdown files changed this week.

Wildcards Core

Wildcards, also called globs, are expanded by the shell into matching file names before the command runs.

• Asterisk wildcard: * matches any characters, so *.log means every log file.
• Wildcard question mark: ? matches exactly one character, so file?.txt matches file1.txt.
• Square bracket wildcards in shell/Bash: [abc] or [0-9] match one character from a set.
• Curly braces wildcards: {jpg,png} is brace expansion, which generates words even if no file exists: mkdir {src,test,docs}.

If nothing matches, Bash passes the pattern unchanged, which can surprise scripts; shopt -s nullglob changes that. Globs do not match hidden files unless you set dotglob.

Practice. Create files named img1.jpg to img12.png with brace expansion, then select subsets with each wildcard type.

File Permissions Core

Each file has an owner, a group, and rwx file permissions for owner, group and others: r to read, w to write, x to execute or, on directories, to enter.

ls -l run.sh        # -rwxr-x--- 1 asha devs
chmod +x run.sh     # make a script executable
chmod 640 secrets.env   # owner rw, group r, others none
chmod -R g+w shared/
sudo chown asha:devs report.csv
chgrp devs report.csv

chmod changes the bits, using letters or octal numbers where read is 4, write 2 and execute 1. chown changes the owner and optionally the group, and chgrp changes only the group. A script needs execute permission to run as ./script.sh.

Practice. Make a script executable only by you and readable by your group, and check with ls -l.

stdin, stdout, and stderr Core

Every command has three standard streams: stdin, file descriptor 0, for input; stdout, 1, for normal output; and stderr, 2, for error messages. By default stdin is the keyboard and both outputs go to the terminal.

Separating errors from results lets you save or pipe data while still seeing problems. When you write scripts, follow the same rule: print results to stdout and messages, warnings and errors to stderr with echo "error" >&2. That keeps your script usable in pipelines.

Practice. Write a script that prints data to stdout and a warning to stderr, and show they can be separated.

Output Redirection, Input Redirection and Error Redirection Core

ls > files.txt              # output redirection, overwrite
date >> files.txt           # append
wc -l < files.txt           # input redirection
grep x /etc/* 2> errors.txt # error redirection
cmd > out.txt 2>&1          # both streams to one file
cmd &> out.txt              # Bash shorthand
cmd 2>/dev/null             # discard errors
set -o noclobber            # refuse to overwrite with >

The order of redirections matters: 2>&1 copies wherever stdout points at that moment, so cmd 2>&1 > file sends errors to the terminal, not the file. /dev/null discards anything written to it.

Practice. Show the difference between cmd > f 2>&1 and cmd 2>&1 > f with a command that writes to both streams.

Pipes Core

A pipe, |, sends one command's stdout into the next command's stdin. Each program does one job, and pipes chain them into a processing line that streams data without temporary files.

history | grep ssh | tail -5
ps aux | sort -k4 -rn | head -5     # top memory users
cut -d, -f3 sales.csv | sort | uniq -c | sort -rn
cmd 2>&1 | tee run.log              # see and save output
cmd |& less                         # pipe stdout and stderr

By default, a pipeline's exit status is that of the last command, so earlier failures are hidden. In scripts, set -o pipefail makes the pipeline fail if any part fails.

Practice. Build a pipeline that lists the five largest files in a directory tree with their sizes.

Here Documents and Here Strings Core

A here document feeds several lines of text to a command's stdin, ideal for writing config files or multi-line input from a script.

cat > app.conf <<EOF
host=$HOSTNAME
port=8080
EOF
cat <<'EOF'          # quoted: no expansion
Price: $5
EOF
grep -c a <<< "banana"    # here string

Variables expand inside an unquoted here document; quote the delimiter to keep text literal. <<- strips leading tabs, so you can indent the block. A here string, <<<, passes a single string as stdin, handy for read and quick tests.

Practice. Generate a config file with a here document that includes the current user and date.

Command Substitution and Process Substitution Core

Command substitution, $(command), runs a command and inserts its output into the command line. Use it to capture output into variables.

today=$(date +%F)
files=$(ls *.csv | wc -l)
echo "Kernel: $(uname -r)"
diff <(sort a.txt) <(sort b.txt)     # process substitution
while read -r line; do echo "$line"; done < <(grep x f)

Prefer $( ) to old backticks; it nests cleanly. Trailing newlines are removed from the output. Process substitution, <(command), gives a command's output a temporary file name, so tools that expect files, such as diff, can compare command outputs directly. Unlike a pipe into while, it keeps the loop in the current shell, so variables set inside survive.

Practice. Compare the installed package lists of two machines with diff and process substitution.

grep Core

grep prints lines that match a pattern. It is the main tool for searching logs and code.

grep "ERROR" app.log
grep -i "timeout" app.log       # ignore case
grep -rn "api_key" src/          # recursive with line numbers
grep -v "DEBUG" app.log          # invert
grep -c "200" access.log         # count lines
grep -o "[0-9]\+ms" app.log      # print only the match
grep -l "TODO" *.py              # file names only

grep exits 0 when it finds a match and 1 when it does not, so it doubles as a test in scripts: if grep -q pattern file; then.

Practice. Find every line with ERROR or WARN in a log, excluding a noisy component, and count them per type.

Regular Expressions Core

Regular expressions describe text patterns. Basic regex syntax: . any character, * zero or more of the previous item, ^ start of line, $ end, [a-z] a set, [^0-9] not in a set, and \ to escape.

Extended regular expressions, with grep -E or sed -E, add + one or more, ? optional, | alternation, () groups and {2,4} counts without backslashes.

grep -E '^[0-9]{1,3}(\.[0-9]{1,3}){3}' ips.txt
grep -E 'error|fail(ed|ure)' app.log
[[ $email =~ ^[^@]+@[^@]+\.[a-z]+$ ]] && echo ok

Note that regex * is not the same as the shell's wildcard *. Bash's =~ operator matches extended regexes inside [[ ]].

Practice. Write regexes for a date in YYYY-MM-DD form, an IPv4-like address and a line ending in a number.

sed Core

sed, the stream editor, transforms text line by line. Its most common job is find and replace.

sed 's/http:/https:/' urls.txt        # first match per line
sed 's/foo/bar/g' file                # every match
sed -n '5,10p' file                   # print lines 5 to 10
sed '/^#/d' config                    # delete comment lines
sed -E 's/([0-9]+)ms/\1 ms/g' log
sed -i.bak 's/DEBUG/INFO/' app.conf   # edit in place, keep backup

In-place editing differs between GNU sed on Linux and BSD sed on macOS: on macOS -i needs an argument, such as -i ''. Using a backup suffix works on both.

Practice. Use sed to remove comments and blank lines from a config file and change one value in place.

awk Core

awk is a small language for column-based text. It splits each line into fields, $1 to $NF, and runs actions for lines matching a pattern. NR is the line number and -F sets the separator.

awk '{print $1}' access.log
awk -F, '$3 > 100 {print $1, $3}' sales.csv
awk -F: '{print $1}' /etc/passwd
awk 'NR > 1' data.csv              # skip header

awk for numeric operations is where it beats other tools: sums, averages and grouping in one line.

awk -F, '{t[$2] += $4} END {for (k in t) print k, t[k]}' sales.csv
awk '{s += $NF} END {printf "%.2f\n", s/NR}' times.txt

Practice. Compute total and average sales per region from a CSV with one awk command.

Sort, uniq, wc and nl Core

These filters summarise text.

sort names.txt              # alphabetical
sort -n sizes.txt           # numeric
sort -t, -k3 -rn data.csv   # by third CSV field, descending
sort -u list.txt            # unique while sorting
sort file | uniq -c         # count duplicates
uniq -d                     # only duplicated lines
wc -l file                  # count lines (-w words, -c bytes)
nl script.sh                # number non-empty lines

uniq only removes adjacent duplicates, so always sort first. The pattern sort | uniq -c | sort -rn | head is the classic "top N" report.

Practice. Find the ten most common words in a text file using tr, sort, uniq and head.

cut and paste, tr Command and Case Conversion Core

cut extracts columns, paste joins files line by line, and the tr command translates or deletes characters.

cut -d, -f1,3 data.csv
cut -c1-8 ids.txt              # by character position
paste names.txt ages.txt       # side by side, tab separated
paste -sd, list.txt            # join all lines with commas
tr ',' '\t' < data.csv
tr -d '\r' < win.txt > unix.txt   # remove Windows line endings
tr 'a-z' 'A-Z' <<< "hello"

Case conversion in Bash itself needs no external tool: ${name^^} gives upper case and ${name,,} lower case. cut cannot handle quoted CSV fields containing commas; use a real CSV tool for that.

Practice. Convert a Windows CSV to Unix line endings, keep two columns and upper-case one of them.

Join and Split Optional

join merges two files on a shared key field, like a database join; both inputs must be sorted on that field. split breaks a large file into smaller pieces by lines or size.

join -t, -1 1 -2 1 <(sort users.csv) <(sort orders.csv)
split -l 100000 big.csv part_      # 100,000 lines each
split -b 50M backup.tar chunk_
cat chunk_* > backup.tar           # reassemble

Split is useful for uploading big files in parts or processing data in parallel. For anything more than a simple join, a small script or SQL is easier to read.

Practice. Split a large CSV into parts, process each part, and recombine the results.

What is Scripting? Core

Scripting means writing a sequence of commands into a file so the computer can run them for you. A shell script is ideal for gluing tools together: setting up a machine, running a build, backing up files, rotating logs or wrapping a deployment.

Bash is the right tool when the task is mostly running other programs and moving files. When you need complex data structures, careful error handling across many steps, or more than a couple of hundred lines, switch to a language such as Python. Knowing where that line is matters as much as the syntax.

Practice. List three tasks you repeat by hand each week and decide which suit a shell script.

Bash Script Anatomy Core

A well-formed script has a shebang, strict mode, comments and a clear structure.

#!/usr/bin/env bash
# backup.sh: archive a directory to a dated file.
# Usage: backup.sh SOURCE DEST
set -euo pipefail

main() {
  local src=$1 dest=$2
  tar -czf "$dest/backup-$(date +%F).tar.gz" "$src"
}

main "$@"

The shebang line tells the kernel which interpreter runs the file. Comments start with #; use them to explain why and how to call the script, not to restate each command. Putting the logic in functions with a main call at the end keeps scripts readable.

Practice. Write a script with a shebang, usage comment, strict mode and a main function.

Running Shell Scripts Core

There are three ways to run a script, and they behave differently.

• Running with Bash: bash script.sh starts a new Bash process; no execute bit needed, and the shebang is ignored.
• Direct execution: chmod +x script.sh then ./script.sh; the kernel reads the shebang to pick the interpreter. To run it by name alone, put it in a directory on PATH.
• Running shell scripts with source: source script.sh or . script.sh runs it inside your current shell, so its variables, functions and cd change your session.

Use source for files that set up your environment, such as .bashrc or a virtualenv activate script. Use the other two for normal programs.

Practice. Write a script that changes directory and sets a variable, run it all three ways and compare your shell afterwards.

Variables in Shell Scripting Core

Assign with name=value, no spaces around the equals sign, and read with $name or ${name}.

user="asha"
log_dir="/var/log/${user}_app"
readonly MAX_RETRIES=3
port=${PORT:-8080}         # default if unset or empty
: "${API_KEY:?API_KEY must be set}"   # fail if missing
unset user

Variables best practices: always double-quote expansions to avoid word splitting and glob surprises; use lowercase names for script variables so you do not clash with environment ones; use readonly for constants and local inside functions; and give defaults or fail early for required settings.

Practice. Write a script that reads a port from the environment with a default and fails clearly if a required variable is missing.

Environment vs. Shell Variables Core

A shell variable exists only in the current shell. An environment variable is exported, so child processes, the programs you start, receive a copy.

greeting=hi
bash -c 'echo "$greeting"'   # prints nothing
export greeting
bash -c 'echo "$greeting"'   # prints hi
DEBUG=1 ./app.sh            # set for one command only
env | grep PATH

Children can never change their parent's variables; that is why a script that sets PATH must be sourced to affect your shell. Common environment variables include PATH, HOME, USER, PWD, SHELL and LANG.

Practice. Demonstrate with a child process that only exported variables are inherited.

Bash Data Types and Strings in Bash Core

Bash data types are few: everything is a string, and context decides whether it is treated as a number. Beyond strings, Bash has indexed arrays and associative arrays, and declare -i marks a variable as integer.

Strings in Bash can be quoted three ways. Single quotes keep text exactly. Double quotes allow $var, $(cmd) and \ escapes but prevent splitting. ANSI-C quoting, $'...', interprets escapes such as \n and \t.

a='$HOME stays literal'
b="Home is $HOME"
c=$'line one\nline two'
declare -i n=5; n+=2; echo "$n"   # 7

Practice. Write examples showing the output of the same text in single, double and ANSI-C quotes.

Echo and printf Formatting Core

Echo prints its arguments followed by a newline. It is fine for simple messages, but its options and escape handling differ between shells.

printf formatting is predictable and powerful: a format string with placeholders, then values.

echo "Deploying $app"
printf '%s\n' "$line"                 # safe for any text
printf '%-10s %5d\n' "$name" "$count"  # padded columns
printf '%.2f\n' 3.14159
printf -v stamp '%(%F %T)T' -1        # date into a variable

printf does not add a newline unless you include \n. Use it for tables, numbers and any text that might start with a dash or contain backslashes.

Practice. Print a neat two-column table of file names and sizes using printf.

Read User Input Core

The read builtin reads a line from stdin into variables.

read -rp "Environment (staging/prod): " env
read -rsp "Password: " pass; echo     # silent
read -t 10 -rp "Continue? [y/N] " ok || ok=n
IFS=, read -r name age city <<< "Asha,31,Leeds"
while IFS= read -r line; do echo "> $line"; done < notes.txt

Always use -r so backslashes are kept. Setting IFS controls how the line is split. Remember that scripts may run without a terminal, in CI or cron, so take values from arguments or environment variables and keep prompts optional.

Practice. Write a confirmation prompt that defaults to no and times out after ten seconds.

Exit Codes Core

Every command returns an exit code from 0 to 255. Exit codes: success vs. failure is simple: 0 means success and anything else means failure, with the meaning of non-zero values set by each program.

grep -q root /etc/passwd
echo $?              # exit codes and $?: 0 here
mkdir /root/x 2>/dev/null || echo "failed with $?"
cmd && echo ok || echo failed
exit 2               # end a script with a code

$? holds the exit code of the last command, so read it immediately. Conventions: 1 general error, 2 misuse of arguments, 126 not executable, 127 command not found, and 128 plus N when killed by signal N. Always exit non-zero from your scripts on failure so callers and CI notice.

Practice. Write a script that exits 0, 1 or 2 depending on its input, and check $? after each run.

Conditionals - if Core

In Bash, if runs a command and takes the then-branch when it exits 0. Tests are commands too: [ ] is the POSIX test and [[ ]] is Bash's safer version with pattern and regex matching.

if [[ -z ${1:-} ]]; then
  echo "usage: $0 NAME" >&2; exit 2
elif [[ $1 == admin* ]]; then
  echo "admin user"
else
  echo "user $1"
fi
if ping -c1 -W1 db >/dev/null; then echo up; fi

Spaces inside the brackets are required. Inside [[ ]] unquoted variables are not split, but quoting is still a good habit, and quoting the right side of == makes it a literal string instead of a pattern.

Practice. Write a script that checks its argument and prints different messages for empty, numeric and other input.

Bash Operators Core

Bash has separate operator sets for strings, numbers, files and logic.

A classic bug is [[ $a > $b ]] for numbers: inside double brackets that compares strings alphabetically, so "10" is less than "9". Use -gt or (( a > b )).

Practice. Write tests that show the difference between string and numeric comparison of 10 and 9.

Case Statements Core

A case statement matches a value against glob patterns and runs the first branch that fits. It is cleaner than a long if/elif chain, especially for command-line options and subcommands.

case "$1" in
  start)        start_app ;;
  stop|halt)    stop_app ;;
  -h|--help)    usage; exit 0 ;;
  *.tar.gz)     tar -xzf "$1" ;;
  *)            echo "unknown: $1" >&2; exit 2 ;;
esac

Each branch ends with ;;, | separates alternative patterns, and * catches everything else. Patterns are globs, not regular expressions.

Practice. Write a script with start, stop, status and help subcommands using case.

Loops Core

Bash has three loops. For loops iterate over a list; while loops repeat while a command succeeds; until loops repeat until a command succeeds.

for f in *.csv; do echo "$f"; done
for ((i = 1; i <= 5; i++)); do echo "$i"; done
for host in "${hosts[@]}"; do ssh "$host" uptime; done
while IFS= read -r line; do echo "$line"; done < list.txt
until curl -fs localhost:8080/health; do sleep 2; done

Never loop over $(ls) or $(cat file): names and lines with spaces are split apart. Loop over a glob for files and use while read for lines.

Practice. Write a loop that retries a flaky command up to five times with a delay between tries.

Break and Continue in Loops Core

break leaves a loop immediately; continue skips to the next iteration. Both take an optional number to act on outer loops in nested ones.

for f in logs/*; do
  [[ -s $f ]] || continue        # skip empty files
  if grep -q FATAL "$f"; then
    echo "fatal in $f"; break
  fi
done

Using continue early for cases you want to skip keeps the main body flat and readable, the same guard-clause style as in other languages.

Practice. Loop over files, skipping hidden or empty ones, and stop at the first one containing a given word.

Script Arguments and Positional Parameters Core

Script arguments arrive as positional parameters: $1, $2 and so on, with ${10} for two digits. $0 in shell scripting holds the script's name as it was called, which is useful in usage messages.

echo "script: $0"
echo "first: $1"
echo "number of script arguments: $#"
for arg in "$@"; do echo "[$arg]"; done
shift             # drop $1, move the rest down
shift 2

Script arguments using $@ must be quoted: "$@" keeps each argument separate even if it has spaces. $* joins all arguments into one string, separated by the first character of IFS. The shift builtin removes leading arguments, which is the basis of simple option loops.

Practice. Write a script that prints each argument in brackets, and compare "$@" with "$*" using an argument containing spaces.

Parsing Options Recommended

Real scripts accept flags. For short options, the getopts builtin handles combined flags and values.

verbose=0 out=out.txt
while getopts ":vo:h" opt; do
  case $opt in
    v) verbose=1 ;;
    o) out=$OPTARG ;;
    h) usage; exit 0 ;;
    *) usage >&2; exit 2 ;;
  esac
done
shift $((OPTIND - 1))   # the rest are positional

For long options such as --output, loop over arguments with case and shift. Always print a usage message for -h and for invalid input.

Practice. Add -v, -o FILE and -h options to one of your scripts with getopts.

Special Variables Core

Bash sets several special variables automatically.

• $?: exit status of the last command.
• $$: PID of the current shell, handy for unique temp names.
• $!: PID of the last background job.
• $#, $@ and $*: argument count and lists.
• $_: last argument of the previous command.
• $LINENO, $SECONDS, $RANDOM and $BASH_SOURCE: current line, seconds since start, a random number and the script's path.

"$(dirname "${BASH_SOURCE[0]}")" gives the directory of the script itself, which lets it find its own files wherever it is called from.

Practice. Write a script that starts a background sleep, prints its PID with $!, waits for it and reports the time taken.

Functions Core

Functions group commands under a name. They take arguments as $1, $2 and $@, just like scripts, and return an exit status with return.

log() { printf '%s %s\n' "$(date +%T)" "$*" >&2; }

is_up() {
  curl -fs "$1/health" >/dev/null
}

if is_up "localhost:8080"; then log "service up"; fi
version=$(get_version)      # return data via stdout

return only sets a status from 0 to 255; to return data, print it and capture it with command substitution. Define functions before calling them.

Practice. Refactor a long script into functions for logging, validation and the main work.

Variable Scopes and Function Scopes Core

Variable scopes in Bash are simple but surprising: variables are global by default, even when set inside a function. Function scopes start only when you declare variables with local.

count=0
bump() { local count=10; count=$((count + 1)); echo "$count"; }
bump            # 11
echo "$count"   # 0, untouched

Bash uses dynamic scoping: a function can see the local variables of the function that called it. Always use local inside functions to avoid one function silently changing another's data. Subshells, such as ( ... ) or the parts of a pipeline, get copies, so changes inside them do not survive.

Practice. Show a bug caused by a missing local, then fix it.

Recursive Functions Optional

Recursive functions call themselves. Bash supports them, though each call is slow and deep recursion can hit limits.

walk() {
  local dir=$1 entry
  for entry in "$dir"/*; do
    if [[ -d $entry ]]; then walk "$entry"
    else echo "$entry"; fi
  done
}
factorial() { (( $1 <= 1 )) && echo 1 || echo $(( $1 * $(factorial $(( $1 - 1 ))) )); }

Use local for every variable in a recursive function, or calls will overwrite each other. For directory walks, find is faster and simpler; recursion is mainly useful to show you understand scopes.

Practice. Write a recursive function that prints a directory tree with indentation by depth.

Arrays and Associative Arrays Core

Indexed arrays hold ordered lists; associative arrays map keys to values and need declare -A, available from Bash 4.

hosts=(web1 web2 "db 1")
hosts+=(cache1)
echo "${hosts[0]}" "${#hosts[@]}"
for h in "${hosts[@]}"; do echo "$h"; done

declare -A port=([web]=80 [db]=5432)
port[cache]=6379
for k in "${!port[@]}"; do echo "$k=${port[$k]}"; done
mapfile -t lines < file.txt     # file into array

Quote "${arr[@]}" to keep elements with spaces intact. Arrays are a good way to build command arguments safely instead of a single string.

Practice. Count file extensions in a directory with an associative array and print them sorted by count.

String Manipulation Core

Parameter expansion handles most string manipulation without calling external tools.

f="/var/log/app/error.log"
echo "${#f}"            # string length
echo "${f:9:3}"         # substring extraction: app
echo "${f##*/}"         # error.log (strip longest prefix)
echo "${f%.log}"        # remove suffix
echo "${f%/*}"          # directory part
echo "${f/error/access}"   # pattern replacement, first
echo "${f//\//_}"        # replace all slashes

The # and % forms remove the shortest match from the start or end, and doubled forms remove the longest. These are fast because they run inside Bash rather than starting sed or basename.

Practice. Rename every file.JPG in a directory to file.jpg using only parameter expansion in a loop.

Numeric Data in Bash Core

Bash does integer maths only. Arithmetic expansion, $(( )), is the modern way; (( )) on its own evaluates a condition or updates a variable.

total=$(( price * qty + 5 ))
(( count++ ))
(( size > 100 )) && echo big
echo $(( 17 / 5 )) $(( 17 % 5 )) $(( 2 ** 10 ))   # 3 2 1024
let "n = n + 1"            # let command, older style
expr 3 + 4                 # expr, external and legacy
echo "scale=2; 10 / 3" | bc   # 3.33

Arithmetic operators in Bash include + - * / % **, comparisons and bit operators. Division truncates. The let command and expr still appear in old scripts, but prefer $(( )). For decimals use bc or awk.

Practice. Write a script that computes the average file size in a directory to two decimal places.

Error Handling with set -e, set -u and set -o Core

By default Bash keeps going after a failed command, which can turn one error into a disaster. Strict mode changes that.

• set -e: exit when a command fails. It does not trigger in if tests or before || and &&, which is why it is a safety net, not full error handling.
• set -u: treat unset variables as an error, catching typos such as $DIRR.
• set -o pipefail: a pipeline fails if any part fails. set -o on its own lists all options.

set -euo pipefail
cp "$src" "$dest" || { echo "copy failed" >&2; exit 1; }

Check important commands explicitly with || and a clear message.

Practice. Run a script that uses an undefined variable and a failing pipeline with and without strict mode.

Trap and Stop Execution Core

trap runs code when the script receives a signal or exits. Use it to delete temporary files, release locks or print where a failure happened.

tmp=$(mktemp -d)
cleanup() { rm -rf "$tmp"; }
trap cleanup EXIT
trap 'echo "failed at line $LINENO" >&2' ERR
trap 'echo interrupted; exit 130' INT TERM

To stop execution deliberately, call exit with a non-zero code from the script, or return from a function. The EXIT trap still runs, so clean-up always happens. Create temp files with mktemp, never with a fixed name in /tmp.

Practice. Write a script that creates a temp directory, is interrupted with Ctrl+C, and still removes it.

Error Logging Recommended

Good scripts tell you what went wrong and where. Send messages to stderr, include a timestamp and level, and also keep a log file for scripts that run unattended.

log()  { printf '%s [%s] %s\n' "$(date '+%F %T')" "$1" "${*:2}" >&2; }
die()  { log ERROR "$@"; exit 1; }
exec > >(tee -a /var/log/backup.log) 2>&1   # copy all output to a file
log INFO "starting backup"
[[ -d $src ]] || die "source $src missing"

For scripts run by systemd, write to stderr and let the journal store it, or use logger to send messages to the system log.

Practice. Add log and die helpers to a script and make it write all output to a log file as well as the screen.

Bash Debug Core

Debugging Bash starts with tracing. set -x, or running bash -x script.sh, prints each command after expansion with a + prefix, so you see the real values used.

bash -n script.sh                       # bash -n: syntax check, no run
bash -x script.sh arg1
PS4='+ ${BASH_SOURCE}:${LINENO}: ' bash -x script.sh
set -x; risky_part; set +x               # trace one section

Setting PS4 adds file and line numbers to the trace. Common causes of bugs: unquoted variables, spaces around =, missing spaces inside [[ ]], Windows line endings, and scripts that work in your shell but not in cron because of a different PATH.

Practice. Debug a script that fails only for file names with spaces using bash -x.

Shellcheck Core

Shellcheck reads your script without running it and reports likely bugs. It finds unquoted variables, useless cat, wrong test syntax, unreachable code, Bash features in sh scripts and many subtle bugs, each with a code you can look up.

shellcheck deploy.sh
# SC2086: Double quote to prevent globbing and word splitting.
# shellcheck disable=SC2034   # silence one warning, with a reason

Run it in your editor and in CI so every script is checked. Formatters such as shfmt keep style consistent. Disable a warning only with a comment explaining why.

Practice. Run shellcheck on three of your scripts and fix every warning.

ps, top and htop Core

Process management starts with seeing what runs. ps prints a snapshot; top and htop show a live, sorted view.

ps aux                   # every process
ps -ef --forest          # with parent tree
ps -o pid,%cpu,%mem,cmd -C python3
pgrep -fa worker         # find by command line
top                      # P sorts by CPU, M by memory, k kills
htop                     # colours, mouse, tree view

Key columns are PID, user, CPU and memory use, state and command. kill PID sends SIGTERM to ask a process to stop; use kill -9 only when it ignores that.

Practice. Find the process using the most CPU, show its parent and stop it politely.

Jobs: fg and bg, nohup and disown Core

Jobs are commands your shell started. Add & to start one in the background. Ctrl+Z suspends the foreground job.

./build.sh &
jobs -l          # list jobs with PIDs
fg %1            # bring to foreground
bg %1            # resume a stopped job in background
wait             # wait for all background jobs
nohup ./sync.sh > sync.log 2>&1 &
disown %1        # detach a running job from the shell

Background jobs normally receive a hang-up signal when you close the terminal. nohup starts a command immune to that, and disown removes an already running job from the shell's list. For long work over SSH, tmux is usually better.

Practice. Run three slow commands in parallel with &, wait for all of them, and print the total time.

cron and crontab Core

cron runs commands on a schedule. Each user has a crontab, edited with crontab -e. A line has five time fields then the command: minute, hour, day of month, month and day of week.

# m  h  dom mon dow  command
30   2  *   *   *    /home/asha/bin/backup.sh >> /home/asha/backup.log 2>&1
*/15 *  *   *   1-5  /usr/local/bin/check-api.sh
crontab -l           # list

Cron runs with a minimal environment and a short PATH, so use full paths, set variables in the script, and redirect output to a log or you will not see errors. Use flock to stop overlapping runs. at schedules a one-off command: echo ./report.sh | at 17:00.

Practice. Schedule a script to run every ten minutes, log its output and confirm it ran.

Systemd Timers Recommended

Systemd timers are the modern alternative to cron on systemd-based Linux. A timer unit triggers a matching service unit.

# /etc/systemd/system/backup.timer
[Timer]
OnCalendar=*-*-* 02:30:00
Persistent=true
[Install]
WantedBy=timers.target

systemctl enable --now backup.timer
systemctl list-timers

Advantages over cron: output goes to the journal, the job can use service settings such as user, resource limits and dependencies, and Persistent=true runs missed jobs after downtime. Cron remains simpler for quick personal jobs.

Practice. Convert one cron job into a systemd service and timer and check its log with journalctl.

Package Management Core

Each platform has a package manager that installs, upgrades and removes software with dependencies.

On modern Red Hat systems, yum is an alias for dnf. In scripts, use non-interactive flags such as apt-get install -y and check whether a command exists with command -v jq before installing.

Practice. Write a script that installs jq with whichever package manager the machine has.

System Monitoring Commands Core

A few commands answer most "is the machine healthy?" questions.

uptime          # how long up, load averages
free -h         # memory; read the available column
df -h           # disk space per filesystem
du -sh *        # size of each item here
vmstat 1 5      # CPU, memory, swap and I/O each second
iostat -x 1 3   # disk utilisation, from sysstat

df and du commands differ: df reports filesystem totals, du adds up files. Load averages from Uptime should be compared with the CPU count. iostat and vmstat show whether a slow machine is waiting on CPU, memory or disk.

Practice. Write a script that prints load, free memory and the fullest filesystem in one line.

Ping, ifconfig and ip Commands, netstat and ss Core

Ping tests whether a host replies and how long it takes. For interfaces and addresses, ifconfig and ip commands both exist, but ifconfig is deprecated on Linux; use ip. Likewise netstat and ss list sockets, and ss is the modern tool.

ping -c 3 example.com
ip -br addr          # interfaces and addresses
ip route             # default gateway
ss -tulpn            # listening ports and processes
nc -zv db 5432       # test a TCP port

A failed ping does not always mean a host is down, because ICMP may be blocked; test the real port too. The Linux Roadmap covers networking in more depth.

Practice. Find your IP address, default gateway and every listening port with the process behind it.

curl and wget Core

curl transfers data over HTTP and many other protocols and is the standard way to call APIs from the shell. wget focuses on downloading files and can fetch recursively and resume.

curl -fsSL "$url" -o tool.tar.gz       # fail on HTTP errors, follow redirects
curl -s "$api/users/1" | jq '.name'
curl -X POST -H 'Content-Type: application/json' \
     -d '{"name":"asha"}' "$api/users"
curl -I "$url"                          # headers only
wget -c "$url"                          # resume download

Use -f in scripts so HTTP errors give a non-zero exit code. Never pipe a downloaded script straight into a shell from a source you have not checked.

Practice. Write a script that calls a public JSON API with curl, extracts a field with jq and handles HTTP errors.

SSH, scp and rsync Core

SSH opens an encrypted shell on a remote machine and can run single commands. Use key-based login with ed25519 keys.

ssh deploy@web1 'df -h /'
ssh -i ~/.ssh/id_ed25519 -p 2222 user@host
scp report.csv user@host:/tmp/
rsync -avz --delete ./site/ user@host:/srv/site/
rsync -avn src/ dest/      # dry run first

scp copies single files simply. rsync transfers only differences, preserves permissions, resumes and can mirror directories; the trailing slash on the source changes whether the folder itself is copied. Keep hosts in ~/.ssh/config with aliases.

Practice. Set up key-based SSH to a VM, add an alias in ~/.ssh/config and mirror a folder with rsync after a dry run.

Archives: tar, gzip, bzip2, xz and zip Core

tar bundles files into one archive, keeping permissions and directory structure, and can compress at the same time.

tar -czf app.tar.gz app/     # create with gzip
tar -cJf app.tar.xz app/     # create with xz
tar -tf app.tar.gz           # list
tar -xzf app.tar.gz -C /opt  # extract elsewhere
gzip big.log; gunzip big.log.gz
zip -r site.zip site/; unzip site.zip

gzip and gunzip are fast and universal. bzip2 and xz compress smaller but more slowly; xz is common for software releases. zip and unzip suit sharing with Windows users, and zip compresses each file separately.

Practice. Compress the same directory with gzip, bzip2 and xz and compare size and time.