POSIX stands for "Portable Operating System Interface". It is a set of standards designed to maintain compatibility between various UNIX and UNIX-like operating systems. POSIX defines everything from system-level APIs and user facing interfaces. These standards make it easier for us to share programs.
In order to save ourselves time and effort, we should focus on writing portable code. In computing, "Portability" is the measure of how easy it is to transfer a program to a different environment. Since POSIX is a formal specification implemented by any operating system worth using, it is the Lowest Common Denominator. Writing POSIX programs means that we can write our program once, copy it to any system we might want to run it on, then run it with little to no extra effort required. POSIX means compatibility.
Although there are many shells to choose from, we should avoid any non-POSIX shells when scripting. Shells like bash , csh , zsh , and fish are not POSIX and therefore programs we write in them are not compatible with a pure POSIX system. Specifically, we must avoid bash and bashisms . bash is a popular shell but it is only present on systems infected with GNU and related GPL software. If we write our programs in bash they will not run on systems without GNU. We must target the lowest common denominator (POSIX) if we want our programs to be usable by everyone, everywhere.
If you want official documentation, see The POSIX sh specification
man 1p to get the POSIX manual for a command [[ option ]] is bash exclusive. Use [ option ] instead. If you do not care to do any preliminary research whatsoever, know that a UNIX system is a collection of small programs used via the shell. The user interacts with a UNIX system by the shell and can automate the system by creating shell scripts which are a series of commands placed into a script file. The shell is a fully featured programming language that makes it easy to create new programs on top of the existing UNIX utilities. All of this means that we can easily write simple programs without ever needing to learn a more complicated language like C. For most basic programs, shell scripts are more than enough.
Before we go any further, we should verify which shell we are running. Typically, we can just run echo $SHELL bu this will only tell us our default shell. In order to get the currently running shell , we should run echo $0 . The $0 variable contains the name of the currently running program. Below is an example of some various shells.
user@localhost:~ % echo $SHELL
/bin/tcsh
user@localhost:~ % bash
[user@localhost ~]$ echo $SHELL
/bin/tcsh
[user@localhost ~]$ echo $0
bash
[user@localhost ~]$ sh
$ echo $SHELL
/bin/tcsh
$ echo $0
sh
$ ^D
[user@localhost ~]$
exit
user@localhost:~ %
Before we continue, we also need to verify that /bin/sh is actually a real program. If you have a GNU system, /bin/sh is actually a symlink to bash. If you are on a GNU system, you must start bash in POSIX mode with bash --posix . Modify your shebang accordingly. When you see #!/bin/sh , replace it with #!/bin/bash --posix . Additionally, when you see sh ./script.sh , replace it with bash --posix ./script.sh
This is a POSIX system
user@localhost:~ % file /bin/sh
/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300139), FreeBSD-style, stripped
user@localhost:~ %
And this is a GNU system
[user@localhost ~]$ file /bin/sh
/bin/sh: symbolic link to bash
[user@localhost ~]$
A shell script is quite literally a list of commands we want the shell to run for us. If we are clever, we can use this power to automate a series of actions. If we are sub-genius, we can use this power to write fully featured programs.
To write a shell script, we simply create a plaintext file and open it in any editor. I will be using vim but you should replace this with your editor of choice. Typically, a shell script will end in the .sh extension. This is not required but it will help future you quickly determine what is and isn't a shell script.
user@localhost:~ % vim script.sh
UNIX Graybeards often refer to the first two characters of a shell script as a shebang . Some people call it a crunchbang but I call it a hashbang . These names are largely meaningless . . . but it's useful to remember that they stand for #! . Every shell script starts with a #! followed by whatever program you want to execute. Typically, the first like will be #!/bin/sh but if we wanted to create an awk script we could just as easily use #!/usr/local/bin/awk .
#!/bin/sh
First, let's write a simple shell script that tells us where we are at and lists our files. Notice that it's quite literally just a list of commands separated by newlines.
#!/bin/sh
pwd
ls
In order to run this script, we have two options. We can simply type sh script.sh
user@localhost:~ % sh script.sh
/home/user
R cr public_html src
bin mbox script.sh
user@localhost:~ %
Or, if we want to run this program like a normal program, we can chmod u+x it then run it with ./script.sh
user@localhost:~ % chmod u+x ./script.sh
user@localhost:~ % ./script.sh
/home/user
R cr public_html src
bin mbox script.sh
user@localhost:~ %
echo is a command that allows us to print some text onto the screen. It's useful in that it can help us format our text and communicate progress ot the user. Let's modify script.sh from before.
#!/bin/sh
echo "you are here:"
pwd
echo "here are your files:"
ls
user@localhost:~ % ./script.sh
you are here:
/home/user
here are your files:
R cr public_html src
bin mbox script.sh
user@localhost:~ %
hmmm, we might be able to improve this output with command substitution . . .
The shell allows us to use command substitution. Command substitution allows us to place a command within what we tell echo to say. This allows for greater formatting.
#!/bin/sh
echo "you are here: $(pwd)"
echo "here are your files: "
ls
user@localhost:~ % ./script.sh
you are here: /home/user
here are your files:
R cr public_html src
bin mbox script.sh
user@localhost:~ %
In shell scripts, quotation marks are important. When we surround multiple words in quotes, we create a string . Usually, the shell will split up our arguments, treating whitespace as a delimiter. To avoid chopping up our sentences, we use quotes. But there is a caveat: if we want to use quotation symbols inside our strings we must either mix quotation marks or escape them.
We can choose to begin and end our string with either a " or a ' . For now, this choice doesn't matter. Let's choose the " character. Notice how our doublequote is completely ignored
user@localhost:~ % echo "echo says "hello""
echo says hello
user@localhost:~ % echo "echo says 'hello'"
echo says 'hello'
user@localhost:~ %
Alternative, we can use escape sequences. To escape the double quote, we can use the \ character. On some UNIX systems, echo cannot handle escape sequences. Instead, we can use the POSIX program printf . printf is somewhat strange in that escapes only work when our string is surrounded in single quotes. We must also supply a \n escape sequence if we want a newline after printf finishes.
user@localhost:~ % echo "echo says "hello""
echo says hello
user@localhost:~ % echo "echo says \"hello\""
Unmatched '"'.
user@localhost:~ % printf "echo says \"hello\""
Unmatched '"'.
user@localhost:~ % printf 'echo says \"hello\"'
echo says "hello"user@localhost:~ %
user@localhost:~ % printf 'echo says \"hello\"\n'
echo says "hello"
user@localhost:~ %
If you can, try to prefer to mix quotation marks instead of use escape sequences. Escape sequences are finicky and can be very difficult for humans to understand quickly.
In the shell, variables start with the $ character. We can place variables inside of strings
#!/bin/sh
echo "My home directory is '$HOME' and my username is '$USER'"
printf "here are my files: \n$(ls) \n"
user@localhost:~ % ./script.sh
My home directory is '/home/user' and my username is 'user'
here are my files:
R
bin
cr
mbox
public_html
script.sh
src
user@localhost:~ %
The shell supports variables but there are some caveats. The name for a variable can contain letters, numbers, and underscore characters. Variable names cannot start with a space or a number. Also, spaces cannot be placed between the variable name, equals sign, and the value of the variable.
Good variables: | Bad variables:
------------------------+---------------
x=1 | x= 1
y=2 | y = 2
baz="foo bar" | baz=foo bar
executables=$(ls /bin) | exes =$(ls /bin)
let's add some variables to our shell script
#!/bin/sh
homedir="My home ditectory is"
username="and my username is"
echo "$homedir '$HOME' $username '$USER'"
user@localhost:~ % ./script.sh
My home ditectory is '/home/user' and my username is 'user'
user@localhost:~ %
Comments allow us to annotate and explain out code. In shell, everything that comes after a # character is a comment. We can comment entire lines or only part of lines. Comments are also useful because they let us remove segments of code for testing and debugging purposes. Keep in mind, we can still escape the # character.
#!/bin/sh
# this is a comment
echo "hello world" # this line echoes hello world
# this function commented out because it's broken
#files=$(ls)
#for file in $files; do
#echo $flie
#done
echo \# done
user@localhost:~ % ./script.sh
hello world
# done
user@localhost:~ %
The ; character can be used to separate our commands in a way that lets us write multiple commands on the same line. Unlike the && pipe, the ; separator will execute the following commands regardless of the exit code of the program before it.
user@localhost:~ % echo "hey mom" | figlet; echo "I'm on" | figlet; echo "UNIX" | figlet;
_
| |__ ___ _ _ _ __ ___ ___ _ __ ___
| '_ \ / _ \ | | | | '_ ` _ \ / _ \| '_ ` _ \
| | | | __/ |_| | | | | | | | (_) | | | | | |
|_| |_|\___|\__, | |_| |_| |_|\___/|_| |_| |_|
|___/
___ _
|_ _( )_ __ ___ ___ _ __
| ||/| '_ ` _ \ / _ \| '_ \
| | | | | | | | | (_) | | | |
|___| |_| |_| |_| \___/|_| |_|
_ _ _ _ _____ __
| | | | \ | |_ _\ \/ /
| | | | \| || | \ /
| |_| | |\ || | / \
\___/|_| \_|___/_/\_\
user@localhost:~ %
the || and && characters can be used to control flow. The || operator means "Or" and the && operator means "and"
The && operator will execute the command that follows it only if the exit code of the command before it is equal to 0.
The || operator will execute the command that follows it only if the exit code of the command before it is not equal to 0.
#!bin/sh
true && echo "this line is printed"
false || echo "this line also printed"
false && echo "this line is not printed"
true || echo "this line is not printed"
user@localhost:~ % sh ./control.sh
this line is printed
this line also printed
user@localhost:~ %
Just like how we can write multiple commands on one line, we can write one command on multiple lines using the \ character to escape the newlines. In this example, we are listing all users with a UID greater than 1000. In UNIX, all non-daemon users have a UID greater than 1000.
user@localhost:~ % cat /etc/passwd | sed s/:/\ /g | awk '$3>1000{print$1}'
nobody
user
nuit
hadid
ra-hoor-khuit
is equivalent to
user@localhost:~ % cat /etc/passwd | \
? sed s/:/\ /g | \
? awk '$3>1000{print$1}'
nobody
nuit
hadid
ra-hoor-khuit
user@localhost:~ %
When we log in, our shell provides some basic environmental variables. They are called environmental variables because they provide information about (and configure) our environment.
We can list them using either the env or set command.
user@localhost:~ % env
USER=user
LOGNAME=user
HOME=/home/user
MAIL=/var/mail/user
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/home/user/bin
TERM=xterm-256color
BLOCKSIZE=K
MM_CHARSET=UTF-8
LANG=C.UTF-8
SHELL=/bin/tcsh
SSH_CLIENT=::1 58353 22
SSH_CONNECTION=::1 58353 ::1 22
SSH_TTY=/dev/pts/4
HOSTTYPE=FreeBSD
VENDOR=amd
OSTYPE=FreeBSD
MACHTYPE=x86_64
SHLVL=1
PWD=/home/user
GROUP=user
HOST=localhost.localdomain
REMOTEHOST=localhost
EDITOR=vim
PAGER=less
user@localhost:~ %
the set command provides much more information
user@localhost:~ % set
_ env
addsuffix
anyerror
argv ()
autoexpand
autolist ambiguous
autorehash
cdtohome
csubstnonl
cwd /home/user
dirstack /home/user
echo_style bsd
edit
euid 1002
euser user
filec
gid 1003
group user
history 1000
home /home/user
killring 30
loginsh
mail /var/mail/user
owd
path (/sbin /bin /usr/sbin /usr/bin /usr/local/sbin /usr/local/bin /home/user/bin)
prompt %{\033[1;32m%}%N@%m:%{\033[1;34m%}%~ %#%{\033[0m%}
prompt2 %R?
prompt3 CORRECT>%R (y|n|e|a)?
promptchars %#
savehist (1000 merge)
shell /bin/tcsh
shlvl 1
status 0
tcsh 6.21.00
term xterm-256color
tty pts/4
uid 1002
user user
version tcsh 6.21.00 (Astron) 2019-05-08 (x86_64-amd-FreeBSD) options wide,nls,dl,al,kan,sm,rh,color,filec
user@localhost:~ %
$USER your username
$LOGNAME your login name
$HOME /path/to/your/home/directory
$MAIL /path/to/your/mailbox
$PATH series of directories your shell searches for programs in
$TERM terminal type
$LANG your locale
$SHELL your login shell
$OSTYPE the UNIX variant on your computer
$MACHTYPE your CPU's architecture
$PWD the directory you are currently in
$GROUP the primary group your user is in
$HOST the hostname of your computer
$EDITOR your default editor
$PAGER your default pager
We can combine our variables with strings using quotes and curly braces
user@localhost:~ % echo $OSTYPE" is cool"
FreeBSD is cool
user@localhost:~ % echo ${OSTYPE}\'s fast
FreeBSD's fast
user@localhost:~ %
Sometimes we want to remove a variable from our program. There are a couple ways to do it.
The first way keeps the variable in our program but removes the value we've assigned to it.
variable=
The second way actually removes the variable from our program entirely
unset variable
Here is a demonstration. We are using the set command to list all variables, then piping it's output through grep to check if there is a match. The grep command will return 1 if there were no matches.
#!/bin/sh
variable="hello world"
set | grep variable
echo "Exit code: $?"
variable=
set | grep variable
echo "Exit code: $?"
unset variable
set | grep variable
echo "Exit code: $?"
user@localhost:~ % sh ./vars.sh
variable='hello world'
Exit code: 0
variable=''
Exit code: 0
Exit code: 1
user@localhost:~ %
We can pass arguments to our shell script just like with other commands. These are indexed, in order, as variables. $1 represents the first argument, $2 represents the second argument, $3 represents the third, etc. $0 is special in that it contains the name of the program.
Let's write a script called rename.sh that we can use to rename files.
#!/bin/sh
mv -i $1 $2
user@localhost:~ % touch foo
user@localhost:~ % ./rename.sh foo bar
user@localhost:~ % ls
R bin mbox recycle.sh script.sh
bar cr public_html rename.sh src
user@localhost:~ % touch baz
user@localhost:~ % ./rename.sh baz bar
overwrite bar? (y/n [n]) y
user@localhost:~ % ls
R bin mbox recycle.sh script.sh
bar cr public_html rename.sh src
user@localhost:~ %
Let's write a script called recycle.sh that we can use to place our files into a recycling bin. This script contains some concepts we have not covered yet.
$? is a variable that holds the exit code of the last program. Every well behaved program in UNIX has an exit code. An exit code other than 0 indicates an error.
We are also using an if statement to compare the exit code of the last command (the move command) to zero. The -eq flag stands for "equals". If $? == 0 , then we print the success message. We'll learn more about this later.
#!/bin/sh
recyclebin="$HOME/.recyclebin"
# uncomment if $HOME/.recyclebin does not exist
# mkdir $recyclebin
# move file to recycle bin
mv $1 $recyclebin/$1
# if the file was sucessfully moved, print a success message
if [ $? -eq 0 ]; then
echo $0 has sucessfully recycled $1
fi
user@localhost:~ % ./recycle.sh foo
./recycle.sh has sucessfully recycled foo
user@localhost:~ % ./recycle.sh bar
mv: rename bar to /home/user/.recyclebin/bar: No such file or directory
user@localhost:~ %
The $# variable represents the number of arguments we passed to our program. We can use this to loop through all of the supplied arguments. The $@ variable represents all command line arguments.
In some shells, a plain old $@ gets ignored so for maximum portability we should use ${1+"$@"} instead. This format basically translates to "if $1 is defined, do something. If $i is not defined, do nothing."
We are also using a for loop here. For loops iterate until some condition is met. We'll learn more about this later.
#!/bin/sh
echo "total number of arguments: $#"
for i in ${1+"$@"}; do
echo $i
done
user@localhost:~ % ./counter.sh foo bar baz
total number of arguments: 3
foo
bar
baz
user@localhost:~ %
In programming, a loop statement can be used to repeat a command n times. This means we can write less code that does more work. When we add logic statements, we can make decisions. Combining loops and logic allows us to create very powerful tools.
Comparison operators in the shell might be confusing at first. Intead of using mathematical operators, we use a - character followed by whaver condition we are testing for. Mathetimacal operators are bashisms and should be avoided to ensure maximum portability.
These operators are used to compare two variables or values. We can use these expressoins to make decisions in our programs.
| mathematical expression | shell equivalent | what it does | Does the mathematical expression works in POSIX shell? |
|---|---|---|---|
| == | -eq | tests if equal | YES |
| != | -nq | tests if not equal | YES |
| < | -lt | tests if less than | NO |
| > | -gt | tests if greater than | NO |
| <= | -le | tests if less than or equal to | NO |
| >= | -ge | tests if greater than or equal to | NO |
An if statement allows us to make decisions based on some condition. It follows the general structure of if expression is equal to value then do action . Notice how each if statement ends with "fi". "fi" is "if" but backwards. Notice how the statement do continue after one of the checks is true.
#!/bin/sh
a=$1
b=$2
if [ $a -eq $b ]
then
echo "they are equal"
fi
if [ $a -ne $b ]
then
echo "they are not equal"
fi
user@localhost:~ % sh ./equality.sh 10 20
they are not equal
user@localhost:~ % sh ./equality.sh 10 10
they are equal
user@localhost:~ %
What if we wanted to write a more concise if statement? We can use the "elif" statement. "elif" is short for "else if" and is used to check for a secondary condition if the first test is false. Notice how the statement does not continue after one of the checks is true. The "else" statement is used as a backup of sorts. If none of the statements before the "else" are true, then whatever is inside of the else block is done.
Also notice how we are placing if and then on the same line. We are separating them with the ; character.
#!/bin/sh
a=$1
b=$2
if [ $a -eq $b ]; then
echo "they are equal"
elif [ $a -ne $b ]; then
echo "they are not equal"
elif [ $a -eq 19 ] && [ $b -eq 70 ]; then # check if $a is 19 AND $b is 70
echo "$a$b, an easter egg for the UNIX epoch"
else
echo "some error occured"
fi
user@localhost:~ % sh equality.sh 19 70
they are not equal
user@localhost:~ %
Let's modify our shell script so that we can see the easter egg. We need to move the conditional with the highest priority to the top of our if-else-chain. Let's also add an exit code to our final else statement to indicate that there was some error in the program if it happens to go through all our checks without matching one.
#!/bin/sh
a=$1
b=$2
if [ $a -eq 19 ] && [ $b -eq 70 ]; then # check if $a is 19 AND $b is 70
echo "$a$b, an easter egg for the UNIX epoch"
elif [ $a -eq $b ]; then
echo "they are equal"
elif [ $a -ne $b ]; then
echo "they are not equal"
else
echo "some error occured"
exit 1 # exit with an error
fi
user@localhost:~ % sh ./equality.sh 19 70
1970, an easter egg for the UNIX epoch
user@localhost:~ %
A case statement (sometimes called a switch statement) is similar to to an if-else statement but with more options. Notice how we end the case statement with 'esac'. 'esac' is just 'case' spelled backwards. Every block ends with the ;; character. These mean "Hey we are done doing $stuff, go to the next check".
#!/bin/sh
printf "Answer Yes or No: "
read userinput # read command gets input from the user
case $userinput in
yes | YES ) # if $userinput matches yes or YES
echo "You answered yes"
;;
no | NO ) # if $userinput matches no or NO
echo "You answered no"
;;
esac
user@localhost:~ % sh cases.sh
Answer Yes or No: yes
You answered yes
user@localhost:~ % sh cases.sh
Answer Yes or No: YES
You answered yes
user@localhost:~ % sh cases.sh
Answer Yes or No: no
You answered no
user@localhost:~ % sh cases.sh
Answer Yes or No: No
user@localhost:~ %
Hmmm, looks like some of our arguments got ignored. Since we are only checking for all caps or all lowercase, a mixed case word is ignored. Let's use some regex to match more generously.
Instead of an "else" catching anything that slips through the cracks, a case statement uses "default". Just like in regex, the * character matches everything. Let's also add a "default" block to catch any user input that doesn't get matched.
#!/bin/sh
printf "Answer Yes or No: "
read userinput # read command gets input from the user
case $userinput in
[yY][eE][sS] ) # check for mixed case y-e-s
echo "You answered yes"
;;
[nN][oO] ) # check for mixed case n-o
echo "You answered no"
;;
*) # what to do if we didn't get a match
echo "I couldn't understand '$userinput'"
exit 1 # exit with an error
;;
esac
user@localhost:~ % sh cases.sh
Answer Yes or No: yEs
You answered yes
user@localhost:~ % sh cases.sh
Answer Yes or No: No
You answered no
user@localhost:~ % sh cases.sh
Answer Yes or No: YeS
You answered yes
user@localhost:~ % sh cases.sh
Answer Yes or No: I don't want to answer
I couldn't understand 'I don't want to answer'
user@localhost:~ %
The "for", "while", and "until" loops execute the same lines of code more than once. The "break" command will immediately exit the loop. The "continue" command will immediately restart the loop from the top. Keep this in mind as we go forward.
A "while" loop runs while some condition is true. When the condition is no longer true, the loop stops. A while loop is useful for processing user input or for executing a section of code multiple times until the desired number of iterations is achieved.
#!/bin/sh
i=0
# loop while $i is less than or equal to 10
while [ $i -le 10 ]; do
echo "Loop #$i"
i=$(( i + 1 )) # increment $i by 1 every loop
done
user@localhost:~ % sh while.sh
Loop #0
Loop #1
Loop #2
Loop #3
Loop #4
Loop #5
Loop #6
Loop #7
Loop #8
Loop #9
Loop #10
user@localhost:~ %
We can also use while loops to create infinite loops.
#!/bin/sh
while [ true ]; do
echo "I can't stop aaaaAAA"
done
user@localhost:~ % sh while.sh
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
I can't stop aaaaAAA
^C
user@localhost:~ %
Let's use a while loop to generate all 256 colors supported by a 256-color terminal emulator. We can use ANSI escape sequences to make the colors
#!/bin/sh
i=0
while [ $i -le 255 ]; do
printf "\033[48:5:${i}m \033[0m"
i=$(( i + 1 ))
done
printf "\n"
user@localhost:~ % ./256test.sh
user@localhost:~ %
An until loop is very similar to a while loop. An until loop will continue looping until some condition is met. Some people consider until loops to be more danger than while loops but they're almost identical.
#!/bin/sh
i=0
# loop until $i is equal to 10
until [ $i -eq 10 ]; do
echo "Loop #$i"
i=$(( i + 1 )) # increment $i by 1 every loop
done
user@localhost:~ % sh while.sh
Loop #0
Loop #1
Loop #2
Loop #3
Loop #4
Loop #5
Loop #6
Loop #7
Loop #8
Loop #9
user@localhost:~ %
Let's use an until loop to generate all 256 colors supported by a 256-color terminal emulator.
#!/bin/sh
i=0
until [ $i -eq 256 ]; do
printf "\033[48:5:${i}m \033[0m"
i=$(( i + 1 ))
done
printf "\n"
user@localhost:~ % ./256test.sh
user@localhost:~ %
A for loop operates on a list of items. It's useful for iterating over a series of files or a string delimited by spaces. Since there are no arrays in POSIX shell, we can use a string instead. A for loop will do an action for every item in the list we provide.
#!/bin/sh
# iterating over a list
for item in 1 2 3 4 5; do
echo "Loop #$item"
done
# no arrays, only strings :)
string="one two three four five"
#iterating over a list of items stored in a variable
for item in $string; do
echo "Number $item"
done
user@localhost:~ % sh for.sh
Loop #1
Loop #2
Loop #3
Loop #4
Loop #5
Number one
Number two
Number three
Number four
Number five
user@localhost:~ %
This script is . . . not entirely portable. It requires a program called 'viu'. You can probably use your package manager to install it. If not, you can build it from the viu repo
#!/bin/sh
while true; do
# make pictures variable empty
pictures=
# fill pictures variable with a list of files, randomly sorted
pictures=$(ls Pictures/ | sort -R)
for item in $pictures; do # loop through files
clear # clear screen
viu Pictures/$item # display image with viu
sleep 2 # wait 2 seconds
done
done
But I think we can improve this gallery tool using other skills we've learned so far.
#!/bin/sh
# the path to our images is supplied the command line
path=$1
# use test command, -z flag tests if the string is empty
# if there are no arguments (ie empty string), then exit with an error code
if test -z $path ; then
echo "you must supply a path to your images"
exit 1
fi
echo "press enter to go to the next image"
echo "press q then enter to quit"
echo
echo "press enter to continue"
# wait for enter key
read keypress
# fill pictures variable with a list of files
pictures=$(ls -d ${PWD}/${path}/*)
for item in $pictures; do # loop through files
clear # clear screen
viu $item # display image with viu
printf "\033[1mImage: \033[0m$item\n" # display image name
read keypress # wait for keypress
case $keypress in
q) # if q then enter, exit program
break
;;
*) # any key or enter, continue
continue
;;
esac
done
We don't need to remember fancy programs for very basic mathematical operations because we can call the eval command with a fancy trick that looks a lot like command substitution. Let's build a loop that calculates the Fibonacci sequence to demonstrate.
Here we are doing some math. We need four variables: a counter, the current number, the last number, and the last last number. We then run a test that will safely exit the program if the user does not specify a number of operations. After that, we run a loop. "Continue looping while the number of iterations is less than the number specified by the user". Inside this loop, we do the math required to generate a Fibonacci sequence. At the very end, we print a newline for clean output.
the spaces within the $(( [ math here ] )) are significant! It might work without spaces for you but the eval program on someone else's system might not work without the spaces!
#!/bin/sh
i=0 # iteration counter
a=1 # start at 1
b=0 # remember last number
c=0 # remember last last number
# safety check to prevent messy errors
if test -z $1; then
echo "please specify a number of iterations"
exit 1
fi
# while the number of iterations is less than or
# or equal to the number specified in the first argument, $1
while [ $i -le $1 ]; do
printf "$a " # print value
c=$b # last last value = last value
b=$a # last value = current
a=$(( $c + $b )) # current = last + last last
i=$(( $i + 1 )) # iterate counter by 1
done
printf "\n" # for formatting
user@localhost:~ % ./fib.sh
please specify a number of iterations
user@localhost:~ % ./fib.sh 10
1 1 2 3 5 8 13 21 34 55 89
user@localhost:~ % ./fib.sh 100
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1346269 2178309 3524578 5702887 9227465 14930352 24157817 39088169 63245986 102334155 165580141 267914296 433494437 701408733 1134903170 1836311903 2971215073 4807526976 7778742049 12586269025 20365011074 32951280099 53316291173 86267571272 139583862445 225851433717 365435296162 591286729879 956722026041 1548008755920 2504730781961 4052739537881 6557470319842 10610209857723 17167680177565 27777890035288 44945570212853 72723460248141 117669030460994 190392490709135 308061521170129 498454011879264 806515533049393 1304969544928657 2111485077978050 3416454622906707 5527939700884757 8944394323791464 14472334024676221 23416728348467685 37889062373143906 61305790721611591 99194853094755497 160500643816367088 259695496911122585 420196140727489673 679891637638612258 1100087778366101931 1779979416004714189 2880067194370816120 4660046610375530309 7540113804746346429 printf: Illegal option -6
1293530146158671551 printf: Illegal option -4
printf: Illegal option -3
printf: Illegal option -8
6174643828739884737 printf: Illegal option -2
3736710778780434371 1298777728820984005
user@localhost:~ %
Oh no! it looks like the number got too large and caused a buffer overflow in expr . Let's try to re-write this program so that it can calculate the first 100 numbers in the Fibonacci sequence.
bc is a slightly more robust and feature complete and extensible math program(ming language). We can remember bc by thinking that it stands for "basic calculator". By default, bc is very similar to expr but it supports larger numbers. We can do square roots, absolute values, arithmetic, etc.
Let's re-write our program to use bc Instead of using the method from the previous section, we are using command substitution to do our math.
#!/bin/sh
i=0 # iteration counter
a=1 # start at 1
b=0 # remember last number
c=0 # remember last last number
# safety check to prevent messy errors
if test -z $1; then
echo "please specify a number of iterations"
exit 1
fi
# while the number of iterations is less than or
# or equal to the number specified in the first argument, $1
while [ $i -le $1 ]; do
printf "$a " # print value
c=$b # last last value = last value
b=$a # last value = current
a=$( echo $c + $b | bc ) # current = last + last last
i=$( echo $i + 1 | bc ) # iterate counter by 1
done
printf "\n" # for formatting
user@localhost:~ % ./fib-bc.sh
please specify a number of iterations
user@localhost:~ % ./fib-bc.sh 10
1 1 2 3 5 8 13 21 34 55 89
user@localhost:~ % ./fib-bc.sh 100
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765 10946 17711 28657 46368 75025 121393 196418 317811 514229 832040 1346269 2178309 3524578 5702887 9227465 14930352 24157817 39088169 63245986 102334155 165580141 267914296 433494437 701408733 1134903170 1836311903 2971215073 4807526976 7778742049 12586269025 20365011074 32951280099 53316291173 86267571272 139583862445 225851433717 365435296162 591286729879 956722026041 1548008755920 2504730781961 4052739537881 6557470319842 10610209857723 17167680177565 27777890035288 44945570212853 72723460248141 117669030460994 190392490709135 308061521170129 498454011879264 806515533049393 1304969544928657 2111485077978050 3416454622906707 5527939700884757 8944394323791464 14472334024676221 23416728348467685 37889062373143906 61305790721611591 99194853094755497 160500643816367088 259695496911122585 420196140727489673 679891637638612258 1100087778366101931 1779979416004714189 2880067194370816120 4660046610375530309 7540113804746346429 12200160415121876738 19740274219868223167 31940434634990099905 51680708854858323072 83621143489848422977 135301852344706746049 218922995834555169026 354224848179261915075 573147844013817084101
user@localhost:~ %
By default, bc will automatically floor (ie round down to the nearest whole number). To get decimal precision, we can include the scale indicator in the string we echo to bc
user@localhost:~ % echo "1/3" | bc
0
user@localhost:~ % echo "scale=2; 1/3" | bc
.33
user@localhost:~ % echo "scale=4; 1/3" | bc
.3333
user@localhost:~ % echo "scale=1; .3 * 7" | bc
2.1
user@localhost:~ %
If we run bc -l , we will get support for trig, exponentials, logs, etc. If you need more than the basic arithmetic shown above, you should read man bc(1) . I cannot demonstrate it all here.
The test command is another useful tool for shell scripting. It can be used to easily test for a wide variety of things we might need in a script. If the test is true, test will return 0. If the test is false, test will return a 1.
I have included some of the most useful tests but you really should read the man page. It's much more useful than this.
| test | what it does |
|---|---|
| -e file | True if file exists (regardless of type). |
| -n string | True if the length of string is nonzero. |
| -s file | True if file exists and has a size greater than zero. |
| -z string | True if the length of string is zero. |
| file1 -ef file2 | True if file1 and file2 exist and refer to the same file. |
| string | True if string is not the null string. |
| s1 = s2 | True if the strings s1 and s2 are identical. |
| s1 != s2 | True if the strings s1 and s2 are not identical. |
The test command also supports multi-condition statements. The -a operator has higher precedence than the -o operator.
| ! expression | True if expression is false. |
| expression1 -a expression2 | True if both expression1 and expression2 are true. |
| expression1 -o expression2 | True if either expression1 or expression2 are true. |
| ( expression ) | True if expression is true. |
The expr command can be used for expression evaluation. If we pass a mathematical operation, expr will return the result. If we pass a any other type of operator, expr will return 0 for false, 1 for true, and 2 for a program error.
The mathematical operators are very simple but there is one caveat: you must place spaces around all operators and escape the following characters if you use them in your operation: *&|><()/ .
user@localhost:~ % expr 1 + 1
2
user@localhost:~ % expr 2 - 1
1
user@localhost:~ % expr 2 * 2
expr: syntax error
user@localhost:~ % expr 2 \* 2
4
user@localhost:~ % expr 4 \/ 2
2
user@localhost:~ % expr 7 % 5
2
The relational operators should be familliar. We've used them before. As with many other programs, expr will return 0 for a true statement and 1 for a false statement.
#!/bin/sh
# list of variables
a=$( expr 2 "=" 1 )
b=$( expr 2 ">" 1 )
c=$( expr 2 ">=" 1 )
d=$( expr 2 "<" 1 )
e=$( expr 2 "<=" 1 )
f=$( expr 2 "!=" 1 )
# print our variable assignment
cat expr.sh | grep " expr" | grep -v cat | grep =
# set delim character to .
IFS="."
# make an array of the return codes form expr
# echo will print all vars in order, making
# a list separated by the delim character
# that we can iterate over
arr=$( echo $a.$b.$c.$d.$e.$f)
# loop through the array
for i in $arr; do
if [ $i -eq 1 ]; then
printf "$i true"
elif [ $i -eq 0 ]; then
printf "$i false"
elif [ $i -eq 2 ]; then
printf "$i ERROR"
fi
printf "\n"
done
user@localhost:~ % sh expr.sh
a=$( expr 2 = 1 )
b=$( expr 2 ">" 1 )
c=$( expr 2 ">=" 1 )
d=$( expr 2 "<" 1 )
e=$( expr 2 "<=" 1 )
f=$( expr 2 "!=" 1 )
0 false
1 true
1 true
0 false
0 false
1 true
user@localhost:~ %
The boolean operator works on both strings and integers. Null strings and variables are false. Non-null strings and vars are true. 1 = true, 0 = false.
user@localhost:~ % expr 1 \& 1
1
user@localhost:~ % expr 1 \& 0
0
user@localhost:~ % expr 1 \| 1
1
user@localhost:~ % expr 1 \| 0
1
user@localhost:~ % expr 0 \| 0
0
user@localhost:~ %
The string operator, : can be used to compare strings. Any number greater than 0 is true, any number equal to zero is false.
user@localhost:~ % expr foo : foo
3
user@localhost:~ % expr foo : bar
0
user@localhost:~ % expr food : "foo*"
3
user@localhost:~ % expr baz : ".*"
3
user@localhost:~ %
Functions in shell scripts are similar to functions in any other language except that there are only global variables (ie no scoping). While the bash shell runs functions in subshells to avoid variable collision, a POSIX shell will work with global variables. Keep this in mind as you can quickly break your system.
#!/bin/sh
i=1
inc_i() {
# increment variable by 1
i=$(expr $i + 1)
}
while [ "$i" -le 10 ]; do
echo $i
inc_i
done
user@localhost:~ % sh ./functs.sh
1
2
3
4
5
6
7
8
9
10
user@localhost:~ %
In the terminal, we can start a process in the background by placing a & character at the end of the line. When using this method, it's important to use the wait command before continuing the script. wait will wait untill all background jobs are completed before continuing.
Let's build a multithreaded image converter using ImageMagick. This program isn't part of the POSIX spec so you'll need to install it via your package manager or download ImageMagick here . We'll be using ImageMagick to convert all of our images to .jpgs.
#!/bin/sh
# get array of all pictures, excluding subdirectories
files=$(ls -p ~/Pictures | grep -v \/)
cd ~/Pictures
# check if converted dir exists, if not create it
if ! test -e converted; then
mkdir converted
fi
# counter for impatient users
echo "converting images, please wait"
j=1
# iterate through files
for i in $files; do
# using sed to remove the file extension
filename=$(echo $i | sed 's/\.[^\.]*$//')
# convert file to converted/file.png
convert $i converted/${filename}.png &
# counter for impatient users
echo "starting job $j"
j=$(( j + 1 ))
done
# wait until all jobs are finished
wait
# echo then exit
echo "done"
user@localhost:~ % sh ./multithread.sh
converting images, please wait
starting job 1
starting job 2
starting job 3
starting job 4
starting job 5
starting job 6
starting job 7
starting job 8
starting job 9
starting job 10
starting job 11
done
user@localhost:~ % ls Pictures/
1616854535700.jpg 1621241813304.jpg antinatalist.jpg machine.jpg
1618463408483.jpg 1623754061681.png argch1.png scrot.png
1619699442668.jpg 1625650766474.png converted sun.jpg
user@localhost:~ % ls Pictures/converted/
1616854535700.png 1621241813304.png antinatalist.png scrot.png
1618463408483.png 1623754061681.png argch1.png sun.png
1619699442668.png 1625650766474.png machine.png
user@localhost:~ %
This seems to work quite well, but there is one issue: we can lose efficiecy at some point. Starting more jobs than we have resources might create a bottleneck on our system. Be it limited CPUs, limited memory, disk, or network - we can actually slow down our system overall.
The xargs command solves the problem from above. It does most of the hard work for us. Let's re-write our program with xargs so that we can do multithreading without grinding the system to a halt.
#!/bin/sh
# get array of all pictures, excluding subdirectories
maxjobs=$1
files=$(ls -p ~/Pictures | grep -v \/)
cd ~/Pictures
# check if converted dir exists, if not create it
if ! test -e xargs; then
mkdir xargs
fi
echo $files | sed 's/\ /\n/g' | xargs -P $1 -I '{}' convert '{}' xargs/'{}'.png
# echo then exit
echo "done"
Cron jobs are used to automatically run some script or command. Cron jobs are placed in a file called a crontab . A typical crontab will look like this:
* * * * * user command
| | | | |
| | | | day of the week
| | | month
| | day of the month
| hour
minute
here is an example, taken from /etc/crontab
# /etc/crontab - root's crontab for FreeBSD
#
# $FreeBSD$
#
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
#
#minute hour mday month wday who command
#
# Save some entropy so that /dev/random can re-seed on boot.
*/11 * * * * operator /usr/libexec/save-entropy
#
# Rotate log files every hour, if necessary.
0 * * * * root newsyslog
#
# Perform daily/weekly/monthly maintenance.
1 3 * * * root periodic daily
15 4 * * 6 root periodic weekly
30 5 1 * * root periodic monthly
#
# Adjust the time zone if the CMOS clock keeps local time, as opposed to
# UTC time. See adjkerntz(8) for details.
1,31 0-5 * * * root adjkerntz -a
DO NOT edit /etc/crontab. Instead you should use the crontab -e command. This will dump you into vi and automatically check the syntax of the crontab. You can also use crontab -l to list the entries in your crontab. In order to add cron jobs for root, you should use sudo crontab -e
user@localhost:~ % crontab -e
[ vi here ]
user@localhost:~ % crontab -l
# get new ssl certs every month
0 0 1 * * service apache24 stop && acme.sh --standalone --renew -d localhost.localdomain --force && service apache24 restart
# automatically update and reboot every sunday at 3AM
0 3 * * 0 pkg update && pkg upgrade -y && reboot