1. History and Design
1.1 From Wikipedia:
A Unix shell is a command-line interpreter or shell that provides a command line user interface for Unix-like operating systems. The shell is both an interactive command language and a scripting language, and is used by the operating system to control the execution of the system using shell scripts.
Users typically interact with a Unix shell using a terminal emulator; however, direct operation via serial hardware connections or Secure Shell are common for server systems. All Unix shells provide filename wildcarding, piping, here documents, command substitution, variables and control structures for condition-testing and iteration.
The most generic sense of the term shell means any program that users employ to type commands. A shell hides the details of the underlying operating system and manages the technical details of the operating system kernel interface, which is the lowest-level, or "inner-most" component of most operating systems.
- Unix Shell - Wikipedia
Before we begin, we should go over some history and basic systems design. UNIX systems originate from Bell Labs in the 1970s. Computers in the 1970s were much less powerful than they were today. Programs needed to be small due to the constrained nature of the systems at the time. Here is a PDP7 - a computer similar to what UNIX was first designed for. It's a big mainframe system. If you look closely, you'll see that there is no screen! Only a panel of switches, an oscilloscope, a few tape drives, and something that looks like a typewriter. In order to use these machines, the operator would sit at the teletype terminal and enter commands for the system to run. As the system finished running commands, the output would be printed on the same terminal. Since there was no graphical display, programs needed to be simple enough to run with minimal user interference.
Eventually, as these systems became more powerful, they were able to support multiple users at once. As technology progressed, users moved from teletypes to dumb terminals. A dumb terminal was an upgrade because it supported primitive character graphics and even early windowing systems. They were called 'dumb' because they actually only contained just enough hardware and processing power to connect to a remote terminal server (think PDP11). All computing still happened on the mainframe.
Keeping these constraints in mind, we can proceed to look at the design of a UNIX system. A UNIX system is very simple in that it is composed of simple logical components:
Hardware - The physical silicon crunching numbers.
Kernel - The kernel is responsible for managing resources, system processes, and communications with the hardware. The kernel is largely immutable and invisible in that the average user rarely (if ever) needs to manually communicate with the hardware.
Shell - The shell sits on top of the kernel and is how a user interfaces with the system. The shell is a friendly and simple environment designed to get work done. It take commands from the users. The shell knows where all the programs are at starts them upon the user's command.
Utilities - The utilities simultaneously sit on top of and alongside the shell. Although it is possible to run utilities without a shell, it is not an easy or pleasant experience. These programs are usually very small an designed to be strung together to create larger, more useful tools.
1.2 Back to the Shell
The Shell is a text-only interface to a UNIX system. A common misconception is that the shell is hard to learn but consider the following: the shell reads from human language and does an action based on the words. A modern day voice assistant is a similar concept. You tell the computer to do action with modifiers on a Subject and a direct object. If you can use a voice assistant, you can use a terminal.
Another terminal misconception is that it's old, inefficient, and useless. This couldn't be more incorrect. Modern UNIX systems are actually very comfortable to use. Although the shell and it's utilities retain backwards compatibility, many modern terminal programs actually have so many features that they won't work on a teletype or dumb terminal! As you learn the terminal, you will see efficiency gains as you climb the learning curve. It's always faster to type out exactly what you want the computer to do than it is to click through menu after menu, dragging files between programs, manually doing each and every step that could be automated. As for usefulness, everything but your laptop subsidized by microsoft runs some variant of UNIX. Every webserver, every smartphone, every macbook, every playstation, every internet connected smart toaster. It seems like everyone who begins learning UNIX and the shell also increase their skills in general logic, programming, process structuring, and computing in general.
1.3 Naming Conventions
Many UNIX terms are flexible and everyone calls them differently. This can cause confusion for new users. The Shell is also sometimes called the Terminal, the Command Line, the Prompt, Bash, or the Command Interpreter. All of these terms are interchangeable and you can pick whichever you like. Lately I've been calling it 'shell' because it implies UNIX as opposed 'command prompt' which can imply DOS to a room of paid microsoft shills.
Another confusing thing can be the difference between Linux and UNIX. UNIX is a specification based on the POSIX standards. Linux, *BSD, illumos, etc are simply different implementations of UNIX. For the purposes of this article, Linux and UNIX are interchangeable terms unless otherwise specified. This statement is factually incorrect . . . but this doesn't really matter if you are just starting out.
2. Basic Navigation
Here we will actually begin playing with the command line. Be sure to open your terminal and follow along.
2.1 Case Sensitivity
On UNIX, everything is 'case sensitive'. This means that uppercase and lowercase matters! make sure to follow the examples exactly. pwd will work but PWD will not work. Similarly, a file named foo.txt is different from FOO.txt. This shouldn't be a huge issue unless you have CP/M Stockholm syndrome.
2.2. Understanding the prompt
Since the shell is highly customizable, your prompt will most certainly look different. On non-brutalist systems your prompt will look something like this:
user@localhost:~ % Let's examine this. The first thing you see is your username, then an @, then the hostname of the machine you are logged into. After the user@host, we have a ~ symbol. This will change to to the directory you are currently in. Finally, there is a % symbol. In many shells, you might only see a $, %, >, or # symbol without color. All this symbol means is that the shell is listening for commands, ready to take your input.
The terms 'directory' and 'folder' are interchangeable. Similarly, 'subdirectory' is interchangeable with 'subfolder'.
2.3 Where are we at?
The pwd command stands for 'Print Working Directory'. As you'll soon see, many UNIX programs have short, mnemonic names or abbreviations. pwd tells us what directory (folder) you are currently in. This is useful for when you forget where you are.
user@localhost:~ % pwd
/home/user
user@localhost:~ %
2.4 What's here?
The ls command stands for 'List'. We can append flags after the name of the command. These flags act as modifiers to the behavior of the program. The -a flag stands for 'All' and lists hidden files. In UNIX, naming a file with a period in front will hide it. The -l flag stands for 'Long' and will show us extra information about our files. We can combine flags too!
user@localhost:~ % ls
R mbox public_html
user@localhost:~ % ls -l
total 6
drwxr-xr-x 3 user user 3 Jul 31 00:32 R
-rw------- 1 user user 1426 Jul 3 20:52 mbox
lrwxr-xr-x 1 user user 35 Jun 30 01:18 public_html -> /usr/local/www/apache24/data/~user
user@localhost:~ % ls -la
total 66
drwxr-xr-x 7 user user 20 Aug 1 00:17 .
drwxr-xr-x 5 root wheel 5 Jul 31 16:32 ..
-rw-r--r-- 1 user user 420 Jul 31 00:59 .Rhistory
drwxr-xr-x 4 user user 4 Jul 31 00:45 .config
-rw-r--r-- 1 user user 1248 Jun 30 01:00 .cshrc
-rw------- 1 user user 15728 Jul 31 21:24 .history
-rw------- 1 user user 73 Jul 13 01:58 .lesshst
drwxr-xr-x 3 user user 3 Jul 31 00:32 .local
-rw-r--r-- 1 user user 323 Jun 30 01:00 .login
-rw-r--r-- 1 user user 91 Jun 30 01:00 .login_conf
-rw------- 1 user user 301 Jun 30 01:00 .mail_aliases
-rw-r--r-- 1 user user 267 Jun 30 01:00 .mailrc
-rw-r--r-- 1 user user 978 Jun 30 01:00 .profile
-rw-r--r-- 1 user user 695 Jun 30 01:00 .shrc
drwxr-xr-x 2 user user 4 Jul 3 16:13 .ssh
drwxr-xr-x 2 user user 3 Jul 3 20:40 .vim
-rw------- 1 user user 23628 Jul 27 00:37 .viminfo
drwxr-xr-x 3 user user 3 Jul 31 00:32 R
-rw------- 1 user user 1426 Jul 3 20:52 mbox
lrwxr-xr-x 1 user user 35 Jun 30 01:18 public_html -> /usr/local/www/apache24/data/~user
user@localhost:~ %
The ls command also takes arguments. Let's try it out.
user@localhost:~ % ls /bin
[ cp domainname freebsd-version link pax pwd rmdir sync
cat csh echo getfacl ln pgrep realpath setfacl tcsh
chflags date ed hostname ls pkill red sh test
chio dd erl kenv mkdir ps rm sleep unlink
chmod df expr kill mv pwait rmail stty uuidgen
user@localhost:~ %
2.5 Moving around
The cd command stands for 'Change Directory'. Are you seeing the mnemonic conventions yet? like ls, cd can take an argument.
user@localhost:~ % cd /usr/bin
user@localhost:/usr/bin % pwd
/usr/bin
user@localhost:/usr/bin % cd /etc
user@localhost:/etc % pwd
/etc
user@localhost:/etc % cd ~
user@localhost:~ % pwd
/home/user
user@localhost:~ %
2.6 Absolute and relative paths
In UNIX, 'directory' and 'folder' are interchangeable. 'path' means nothing more than 'the path to the directory'. There are two types of paths: absolute and relative. An absolute path starts at the top of the filesystem ('/') and lists all of the subdirectories required to get to the file. A relative path omits everything above our current directory and references subdirectories in our current directory.
Play around with pwd, ls, and cd before continuing.
./ means 'this directory'
../ means 'up a directory'
../../ means 'up two directories'
/ means 'the very top directory'
public_html/ is the relative path to this folder from our current directory
/home/user/public_html/ is the absolute path to this folder
~ is a shortcut to your home directory
3. Man Pages
Before we go further, you should know about man pages. the man command is short for 'manual'. All programs installed on your system come with a 'manual' page. To view a man page, we can use the man command. This command places us inside of a separate program called a 'pager'. The pager makes it easier for us to scroll up and down.
user@localhost:~ % man ls
LS(1) FreeBSD General Commands Manual LS(1)
NAME
ls - list directory contents
SYNOPSIS
ls [-ABCFGHILPRSTUWZabcdfghiklmnopqrstuwxy1,] [--color=when] [-D format]
[file ...]
DESCRIPTION
For each operand that names a file of a type other than directory, ls
displays its name as well as any requested, associated information. For
each operand that names a file of type directory, ls displays the names
of files contained within that directory, as well as any requested,
associated information.
If no operands are given, the contents of the current directory are
displayed. If more than one operand is given, non-directory operands are
displayed first; directory and non-directory operands are sorted
separately and in lexicographical order.
The following options are available:
[pager here, press q to quit]
user@localhost:~ %
Every manual comes with multiple pages. When you see something like ls(1) , it means the first man page for the ls command. Here is an excerpt from the man page for the man command
The sections of the manual are:
1. FreeBSD General Commands Manual
2. FreeBSD System Calls Manual
3. FreeBSD Library Functions Manual
4. FreeBSD Kernel Interfaces Manual
5. FreeBSD File Formats Manual
6. FreeBSD Games Manual
7. FreeBSD Miscellaneous Information Manual
8. FreeBSD System Manager's Manual
9. FreeBSD Kernel Developer's Manual
Another useful command is apropos . The apropos command is used to search man pages for a specific term.
user@localhost:~ % apropos whatis
apropos, whatis(1) - search manual page databases
makewhatis(8) - index UNIX manuals
makewhatis.local(8) - start makewhatis for local file systems
user@localhost:~ %
Notice how apropos tells us which man page to look at. To view the specific page number, we can run:
user@localhost:~ % man 8 makewhatis
[ pager here, press q to quit ]
If you ever need help, be sure to check the man page first before asking online. Some (but not all) man pages contain useful examples too. Developers have spent countless hours documenting their software through manpages. Please use them
4. Files
Everything in UNIX is a file. Well, almost. Everything is represented by a file. Even directories are just a special type of file. Learning file manipulation is an important part of using a UNIX system.
4.1 Special Characters
It can be useful to avoid special characters in your file names. Spaces and characters like ~ ` ^ & * \ / ? | < > all have special meanings in UNIX. If you must use special characters, be sure to surround the entire filename in quotes or use an escape sequence. Escape sequences are a somewhat advanced topic . . . but the most simple way to use them is to prefix a \ character before the character you want to escape. \ essentially means 'hey, ignore whatever character comes after me'. If you don't surround the file name in quotes or use the \ escape sequence for all the special characters in the filename, you will get some unexpected behavior.
user@localhost:~ % touch '
Unmatched '''.
user@localhost:~ % touch \'
user@localhost:~ % ls
' R mbox public_html
user@localhost:~ % rm '
Unmatched '''.
user@localhost:~ % rm \'
user@localhost:~ % ls
R mbox public_html
user@localhost:~ %
4.1 File Extensions
On windows, file extensions must be three characters or shorter and are mandatory for proper function of the system. Ex: if you don't add .txt you cannot open the file, if you don't add .png you can't open the file.
On UNIX, everything is assumed to be a text file. You don't need to add a .jpg or .txt extension because the system determines what type a file is by actually looking at the data inside the file. You can even cat a .jpg or something to confirm that it's literally just a text file with special information at the beginning and end that tells us to treat it like an image instead of a plain text file. Even programs, you can cat then and see that they are just specially encoded text
For compatability though, be sure to add extensions for anything you serve on the web. Windows is stupid and determines file types not by the file itself but by the extension
4.3 What type of file is this?
the file command tells what type of file we are looking at.
user@localhost:~ % ls
R mbox public_html
user@localhost:~ % file R
R: directory
user@localhost:~ % file mbox
mbox: ASCII text
user@localhost:~ % file public_html/
public_html/: directory
user@localhost:~ % file /dev/mem
/dev/mem: character special (0/38)
user@localhost:~ %
4.4 Creating directories
the mkdir command stands for 'Make Directory'. This command is used to create new folders. the -p flag is useful to create a directory with a subdirectory in it
user@localhost:~ % mkdir foo
user@localhost:~ % ls
R foo mbox public_html
user@localhost:~ % mkdir -p foo/bar/baz
user@localhost:~ % ls -aR foo
. .. bar
foo/bar:
. .. baz
foo/bar/baz:
. ..
user@localhost:~ %
4.5 Deleting directories
The rmdir command stands for 'Remove Directory'. This command can be used to delete empty directories.
If you want to delete a directory that is not empty, you can use the rm -rf command. Please, please, please be careful when using the rm -rf command! If you mistype you can accidentally delete everything! If you must use use this command, consider using rm -ri instead!
user@localhost:~ % mkdir foo
user@localhost:~ % ls
R foo mbox public_html
user@localhost:~ % rmdir foo
user@localhost:~ % mkdir foo
user@localhost:~ % cp mbox foo/
user@localhost:~ % rmdir foo
rmdir: foo: Directory not empty
user@localhost:~ % rm -ri foo
remove foo? y
remove foo/mbox? y
user@localhost:~ %
4.6 Creating empty files
The touch command is used to create empty files. Don't worry if the file already exists, it won't overwrite it.
user@localhost:~ % touch foobar
user@localhost:~ % ls
R foobar mbox public_html
user@localhost:~ % file foobar
foobar: empty
user@localhost:~ %
4.7 Copying files and directories
The cp command stands for "Copy". It's used to copy files and directories.
user@localhost:~ % ls
R cr foobar mbox public_html
user@localhost:~ % cp foobar baz
user@localhost:~ % ls
R baz cr foobar mbox public_html
user@localhost:~ %
If we want to copy a directory, we must pass the -r flag. This stands for 'recursive'.
user@localhost:~ % ls
R cr mbox public_html
user@localhost:~ % cp R S
cp: R is a directory (not copied).
user@localhost:~ % ls
R cr mbox public_html
user@localhost:~ % cp -r R S
user@localhost:~ % ls
R S cr mbox public_html
user@localhost:~ %
4.8 Moving files and directories
The mv command stands for "Move". It's used to move files and directories. This command is also used to rename files. Note that we don't need the -r flag to move a directory.
user@localhost:~ % ls
R S cr foobar mbox public_html
user@localhost:~ % mv foobar baz
user@localhost:~ % ls
R S baz cr mbox public_html
user@localhost:~ % mv S T
user@localhost:~ % ls
R T baz cr mbox public_html
user@localhost:~ %
4.9 Deleting files and non-empty directories
the rm command stands for "Remove". This command will delete files and directories. If you would like to delete a directory, you should pass the -rf flag.
Be careful, there is no recycle bin on UNIX. If you accidentally delete a file you will need to use disk recovery software. Consider using the -i flag. It will ask for confirmation before deleting a file
user@localhost:~ % ls
R T baz cr mbox public_html
user@localhost:~ % rm baz
user@localhost:~ % rm T
rm: T: is a directory
user@localhost:~ % rm -rf T
user@localhost:~ % ls
R cr mbox public_html
user@localhost:~ %
4.10 Printing the contents of a file
the cat command stands for "Concatenate". Although it's primary purpose is for concatenating files, it's also useful for printing the contents on screen.
user@localhost:~ % cat /etc/os-release
NAME=FreeBSD
VERSION=13.0-RELEASE-p3
VERSION_ID=13.0
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 13.0-RELEASE-p3"
CPE_NAME=cpe:/o:freebsd:freebsd:13.0
HOME_URL=https://FreeBSD.org/
BUG_REPORT_URL=https://bugs.FreeBSD.org/
user@localhost:~ %
4.11 Pagers
While the cat command is useful, it does not provide any scrolling capabilities. If the file we want to read is very big, cat will quickly print the entire file before we even get a change to read it. This is where the less command comes in. less is a program called a pager. It's used for when we want to scroll through a file on our own time.
less is useful for larger files, cat is useful for short files or pipes
Here are some useful shortcuts for less
arrow keys - scrolling
h - open help menu
q - quit less
/ - search for phrase
n - next occurrence of searched phrase
user@localhost:~ % less /etc/services
[ pager here ]
5. Editors
Editors are an important part of any UNIX system. Since everything in UNIX is a file, most system settings are actually stored in a plain text file somewhere on the system. There are many editors to choose from. We will go over a few (in order of learning curve). You should try all available editors.
5.1 Nano
Nano is an easy to use editor. It's an improved version of an old editor called Pico. Nano even has a friendly menu at the bottom of the screen that reminds you of shortcuts.
GNU nano 5.8 New Buffer Modified
type some text here
^G Help ^O Write Out ^W Where Is ^K Cut ^T Execute ^C Location
^X Exit ^R Read File ^\ Replace ^U Paste ^J Justify ^/ Go To Line
5.2 ee
ee is a another easy to use editor. It's not standard on GNU systems but comes pre installed on FreeBSD.
^[ (escape) menu ^y search prompt ^k delete line ^p prev li ^g prev page
^o ascii code ^x search ^l undelete line ^n next li ^v next page
^u end of file ^a begin of line ^w delete word ^b back 1 char ^z next word
^t top of text ^e end of line ^r restore word ^f forward char
^c command ^d delete char ^j undelete char ESC-Enter: exit
=====line 1 col 0 lines from top 1 ============================================
type some text here
no file
5.3 Vi
Vi stands for 'visual editor'. There is a fork called Vim that stands for 'vi improved'. Many people prefer Vim over Vi because of it's increased feature set. On every UNIX, Vi or vim will be installed by default.
Vi(m) has a fairly high learning curve but it's the most efficient editor on earth. It uses concepts like modal editing, ergonomic shortcuts, and a command interface. It supports macros, syntax highlighting, auto indent, search and replace, etc. If you have the time to invest, Vi(m) is definitely worth Learning.
Some useful shortcuts:
h,j,k,l - These are your cursor keys. in order, they are left, down, up, and right.i - this will place you into insert mode. You can type as normal
Escape - this will place you into normal mode. This mode is used for moving around
: - this will place you into command mode
:wq - Save and quit file (aka Write Quit)
type some text here
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
~
"file" 1L, 20B 1,19 All
5.4 Other editors
There are many other editors available on a UNIX system. Ed is the standard line editor, Joe is similar to nano, Jed has pseudo menus, Micro is a terminal based editor with 'modern' (aka familiar) keyboard shortcut (ex: Control + c, Control + v), and Emacs is an editor that doesn't work in POSIX compliant terminals.
6. Regex
'Regex' is short for 'Regular Expressions'. Some people call these 'wildcards'. Let's look at some basic regex.
* - match zero or more characters
? - match a single character
[] - match a range of characters
user@localhost:~ % ls /bin/*sh*
/bin/csh /bin/sh /bin/tcsh
user@localhost:~ % ls /bin/?sh
/bin/csh
user@localhost:~ % ls /bin/*[a-f]
/bin/chmod /bin/date /bin/dd /bin/df /bin/domainname /bin/ed /bin/hostname /bin/pwd /bin/red /bin/sync
user@localhost:~ %
Let's look at more practical examples. If we want to list all files with a specific extension, we can use *.extension
user@localhost:~ % ls *.png
android-chrome-192x192.png android-chrome-512x512.png apple-touch-icon.png favicon-16x16.png favicon-32x32.png
If we want to list files with a specific numeber of characters in the name, we can do this:
user@localhost:~ % ls ?????.*
index.php
What if we want to find all files with numbers in the name?
user@localhost:~ % ls *[0-9]*
android-chrome-192x192.png android-chrome-512x512.png favicon-16x16.png favicon-32x32.png
7. Users and Permissions
7.1 Users, Groups, and Permissions explained
UNIX is a multi user, time sharing operating system. This means that multiple users can be logged in to one machine at the same time. In contrast to windows where ther is effectively only one user at any given time, a UNIX system has many users and 'Daemons' running concurrently. A daemon is a special type of user that runs a server or process (ex sshd runs under the ssh daemon, httpd runs under the http daemon, etc). Additionally, on UNIX there is no 'Administrator' 'account'. The UNIX admin user is called 'root'. 'root' is a real account that has unrestricted access to the entire system. In order to avoid breaking things and creating additional vulnerabilities, many sysadmins choose to install a program called sudo . Sudo is similar to the windows concept of "Right Click -> Run as Administrator". sudo supports logging, restrictions on which commands a user can run, how a user must authenticate to run a program as root, etc.
Alongside users, UNIX has groups. A group is used to grant permissions or access to a specific list of accounts. For example: Accounts in the 'wheel' or 'sudoers' groups are allowed to use the sudo command to run commands as root. Users in the 'daemon' group are system accounts and are prevented from logging in as a normal user.
In a real world example, everyone who works in the Accounting department would be in the 'accounting' group. They all have access to the files related to accounting. The Data Entry department does not need to access the documents and files owned by the Accounting department so we would want to restrict their access. Likewise, the users in the Developer department do not need access to the Accounting department's files but they do need access to a communal workspace (owned by the 'developer' group) where they can all work together. The IT department does not need to access what the Accounting department is working on but they all need to share files related to network documentation and the ticket tracker as well as whatever the Development department is working on. A user in the IT department might be in both the 'developer' group and the 'it' group. Groups are useful because they allow us to group users together in a way that allows them to share resources while preventing them from accessing resources they have no need to ever touch. A user can be in multiple groups at once. This makes the entire concept of permissions extremely flexible.
In addition to sudo and groups, the designers of UNIX created a permission to prevent users from harming each other (or the system). The permission system is quite simple.
7.2 Listing permissions
Every file is owned by a User and a Group . Each file has 3 permissions: Read, Write, and Execute. These permissions can be different for the User , the Group and Other Users .
These 3 permissions define what can be done with the file. If the Read bit is set for User, Group, Other then all users on the system can read the file. If the Write bit is set for User, Group , then the user and group of the file can edit or overwrite that file. If the Execute bit is set for only the User , then the user who owns that file can run the file as if it was any other program.
Be careful when playing with permissions! If you accidentially allow other users to modify your or ready files they can destroy or exfiltrate your data!
Let's take a look at some files. In order to get a list of permissions, you can run the ls -lh command.
In order to explain the output of ls -lh , I have provided a diagram
user@localhost:~ % ls -lh /bin/sh
-r-xr-xr-x 1 root wheel 159K Apr 8 23:17 /bin/sh
| | | | | | |
| User Group Size | | File Name
| | |
Permissions Date last edited
The permissions block can be somewhat confusing at first. Let's diagram further
d r w x r w x r w x
| | | | | | | | | |
| | | | | | | | | Other execute permission
| | | | | | | | Other write permission
| | | | | | | Other read permission
| | | | | | Group execute permission
| | | | | Group write permission
| | | | Group read permission
| | | User execute permission
| | User wirte permission
| User read permission
Indicates directory
the most simple way to remember the diagram from above is by remembering the 'UGO' acronym. This stands for 'User, Group, Other'. We can group the permissions together in sets of three to reduce the number of things we need to remember.
-[rwx][r-x][r-x]
| | |
| | Other permissions
| Group permissions
User permissions
7.3 Chaging permissions
In order to change permissions, we can use the chmod command. chmod is short for 'Change file Mode'.
To use chmod , you list which set to change permissions for (UGOA), the + or - symbols to add or remove a permissions, and list of permissions to give or revoke (RWX). We can combine operations and set permissions for multiple sets at once.
An easy way to remember how to use chmod is to remember two acronyms: 'UGOA' and 'RWX'. These stand for 'User, Group, Other, All' and 'Read, Write, Execute'.
user@localhost:~ % ls -lh foobar
-rw-r--r-- 1 user user 0B Aug 8 19:14 foobar
user@localhost:~ % chmod u+x ./foobar
user@localhost:~ % ls -lh foobar
-rwxr--r-- 1 user user 0B Aug 8 19:14 foobar
user@localhost:~ % chmod og-r ./foobar
user@localhost:~ % ls -lh foobar
-rwx------ 1 user user 0B Aug 8 19:14 foobar
user@localhost:~ % chmod a+rwx ./foobar
user@localhost:~ % ls -lh foobar
-rwxrwxrwx 1 user user 0B Aug 8 19:14 foobar
user@localhost:~ %
7.4 Chaging Ownership
The chown command stands for "Change Ownership" and is used for changing which user and group owns a file. This command will not work in many cases unless you are allowed to run the sudo command. The chown command changes ownership in the form of user:group
user@localhost:~ % ls -lh foobar
-rwxrwxrwx 1 user user 0B Aug 8 19:26 foobar
user@localhost:~ % chown user:wheel ./foobar
user@localhost:~ % ls -lh foobar
-rwxrwxrwx 1 user wheel 0B Aug 8 19:26 foobar
user@localhost:~ % sudo chown root:www ./foobar
Password:
user@localhost:~ % ls -lh ./foobar
-rwxrwxrwx 1 root www 0B Aug 8 19:26 ./foobar
user@localhost:~ %
8. Grep
grep is a tool that's used for filtering text. Think of grep as a search function. To use grep , specify a phrase you want to search for then the file you want to search in.
user@localhost:~ % grep "ssh" /etc/services
ssh 22/tcp #Secure Shell Login
ssh 22/udp #Secure Shell Login
ssh 22/sctp #Secure Shell Login
sshell 614/tcp #SSLshell
sshell 614/udp
netconf-ssh 830/tcp #NETCONF over SSH
netconf-ssh 830/udp #NETCONF over SSH
x11-ssh 6010/tcp #Unofficial name, for convenience
x11-ssh 6010/udp
user@localhost:~ % grep "user" /etc/passwd
user:*:1002:1003:User:/home/user:/bin/tcsh
user@localhost:~ %
To find all lines that do not contain a specific string, we can use the -v flag.
user@localhost:~ % cat /etc/os-release
NAME=FreeBSD
VERSION=13.0-RELEASE-p3
VERSION_ID=13.0
ID=freebsd
ANSI_COLOR="0;31"
PRETTY_NAME="FreeBSD 13.0-RELEASE-p3"
CPE_NAME=cpe:/o:freebsd:freebsd:13.0
HOME_URL=https://FreeBSD.org/
BUG_REPORT_URL=https://bugs.FreeBSD.org/
user@localhost:~ % grep -v "FreeBSD" /etc/os-release
VERSION=13.0-RELEASE-p3
VERSION_ID=13.0
ID=freebsd
ANSI_COLOR="0;31"
CPE_NAME=cpe:/o:freebsd:freebsd:13.0
user@localhost:~ %
To search case insensitively, we can use the -i flag.
user@localhost:~ % grep -i ansi /etc/os-release
ANSI_COLOR="0;31"
user@localhost:~ %
to search for multiple phrases, we can use the -e flag. Try combining flags to see what you can do. Be sure to surround your phrases in quotes if they contain any spaces or special characters.
user@localhost:~ % grep -e bug -i -e id /etc/os-release
VERSION_ID=13.0
ID=freebsd
BUG_REPORT_URL=https://bugs.FreeBSD.org/
user@localhost:~ %
9. Pipes
9.1 Data Streams
Every program you can run has three data streams:
STDIN - Standard Input, the data fed to the programSTDOUT - Standard Output, data printed to the terminal by the program
STDERR - Standard Error, Error messages (printed to the terminal by default).
UNIX pipes allow us to redirect these data streams in useful ways. Nearly every UNIX command supports piping. With pipes we can redirect command output to a file, redirect input from a file, and even use the output of one program as input for another program. Using one command's output as input for another program allows us to chain together many commands in order to create programs of various complexity.
Here are the pipes. They allow us to do specific operations
> save command output to a file
>> append output to a file
< read input from a file
2> redirect error messages
| send the output of one program to the input of another
For now, you can ignore standard error. The way to redirect STDERR is not consistent across shells. 2> works in bourne-compatible shells but not in every shell.
9.2 Using Pipes
Once you become familiar with pipes you'll feel a bit strange when working in an environment without them. Let's play with them.
Here we are redirecting output. Notice how it's identical to what's been printed.
user@localhost:~ % ls
R cr mbox procs.svg public_html
user@localhost:~ % ls > files
user@localhost:~ % cat files
R
cr
files
mbox
procs.svg
public_html
user@localhost:~ %
Let's append more data to this file we've created. If we were to use the > pipe again, we'd overwrite the file. So let's use the >> pipe instead.
user@localhost:~ % ls -d .* >> files
user@localhost:~ % cat files
R
cr
files
mbox
procs.svg
public_html
.
..
.Rhistory
.config
.cshrc
.history
.local
.login
.login_conf
.mail_aliases
.mailrc
.profile
.shrc
.ssh
.vim
.viminfo
user@localhost:~ %
Now let's use the < pipe to read a file into a program. In this use case, less < files is equivalent to less files
user@localhost:~ % less < files
R
cr
files
mbox
procs.svg
public_html
.
..
.Rhistory
.config
.cshrc
.history
.local
.login
.login_conf
.mail_aliases
.mailrc
.profile
.shrc
.ssh
.vim
.viminfo
(END)
Now we can try out the pipe operator. The | character is called 'pipe' and is usually on the same key as the \ character. Below are some fun examples. Cowsay and figlet are rarely used in serious business but they provide a good example of how pipes work. Try catting a file and piping it through cowsay.
user@localhost:~ % cat files | grep "." | grep rc | figlet -f small
_
__ __| |_ _ _ __
_/ _(_-< ' \| '_/ _|
(_)__/__/_||_|_| \__|
_ _
_ __ __ _(_) |_ _ __
_| ' \/ _` | | | '_/ _|
(_)_|_|_\__,_|_|_|_| \__|
_
__| |_ _ _ __
_(_-< ' \| '_/ _|
(_)__/_||_|_| \__|
user@localhost:~ % cat files | grep "." | grep rc | cowsay
______________________
< .cshrc .mailrc .shrc >
----------------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
user@localhost:~ %
You can combine pipes and redirection too. Try to redirect the output of cowsay to a file, then pipe that same file into cowsay. See what happens.
One tip for preventing pipe problems is to build your pipes incrementally. Add another pipe only when you have achieved a result you're looking for.
10. Filters
Now that you know about pipes, filters will be useful. In UNIX, a filter command takes whatever data you give it, performs some operation, then outputs the modified data. Be sure to read the man pages for each command. These examples do not demonstrate the full capabilities of each command.
10.1 head
the head command is used for displaying the first few lines of a file. Using the -n flag specifies how many lines to display. By defualt, 10 lines are displayed.
user@localhost:~ % head /etc/protocols
#
# Internet protocols
#
# $FreeBSD$
# from: @(#)protocols 5.1 (Berkeley) 4/17/89
#
# See also http://www.iana.org/assignments/protocol-numbers
#
ip 0 IP # internet protocol, pseudo protocol number
#hopopt 0 HOPOPT # hop-by-hop options for ipv6
user@localhost:~ % head -n 5 /etc/protocols
#
# Internet protocols
#
# $FreeBSD$
# from: @(#)protocols 5.1 (Berkeley) 4/17/89
user@localhost:~ %
10.2 tail
The tail command works similarly to the head command. You can also use the -n flag to specify any number of lines. Another useful flag is the -f flag. This flag will continually print lines as they are added to the file until you press Control + c. tail -f is useful for watching log files in real time.
user@localhost:~ % tail /etc/protocols
manet 138 MANET # MANET Protocols (RFC5498)
hip 139 HIP # Host Identity Protocol (RFC5201)
shim6 140 SHIM6 # Shim6 Protocol (RFC5533)
wesp 141 WESP # Wrapped Encapsulating Security Payload (RFC5840)
rohc 142 ROHC # Robust Header Compression (RFC5858)
# 138-254 # Unassigned
pfsync 240 PFSYNC # PF Synchronization
# 253-254 # Use for experimentation and testing (RFC3692)
# 255 # Reserved
divert 258 DIVERT # Divert pseudo-protocol [non IANA]
user@localhost:~ % tail -n 3 /etc/protocols
# 253-254 # Use for experimentation and testing (RFC3692)
# 255 # Reserved
divert 258 DIVERT # Divert pseudo-protocol [non IANA]
user@localhost:~ %
10.3 sort
The sort command does what it sounds like: sort the file by lines.
user@localhost:~ % cat animals
dog
cat
hippo
duck
penguin
shark
user@localhost:~ % cat animals | sort
cat
dog
duck
hippo
penguin
shark
user@localhost:~ % cat animals | sort > animals.sorted
user@localhost:~ % cat animals.sorted
cat
dog
duck
hippo
penguin
shark
user@localhost:~ %
The sort command can also sort by numbers. Use the -n flag to sort by numeric order instead of alphabetically.
user@localhost:~ % cat animals
2 moose
0 pegasus
4 antelope
6 giraffe
3 walrus
1 snake
user@localhost:~ % cat animals | sort -n
0 pegasus
1 snake
2 moose
3 walrus
4 antelope
6 giraffe
user@localhost:~ % cat animals | sort -n > animals.sorted
user@localhost:~ % cat animals.sorted
0 pegasus
1 snake
2 moose
3 walrus
4 antelope
6 giraffe
user@localhost:~ %
10.4 wc
The wc command stands for "Word Count". This command can be used to count lines, words, or characters in a file. The -l flag means "Lines", the -m flag means "characters", and the -w flag means "Words".
user@localhost:~ % cat animals
ostrich
penguin puffin
pheasant grouse
otter
trout
goose
user@localhost:~ % cat animals | wc -l
6
user@localhost:~ % cat animals | wc -m
57
user@localhost:~ % cat animals | wc -w
8
user@localhost:~ %
10.5 nl
The nl command stands for "Number Lines". This command is useful for when you want to number the lines in your file.
user@localhost:~ % cat animals
ostrich
penguin puffin
pheasant grouse
otter
trout
goose
user@localhost:~ % cat animals | nl
1 ostrich
2 penguin puffin
3 pheasant grouse
4 otter
5 trout
6 goose
user@localhost:~ % cat animals | nl > animals.numbered
user@localhost:~ % cat animals.numbered
1 ostrich
2 penguin puffin
3 pheasant grouse
4 otter
5 trout
6 goose
user@localhost:~ %
10.6 uniq
The uniq command is used for removing duplicates or printing only duplicates. Notice how cat appears twice.
user@localhost:~ % cat animals
cat
wolf
salmon
trout
cat
bear
bear
goose
user@localhost:~ % cat animals | uniq
cat
wolf
salmon
trout
cat
bear
goose
uniq works best when we pass our data through sort first. Be sure to add a sort command before uniq if you want accurate results.
user@localhost:~ % cat animals | sort | uniq
bear
cat
goose
salmon
trout
wolf
The -d flag means "print only duplicate lines".
user@localhost:~ % cat animals | sort | uniq -d
bear
cat
user@localhost:~ %
10.7 tac
The tac command is just the cat command but backwards. Instead of printing the file in order, it will print it in reverse. If tac is not available on your system, you can use tail -r instead.
user@localhost:~ % cat animals
cat
wolf
salmon
trout
cat
bear
bear
goose
user@localhost:~ % tac animals
goose
bear
bear
cat
trout
salmon
wolf
cat
user@localhost:~ % tail -r animals
goose
bear
bear
cat
trout
salmon
wolf
cat
user@localhost:~ %
10.8 shuf
The shuf command stands for "Shuffle". This command will shuflfe the lines in your file. If shuf is not available on your system, you can use sort --random-sort instead.
user@localhost:~ % cat animals
cat
wolf
salmon
trout
cat
bear
bear
goose
user@localhost:~ % cat animals | shuf
trout
goose
cat
cat
salmon
bear
wolf
bear
user@localhost:~ % cat animals | shuf
trout
cat
bear
goose
cat
bear
wolf
salmon
user@localhost:~ % cat animals | sort --random-sort
bear
bear
salmon
cat
cat
trout
goose
wolf
user@localhost:~ %
10.9 rs
The rs command is used for converting a file with many lines into a format that's more friendly for reading into a data array.
user@localhost:~ % cat animals
cat
wolf
salmon
trout
cat
bear
bear
goose
user@localhost:~ % cat animals | rs
cat wolf salmon trout cat bear bear goose
user@localhost:~ %
10.10 tr
The tr command stands for "Translate". It is used for translating on specific character (or phrase) into another.
user@localhost:~ % cat animals
cat
wolf
salmon
trout
cat
bear
bear
goose
user@localhost:~ % cat animals | tr cat bat
bat
wolf
salmon
trout
bat
bear
bear
goose
user@localhost:~ % cat animals | tr "\n" " " && printf "\n"
cat wolf salmon trout cat bear bear goose
user@localhost:~ %
10.11 cut
The cut command is useful for cutting text at a delimiter. It is most useful for data formatted as a .csv file or as a matrix. Use the -d flag to specify a delimiter and the -f flag to specify a field
user@localhost:~ % cat animals.csv
"BIRDS", "FELINES", "CANINES", "FISH"
"pigeon", "ocelot", "wolf", "trout"
"toucan", "tiger", "labrador", "salmon"
"parrot", "lion", "poodle", "swordfish"
"sparrow", "cheetah", "english bulldog", "tuna"
"bluejay", "leopard", "malamute", "sardine"
user@localhost:~ % cat animals.csv | cut -d , -f 1
"BIRDS"
"pigeon"
"toucan"
"parrot"
"sparrow"
"bluejay"
user@localhost:~ % cat animals.csv | cut -d , -f 2
"FELINES"
"ocelot"
"tiger"
"lion"
"cheetah"
"leopard"
user@localhost:~ % cat animals.csv | cut -d , -f 4
"FISH"
"trout"
"salmon"
"swordfish"
"tuna"
"sardine"
user@localhost:~ %
10.12 sed
The sed command stands for "Stream EDitor". It's one of the most useful commands you could possibly learn but has somewhat of a learning curve. Although sed can do some very advanced things, it's most useful for search and replace. Typically, you will run something like:
cat $data | sed s/search/replace/g Here are some examples
user@localhost:~ % cat animals.csv
"BIRDS", "FELINES", "CANINES", "FISH"
"pigeon", "ocelot", "wolf", "trout"
"toucan", "tiger", "labrador", "salmon"
"parrot", "lion", "poodle", "swordfish"
"sparrow", "cheetah", "english bulldog", "tuna"
"bluejay", "leopard", "malamute", "sardine"
user@localhost:~ % cat animals.csv | sed s/\"/\'/g
'BIRDS', 'FELINES', 'CANINES', 'FISH'
'pigeon', 'ocelot', 'wolf', 'trout'
'toucan', 'tiger', 'labrador', 'salmon'
'parrot', 'lion', 'poodle', 'swordfish'
'sparrow', 'cheetah', 'english bulldog', 'tuna'
'bluejay', 'leopard', 'malamute', 'sardine'
user@localhost:~ % cat animals.csv | sed s/\"//g | sed s/", "/" comma "/g
BIRDS comma FELINES comma CANINES comma FISH
pigeon comma ocelot comma wolf comma trout
toucan comma tiger comma labrador comma salmon
parrot comma lion comma poodle comma swordfish
sparrow comma cheetah comma english bulldog comma tuna
bluejay comma leopard comma malamute comma sardine
user@localhost:~ %
10.13 awk
The awk command is like sed but more advanced. If you are dedicated enough, awk can be used a complete programming language. In most cases though, awk is used to achieve a result similar to cut.
user@localhost:~ % cat tally
PERSON ITEM QUANTITY
Alice bananas 21
Elizabeth potatoes 8lbs
Jean-Luc replicatorBurgers 2
Ollie oranges 4
Emme bananas 5
Phillip potatoes 2
Lennon lemons 2
user@localhost:~ % cat tally | awk '{print $1 " - " $2 " : " $3 }'
PERSON - ITEM : QUANTITY
Alice - bananas : 21
Elizabeth - potatoes : 8lbs
Jean-Luc - replicatorBurgers : 2
Ollie - oranges : 4
Emme - bananas : 5
Phillip - potatoes : 2
Lennon - lemons : 2
user@localhost:~ % cat tally | awk '{print $1 " purchased " $3 " " $2 }' | grep -v PERSON
Alice purchased 21 bananas
Elizabeth purchased 8lbs potatoes
Jean-Luc purchased 2 replicatorBurgers
Ollie purchased 4 oranges
Emme purchased 5 bananas
Phillip purchased 2 potatoes
Lennon purchased 2 lemons
user@localhost:~ %
awk is also useful for formatting data so that it can be read by other programs. Here we are converting our plaintext data to a CSV
user@localhost:~ % cat tally
PERSON ITEM QUANTITY
Alice bananas 21
Elizabeth potatoes 8lbs
Jean-Luc replicatorBurgers 2
Ollie oranges 4
Emme bananas 5
Phillip potatoes 2
user@localhost:~ % cat tally | awk '{print "\"" $1 "\"," "\"" $2 "\"," "\"" $3 "\"" }'
"PERSON","ITEM","QUANTITY"
"Alice","bananas","21"
"Elizabeth","potatoes","8lbs"
"Jean-Luc","replicatorBurgers","2"
"Ollie","oranges","4"
"Emme","bananas","5"
"Phillip","potatoes","2"
"Lennon","lemons","2"
user@localhost:~ % cat tally | awk '{print "\"" $1 "\"," "\"" $2 "\"," "\"" $3 "\"" }' > tally.csv
user@localhost:~ % cat tally.csv
"PERSON","ITEM","QUANTITY"
"Alice","bananas","21"
"Elizabeth","potatoes","8lbs"
"Jean-Luc","replicatorBurgers","2"
"Ollie","oranges","4"
"Emme","bananas","5"
"Phillip","potatoes","2"
"Lennon","lemons","2"
user@localhost:~ %
10.14 diff
The diff command is mostly used by developers. The diff tool can be used to show how two files are different. If no output is shown, the files are the same. If we redirect the output of the diff command to a file we can create a patch. We can then email our patch around and other users can apply it to their files in order to benefit from our improvements. The patch command automatically creates a backup of your file with a .orig extension so don't worry about accidentally adding a bad patch because you can always go back.
user@localhost:~ % cat script-v1.sh
#!/bin/sh
echo "hello world"
echo "here is a list of files: "
ls
user@localhost:~ % cat script-v2.sh
#!/bin/sh
echo "Hello World"
echo "here is a list of files: "
ls -l
user@localhost:~ % diff script-v1.sh script-v2.sh
2c2
< echo "hello world"
---
> echo "Hello World"
4c4
< ls
---
> ls -l
user@localhost:~ % diff script-v1.sh script-v2.sh > patch
user@localhost:~ % patch script-v1.sh patch
Hmm... Looks like a normal diff to me...
Patching file script-v1.sh using Plan A...
Hunk #1 succeeded at 2.
Hunk #2 succeeded at 4.
done
user@localhost:~ % diff script-v1.sh script-v2.sh
user@localhost:~ %
10.15 hd
The hd command stands for "Hex Dump". This is another command that's really only useful if you're a developer.
user@localhost:~ % cat script-v1.sh
#!/bin/sh
echo "hello world"
echo "here is a list of files: "
ls
user@localhost:~ % cat script-v1.sh | hd
00000000 23 21 2f 62 69 6e 2f 73 68 0a 65 63 68 6f 20 22 |#!/bin/sh.echo "|
00000010 68 65 6c 6c 6f 20 77 6f 72 6c 64 22 0a 65 63 68 |hello world".ech|
00000020 6f 20 22 68 65 72 65 20 69 73 20 61 20 6c 69 73 |o "here is a lis|
00000030 74 20 6f 66 20 66 69 6c 65 73 3a 20 22 0a 6c 73 |t of files: ".ls|
00000040 0a |.|
00000041
user@localhost:~ %
11. Escape Sequences
From Wikipedia:
ANSI escape sequences are a standard for in-band signaling to control cursor location, color, span styling, and other options on video text terminals and terminal emulators. Certain sequences of bytes, most starting with an ASCII escape character and a bracket character, are embedded into text. The terminal interprets these sequences as commands, rather than text to display verbatim.
ANSI sequences were introduced in the 1970s to replace vendor-specific sequences and became widespread in the computer equipment market by the early 1980s. They are used in development, scientific, commercial text-based applications as well as bulletin board systems to offer standardized functionality.
ANSI sequences were introduced in the 1970s to replace vendor-specific sequences and became widespread in the computer equipment market by the early 1980s. They are used in development, scientific, commercial text-based applications as well as bulletin board systems to offer standardized functionality. Although hardware text terminals have become increasingly rare in the 21st century, the relevance of the ANSI standard persists because a great majority of terminal emulators and command consoles interpret at least a portion of the ANSI standard.
Or more simply, escape sequences are a combination of keys you can press or characters you can type to instruct the terminal to do something specific. An escape seqence is a series of characters that the terminal interprets as a single special control character.
11.1 Useful Shortcuts
Here are some useful keyboard shortcuts. When reading the table, assume that ^ means "Press the control key and whatever key comes after it at the same time".
| Control + key | Description |
|---|---|
| ^C | Cancel running program |
| ^D | logout (runs exit command ) |
| ^L | Clear screen |
| ^E | Move cursor to end of line |
| ^A | Move cursor ot beginning of line |
| ^P or Up Arrow | Scroll upwards through command history |
| ^N or Down Arrow | Scroll downwards through command history |
| TAB or ^I | Command autocompletion (press multiple times) |
| ^R | Search through command history (doesn't work on all shells) |
11.2 ANSI Control Sequences
These escape sequences are useful for when you need to type a specific character when your keyboard doesn't support it. When reading the table, assume that ^ means "Press the control key and whatever key comes after it at the same time". If you want to escape these characters for a command, type \ then the key. These are additional special characters. You should escape them when writing scripts or trying to use a command like sed to replace newlines or tabs.
\t means "hey terminal, this is a tab character"Control + i means "hey terminal, print a tab character right now" | Escape | Control + key | Description |
|---|---|---|
| \a | ^G | Terminal Bell (make it beep) |
| \b | ^H | Backspace |
| \t | ^I | Tab |
| \n | ^J | Newline aka enter key |
| \f | ^L | Clear Screen |
| \r | ^M | Carrige return aka enter key |
| \e | ^[ | Escape key |
11.3 ANSI Color Codes
You can use echo -e to print these escape sequences to the ternimal. If the -e flag is not available on your POSIX version of echo , you can use the printf command instead. These two commands are very simple in that they will print exactly what you tell them to the terminal.
| Foreground | Background | Name |
|---|---|---|
| 30 | 40 | Black |
| 31 | 41 | Red |
| 32 | 42 | Green |
| 33 | 43 | Yellow |
| 34 | 44 | Blue |
| 35 | 45 | Magenta |
| 36 | 46 | Cyan |
| 37 | 47 | White |
How you type ANSI color codes will vary depending on your system. Typically, you will type one of the following prefixes followed by a [ character, the color code, them the m character. If you want to use multiple color codes (eg foreground and background at the same time), put a ; character between them. I personally find \033 the easiest to type.
Control + v then Escape
\033
\u001b
\x1b
Let's play with some color codes. Be sure to end your command with \033[0m to reset your terminal to it's default colors. If you experience any issues, you can run the reset command to fix your terminal. Also notice how I end each printf command with \n , the escape for a newline character.
Setting the foreground color:
user@localhost:~ % printf '\033[31mthis is red\033[0m\n'
this is red
user@localhost:~ % printf '\033[36mthis is cyan\033[0m\n'
this is cyan
user@localhost:~ % printf '\033[33mthis is yellow\033[0m\n'
this is yellow
user@localhost:~ %
Setting the background color:
user@localhost:~ % printf '\033[41mthis is red\033[0m\n'
this is red
user@localhost:~ % printf '\033[44mthis is blue\033[0m\n'
this is blue
user@localhost:~ % printf '\033[42mthis is green\033[0m\n'
this is green
user@localhost:~ %
Setting both the foreground and background color:
user@localhost:~ % printf '\033[34;41mblue on red\033[0m\n'
blue on red
user@localhost:~ % printf '\033[30;47mblack on white\033[0m\n'
black on white
user@localhost:~ % printf '\033[35;46mmagenta on cyan\033[0m\n'
magenta on cyan
user@localhost:~ %
11.4 ANSI Select Graphic Rendition Codes
These escape sequences can be used for formatting text in special ways other than colors. Some of these might not be supported in your terminal emulator. Assume that everything past escape code 55 is unsupported and that everything else is rarely supported. If you must use one of these escape sequences, stick with 0-9 and 21-41. If you want to ensure that your escape sequence will work everywhere, use only bold, underline, and inverted.
| Code | What it does |
|---|---|
| 0 | Reset or normal |
| 1 | Bold or increased intensity |
| 2 | Faint, decreased intensity, or dim |
| 3 | Italic |
| 4 | Underline |
| 5 | Slow blink |
| 6 | Rapid blink |
| 7 | Reverse video or invert |
| 8 | Conceal or hide |
| 9 | Crossed-out, or strike |
| 10 | Primary (default) span |
| 11-19 | Alternative span |
| 20 | Blackletter span |
| 21 | Doubly underlined; or: not bold |
| 22 | Normal intensity |
| 23 | Neither italic, nor blackletter |
| 24 | Not underlined |
| 25 | Not blinking |
| 26 | Proportional spacing |
| 27 | Not reversed |
| 28 | Reveal |
| 29 | Not crossed out |
| 30-37 | Set foreground color |
| 38 | Set foreground color |
| 39 | Default foreground color |
| 40-47 | Set background color |
| 48 | Set background color |
| 49 | Default background color |
| 50 | Disable proportional spacing |
| 51 | Framed |
| 52 | Encircled |
| 53 | Overlined |
| 54 | Neither framed nor encircled |
| 55 | Not overlined |
| 58 | Set underline color |
| 59 | Default underline color |
| 60 | Ideogram underline or right side line |
| 61 | Ideogram double underline, or double line on the right side |
| 62 | Ideogram overline or left side line |
| 63 | Ideogram double overline, or double line on the left side |
| 64 | Ideogram stress marking |
| 65 | No ideogram attributes |
| 73 | Superscript |
| 74 | Subscript |
| 75 | Neither superscript nor subscript |
| 90-97 | Set bright foreground color |
| 100-107 | Set bright background color |
user@localhost:~ % printf "\033[1m bold \033[2m faint \033[0m normal \n"
bold faint normal
user@localhost:~ % printf "\033[4m underlined \033[24;53m overlined \033[55m neither \033[4;53m both \033[0m\n"
underlined overlined neither both
user@localhost:~ % printf "\033[5m blinking \033[0m\n"
blinking
user@localhost:~ % printf "\033[7m inverted \033[0m\n"
inverted
user@localhost:~ %
12. Processes
On UNIX, everything that isn't a file is a process. A process is any command running on the system. If you've ever opened Task Manager on windows, you were viewing various processes running on the system. Learning process management is an important skill for anyone spending time in the shell
12.1 ps
The ps command displays processes running on the system. ps stands for "ProcesS". By default, ps will only show commands that your user is running and have a controlling terminal. The most useful flags for ps are the -a flag which stands for "All users", the -u flag which shows additional information, and the -x flag which displays processes that do not have a controlling terminal. Combining all these flags together, we can see all the processes running on the system by running ps aux .
user@localhost:~ % ps
PID TT STAT TIME COMMAND
32216 0 INs 0:00.05 -tcsh (tcsh)
32272 2 SNs 0:00.18 -tcsh (tcsh)
32603 2 RN+ 0:00.00 ps
user@localhost:~ % ps ux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
user 32215 0.0 0.4 21396 8704 - IN 23:52 0:02.02 sshd: user@pts/0 (sshd)
user 32271 0.0 0.4 21396 8664 - SN 23:55 0:00.21 sshd: user@pts/2 (sshd)
user 32216 0.0 0.2 13984 3828 0 INs 23:52 0:00.05 -tcsh (tcsh)
user 32272 0.0 0.2 13984 4076 2 SNs 23:55 0:00.18 -tcsh (tcsh)
user 32604 0.0 0.1 13472 2580 2 RN+ 00:54 0:00.00 ps ux
user@localhost:~ % ps aux
[ very long list of processes here ]
The most important thing ps tells us can be found in the "PID" column. "PID" stands for Process ID. We can use a process' PID to manipulate it.
12.2 top
top is a command that displays a more interactive verison of ps . I personally do not like top because I prefer ps aux | grep [what I am looking for ]. Some people use htop instead but htop is bloat.
user@localhost:~ % top
last pid: 32630; load averages: 0.64, 0.65, 0.67 up 16+00:16:52 00:58:26
72 processes: 2 running, 70 sleeping
CPU: 2.9% user, 0.0% nice, 1.0% system, 0.2% interrupt, 96.0% idle
Mem: 76M Active, 106M Inact, 64M Laundry, 1368M Wired, 319M Free
ARC: 802M Total, 276M MFU, 353M MRU, 132K Anon, 14M Header, 158M Other
527M Compressed, 1117M Uncompressed, 2.12:1 Ratio
Swap: 4096M Total, 235M Used, 3861M Free, 5% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
45149 pleroma 18 52 0 1227M 72M select 344:09 1.95% beam.smp
45317 postgres 1 20 0 183M 36M select 32:13 0.00% postgres
45319 postgres 1 20 0 183M 36M select 32:10 0.00% postgres
45318 postgres 1 20 0 183M 36M select 32:08 0.00% postgres
45320 postgres 1 20 0 183M 37M select 32:00 0.00% postgres
45323 postgres 1 20 0 183M 37M select 32:00 0.00% postgres
45321 postgres 1 20 0 183M 36M select 32:00 0.00% postgres
45322 postgres 1 20 0 183M 37M select 32:00 0.00% postgres
45324 postgres 1 20 0 183M 37M select 31:58 0.00% postgres
45326 postgres 1 20 0 183M 37M select 31:54 0.00% postgres
45325 postgres 1 20 0 183M 37M select 31:52 0.00% postgres
14491 root 4 23 0 94M 6952K sigwai 5:08 0.00% rserver
45139 postgres 1 20 0 31M 3580K select 3:12 0.00% postgres
47069 prosody 1 20 0 24M 8188K select 2:06 0.00% lua52
user@localhost:~ %
12.3 kill
The kill command sends a message to a process identified by it's PID. Although the name sounds dangerous, it can be used for more than killing processes.
user@localhost:~ % ps
PID TT STAT TIME COMMAND
32216 0 INs 0:00.05 -tcsh (tcsh)
32272 2 SNs 0:00.20 -tcsh (tcsh)
32666 2 SNC 0:00.00 sleep 1000
32667 2 RN+ 0:00.00 ps
user@localhost:~ % kill 32666
user@localhost:~ % ps
PID TT STAT TIME COMMAND
32216 0 INs 0:00.05 -tcsh (tcsh)
32272 2 SNs 0:00.20 -tcsh (tcsh)
32668 2 RN+ 0:00.00 ps
user@localhost:~ %
By default, kill will end a process. If you want to send a different signal, you can use -s $signalNumber . Here are some of the more commonly used signals:
1 HUP (hang up)
2 INT (interrupt)
3 QUIT (quit)
6 ABRT (abort)
9 KILL (non-catchable, non-ignorable kill)
14 ALRM (alarm clock)
15 TERM (software termination signal)
user@localhost:~ % ps
PID TT STAT TIME COMMAND
32216 0 INs 0:00.05 -tcsh (tcsh)
32272 2 SNs 0:00.21 -tcsh (tcsh)
32671 2 INC 0:00.00 sleep 1000
32674 2 RN+ 0:00.00 ps
user@localhost:~ % kill -s 9 32671
user@localhost:~ % ps
PID TT STAT TIME COMMAND
32216 0 INs 0:00.05 -tcsh (tcsh)
32272 2 SNs 0:00.21 -tcsh (tcsh)
32675 2 RN+ 0:00.00 ps
user@localhost:~ %
12.4 pkill
The pkill command works similarly to kill but it is used to kill processes by their names. Unless you are root, you cannot kill processes belonging to other users. Be careful! pkill will kill all processes that match the name you specify!
user@localhost:~ % ps
PID TT STAT TIME COMMAND
32216 0 INs 0:00.05 -tcsh (tcsh)
32272 2 SNs 0:00.22 -tcsh (tcsh)
32723 2 RN+ 0:00.00 ps
32682 3 SNs 0:00.03 -tcsh (tcsh)
32721 3 SN+ 0:00.01 vi
user@localhost:~ % pkill vi
pkill: signalling pid 32471: Operation not permitted
user@localhost:~ % ps
PID TT STAT TIME COMMAND
32216 0 INs 0:00.05 -tcsh (tcsh)
32272 2 SNs 0:00.23 -tcsh (tcsh)
32729 2 RN+ 0:00.00 ps
32682 3 INs+ 0:00.04 -tcsh (tcsh)
user@localhost:~ % ps aux | grep 32471
root 32471 0.0 0.5 22368 10612 0 I+ 00:39 0:20.78 vim cli.html
user@localhost:~ %
12.5 jobs
The jobs command lists the jobs currently running in your terminal. It's most useful for listing processes running in the background. IN order to start a job in the background, you can add a & character to the end of your command.
user@localhost:~ % sleep 999999 &
[1] 32769
user@localhost:~ % vim &
[2] 32770
user@localhost:~ % jobs
[1] - Running sleep 999999
[2] + Suspended (tty output) vim
user@localhost:~ %
12.6 fg
the fg command is used to place a backgrounded job into the foreground.
very@localhost:~ % vim &
[2] 32774
user@localhost:~ % jobs
[1] - Running sleep 999999
[2] + Suspended (tty output) vim
user@localhost:~ % fg 2
[ vim here ]
12.7 Control+z
This keyboard shortcut will suspend a process and place it into the background.
user@localhost:~ % tail -f /var/log/messages
Aug 11 20:40:47 localhost sshd[31501]: error: maximum authentication attempts exceeded for invalid user test from 94.75.36.6 port 49796 ssh2 [preauth]
Aug 11 20:40:53 localhost sshd[31503]: error: maximum authentication attempts exceeded for invalid user test from 94.75.36.6 port 49816 ssh2 [preauth]
Aug 11 20:41:00 localhost sshd[31507]: error: maximum authentication attempts exceeded for invalid user user from 94.75.36.6 port 49851 ssh2 [preauth]
Aug 11 20:41:05 localhost sshd[31510]: error: maximum authentication attempts exceeded for invalid user user from 94.75.36.6 port 49866 ssh2 [preauth]
Aug 11 20:41:16 localhost sshd[31514]: error: maximum authentication attempts exceeded for invalid user ftpuser from 94.75.36.6 port 49898 ssh2 [preauth]
Aug 11 20:41:20 localhost sshd[31516]: error: maximum authentication attempts exceeded for invalid user ftpuser from 94.75.36.6 port 49911 ssh2 [preauth]
Aug 11 20:41:30 localhost sshd[31520]: error: maximum authentication attempts exceeded for invalid user test1 from 94.75.36.6 port 49939 ssh2 [preauth]
Aug 11 20:41:34 localhost sshd[31523]: error: maximum authentication attempts exceeded for invalid user test1 from 94.75.36.6 port 49964 ssh2 [preauth]
Aug 11 20:41:42 localhost sshd[31527]: error: maximum authentication attempts exceeded for invalid user test2 from 94.75.36.6 port 49995 ssh2 [preauth]
Aug 11 20:41:46 localhost sshd[31529]: error: maximum authentication attempts exceeded for invalid user test2 from 94.75.36.6 port 50011 ssh2 [preauth]
^Z
Suspended
user@localhost:~ % jobs
[1] + Suspended tail -f /var/log/messages
12.8 bg
the bg command will start a suspended process in the background.
user@localhost:~ % jobs
[1] + Suspended tail -f /var/log/messages
user@localhost:~ % bg 1
[1] tail -f /var/log/messages &
user@localhost:~ % jobs
[1] Running tail -f /var/log/messages
user@localhost:~ %
12.9 Terminal Multiplexing
There is a class of programs called "Terminal Multiplexers". You can think of these as window managers for your terminal. The most usable one is called tmux but some people prefer screen . Multiplexers are useful for when you need to leave a command running in the background for a long period of time, for pair programming over the internet, or for when you just want more shells without starting multiple ssh connections.
12.9.1 tmux
Tmux is very useful for when we want to run jobs in the background. It also supports multiple windows, panes, and sessions.
Starting a new session:
user@localhost:~ % tmux
[ tmux begins here ]
as you can see, tmux is just a fancy wrapper for your terminal. Let's make some splits. To split the vertically, use Control + b %. To split horizontally, use Control + b ". To switch between panes, you can use Control + b and any arrow key in the direction you want to travel. To cycle through panes, you can use Control + b o.
These keyboard combinations might seem confusing at first but they become easier to remember when we think of them as key-chords. First you press the keyboard shortcut, then the character after it. Control + b combination is a prefix that sends a command to tmux.
Over quota? "du -sh * | sort -h " will give you a sorted list of your
directory sizes.
-- David Scheidt <dscheidt@tumbolia.com>
user@localhost:~ %
────────────────────────────────────────┬───────────────────────────────────────
er to do 'umount /cdrom' before │Want colour in your directory listings?
removing the CD-ROM (it will usually not│ Use "ls -G". "ls -F" is also useful,
be possible to remove the CD-ROM │and they can be combined as "ls -FG".
without doing this.) │user@localhost:~ %
│
Note: This tip may not work in all confi│
gurations. │
user@localhost:~ % │
│
│
│
[1] 0:tcsh* "localhost" 00:19 13-Aug-21
Tmux also supports multiple windows. To create a new window, you can use Control + b c. You can use this command as many times as you'd like. In order to switch between windows, you can use the Control + b prefix then type the number of the window you want to switch to. The bottom status line will list your windows. Here is an example:
[1] 0:tcsh 1:R* 2:vim- "localhost" 00:35 13-Aug-21 The most useful part of tmux is yet to come. With tmux, we can attach and detach sessions at will. With this ability, we can start our long running process in a tmux session, detach tmux, log out, come back a week later, then attach the session and be right back where we left off. The Control + b d shortcut will detach a session. To list sessions, we can use the tmux ls command. In order to re-open the previous session, we can use the tmux attach command. tmux attach -t $session number can be used to re-open a specific previous session. All tmux sessions are automatically numbered.
user@localhost:~ % tmux ls
0: 1 windows (created Fri Aug 13 00:12:47 2021) (attached)
5: 3 windows (created Fri Aug 13 00:32:21 2021)
user@localhost:~ % tmux attach -t 5
[ tmux begins here ]