Introduction to UNIX
Practicals and tape_archive.zip
The objective of this tutorial is to promote understanding of basic UNIX commands and their implementation using a Command-Line Interface. MacOS and various flavours of Linux are UNIX-based operating systems.
The command-line interface is the single most powerful tool in a bioinformaticians toolbox. Some basic understanding of how to use the Command Line to move, modify and view files containing molecular and descriptive data is an absolute requirement for modern biological analysis, and mastery of the command line and its powerful tool set is highly recommended.
Directory structure
To work with a file or directory, it is important to tell the system where you are. There are two ways to do that: the absolute and the relative path. The absolute path gives the whole location of the file, for example: /home/darwin/Desktop/file
The relative path is described with the signs “.” and “/”:
./ = current directory (When no location is specified, UNIX assumes “./”.)
../= parent directory of ./ (one directory up)
/ = root directory (top directory)
~ is your home directory. In Ubuntu Linux, this will be /home/username (for example /home/john if your user id is john)
Some basic commands
Commands are composed this way:
command option filename
|
Command |
Option |
Description |
| pwd | print working directory | |
| ls | list directory contents | |
| -l | list contents in long format (with rights, size, modification…) | |
| -a | list also hidden files/directories (e.g. .profile) | |
| history | display command history | |
| cd | directory | change directory |
| mkdir | new_directory | make directory |
| clear | clear the screen | |
| exit | quit command line |
Rights and Permissions in UNIX
The permission system in UNIX and in Mac OSX is different than in Windows. As you login, you get rights to run programs and to create and delete files in your home directory. That’s sufficient for most of your work. But sooner or later (new program installation, USB drive un-/mounting…), you will need to get more rights, i.e. to become a superuser (root). A safe way is to get “temporary” rights just for one action, type sudo (superuser do)and the command you were not allowed to execute:
sudo command
If you know that you will need the superpower for more commands, type:
sudo su
but be aware that you can destroy your system. When you finish, become a “regular” user by typing:
exit
File permission is another example of special rights. When you work with files created by someone else it could happen that you are not allowed to change it. To display files permissions and sizes use:
ls -l
What does drwxrwxrwx mean?
The permission for a file, you can only read, looks really simple:
-r––—–
A simple command change the file (or directory) signature to give permissions to read and write for all groups:
chmod a+rw file
|
Command |
Option |
Description |
| chmod | +rwx | add permissions to read-write-execute for you and groups |
| -rwx | remove permissions to read-write-execute for you and groups | |
| ugoa | permissions for user-group-other-all |
Copying, Moving, Renamingand Deleting files
On a UNIX file system, moving a file is the same as renaming it. The commands works like:
| Command | Option | Description |
| cp | -i | prompt before overwrite |
| -n | do not overwrite an existing file | |
| -r | copy directory with all files | |
| -u | move only when the SOURCE file is newer or the destination file is missing | |
| -v | explain what is being done (verbose mode) | |
| mv | -i | prompt before overwrite |
| -n | do not overwrite an existing file | |
| -u | move only when the SOURCE file is newer or the destination file is missing | |
| -v | explain what is being done (verbose mode) | |
| rm | -i | prompt before evert removal |
| -r | remove directory with all files | |
| -v | explain what is being done (verbose mode) |
Examples
To copy a file using a terminal, use:
cp original-file new-file
This makes a new file that contains the same information as the original file with the name indicated. cp can also be used to copy a file from one directory to another. For example:
cp ~/directory1/file ./newfile
To copy an entire directory, use the -r option:
cp -rv dir-to-copy/ new-dir/
Note: on some computers cp -r does not copy dot-files (files that start with ‘.’, for example .bashrc). Also, you may run into trouble copying to or from directories on which you do not have appropriate permissions.
Exploring files
| Command | Option | Description |
| head | (-10) | view first 10 lines of the document |
| tail | (-10) | view last 10 lines of the document |
| less | view all document, page by page | |
| cat | file | print whole content of the file on the screen |
| file1 file2 > file3 | merge (concatenate) two files into third | |
| wc | (-l,-w,-c) | print line, word and byte counts for each file |
There are many ways to view a text file. One way for simple viewing is to type:
less
The less command displays as much of the file as can fit onto the screen. To scroll up and down within the document, use the arrow keys. Hitting the space bar will bring a new screen-full of information. To search forward in the file for a given pattern, enter:
/pattern
where “pattern” represents the character string you wish to find. To search for the next occurrence of the “pattern”, press n. To exit the less program and return to the prompt, press q.
Locate file in the filesystem
Used to find the location of files and directories
Note that you would need locate database to be generated for the locate command to work. If that is the case follow the instructions given by the locate program or ignore this part of the tutorial.
Type locate followed by the name of the file you are looking for:
locate
Search files for a pattern
Grep is an acronym for General Regular Expression Print.
In other words, the grep command is excellent for searching for a particular pattern in a file and outputting the results to the screen. This pattern can be literal (exact characters), such as chr_1, but could also be a regular expression (often called regexp). Regular expressions are a powerful way to find and replace strings of a particular format. For example, regular expressions can be used to search for a pattern such as “chr_” followed by any number (e.g. chr_1 as well as chr_999 etc should match). Note: A regexp provides more flexibility in a search, but can be difficult to get right in the beginning so make sure to check that your results are what you expect them to be. It is highly recommended to put the pattern you are searching for within single quotes:
grep option ‘pattern’ filename
| Command | Option/Pattern | Description |
| grep | ‘pattern‘ | find a ‘pattern’ in the document |
| –color | highlights the match on the screen | |
| -E | extends the regular expression capabilities of grep | |
| -E ‘[0-9]’ | [0-9] matches any digit in range (-E could be omitted) | |
| -E ‘[a-z]’ | [a-z] matches any character in range (-E could be omitted) | |
| -E ‘+‘ | give all not empty lines | |
| -c | count lines with matches | |
| -n | print the line number where the ‘pattern’ was found | |
| -r | read all files in the directory | |
| -w | find only ‘pattern’ standing as whole word |
For example to retrieve all lines matching fasta headers:
grep -E ‘>‘ filename
To count how many matches grep got, you can use the -c option:
grep -E -c ‘[0-9]’ filename
Split large files
With the large data sizes typically generated in genomics, it can sometimes be an advantage to split a file into several smaller ones:
split -l 10000 filename
Uncompressing files and directories
tar : create tape archives and add or extract files. Example:
tar –zvxf tarfile.tar.gz
unzip : extract all files of the archive into the current directory and subdrectories:
unzip zippedfile.zip
gunzip : uncompress a gzip’d file:
gunzipgzippedfile.gz
| Command | Option/Pattern | Description |
| tar | -A | append tar files to the archive |
| -c | create a new archive | |
| -r | append files to the end of the archive | |
| -x | extract files from an existing archive | |
| unzip | extract all files of the archive into the current directory and subdrectories | |
| gunzip | uncompress a gzip’d file |
Combining commands into a pipeline
UNIX lets you combine virtually any commands by “glueing” them together with the pipe symbol “|”. The output of the first command will be used as input of the next command. Combining grep and wc will give you the number of lines having a particular pattern:
grep ‘pattern‘filename| wc -l
grep ‘pattern‘ filename > headers_filename
Navigating the command line: As you get more and more comfortable with using the command line you will begin to generate commands that are longer and unwieldily. The length of your commands will also increase when you want to modify multiple files, files with long names or files in complex directory structures. Taken together, a long and detailed command can become difficult to manage. Fortunately, the text of a command can be easily navigated by simple keyboard shortcuts. Of note, there are some differences between the standard linux shortcuts and Mac OS terminal short-cuts. Some of these are detailed below, but you may hear about a different shortcut from somewhere else and it may or may not work the same depending on your OS.
| Hot-keys in Linux (not only) | Description |
| Alt+Tab | jump to the next window (i.e. from Command Line to Firefox) |
| Ctrl+Tab | jump to the next tab (e.g. in Firefox) |
| Ctrl+N | creates a new window |
| Ctrl+T | creates a new tab (different on the Command Line!!!) |
| Ctrl+C, Ctrl+V | copy and paste (different on the Command Line!!!) |
| Hot-keys on a Command Line | Description |
| Arrow up/down | choose previous command |
| Ctrl+Shift+T | creates new tab |
| Ctrl+Shift+C | copy |
| Ctrl+Shift+V | paste |
| Ctrl+A or Home | take you to the beginning of the command line |
| Ctrl+E or End | take you to the end of the command line |
| Ctrl+Left Arrow | jump leftwards to beginning of the words |
| Ctrl+Right Arrow | jump rightwards in the gap between words |
| Ctrl+U | clears the entire line to the left of the cursor (this is the fastest way to clear an entire command if your cursor is at the end) |
| Ctrl+W | delete the word before (to the left) of the cursor |
| Ctrl+K | delete everything from the position of the cursor to the end of the line (everything to the right of the cursor) |
| Ctrl+C | close/exit the program (!!! note the difference from the most common utilization !!!) |
| Ctrl+Z | suspend/interrupt the program – user can later continue by typing the command ‘fg‘ (short for foreground) or by typing ‘bg‘ (short for background) |