Bash

From Christoph's Personal Wiki
Revision as of 01:45, 11 May 2017 by Christoph (Talk | contribs) (→‎Integer mathematics)

Jump to: navigation, search
The correct title of this article is bash. The initial letter is capitalized due to technical restrictions.

Bash is a Linux command shell written for the GNU project. Its name is an acronym for Bourne-again shell—a pun on the Bourne shell (sh), which was an early, important Unix shell.

Bash is the default shell on most Linux systems and was previously my favourite shell. After nearly 14 years of using the bash shell as my primary shell, I switched to Z shell (zsh). It is awesome and extremely powerful! 

see: Bash/scripts for examples

Bash builtins

A shell builtin is a command or a function, called from a shell, that is executed directly in the shell itself, instead of an external executable program which the shell would load and execute.

Shell builtins work significantly faster than external programs, because there is no program loading overhead. However, their code is inherently present in the shell, and thus modifying or updating them requires modifications to the shell. Therefore shell builtins are usually used for simple, almost trivial, functions, such as text output. Because of the nature of Linux, some functions of the operating system have to be implemented as shell builtins. The most notable example is cd, which changes the working directory of the shell. Because each executable program runs in a separate process, and working directories are specific to each process, loading cd as an external program would not affect the working directory of the shell that loaded it. 

bash, :, ., [, alias, bg, bind, break, builtin, cd, command, compgen,
complete, continue, declare, dirs, disown, echo, enable, eval, exec,
exit, export, fc, fg, getopts, hash, help, history, jobs, kill, let,
local, logout, popd, printf, pushd, pwd, read, readonly, return, set,
shift, shopt, source, suspend, test, times, trap, type, typeset,
ulimit, umask, unalias, unset, wait

Bash shell shortcuts

CTRL Key Bound

Basic commands
Command Description
Ctrl + a Jump to the start of the line
Ctrl + b Move back a char
Ctrl + c Terminate the command
Ctrl + d Delete from under the cursor
Ctrl + e Jump to the end of the line
Ctrl + f Move forward a char
Ctrl + h Backspace
Ctrl + k Delete to EOL
Ctrl + l Clear the screen
Ctrl + n Next command line (useful for "scrolling" with Ctrl + p)
Ctrl + p Previous command line
Ctrl + r Search the history backwards
Ctrl + R Search the history backwards with multi occurrence
Ctrl + u Delete backward from cursor
Ctrl + xx Move between EOL and current cursor position
Ctrl + x @ Show possible hostname completions
Ctrl + w deletes the token left of the cursor
Ctrl + z Suspend / Stop the command
Ctrl + / Undo last command-line edit


ALT Key Bound

Basic commands
Command Description
Alt + < Move to the first line in the history
Alt + > Move to the last line in the history
Alt + ? Show current completion list
Alt + * Insert all possible completions
Alt + / Attempt to complete filename
Alt + . Yank last argument to previous command
Alt + b Move backward
Alt + c Capitalize the word
Alt + d Delete word
Alt + f Move forward
Alt + l Make word lowercase
Alt + n Search the history forwards non-incremental
Alt + p Search the history backwards non-incremental
Alt + r Recall command
Alt + t Move words around
Alt + u Make word uppercase
Alt + back-space Delete backward from cursor


More Special Keybindings

Basic commands
Command Description
$ 2T All available commands(common)
$ (string)2T All available commands starting with (string)
$ /2T Entire directory structure including Hidden one
$ 2T Only Sub Dirs inside including Hidden one
$ *2T Only Sub Dirs inside without Hidden one
$ ~2T All Present Users on system from "/etc/passwd"
$ $2T All Sys variables
$ @2T Entries from "/etc/hosts"
$ =2T Output like ls or dir
Note: Here "2T" means Press TAB twice


Bash Bang (!) commands

Re-run all or part of a previous command. 

!!       Run the last command again
!foo     Run the most recent command that starts with 'foo' (e.g. !ls)
!foo:p   Print out the command that !foo would run
         also add it to the command history
!$       Run the last word of the previous command (same as Alt + .)
!$:p     Print out the word that !$ would substitute
!*       Run the previous command except for the last word
!*:p     Print out the previous command except for the last word
^foo^bar  Run the previous command replacing foo with bar

Bash syntax highlights

Bash's command syntax is a superset of the Bourne shell's command syntax. The definitive specification of Bash's command syntax is the Bash Reference Manual distributed by the GNU project. This section highlights some of Bash's unique syntax features.

The vast majority of Bourne shell scripts can be executed without alteration by Bash, with the exception of those Bourne shell scripts that happen to reference a Bourne special variable or to use a Bourne builtin command. The Bash command syntax includes ideas drawn from the Korn shell (ksh) and the C shell (csh), such as command-line editing, command history, the directory stack, the $RANDOM and $PPID variables, and POSIX command substitution syntax: $(...). When being used as an interactive command shell, Bash supports completion of partly typed-in program names, filenames, variable names, etc. when the user presses the TAB key.

Bash syntax has many extensions that the Bourne shell lacks. Several of those extensions are enumerated here.

Integer mathematics

A major limitation of the Bourne shell is that it cannot perform integer calculations without spawning an external process. Bash can perform in-process integer calculations using the ((...)) command and the $[...] variable syntax, as follows: 

VAR=55             # Assign integer 55 to variable VAR.
((VAR = VAR + 1))  # Add one to variable VAR.  Note the absence of the '$' character.
((++VAR))          # Another way to add one to VAR.  Performs C-style pre-increment.
((VAR++))          # Another way to add one to VAR.  Performs C-style post-increment.
echo $[VAR * 22]   # Multiply VAR by 22 and substitute the result into the command.
echo $((VAR * 22)) # Another way to do the above.

The ((...)) command can also be used in conditional statements, because its exit status is 0 or 1 depending on whether the condition is true or false: 

if ((VAR == Y * 3 + X * 2)); then
  echo Yes
fi

((Z > 23)) && echo Yes

The ((...)) command supports the following relational operators: '==', '!=', '>', '<', '>=', and '<='.

Bash cannot perform in-process floating point calculations. The only Unix command shells capable of this are Korn Shell (1993 version) and zsh (starting at version 4.0).

I/O redirection

Bash has several I/O redirection syntaxes that the traditional Bourne shell lacks. Bash can redirect standard output and standard error at the same time using this syntax: 

command &> file

which is simpler to type than the equivalent Bourne shell syntax, "command > file 2>&1". Bash, since version 2.05b, can redirect standard input from a string using the following syntax (sometimes called "here strings"): 

command <<< "string to be read as standard input"

If the string contains whitespace, it must be quoted.

Example: Redirect standard output to a file, write data, close file, reset stdout 

# make Filedescriptor(FD) 6 a copy of stdout (FD 1)
exec 6>&1
# open file "test.data" for writing
exec 1>test.data
# produce some content
echo "data:data:data"
# close file "test.data"
exec 1>&-
# make stdout a copy of FD 6 (reset stdout)
exec 1>&6
# close FD6
exec 6>&-

Open and close files 

# open file test.data for reading
exec 6<test.data
# read until end of file
while read -u 6 dta
do
  echo "$dta" 
done
# close file test.data
exec 6<&-

Catch output of external commands 

 # execute 'find' and store results in VAR
 # search for filenames which end with the letter "h"
 VAR=$(find . -name "*h")

EOF

The `cat <<EOF` Bash syntax is very useful when one needs to work with multi-line strings in Bash (e.g., when passing a multi-line string to a variable, file, or a piped command).

  • Pass a multiline string to a variable:
$ sql=$(cat <<EOF
SELECT foo, bar FROM db
WHERE foo='baz'
EOF
)

The $sql variable now holds newlines as well, you can check it with `echo -e "$sql"`: 

SELECT foo, bar FROM db WHERE foo='baz'
  • Pass a multiline string to a file:
$ cat <<EOF > print.sh
#!/bin/bash
echo \$PWD
echo $PWD
EOF

The print.sh file now contains: 

#!/bin/bash
echo $PWD
echo /home/user
  • Pass a multiline string to a command/pipe:
$ cat <<EOF | grep 'b' | tee b.txt | grep 'r'
foo
bar
baz
EOF

This creates b.txt file with both bar and baz lines but prints only the bar.

In-process regular expressions

Bash 3.0 supports in-process regular expression matching using the following syntax, reminiscent of Perl: 

[[ string =~ regex ]]

The regular expression syntax is the same as that documented by the regex(3) man page. The exit status of the above command is 0 if the regex matches the string, 1 if it does not match. Parenthesized subexpressions in the regular expression can be accessed using the shell variable BASH_REMATCH, as follows: 

if [[ abcfoobarbletch =~ 'foo(bar)bl(.*)' ]]; then
  echo The regex matches!
  echo $BASH_REMATCH      -- outputs: foobarbletch
  echo ${BASH_REMATCH[1]} -- outputs: bar
  echo ${BASH_REMATCH[2]} -- outputs: etch
fi

This syntax gives performance superior to spawning a separate process to run a grep command, because the regular expression matching takes place within the Bash process. If the regular expression or the string contain whitespace or shell metacharacters (such as '*' or '?'), they should be quoted.

Backslash escapes

Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the C programming language. Backslash escape sequences, if present, are decoded as follows:

Backslash Escapes
Backslash
Escape 		Expands To ...
\a An alert (bell) character
\b A backspace character
\e An escape character
\f A form feed character
\n A new line character
\r A carriage return character
\t A horizontal tab character
\v A vertical tab character
\\ A backslash character
\' A single quote character
\nnn The eight-bit character whose value is the octal value nnn (one to three digits)
\xHH The eight-bit character whose value is the hexadecimal value HH (one or two hex digits)
\cx A control-X character

The expanded result is single-quoted, as if the dollar sign had not been present.

A double-quoted string preceded by a dollar sign ($"...") will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.

The full list is as follows: 

# \a     an ASCII bell character (07)
# \d     the date in "Weekday Month Date" format (e.g., "Tue May 26")
# \D{format}  the format is passed to strftime(3) and the result is inserted into the prompt string;
# \e     an ASCII escape character (033)
# \h     the hostname up to the first `.'
# \H     the hostname
# \j     the number of jobs currently managed by the shell
# \l     the basename of the shell's terminal device name
# \n     newline
# \r     carriage return
# \s     the name of the shell, the basename of $0 (the portion following the final slash)
# \t     the current time in 24-hour HH:MM:SS format
# \T     the current time in 12-hour HH:MM:SS format
# \@     the current time in 12-hour am/pm format
# \A     the current time in 24-hour HH:MM format
# \u     the username of the current user
# \v     the version of bash (e.g., 2.00)
# \V     the release of bash, version + patchelvel (e.g., 2.00.0)
# \w     the current working directory
# \W     the basename of the current working directory
# \!     the history number of this command
# \#     the command number of this command
# \$     if the effective UID is 0, a #, otherwise a $
# \nnn   the character corresponding to the octal number nnn
# \\     a backslash
# \[     begin a sequence of non-printing characters, which could be used to embed a terminal control sequence into the prompt
# \]     end a sequence of non-printing characters

Variables

When variables are used they are referred to with the $ symbol in front of them. There are several useful variables available in the shell program. Here are a few:

  • $$ = The PID number of the process executing the shell.
  • $? = Exit status variable.
  • $0 = The name of the command you used to call a program.
  • $1 = The first argument on the command line.
  • $2 = The second argument on the command line.
  • $n = The nth argument on the command line.
  • $* = All the arguments on the command line.
  • $# = The number of command line arguments.

The "shift" command can be used to shift command line arguments to the left, i.e., $1 becomes the value of $2, $3 shifts into $2, etc. The command, "shift 2" will shift 2 places meaning the new value of $1 will be the old value of $3 and so forth.

  • Print specific characters stored in a variable:
# The syntax is ${variable:start:length}. Omitting "length" value gives rest of string.
$ val="hello"
$ echo ${val:0:1} # Print out the first character of $val ("h" in this example)

Tests

There is a function provided by bash called test which returns a true or false value depending on the result of the tested expression (see: wikipedia:test (Unix) for more details). Its syntax is: 

test expression

It can also be implied as follows: 

[ expression ]

The tests below are test conditions provided by the shell:

  • -b file = True if the file exists and is block special file.
  • -c file = True if the file exists and is character special file.
  • -d file = True if the file exists and is a directory.
  • -e file = True if the file exists.
  • -f file = True if the file exists and is a regular file
  • -g file = True if the file exists and the set-group-id bit is set.
  • -k file = True if the files' "sticky" bit is set.
  • -L file = True if the file exists and is a symbolic link.
  • -p file = True if the file exists and is a named pipe.
  • -r file = True if the file exists and is readable.
  • -s file = True if the file exists and its size is greater than zero.
  • -s file = True if the file exists and is a socket.
  • -t fd = True if the file descriptor is opened on a terminal.
  • -u file = True if the file exists and its set-user-id bit is set.
  • -w file = True if the file exists and is writable.
  • -x file = True if the file exists and is executable.
  • -O file = True if the file exists and is owned by the effective user id.
  • -G file = True if the file exists and is owned by the effective group id.
  • file1 -nt file2 = True if file1 is newer, by modification date, than file2.
  • file1 -ot file2 = True if file1 is older than file2.
  • file1 -ef file2 = True if file1 and file2 have the same device and inode numbers.
  • -z string = True if the length of the string is 0.
  • -n string = True if the length of the string is non-zero.
  • string1 = string2 = True if the strings are equal.
  • string1 != string2 = True if the strings are not equal.
  • !expr = True if the expr evaluates to false.
  • expr1 -a expr2 = True if both expr1 and expr2 are true.
  • expr1 -o expr2 = True if either expr1 or expr2 is true.

The syntax is: 

arg1 OP arg2

where OP is one of -eq, -ne, -lt, -le, -gt, or -ge. Arg1 and arg2 may be positive or negative integers or the special expression "-l string" which evaluates to the length of string.

  • Examples:
if [ ! -e foo ]; then echo "NO FILE"; else cat foo; fi
if [ -d "/home/bob" -a ! -d "/home/alice" ]; then echo "Bob exists, but not Alice"; fi

Colours in bash

Black       0;30     Dark Gray     1;30
Blue        0;34     Light Blue    1;34
Green       0;32     Light Green   1;32
Cyan        0;36     Light Cyan    1;36
Red         0;31     Light Red     1;31
Purple      0;35     Light Purple  1;35
Brown       0;33     Yellow        1;33
Light Gray  0;37     White         1;37

Here is an example borrowed from the Bash-Prompt-HOWTO: 

PS1="\[\033[1;34m\][\$(date +%H%M)][\u@\h:\w]$\[\033[0m\] "

This turns the text blue, displays the time in brackets (very useful for not losing track of time while working), and displays the user name, host, and current directory enclosed in brackets. The "\[\033[0m\]" following the $ returns the colour to the previous foreground colour.

  • Another example:
PS1="\[\033[1;30m\][\[\033[1;34m\]\u\[\033[1;30m\]@\[\033[0;35m\]\h\[\033[1;30m\]] \[\033[0;37m\]\W \[\033[1;30m\]\$\[\033[0m\] "

yields: 

[user@host] directory $

Break down: 

\[\033[1;30m\] - Sets the color for the characters that follow it.
                 Here 1;30 will set them to Dark Gray.
\u \h \W \$    - From the table above
\[\033[0m\]    - Sets the colours back to how they were originally.

Bash startup scripts

When Bash starts, it executes the commands in a variety of different scripts.

When Bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable. The --noprofile option may be used when the shell is started to inhibit this behavior.

When a login shell exits, Bash reads and executes commands from the file ~/.bash_logout, if it exists.

When an interactive shell that is not a login shell is started, Bash reads and executes commands from ~/.bashrc, if that file exists. This may be inhibited by using the --norc option. The --rcfile file option will force Bash to read and execute commands from file instead of ~/.bashrc.

When Bash is started non-interactively, to run a shell script, for example, it looks for the variable BASH_ENV in the environment, expands its value if it appears there, and uses the expanded value as the name of a file to read and execute. Bash behaves as if the following command were executed: 

if [ -n "$BASH_ENV" ]; then . "$BASH_ENV"; fi

but the value of the PATH variable is not used to search for the file name.

If Bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well. When invoked as an interactive login shell, or a non-interactive shell with the --login option, it first attempts to read and execute commands from /etc/profile and ~/.profile, in that order. The --noprofile option may be used to inhibit this behavior. When invoked as an interactive shell with the name sh, Bash looks for the variable ENV, expands its value if it is defined, and uses the expanded value as the name of a file to read and execute. Since a shell invoked as sh does not attempt to read and execute commands from any other startup files, the --rcfile option has no effect. A non-interactive shell invoked with the name sh does not attempt to read any other startup files. When invoked as sh, Bash enters posix mode after the startup files are read.

When Bash is started in posix mode, as with the --posix command line option, it follows the POSIX standard for startup files. In this mode, interactive shells expand the ENV variable and commands are read and executed from the file whose name is the expanded value. No other startup files are read.

Bash attempts to determine when it is being run by the remote shell daemon, usually rshd. If Bash determines it is being run by rshd, it reads and executes commands from ~/.bashrc, if that file exists and is readable. It will not do this if invoked as sh. The --norc option may be used to inhibit this behavior, and the --rcfile option may be used to force another file to be read, but rshd does not generally invoke the shell with those options or allow them to be specified.

Environment variables

$CDPATH 
does for the cd built-in what PATH does for executables. By setting this wisely, you can cut down on the number of key-strokes you enter per day.
Example
$ export CDPATH=.:~:~/docs:~/src:~/src/ops/docs:/mnt:/usr/src/redhat:/usr/src/redhat/RPMS:/usr/src:/usr/lib:/usr/local:/software:/software/redhat
Using this, cd i386 would likely take you to /usr/src/redhat/RPMS/i386 on a Red Hat Linux system. Make sure that you do include . in the list or you'll find that you can't change to directories relative to your current one without prefixing them with ./
$HISTIGNORE 
Set this to to avoid having consecutive duplicate commands and other not so useful information appended to the history list. This will cut down on hitting the up arrow endlessly to get to the command before the one you just entered twenty times. It will also avoid filling a large percentage of your history list with useless commands.
Example
$ export HISTIGNORE="&:ls:ls *:mutt:[bf]g:exit"
Using this, consecutive duplicate commands, invocations of ls, executions of the mutt mail client without any additional parameters, plus calls to the bg, fg and exit built-ins will not be appended to the history list.
$MAILPATH 
bash will warn you of new mail in any folder appended to MAILPATH. This is very handy if you use a tool like procmail to presort your e-mail into folders.
Try adding the following to your ~/.bash_profile to be notified when any new mail is deposited in any mailbox under ~/Mail.
MAILPATH=/var/spool/mail/$USER
for i in ~/Mail/[^.]*
do
    MAILPATH=$MAILPATH:$i'?You have new mail in your ${_##*/} folder'
done
export MAILPATH
unset i
$TMOUT 
If you set this to a value greater than zero, bash will terminate after this number of seconds have elapsed if no input arrives.

shopt

Check your shell options by issuing the following command: 

shopt  # see: help shopt

A typical output should look something like the following: 

cdable_vars     off
cdspell         off
checkhash       off
checkwinsize    off
cmdhist         on
dotglob         off
execfail        off
expand_aliases  on
extglob         off
histreedit      off
histappend      off
histverify      off
hostcomplete    on
huponexit       off
interactive_comments    on
lithist         off
login_shell     on
mailwarn        off
no_empty_cmd_completion off
nocaseglob      off
nullglob        off
progcomp        on
promptvars      on
restricted_shell        off
shift_verbose   off
sourcepath      on
xpg_echo        off

where each of the above mean (see man bash or help set for more info.):

cdable_vars 
an argument to the cd builtin command that is not a directory is assumed to be the name of a variable dir to change to.
cdspell 
minor errors in the spelling of a directory component in a cd command will be corrected.
checkhash 
bash checks that a command found in the hash table exists before execute it. If no longer exists, a path search is performed.
checkwinsize 
bash checks the window size after each command and, if necessary, updates the values of LINES and COLUMNS.
cmdhist 
bash attempts to save all lines of a multiple-line command in the same history entry. Allows re-editing of multi-line commands.
dotglob 
bash includes filenames beginning with a `.' in the results of pathname expansion.
execfail 
a non-int shell will not exit if it cannot execute the file specified as an argument to the exec builtin command, like int sh.
expand_aliases 
aliases are expanded as described above under ALIASES. This option is enabled by default for interactive shells.
extglob 
the extended pattern matching features described above under Pathname Expansion are enabled.
histappend 
the history list is appended to the file named by the value of the HISTFILE variable when shell exits, no overwriting the file.
hostcomplete 
and readline is being used, bash will attempt to perform hostname completion when a word containing a @ is being completed
huponexit 
bash will send SIGHUP to all jobs when an interactive login shell exits.
interactive_comments 
allow a word beginning with # to cause that word and all remaining characters on that line to be ignored in an interactive shell
lithist 
if cmdhist option is enabled, multi-line commands are saved to the history with embedded newlines rather than using semicolon
login_shell 
shell sets this option if it is started as a login shell (see INVOCATION above). The value may not be changed.
mailwarn 
file that bash is checking for mail has been accessed since the last checked, ``The mail in mailfile has been read is displayed.
no_empty_cmd_completion 
bash will not attempt to search the PATH for possible completions when completion is attempted on an empty line.
nocaseglob 
bash matches filenames in a case-insensitive fashion when performing pathname expansion (see Pathname Expansion above).
nullglob 
bash allows patterns which match no files (see Pathname Expansion above) to expand to a null string, rather than themselves.
progcomp 
the programmable completion facilities (see Programmable Completion above) are enabled. This option is enabled by default.
promptvars 
prompt strings undergo variable and parameter expansion after being expanded as described in PROMPTING above.
shift_verbose 
the shift builtin prints an error message when the shift count exceeds the number of positional parameters.
sourcepath 
the source (.) builtin uses the value of PATH to find the directory containing the file supplied as an argument.
xpg_echo 
the echo builtin expands backslash-escape sequences by default.

Bash (Shellshock) vulnerability

  • Run the following command from a Bash shell:
$ env 'x=() { :;}; echo vulnerable' 'BASH_FUNC_x()=() { :;}; echo vulnerable' bash -c "echo test"

If you see the following output, you are VULNERABLE: 

vulnerable
bash: BASH_FUNC_x(): line 0: syntax error near unexpected token `)'
bash: BASH_FUNC_x(): line 0: `BASH_FUNC_x() () { :;}; echo vulnerable'
bash: error importing function definition for `BASH_FUNC_x'
test

If you see the following output, you are NOT VULNERABLE: 

bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `BASH_FUNC_x'
test

If you are vulnerable, make sure you update Bash to the latest version your Linux distribution has to offer. If you still see the same vulnerability after updating from a repository, you should probably down the the latest source code of Bash and compile it on your own. Do not take this bug lightly!

Christoph's Additions

PS1='\[\033[0;31m\][\u@christophchamp]\[\033[00m\]:\[\033[0;33m\]`pwd`\[\033[00m\]> '

[christoph@christophchamp]:/home/christoph>

References

External links

Bash guides from the Linux Documentation Project

Other guides and tutorials

Custom .bashrc or .bash_profile

Retrieved from "http://wiki.christophchamp.com/index.php?title=Bash&oldid=6967"