Difference between revisions of "Redirection (Linux)"

Latest revision as of 20:29, 22 September 2020

Redirection is a function common to most Linux shells which allow standard streams to be redirected to user-specified locations.

In Unix-like systems, there are two output paths that, if left unmodified, will send output to your screen. Standard output (STDOUT) captures the normal output of a given command. Standard error (STDERR) captures most failures and error conditions. To pass the permission denied message in the stderr to the same output stream as "regular output" you must combine the two. In your example, in order for your grep -v to properly operate on it, you combine stdout (standard output) and stderr with the arcane syntax you see.

Redirecting standard input and standard output

Redirection is usually implemented by placing certain characters (or tokens) between commands. Typically, the syntax of these characters is as follows:

command1 > file1  

executes command1, placing the output in file1.

command1 < file1 

executes command1, using file1 as the source of input (as opposed to the keyboard).

command1 < infile > outfile

combines the two capabilities: command1 reads from infile and writes to outfile

Redirecting with tee

$ command > >(tee stdout.log) 2> >(tee stderr.log >&2)

The ">(...)" (process substitution) part creates a FIFO and lets tee listen on it. Then, it uses > (file redirection) to redirect the STDOUT of command to the FIFO that your first tee is listening on.

• Other methods:

 $ ./a.out 2>&1 | tee output
 #~OR~
 $ ./a.out |& tee output

Piping

Programs can be run together such that one program reads the output from another with no need for an explicit intermediate file:

command1 | command2   

executes command1, using its output as the input for command2 (commonly called piping, since the "|" character is known as a "pipe").

A good example for command piping is combining echo with another command to achieve something interactive in a non-interactive shell, e.g.

echo -e "user\npass" | ftp localhost

This would run the ftp client and enter user, press return, then pass.

Redirecting to and from the standard file handles

In shells derived from the original Bourne shell, the first two actions can be further modified by placing a number (the file descriptor) immediately before the character; this will affect which stream is used for the redirection. The Linux standard I/O streams are:

Each open file gets assigned a file descriptor. The file descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively. For opening additional files, there remain descriptors 3 to 9. It is sometimes useful to assign one of these additional file descriptors to stdin, stdout, or stderr as a temporary duplicate link. This simplifies restoration to normal after complex redirection and reshuffling.

For example:

command1 2> file1

executes command1, directing the standard error stream to file1 (useful since standard error outputs to the terminal by default and is unaffected by redirection unless so specified).

In shells derived from csh (the C shell), the syntax instead appends the & character to the redirect characters, thus achieving a similar result.

Another useful capability is to redirect one standard file handle to another. The most popular variation is to merge standard error into standard output so error messages can be processed together with (or alternately to) the usual output. Example:

find / -name .profile > results 2>&1

will try to find files/directories named CVS in the /root directory. Executed without redirection, it will output hits to stdout and errors (e.g. for lack of privilege to read certain directories) to Stderr. If we redirect standard output to file results, the error messages will continue to spam the console. To see both hits and error messages in file results, we merge stderr (handle 1) into stdout using 2>&1 .

If the merged output is to be piped into another program, the file merge sequence 2>&1 must precede the pipe symbol, thus:

find / -name .profile 2>&1 | less

• Find all "*.txt" files under the /etc directory and suppress any "Permission denied" messages:

find /etc -name '*.txt' |& grep -v "Permission denied"

• Redirect both STDOUT and STDERR to /dev/null:

git pull >/dev/null 2>&1

EOF

• Save text from the CLI to a file without using an editor:

cat << EOF > /path/to/output/file.txt
line1
line2
EOF

• Multiple EOFs:

cat <<eof1; cat <<eof2
Hello
eof1
World
eof2

Redirection order priorities

If one were to execute the following command

$ cat file1 file2 1> file.txt 2>&1  # EXAMPLE 1

one would get,

1. concatenate files file1 and file2;
2. send STDOUT from this operation to file.txt; and
3. send STDERR to STDOUT.

This is not the same as executing

$ cat file1 fil2 2>&1 > file.txt  # EXAMPLE 2

To understand why, note that by default:

1 = /dev/tty
2 = /dev/tty

So, EXAMPLE 1 (1> file.txt 2>&1) does:

1 = file.txt
2 = 1         # and currently 1 = file.txt

with result:

1 = file.txt
2 = file.txt

EXAMPLE 2 also starts with:

1 = /dev/tty
2 = /dev/tty

However, 2>&1 > file.txt does

2 = 1         # and currently 1 = /dev/tty
1 = file.txt

with result:

1 = file.txt
2 = /dev/tty

Thus, only STDOUT has been redirected, not STDERR.

Swapping STDOUT and STDERR using file descriptors

Note: According to the Advanced Bash-Scripting Guide, if "|&" is used, the STDERR of command1 is connected to command2's standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error is performed after any redirections specified by the command.

One can either redirect a file descriptor to a file with > or redirect it to another file descriptor (FD) with >&. In other words:

>name means redirect output to file name.

>&number means redirect output to file descriptor number.

The '&' is needed to tell the shell you mean a file descriptor, not a file name.

A file descriptor is a number that refers to an already open file. The standard ones are 0 for standard input, 1 for standard output (STDOUT), and 2 for standard error (STDERR). One can also use any other number, which will create a new file descriptor, in much the same way when one creates a new variable with var=value.

By default, both file descriptor 1 and 2 go to /dev/tty, so if one were to run somecommand 3>&1 1>&2 2>&3 in a new shell, it will not change anything (except now there is a file descriptor number 3).

But if somewhere earlier in the script it does a redirection using exec (e.g., exec 2>error.log), or the script is run with a command line including redirection (e.g., ./myscript 2>error.log), then swapping STDOUT and STDERR will do something.

So, one could create the following to swap STDOUT and STDERR:

$ exec 3>&1 1>&2 2>&3
$ # ~OR~
$ ./myscript 3>&1 1>&2 2>&3

The 3>&1 in the command line will create a new file descriptor and redirect it to 1, which is STDOUT. Now 1>&2 will redirect the FD 1 to STDERR and 2>&3 will redirect FD 2 to 3, which is STDOUT.

Thus, STDOUT and STDERR have been switched. These are the steps:

1. create a new FD 3 and point it to the FD 1;
2. redirect FD 1 to FD 2. (Note: If the FD in 3 is not saved first, the target is lost.); and
3. redirect FD 2 to FD 3. Now FDs one and two are switched.

If the program now prints something to the FD 1 it will be printed to the FD 2, et vice versa.

One can list the open file descriptors with

$ ls -l /proc/self/fd

• Check that "3<&1" is the same as "3>&1" (see: `strace`):

$ sudo strace -f -s 200 -e trace=dup2 sh -c 'exec 3<&1'
dup2(1, 3)                              = 3
$ sudo strace -f -s 200 -e trace=dup2 sh -c 'exec 3>&1'
dup2(1, 3)                              = 3
$ sudo strace -f -s 200 -e trace=dup2 sh -c 'exec 2>&1'
dup2(1, 2)                              = 2

It appears they are the same.

Note:

$ echo "exec 3<&1" > redirect.sh
$ sudo strace -f -s 200 -e trace=dup2 bash redirect.sh
dup2(3, 255)                            = 255
dup2(1, 3)                              = 3

Check this page and the "dup2" man page for details:

$ man 2 dup2

Also, from the bash manpage:

Duplicating File Descriptors
       The redirection operator

              [n]<&word

       is used to duplicate input file descriptors.  If word expands to one or
       more  digits,  the file descriptor denoted by n is made to be a copy of
       that file descriptor.  If the digits in word  do  not  specify  a  file
       descriptor  open for input, a redirection error occurs.  If word evalu‐
       ates to -, file descriptor n is closed.  If n  is  not  specified,  the
       standard input (file descriptor 0) is used.

       The operator

              [n]>&word

       is  used  similarly  to duplicate output file descriptors.  If n is not
       specified, the standard output (file descriptor 1)  is  used.   If  the
       digits  in word do not specify a file descriptor open for output, a re‐
       direction error occurs.  As a special case, if n is omitted,  and  word
       does not expand to one or more digits, the standard output and standard
       error are redirected as described previously.

Chained pipelines

The redirection and piping tokens can be chained together to create complex commands, for example:

ls | grep '.sh' | sort > shlist    

lists the contents of the current directory, where this output is filtered to only contain lines which contain .sh, sort this resultant output alphabetically, and place the final output in shlist.

This type of construction is used very commonly in Linux shell scripts.

Common standard I/O redirections

Note: These have been tested on a bash shell.

External links

• Redirections in the Bash Reference Manual
• Advanced Bash-Scripting Guide
• Using Standard Input and Output
• pv - Pipe Viewer