How do file descriptors work?

Question

Can someone tell me why this does not work? I'm playing around with file descriptors, but feel a little lost.

#!/bin/bash
echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4

The first three lines run fine, but the last two error out. Why?

Solution 1

File descriptors 0, 1 and 2 are for stdin, stdout and stderr respectively.

File descriptors 3, 4, .. 9 are for additional files. In order to use them, you need to open them first. For example:

exec 3<> /tmp/foo  #open fd 3.
echo "test" >&3
exec 3>&- #close fd 3.

For more information take a look at Advanced Bash-Scripting Guide: Chapter 20. I/O Redirection.

Solution 2

It's an old question but one thing needs clarification.

While the answers by Carl Norum and dogbane are correct, the assumption is to change your script to make it work.

What I'd like to point out is that you don't need to change the script:

#!/bin/bash
echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4

It works if you invoke it differently:

./fdtest 3>&1 4>&1

which means to redirect file descriptors 3 and 4 to 1 (which is standard output).

The point is that the script is perfectly fine in wanting to write to descriptors other than just 1 and 2 (stdout and stderr) if those descriptors are provided by the parent process.

Your example is actually quite interesting because this script can write to 4 different files:

./fdtest >file1.txt 2>file2.txt 3>file3.txt 4>file4.txt

Now you have the output in 4 separate files:

$ for f in file*; do echo $f:; cat $f; done
file1.txt:
This
file2.txt:
is
file3.txt:
a
file4.txt:
test.

What is more interesting about it is that your program doesn't have to have write permissions for those files, because it doesn't actually open them.

For example, when I run sudo -s to change user to root, create a directory as root, and try to run the following command as my regular user (rsp in my case) like this:

# su rsp -c '../fdtest >file1.txt 2>file2.txt 3>file3.txt 4>file4.txt'

I get an error:

bash: file1.txt: Permission denied

But if I do the redirection outside of su:

# su rsp -c '../fdtest' >file1.txt 2>file2.txt 3>file3.txt 4>file4.txt

(note the difference in single quotes) it works and I get:

# ls -alp
total 56
drwxr-xr-x 2 root root 4096 Jun 23 15:05 ./
drwxrwxr-x 3 rsp  rsp  4096 Jun 23 15:01 ../
-rw-r--r-- 1 root root    5 Jun 23 15:05 file1.txt
-rw-r--r-- 1 root root   39 Jun 23 15:05 file2.txt
-rw-r--r-- 1 root root    2 Jun 23 15:05 file3.txt
-rw-r--r-- 1 root root    6 Jun 23 15:05 file4.txt

which are 4 files owned by root in a directory owned by root - even though the script didn't have permissions to create those files.

Another example would be using chroot jail or a container and run a program inside where it wouldn't have access to those files even if it was run as root and still redirect those descriptors externally where you need, without actually giving access to the entire file system or anything else to this script.

The point is that you have discovered a very interesting and useful mechanism. You don't have to open all the files inside of your script as was suggested in other answers. Sometimes it is useful to redirect them during the script invocation.

To sum it up, this:

echo "This"

is actually equivalent to:

echo "This" >&1

and running the program as:

./program >file.txt

is the same as:

./program 1>file.txt

The number 1 is just a default number and it is stdout.

But even this program:

#!/bin/bash
echo "This"

can produce a "Bad descriptor" error. How? When run as:

./fdtest2 >&-

The output will be:

./fdtest2: line 2: echo: write error: Bad file descriptor

Adding >&- (which is the same as 1>&-) means closing the standard output. Adding 2>&- would mean closing the stderr.

You can even do a more complicated thing. Your original script:

#!/bin/bash
echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4

when run with just:

./fdtest

prints:

This
is
./fdtest: line 4: 3: Bad file descriptor
./fdtest: line 5: 4: Bad file descriptor

But you can make descriptors 3 and 4 work, but number 1 fail by running:

./fdtest 3>&1 4>&1 1>&-

It outputs:

./fdtest: line 2: echo: write error: Bad file descriptor
is
a
test.

If you want descriptors both 1 and 2 fail, run it like this:

./fdtest 3>&1 4>&1 1>&- 2>&-

You get:

a
test.

Why? Didn't anything fail? It did but with no stderr (file descriptor number 2) you didn't see the error messages!

I think it's very useful to experiment this way to get a feeling of how the descriptors and their redirection work.

Your script is a very interesting example indeed - and I argue that it is not broken at all, you were just using it wrong! :)

Solution 3

It's failing because those file descriptors don't point to anything! The normal default file descriptors are the standard input 0, the standard output 1, and the standard error stream 2. Since your script isn't opening any other files, there are no other valid file descriptors. You can open a file in bash using exec. Here's a modification of your example:

#!/bin/bash
exec 3> out1     # open file 'out1' for writing, assign to fd 3
exec 4> out2     # open file 'out2' for writing, assign to fd 4

echo "This"      # output to fd 1 (stdout)
echo "is" >&2    # output to fd 2 (stderr)
echo "a" >&3     # output to fd 3
echo "test." >&4 # output to fd 4

And now we'll run it:

$ ls
script
$ ./script 
This
is
$ ls
out1    out2    script
$ cat out*
a
test.
$

As you can see, the extra output was sent to the requested files.

Solution 4

To add on to the answer from rsp and respond the question in the comments of that answer from @MattClimbs.

You can test if the file descriptor is open or not by attempting to redirect to it early and if it fails, open the desired numbered file descriptor to something like /dev/null. I do this regularly within scripts and leverage the additional file descriptors to pass back additional details or responses beyond return #.

script.sh

#!/bin/bash
2>/dev/null >&3 || exec 3>/dev/null
2>/dev/null >&4 || exec 4>/dev/null

echo "This"
echo "is" >&2
echo "a" >&3
echo "test." >&4

The stderr is redirected to /dev/null to discard the possible bash: #: Bad file descriptor response and the || is used to process the following command exec #>/dev/null when the previous one exits with a non zero status. In the event that the file descriptor is already opened, the two tests would return a zero status and the exec ... command would not be executed.

Calling the script without any redirections yields:

# ./script.sh
This
is

In this case, the redirections for a and test are shipped off to /dev/null

Calling the script with a redirection defined yields:

# ./script.sh 3>temp.txt 4>>temp.txt
This
is
# cat temp.txt
a
test.

The first redirection 3>temp.txt overwrites the file temp.txt while 4>>temp.txt appends to the file.

In the end, you can define default files to redirect to within the script if you want something other than /dev/null or you can change the execution method of the script and redirect those extra file descriptors anywhere you want.

View more solutions

Comments

[Trcx]

That's what I'm looking for! So I need to specify a file for it to use as a temporary storage place with the exec command, and then close them when i'm done? Sorry, I'm a little fuzzy with the exec command, I don't use it much.

[dogbane]

yes, but it's not temporary. The file will exist even after your program completes.

[Trcx]

Is there a way I could have it write the out put to the terminal? I want to be able to be able to see it all in terminal, but want to be able to send the output where I want. i.e. ./script 2>out.2 3>out.3 4>out.4

[Carl Norum]

@Trcx, if you want to write to the terminal, use stdout or stderr. Why would you need or want to use other files for that?

[Trcx]

That could work out nicely then, I'm trying to port some scripts to be compatible with crontab task scheduler, but I am having trouble as cron does not allow for the piping of stdout in scripts.

[Trcx]

The scripts that I am trying to make compatible with crontab need to write multiple files out, but crontab does not allow for files to be written to from scripts (because of it lacks support from stdout) However I can use crontab to write the output of the scripts to a file. I was thinking that I could have the scripts write to the various outputs, and then have crontab separate everything into the appropriate files. I was just looking for a cleaver way to write to the files with out using stdout. But thanks to you guys I found out that I was overthinking it. (again :P ) Thanks for the help!

[Felipe Alvarez]

You don't have to specify a file. You can also make fd 3 point to the same file that fd 1 points to exec 3>&1 causes echo hi >&3 to print "hi" to stdout.

[Gunith D]

V interesting response.. Thanks

[programmerjake]

actually, file descriptor 1 is stdout; stdin is file descriptor 0.

[rsp]

@programmerjake Oops. Typo fixed. Thanks for pointing it out.

[solfish]

why echo "is" make stdout to stderr >&2 ?

[MattCochrane]

This is a fantastic response. Thanks! Is there a way to check if a file exists? ie. if you run ./fdtest without 3>&1 4>&1, is there a way for fdtest to determine that file descriptors 3 and 4 do not exist?

[clapas]

Why 3<> instead of just 3> in the first line?

[Yzmir Ramirez]

@clapas , 3<> is for reading and writing. Since we only write to this example, then yes 3> would be correct in this snippet; however, typically we want to read from the descriptor something we wrote to.

[Ajax]

...this response, while excellent in many ways, gets one important thing wrong: the magic of "put the 3>file.txt outside of single quotes" is because su is running the program as another user, and root is redirecting that to a file in a directory your ls -la shows is also owned by root. There's no permission-breaking trick here, just that "putting your redirections inside a string to be executed by a user-switching subprogram (su in your case) will be checked as the user the subprogram executes as". Semantically, the redirects apply to the command immediately before they are declared.