When I redirect the output of ls to a file, the filename is included in that file. How can I avoid this?

Question

Observe:

$ ls
$ ls > list
$ cat list
list

This appears to indicate that when ls is executed that the redirection into file list has already begun and the list file is already created. A fine enough explanation at any rate, but the question is this: How can I prevent this from happening? What I expected to happen was ls would execute and its output dumped into list and that is what I want.

Solution 1

As you've noticed, the file is created before ls is run. This is due to how the shell handles its order of operations. In order to do

ls > file

the shell needs to create file and then set stdout to point to that and the finally run the ls program.

So you have some options.

1. Create the file in another directory (eg /tmp) and then mv it to the final directory
2. Create it as a hidden file (.file) and rename it
3. Use grep to remove the file from the output
4. Cheat :-)

The cheat would be something like

x=$(ls) ; printf "%s\n" "$x" > file

This causes the output of ls to be held in a variable, and then we write that out.

Solution 2

The output file is created by the shell before ls begins. You can get around this by using tee:

ls | tee list

To thoroughly defeat any race condition, there is always

ls | grep -vx 'list' > list

Or if you like that tee displays the results as well:

ls | grep -vx 'list' | tee list

However, as pointed out in comments, things like this often break when filenames contain strange characters. Unix filenames can generally contain any characters except for NUL and /, so parsing the output of ls is extremely difficult:

• Assigning to a shell variable can fail if a filename ends in one or more \n.
• Filtering with grep fails when the search term lies between \n.
• You can separate filenames with NUL instead of \n using find, but it can be difficult to convert this into something resembling the traditional sorted, newline-separated output of ls.
• Removing the output filename from the list may be incorrect if it already exists.

So the only truly effective way to do this is to create the output file somewhere else, and move it into place. If you will never use ls -a, then this works:

ls > .list && mv .list list

If you might be using ls -a, then .list could appear in your output but no longer exist in the directory. So then you would use a different directory, like /tmp to store the intermediate result. Of course, if you always use /tmp you run into trouble there, so you can write a script:

#!/bin/sh
OUTDIR='/tmp'
if [ "${PWD}" = '/tmp' ]; then
  OUTDIR="${HOME}"
fi
ls > "${OUTDIR}/list" && mv "${OUTDIR}/list" list

This seems overly complicated for the task, though.

But the entire cause of the issue is that the shell is creating the output file before the command begins. We can take that into consideration and just have the shell list the files for us. Then we don't even need ls at all!

printf '%s\n' * > list

This will work until you have too many files in the directory to fit into an argument list.

Solution 3

You can use moreutils sponge:

ls | sponge list

Or with zsh:

cp =(ls) list

With GNU ls:

ls -I list > list

(though if there had been a file called list before, that means it won't be listed).

Since ls output is sorted anyway, you can also use (assuming your filenames don't contain newline characters):

ls | sort -o list

Or to avoid the double sorting, if your ls supports -U for Unsorted (beware some ls implementations have a -U for something else):

ls -U | sort -o list

Solution 4

You can make the filename temporarily hidden:

ls >.list && mv .list list

Solution 5

Partial/most credit goes to @StephenHarris...

echo "`ls`" > list

equivalent to

echo "$(ls)" > list

View more solutions

Comments

[Steven Lu]

OK i hadn't thought of this yet. It touches the filesystem a few times but it's somewhat intelligible, which is a big plus... Yes I'm looking for something concise and not too convoluted. Thanks

[Steven Lu]

Very neat! so here we are using the tee program to write to the file rather than leaving that task to the shell to do (which the shell ends up being overeager with for this particular situation)

[Steven Lu]

I actually love that it shows the output on stdout... tee is really useful.

[Stephen Harris]

This fails to solve the problem if there are too many files in the directory. There's an inherent race condition. In an empty (or few files) directory it works, but with a lot of files the tee process can create the file before the ls has had a chance to read the whole directory.

[Stephen Harris]

I did it with 10,000 files and saw the race condition :-) I was in the middle of writing it as an answer until I spotted it. We can kludge with something nasty like (sleep 1 ; tee file), but it's nasty.

[Stephen Harris]

Yup, grep is one of the options I listed as well. Mostly I try to avoid this problem by using hidden files or other directories 'cos it's just nasty :-)

[Steven Lu]

Nice. Your last paragraph prompted me to try echo `ls` > ls which I love for being so concise, but this doesnt put an entry on each line which my particular situation has issues with (as I very much do need to deal with directories with spaces in them today)...

[Stephen Harris]

ls -1 will force single-column output

[Steven Lu]

@StephenHarris Yeah I think i might as well give you the accept so somebody gets points, this answer is 99% of the way there. Do you think there are any race conditions or failure modes of echo "$(ls -1)" > file?

[Stephen Harris]

Yes, there are potential race conditions. If you want to use this sort of cheat you really should split the ls and the file creation into two steps. Also beware of echo potentially interpreting the output of the ls; (what if you have a file called -ne ?). That's why I used printf in my example.

[Stéphane Chazelas]

grep -v '^list$' or more simply: grep -vx list removes the line corresponding to the file called list, but also all those coming from filenames containing <newline>list<newline>

[Stéphane Chazelas]

Note that x=$(ls) removes all the trailing newline characters from the output of ls while your printf '%s\n' adds only one back.

[Overmind Jiang]

@StevenLu: you could use echo with double quotes — echo "`ls`" > ls or echo "$(ls)".

[Gilles 'SO- stop being evil']

You don't need the -1 option. It's implicit whenever ls isn't writing to a terminal (such as here, where it's writing to a pipe that the shell reads from).

[Gilles 'SO- stop being evil']

@StevenLu There are no race conditions, the order of evaluation is well-defined: the command substitution $(ls) is evaluated before the redirection >file. The reason you aren't getting one file per line is that you didn't use double quotes: echo "$(ls)" >file

[Steven Lu]

@Gilles Right you are. I started trying it with -1 in hopes it would fix it and it stuck around. I've edited the answer.