Making BASH script `for` handle filenames with spaces (or workaround)

Question

Whilst I have been using BASH for several years, my experience with BASH scripting is relatively limited.

My code is as below. It should grab the entire directory structure from within the current directory and replicate it into $OUTDIR.

for DIR in `find . -type d -printf "\"%P\"\040"`
do
  echo mkdir -p \"${OUTPATH}${DIR}\"        # Using echo for debug; working script will simply execute mkdir
  echo Created $DIR
done

The problem is, here is a sample of my file structure:

$ ls
Expect The Impossible-Stellar Kart
Five Iron Frenzy - Cheeses...
Five Score and Seven Years Ago-Relient K
Hello-After Edmund
I Will Go-Starfield
Learning to Breathe-Switchfoot
MMHMM-Relient K

Note the spaces :-S And for takes parameters word by word, so my script's output looks something like this:

Creating directory structure...
mkdir -p "/myfiles/multimedia/samjmusicmp3test/Learning"
Created Learning
mkdir -p "/myfiles/multimedia/samjmusicmp3test/to"
Created to
mkdir -p "/myfiles/multimedia/samjmusicmp3test/Breathe-Switchfoot"
Created Breathe-Switchfoot

But I need it to grab whole filenames (one line at a time) from the output of find. I have also tried making find put double-quotes around each filename. But this doesn't help.

for DIR in `find . -type d -printf "\"%P\"\040"`

And output with this changed line:

Creating directory structure...
mkdir -p "/myfiles/multimedia/samjmusicmp3test/"""
Created ""
mkdir -p "/myfiles/multimedia/samjmusicmp3test/"Learning"
Created "Learning
mkdir -p "/myfiles/multimedia/samjmusicmp3test/to"
Created to
mkdir -p "/myfiles/multimedia/samjmusicmp3test/Breathe-Switchfoot""
Created Breathe-Switchfoot"

Now, I need some way that I can iterate through like this, because I also wish to run a more complicated command involving gstreamer on each file in a following similar structure. How should I be doing this?

Edit: I need a code structure which will allow me to run multiple lines of code for each directory/file/loop. Sorry if I was unclear.

Solution: I initially tried:

find . -type d | while read DIR
do
  mkdir -p "${OUTPATH}${DIR}"
  echo Created $DIR
done

This worked fine for the most part. However, I later found that since the pipe results in the while loop running in a subshell, any variables set in the loop were later unavailable which made implementing an error counter quite difficult. My final solution (from this answer on SO):

while read DIR
do
  mkdir -p "${OUTPATH}${DIR}"
  echo Created $DIR
done < <(find . -type d)

This later allowed me to conditionally increment variables within the loop which would remain available later in the script.

Solution 1

You need to pipe the find into a while loop.

find ... | while read -r dir
do
    something with "$dir"
done

Also, you won't need to use -printf in this case.

You can make this proof against files with newlines in their names, if you wish, by using a nullbyte delimiter (that being the only character which cannot appear in a *nix filepath):

find ... -print0 | while read -d '' -r dir
do
    something with "$dir"
done

You will also find using $() instead of backticks to be more versatile and easier. They can be nested much more easily and quoting can be done much more easily. This contrived example will illustrate these points:

echo "$(echo "$(echo "hello")")"

Try to do that with backticks.

Solution 2

See this answer I wrote a few days ago for an example of a script that handles filenames with spaces.

There's a slightly more convoluted (but more concise) way to achieve what you're trying to do though:

find . -type d -print0 | xargs -0 -I {} mkdir -p ../theredir/{}

-print0 tells find to separate the arguments with a null; the -0 to xargs tells it to expect arguments seperated by nulls. This means that it handles spaces fine.

-I {} tells xargs to replace the string {} with the filename. This also implies that only one filename should be used per commandline (xargs will normally stuff as many as will fit on the line)

The rest should be obvious.

Solution 3

The issue you're encountering is the for statement is responding to the find as separate arguments. The space delimiter. You need to use bash's IFS variable to not split on space.

Here is a link that explains how to do this.

The IFS internal variable

One way around this problem is to change Bash's internal IFS (Internal Field Separator) variable so that it splits fields by something other than the default whitespace (space, tab, newline), in this case, a comma.

#!/bin/bash
IFS=$';'

for I in `find -type d -printf \"%P\"\;`
do
   echo "== $I =="
done

Set your find to output your field delimiter after the %P and set your IFS appropriately. I picked semi-colon since it's highly unlikely to found in your filenames.

The other alternative is to call mkdir from the find directly via -exec do you can skip the for loop altogether. That's if you don't need to do any additional parsing.

Solution 4

If the body of your loop is more than a single command, it is possible to use xargs to drive a shell script:

export OUTPATH=/some/where/else/
find . -type d -print0 | xargs -0 bash -c 'for DIR in "$@"; do
  printf "mkdir -p %q\\n" "${OUTPATH}${DIR}"        # Using echo for debug; working script will simply execute mkdir
  echo Created $DIR
done' -

Be sure to include the trailing dash (or some other ‘word’) if the shell is of the Bourne/POSIX variety (it is used to set $0 in the shell script). Also, care must be taken with quoting, since the shell script is being written inside a quoted string instead of directly at the prompt.

Solution 5

find . -type d -exec mkdir -p "{}\040" ';' -exec echo "Created {}\040" ';'

View more solutions

Comments

[Admin]

Why_would_you_ever_need_a_space_in_a_file_name?

[Admin]

True, not my preference. Though, to remove spaces, you need to handle files with spaces first ;)

[Admin]

Actually, file names should allow spaces. I would allow anything but / and unprintable chars. But anything is allowed except / and \0 so you must allow them.

[Darren Hall]

You can pick / on POSIX, and : on DOS filesystems. There are illegal characters for different filesystems that you can pick for the IFS. Anything more complicated and you're better off using perl.

[James Polley]

Also, rather than "$dir", it's preferable to use "${dir}" - it's easy to tell the difference between ${dir}name and ${dirname}, but $dirname could be interpreted either way.

[James Polley]

Dennis Williamson's suggestion is, however (aside from the typos) much more readable, and thus preferable in almost every way.

[Dennis Williamson]

The problem with using / is that it's the directory delimiter and find returns filenames with paths including a slash. Try changing the semicolon in your script to a slash and the echo will print the directory and the filename on separate lines.

[Dennis Williamson]

Thanks for finding the $/" typo. The braces aren't necessary if there's nothing following the variable name.

[quack quixote]

@James Polley: only humans would interpret "$dirname" as "$dir"+"name"; bash always sees it as "$dirname". but you're right, i find the ${var-name} syntax clearer.

[Chris Johnsen]

This will handle pathnames with spaces (U+0020), but it will still fail to properly handle pathnames with line feeds (U+000A). I prefer find … -print0 | xargs -0 … because the delimiter it uses corresponds exactly to the only character that is not allowed in POSIX pathanames: NUL (U+0000).

[Samuel Jaeschke]

Perfect! Just what I was looking for. It had never occurred to me that you might be able to pipe to while. @Chris Johnsen: True, but even music ripping programs don't tend to put linefeeds in their filenames. And if they do, I want to know (ie: something goes wrong) and get rid of them immediately...

[Samuel Jaeschke]

Works, for mkdir, but sorry I should have been more clear - I wish to run a series of commands for each file. You see, for my similar routine later I wish to generate an output filename based on the input filename (which involves stripping the .ogg extension and adding .mp3) and then use these multiple variables in my pipline when invoking gst-launch.

[Samuel Jaeschke]

That also looks quite useful. I've gone with the pipe to while option, but this also looks quite workable. Yes, in my similar structure later I needed to do further parsing. (The input filename would be .ogg, which would be passed as filesrc in the gst pipeline, but an equivalent ending in .mp3 based in the output directory would be generated and also passed to the pipeline as filesink, and this of course needs to be done for each file, along with some echo to the user.)

[Samuel Jaeschke]

No, I need an iterative structure like so, which allows me to run multiple lines of code for each file. "Now, I need some way that I can iterate through like this, because I also wish to run a more complicated command involving gstreamer on each file in a following similar structure." Sorry if I was unclear.

[Samuel Jaeschke]

Another interesting concept. Thanks - I'm sure I'll find a use for this later :)

[akira]

the command i gave solves the problem you asked for, it does not matter if this is just a part of a bigger 'pipeline' on your side. for someone else having the problem as described in the question the rsync-approach will work. so, no need to be sorry about potential unclearity :)

[Samuel Jaeschke]

Thanks. Fixed. It was also reading to FILENAME instead of DIR - copy-paste :P

[Samuel Jaeschke]

Yea. No, I mean I would be using a similar while ... do ... done structure later to do similar processing from find, which would require several lines of code to be run on each file (modify string, echo, gst-launch, etc.) and rsync would not achieve this. That's why I specified that I needed to be able to run a more complicated set of commands within a similar strucutre. My script uses this loop structure twice, so for the question I posted the one with less crud in the middle.