recursive mkdir

Question

Is there a linux command that I'm overlooking that makes it possible to do something along the lines of: (pseudo)

$ mkdir -R foo/bar/zoo/andsoforth

Or is there no alternative but to make the directories one at a time?

Solution 1

$ mkdir -p foo/bar/zoo/andsoforth

Parameter p stands for 'parents'.

Solution 2

Using mkdir -p is a simple way for most modern OSes:

mkdir -p foo/bar/zoo/andsoforth

However, mkdir -p is not recommended in many manuals. Read documentation for of GNU make and autoconf about problems with using mkdir -p:

• http://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Limitations-of-Usual-Tools.html#Limitations-of-Usual-Tools
• http://www.gnu.org/savannah-checkouts/gnu/make/manual/html_node/Utilities-in-Makefiles.html

The cross platform installation and configuration systems have their own safe alternatives for mkdir -p.

CMake to use in shell command line:

 cmake -E make_directory foo/bar/zoo/andsoforth

Autoconf to use in script with preprocessing:

AS_MKDIR_P(foo/bar/zoo/andsoforth)

or:

AC_PROG_MKDIR_P(foo/bar/zoo/andsoforth)

But these solutions require cmake or autoconf (M4) tools to be installed (and possible preprocessing)

You can use also install-sh script with -d option:

install-sh -d foo/bar/zoo/andsoforth

This script is used by autoconf and automake project. I think it must be the safest solution.

At the time I was searching for a cross platform solution for standard /bin/sh without dependences, but haven't found one. Therefore I wrote the next script that may be not ideal, but I think it is compliant to most cross platform requirements:

#! /bin/sh

cdirname() # cross platform alternative for 'dirname'
{
  # $1 - path
  test $# -eq 1 || { echo "Procedure 'cdirname' must have only one parameter. Scripting error."; exit 1; }
  echo "$1" | sed -n -e '1p' | sed  -e 's#//*#/#g' -e 's#\(.\)/$#\1#' -e 's#^[^/]*$#.#' -e 's#\(.\)/[^/]*$#\1#' -
}

mkd() # cross platform alternative for 'mkdir -p'
{
  # $1 - directory to create
  test $# -eq 1 || { echo "Function 'mkd' can create only one directory (with it's parent directories)."; exit 1; }
  test -d "$1"  && return 0
  test -d "$(cdirname "$1")" || { mkd "$(cdirname "$1")" || return 1; }
  test -d "$1" || { mkdir "$1" || return 1; }
  return 0
}

This script can be used for old systems, where option -p for mkdir is absent.

sed-based cross platform version of dirname was added to the code. It works with a way similar to dirname (correct with path /, paths with base name only, paths with trailing /, paths with and without trailing \ns). This function can't work correct if the path has newlines or some invalid characters for current locale. It also replaces any combination of / (//, ///) with single /

Changed line mkdir "$1" || return 1 to test -d "$1" || { mkdir "$1" || return 1; } because mkdir terminates with error if path exists and this check is needed for paths containing constructions like aaa\. (If aaa doesn't exist previous version creates aaa and then tries to create it again).

This version of mkd doesn't generate an error if path already exist (but it still has the possibility to generate such an error in parallel execution) and can't get several directories in command line.

Solution 3

Option -p in command mkdir makes parent directories as needed (no error if existing):

mkdir -p foo/bar/zoo/andsoforth

Another way is, for example using && (error if the specified folder exists):

mkdir foo && mkdir foo/bar && mkdir foo/bar/zoo && mkdir foo/bar/zoo/andsoforth

The chaining operator && is used to chain commands together, such that the next command is run if and only if the preceding command exited without errors.

Of course, the first way is better.

Comments

[Admin]

linux.die.net/man/1/mkdir

[Admin]

funny, I was looking at that page but totally overlooked "parent" because I was thinking of them as children (left to right).

[Admin]

Protip: In some shells you can even do mkdir -p foo/{bar,baz}/zoo/andsoforth. Very useful!

[Admin]

Personally, I overlooked "parent" in the man page as well because, well, I feel like the flag should be "-r" for "recursive" - or at the very least, there should be an alias for such since mkdir has only 6 documented flags in total. Hell, I feel like the command should create directories recursively by default, and if you don't want this then you should have to specify so explicitly.

[Tim B]

"mkdir -p" is defined by posix. What unix or unix-like variants don't support it? (and it shouldn't be a shell built-in, though I guess there isn't any reason it couldn't be.) pubs.opengroup.org/onlinepubs/9699919799/utilities/mkdir.htm‌​l

[Роман Коптев]

@TimB For example: don’t use ‘mkdir -p’, convenient as it may be, because a few systems don’t support it at all and with others, it is not safe for parallel execution. For a list of known incompatibilities, see Portable Shell Programming in Autoconf. Source: gnu.org/savannah-checkouts/gnu/make/manual/html_node/…

[Роман Коптев]

@TimB Read also here about problems with mkdir -p gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69‌​/…

[Marius]

Some of the information in that page is only 15 years old. Some is older.

[Роман Коптев]

@ThomasDickey It's incorrect. Information on those pages is written after 2012 as autoconf 2.69 is actual version of autoconf 2012. Now the year is 2016. The make documentation is actual too.

[Stéphane Chazelas]

Systems that wouldn't support mkdir -p would be too old (and we're talking at least a couple of decades old) to have a printf utility or support -- to mark the end of options.

[Stéphane Chazelas]

printf -- "$1\n" is wrong. The first non-option argument to printf is the format. It should be printf '%s\n' "$1"

[Роман Коптев]

@StéphaneChazelas printf is marked as one of the safest functions and must be present even in minimal shell installation. It's more safe then using echo as echo uses different options on some platforms and has a question with the newline. About "--" I can't say 100% from there it started to be used. It can be omited. It is here for safety only.

[Stéphane Chazelas]

Your cdirname would give the wrong answer for foo/bar//, or directory names containing newline characters, or invalid characters in the current locale.

[Роман Коптев]

@StéphaneChazelas I think no one installation system now can work with newlines and invalid characters in the names, even cmake can't. Combination // is not possible in my script, but I'll patch this.

[Роман Коптев]

@StéphaneChazelas Mostly I follow to recommendations you can see by the links. The make documentation notes: "The configure script and the Makefile rules for building and installation should not use any utilities directly except these: awk cat cmp cp diff echo egrep expr false grep install-info ln ls mkdir mv printf pwd rm rmdir sed sleep sort tar test touch tr true".So I think those configurations now are not supported by installation systems at all. Not only by this script. I don't tested cmake -E make_directory for paths with newlines, but generally cmake doesn't support such installations.

[Tim B]

The only complaint I see in those links about mkdir -p is the race condition in simultaneous execution of mkdir -p a/b and mkdir -p a/c, where both see a/ needs to be created, but the second process to call mkdir(2) finds that the other process has already created it. That isn't a concern in interactive use, and it certainly isn't "a few systems don't support it." Further, I don't think gnu make docs on on what utilities to use in makefiles is appropriately applied beyond that context. What's more likely to be present, mkdir that supports -p, or autoconf?. For me, the former.

[Stéphane Chazelas]

While the autoconf doc makes sense (but bear in mind that autoconf's target includes very old systems), that make documentation (actually that section comes from gnustandards, not make) doesn't make much sense. printf was added without much of a justification to that OK list in 2010.

[Роман Коптев]

@Tim 1. For that problem with simultaneous execution my script also doesn't have any problem, as I don't treat the existence of the folder as error. This pointed only as one problems with mkdir -p. I've read many manuals that say don't use mkdir -p. 2. The recommended rules of make files doesn't recommend using mkdir -p and I follow them. I need only at least unix make working on the platform for my projects, and I don't interesting for systems that doesn't support make files and their minimal syntax as written in make documentation. At this point the question must be closed.

[Tim B]

I commented initially because I think your comment that mkdir -p is not cross platform is wrong and misleading. I still believe it is.

[Роман Коптев]

@Tim. It's not my personal opinion. I wrote because I can't use CPack in my project and I write an installation shell script by hand. The cmake and autoconf use unix make files anyway. The unix makefile have some minimal requirements on shell. I need to write a shell script and I follow those requirements. If system can't support them, I don't know that I would can to do except to use some other tools or file syntax that that patform supports. At the moment I support only system compliant to those make file requirements (and even less any way).

[Роман Коптев]

@TimB I have mistaked, it can have an error in parallel execution. And may be I'll change the part of cdirname for working with '//'. I don't want to explore all possible operation systems, I trust make documentation, if I'll have problems in the future, I'll solve them separately.

[Stéphane Chazelas]

For the record, I've started a thread on the gnustandards mailing list about that list of commands in the GNU make manual.