mkdir -p functionality in Python

Question

Is there a way to get functionality similar to mkdir -p on the shell from within Python. I am looking for a solution other than a system call. I am sure the code is less than 20 lines, and I am wondering if someone has already written it?

Solution 1

For Python ≥ 3.5, use pathlib.Path.mkdir:

import pathlib
pathlib.Path("/tmp/path/to/desired/directory").mkdir(parents=True, exist_ok=True)

The exist_ok parameter was added in Python 3.5.

---

For Python ≥ 3.2, os.makedirs has an optional third argument exist_ok that, when True, enables the mkdir -p functionality—unless mode is provided and the existing directory has different permissions than the intended ones; in that case, OSError is raised as previously:

import os
os.makedirs("/tmp/path/to/desired/directory", exist_ok=True)

---

For even older versions of Python, you can use os.makedirs and ignore the error:

import errno    
import os

def mkdir_p(path):
    try:
        os.makedirs(path)
    except OSError as exc:  # Python ≥ 2.5
        if exc.errno == errno.EEXIST and os.path.isdir(path):
            pass
        # possibly handle other errno cases here, otherwise finally:
        else:
            raise

Solution 2

In Python >=3.2, that's

os.makedirs(path, exist_ok=True)

In earlier versions, use @tzot's answer.

Solution 3

This is easier than trapping the exception:

import os
if not os.path.exists(...):
    os.makedirs(...)

Disclaimer This approach requires two system calls which is more susceptible to race conditions under certain environments/conditions. If you're writing something more sophisticated than a simple throwaway script running in a controlled environment, you're better off going with the accepted answer that requires only one system call.

UPDATE 2012-07-27

I'm tempted to delete this answer, but I think there's value in the comment thread below. As such, I'm converting it to a wiki.

Solution 4

Recently, I found this distutils.dir_util.mkpath:

In [17]: from distutils.dir_util import mkpath

In [18]: mkpath('./foo/bar')
Out[18]: ['foo', 'foo/bar']

Solution 5

mkdir -p gives you an error if the file already exists:

$ touch /tmp/foo
$ mkdir -p /tmp/foo
mkdir: cannot create directory `/tmp/foo': File exists

So a refinement to the previous suggestions would be to re-raise the exception if os.path.isdir returns False (when checking for errno.EEXIST).

(Update) See also this highly similar question; I agree with the accepted answer (and caveats) except I would recommend os.path.isdir instead of os.path.exists.

(Update) Per a suggestion in the comments, the full function would look like:

import os
def mkdirp(directory):
    if not os.path.isdir(directory):
        os.makedirs(directory) 

View more solutions

Comments

[Ishbir]

This way, you make it less probable but not impossible that makedirs will fail, in all multitasking operating systems. It's like saying "256 chars should be enough for any path created".

[Setjmp]

I like this solution, though a friend pointed out that the longer exception based solution would only need one file system call. So on a really high latency corporate NFS server, I would want to go with the exception based approach.

[Joe Holloway]

@setjmp I agree. You probably don't want to use this approach in an enterprise setting, but would work just fine for some simple scripting. I inferred that is what you were looking for from the way you phrased your question.

[anonymous]

consider what happens in this code if the path doesn't exist, but you don't have permission to create the folder. You still get an exception.

[Joe Holloway]

@Asa Of course. And mkdir -p would complain about that too. Did I miss your point?

[Ishbir]

@jholloway7: based on the requirements ("mkdir -p"-like functionality) Asa's comment is unnecessary. However, I would like to know whether you do acknowledge that it's possible that the directory can be non-existent when .exists is called, and existent when .makedirs is called.

[Ishbir]

You are absolutely correct about this case; however, the program should catch exceptions later on e.g. when trying to open("/tmp/foo/a_file", "w"), so I don't think an update is necessary. You could update your answer with Python code instead, and watch it being upvoted ;)

[Jacob Gabrielson]

In a lot of cases that would probably be fine. In general, though, I would prefer the code to fail as early as possible so it's clear what really caused the problem.

[Joe Holloway]

@TZ Yes, I certainly acknowledge that. Again, without complete specifications from the original poster, my assumption was that he/she wanted a solution that could be used to create a directory tree if not already existing in a simple script, not a HA enterprisey production solution with SLAs.

[jpsimons]

@Asa That's what exceptions are for, something unexpected went wrong. If you don't have permissions the exception bubbles all the way up and you notice to fix the permissions. As it should be.

[Johan Dahlin]

This version also avoids races as the file system might change between the os.path.exist() and os.makedirs()

[Will Hardy]

In the spirit of micro-improving something lots of people will copy+paste: how about replacing == with != and removing the pass/else :-)

[Ishbir]

@Will: I prefer this method (else: raise), because this way is easier to add handling of more exc.errno cases (elif exc.errno == errno.EACCES: etc)

[Jens Timmerman]

@JohnnyLambada you mean python < 3.0 ?

[Ishbir]

@JensTimmerman: it's obligatory for Python ≤ 2.5 and optional (but still valid) for 2.5 < Python < 3.

[Sam Watkins]

this is clearly a design bug in os.makedirs(): zero and one should not be handled in a special way. ah well, unix made the same mistake for many tools

[jfs]

@Sam there is exist_ok parameter

[jfs]

@SetJmp: it requires at least 2 calls: stat() to check that the parent dir exists and mkdir() to create the directory.

[elhefe]

This appears to fail if the last portion of path is a file, as exc.errno equals errno.EEXIST and so everything seems ok, but actually using the directory later will obviously fail.

[Frank Klotz]

If it already exists AS A DIRECTORY, mkdir -p does not error. It does error if you ask it to create a directory, and a FILE by that name already exists.

[Jacob Gabrielson]

@FrankKlotz that's why I'm calling os.path.isdir not os.path.exists

[auraham]

What about distutils.dir_util.mkpath? It's pretty simple as mkpath('./foo/bar')

[Julian]

Optimize exception handling? except OSError as exc: if exc.errno != errno.EEXIST or not os.path.isdir(path): raise

[romanows]

Beware, mkpath() caches the directory so that you can't re-mkpath() a directory that has been removed with a different method: bugs.python.org/issue10948.

[romanows]

@auraham, mkpath has some unexpected behavior due to undocumented caching that may cause problems if you try to use it exactly like mkdir -p: bugs.python.org/issue10948.

[tsbertalan]

On Python 2.7.6 ... [GCC 4.8.2] on linux2, at least, it seems that it should be os.mkdir, not os.mkdirs.

[Ronan Paixão]

I actually receive an UnicodeDecodeError if the path already exists. Is this a bug? Python (2.7) is probably passing the string received from the OS without checking. It probably contains some non-ascii characters, in which case the above exception fails: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 13: ordinal not in range(128)

[jtpereyda]

@romanows Moreover the method is intended to be private, in case anyone else is tempted to read the bug report to see if it's been 'fixed' (it's not a bug).

[user5359531]

any explanation of what is going on in the except .... raise part?

[Jim Oldfield]

@lambacck: "Other implementations ... have a potential race condition bug." Since you can't lock the file system, all implementations, including this one, have a race condition when considered as part of the code that calls it: another process could delete the directory between you creating it and putting something in it. I think it's best not to worry about such things in the lower level code but have a try block around the higher-level code block to catch them. That's closer to the Python spirit of asking for forgiveness.

[hyperknot]

But this one solves the very common race condition of two threads trying to create the same directory at the exact same time. The not os.path.exists has a high chance of calling the os.makedirs at the same time.

[Eyal Levin]

For python < 3.5 you can use pathlib2. pip install pathlib2; from pathlib2 import Path

[Dave00Galloway]

this is a nice answer for 2.7, seems cleaner than trapping an error

[Dave00Galloway]

fails if part of the tree already exists though, so here's a fix:- import os; from os.path import join as join_paths def mk_dir_recursive(dir_path): if os.path.isdir(dir_path): return h, t = os.path.split(dir_path) # head/tail if not os.path.isdir(h): mk_dir_recursive(h) new_path = join_paths(h, t) if not os.path.isdir(new_path): os.mkdir(new_path)

[jkdev]

To protect against race conditions, you can install IPython and use IPython.utils.path.ensure_dir_exists()

[Erik Aronesty]

Or just: os.makedirs(path, exist_ok=True)

[wchargin]

−1 because the only part of this answer that actually answers the question (the last code block) answers it incorrectly and also duplicates other answers.

[Ryan Jensen]

adding simplicity to tzot's original solution: def mkdir_p(path): if not os.path.isdir(path): os.makedirs(path);

[Joe Holloway]

@RyanJensen The beef everyone has with my answer is that it makes two system calls and is not a single atomic operation. Yours has the same flaw, so just be mindful that it's only usable in very constrained applications and not the best idea for a general purpose solution

[Boris Verkhovskiy]

@MauroBaraldi the point is that if you create a directory with this method, it gets deleted and you try to create it again using this method from the same program, it won't work. Don't use this.

[Boris Verkhovskiy]

If you use pathlib, then Path("/tmp/my/file").mkdir(parents=True, exist_ok=True) it behaves the same way too. It'll raise a FileExistsError

[j4hangir]

Wish they'd named this mkdirs in accordance with mkdir and not suddenly makedirs ...

[Milan Lakhani]

Is pathlib preferable to OS? Why?

[JGFMK]

I don't like this method - prefer the os.mkdir option. SIlent continue if you delete and recreate folders in conjunction with shutil.rmtree - leaving a lock. os version gives access denied -and abends early rather than later. Setting up a folder of results from a huge conversion which would not be able to output results

[Clint Eastwood]

the first option is susceptible to race conditions (at one instant, the dir is not there, so we proceed to create it but in the middle something else creates it and boom!) second option is the way to go in Python 2