os.mkdir(path) returns OSError when directory does not exist

Question

I am calling os.mkdir to create a folder with a certain set of generated data. However, even though the path I specified has not been created, the os.mkdir(path) raises an OSError that the path already exists.

For example, I call:

os.mkdir(test)

This call results in OSError: [Errno 17] File exists: 'test' even though I don't have a test directory or a file named test anywhere.

NOTE: the actual path name I use is not "test" but something more obscure that I'm sure is not named anywhere.

Help, please?

Solution 1

Greg's answer is correct but doesn't go far enough. OSError has sub-error conditions, and you don't want to suppress them all every time. It's prudent to trap just expected OS errors.

Do additional checking before you decide to suppress the exception, like this:

import errno
import os

try:
    os.mkdir(dirname)
except OSError as exc:
    if exc.errno != errno.EEXIST:
        raise
    pass

You probably don't want to suppress errno.EACCES (Permission denied), errno.ENOSPC (No space left on device), errno.EROFS (Read-only file system) etc. Or maybe you do want to -- but that needs to be a conscious decision based on the specific logic of what you're building.

Greg's code suppresses all OS errors; that's unsafe just like except Exception is unsafe.

As others have pointed out, newer versions of Python provide os.makedirs() that attempts to create the dir only if it doesn't exist, equivalent to mkdir -p from a unix command line.

Solution 2

In Python 3.2 and above, you can use:

os.makedirs(path, exist_ok=True)

to avoid getting an exception if the directory already exists. This will still raise an exception if path exists and is not a directory.

Solution 3

Just check if the path exist. if not create it

import os    
if not os.path.exists(test):
    os.makedirs(test)

Solution 4

I also faced the same problem, specially, when the string 'test' contains the multiple directory name. So when 'test' contains the single directory -

if not os.path.exists(test):
    try:
        os.makedir(test)
    except:
        raise OSError("Can't create destination directory (%s)!" % (test))  

If the 'test' contains multiple directory like '\dir1\dir2' then -

if not os.path.exists(test):
    try:
        os.makedirs(test)
    except:
        raise OSError("Can't create destination directory (%s)!" % (test))  

Solution 5

Happened to me on Windows, maybe this is the case:

Like you I was trying to :

os.mkdir(dirname)

and got OSError: [Errno 17] File exists: '<dirname>'. When I ran:

os.path.exists(dirname)

I got false, and it drove me mad for a while :)

The problem was: In a certain window I was at the specific directory. Even though it did not exists at that time (I removed it from linux). The solution was to close that window \ navigate to somewhere else. Shameful, I know ...

View more solutions

Comments

[krait]

This is the only correct answer. The "look before you leap" approach using os.path.exists that others have suggested creates a race condition: it's entirely possible for the file/dir to be created between the exists and mkdir/makedirs calls, which would generate an unhandled exception.

[Chris Johnson]

This answer is incorrect and dangerous. It creates a race condition, since the directory could be created by another process after the exists call and before the makedirs call. See my answer for a proper solution.

[lordkain]

hehe how many miliseconds do you think there will be between the two statements :) but yes you are correct :)

[Chris Johnson]

It will happen just often enough to drive you crazy trying to debug it :)

[lordkain]

in was kind of environment are you working? and how many processes are there?

[oseiskar]

This is not the "only correct answer" since the race condition is not a relevant problem in all (perhaps even most) use cases.

[Chris Johnson]

@oseiskar, when the right way is as easy or easier than doing it the wrong way, to not do it the right way every time is irresponsible. If you do it the right way every time, you don't have to waste time & energy thinking about it. Good developers focus on developing sound habits.

[oseiskar]

@ChrisJohnson true, but in this case the "as easy" is questionable in terms of readability. The correct balance between these different aspects depends on your use case. Sacrificing readability for resistance to race conditions that never happen in your environment is premature optimization.

[Chris Johnson]

@oseiskar, what I showed is the normal approach for a Python developer. If it's not readable, then whoever's reading it doesn't understand Python very well. Unless you're running on a single-threaded OS, the potential for this particular race condition always exists.

[Chris Johnson]

The if serves no purpose.

[Anupam Bera]

This if statement protects from overwriting the existing directory with the same name.

[Chris Johnson]

No it doesn't. The operating system will prevent the overwriting whether you check before or not. Unless you are running on a single-threaded operating system, another process can create the directory after the time you check the directory doesn't exist and the time you try to create it. The whole point of catching the exception is to deal with what happens if the directory already exists, whether it came into existence last week or 1 microsecond ago. The if serves no purpose. The try/except is the only reliable approach.

[Russia Must Remove Putin]

Two criticisms: 1) the pass is superfluous and 2) I'd rather the raise be bare (preserving the traceback - maybe unimportant here, but I've had problems with losing tracebacks due to well-meaning coders raising a new error, losing the traceback, and thus my clue for what to fix, before.)

[Be Kind To New Users]

Someone famous once said: A one in a million chance is next Tuesday.

[Chris Johnson]

Aaron, I agree. The pass is just a harmless habit, but the raise has a real effect. I'll revise that.

[Chris Johnson]

The error message is incorrect. The OSError exception can happen for many reasons, e.g. insufficient permissions or a read-only file system. You can't conclude that the directory already exists. You need to check the value of exc.errno to determine the cause of the exception.

[Chris Johnson]

This answer is incorrect and dangerous. It creates a race condition, since the directory could be created by another process after the exists call and before the makedirs call. See my answer for a proper solution.

[Sumudu Fernando]

Seems like the scheduler likes to slice between OS calls because this particular case is more likely than you'd think. I just ran into it today with two threads calling into the same library that has basically this code in a @property & with a hardcoded directory name.

[nadrimajstor]

Using exist_ok=True in makedirs would still raise FileExistsError if target path exists and it is not a directory (file, block device, ...) :(

[1'']

@nadrimajstor That's a good point, this isn't a drop-in replacement for catching FileExistsError, although often you do want an exception when you're trying to create a directory on top of a file.

[Akif]

Seems it is a kind of “Ask forgiveness not permission” stackoverflow.com/questions/12265451/…

[Chris Johnson]

@akif yes exactly.

[sox with Monica]

If it's hidden, it can't possibly be called test, right? It must be .test which is a different string

[Roger Heathcote]

Just to play devils advocate - if your code is the only thing on the system that would or could create that folder then your race condition will never occur. Still, I'd prefer errno.EEXIST as then you don't even have to ask that question.

[Michael Krebs]

Race conditions are good to eliminate, but this thread seems overly focused on the race condition in the example code, when a zillion other more common issues can arise and there is no exception handling here at all. Permission errors? Folder in which you are creating the new folder doesn't exist, or worse, is actually a file and not a folder? Out of drive space? Folder name contains invalid characters or is too long? Networked drive is not accessible? Disk is corrupt? Out of memory? Every one of these and many more are not handled.

[gmargari]

Thank you for bringing my sanity back after 3 hours of desperate debugging! From one terminal I had 'cd mydir', then ssh'ed to another machine, and there was no way I could see that I was still inside 'mydir'...