How "with" is better than try/catch to open a file in Python?

Question

I got that the with statement help you to turn this:

try:
    f = open(my_file)
    do_stuff_that_fails()
except:
    pass
finally:
    f.close()

Into:

with open(my_file) as f:
    do_stuff_that_fails()

But how is that better? You still have to handle the case with the file not being able to be opened (like prompting the user to tell him he doesn't have permissions), so in reality you'd have:

try:
    with open(my_file) as f:
        do_stuff_that_fails()
except (IOError, OSError, Failure) as e:
    do_stuff_when_it_doesnt_work()

Which is equivalent to:

try:
    f = open(my_file)
    do_stuff_that_fails()
except (IOError, OSError, Faillure) as e:
    do_stuff_when_it_doesnt_work()
finally:
    f.close()

Yes, you gained two lines, but you added a level of nesting which doesn't make it easier to read. Is the purpose of the with statement to save you two lines or am I missing something?

It seems a lot to add a keyword just for that, so I feel like there is some syntax to handle the additional try/except that I don't know about.

Solution 1

For a start, it helps prevent the problem you've introduced in your try ... finally ... example.

The way you've structured it, if an exception is thrown while trying to open the file then you will never bind an open file to the name f, leading to either a NameError in the finally clause (if f has never been bound within scope) or something entirely unexpected (if it has).

The correct structure (equivalent to the with) is:

f = open(my_file)

try:
    do_stuff_that_fails()
finally:
    f.close()

(note - no need for an except clause if you've got nothing to do there).

Your second example similarly is wrong, and should be structured like:

try:
    f = open(my_file)

    try:
        do_stuff_that_fails()
    except EXPECTED_EXCEPTION_TYPES as e:
        do_stuff_when_it_doesnt_work()
    finally:
        f.close()

except (IOError, OSError) as e:
    do_other_stuff_when_it_we_have_file_IO_problems()

The second is (as stated in another answer) that you can't forget to call f.close().

BTW, the term is "context management", not "resource management" - the with statement manages contexts, some of which may be resources, but others not. For example, it's also used with decimal to establish a decimal context for a particular block of code.

Finally (responding to your comment to the previous answer) you should never rely on refcount semantics for handling resources in Python. Jython, IronPython and PyPy all have non-refcount semantics, and there's nothing preventing CPython from going the other way (though it's highly unlikely for the immediate future). In a tight loop (e.g. os.walk) it is very very easy to run out of file handles if code relying on refcount semantics is run on a VM with different behaviour.

Solution 2

In the example you give, it's not better. It's best practice to catch exceptions as close to the point they're thrown to avoid catching unrelated exceptions of the same type.

try:
    file = open(...)
except OpenErrors...:
    # handle open exceptions
else:
    try:
        # do stuff with file
    finally:
        file.close()

As unfortunately verbose as this is, the with statement doesn't allow you to catch exceptions thrown during its evaluation. There was a suggestion to add exception handling to this effect on the mailing list:

with open(...) as file:
    # do stuff with file
except OpenErrors...:
    # handle open exceptions

But this was shot down.

Finally it's worth noting that you can directly enter and exit context managers like so:

file = open(...).__enter__()
file.__exit__(typ, val, tb)

This is described in more detail here and here.

As a general guideline, with statements excel for cases where exceptions are not expected, and the default "enter/open/acquire" behaviour is adequate. Examples include required files, and simple locking.

Solution 3

It is for resource management ... not for how you react to an exception otherwise :)

There is no way to "forget" f.close() when using with. In this manner it serves the same role as using in C#.

Happy coding.

Comments

[Charles]

"with" is going to be an awful tag name, subject to newbie abuse. Can we find something better?

[Will]

Python needs a: trywith open('whatever', 'r') as f: .... except(blah): ....

[e-satis]

So it's really all about doing the right thing wihtout having to write it. I like the term resource management, it makes sense. But making a new keyword and a new way of thinking basic operations to save you a finally feels overkill since ressource management is automatically handle in destructors called at garbage collected. Which, unless you write gigantic blocks or a loads of recursive codes, quickly happens at the end of your function or method. I know you should close them explicitly, but it works if you don't. Eventually, how many use cases really benefit from with ?

[Admin]

@e-satis Destructors are not necessarily deterministic in all implementations, even though CPython has been based on ref-counting historically (but this can still bonkered by variable scope, with with it is handled now). I'm not sure why the syntax was chosen, but I love the feature. Perhaps an additional trywith(...): construct would have been neat.

[e-satis]

Right, it's about being pythonic. You make the right thing, it's obvious, automatic, shorter and deterministic. Just about making things a little easier, like decorators that are just syntaxi sugar. Just for that comment, I'll accept the answer.

[e-satis]

I like this idea: mail.python.org/pipermail/python-ideas/2009-May/004903.html with...except But it's been rejected by guido. His answer make sense though: you want exception to be explicitly handled, in a very clear way. But try/except is so ugly around such an elegant with :-)

[e-satis]

+1 I didn't even notice I made such a stupid mistake. Thanks for pointing that out.

[e-satis]

As much as I like your answer, I think @tim's deserve to be the accepted one.

[Admin]

@e-satis It's okay, share the rep :p~