Create a "with" block on several context managers?

Question

Suppose you have three objects you acquire via context manager, for instance A lock, a db connection and an ip socket. You can acquire them by:

with lock:
   with db_con:
       with socket:
            #do stuff

But is there a way to do it in one block? something like

with lock,db_con,socket:
   #do stuff

Furthermore, is it possible, given an array of unknown length of objects that have context managers, is it possible to somehow do:

a=[lock1, lock2, lock3, db_con1, socket, db_con2]
with a as res:
    #now all objects in array are acquired

If the answer is "no", is it because the need for such a feature implies bad design, or maybe I should suggest it in a pep? :-P

Solution 1

In Python 2.7 and 3.1 and above, you can write:

with A() as X, B() as Y, C() as Z:
    do_something()

This is normally the best method to use, but if you have an unknown-length list of context managers you'll need one of the below methods.

---

In Python 3.3, you can enter an unknown-length list of context managers by using contextlib.ExitStack:

with ExitStack() as stack:
    for mgr in ctx_managers:
        stack.enter_context(mgr)
    # ...

This allows you to create the context managers as you are adding them to the ExitStack, which prevents the possible problem with contextlib.nested (mentioned below).

contextlib2 provides a backport of ExitStack for Python 2.6 and 2.7.

---

In Python 2.6 and below, you can use contextlib.nested:

from contextlib import nested

with nested(A(), B(), C()) as (X, Y, Z):
    do_something()

is equivalent to:

m1, m2, m3 = A(), B(), C()
with m1 as X:
    with m2 as Y:
        with m3 as Z:
            do_something()

Note that this isn't exactly the same as normally using nested with, because A(), B(), and C() will all be called initially, before entering the context managers. This will not work correctly if one of these functions raises an exception.

contextlib.nested is deprecated in newer Python versions in favor of the above methods.

Solution 2

UPDATE: Starting in python 3.10, you'll be able to use parenthesized context managers! Thanks @iforapsy!

with (
    mock.patch('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa') as a,
    mock.patch('bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb') as b,
    mock.patch('cccccccccccccccccccccccccccccccccccccccccc') as c,
):
    do_something()

Original Answer For python versions < 3.10

@interjay's Answer is correct. However, if you need to do this for long context managers, for example mock.patch context managers, then you quickly realize you want to break this across lines. Turns out you can't wrap them in parens, so you have to use backslashes. Here's what that looks like:

with mock.patch('aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa') as a, \
        mock.patch('bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb') as b, \
        mock.patch('cccccccccccccccccccccccccccccccccccccccccc') as c:
    do_something()

Solution 3

The first part of your question is possible in Python 3.1.

With more than one item, the context managers are processed as if multiple with statements were nested:

with A() as a, B() as b:
    suite

is equivalent to

with A() as a:
    with B() as b:
        suite

Changed in version 3.1: Support for multiple context expressions

Solution 4

The second part of your question is solved with contextlib.ExitStack in Python 3.3.

Solution 5

Following on from @sage88's response you can always assign those patches to have meaningful variable names before you enter into them.

You could create those patches in multiple lines

a_patch = mock.patch('aaaaaaa') 
b_patch = mock.patch('bbbbbbb')
c_patch = mock.patch('ccccccc') 
with a_patch as a, b_patch as b, c_patch as c:    
    do_something()

View more solutions

Comments

[Graham]

I've started a meta discussion about reversing the duplicate target for this question.

[Graham]

@timgeb I've started a discussion about this duplicate closure. My apologies for not @-notifying you sooner; it slipped my mind.

[olamundo]

thanks! but that still didn't answer my whole question: what about the 2nd case I mentioned, where the context managers are given in an array, without knowing how many mangers are there in the array. will it be possible in some python3.X to do with [cm1,cm2,cm3,cm4,cm5] as result: ....

[Mark Byers]

@noam: To solve the second part of your question you could write a class to wrap a number of resources and implement __enter__ and __exit__ for that class. I'm not sure if there's a standard library class that does this already.

[olamundo]

thanks! So I can use this for an array of context managers with contextlib.nested(*arr).<br> Is this possible somehow in python 2.7 and 3.1's new syntax?

[interjay]

@noam: No, in fact the docstring for nested in 3.1 says: "The one advantage of this function over the multiple manager form of the with statement is that argument unpacking allows it to be used with a variable number of context managers as follows: with nested(*managers): do_something()"

[olamundo]

Odd, on one hand it's deprecated, but on the other they acknowledge an advantage of the deprecated module over the replacement?

[glglgl]

@Mark I don't think it is that easy - that's why contextlib.nested() is deprecated. If something happens between the generation of the other things and the activation of the context manager, it might happen that the cleanup doesn't happen as wanted.

[Martijn Pieters]

@noam: no, contextlib.ExitStack is the replacement for contextlib.nested().

[Martijn Pieters]

But usually this makes no difference. It makes a huge difference if B() or C() raises an exception. The nested with statement will call A().__exit__(), but the contextlib.nested() setup will not!

[interjay]

@MartijnPieters: Yes, "usually" was definitely too strong a word. I've clarified it. And thanks for the reference to the new ExitStack, I added an example.

[Neil G]

+1: for adding ExitStack and completing your answer

[Darren Ringer]

One issue: Using the simple "with open(A) as a, open(B) as b:" style syntax, line breaks make pep8 compliance as reported by standard tools seemingly impossible. I use a backslash to signify the line break, because surrounding the comma-separated expressions with parentheses results in the report of a syntax error. With the backslash, I get an "E127 continuation line over-indented" warning. I have yet to find a way to use this syntax while suppressing all warnings.

[igordc]

And any ideas on having a full replacement of contextlib.nested in Python 2.7 itself?

[interjay]

@igordcard contextlib.nested is still in Python 2.7, just marked as deprecated. The docs say "Developers that need to support nesting of a variable number of context managers can either use the warnings module to suppress the DeprecationWarning raised by this function or else use this function as a model for an application specific implementation.".

[sage88]

@DarrenRinger I had this same problem. I, too, used backslashes, which I abhor, to accomplish this. Double indent all the context managers after the initial with line. Only single indent the content being wrapped. It passes flake8 for me.

[jpmc26]

@MartijnPieters Importantly, though, the nested call won't call the __enter__ methods of any of the context managers if B() or C() raise an exception. That could be a desirable difference. This is a good reason to keep any side effects out of the initializers and put it in __enter__, which is what you should be doing anyway. I'd argue breaking that convention is un-Pythonic. As long as the context managers follow this convention, there's no risk involved. (interjay: I'd recommend incorporating that detail into the answer, or I could do so if you don't mind.)

[interjay]

@jpmc26 Unfortunately, this won't help with file objects, which are probably the most common use for the with statement. And the same will be true for other similar objects for which using the with statement is optional. Also, there is another problem with nested mentioned in the documentation: "if the __enter__() method of one of the inner context managers raises an exception that is caught and suppressed by the __exit__() method of one of the outer context managers, this construct will raise RuntimeError rather than skipping the body of the with statement."

[John Deighan]

ugly. Why not allow wrapping in parens (I got bit by this)

[sage88]

@JohnDeighan I agree, it is ugly. I also got bit by it, which is why I posted this. I hope this was helpful, though I agree, I wish it supported paren wrapping.

[isarandi]

@JohnDeighan Why are line breaks with \ uglier than using parens?

[Stef]

To avoid the backslashes you can format it like this (had to link it as an image, as comments don't allow line breaks and this question is closed for additional answers).

[Boris Verkhovskiy]

This answer is perfect. ExitStack is exactly what I was looking for. Thank you very much for keeping this updated.

[ThorSummoner]

mock.patch should evaluate to an object. can we also write a_ctx = mock.patch('a'); ...; with a_ctx as a, ...:

[iforapsy]

Good news, folks. Parenthesized context managers will be valid syntax starting in Python 3.10, thanks to a new parser.

[sage88]

@iforapsy fantastic, I'll look into it and update my answer accordingly.