Pythonic way to combine FOR loop and IF statement

Question

I know how to use both for loops and if statements on separate lines, such as:

>>> a = [2,3,4,5,6,7,8,9,0]
... xyz = [0,12,4,6,242,7,9]
... for x in xyz:
...     if x in a:
...         print(x)
0,4,6,7,9

And I know I can use a list comprehension to combine these when the statements are simple, such as:

print([x for x in xyz if x in a])

But what I can't find is a good example anywhere (to copy and learn from) demonstrating a complex set of commands (not just "print x") that occur following a combination of a for loop and some if statements. Something that I would expect looks like:

for x in xyz if x not in a:
    print(x...)

Is this just not the way python is supposed to work?

Solution 1

You can use generator expressions like this:

gen = (x for x in xyz if x not in a)

for x in gen:
    print(x)

Solution 2

As per The Zen of Python (if you are wondering whether your code is "Pythonic", that's the place to go):

• Beautiful is better than ugly.
• Explicit is better than implicit.
• Simple is better than complex.
• Flat is better than nested.
• Readability counts.

The Pythonic way of getting the sorted intersection of two sets is:

>>> sorted(set(a).intersection(xyz))
[0, 4, 6, 7, 9]

Or those elements that are xyz but not in a:

>>> sorted(set(xyz).difference(a))
[12, 242]

But for a more complicated loop you may want to flatten it by iterating over a well-named generator expression and/or calling out to a well-named function. Trying to fit everything on one line is rarely "Pythonic".

---

Update following additional comments on your question and the accepted answer

I'm not sure what you are trying to do with enumerate, but if a is a dictionary, you probably want to use the keys, like this:

>>> a = {
...     2: 'Turtle Doves',
...     3: 'French Hens',
...     4: 'Colly Birds',
...     5: 'Gold Rings',
...     6: 'Geese-a-Laying',
...     7: 'Swans-a-Swimming',
...     8: 'Maids-a-Milking',
...     9: 'Ladies Dancing',
...     0: 'Camel Books',
... }
>>>
>>> xyz = [0, 12, 4, 6, 242, 7, 9]
>>>
>>> known_things = sorted(set(a.iterkeys()).intersection(xyz))
>>> unknown_things = sorted(set(xyz).difference(a.iterkeys()))
>>>
>>> for thing in known_things:
...     print 'I know about', a[thing]
...
I know about Camel Books
I know about Colly Birds
I know about Geese-a-Laying
I know about Swans-a-Swimming
I know about Ladies Dancing
>>> print '...but...'
...but...
>>>
>>> for thing in unknown_things:
...     print "I don't know what happened on the {0}th day of Christmas".format(thing)
...
I don't know what happened on the 12th day of Christmas
I don't know what happened on the 242th day of Christmas

Solution 3

The following is a simplification/one liner from the accepted answer:

a = [2,3,4,5,6,7,8,9,0]
xyz = [0,12,4,6,242,7,9]

for x in (x for x in xyz if x not in a):
    print(x)

12
242

Notice that the generator was kept inline. This was tested on python2.7 and python3.6 (notice the parens in the print ;) )

It is honestly cumbersome even so: the x is mentioned four times.

Solution 4

I personally think this is the prettiest version:

a = [2,3,4,5,6,7,8,9,0]
xyz = [0,12,4,6,242,7,9]
for x in filter(lambda w: w in a, xyz):
  print x

Edit

if you are very keen on avoiding to use lambda you can use partial function application and use the operator module (that provides functions of most operators).

https://docs.python.org/2/library/operator.html#module-operator

from operator import contains
from functools import partial
print(list(filter(partial(contains, a), xyz)))

Solution 5

I would probably use:

for x in xyz: 
    if x not in a:
        print(x...)

View more solutions

Comments

[ChewyChunks]

This is a bit more useful to me. I've never looked at generators. They sound scary (because I saw them in modules that were generally a pain to use).

[ChewyChunks]

Sounds like from the comments below, I should be studying up on generators. I've never used them. Thanks. Is a generator faster than the equivalent combination of FOR and IF statements? I've also used sets, but sometimes redundant elements in a list are information I can't discard.

[ChewyChunks]

Very Zen, @lazyr, but would not help me improve a complex code block that depends on iterating through one list and ignoring matching elements in another list. Is it faster to treat the first list as a set and compare union / difference with a second, growing "ignore" list?

[ChewyChunks]

Agreed. This looks exactly as I think it ought to look, and I understand how it will speed up and clean up my code.

[Kracekumar]

Try this import time a = [2,3,4,5,6,7,8,9,0] xyz = [0,12,4,6,242,7,9] start = time.time() print (set(a) & set(xyz)) print time.time() - start

[johnsyweb]

@ChewyChunks: Generators are not the only way to be Pythonic!

[ChewyChunks]

Sorry another question: if I create a generator with enumerate, will it still work:

[ChewyChunks]

gen = (y for (x,y) in enumerate(xyz) if x not in a) returns >>> 12 when I type for x in gen: print x -- so why the unexpected behavior with enumerate?

[Lauritz V. Thaulow]

@ChewyChunks if either of the lists change during the iteration it will probably be faster to check each element against the ignore list -- except you should make it an ignore set. Checking for membership in sets is very fast: if x in ignore: ....

[johnsyweb]

@ChewyChunks: Please see the last sentence of my answer! enumerate is giving you ((0, 0), (1, 12), (2, 4), (3, 6), (4, 242), (5, 7), (6, 9)), 1 is the only x not in a and 12 is its partner. This is not unexpected behaviour, you're just trying to do too much in one line.

[Mike Graham]

Possible, but not nicer than the original for and if blocks.

[Wooble]

@Johnsyweb, if you're going to quote the Zen of Python: "There should be one-- and preferably only one --obvious way to do it."

[ChewyChunks]

@lazyr I just rewrote my code using an ignore set over an ignore list. Appears to process time much slower. (To be fair I was comparing using if set(a) - set(ignore) == set([]): so perhaps that's why it was much slower than checking membership. I'll test this again in the future on a much simpler example than what I'm writing.

[ChewyChunks]

So @Johnsyweb it will work given gen = (y for (x,y) in enumerate(xyz) if y not in a) ??

[johnsyweb]

@ChewyChunks. That would work but the call to enumerate is redundant.

[johnsyweb]

@Wooble: There should. I quoted that section in my answer to another question around the same time!

[bgusach]

I really miss in python being able to say for x in xyz if x:

[Veky]

@ikaros45 for x in filter(None, xyz): (of course, you can rebind partial(filter, None) if you need it often).

[Veky]

filter(a.__contains__, xyz). Usually when people use lambda, they really need something much simpler.

[bgusach]

@Veky that's fine for basic filtering, but when you do more complex things you have to define lambdas and the beauty is lost.

[Veky]

You mean condition like "if x**2 < 5"? Yes, that might be ugly to write in one line, but what exactly is the problem with two lines? [Comprehensions/genexps have to be inline since they are expressions. When you're writing statements, as you are if you do complicated things in the suite of the loop, it's natural to use lines for them.] If you're afraid of indenting your code too much, you don't have to use 4 spaces every time. for x in xyz:/_if x**2 < 5:/____do something (/ is newline, _ is space).

[Rob]

@bgusach You can always do for x in filter(None, xyz):.

[Veky]

I think you misunderstood something. __contains__ is a method like any other, only it is a special method, meaning it can be called indirectly by an operator (in in this case). But it can also be called directly, it is a part of the public API. Private names are specifically defined as having at most one trailing underscore, to provide exception for special method names - and they are subject to name mangling when lexically in class scopes. See docs.python.org/3/reference/datamodel.html#specialnames and docs.python.org/3.6/tutorial/classes.html#private-variables .

[Veky]

It is certainly ok, but two imports just to be able to refer to a method that's accessible using just an attribute seems weird (operators are usually used when double dispatch is essential, but in is singly dispatched wrt right operand). Besides, note that operator also exports contains method under the name __contains__, so it surely is not a private name. I think you'll just have to learn to live with the fact that not every double underscore means "keep away". :-]

[Diansheng]

how should i turn it into a one-liner when having a break in my loop. for example result=[]; for x in [1,2,3,4,5]: if x>3: result.append(x); break;

[WestCoastProjects]

@bgusach I miss those kinds of statements from other languages: don't know when it were removed from python. I disagree with whatever mindset the python controllers (Guido .. ) have on most of these items - (mostly to keep things simple .. at the expense of capabilities)

[thomas.mc.work]

What about for x in xyz or ():?

[Matti Wens]

for x in (x for x in xyz if x not in a): works for me, but why you shouldn't just be able to do for x in xyz if x not in a:, I'm not sure...

[Qian Chen]

Great answer thanks. But just wondering why Python is making so ugly syntax.

[Dr_Zaszuś]

Is this supposed to work if a is dynamically updated within the cycle?

[WestCoastProjects]

I think your lambda needs fixing to include not : lambda w: not w in a, xyz

[Khanis Rok]

The filter seems more elegant, especially for complex conditions that would become defined functions instead of lambdas, maybe naming the lambda function would add some readability, The generator seems better when the iterated elements are some modification on the list items

[Tadhg McDonald-Jensen]

better solution to lambda or partial(contains, a) just define a one line function above: def isKnown(x): return x in a; This makes the loop read beautifully: for x in filter(isKnown, xyz): pass

[Alexander Oh]

@TadhgMcDonald-Jensen split your code into functions however you like, this is just example code. Bear in mind that it depends on the complexity of the passed function, whether it's worth introducing an extra symbol vs a provided one. This depends on whether you have for instance documentation guarantees that you need, and unit test coverage.

[WestCoastProjects]

@KirillTitov Yes python is a fundamentally non-functional language (this is a purely imperative coding - and I agree with this answer's author that it is the way python is set up to be written. Attempting to use functionals leads to poorly reading or non-pythonic results. I can code functionally in every other language I use (scala, kotlin, javascript, R, swift, ..) but difficult/awkward in python

[James Bedford]

Why is it ugly? To me by starkly declaring a for keyword after a variable it makes it clear early on in reading that this is potentially going to be a reasonably complicated expression for the thing being assigned (or looped through). It's possible to have a few small expressions here for each of x, xyz, a. The first x could even be used to transform each result (e.g. x.lowercase() for x in xyz if x not in a). By separating it out in this way it makes the overall context of creating an iterator clearer.

[WestCoastProjects]

python3 makes this significantly uglier due to need to add list(..) in front of the filter() The lazy evaluation makes all collections processing more cumbersome and difficult to read

[WestCoastProjects]

the python language fails on three counts of the zen of python: and I disagree with the other three (explicit, simple, flat). I'm no newbie: it has been my primary language for 30 months and I did major project[s] using it every year since 2012 . Is this comment off topic? Given the zen was put in relief in the question not necessarily

[acidjunk]

@Wooble Except if you're Dutch