How does this lambda/yield/generator comprehension work?

Question

I was looking through my codebase today and found this:

def optionsToArgs(options, separator='='):
    kvs = [
        (
            "%(option)s%(separator)s%(value)s" %  
            {'option' : str(k), 'separator' : separator, 'value' : str(v)}
        ) for k, v in options.items()
    ]
    return list(
        reversed(
            list(
                    (lambda l, t: 
                        (lambda f: 
                            (f((yield x)) for x in l)
                        )(lambda _: t)
                    )(kvs, '-o')
                )
            )
        )

It seems to take a dict of parameters and turn them into a list of parameters for a shell command. It looks like it's using yield inside a generator comprehension, which I thought would be impossible...?

>>> optionsToArgs({"x":1,"y":2,"z":3})
['-o', 'z=3', '-o', 'x=1', '-o', 'y=2']

How does it work?

Solution 1

Since Python 2.5, yield <value> is an expression, not a statement. See PEP 342.

The code is hideously and unnecessarily ugly, but it's legal. Its central trick is using f((yield x)) inside the generator expression. Here's a simpler example of how this works:

>>> def f(val):
...     return "Hi"
>>> x = [1, 2, 3]
>>> list(f((yield a)) for a in x)
[1, 'Hi', 2, 'Hi', 3, 'Hi']

Basically, using yield in the generator expression causes it to produce two values for every value in the source iterable. As the generator expression iterates over the list of strings, on each iteration, the yield x first yields a string from the list. The target expression of the genexp is f((yield x)), so for every value in the list, the "result" of the generator expression is the value of f((yield x)). But f just ignores its argument and always returns the option string "-o". So on every step through the generator, it yields first the key-value string (e.g., "x=1"), then "-o". The outer list(reversed(list(...))) just makes a list out of this generator and then reverses it so that the "-o"s will come before each option instead of after.

However, there is no reason to do it this way. There are a number of much more readable alternatives. Perhaps the most explicit is simply:

kvs = [...] # same list comprehension can be used for this part
result = []
for keyval in kvs:
   result.append("-o")
   result.append(keyval)
return result

Even if you like terse, "clever" code, you could still just do

return sum([["-o", keyval] for keyval in kvs], [])

The kvs list comprehension itself is a bizarre mix of attempted readability and unreadability. It is more simply written:

kvs = [str(optName) + separator + str(optValue) for optName, optValue in options.items()]

You should consider arranging an "intervention" for whoever put this in your codebase.

Solution 2

Oh god. Basically, it boils down to this,:

def f(_):              # I'm the lambda _: t
    return '-o'

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        yield f((yield x))

So when iterated over, thegenerator yields x (a member of kvs) and then the return value of f, which is always -o, all in one iteration over kvs. Whatever yield x returns and what gets passed to f is ignored.

Equivalents:

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        whatever = (yield x)
        yield f(whatever)

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        yield x
        yield f(None)

def thegenerator():   # I'm (f((yield x)) for x in l)
    for x in kvs:
        yield x
        yield '-o'

There are lots of ways to do this much simpler, of course. Even with the original double-yield trick, the entire thing could've been

return list(((lambda _: '-o')((yield x)) for x in kvs))[::-1]

Comments

[BenDundee]

Dang. Talk about unreadable code.

[ch3ka]

the funniest part is the list(reversed(list( part to get the -o switches right, though

[Pavel Anossov]

Also all the lambdas could've been just ((lambda _: '-o')((yield x)) for x in kvs)

[l4mpi]

... and also the return statement could just be expressed as [v for o in kvs for v in ["-o", o]].

[Eric des Courtis]

This reminds me of duffs device. From this day on this will forever be known as Dogs device lol.

[Caltor]

Looks like somebody got Maslow's hammer and kept banging until this came out.

[Admin]

Lol man for that project you're better with Lisp I suppose aha ..

[Dog]

Looking at the history, it used to be: return list(itertools.chain(*[['-o', v] for v in kvs])). It's unclear why it was changed from this.

[Bakuriu]

@Dog The only change I'd do to the code in your comment is to use itertools.chain.from_iterable to avoid using the *(which can become expensive if the list is big)...

[Alexander]

list(f((yield a)) for a in x) works here in Python 2.7, but I am relieved to see that it does not work in Python 3.9 and results in a SyntaxError: 'yield' inside generator expression