Python: Why is functools.partial necessary?

Question

Partial application is cool. What functionality does functools.partial offer that you can't get through lambdas?

>>> sum = lambda x, y : x + y
>>> sum(1, 2)
3
>>> incr = lambda y : sum(1, y)
>>> incr(2)
3
>>> def sum2(x, y):
    return x + y

>>> incr2 = functools.partial(sum2, 1)
>>> incr2(4)
5

Is functools somehow more efficient, or readable?

Solution 1

What functionality does functools.partial offer that you can't get through lambdas?

Not much in terms of extra functionality (but, see later) – and, readability is in the eye of the beholder.
Most people who are familiar with functional programming languages (those in the Lisp/Scheme families in particular) appear to like lambda just fine – I say "most", definitely not all, because Guido and I assuredly are among those "familiar with" (etc) yet think of lambda as an eyesore anomaly in Python...
He was repentant of ever having accepted it into Python whereas planned to remove it from Python 3, as one of "Python's glitches".
I fully supported him in that. (I love lambda in Scheme... while its limitations in Python, and the weird way it just doesn't fit in with the rest of the language, make my skin crawl).

Not so, however, for the hordes of lambda lovers -- who staged one of the closest things to a rebellion ever seen in Python's history, until Guido backtracked and decided to leave lambda in.
Several possible additions to functools (to make functions returning constants, identity, etc) didn't happen (to avoid explicitly duplicating more of lambda's functionality), though partial did of course remain (it's no total duplication, nor is it an eyesore).

Remember that lambda's body is limited to be an expression, so it's got limitations. For example...:

>>> import functools
>>> f = functools.partial(int, base=2)
>>> f.args
()
>>> f.func
<type 'int'>
>>> f.keywords
{'base': 2}
>>> 

functools.partial's returned function is decorated with attributes useful for introspection -- the function it's wrapping, and what positional and named arguments it fixes therein. Further, the named arguments can be overridden right back (the "fixing" is rather, in a sense, the setting of defaults):

>>> f('23', base=10)
23

So, as you see, it's definely not as simplistic as lambda s: int(s, base=2)!-)

Yes, you could contort your lambda to give you some of this – e.g., for the keyword-overriding,

>>> f = lambda s, **k: int(s, **dict({'base': 2}, **k))

but I dearly hope that even the most ardent lambda-lover doesn't consider this horror more readable than the partial call!-). The "attribute setting" part is even harder, because of the "body's a single expression" limitation of Python's lambda (plus the fact that assignment can never be part of a Python expression)... you end up "faking assignments within an expression" by stretching list comprehension well beyond its design limits...:

>>> f = [f for f in (lambda f: int(s, base=2),)
           if setattr(f, 'keywords', {'base': 2}) is None][0]

Now combine the named-arguments overridability, plus the setting of three attributes, into a single expression, and tell me just how readable that is going to be...!

Solution 2

Well, here's an example that shows a difference:

In [132]: sum = lambda x, y: x + y

In [133]: n = 5

In [134]: incr = lambda y: sum(n, y)

In [135]: incr2 = partial(sum, n)

In [136]: print incr(3), incr2(3)
8 8

In [137]: n = 9

In [138]: print incr(3), incr2(3)
12 8

These posts by Ivan Moore expand on the "limitations of lambda" and closures in python:

• Closures in Python (part 2)
• Closures in Python (part 3)

Solution 3

In the latest versions of Python (>=2.7), you can pickle a partial, but not a lambda:

>>> pickle.dumps(partial(int))
'cfunctools\npartial\np0\n(c__builtin__\nint\np1\ntp2\nRp3\n(g1\n(tNNtp4\nb.'
>>> pickle.dumps(lambda x: int(x))
Traceback (most recent call last):
  File "<ipython-input-11-e32d5a050739>", line 1, in <module>
    pickle.dumps(lambda x: int(x))
  File "/usr/lib/python2.7/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
    self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <function <lambda> at 0x1729aa0>: it's not found as __main__.<lambda>

Solution 4

Is functools somehow more efficient..?

As a partly answer to this I decided to test the performance. Here is my example:

from functools import partial
import time, math

def make_lambda():
    x = 1.3
    return lambda: math.sin(x)

def make_partial():
    x = 1.3
    return partial(math.sin, x)

Iter = 10**7

start = time.clock()
for i in range(0, Iter):
    l = make_lambda()
stop = time.clock()
print('lambda creation time {}'.format(stop - start))

start = time.clock()
for i in range(0, Iter):
    l()
stop = time.clock()
print('lambda execution time {}'.format(stop - start))

start = time.clock()
for i in range(0, Iter):
    p = make_partial()
stop = time.clock()
print('partial creation time {}'.format(stop - start))

start = time.clock()
for i in range(0, Iter):
    p()
stop = time.clock()
print('partial execution time {}'.format(stop - start))

on Python 3.3 it gives:

lambda creation time 3.1743163756961392
lambda execution time 3.040552701787919
partial creation time 3.514482823352731
partial execution time 1.7113973411608114

Which means that partial needs a bit more time for creation but considerably less time for execution. This can well be the effect of the early and late binding which are discussed in the answer from ars.

Solution 5

Besides the extra functionality Alex mentioned, another advantage of functools.partial is speed. With partial you can avoid constructing (and destructing) another stack frame.

Neither the function generated by partial nor lambdas have docstrings by default (though you can set the doc string for any objects via __doc__).

You can find more details in this blog: Partial Function Application in Python

View more solutions

Comments

[David Z]

Hm, I'm actually of the opposite persuasion, I took a lot longer to parse the functools.partial call, whereas the lambdas are self-evident.

[sidgeon smythe]

Good example. To me, this seems more a "bug" with lambda, actually, but I understand others may disagree. (Something similar happens with closures defined within a loop, as implemented in several programming languages.)

[Alex Martelli]

The fix to this "early vs late binding dilemma" is to explicitly use early binding, when you want that, by lambda y, n=n: .... Late binding (of names appearing only in a function's body, not in its def or equivalent lambda) is anything but a bug, as I've shown at length in long SO answers in the past: you early-bind explicitly when that's what you want, use the late-binding default when that is what you want, and that's exactly the right design choice given the context of the rest of Python's design.

[Nick Heiner]

Yeah, I'd say that the extra functionality of functools.partial that you mentioned makes it superior to lambda. Perhaps this is the topic of another post, but what is it on a design level that bothers you so much about lambda?

[sidgeon smythe]

@Alex Martelli: Yeah, sorry. I just fail to get used to late binding properly, perhaps because I think when defining functions that I'm actually defining something for good, and the unexpected surprises only cause me headaches. (More when I try to do functional things in Javascript than in Python, though.) I understand that many people are comfortable with late binding, and that it's consistent with the rest of Python's design. I would still like to read your other long SO answers, though -- links? :-)

[ars]

Alex is right, it's not a bug. But it's a "gotcha" that traps many lambda enthusiasts. For the "bug" side of the argument from a haskel/functional type, see Andrej Bauer's post: math.andrej.com/2009/04/09/pythons-lambda-is-broken

[sidgeon smythe]

@ars: Ah yes, thanks for the link to Andrej Bauer's post. Yeah, the effects of late binding are certainly something that we mathematics-types (worse, with a Haskell background) keep finding grossly unexpected and shocking. :-) I'm not sure I'd go as far as Prof. Bauer and call it a design error, but it is hard for human programmers to completely switch between one way of thinking and another. (Or perhaps this is just my insufficient Python experience.)

[sidgeon smythe]

BTW, this issue (here and in Andrej Bauer's post) is a good reason to remove lambda from Python: it doesn't do the sensible thing that many would expect. ;-)

[Alex Martelli]

@Rosarch, as I said: first, it limitations (Python sharply distinguishes expressions and statements -- there's much you can't do, or can't do sensibly, within a single expression, and that's what a lambda's body is); second, its absolutely weird syntax sugar. If I could go back in time and change one thing within Python, it would be the absurd, meaningless, eyesore def and lambda keywords: make them both function (one name choice Javascript got really right), and at least 1/3 of my objections would vanish!-). As I said, I have no objection to lambda in Lisp...!-)

[Alex Martelli]

@ShreevatsaR, too late to remove anything -- Python 3 is way finalized (wait another 20 years for Python 4;-). Re some of my answers above mentioned, "let me google it for you": google.com/… (I don't know how to do the equivalent search on SO directly, but this one has at least some relevant ones, it seems).

[Peter Long]

@Alex Martelli, Why did Guido set such a limitation for lambda: "body's a single expression" ? C#'s lambda body could be anything valid in a function's body. Why don't Guido just remove the limitation for python lambda?

[new123456]

@PeterLong Hopefully Guido can answer your question. The gist of it is that it would be too complex, and that you can use a def anyway. Our benevolent leader has spoken!

[David]

@AlexMartelli DropBox has had an interesting influence on Guido - twitter.com/gvanrossum/status/391769557758521345

[wting]

Unfortunately partial functions fail to pickle for multiprocessing.Pool.map(). stackoverflow.com/a/3637905/195139

[Fred Foo]

@wting That post is from 2010. partial is pickleable in Python 2.7.

[Trilarion]

If you have tested the speed advantage, what speed improvement of partial over lambda can be expected?

[chepner]

More importantly, partial is written in C, rather than pure Python, meaning it can produce a more efficient callable than simply creating a function that calls another function.

[jan]

When you say that the docstring is inherited, which Python version do you refer to? In Python 2.7.15 and Python 3.7.2 they are not inherited. Which is a good thing, because the original docstring is not necessarily correct for the function with partially applied arguments.

[Yaroslav Nikitenko]

For python 2.7 (docs.python.org/2/library/functools.html#partial-objects): "the name and doc attributes are not created automatically". Same for 3.[5-7].

[Yaroslav Nikitenko]

There is a mistake in your link: log_info = partial(log_template, level="info") - it's not possible because level is not a keyword argument in the example. Both python 2 and 3 say: "TypeError: log_template() got multiple values for argument 'level'".

[Yaroslav Nikitenko]

In fact, I created a partial(f) by hand and it gives doc field as 'partial(func, *args, **keywords) - new function with partial application\n of the given arguments and keywords.\n' (both for python 2 and 3).

[user]

Can you explain with words what lambdas are doing "wrong" in the question example? Just looking the code is not easy to get how it is happening and going wrong. After looking, it seems that lambdas do not "copy" the parameters values, and they use it from the current context, while partial does "copy" the values on the moment of creation. Meaming lambdas have a dynamic context, while partial has a fixed one for the function parameter's variables used.

[Elijas Dapšauskas]

You can pickle lambdas with dill

[Stéphane Gourichon]

Beware, the linked articles "Closures in Python" were written in 2004, things have probably evolved a lot since then.

[RCCG]

Note that partials still have an overhead compared to handing the arguments over explicitly. See stackoverflow.com/questions/17388438/…

[mtraceur]

Readability is partially in the eye of the beholder. It has objective components - like the number of eye saccades it takes to see all the details, how much visual detail you have to resolve before you can confidently know what you are looking at, and so on. Even the eye-of-the-beholder stuff is often relativistic but still varies along predictable objective principles (visual acuity, distance to screen, how much code fits on screen, the brain getting habitualized to expect and parse specific patterns visual acuity, etc), not just randomly different as arbitrary matters of taste.

[mtraceur]

+1 but convoluted lambda example detracts from the answer: lambdas can have default arguments too: f = lambda s, base=2: int(s, base=base).