How does functools partial do what it does?
Solution 1
Roughly, partial
does something like this (apart from keyword args support etc):
def partial(func, *part_args):
def wrapper(*extra_args):
args = list(part_args)
args.extend(extra_args)
return func(*args)
return wrapper
So, by calling partial(sum2, 4)
you create a new function (a callable, to be precise) that behaves like sum2
, but has one positional argument less. That missing argument is always substituted by 4
, so that partial(sum2, 4)(2) == sum2(4, 2)
As for why it's needed, there's a variety of cases. Just for one, suppose you have to pass a function somewhere where it's expected to have 2 arguments:
class EventNotifier(object):
def __init__(self):
self._listeners = []
def add_listener(self, callback):
''' callback should accept two positional arguments, event and params '''
self._listeners.append(callback)
# ...
def notify(self, event, *params):
for f in self._listeners:
f(event, params)
But a function you already have needs access to some third context
object to do its job:
def log_event(context, event, params):
context.log_event("Something happened %s, %s", event, params)
So, there are several solutions:
A custom object:
class Listener(object):
def __init__(self, context):
self._context = context
def __call__(self, event, params):
self._context.log_event("Something happened %s, %s", event, params)
notifier.add_listener(Listener(context))
Lambda:
log_listener = lambda event, params: log_event(context, event, params)
notifier.add_listener(log_listener)
With partials:
context = get_context() # whatever
notifier.add_listener(partial(log_event, context))
Of those three, partial
is the shortest and the fastest.
(For a more complex case you might want a custom object though).
Solution 2
partials are incredibly useful.
For instance, in a 'pipe-lined' sequence of function calls (in which the returned value from one function is the argument passed to the next).
Sometimes a function in such a pipeline requires a single argument, but the function immediately upstream from it returns two values.
In this scenario, functools.partial
might allow you to keep this function pipeline intact.
Here's a specific, isolated example: suppose you want to sort some data by each data point's distance from some target:
# create some data
import random as RND
fnx = lambda: RND.randint(0, 10)
data = [ (fnx(), fnx()) for c in range(10) ]
target = (2, 4)
import math
def euclid_dist(v1, v2):
x1, y1 = v1
x2, y2 = v2
return math.sqrt((x2 - x1)**2 + (y2 - y1)**2)
To sort this data by distance from the target, what you would like to do of course is this:
data.sort(key=euclid_dist)
but you can't--the sort method's key parameter only accepts functions that take a single argument.
so re-write euclid_dist
as a function taking a single parameter:
from functools import partial
p_euclid_dist = partial(euclid_dist, target)
p_euclid_dist
now accepts a single argument,
>>> p_euclid_dist((3, 3))
1.4142135623730951
so now you can sort your data by passing in the partial function for the sort method's key argument:
data.sort(key=p_euclid_dist)
# verify that it works:
for p in data:
print(round(p_euclid_dist(p), 3))
1.0
2.236
2.236
3.606
4.243
5.0
5.831
6.325
7.071
8.602
Or for instance, one of the function's arguments changes in an outer loop but is fixed during iteration in the inner loop. By using a partial, you don't have to pass in the additional parameter during iteration of the inner loop, because the modified (partial) function doesn't require it.
>>> from functools import partial
>>> def fnx(a, b, c):
return a + b + c
>>> fnx(3, 4, 5)
12
create a partial function (using keyword arg)
>>> pfnx = partial(fnx, a=12)
>>> pfnx(b=4, c=5)
21
you can also create a partial function with a positional argument
>>> pfnx = partial(fnx, 12)
>>> pfnx(4, 5)
21
but this will throw (e.g., creating partial with keyword argument then calling using positional arguments)
>>> pfnx = partial(fnx, a=12)
>>> pfnx(4, 5)
Traceback (most recent call last):
File "<pyshell#80>", line 1, in <module>
pfnx(4, 5)
TypeError: fnx() got multiple values for keyword argument 'a'
another use case: writing distributed code using python's multiprocessing
library. A pool of processes is created using the Pool method:
>>> import multiprocessing as MP
>>> # create a process pool:
>>> ppool = MP.Pool()
Pool
has a map method, but it only takes a single iterable, so if you need to pass in a function with a longer parameter list, re-define the function as a partial, to fix all but one:
>>> ppool.map(pfnx, [4, 6, 7, 8])
Solution 3
short answer, partial
gives default values to the parameters of a function that would otherwise not have default values.
from functools import partial
def foo(a,b):
return a+b
bar = partial(foo, a=1) # equivalent to: foo(a=1, b)
bar(b=10)
#11 = 1+10
bar(a=101, b=10)
#111=101+10
Solution 4
Partials can be used to make new derived functions that have some input parameters pre-assigned
To see some real world usage of partials, refer to this really good blog post here
A simple but neat beginner's example from the blog, covers how one might use partial
on re.search
to make code more readable. re.search
method's signature is:
search(pattern, string, flags=0)
By applying partial
we can create multiple versions of the regular expression search
to suit our requirements, so for example:
is_spaced_apart = partial(re.search, '[a-zA-Z]\s\=')
is_grouped_together = partial(re.search, '[a-zA-Z]\=')
Now is_spaced_apart
and is_grouped_together
are two new functions derived from re.search
that have the pattern
argument applied(since pattern
is the first argument in the re.search
method's signature).
The signature of these two new functions(callable) is:
is_spaced_apart(string, flags=0) # pattern '[a-zA-Z]\s\=' applied
is_grouped_together(string, flags=0) # pattern '[a-zA-Z]\=' applied
This is how you could then use these partial functions on some text:
for text in lines:
if is_grouped_together(text):
some_action(text)
elif is_spaced_apart(text):
some_other_action(text)
else:
some_default_action()
You can refer the link above to get a more in depth understanding of the subject, as it covers this specific example and much more..
Solution 5
In my opinion, it's a way to implement currying in python.
from functools import partial
def add(a,b):
return a + b
def add2number(x,y,z):
return x + y + z
if __name__ == "__main__":
add2 = partial(add,2)
print("result of add2 ",add2(1))
add3 = partial(partial(add2number,1),2)
print("result of add3",add3(1))
The result is 3 and 4.
Related videos on Youtube
user1865341
Updated on June 21, 2021Comments
-
user1865341 almost 3 years
I am not able to get my head on how the partial works in functools. I have the following code from here:
>>> sum = lambda x, y : x + y >>> sum(1, 2) 3 >>> incr = lambda y : sum(1, y) >>> incr(2) 3 >>> def sum2(x, y): return x + y >>> incr2 = functools.partial(sum2, 1) >>> incr2(4) 5
Now in the line
incr = lambda y : sum(1, y)
I get that whatever argument I pass to
incr
it will be passed asy
tolambda
which will returnsum(1, y)
i.e1 + y
.I understand that. But I didn't understand this
incr2(4)
.How does the
4
gets passed asx
in partial function? To me,4
should replace thesum2
. What is the relation betweenx
and4
?-
Ricardo over 2 yearsSimple answer: it doesn't! It's actually passed as
y
. Try adding the lineprint(f'x: {x}, y: {y}')
to the top ofsum2
and you'll see, Check @MSK's answer below.
-
-
user1865341 about 11 yearsfrom where did u get the
extra_args
variable -
bereal about 11 years
extra_args
is something that passed by the partial caller, in the example withp = partial(func, 1); f(2, 3, 4)
it is(2, 3, 4)
. -
user1865341 about 11 yearsbut why we would do that , any special use case where something has to be done by partial only and can't be done with other thing
-
bereal about 11 years@user1865341 I added an example to the answer.
-
user1865341 about 11 yearsis there any practical use of this function somewher
-
user1865341 about 11 yearswith your example , what is the relation between
callback
andmy_callback
-
bereal about 11 yearsAfter
my_callback
is wrapped with partial it will mimiccallback
's API and you may callmy_callback
wherever could callcallback
. -
user1865341 about 11 yearsSorry i am getting little bit . iam 70% clear but still not 100% clear. can you give some more code example for current scenario like u did before. . i get to the point that
partial(my_callback, context)(event, params) == my_callback(context, event, params)
but i am not able to get , how can we use that wherecallback is reuired
-
Deepak Mathpal about 11 years@user1865341 added two exemplarly use cases to my answer
-
bereal about 11 yearsWell callback is not a function to be replaced, it's more of an API description. I've updated the example code to reflect some semi-real-life situation.
-
user1865341 about 11 yearssorry for the trouble . can you tell me how the solution would look like for the other two options like
object creating
and lambda. i am not able to comprehend yet. i am new to python -
bereal about 11 years@user1865341 I updated the answer with object and lambda solutions.
-
user1865341 about 11 yearswow , thanks buddy , now i undestood your complete solution. thanks for your patience
-
Can Lu over 10 yearsIn your example: >>> incr2 = functools.partial(sum2, 1) >>> incr2(4) Am i right in thinking, the 4 is extra_args, and 1 is part_args, so effectively the incr2(4) == sum2(1, 4)
-
bereal over 10 years@CanLu yes, or, more general,
partial(f, x)(y) == f(x, y)
for allf
,x
,y
. -
Brent over 5 yearsDoes this perform any constant folding to achieve performance gains, like currying in functional languages?
-
bereal over 5 years@Brent I don't think so, that would require static analysis, while in Python types are unknows until the execution.
-
Azat Ibrakov over 5 yearsthis is half true because we can override default values, we can even override overriden parameters by subsequent
partial
and so on -
user1071847 about 5 yearsGood answer, though IMHO it's important to note the role of closures, as in "Closures, Partials and Decorators".
-
DylanYoung over 4 yearsHuh? You do the leftmost parameter like this:
prt=partial(func, 7)
-
akhan about 4 yearsIMHO, this is a better answer as it keeps out unrelated concepts like objects and classes and focuses on functions which is what this is all about.
-
Aristide about 4 yearsIsn't this equivalent to
is_spaced_apart = re.compile('[a-zA-Z]\s\=').search
? If so, is there a guarantee that thepartial
idiom compiles the regular expression for faster reuse? -
Andrew Magerman over 2 yearsHmm, not exactly. Currying is dividing a function with n parameters into n successive functions with one parameter. Partial application is 'pre-filling' a function with some parameters, then returning a function with a smaller number of parameters