Python function overloading

Question

I know that Python does not support method overloading, but I've run into a problem that I can't seem to solve in a nice Pythonic way.

I am making a game where a character needs to shoot a variety of bullets, but how do I write different functions for creating these bullets? For example suppose I have a function that creates a bullet travelling from point A to B with a given speed. I would write a function like this:

def add_bullet(sprite, start, headto, speed):
    # Code ...

But I want to write other functions for creating bullets like:

def add_bullet(sprite, start, direction, speed):
def add_bullet(sprite, start, headto, spead, acceleration):
def add_bullet(sprite, script): # For bullets that are controlled by a script
def add_bullet(sprite, curve, speed): # for bullets with curved paths
# And so on ...

And so on with many variations. Is there a better way to do it without using so many keyword arguments cause its getting kinda ugly fast. Renaming each function is pretty bad too because you get either add_bullet1, add_bullet2, or add_bullet_with_really_long_name.

To address some answers:

1. No I can't create a Bullet class hierarchy because thats too slow. The actual code for managing bullets is in C and my functions are wrappers around C API.

2. I know about the keyword arguments but checking for all sorts of combinations of parameters is getting annoying, but default arguments help allot like acceleration=0

Solution 1

What you are asking for is called multiple dispatch. See Julia language examples which demonstrates different types of dispatches.

However, before looking at that, we'll first tackle why overloading is not really what you want in Python.

Why Not Overloading?

First, one needs to understand the concept of overloading and why it's not applicable to Python.

When working with languages that can discriminate data types at compile-time, selecting among the alternatives can occur at compile-time. The act of creating such alternative functions for compile-time selection is usually referred to as overloading a function. (Wikipedia)

Python is a dynamically typed language, so the concept of overloading simply does not apply to it. However, all is not lost, since we can create such alternative functions at run-time:

In programming languages that defer data type identification until run-time the selection among alternative functions must occur at run-time, based on the dynamically determined types of function arguments. Functions whose alternative implementations are selected in this manner are referred to most generally as multimethods. (Wikipedia)

So we should be able to do multimethods in Python—or, as it is alternatively called: multiple dispatch.

Multiple dispatch

The multimethods are also called multiple dispatch:

Multiple dispatch or multimethods is the feature of some object-oriented programming languages in which a function or method can be dynamically dispatched based on the run time (dynamic) type of more than one of its arguments. (Wikipedia)

Python does not support this out of the box¹, but, as it happens, there is an excellent Python package called multipledispatch that does exactly that.

Solution

Here is how we might use multipledispatch² package to implement your methods:

>>> from multipledispatch import dispatch
>>> from collections import namedtuple
>>> from types import *  # we can test for lambda type, e.g.:
>>> type(lambda a: 1) == LambdaType
True

>>> Sprite = namedtuple('Sprite', ['name'])
>>> Point = namedtuple('Point', ['x', 'y'])
>>> Curve = namedtuple('Curve', ['x', 'y', 'z'])
>>> Vector = namedtuple('Vector', ['x','y','z'])

>>> @dispatch(Sprite, Point, Vector, int)
... def add_bullet(sprite, start, direction, speed):
...     print("Called Version 1")
...
>>> @dispatch(Sprite, Point, Point, int, float)
... def add_bullet(sprite, start, headto, speed, acceleration):
...     print("Called version 2")
...
>>> @dispatch(Sprite, LambdaType)
... def add_bullet(sprite, script):
...     print("Called version 3")
...
>>> @dispatch(Sprite, Curve, int)
... def add_bullet(sprite, curve, speed):
...     print("Called version 4")
...

>>> sprite = Sprite('Turtle')
>>> start = Point(1,2)
>>> direction = Vector(1,1,1)
>>> speed = 100 #km/h
>>> acceleration = 5.0 #m/s**2
>>> script = lambda sprite: sprite.x * 2
>>> curve = Curve(3, 1, 4)
>>> headto = Point(100, 100) # somewhere far away

>>> add_bullet(sprite, start, direction, speed)
Called Version 1

>>> add_bullet(sprite, start, headto, speed, acceleration)
Called version 2

>>> add_bullet(sprite, script)
Called version 3

>>> add_bullet(sprite, curve, speed)
Called version 4

_{1. Python 3 currently supports single dispatch 2. Take care not to use multipledispatch in a multi-threaded environment or you will get weird behavior.}

Solution 2

Python does support "method overloading" as you present it. In fact, what you just describe is trivial to implement in Python, in so many different ways, but I would go with:

class Character(object):
    # your character __init__ and other methods go here

    def add_bullet(self, sprite=default, start=default, 
                 direction=default, speed=default, accel=default, 
                  curve=default):
        # do stuff with your arguments

In the above code, default is a plausible default value for those arguments, or None. You can then call the method with only the arguments you are interested in, and Python will use the default values.

You could also do something like this:

class Character(object):
    # your character __init__ and other methods go here

    def add_bullet(self, **kwargs):
        # here you can unpack kwargs as (key, values) and
        # do stuff with them, and use some global dictionary
        # to provide default values and ensure that ``key``
        # is a valid argument...

        # do stuff with your arguments

Another alternative is to directly hook the desired function directly to the class or instance:

def some_implementation(self, arg1, arg2, arg3):
  # implementation
my_class.add_bullet = some_implementation_of_add_bullet

Yet another way is to use an abstract factory pattern:

class Character(object):
   def __init__(self, bfactory, *args, **kwargs):
       self.bfactory = bfactory
   def add_bullet(self):
       sprite = self.bfactory.sprite()
       speed = self.bfactory.speed()
       # do stuff with your sprite and speed

class pretty_and_fast_factory(object):
    def sprite(self):
       return pretty_sprite
    def speed(self):
       return 10000000000.0

my_character = Character(pretty_and_fast_factory(), a1, a2, kw1=v1, kw2=v2)
my_character.add_bullet() # uses pretty_and_fast_factory

# now, if you have another factory called "ugly_and_slow_factory" 
# you can change it at runtime in python by issuing
my_character.bfactory = ugly_and_slow_factory()

# In the last example you can see abstract factory and "method
# overloading" (as you call it) in action 

Solution 3

You can use "roll-your-own" solution for function overloading. This one is copied from Guido van Rossum's article about multimethods (because there is little difference between multimethods and overloading in Python):

registry = {}

class MultiMethod(object):
    def __init__(self, name):
        self.name = name
        self.typemap = {}
    def __call__(self, *args):
        types = tuple(arg.__class__ for arg in args) # a generator expression!
        function = self.typemap.get(types)
        if function is None:
            raise TypeError("no match")
        return function(*args)
    def register(self, types, function):
        if types in self.typemap:
            raise TypeError("duplicate registration")
        self.typemap[types] = function


def multimethod(*types):
    def register(function):
        name = function.__name__
        mm = registry.get(name)
        if mm is None:
            mm = registry[name] = MultiMethod(name)
        mm.register(types, function)
        return mm
    return register

The usage would be

from multimethods import multimethod
import unittest

# 'overload' makes more sense in this case
overload = multimethod

class Sprite(object):
    pass

class Point(object):
    pass

class Curve(object):
    pass

@overload(Sprite, Point, Direction, int)
def add_bullet(sprite, start, direction, speed):
    # ...

@overload(Sprite, Point, Point, int, int)
def add_bullet(sprite, start, headto, speed, acceleration):
    # ...

@overload(Sprite, str)
def add_bullet(sprite, script):
    # ...

@overload(Sprite, Curve, speed)
def add_bullet(sprite, curve, speed):
    # ...

Most restrictive limitations at the moment are:

• methods are not supported, only functions that are not class members;
• inheritance is not handled;
• kwargs are not supported;
• registering new functions should be done at import time thing is not thread-safe

Solution 4

A possible option is to use the multipledispatch module as detailed here: http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch

Instead of doing this:

def add(self, other):
    if isinstance(other, Foo):
        ...
    elif isinstance(other, Bar):
        ...
    else:
        raise NotImplementedError()

You can do this:

from multipledispatch import dispatch
@dispatch(int, int)
def add(x, y):
    return x + y    

@dispatch(object, object)
def add(x, y):
    return "%s + %s" % (x, y)

With the resulting usage:

>>> add(1, 2)
3

>>> add(1, 'hello')
'1 + hello'

Solution 5

In Python 3.4 PEP-0443. Single-dispatch generic functions was added.

Here is a short API description from PEP.

To define a generic function, decorate it with the @singledispatch decorator. Note that the dispatch happens on the type of the first argument. Create your function accordingly:

from functools import singledispatch
@singledispatch
def fun(arg, verbose=False):
    if verbose:
        print("Let me just say,", end=" ")
    print(arg)

To add overloaded implementations to the function, use the register() attribute of the generic function. This is a decorator, taking a type parameter and decorating a function implementing the operation for that type:

@fun.register(int)
def _(arg, verbose=False):
    if verbose:
        print("Strength in numbers, eh?", end=" ")
    print(arg)

@fun.register(list)
def _(arg, verbose=False):
    if verbose:
        print("Enumerate this:")
    for i, elem in enumerate(arg):
        print(i, elem)

View more solutions

Comments

[leewz]

Works for only one parameter, but here (for people coming here from a search engine): docs.python.org/3/library/…

[Andrew Scott Evans]

this seems like a good place for default values. you can set some to None and just check for them. the extra boolean impact seems negligable

[Deqing]

Have to use default value + if + else to do the same as C++ do. This is one of the very few things that C++ has better readability than Python...

[Calculus]

I'm confused on why kwargs is not a valid answer. You say that you don't want to use many keyword arguments because it gets ugly fast... well that's just the nature of the problem. If you have many arguments and it's messy because you have many arguments than what did you expect? Do you want to use many arguments without specifying them anywhere??? Python is not a mind reader.

[smci]

We don't know what sort of objects script, curve are, do they have a common ancestor, what methods they support. With duck-typing, it's up to you for class design to figure out what methods they need to support. Presumably Script supports some sort of timestep-based callback (but what object should it return? the position at that timestep? the trajectory at that timestep?). Presumably start, direction, speed and start, headto, spead, acceleration both describe types of trajectories, but again it's up to you to design the receiving class to know how to unpack them and process them.

[Peter Mortensen]

Related (not duplicate): How can I detect duplicate method names in a Python class?

[John Zwinck]

I was going to suggest the second approach: make some BulletParams... classes to specify the bullet details.

[Bullets]

Can you elaborate on this? I tried to create a class hierarchy with different bullets but this does not work, because Python is too slow. It can't calculate the motions of the required number of bullets fast enough, so I had to write that part in C. All the add_bullet variants just call the corresponding C function.

[Bullets]

That was my initial approach, but for performance reasons I had to rewrite that code in C.

[Josh Smeaton]

@Bullets, I would suggest that there may be a number of different options available to improve performance rather than writing a whole lot of c functions that probably won't be doing a whole lot. For example: creating an instance may be expensive, so maintain an object pool. Though I say this without knowing what you found to be too slow. Out of interest, what exactly was slow about this approach? Unless significant time is going to be spent in the C side of the boundary, I can't think that Python (itself) is the real problem.

[Bullets]

Maybe there are other ways to improve the performance, but I am much better with C than with Python. The problem was calculating the motions of the bullets and detecting when they go out of screen bounds. I had a methods for calculating position of the bullet pos+v*t and then comparing to screen boundaries if x > 800 and so on. Calling these functions several hundred times per frame turned out to be unacceptably slow. It was something like 40 fps at 100% cpu with pure python to 60 fps with 5%-10% when done in C.

[Josh Smeaton]

@Bullets, fair enough then. I'd still use the approach I went with for encapsulating data. Pass an instance of bullet to add_bullet, and extract all the fields that you need. I'll edit my answer.

[Bullets]

Hmm this is still just a fancy way to name the functions as add_bullet1, add_bullet2 and so on.

[martineau]

@Bullets: Perhaps it is, or maybe it's just a slightly elaborate way to create a factory function. A nice thing about it is that it supports a hierarchy of Bullet subclasses without having to modify the base class or factory function every time you add another subtype. (Of course, if you're using C rather than C++, I guess you don't have classes.) You could also make a smarter metaclass that figured-out on its own what subclass to create based on the type and/or number of arguments passed (like C++ does to support overloading).

[jfs]

@Bullets: You can combine your C functions and the OOP approach suggested by Josh using Cython. It allows early binding so there should not be a speed penalty.

[Eloims]

+1 for decorators for extending the language in this use case.

[Efren]

All these look as examples of variable arguments, rather than overloading. Since overloading allows you to have the same function, for different types as arguments. eg: sum(real_num1, real_num2) and sum(imaginary_num1, imaginary_num2) Will both have same calling syntax, but are actually expecting 2 different types as input, and the implementation has to change also internally

[Escualo]

+1 because this is a great idea (and probably what the OP should go with) --- I had never seen a multimethod implementation in Python.

[Greg Ennis]

Using the answer you would go with, how would you present to the caller which arguments make sense together? Just putting a bunch of arguments each with a default value may provide the same functionality but in terms of an API it is much less elegant

[Andriy Drozdyuk]

Why doesn't this get more votes? I'm guessing due to lack of examples... I've created an answer with an example of how to implement a solution to OP's problem with multipledispatch package.

[danzeer]

What is the problem with 'multipledispatch' in a multi-threaded environment ? Since server-side's code is usually in multi-threaded environment! Just trying to dig it out!

[Andriy Drozdyuk]

@danzeer It was not thread-safe. I saw the argument being modified by two different threads (i.e. the value of speed might change in the middle of the function when another thread sets it's own value of speed)!!! It took me long time to realize that it was the library that was the culprit.

[Daniel Möller]

This inheritance idea would be my first option as well.

[Roee Gavirel]

Non of the above is overloading, the implementation will have to check all combinations of parameters inputs (or ignore parameters) like: if sprite and script and not start and not direction and not speed... just to know it is in a specific action. because a caller can call the function providing all the parameters available. While overloading define for you the exact sets of relevant parameters.

[Howard Swope]

It is very upsetting when people say that python supports method overloading. It does not. The fact that you put "method overloading" in quotations indicates that you aware of this fact. You can get similar functionality with several techniques, like the one mentioned here. But method overloading has a very specific definition.

[rawr rang]

I think the intended point is while method overloading is not a feature of python, the above mechanisms can be used to achieve the equivalent effect.

[smci]

This example crucially doesn't illustrate which method got called on x for dynamic dispatch, nor in which order both methods got called in for static dispatch. Recommend you edit the print statements to print('number called for Integer') etc.

[gerrit]

Can you add some examples?

[Andrew]

I like this response. It could be used also to detect types and then make overloads based on arg count and types

[mtraceur]

+1, but an example which uses single dispatch to implement the question's example use case (in other words, how to implement multiple dispatch on top of single dispatch) would make this answer a lot better. If someone doesn't think of how to solve multiple dispatch with single dispatch, this answer might feel irrelevant or useless to people looking at problems like the one in the question.

[Admin]

Biggest downside is the parser cannot "see" or hint at the parameter names nor the types the method accepts anymore. Making it necessary to use a docstring, or else someone just using your code is having to read it once.

[Dr_Zaszuś]

The advantage of multipledispatch as compared to single_dispatch is that it also works with class methods in python<3.8.

[Peter Mortensen]

Re "passing keyword args": Don't you mean "passing keyword kwargs"?

[Make42]

How does this compare to the multipledispatch library?

[Cas]

I agree an example would be nice here as this is nice syntactic sugar rather than separating logic in separate functions. This are better details of the workings on mypy page: mypy.readthedocs.io/en/stable/…

[Thijs Kramer]

@PeterMortensen I don't think so since kwarg is short for keyword argument.

[SKSKSKSK]

how to dispatch the class method with self as argument? what's the type of it? thanks

[Iqigai]

Isn't the + overloaded in python: we can write a = 1 + 2 and my_string = 'abc' + 'def' as far as I know. So saying that overloading does not exist in python is not accurate.

[IntegrateThis]

Is there not scenarios where overloading in other languages also entails a different number of arguments for each function definition, and not simply a different input type?

[Andriy Drozdyuk]

@Iqigai The + is not overloaded. It is simply sugar for __add__(self, other), which is a method defined on specific class. If the class does not define this method then you get an error. For example {} + {} gives TypeError: unsupported operand type(s) for +: 'dict' and 'dict'.

[darda]

What about typing.overload?

[Andriy Drozdyuk]

That's just for type checker. You have to check for each specific type inside the final non-decorated function. Those functions have ... as their body.

[Ryan McGrath]

Need to add that you're referencing a overloading pip package, and you need to import the decorators first