Named tuple and default values for optional keyword arguments

Question

I'm trying to convert a longish hollow "data" class into a named tuple. My class currently looks like this:

class Node(object):
    def __init__(self, val, left=None, right=None):
        self.val = val
        self.left = left
        self.right = right

After conversion to namedtuple it looks like:

from collections import namedtuple
Node = namedtuple('Node', 'val left right')

But there is a problem here. My original class allowed me to pass in just a value and took care of the default by using default values for the named/keyword arguments. Something like:

class BinaryTree(object):
    def __init__(self, val):
        self.root = Node(val)

But this doesn't work in the case of my refactored named tuple since it expects me to pass all the fields. I can of course replace the occurrences of Node(val) to Node(val, None, None) but it isn't to my liking.

So does there exist a good trick which can make my re-write successful without adding a lot of code complexity (metaprogramming) or should I just swallow the pill and go ahead with the "search and replace"? :)

Solution 1

Python 3.7

Use the defaults parameter.

>>> from collections import namedtuple
>>> fields = ('val', 'left', 'right')
>>> Node = namedtuple('Node', fields, defaults=(None,) * len(fields))
>>> Node()
Node(val=None, left=None, right=None)

Or better yet, use the new dataclasses library, which is much nicer than namedtuple.

>>> from dataclasses import dataclass
>>> from typing import Any
>>> @dataclass
... class Node:
...     val: Any = None
...     left: 'Node' = None
...     right: 'Node' = None
>>> Node()
Node(val=None, left=None, right=None)

Before Python 3.7

Set Node.__new__.__defaults__ to the default values.

>>> from collections import namedtuple
>>> Node = namedtuple('Node', 'val left right')
>>> Node.__new__.__defaults__ = (None,) * len(Node._fields)
>>> Node()
Node(val=None, left=None, right=None)

Before Python 2.6

Set Node.__new__.func_defaults to the default values.

>>> from collections import namedtuple
>>> Node = namedtuple('Node', 'val left right')
>>> Node.__new__.func_defaults = (None,) * len(Node._fields)
>>> Node()
Node(val=None, left=None, right=None)

Order

In all versions of Python, if you set fewer default values than exist in the namedtuple, the defaults are applied to the rightmost parameters. This allows you to keep some arguments as required arguments.

>>> Node.__new__.__defaults__ = (1,2)
>>> Node()
Traceback (most recent call last):
  ...
TypeError: __new__() missing 1 required positional argument: 'val'
>>> Node(3)
Node(val=3, left=1, right=2)

Wrapper for Python 2.6 to 3.6

Here's a wrapper for you, which even lets you (optionally) set the default values to something other than None. This does not support required arguments.

import collections
def namedtuple_with_defaults(typename, field_names, default_values=()):
    T = collections.namedtuple(typename, field_names)
    T.__new__.__defaults__ = (None,) * len(T._fields)
    if isinstance(default_values, collections.Mapping):
        prototype = T(**default_values)
    else:
        prototype = T(*default_values)
    T.__new__.__defaults__ = tuple(prototype)
    return T

Example:

>>> Node = namedtuple_with_defaults('Node', 'val left right')
>>> Node()
Node(val=None, left=None, right=None)
>>> Node = namedtuple_with_defaults('Node', 'val left right', [1, 2, 3])
>>> Node()
Node(val=1, left=2, right=3)
>>> Node = namedtuple_with_defaults('Node', 'val left right', {'right':7})
>>> Node()
Node(val=None, left=None, right=7)
>>> Node(4)
Node(val=4, left=None, right=7)

Solution 2

I subclassed namedtuple and overrode the __new__ method:

from collections import namedtuple

class Node(namedtuple('Node', ['value', 'left', 'right'])):
    __slots__ = ()
    def __new__(cls, value, left=None, right=None):
        return super(Node, cls).__new__(cls, value, left, right)

This preserves an intuitive type hierarchy, which the creation of a factory function disguised as a class does not.

Solution 3

Wrap it in a function.

NodeT = namedtuple('Node', 'val left right')

def Node(val, left=None, right=None):
  return NodeT(val, left, right)

Solution 4

With typing.NamedTuple in Python 3.6.1+ you can provide both a default value and a type annotation to a NamedTuple field. Use typing.Any if you only need the former:

from typing import Any, NamedTuple


class Node(NamedTuple):
    val: Any
    left: 'Node' = None
    right: 'Node' = None

Usage:

>>> Node(1)
Node(val=1, left=None, right=None)
>>> n = Node(1)
>>> Node(2, left=n)
Node(val=2, left=Node(val=1, left=None, right=None), right=None)

Also, in case you need both default values and optional mutability, Python 3.7 is going to have data classes (PEP 557) that can in some (many?) cases replace namedtuples.

---

Sidenote: one quirk of the current specification of annotations (expressions after : for parameters and variables and after -> for functions) in Python is that they are evaluated at definition time^*. So, since "class names become defined once the entire body of the class has been executed", the annotations for 'Node' in the class fields above must be strings to avoid NameError.

This kind of type hints is called "forward reference" ([1], [2]), and with PEP 563 Python 3.7+ is going to have a __future__ import (to be enabled by default in 4.0) that will allow to use forward references without quotes, postponing their evaluation.

^{* AFAICT only local variable annotations are not evaluated at runtime. (source: PEP 526)}

Solution 5

This is an example straight from the docs:

Default values can be implemented by using _replace() to customize a prototype instance:

>>> Account = namedtuple('Account', 'owner balance transaction_count')
>>> default_account = Account('<owner name>', 0.0, 0)
>>> johns_account = default_account._replace(owner='John')
>>> janes_account = default_account._replace(owner='Jane')

So, the OP's example would be:

from collections import namedtuple
Node = namedtuple('Node', 'val left right')
default_node = Node(None, None, None)
example = default_node._replace(val="whut")

However, I like some of the other answers given here better. I just wanted to add this for completeness.

View more solutions

Comments

[sasuke]

Ah, not possible to use a third party package though recordtype certainly looks interesting for future work. +1

[jterrace]

The module is quite small and only a single file so you could always just add it to your project.

[sasuke]

Fair enough, though I'll wait for some more time for a pure named tuple solution is there is one out there before marking this accepted! :)

[jterrace]

Agreed pure python would be nice, but I don't think there is one :(

[Gabriel Grant]

This is clever, and can be a good option, but can also cause problems by breaking isinstance(Node('val'), Node): it will now raise an exception, rather than returning True. While a bit more verbose, @justinfay's answer (below) preserves type hierarchy information properly, so is probably a better approach if others are going to interact with Node instances.

[bavaza]

Just to note that recordtype is mutable whereas namedtuple is not. This might matter if you want the object to be hashable (which I guess you don't, since it started out as a class).

[jorgeca]

Beware that Node(1, 2) doesn't work with this recipe, but works in @justinfay's answer. Otherwise, it's quite nifty (+1).

[Pepijn]

This might need slots and fields properties in order to maintain the space efficiency of a named tuple.

[Gerrat]

Let's see...your one-liner: a) is the shortest/simplest answer, b) preserves space efficiency, c) doesn't break isinstance ...all pros, no cons...too bad you were a little late to the party. This is the best answer.

[Michael Scott Asato Cuthbert]

One problem with the wrapper version: unlike the builtin collections.namedtuple, this version is not pickleable/multiprocess serializable if the def() is included in a different module.

[Admin]

For some reason, __new__ is not being called when _replace is used.

[Justin Fay]

Please take a look at @marc-lodato answer below which IMHO is a better solution than this.

[Justin Fay]

I have given this answer an upvote as it is preferable to my own. Its a pity however that my own answer keeps getting upvoted :|

[Robert Siemer]

How can you make only left and right optional? val should not be optional!

[Robert Siemer]

You should have mentioned my name @Robert... :-P

[user3467349]

Is there a way to set defaults in a Class subclassing namedtuple or is @justinfay 's answer the only way ?

[Mark Lodato]

@user3467349, if you're subclassing from namedtuple, use @justinfay's method. You might be able to muck around with __defaults__ but why would you want to?

[user3467349]

Sometimes you have tuples with long argument lists -- your approach is a lot more succinct. It's only a stylistic preference though.

[1'']

Nice answer! Minor quibble: you don't need the parentheses around None, None, None.

[ishaaq]

@MarkLodato I would like to chime in and say I think this is the best solution to the OP's question.However it doesn't seem to work with just the last parameter as optional, i.e. if you set Node.__new__.__defaults__ = (None) then you seem to have to always specify all three args, it seems to only work if you have two or more optional args.

[Mark Lodato]

@ishaaq, the problem is that (None) is not a tuple, it's None. If you use (None,) instead, it should work fine.

[sasuke]

+1. It's very strange that they decided to go with an _ method (which basically means a private one) for something like replace which seems pretty useful..

[Tim Tisdall]

@sasuke - I was wondering that too. It's already a little odd that you define the elements with a space separated string instead of *args. It may just be that it was added to the language before a lot of those things were standardized.

[ankostis]

Excellent! You can generalize the setting-of-defaults with: Node.__new__.__defaults__= (None,) * len(Node._fields)

[Søren Løvborg]

The _ prefix is to avoid colliding with the names of the user defined tuple fields (relevant doc quote: "Any valid Python identifier may be used for a fieldname except for names starting with an underscore."). As for the space separated string, I think that's just to save a few keystrokes (and you can pass a sequence of strings if you prefer).

[Tim Tisdall]

Ah, yeah, I forgot you access the elements of the named tuple as attributes, so the _ makes a lot of sense then.

[ChaimG]

Can we use the one liner version with [] without running into the mutable default gotcha?

[Mark Lodato]

@ankostis - Good idea! I updated the answer accordingly.

[Mark Lodato]

@ChaimG: This code is not affected by the mutable default gotcha, but I changed the default value of default_values to be a tuple to make it more clear that it cannot be changed.

[Pavel Hanpari]

Your solution is simple and the best. The rest is IMHO rather ugly. I would do only one small change. Instead of default_node I would prefer node_default because it makes better experience with IntelliSense. In case you start typing node you received everything you need :)

[user1556435]

I like the brevity of this answer. Perhaps the concern in the comment above can be addressed by naming the function def make_node(...): rather than pretending it is a class definition. In that way users are not tempted to check for type polymorphism on the function but use the tuple definition itself.

[Elliot Cameron]

See my answer for a variation of this that doesn't suffer from misleading people to use isinstance incorrectly.

[Ethan Furman]

dict has no guarantee of ordering.

[Jason S]

but @marc-lodato's answer doesn't provide the ability for a subclass to have different defaults

[gps]

Please never write code that assigns to .__defaults__. That is obscure and unintelligible.

[101]

This seems like the cleanest solution for 3.6.1+ users. Note that this example is (slightly) confusing as the type hint for the fields left and right (i.e. Node) is the same type as the class being defined and therefore must be written as strings.

[monk-time]

@101, thank you, I've added a note about this to the answer.

[Alexey]

@Pepijn, could you please explain what is the problem here with space efficiency?

[Alexey]

@JasonS, i suspect that for a subclass to have different defaults could violate the LSP. However, a subclass may very well want to have more defaults. In any case, it would be for the subclass to use justinfay's method, and the base class would be fine with Marc's method.

[Jason S]

@Alexey -- default constructor values have nothing to do with LSP, they are used to provide specific instance information for a constructor. Let's use the "is-a" test in real life: If I say, buy me a car, the safe default assumption if I don't specify otherwise is that it will have 4 wheels, 4 doors, a trunk, and a gasoline engine. But there are more specific cars or types of cars that have 3 wheels, or 2 doors, or a hatchback, or an electric engine with battery.

[weberc2]

What's the analog for the idiom my_list: List[T] = None self.my_list = my_list if my_list is not None else []? Can we not use default parameters like this?

[monk-time]

@weberc2 Great question! I'm not sure if this workaround for mutable def. values is possible with typing.NamedTuple. But with data classes you can use Field objects with a default_factory attr. for this, replacing your idiom with my_list: List[T] = field(default_factory=list).

[Matthew D. Scholefield]

You can support required arguments via [:len(default_values)] after tuple(prototype).

[K.-Michael Aye]

using this with ([],) or (list(),) instead of (None,) has the problem that all attributes receive the same list, amazingly. :o Doing Node.left.append('item') adds the item to all three lists.

[cglacet]

Using dataclass this way is not strictly better than using namedtuple because the later produces immutable instances. On the other hand you could use @dataclass(frozen=True) together with dataclasses.replace to modify your instances. That also seems a bit nicer than using the ._replace method on a namedtuple.

[nixon]

Thanks for the suggestion of using dataclasses, I find this very useful