Override the {...} notation so i get an OrderedDict() instead of a dict()?

Question

Update: dicts retaining insertion order is guaranteed for Python 3.7+

I want to use a .py file like a config file. So using the {...} notation I can create a dictionary using strings as keys but the definition order is lost in a standard python dictionary.

My question: is it possible to override the {...} notation so that I get an OrderedDict() instead of a dict()?

I was hoping that simply overriding dict constructor with OrderedDict (dict = OrderedDict) would work, but it doesn't.

Eg:

dict = OrderedDict
dictname = {
   'B key': 'value1',
   'A key': 'value2',
   'C key': 'value3'
   }

print dictname.items()

Output:

[('B key', 'value1'), ('A key', 'value2'), ('C key', 'value3')]

Solution 1

Here's a hack that almost gives you the syntax you want:

class _OrderedDictMaker(object):
    def __getitem__(self, keys):
        if not isinstance(keys, tuple):
            keys = (keys,)
        assert all(isinstance(key, slice) for key in keys)

        return OrderedDict([(k.start, k.stop) for k in keys])

ordereddict = _OrderedDictMaker()

from nastyhacks import ordereddict

menu = ordereddict[
   "about" : "about",
   "login" : "login",
   'signup': "signup"
]

Edit: Someone else discovered this independently, and has published the odictliteral package on PyPI that provides a slightly more thorough implementation - use that package instead

Solution 2

To literally get what you are asking for, you have to fiddle with the syntax tree of your file. I don't think it is advisable to do so, but I couldn't resist the temptation to try. So here we go.

First, we create a module with a function my_execfile() that works like the built-in execfile(), except that all occurrences of dictionary displays, e.g. {3: 4, "a": 2} are replaced by explicit calls to the dict() constructor, e.g. dict([(3, 4), ('a', 2)]). (Of course we could directly replace them by calls to collections.OrderedDict(), but we don't want to be too intrusive.) Here's the code:

import ast

class DictDisplayTransformer(ast.NodeTransformer):
    def visit_Dict(self, node):
        self.generic_visit(node)
        list_node = ast.List(
            [ast.copy_location(ast.Tuple(list(x), ast.Load()), x[0])
             for x in zip(node.keys, node.values)],
            ast.Load())
        name_node = ast.Name("dict", ast.Load())
        new_node = ast.Call(ast.copy_location(name_node, node),
                            [ast.copy_location(list_node, node)],
                            [], None, None)
        return ast.copy_location(new_node, node)

def my_execfile(filename, globals=None, locals=None):
    if globals is None:
        globals = {}
    if locals is None:
        locals = globals
    node = ast.parse(open(filename).read())
    transformed = DictDisplayTransformer().visit(node)
    exec compile(transformed, filename, "exec") in globals, locals

With this modification in place, we can modify the behaviour of dictionary displays by overwriting dict. Here is an example:

# test.py
from collections import OrderedDict
print {3: 4, "a": 2}
dict = OrderedDict
print {3: 4, "a": 2}

Now we can run this file using my_execfile("test.py"), yielding the output

{'a': 2, 3: 4}
OrderedDict([(3, 4), ('a', 2)])

Note that for simplicity, the above code doesn't touch dictionary comprehensions, which should be transformed to generator expressions passed to the dict() constructor. You'd need to add a visit_DictComp() method to the DictDisplayTransformer class. Given the above example code, this should be straight-forward.

Again, I don't recommend this kind of messing around with the language semantics. Did you have a look into the ConfigParser module?

Solution 3

OrderedDict is not "standard python syntax", however, an ordered set of key-value pairs (in standard python syntax) is simply:

[('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')]

To explicitly get an OrderedDict:

OrderedDict([('key1 name', 'value1'), ('key2 name', 'value2'), ('key3 name', 'value3')])

Another alternative, is to sort dictname.items(), if that's all you need:

sorted(dictname.items())

Solution 4

As of python 3.6, all dictionaries will be ordered by default. For now, this is an implementation detail of dict and should not be relied upon, but it will likely become standard after v3.6.

Insertion order is always preserved in the new dict implementation:

>>>x = {'a': 1, 'b':2, 'c':3 }
>>>list(x.keys())
['a', 'b', 'c']

As of python 3.6 **kwargs order [PEP468] and class attribute order [PEP520] are preserved. The new compact, ordered dictionary implementation is used to implement the ordering for both of these.

Solution 5

The one solution I found is to patch python itself, making the dict object remember the order of insertion.

This then works for all kind of syntaxes:

x = {'a': 1, 'b':2, 'c':3 }
y = dict(a=1, b=2, c=3)

etc.

I have taken the ordereddict C implementation from https://pypi.python.org/pypi/ruamel.ordereddict/ and merged back into the main python code.

If you do not mind re-building the python interpreter, here is a patch for Python 2.7.8: https://github.com/fwyzard/cpython/compare/2.7.8...ordereddict-2.7.8.diff .A

View more solutions

Comments

[fdb]

my question isn't if OrderedDict is "standard python syntax" but if is possible to override the {...} notation

[Sven Marnach]

@fdb: In Python {} creates a dict object, which is unordered by definition. You can of course define your own language with {} denoting an orderd dictionary. You can even write a small wrapper that translates your new language to Python. Is this what you actually want?

[fdb]

@SvenMarnach: yes! but was hoping that simply overriding dict constructor with OrderedDict (dict = OrderedDict) would work.

[Daenyth]

@fdb: That only works if you make your dictionary by calling dict()

[Sven Marnach]

"Impossible" might be a bit too strong a word -- see my answer.

[mrivard]

@Sven: Yes, I totally enjoyed your answer! :) I think I will let my wording stand, though. Please adjust your understanding of "impossible" in this context to match reality ;)

[fdb]

Yes I will use ConfigParser...but your solution is illuminating. Thank you very much.

[Alex Bitek]

json.loads and json.load have been also updated since Python 3.1 with support for object_pairs_hook docs.python.org/3.4/library/json.html#json.load

[PaulMcG]

sorted(dictname.items()) gives the items in ascending lexical order of the keys. OrderedDict ordering is by insertion order, not by key lexical order. You may be thinking of SortedDict (which does not currently exist).

[Tony Suffolk 66]

shudder - i can see why you call that a hack - please - don't use that in production

[Tony Suffolk 66]

before you think of changing the language semantics - think about the principle 'Explicit is better than implicit' - if you try to override '{}' or hide to avoid having to type 'OrderedDict' - you will end up making your code far more difficult to read for others - of for yourself 6 months down the line. Just type 'OrderedDict' - it is understood, and does what you want - more typing, but improved readability.

[Tony Suffolk 66]

@fdb - before you think of changing the language semantics - think about the principle 'Explicit is better than implicit' - if you try to override '{}' or hide to avoid having to type 'OrderedDict' - you will end up making your code far more difficult to read for others - of for yourself 6 months down the line. Just type 'OrderedDict' - it is understood, and does what you want - more typing, but improved readability.

[Régis B.]

This is genius. Evil genius.

[Eric]

@RégisB.: This post has got a lot of attention today - was it linked to from elsewhere?

[Tony Baguette]

@Eric I came here from reddit reddit.com/r/Python/comments/4xyfh7/…

[Ashwini Chaudhary]

TIL we can use strings with slice object.

[user982201]

To help decipher how this works, see stackoverflow.com/questions/2936863/…

[Nick Sweeting]

As of 2016/12, the Pypy implementation will become the standard python dict implementation, nice job predicting this 2 years in advance!

[Samuel]

Maybe it has to do with what docs.python.org/3/whatsnew/3.6.html#new-dict-implementation says: "The order-preserving aspect of this new implementation is considered an implementation detail and should not be relied upon". Still, I found the information interesting, so here's an upvote!

[TatianaP]

@Eric it doesn't work if one key: menu = ordereddict["about" : "about"]; but it seems easy to fix it.

[Eric]

@TatianaP: Nice catch, fixed. But use the pypi version not mine anyway!

[ShadowRanger]

Note: Both of these violate the expectations of their operators in extreme ways. Both __add__ and __getitem__ are intended to be non-mutating, and slicing support is expected to be an aggregate form of indexing support, not a completely unrelated behavior. Violating those expectations is asking for maintainability nightmare. The slice hack is much better used to achieve the result given in the accepted answer, where it's a factory object that makes a normal OrderedDict, not an OrderedDict replacement with ongoing weird behaviors.

[Nick Sweeting]

Update: Insert order being preserved is now standard in 3.7 and can be relied upon.