Can PyYAML dump dict items in non-alphabetical order?

Question

I'm using yaml.dump to output a dict. It prints out each item in alphabetical order based on the key.

>>> d = {"z":0,"y":0,"x":0}
>>> yaml.dump( d, default_flow_style=False )
'x: 0\ny: 0\nz: 0\n'

Is there a way to control the order of the key/value pairs?

In my particular use case, printing in reverse would (coincidentally) be good enough. For completeness though, I'm looking for an answer that shows how to control the order more precisely.

I've looked at using collections.OrderedDict but PyYAML doesn't (seem to) support it. I've also looked at subclassing yaml.Dumper, but I haven't been able to figure out if it has the ability to change item order.

Solution 1

There's probably a better workaround, but I couldn't find anything in the documentation or the source.

---

Python 2 (see comments)

I subclassed OrderedDict and made it return a list of unsortable items:

from collections import OrderedDict

class UnsortableList(list):
    def sort(self, *args, **kwargs):
        pass

class UnsortableOrderedDict(OrderedDict):
    def items(self, *args, **kwargs):
        return UnsortableList(OrderedDict.items(self, *args, **kwargs))

yaml.add_representer(UnsortableOrderedDict, yaml.representer.SafeRepresenter.represent_dict)

And it seems to work:

>>> d = UnsortableOrderedDict([
...     ('z', 0),
...     ('y', 0),
...     ('x', 0)
... ])
>>> yaml.dump(d, default_flow_style=False)
'z: 0\ny: 0\nx: 0\n'

---

Python 3 or 2 (see comments)

You can also write a custom representer, but I don't know if you'll run into problems later on, as I stripped out some style checking code from it:

import yaml

from collections import OrderedDict

def represent_ordereddict(dumper, data):
    value = []

    for item_key, item_value in data.items():
        node_key = dumper.represent_data(item_key)
        node_value = dumper.represent_data(item_value)

        value.append((node_key, node_value))

    return yaml.nodes.MappingNode(u'tag:yaml.org,2002:map', value)

yaml.add_representer(OrderedDict, represent_ordereddict)

But with that, you can use the native OrderedDict class.

Solution 2

If you upgrade PyYAML to 5.1 version, now, it supports dump without sorting the keys like this:

yaml.dump(data, sort_keys=False)

As shown in help(yaml.Dumper), sort_keys defaults to True:

Dumper(stream, default_style=None, default_flow_style=False,
  canonical=None, indent=None, width=None, allow_unicode=None,
  line_break=None, encoding=None, explicit_start=None, explicit_end=None,
  version=None, tags=None, sort_keys=True)

(These are passed as kwargs to yaml.dump)

Solution 3

For Python 3.7+, dicts preserve insertion order. Since PyYAML 5.1.x, you can disable the sorting of keys (#254). Unfortunately, the sorting keys behaviour does still default to True.

>>> import yaml
>>> yaml.dump({"b":1, "a": 2})
'a: 2\nb: 1\n'
>>> yaml.dump({"b":1, "a": 2}, sort_keys=False)
'b: 1\na: 2\n'

My project oyaml is a monkeypatch/drop-in replacement for PyYAML. It will preserve dict order by default in all Python versions and PyYAML versions.

>>> import oyaml as yaml  # pip install oyaml
>>> yaml.dump({"b":1, "a": 2})
'b: 1\na: 2\n'

Additionally, it will dump the collections.OrderedDict subclass as normal mappings, rather than Python objects.

>>> from collections import OrderedDict
>>> d = OrderedDict([("b", 1), ("a", 2)])
>>> import yaml
>>> yaml.dump(d)
'!!python/object/apply:collections.OrderedDict\n- - - b\n    - 1\n  - - a\n    - 2\n'
>>> yaml.safe_dump(d)
RepresenterError: ('cannot represent an object', OrderedDict([('b', 1), ('a', 2)]))
>>> import oyaml as yaml
>>> yaml.dump(d)
'b: 1\na: 2\n'
>>> yaml.safe_dump(d)
'b: 1\na: 2\n'

Solution 4

One-liner to rule them all:

yaml.add_representer(dict, lambda self, data: yaml.representer.SafeRepresenter.represent_dict(self, data.items()))

That's it. Finally. After all those years and hours, the mighty represent_dict has been defeated by giving it the dict.items() instead of just dict

Here is how it works:

This is the relevant PyYaml source code:

    if hasattr(mapping, 'items'):
        mapping = list(mapping.items())
        try:
            mapping = sorted(mapping)
        except TypeError:
            pass
    for item_key, item_value in mapping:

To prevent the sorting we just need some Iterable[Pair] object that does not have .items().

dict_items is a perfect candidate for this.

Here is how to do this without affecting the global state of the yaml module:

#Using a custom Dumper class to prevent changing the global state
class CustomDumper(yaml.Dumper):
    #Super neat hack to preserve the mapping key order. See https://stackoverflow.com/a/52621703/1497385
    def represent_dict_preserve_order(self, data):
        return self.represent_dict(data.items())    

CustomDumper.add_representer(dict, CustomDumper.represent_dict_preserve_order)

return yaml.dump(component_dict, Dumper=CustomDumper)

Solution 5

This is really just an addendum to @Blender's answer. If you look in the PyYAML source, at the representer.py module, You find this method:

def represent_mapping(self, tag, mapping, flow_style=None):
    value = []
    node = MappingNode(tag, value, flow_style=flow_style)
    if self.alias_key is not None:
        self.represented_objects[self.alias_key] = node
    best_style = True
    if hasattr(mapping, 'items'):
        mapping = mapping.items()
        mapping.sort()
    for item_key, item_value in mapping:
        node_key = self.represent_data(item_key)
        node_value = self.represent_data(item_value)
        if not (isinstance(node_key, ScalarNode) and not node_key.style):
            best_style = False
        if not (isinstance(node_value, ScalarNode) and not node_value.style):
            best_style = False
        value.append((node_key, node_value))
    if flow_style is None:
        if self.default_flow_style is not None:
            node.flow_style = self.default_flow_style
        else:
            node.flow_style = best_style
    return node

If you simply remove the mapping.sort() line, then it maintains the order of items in the OrderedDict.

Another solution is given in this post. It's similar to @Blender's, but works for safe_dump. The common element is the converting of the dict to a list of tuples, so the if hasattr(mapping, 'items') check evaluates to false.

Update:

I just noticed that The Fedora Project's EPEL repo has a package called python2-yamlordereddictloader, and there's one for Python 3 as well. The upstream project for that package is likely cross-platform.

View more solutions

Comments

[mwcz]

Very nice, I like your style. I'll go with the first solution because I think it's a little more clear. I'll have to rebuild the dict either way, and the MappingNode call and strange unicode string in the representer make it kind of opaque (to me!). Thanks!

[Blender]

@mwcz: The only problem with the first one is subclassing OrderedDict, so if it works, it works.

[Hayk Martiros]

I'm not sure if it's my version of Python (3.4), but this isn't working. I looked in the source at yaml/representer.py:111, and you can see mapping = sorted(mapping). It is using the sorted builtin, not the .sort() method of UnsortableList. Any ideas?

[Michael Scheper]

Could you add a few lines of code as an example? Although this isn't quite a link-only answer, it won't leave a lot to go on if the links break, and example code is more convenient for us lazy people. 😉 For more, please refer to this question about StackOverflow style: meta.stackexchange.com/questions/8231/… (this link is unlikely to rot 😉)

[orodbhen]

Looking at the PyYAML source, it turns out that dumper.represent_mapping would do this if a single line was removed. See my answer for details. I think it would be worth submitting a request to have this as an option.

[Louis]

The approach of adding a representer for dict won't work reliably on versions of Python prior to 3.7. See this Q and its answers. I was looking at your answer and puzzled over the fact that the output was ordered in key insertion order despite using dict rather than OrderedDict. Fortunately, the method used here can be easily adapted to OrderedDict for those who need it: add a representer for OrderedDict instead of dict, with the same implementation and it works.

[Nelson Rodrigues]

thanks @Cooper.Wu This is a clear example where looking for the most recent answer helped a lot.

[Blessy]

This works for me, without needing to use OrderedDict.

[wesinat0r]

this does not preserve the order of a normal dict for me, trying with ordereddict, I still need the representer function above for this to work