Simple tool/library to visualize huge python dict

Question

I have a huge dict structure like this one:

my_data = {
    'key1': {
        '_': 'value1': 'aaa'
    },
    'key2': {
        '_': 'value2': 'bbb',
        'key2.1': {
            '_': 'ccc',
            'key2.1.1': {
                '_': 'ddd'
            }
        }
        'key2.2': {
            '_': 'eee',
            'key2.2.1': {
                '_': 'fff'
            }
            'key2.2.2': {
                '_': 'ggg'
            }               
        }
    }
}

and so on.

I want to display it to user in a kind of tree representation, using GTK, TK or anything to be able to browse it collapse and expand branches and probably search keys and values.

May be I do not need to develop such a tool by hands and there is already something that can visualize this kind of data out of the box?

Solution 1

I'm finally ended up with converting my data into json as @PavelAnossov suggested and using d3 Tree Layout.

Solution 2

I do not know of a ready-to-use tool, but you could use Traits UI to swiftly develop your own

from enthought.traits.api \
    import HasTraits, Instance

from enthought.traits.ui.api \
    import View, VGroup, Item, ValueEditor

class DictEditor(HasTraits):
    Object = Instance( object )

    def __init__(self, obj, **traits):
        super(DictEditor, self).__init__(**traits)
        self.Object = obj

    def trait_view(self, name=None, view_elements=None):
        return View(
          VGroup(
            Item( 'Object',
                  label      = 'Debug',
                  id         = 'debug',
                  editor     = ValueEditor(),
                  style      = 'custom',
                  dock       = 'horizontal',
                  show_label = False
            ),
          ),
          title     = 'Dictionary Editor',
          width     = 800,
          height    = 600,
          resizable = True,
        )


def build_sample_data():
    my_data = dict(zip(range(10),range(10,20)))
    my_data[11] = dict(zip(range(10),range(10,20)))
    my_data[11][11] = dict(zip(range(10),range(10,20)))
    return my_data

# Test
if __name__ == '__main__':
    my_data = build_sample_data()
    b = DictEditor(my_data)
    b.configure_traits()

That's it. You will have a GUI like:

Traits UI uses the Model-View-Controller approach to create GUI without having the need to programatically create every widget. Here, I use the predefined ValueEditor to display arbitrary types. You can now extend it to support searching, filtering etc...

EDIT

Simple extension to support filtering:

# -*- coding: utf-8 -*-
"""
Created on Fri Feb 22 12:52:28 2013

@author: kranzth
"""
from enthought.traits.api \
    import HasTraits, Instance, Str, on_trait_change

from enthought.traits.ui.api \
    import View, VGroup, Item, ValueEditor, TextEditor

from copy import deepcopy

class DictEditor(HasTraits):
    SearchTerm = Str()
    Object = Instance( object )

    def __init__(self, obj, **traits):
        super(DictEditor, self).__init__(**traits)
        self._original_object = obj
        self.Object = self._filter(obj)

    def trait_view(self, name=None, view_elements=None):
        return View(
          VGroup(
            Item( 'SearchTerm',
                  label      = 'Search:',
                  id         = 'search',
                  editor     = TextEditor(),
                  #style      = 'custom',
                  dock       = 'horizontal',
                  show_label = True
            ),
            Item( 'Object',
                  label      = 'Debug',
                  id         = 'debug',
                  editor     = ValueEditor(),
                  style      = 'custom',
                  dock       = 'horizontal',
                  show_label = False
            ),
          ),
          title     = 'Dictionary Editor',
          width     = 800,
          height    = 600,
          resizable = True,
        )

    @on_trait_change("SearchTerm")
    def search(self):
        self.Object = self._filter(self._original_object, self.SearchTerm)

    def _filter(self, object_, search_term=None):
        def has_matching_leaf(obj):
            if isinstance(obj, list):
                return any(
                        map(has_matching_leaf, obj))
            if isinstance(obj, dict):
                return any(
                        map(has_matching_leaf, obj.values()))
            else:
                try:
                    if not str(obj) == search_term:
                        return False
                    return True
                except ValueError:
                    False

        obj = deepcopy(object_)
        if search_term is None:
            return obj

        if isinstance(obj, dict):
            for k in obj.keys():
                if not has_matching_leaf(obj[k]):
                    del obj[k]

            for k in obj.keys():
                if isinstance(obj, dict):
                    obj[k] = self._filter(obj[k], search_term)
                elif isinstance(obj, list):
                    filter(has_matching_leaf,obj[k])

        return obj



def build_sample_data():
    def make_one_level_dict():
        return dict(zip(range(100),
                        range(100,150) + map(str,range(150,200))))

    my_data = make_one_level_dict()
    my_data[11] = make_one_level_dict()
    my_data[11][11] = make_one_level_dict()
    return my_data

# Test
if __name__ == '__main__':
    my_data = build_sample_data()
    b = DictEditor(my_data)
    b.configure_traits()

will give you a textbox with "filter-as-you-type". The search isn't completely correct for all cases, but you can figure out the idea.

Please note that in this sample the data in the dict are partly integers and partly strings, and both types will be found.

Solution 3

There are some great answers already in here, but I believe this one qualifies as "simple" (it uses only python bult-in libraries tkinter and uuid).

It is based on John Gaines Jr.'s answer in another question, modified by Will Ware to support lists, modified by me to also support tuples (runs on python 3).

I've also reorganized it so that you can call the viewer with something as simple as tk_tree_view(data), passing in a dictionary (as in the example at the end).

import uuid
import tkinter as tk
from tkinter import ttk


def j_tree(tree, parent, dic):
    for key in sorted(dic.keys()):
        uid = uuid.uuid4()
        if isinstance(dic[key], dict):
            tree.insert(parent, 'end', uid, text=key)
            j_tree(tree, uid, dic[key])
        elif isinstance(dic[key], tuple):
            tree.insert(parent, 'end', uid, text=str(key) + '()')
            j_tree(tree, uid,
                   dict([(i, x) for i, x in enumerate(dic[key])]))
        elif isinstance(dic[key], list):
            tree.insert(parent, 'end', uid, text=str(key) + '[]')
            j_tree(tree, uid,
                   dict([(i, x) for i, x in enumerate(dic[key])]))
        else:
            value = dic[key]
            if isinstance(value, str):
                value = value.replace(' ', '_')
            tree.insert(parent, 'end', uid, text=key, value=value)


def tk_tree_view(data):
    # Setup the root UI
    root = tk.Tk()
    root.title("tk_tree_view")
    root.columnconfigure(0, weight=1)
    root.rowconfigure(0, weight=1)

    # Setup the Frames
    tree_frame = ttk.Frame(root, padding="3")
    tree_frame.grid(row=0, column=0, sticky=tk.NSEW)

    # Setup the Tree
    tree = ttk.Treeview(tree_frame, columns=('Values'))
    tree.column('Values', width=100, anchor='center')
    tree.heading('Values', text='Values')
    j_tree(tree, '', data)
    tree.pack(fill=tk.BOTH, expand=1)

    # Limit windows minimum dimensions
    root.update_idletasks()
    root.minsize(root.winfo_reqwidth(), root.winfo_reqheight())
    root.mainloop()


if __name__ == "__main__":
    # Setup some test data
    data = {
        "firstName": "John",
        "lastName": "Smith",
        "gender": "male",
        "age": 32,
        "address": {
            "streetAddress": "21 2nd Street",
            "city": "New York",
            "state": "NY",
            "postalCode": "10021"},
        "phoneNumbers": [
            {"type": "home", "number": "212 555-1234"},
            {"type": "fax",
             "number": "646 555-4567",
             "alphabet": [
                 "abc",
                 "def",
                 "ghi"]
             }
        ]}

    # call it with
    tk_tree_view(data)

It looks like this:

Solution 4

If you're using an IDE set a breakpoint after the dictionary is initialized and has the data you want to explore then run in debug mode. There should be a "Variables" view in debug mode where you can expand and collapse the dictionary as you mentioned.

Solution 5

This simple function prints a dictionary in table form. It can also deal with nested dictionaries.

def visualise_dict(d,lvl=0):

    # go through the dictionary alphabetically 
    for k in sorted(d):

        # print the table header if we're at the beginning
        if lvl == 0 and k == sorted(d)[0]:
            print('{:<25} {:<15} {:<10}'.format('KEY','LEVEL','TYPE'))
            print('-'*79)

        indent = '  '*lvl # indent the table to visualise hierarchy
        t = str(type(d[k]))

        # print details of each entry
        print("{:<25} {:<15} {:<10}".format(indent+str(k),lvl,t))

        # if the entry is a dictionary
        if type(d[k])==dict:
            # visualise THAT dictionary with +1 indent
            visualise_dict(d[k],lvl+1)

With a sample dictionary:

d = {}
d.update({1:{},2:{}})
d[1]['foo'] = {}
d[1]['foo']['bar'] = 1
d[2]['bar'] = 5.2

visualise_dict(d)

returns

In [1]: visualise_dict(d)
KEY                       LEVEL           TYPE      
-------------------------------------------------------------------------------
1                         0               <class 'dict'>
  foo                     1               <class 'dict'>
    bar                   2               <class 'int'>
2                         0               <class 'dict'>
  bar                     1               <class 'float'>

View more solutions

Comments

[Pavel Anossov]

If your dict only contains simple types, it will probably be easier to encode it to JSON and find a JSON viewer.

[lig]

@PavelAnossov agree. it is a possible way to solution.

[lig]

nice one. will give it a try

[lig]

sorry. cannot mark this as simple. after half of an hour i'm still getting NotImplemented errors from the installed modules. Nor PyPi nor official github version works. It is likely that enthought.traits.ui package works but i cannot make it work.

[Thorsten Kranz]

Which OS are you on? On Linux installation is Easy, on windows I'd recommend Python(x,y)

[lig]

I'm on Linux. I will be glad to see working installation guide.

[Thorsten Kranz]

If you are on Debian / Ubuntu, all you have to do is sudo apt-get install python-traitsui. I guess other distros will have it in their repositories as well, but I never tried this.

[Thorsten Kranz]

I tried it on Ubuntu, and there the modules have different names. Write from traits.api ... and from traitsui.api ...

[lig]

That way did not worked for me. I'm getting NotImplemented error.

[chisaipete]

Did you conform to the "name" "children" json format as shown in the d3 link you included? If so, could you share more of the complete solution? I'm having a hard time converting my similar dict structure into that json format.

[lig]

That was a bunch of ugly hardcode made by my collegue at work. I dont think I'm able (and do not want) to share this proprietary piece of code anywhere.

[chisaipete]

I totally understand! Thanks for the reply--I was hoping someone had solved the issue :)

[lig]

Is there a way to install it using pip? As I do not use Ubuntu for a while. And I want to be able to install it on Fedora and on Windows.

[czxttkl]

i think you could just use "pip install traitsui", since it is enlisted on pypi website: pypi.python.org/pypi/traitsui

[K. Brafford]

I wish I could +1e6 this. Thank you so much. For people in the future, czxttkl's answer down there works fine. This little util has helped me get over a hump

[K. Brafford]

Just letting everyone know, this still works in 2018. I install pyqt4 from Christoph Gohlke's page, and pip install traitsui. Then these mods to the Thorsten's answer work nicely.

[K. Brafford]

One more observation...it alphabetizes the keys, instead of traversing the dict in iteration order. There doesn't seem to be an obvious way to fix this in your example.

[K. Brafford]

If you want to preserve order in keys (if you are using an ordereddict) change this for key in sorted(dic.keys()) to for key in dic.keys()

[Pacific Stickler]

How do you edit this dict/json it in place and save it in a variable to be written to file?

[canbax]

sorted(dic.keys()) does not work for str type. I think sorted should not be used at all because usually, people use strings as their keys.