Funnelweb error, cannot import OrderedDict

Question

I am trying to import some static pages from a site using funnelweb add-on for plone. Here is what I write: bin/funnelweb --crawler:url=http://wiki.scandiatransplant.com --crawler:max=50 --ploneupload=http://admin:admin@localhost:8080/TestPage.

I get get this error:

ImportError: cannot import name OrderedDict

The full traceback is added below. Does anyone know how to solve this?

Traceback (most recent call last):
  File "bin/funnelweb", line 116, in <module>
    mr.migrator.runner.runner({},"funnelweb.remote")
  File "/home/magiq/Plone/buildout-cache/eggs/mr.migrator-1.0.1-py2.6.egg/mr/migrator/runner/__init__.py", line 69, in runner
    load_config('autoinclude.zcml', mr.migrator)
  File "/home/magiq/Plone/buildout-cache/eggs/Zope2-2.13.12-py2.6.egg/Zope2/App/zcml.py", line 55, in load_config
    _context = xmlconfig.file(config, package, _context, execute=execute)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/xmlconfig.py", line 649, in file
    include(context, name, package)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/expatreader.py", line 207, in feed
    self._parser.Parse(data, isFinal)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/expatreader.py", line 349, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/config.py", line 706, in finish
    actions = self.handler(context, **args)
  File "/home/magiq/Plone/buildout-cache/eggs/z3c.autoinclude-0.3.4-py2.6.egg/z3c/autoinclude/zcml.py", line 104, in includePluginsDirective
    includeZCMLGroup(_context, info, filename)
  File "/home/magiq/Plone/buildout-cache/eggs/z3c.autoinclude-0.3.4-py2.6.egg/z3c/autoinclude/zcml.py", line 30, in includeZCMLGroup
    include(_context, filename, includable_package)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/xmlconfig.py", line 548, in include
    processxmlfile(f, context)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/xmlconfig.py", line 380, in processxmlfile
    parser.parse(src)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/expatreader.py", line 107, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/expatreader.py", line 207, in feed
    self._parser.Parse(data, isFinal)
  File "/home/magiq/Plone/Python-2.6/lib/python2.6/xml/sax/expatreader.py", line 349, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/xmlconfig.py", line 359, in endElementNS
    self.context.end()
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/config.py", line 558, in end
    self.stack.pop().finish()
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/config.py", line 705, in finish
    args = toargs(context, *self.argdata)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/config.py", line 1397, in toargs
    args[str(name)] = field.fromUnicode(s)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/fields.py", line 137, in fromUnicode
    value = self.context.resolve(name)
  File "/home/magiq/Plone/buildout-cache/eggs/zope.configuration-3.7.4-py2.6.egg/zope/configuration/config.py", line 179, in resolve
    mod = __import__(mname, *_import_chickens)
  File "/home/magiq/Plone/buildout-cache/eggs/transmogrify.webcrawler-1.2.1-py2.6.egg/transmogrify/webcrawler/webcrawler.py", line 21, in <module>
    from collections import OrderedDict
zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/home/magiq/Plone/buildout-cache/eggs/mr.migrator-1.0.1-py2.6.egg/mr/migrator/autoinclude.zcml", line 10.4-10.85
    ZopeXMLConfigurationError: File "/home/magiq/Plone/buildout-cache/eggs/transmogrify.webcrawler-1.2.1-py2.6.egg/transmogrify/webcrawler/configure.zcml", line 9.4-10.46
    ImportError: cannot import name OrderedDict

Solution

OrderedDict is a new class added to Python 2.7, so it is missing in your case.

You can install a backport:

pip install ordereddict

or add that as a dependency in your buildout, and edit Plone/buildout-cache/eggs/transmogrify.webcrawler-1.2.1-py2.6.egg/transmogrify/webcrawler/webcrawler.py line 21 to change the import from:

from collections import OrderedDict

to

try:
    from collections import OrderedDict
except ImportError:
    # python 2.6 or earlier, use backport
    from ordereddict import OrderedDict

I have filed an issue in the funnelweb issue tracker to request that this is added to a future release of the package, see issue 22.

Comments

[user1960836]

I have installed funnelweb on a 3rd machine, and having trouble with the message that it can't import OrderedDict. I have done like suggested here (which worked for me earlier on 2 other machines), but on this 3rd machine, it still complains. Is there another approach than listed above? Also, I haven't got the newest version of Plone on this machine (But it is still version. 4.x). Can this affect the installation of ordereddict? I rather not upgrade plone if that is not the case, cause of add-ons incompabilities.

[Martijn Pieters]

Earlier versions of Plone do not support Python 2.7 (officially, it'll probably run), so you are probably running with Python 2.6 or earlier. You need to install the ordereddict package then, into the right Python installation. If import ordereddict fails in your python prompt (and make sure you are 100% certain that you have the right python, the one that runs your Plone server) then you didn't install it in the right location.

[user1960836]

Actually, here is some of the stacktrace: from collections import OrderedDict zope.configuration.xmlconfig.ZopeXMLConfigurationError: File "/usr/local/Plone/buildout-cache/eggs/mr.migrator-1.0.1-py2.‌​6.egg/mr/migrator/au‌​toinclude.zcml", line 10.4-10.85 ZopeXMLConfigurationError: File "/usr/local/Plone/buildout-cache/eggs/transmogrify.ploneremo‌​te-1.3-py2.6.egg/tra‌​nsmogrify/ploneremot‌​e/configure.zcml", line 24.4-25.67 ImportError: cannot import name OrderedDict

[user1960836]

Import orderreddict works fine in the python promt, but import OrderededDict fails. Is this ok?

[Martijn Pieters]

@user1960836: Yes, of course it is. You need to import from the module: from ordereddict import OrderedDict.

[user1960836]

Importing from the module also seems to work, it doesn't complain. Regarding installing it in the right/wrong location. I have useed the unified installer on all 3 machines, and installed in the same way. But can they deviate in where they place python as an example?

[Martijn Pieters]

@user1960836: What does head bin/funnelweb tell you is used for Python? Did you 'patch' the webcrawler.py file in all buildouts? Check what egg is used by using grep transmogrify.webcrawler bin/funnelweb to make sure you got the right one.

[user1960836]

That is not very readable. Is there a way I can make it more readable as a comment? And that is: head bin/funnelweb by the way

[Martijn Pieters]

I didn't ask you to give me the full list :-) I wanted you to go look for the correct transmogrify.webcrawler egg and make sure you had patched it like in my answer. :-) You could also add ordereddict as an egg to the funnelweb recipe part, btw.

[user1960836]

Okay, I notice that it is using python 2.6. I'm not really sure how to change it so that it uses version 2.7 of python?

[user1960836]

Here is the result of the grep: root@debian:/usr/local/Plone/zinstance# grep transmogrify.webcrawler bin/funnelweb '/usr/local/Plone/buildout-cache/eggs/transmogrify.webcrawle‌​r-1.2.1-py2.6.egg', root@debian:/usr/local/Plone/zinstance#

[Martijn Pieters]

@user1960836: Stick to 2.6. Open up the buildout.cfg file where you added a [funnelweb] section. Add a line eggs = ordereddict underneath the recipe = funnelweb line. That'll make sure it's available and imported. Re-run buildout.

[Martijn Pieters]

let us continue this discussion in chat

[user1960836]

adding ordereddict as an egg to buildout sounds good. Do you know how the syntax is? But will it correct anything, since it seems to be the python version that is the problem?

[Ida]

I ran into the same problem and as usual Martijn's answer helped and is explained very detailed and comprehensible. @user1960836: Would you mind to accept the answer?