setuptools vs. distutils: why is distutils still a thing?

Question

Python has a confusing history of tools that can be used to package and describe projects: these include distutils in the Standard Library, distribute, distutils2, and setuptools (and maybe more). It appears that distribute and distutils2 were discontinued in favor of setuptools, which leaves two competing standards.

To my understanding setuptools offers far more options (e.g. declaring dependencies, tests, etc.) than distutils, however it is not included in the Python standard library (yet?).

The Python Packaging User Guide[1] recommends now:

Use setuptools to define projects and create Source Distributions.

And explains:

Although you can use pure distutils for many projects, it does not support defining dependencies on other projects and is missing several convenience utilities for automatically populating package metadata correctly that are provided by setuptools. Being outside the standard library, setuptools also offers a more consistent feature set across different versions of Python, and (unlike distutils), setuptools will be updated to produce the upcoming “Metadata 2.0” standard formats on all supported versions.

Even for projects that do choose to use distutils, when pip installs such projects directly from source (rather than installing from a prebuilt wheel file), it will actually build your project using setuptools instead.

However, looking into various project's setup.py files reveals that this does not seem to be an actual standard. Many packages still use distutils and those that support setuptools often mix setuptools with distutils e.g. by doing a fallback import:

try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup

Followed by an attempt to find a way to write a setup that can be installed by both setuptools and distutils. This often includes various ways of error-prone dependency checking, since distutils does not support dependencies in the setup function.

Why are people still making the extra effort to support distutils - is the fact that setuptools is not in the standard library the only reason? What are the advantages of distutils and are there any drawbacks of writing setup.py files that only support setuptools.

Solution 1

Have a look at this SO question. It explains all the packaging methods very well, and might help answer your question to some extent: Differences between distribute, distutils, setuptools and distutils2?

Distutils is still the standard tool for packaging in Python. It is included in the standard library (Python 2 and Python 3.0 to 3.3). It is useful for simple Python distributions, but lacks features. It introduces the distutils Python package that can be imported in your setup.py script.

Setuptools was developed to overcome Distutils' limitations, and is not included in the standard library. It introduced a command-line utility called easy_install. It also introduced the setuptools Python package that can be imported in your setup.py script, and the pkg_resources Python package that can be imported in your code to locate data files installed with a distribution. One of its gotchas is that it monkey-patches the distutils Python package. It should work well with pip. The latest version was released in July 2013.

So, as you can see setuptools should be preferred to distutils, and I see where your question comes from, however I don't see distutils losing support anytime soon, as, simply put, it is used in many cases with some popular legacy programs. And as you probably know changing these sorts of things in legacy programs can be quite a pain and come with quite a few problems, for example incompatibilities, which would then lead to the developer having to rewrite the source code. So there is that, and also the fact that distutils is a part of the standard python library whereas setuptools is not. So, if you are creating a python program, in this day and age, use setuptools, however keep in mind that without distutils, setuptools would have never existed.

Solution 2

is the fact that setuptools is not in the standard library the only reason

That's one reason. The following is straight from the NumPy setup.py:

if len(sys.argv) >= 2 and ('--help' in sys.argv[1:] or
        sys.argv[1] in ('--help-commands', 'egg_info', '--version',
                        'clean')):
    # Use setuptools for these commands (they don't work well or at all
    # with distutils).  For normal builds use distutils.
    try:
        from setuptools import setup
    except ImportError:
        from distutils.core import setup

So NumPy prefers setuptools if it can find it. But then SciPy used to do this, until it was patched to prefer distutils in some situations. Citing the commit log:

Setuptools sets mode +x on the test scripts, so that Nose refuses to run
them. Better not do that.

Of course, a merger between setuptools and distribute should resolve all this in due time, but many packages still need to support Python 2.6 installations.

Solution 3

There are several reasons we still talk about and use distutils, even though setuptools is without a doubt the better tool set.

Firstly, distutils is available everywhere. If you are looking to build a module for sharing with others, and don't have any complicated requirements, it is guaranteed to be available on your work machine. This is particularly important if you have to support older versions of python, or if you find yourself working in an unfamiliar environment.

Secondly, setuptools provides enhancements to distutils. It is therefore modeled after the distutils tool set and takes all of it's structure from there. The documentation for setuptools assumes the reader is familiar with distutils and only documents how it enhances the base tool set. You can think of it that distutils defines the dialect and setuptools enhances that dialect.

My personal approach for new projects is start with the assumption I'm going to use distutils. Only as the project grows to require a feature of setuptools do I make the upgrade. The setuptools is a drop-in-replacement for distutils, it's a one-line change to my setup.py.

Solution 4

Basically, it's due to the division of responsibilities.

setuptools is not a part of Python standard library because it's maintained by a 3rd party rather than Python core team. Which means, among other things:

• it isn't covered by the core test suite and isn't relied upon by core functionality
• it doesn't itself set core standards for add-on modules (their location, means of import, C extensions' binary interface etc.).
• it's updated and released independently from Python releases

Effectively, the core team has narrowed down the scope of distutils, reserving the "core standards" and "minimal necessary compilation" parts for themselves while leaving all the stuff beyond that (extended compiler/package format/whatever support) to 3rd parties. The code that was previously covering those "extended parts" was left stale for backwards compatibility.

From Distributing Python Modules — Python 2.7.12 documentation:

While direct use of distutils is being phased out, it still laid the foundation for the current packaging and distribution infrastructure, and it not only remains part of the standard library, but its name lives on in other ways (such as the name of the mailing list used to coordinate Python packaging standards development).

Packages for other OSes are likewise likely to provide setuptools and pip separately - for the aforementioned reasons

• and because they aren't necessary - or are even detrimental for maintainability - when there's already another package manager on the system.

View more solutions

Comments

[metatoaster]

distutils has been merged back into setuptools, but there are legacy apps that were written to use distutils and there are costs involved to migrate to correct standards.

[kay]

"It appears that distribute and distutils2 were discontinued in favor of setuptools", correct, distribute is only a wrapper for setuptools now, and distutils2 is dead.

[user2314737]

setuptools is an enhanced alternative to distutils but note that "The recommended pip installer runs all setup.py scripts with setuptools, even if the script itself only imports distutils" (source)

[Stevoisiak]

Related: Differences between distribute, distutils, setuptools and distutils2?

[cel]

"Distutils is still the standard tool for packaging in Python." contradicts the Python Packaging User Guide.

[Admin]

I don't believe so, does it explicitly say that setuptools is the standard. Also note that, that phrase was quoted from the website I provided, so, those are not my words. However it is an opinion which I, among many other people agree with.

[cel]

I awarded you the bounty since the community seems to agree with you. Unfortunately this question did not get as much attention as I wished it would get.

[cel]

Thanks for your answer. I think the availability argument can't be that important, since the setuptools installation can be bootstrapped. I see that if distutils provides enough functionallity - it does make sense to use it. But mixing distutils and setuptools in my opinion is not a very clean way to achieve one's goals. Although, @larsmans showed in his answer some difficulties with setuptools which force to use distutils for some tasks.

[R4444]

distribute was a fork of setuptools and is now merged back.

[Stuart Gathman]

easy_install is the main reason I rejected setuptools - it has been a huge source of problems for me in packages that use it (easier to just repackage). Other features are ok.