Using setup.py to install python project as a systemd service

Question

I have a python project and I want to be able to install it using something like python setup.py install so that the installation automatically creates a systemd service.

I'm having some trouble, most probably setting the paths or imports correctly.

My environment:

• Ubuntu 15.04
• Python 2.7 (although it would be great to make it work in py3 too).

Project Structure:

+ top-folder
  + super_project
    + folder1
      __init__.py
      file1.py
    + folder2
      __init__.py
      file2.py
    __init__.py
    main.py
  setup.py
  setup.cfg

setup.py:

from setuptools.command.install import install
from setuptools import setup, find_packages
import subprocess
import os


class CustomInstallCommand(install):

  def run(self):
    install.run(self)
    current_dir_path = os.path.dirname(os.path.realpath(__file__))
    create_service_script_path = os.path.join(current_dir_path, 'super_project', 'install_scripts', 'create_service.sh')
    subprocess.check_output([create_service_script_path])

setup(
  name='super-project',
  author='Myself',
  version='0.0.1',
  description='My Description',
  packages=find_packages(exclude=['contrib', 'docs']),
  # this will create the /usr/local/bin/super-project entrypoint script
  entry_points={
    'console_scripts': [
      'super-project = super_project.main:main'
    ]
  },
  cmdclass={'install': CustomInstallCommand}
)

main.py

from super_project.folder1.file1 import Class1
from super_project.folder2.file2 import Class2
import logging


def main():
  logging.info('Executing super-project...')
  (...)
  logging.info('super-project execution finished.')

if __name__ == '__main__':
  main()

setup.cfg

[bdist_wheel]
universal=1

create_service.sh (more or less):

SYSTEMD_SCRIPT_DIR=$( cd  $(dirname "${BASH_SOURCE:=$0}") && pwd)
cp -f "$SYSTEMD_SCRIPT_DIR/super-project.service" /lib/systemd/system
chown root:root /lib/systemd/system/super-project.service

systemctl daemon-reload
systemctl enable super-project.service

super-project.service

[Unit]
Description=Super Description

[Service]
Type=simple
ExecStart=/usr/local/bin/super-service
Restart=always

[Install]
WantedBy=multi-user.target

The installation of the package generates the following output:

$ sudo python setup.py install --record files.txt
running install
running build
running build_py
copying super_project/main.py - build/lib.linux-x86_64-2.7/super_project
running install_lib
copying build/lib.linux-x86_64-2.7/super_project/__init__.py - /usr/local/lib/python2.7/dist-packages/super_project
copying build/lib.linux-x86_64-2.7/super_project/main.py - /usr/local/lib/python2.7/dist-packages/super_project
copying build/lib.linux-x86_64-2.7/super_project/db/__init__.py - /usr/local/lib/python2.7/dist-packages/super_project/db
copying build/lib.linux-x86_64-2.7/super_project/db/db_gateway.py - /usr/local/lib/python2.7/dist-packages/super_project/db
(...)
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/__init__.py to
__init__.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/main.py to
main.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/db/__init__.py to
__init__.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/super_project/db/db_gateway.py
to db_gateway.pyc
(...)
running install_egg_info
running egg_info
writing requirements to super_project.egg-info/requires.txt
writing super_project.egg-info/PKG-INFO
writing top-level names to super_project.egg-info/top_level.txt
writing dependency_links to super_project.egg-info/dependency_links.txt
writing entry points to super_project.egg-info/entry_points.txt
reading manifest file 'super_project.egg-info/SOURCES.txt'
writing manifest file 'super_project.egg-info/SOURCES.txt'
Copying super_project.egg-info to /usr/local/lib/python2.7/dist-packages/super_project-0.0.1.egg-info
running install_scripts
Installing ai-scenario-qa script to /usr/local/bin
writing list of installed files to 'files.txt'

The super-project file is created in /usr/local/bin:

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'super-project==0.0.1','console_scripts','super-project'
__requires__ = 'super-project==0.0.1'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('super-project==0.0.1', 'console_scripts', 'super-project')()
    )

The installation seems successful, although:

$ systemctl status super-project.service
● super-project.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

The error I can see in /var/log/syslog:

 Feb 16 20:48:34  systemd[1]: Starting  Super Description...
 Feb 16 20:48:34  super-project[22517]: Traceback (most recent call last):
 Feb 16 20:48:34  super-project[22517]: File "/usr/local/bin/super-project", line 9, in <module
 Feb 16 20:48:34  super-project[22517]: load_entry_point('super-project==0.0.1', 'console_scripts', 'super-project')()
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 521, in load_entry_point
 Feb 16 20:48:34  super-project[22517]: return get_distribution(dist).load_entry_point(group, name)
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2632, in load_entry_point
 Feb 16 20:48:34  super-project[22517]: return ep.load()
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2312, in load
 Feb 16 20:48:34  super-project[22517]: return self.resolve()
 Feb 16 20:48:34  super-project[22517]: File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 2318, in resolve
 Feb 16 20:48:34  super-project[22517]: module = __import__(self.module_name, fromlist=['__name__'], level=0)
 Feb 16 20:48:34  super-project[22517]: ImportError: No module named main
 Feb 16 20:48:34  systemd[1]: super-project.service: main process exited, code=exited, status=1/FLURE
 Feb 16 20:48:34  systemd[1]: Unit super-project.service entered fled state.
 Feb 16 20:48:34  systemd[1]: super-project.service failed.
 Feb 16 20:48:34  systemd[1]: super-project.service holdoff time over, scheduling restart.
 Feb 16 20:48:34  systemd[1]: start request repeated too quickly for super-project.service
 Feb 16 20:48:34  systemd[1]: Failed to start Super Description.
 Feb 16 20:48:34  systemd[1]: Unit super-project.service entered fled state.
 Feb 16 20:48:34  systemd[1]: super-project.service failed.

As can be seen, the module main cannot be found. This is the main problem.

When changing code/conf, I remove the super-project/service as follows:

$ sudo systemctl disable super-project.service
$ sudo rm -f /lib/systemd/system/super-project.service
$ sudo systemctl daemon-reload
$ su
# cat files.txt | xargs rm -r

On the other hand:

• If I execute $ super-project from /usr/local/bin/, the script starts correctly (no import exception) but the configuration files cannot be read (most probably because of relative/absolute path issues).
• If I execute $ super-project from top-folder (folder containing the project code/files) the script runs perfectly

What am I missing? I've spend a lot of time searching what the issue might be. It seems that the package is correctly set up in the dist-packages directory and all the service files are correctly created once the setup is executed.

I've read things about using from __future__ import absolute_import, but I'm not sure if I have to add that to my main.py (it does not work) or to all the files in my project.

Solution

You get an ImportError, because the module in question is not in sys.path or not accessible, because of some file system permissions.
Here's a script to check file system permissions of a given distribution, group and name.

chk_perm.py

from pkg_resources import get_distribution
import os
import sys

dist, group, name = sys.argv[1:]
dist = get_distribution(dist)
location = dist.location
einfo = dist.get_entry_info(group, name)
if not einfo:
    print('No such group "{}" or name "{}"'.format(group, name))
    sys.exit(1)
m_name = einfo.module_name
path = format(os.path.join(location, *m_name.split('.')))
path = path if os.access(path, os.F_OK) else '{}.py'.format(path)
print('If path "{}" exists: {}'.format(path, os.access(path, os.F_OK) if path.endswith('.py') else True))
print('If path "{}" readable: {}'.format(path, os.access(path, os.R_OK)))

Test;

$ python chk_perm.py setuptools console_scripts easy_install
If path "lib/python2.7/site-packages/setuptools/command/easy_install.py" exists: True
If path "lib/python2.7/site-packages/setuptools/command/easy_install.py" readable: True

$ foo
Traceback (most recent call last):
  File "bin/foo", line 9, in <module>
    load_entry_point('mypkg==0.0.4', 'console_scripts', 'foo')()
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 549, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 2542, in load_entry_point
    return ep.load()
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 2202, in load
    return self.resolve()
  File "lib/python2.7/site-packages/pkg_resources/__init__.py", line 2208, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ImportError: No module named main

$ python chk_perm.py mypkg console_scripts foo
If path "lib/python2.7/site-packages/pkg/main.py" exists: True
If path "lib/python2.7/site-packages/pkg/main.py" readable: False

$ ls -l lib/python2.7/site-packages/pkg/main.py 
-rw-rw---- 1 root root 104 Mar  6 22:52 lib/python2.7/site-packages/pkg/main.py

$ sudo chmod o+r lib/python2.7/site-packages/pkg/main.py
$ ls -l lib/python2.7/site-packages/pkg/main.py 
-rw-rw-r-- 1 root root 104 Mar  6 22:52 lib/python2.7/site-packages/pkg/main.py

$ python chk_perm.py mypkg console_scripts foo
If path "lib/python2.7/site-packages/pkg/main.py" exists: True
If path "lib/python2.7/site-packages/pkg/main.py" readable: True

$ foo
App is running