systemd service will start manually, but not at boot?

Question

I've written a systemd service to start groundcontrol on my Raspberry Pi.

[Unit]
Description=Groundcontrol status monitor

[Service]
ExecStart=/opt/groundcontrol/groundcontrol/start.sh
Type=forking

[Install]
WantedBy=multi-user.target  

I'm using the script because groundcontrol will not work properly unless started from the bin directory. Here is the script:

cd /opt/groundcontrol/groundcontrol
./groundcontrol &

This works perfectly when I start it manually, but when I start up my Pi and run systemctl it says it has failed. systemctl status groundcontrol.service prints

groundcontrol.service - Groundcontrol status monitor
   Loaded: loaded (/etc/systemd/system/groundcontrol.service; enabled)
   Active: failed (Result: exit-code) since Wed 1969-12-31 17:00:14 MST; 43 years 11 months ago
  Process: 111 ExecStart=/opt/groundcontrol/groundcontrol/start.sh (code=exited, status=0/SUCCESS)
 Main PID: 116 (code=exited, status=2)

Dec 31 17:00:11 waldo systemd[1]: Starting Groundcontrol status monitor...
Dec 31 17:00:12 waldo systemd[1]: Started Groundcontrol status monitor.
Dec 31 17:00:14 waldo systemd[1]: groundcontrol.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 31 17:00:14 waldo systemd[1]: Unit groundcontrol.service entered failed state. 

When I run it manually the status is

groundcontrol.service - Groundcontrol status monitor
   Loaded: loaded (/etc/systemd/system/groundcontrol.service; enabled)
   Active: active (running) since Thu 2013-12-26 15:38:02 MST; 1s ago
  Process: 296 ExecStart=/opt/groundcontrol/groundcontrol/start.sh (code=exited, status=0/SUCCESS)
 Main PID: 297 (groundcontrol)
   CGroup: /system.slice/groundcontrol.service
           `-297 ./groundcontrol

Dec 26 15:38:02 waldo systemd[1]: Started Groundcontrol status monitor. 

There was a System V init script provided with groundcontrol but I didn't know how to use it with systemd - is this possible, and will it work better than my service? If not, how can I fix this service? Thanks.

Solution

The logs say:

main process exited, code=exited, status=2/INVALIDARGUMENT

This means that the problem is with groundcontrol itself; it has returned the status 2 (some sort of failure).

Usually this specific problem – service failing on boot only – is caused by the service starting too early, that is, when it requires some hardware device that hasn't been discovered by the system yet. (Remember that on modern Linux systems, practically all devices are discovered dynamically; there is no point where it says "oh, I have all the devices, let's start init.")

The solution would be to rewrite the program to use libudev and dynamically add devices.

The workaround is to order the service after the specific device (I don't know which device it needs though, so I cannot give a full answer), or use Wants= + After= to pull in systemd-udev-settle.service which waits until udev has processed the first batch of "new device" events.

---

Also, why do you have a whole separate .sh script for the sole purpose of cd'ing to a directory? WorkingDirectory= + Type=simple would suffice. (The & is also unnecessary, as systemd itself – being a service manager – runs everything in "background".)

Comments

[Admin]

You need to understand groundcontrol and its requirements so that you can order it correctly in the boot process.

[Lily Hahn]

In the init script that came with it (it was for upstart so I couldn't use it), it had # Required-Start: $local_fs $network $remote_fs. Does this mean I would need to check for the local_fs, network, and remote_fs devices before it starts? Is there a way to do this in the init job, or do I just use After=?

[Lily Hahn]

I added Requires=systemd-udev-settle.service After=systemd-udev-settle.service and it seems like it started all the way up and then failed. Here's the status: pastebin.com/kYietiVA

[Pavel Šimerda]

Edit the question instead of using paste services. You can simply sed 's/^/ / the lines to make them appear in a preformatted block.