Systemd State 'stop-sigterm' timed out

Question

I have a simple java server + a couple of scripts that I want to run and keep alive using systemd (Ubuntu 16.04) I'm new to systemd The service keeps restarting with this log

Nov 16 10:20:12 systemd[1]: app.service: Unit entered failed state.
Nov 16 10:20:12 systemd[1]: app.service: Failed with result 'resources'.
Nov 16 10:29:25 systemd[1]: app.service: State 'stop-sigterm' timed out. Killing.
Nov 16 10:29:25  systemd[1]: app.service: Unit entered failed state.
Nov 16 10:29:25  systemd[1]: app.service: Failed with result 'timeout'.
Nov 16 10:31:06 systemd[1]: app.service: Service hold-off time over, scheduling restart.

I have 2 simple bash script to run/control the server;

Here is my service defination app.service

   [Unit]
    Description=java server
    [Service]
    ExecStart=-/home/deploy/server/serverctl.sh start
    ExecStop=-/home/deploy/server/serverctl.sh stop
    ExecReload=-/home/deploy/server/serverctl.sh restart
    SyslogIdentifier=my-app
    User=deploy
    Restart=always
    RestartSec=100
    Type=simple
    [Install]
    WantedBy=multi-user.target

My start.sh

#!/bin/bash
cd "$(dirname "$0")"
java -mx500m -cp lib/* mylcass  -port 8080 -arg val > server.log 2>&1 & echo $!

it runs the server and returns the PID

I have a control script to start/stop/status restart the server, which works fine

#!/bin/bash

PID_FILE='/tmp/myserver.pid'

# ***********************************************
start() {
  PID=`/path/to/server-start.sh`
}

case "$1" in
start)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep -w ${PID} | grep -v grep`" ]; then
            start
        else
            exit 0
        fi
    else
        start
    fi

    if [ -z $PID ]; then
        exit 3
    else
        echo $PID > $PID_FILE
        exit 0
    fi
;;

status)
    echo "status"   
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep -w ${PID} | grep -v grep`" ]; then
            echo "Not running (process dead but pidfile exists)"
            exit 1
        else
            echo "Running [$PID]"
            exit 0
        fi
    else
        echo "Not running"
        exit 3
    fi
;;

stop)
    if [ -f $PID_FILE ]; then
        PID=`cat $PID_FILE`
        if [ -z "`ps axf | grep -w ${PID} | grep -v grep`" ]; then
            exit 1
        else
            PID=`cat $PID_FILE`
            kill -HUP $PID
            rm -f $PID_FILE
            exit 0
        fi
    else
        exit 3
    fi
;;

restart)
    $0 stop
    $0 start
;;

*)
    echo "Usage: $0 {status|start|stop|restart}"
    exit 1
esac

Any ideas? How can i get systemd to ensure the server is up and running ?

Solution 1

First, please stop using your "server control script". That is systemd's job. The init system already keeps track of service PIDs and provides start/stop commands.

Also, don't use the - flag in ExecStart unless you really know that it's necessary. It tells systemd to ignore startup failures, and surely you would want to know when the service failed.

Finally, try to avoid myapp & in the startup scripts. (Init doesn't need that – services already run in background by definition.) If you must use it, then Type=forking would be more correct.

(Setting the correct Type= tells systemd what to expect, i.e. when to consider the server as "starting" vs "started" vs "finished". Type=simple means the initial process never 'backgrounds' itself; Type=forking is the opposite.)

With that, the following should work better:

app.service

[Unit]
Description=java server

[Service]
Type=simple
User=deploy
ExecStart=/home/deploy/server/start.sh
SyslogIdentifier=my-app
Restart=always
RestartSec=100

[Install]
WantedBy=multi-user.target

start.sh

#!/bin/sh
cd "$(dirname "$0")"
exec java -mx500m -cp lib/* mylcass -port 8080 -arg val

Solution 2

If your start.sh is really that simple, you could also just not use it at all.

IMHO, this is actually the prefered way:

[Unit]
Description=My Java App

[Service]
Type=simple
User=deploy
WorkingDirectory=/home/deploy/server
ExecStart=/usr/bin/java -mx500m -cp lib/* mylcass -port 8080 -arg val
SyslogIdentifier=my-app
Restart=always
RestartSec=100

[Install]
WantedBy=multi-user.target

No need to replicate the functionality that Systemd gives to you for free.

Note, this does not replicate the > server.log functionality of your start.sh script. Instead, the output is handled by journald which is much better, imho.

If you need user configuration, either read the configuration files directly in your program or use environment variables with systemd. See Environment= and EnvironmentFile= service options.

You can even convert those environment variables into arguments passed to the command with ${FOO}.

Use override files in app.service.d/* to keep the ENV variables (or other configuration) separate.

Comments

[macarthy]

Thanks that is very useful, I thought I might not need the control script. If I wanted a stop/start functionaltity can I just specify a PIDfile and kill $pid in ExecStop ?

[macarthy]

I was using the "-" in the exec command because my control script was echoing more than the return code before :-)

[user1686]

Um, output (from echo) is completely different from the exit code. Your script wasn't "returning" the PID – it was outputting the PID to syslog, but returning 0, the usual 'success' exit code.