chkconfig bootup script failing, script runs good manually

Question

I recently purchased a vps and I attempted to create a startup/restart script to start up my java program. I am running CentOS 6. The script works if I execute it manually with this command:

service policyserver start

But fails to run automatically, creates a file in the /tmp/test.log with the following error:

Starting Policy Server: failed

Here is the startup script inside /etc/init.d/

#!/bin/bash
#
#policyserver:          Startup script for CamSpark Policy Server Application.
#
#chkconfig: 3 80 05
#description:      Startup script for Cam Spark Policy Server Application.

CAMSPARK_POLICY_HOME=/tools/camsparkserver/Policy;
export CAMSPARK_POLICY_HOME

start() {
        echo -n "Starting Policy Server: "
        $CAMSPARK_POLICY_HOME/Policy.sh start >> /tmp/test.log
        sleep 2
        echo "done"
}

stop() {
        echo -n "Stopping Policy Server: "
        $CAMSPARK_POLICY_HOME/Policy.sh stop
        echo "done"
}

# See how we were called.
case "$1" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                stop
                start
                ;;
        *)
                echo $"Usage: policyserver {start|stop|restart}"
                exit
esac

Here is the main script (can't move this script)

#!/bin/bash
#
# chkconfig: 345 99 05 
# description: Java deamon script
#
# A non-SUSE Linux start/stop script for Java daemons.
#
# Set this to your Java installation
JAVA_HOME=/usr/java/latest

scriptFile=$(readlink -fn $(type -p $0))                   # the absolute, dereferenced path of this script file
scriptDir=$(dirname $scriptFile)                           # absolute path of the script directory
serviceNameLo="policyserver"                                  # service name with the first letter in lowercase
serviceName="PolicyServer"                                    # service name
serviceUser="root"                                      # OS user name for the service
serviceGroup="root"                                    # OS group name for the service
applDir="/tools/camsparkserver/Policy"                          # home directory of the service application
serviceUserHome="/home/$serviceUser"                       # home directory of the service user
serviceLogFile="/var/log/$serviceNameLo.log"               # log file for StdOut/StdErr
maxShutdownTime=15                                         # maximum number of seconds to wait for the daemon to terminate normally
pidFile="/var/run/$serviceNameLo.pid"                      # name of PID file (PID = process ID number)
javaCommand="java"                                         # name of the Java launcher without the path
javaExe="$javaCommand"                      # file name of the Java application launcher executable
javaArgs="PolicyServer"                     # arguments for Java launcher
javaCommandLine="$javaExe $javaArgs"                       # command line to start the Java service application
javaCommandLineKeyword="PolicyServer"                     # a keyword that occurs on the commandline, used to detect an already running service process and to distinguish it from others

# Makes the file $1 writable by the group $serviceGroup.
function makeFileWritable {
   local filename="$1"
   touch $filename || return 1
   chgrp $serviceGroup $filename || return 1
   chmod g+w $filename || return 1
   return 0; }

# Returns 0 if the process with PID $1 is running.
function checkProcessIsRunning {
   local pid="$1"
   if [ -z "$pid" -o "$pid" == " " ]; then return 1; fi
   if [ ! -e /proc/$pid ]; then return 1; fi
   return 0; }

# Returns 0 if the process with PID $1 is our Java service process.
function checkProcessIsOurService {
   local pid="$1"
   if [ "$(ps -p $pid --no-headers -o comm)" != "$javaCommand" ]; then return 1; fi
   grep -q --binary -F "$javaCommandLineKeyword" /proc/$pid/cmdline
   if [ $? -ne 0 ]; then return 1; fi
   return 0; }

# Returns 0 when the service is running and sets the variable $pid to the PID.
function getServicePID {
   if [ ! -f $pidFile ]; then return 1; fi
   pid="$(<$pidFile)"
   checkProcessIsRunning $pid || return 1
   checkProcessIsOurService $pid || return 1
   return 0; }

function startServiceProcess {
   cd $applDir || return 1
   rm -f $pidFile
   makeFileWritable $pidFile || return 1
   makeFileWritable $serviceLogFile || return 1
   cmd="nohup $javaCommandLine >>$serviceLogFile 2>&1 & echo \$! >$pidFile"
   # Don't forget to add -H so the HOME environment variable will be set correctly.
   sudo -u $serviceUser -H $SHELL -c "$cmd" || return 1
   sleep 0.1
   pid="$(<$pidFile)"
   if checkProcessIsRunning $pid; then :; else
      echo -ne "\n$serviceName start failed, see logfile."
      return 1
   fi
   return 0; }

function stopServiceProcess {
   kill $pid || return 1
   for ((i=0; i<maxShutdownTime*10; i++)); do
      checkProcessIsRunning $pid
      if [ $? -ne 0 ]; then
         rm -f $pidFile
         return 0
         fi
      sleep 0.1
      done
   echo -e "\n$serviceName did not terminate within $maxShutdownTime seconds, sending SIGKILL..."
   kill -s KILL $pid || return 1
   local killWaitTime=15
   for ((i=0; i<killWaitTime*10; i++)); do
      checkProcessIsRunning $pid
      if [ $? -ne 0 ]; then
         rm -f $pidFile
         return 0
         fi
      sleep 0.1
      done
   echo "Error: $serviceName could not be stopped within $maxShutdownTime+$killWaitTime seconds!"
   return 1; }

function startService {
   getServicePID
   if [ $? -eq 0 ]; then echo -n "$serviceName is already running"; RETVAL=0; return 0; fi
   echo -n "Starting $serviceName   "
   startServiceProcess
   if [ $? -ne 0 ]; then RETVAL=1; echo "failed"; return 1; fi
   echo "started PID=$pid"
   RETVAL=0
   return 0; }

function stopService {
   getServicePID
   if [ $? -ne 0 ]; then echo -n "$serviceName is not running"; RETVAL=0; echo ""; return 0; fi
   echo -n "Stopping $serviceName   "
   stopServiceProcess
   if [ $? -ne 0 ]; then RETVAL=1; echo "failed"; return 1; fi
   echo "stopped PID=$pid"
   RETVAL=0
   return 0; }

function checkServiceStatus {
   echo -n "Checking for $serviceName:   "
   if getServicePID; then
    echo "running PID=$pid"
    RETVAL=0
   else
    echo "stopped"
    RETVAL=3
   fi
   return 0; }

function main {
   RETVAL=0
   case "$1" in
      start)                                               # starts the Java program as a Linux service
         startService
         ;;
      stop)                                                # stops the Java program service
         stopService
         ;;
      restart)                                             # stops and restarts the service
         stopService && startService
         ;;
      status)                                              # displays the service status
         checkServiceStatus
         ;;
      *)
         echo "Usage: $0 {start|stop|restart|status}"
         exit 1
         ;;
      esac
   exit $RETVAL
}

main $1

Does anyone know what could be the problem?

EDIT: ran one of the answers posted here.

Here is what I get when I just run that command

[root@camspark Policy]# service policyserver stop
Stopping Policy Server: Stopping PolicyServer   stopped PID=683
done
[root@camspark Policy]# bash -x service policyserver start |& tee policyserver_startup.log
+ . /etc/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
++ '[' -f /etc/sysconfig/i18n -a -z '' -a -z '' ']'
++ . /etc/profile.d/lang.sh
++ unset LANGSH_SOURCED
++ '[' -z '' ']'
++ '[' -f /etc/sysconfig/init ']'
++ . /etc/sysconfig/init
+++ BOOTUP=color
+++ RES_COL=60
+++ MOVE_TO_COL='echo -en \033[60G'
+++ SETCOLOR_SUCCESS='echo -en \033[0;32m'
+++ SETCOLOR_FAILURE='echo -en \033[0;31m'
+++ SETCOLOR_WARNING='echo -en \033[0;33m'
+++ SETCOLOR_NORMAL='echo -en \033[0;39m'
+++ PROMPT=no
+++ AUTOSWAP=no
+++ ACTIVE_CONSOLES='/dev/tty[1-6]'
+++ SINGLE=/sbin/sushell
++ '[' pty = serial ']'
++ __sed_discard_ignored_files='/\(~\|\.bak\|\.orig\|\.rpmnew\|\.rpmorig\|\.rpmsave\)$/d'
++ basename service
+ VERSION='service ver. 0.91'
++ basename service
+ USAGE='Usage: service < option > | --status-all | [ service_name [ command | --full-restart ] ]'
+ SERVICE=
+ SERVICEDIR=/etc/init.d
+ OPTIONS=
+ '[' 2 -eq 0 ']'
+ cd /
+ '[' 2 -gt 0 ']'
+ case "${1}" in
+ '[' -z '' -a 2 -eq 1 -a policyserver = --status-all ']'
+ '[' 2 -eq 2 -a start = --full-restart ']'
+ '[' -z '' ']'
+ SERVICE=policyserver
+ shift
+ '[' 1 -gt 0 ']'
+ case "${1}" in
+ '[' -z policyserver -a 1 -eq 1 -a start = --status-all ']'
+ '[' 1 -eq 2 -a '' = --full-restart ']'
+ '[' -z policyserver ']'
+ OPTIONS=' start'
+ shift
+ '[' 0 -gt 0 ']'
+ '[' -f /etc/init.d/policyserver ']'
+ env -i PATH=/sbin:/usr/sbin:/bin:/usr/bin TERM=xterm /etc/init.d/policyserver start
Starting Policy Server: done
[root@camspark Policy]#

Here is what is inside that log file

[root@camspark Policy]# head policyserver_startup.log
+ . /etc/init.d/functions
++ TEXTDOMAIN=initscripts
++ umask 022
++ PATH=/sbin:/usr/sbin:/bin:/usr/bin
++ export PATH
++ '[' -z '' ']'
++ COLUMNS=80
++ '[' -z '' ']'
+++ /sbin/consoletype
++ CONSOLETYPE=pty
[root@camspark Policy]#

Solution

You can debug an issue like this by running the start script in a more verbose manner.

$ bash -x service policyserver start

You might want to capture this output to a log file too.

$ bash -x service policyserver start |& tee policyserver_startup.log

You can then go through the log file and see which command is failing.

Comments

[schaiba]

I'd replace this line $CAMSPARK_POLICY_HOME/Policy.sh start >> /tmp/test.log with this : $CAMSPARK_POLICY_HOME/Policy.sh start >> /tmp/test.log 2>&1 and check what you see in /tmp/test.log .

[SSpoke]

schaiba that told me more descriptive error but I still don't know what I must change in the boot script [root@camspark tmp]# head test.log Starting PolicyServer sudo: sorry, you must have a tty to run sudo failed

[SSpoke]

Okay after some reason I figured out disabling that tty will cause a security vulnerability Only sudo I found in the script says this command above it # Don't forget to add -H so the HOME environment variable will be set correctly. What does that comment mean exactly what should I do?

[SSpoke]

the file policyserver_startup.log has nothing in there that seems useful pastebin.com/TFZyaGzx but when I run that command which generates that new log file it creates a bunch of text before it makes the log file which is here pastebin.com/7dyNk7Br