What is "Signal 15 received"

Question

What might cause a C, MPI program using a library called SUNDIALS/CVODE (a numerical ODE solver) running on a Gentoo Linux cluster to give me repeated Signal 15 received.?

Is that code being issued by MPI, Sundials, Linux, C or who?

Note that I am pretty much a beginner with the following technologies: C, MPI, SUNDIALS/CVODE, and Linux.

I can find nothing that seems related by googling the message. I don't even know where to begin to look. (This is one of those questions where "anything helps" is to be taken quite literally.)

(As an aside/afterthought why doesn't Chrome's dictionary recognize the word "googling"?).

Solution

This indicates the linux has delivered a SIGTERM to your process. This is usually at the request of some other process (via kill()) but could also be sent by your process to itself (using raise()). This signal requests an orderly shutdown of your process.

If you need a quick cheatsheet of signal numbers, open a bash shell and:

$ kill -l
 1) SIGHUP   2) SIGINT   3) SIGQUIT  4) SIGILL
 5) SIGTRAP  6) SIGABRT  7) SIGBUS   8) SIGFPE
 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2
13) SIGPIPE 14) SIGALRM 15) SIGTERM 16) SIGSTKFLT
17) SIGCHLD 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
21) SIGTTIN 22) SIGTTOU 23) SIGURG  24) SIGXCPU
25) SIGXFSZ 26) SIGVTALRM   27) SIGPROF 28) SIGWINCH
29) SIGIO   30) SIGPWR  31) SIGSYS  34) SIGRTMIN
35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3  38) SIGRTMIN+4
39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12
47) SIGRTMIN+13 48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14
51) SIGRTMAX-13 52) SIGRTMAX-12 53) SIGRTMAX-11 54) SIGRTMAX-10
55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7  58) SIGRTMAX-6
59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX    

You can determine the sender by using an appropriate signal handler like:

#include <signal.h>
#include <stdio.h>
#include <stdlib.h>

void sigterm_handler(int signal, siginfo_t *info, void *_unused)
{
  fprintf(stderr, "Received SIGTERM from process with pid = %u\n",
      info->si_pid);
  exit(0);
}

int main (void)
{
  struct sigaction action = {
    .sa_handler = NULL,
    .sa_sigaction = sigterm_handler,
    .sa_mask = 0,
    .sa_flags = SA_SIGINFO,
    .sa_restorer = NULL
  };

  sigaction(SIGTERM, &action, NULL);
  sleep(60);

  return 0;
}

Notice that the signal handler also includes a call to exit(). It's also possible for your program to continue to execute by ignoring the signal, but this isn't recommended in general (if it's a user doing it there's a good chance it will be followed by a SIGKILL if your process doesn't exit, and you lost your opportunity to do any cleanup then).