How to write a Java daemon

Question

This will be a network application that will always (or near as always as I can manage) be listening on a given port.

I'm fairly new to Java, and very new to non-web server side programming, so I'd like to get feedback from the community on my assumptions and preliminary plans.

I've read about jsvc ( http://commons.apache.org/daemon/jsvc.html ) and am currently operating on the assumption that this is the "best" way to write a daemon in java for a linux box (likely running centOS).

Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't? (I assume yes, but I'm not a very talented sys admin type)

This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP. However, I know that it's more "traditional" to write a daemon in C/C++. With modern Java, performing fairly uncomplicated tasks, am I likely to run into any major disadvantages?

What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?

Any other input would be greatly appreciated.

Solution 1

How to write a Java daemon that has 24/7 uptime...

We run a number of 24/365 applications on our Linux servers that just call the Java like the following -- no need for any C wrappers:

nohup java -D... -X... -jar something.jar ... < /dev/null > output.log 2>&1 &

That will put the jar running in the background (nohup ... &) with no input (< /dev/null) and the output (stdout and stderr) redirected to a logfile (> output.log 2>&1). We have distributed logging infrastructure but some console output (such as thread dumps) is still expected. These applications can run for months until we upgrade them.

Can nagios be configured to monitor whether or not my daemon is running, and to alert me or the sys admin when it isn't?

In terms of monitoring, there is much you can do. Nagios looks to have a JMX plugin for testing the information that jconsole displays. There are also a lot of native JMX logging and monitoring utilities out there. We have internal green/yellow/red indicators that can be pulled up using JMX and easily checked. I also have exported a simple JMX/HTTP service from each application to provide status information making it easy for 3rd party monitoring tools to detect failures.

This will be an SMPP client app (or ESME app I guess) which is why I've chosen Java as it seems to be a very mature platform for SMPP.

I assume you mean SMPP? If so then I see no reason why Java couldn't do a good job. Our applications do a wide variety of HTTP, UDP, SMTP, JDBC, LDAP, and other protocols in real time. We use Jgroups at lot which accomplishes a complete authenticated, encrypted, network stack in Java.

What's the best way to manage deployment of new builds? Just stop the daemon and replace the binary as quickly as possible and restart?

In terms of replacing a running binary on the fly, that it more complicated. We have VIPs up front and replace the binaries at our leisure. Our internal protocols are designed to failover. If you do not have a VIP then one thing to consider would be an orderly handoff. You boot the new jar and it talks to the application running the old jar when it is ready to bind to the new port. Then the old application unbinds and the new one binds immediately afterwards. Something like that.

Hope this helps.

Solution 2

If you really want to have something running non-stop on *nix, I recommend you have a look at daemontools.

There are some examples on how to do this here and here.

Basically svscan will spawn a process that monitors your java process from init, and every time it crashes, it gets restarted.

Comments

[Gray]

Java programs really don't "crash" @krico. They can get VM faults but that happens extremely rarely in my experience. They can, however, run out of memory and become unresponsive depending on memory leaks or other application issues. In that case I'm not sure like daemontools would help. I think a more active monitor which pings the application is in order.

[Alexander Pogrebnyak]

@Gray. While in development they don't, but, once in production, they do. to krico +1 daemontools code is ugly and old, but it still works.

[Gray]

Huh. I've run java applications in a cluster in production for years with maybe a handful of VM faults. Guess it depends on the JVM / OS.

[Aaron Bruce]

My question is vague enough that there's no "real" answer, but this post is awesome and helpful, so check-mark you get!

[Gray]

Just talked to my ops guy. Back of the napkin calculation is that we have accumulated 900+ months of 24/7 java application runtime in production (~18 servers, 4+ years) and he can remember 0 (zero) times that the VM faulted. I'm pretty sure I remember 1 time but it might have been a kernel panic. Again, we have had out-of-memory and other application faults but very few (if any) VM "crashes".

[krico]

@Gray: It is quite rare for VMs to fault, but daemontools is already helpfull just for those rare cases. My approach sometimes is to actually force a shutdown in case of an unrecoverable exception, and count on daemontools to bring the app back to life. Very simple example would be catch(OOM e){ System.exit(-1); }.

[Basit Anwer]

Can some one please describe the Linux script written ?