Why do people ship logs to Logstash with NXLog and not Logstash itself?

Question

I read a blog post about setting up Logstash, Elasticsearch and Kibana, and the author suggested to use NXLog to ship logs from different machines. A typical distributed scenario presented in "The Logstash Book" shows how Logstash can be used in both shipping and indexing role. We are currently experimenting with Logstash and setting up it to ship logs to Elasticsearch was straighforward. So I wonder why people choose NXLog as a log shipper to Logstash instead of using Logstash on both ends.

Solution 1

The logstash-forwarder project, formerly known as "Lumberjack", explains it like so:

Resource Usage Concerns

Perceived Problems: Some users view logstash releases as "large" or have a generalized fear of Java.

Actual Problems: Logstash, for right now, runs with a footprint that is not friendly to underprovisioned systems such as EC2 micro instances; on other systems it is fine. This project will exist until that is resolved.

Transport Problems

Few log transport mechanisms provide security, low latency, and reliability.

The lumberjack protocol used by this project exists to provide a network protocol for transmission that is secure, low latency, low resource usage, and reliable.

A Logstash shipper instance isn't especially heavyweight, but if your machine only has 1-2GB of RAM it can be hard to casually allocate several hundred MB to another JVM instance.

Another consideration: what if your shipper nodes are running an OS that isn't supported by Logstash? Logstash can run on Windows, now, but it's still on the buggy side of things. I can't specifically vouch for NXLog in that area, but I gather it's a popular choice.

Solution 2

We've implemented shipping with RSyslog rather than LogStash due to a desire to keep java off of our hosts where possible. It makes the LogStash config on the loghost/filer a little more complex, but not having to keep up with Java security upgrades on front-end hosts is more of a pain than dealing with ripping logs apart on the central filer.