Using nxlog to ship logs in to logstash from Windows using om_ssl

Question

I have been looking at options to ship logs from Windows, I have already got logstash set up, and I currently ship logs from Linux (CentOS) servers to my ELK stack using the logstash-forwarder and ssl encryption.

For compliance reasons encryption is pretty much essential in this environment.

I was hoping to use logstash-forwarder in Windows as well, but after compiling with Go I ran in to issues shipping Event Logs, and I found some people saying that it wasn't possible because of file locking issues, which the logstash-forwarder people appear to be working on, but I can't really wait.

Anyway, eventually I found out that nxlog seems to be able to ship logs in an encrypted format using ssl, I've found a few posts about similar topics and while I've learned quite a bit about how to ship the logs across and how to set up nxlog, I am still at a loss with how to set up logstash to accept the logs so I can process them.

I've asked in the #nxlog and #logstash irc channels, and got some confirmation in #nxlog that it is possible, no further information on how it should be configured.

Anyway, I have taken the crt file created for use with my logstash-forwarder (I will create a new one if needed when I am happy that this will work) and renamed it with a pem extension, which I believe should work as it is readable in ASCII format. I have created the environment variable for %CERTDIR% and put my file in there, I have written the following config file for nxlog from the other articles I have read, I think it is right, but I am not 100% sure:

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Enable json extension
<Extension json>
    Module xm_json
</Extension>

# Nxlog internal logs
<Input internal>
    Module im_internal
    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>

# Windows Event Log
<Input eventlog>
  # Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog
  # Uncomment im_mseventlog for Windows XP/2000/2003
  # Module im_mseventlog
    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>

<Output sslout>
    Module          om_ssl
    Host            lumberjack.domain.com
    Port            5000
    CertFile        %CERTDIR%/logstash-forwarder.crt
    AllowUntrusted  TRUE
    OutputType      Binary
</Output>

<Route 1>
    Path     eventlog, internal => sslout
</Route>

What I want to know is what input format to use in logstash I have tried shipping logs in to a lumberjack input type (using the same config as my logstash-forwarders use) with the following config:

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

But when the service started I get the following in the nxlog logfiles:

2014-11-06 21:16:20 INFO connecting to lumberjack.domain.com:5000
2014-11-06 21:16:20 INFO nxlog-ce-2.8.1248 started
2014-11-06 21:16:21 INFO successfully connected to lumberjack.domain.com:5000
2014-11-06 21:16:22 INFO remote closed SSL socket
2014-11-06 21:16:22 INFO reconnecting in 1 seconds
2014-11-06 21:16:23 INFO connecting to lumberjack.domain.com:5000
2014-11-06 21:16:24 INFO reconnecting in 2 seconds
2014-11-06 21:16:24 ERROR couldn't connect to ssl socket on lumberjack.antmarketing.com:5000; No connection could be made because the target machine actively refused it.

When I turned the logging up to DEBUG I see a massive amount of logs flying through, but I think the key part is:

2014-11-06 21:20:18 ERROR Exception was caused by "rv" at om_ssl.c:532/io_err_handler(); [om_ssl.c:532/io_err_handler()] -; [om_ssl.c:501/om_ssl_connect()] couldn't connect to ssl socket on lumberjack.domain.com:5000; No connection could be made because the target machine actively refused it.

I assume this points to me using the wrong input method on logstash, but I guess it could also be an issue with my ssl certs or the way it is configured. I don't appear to be getting any logs on the logstash server being generated at the time I make the connection from my Windows machine.

Solution 1

Thanks to b0ti for the help, there were a number of issues, my logstash config was crashing the service, but I also had issues with my nxlog setup as well as my ssl certs being set up in the correct way.

I found this post about creating ssl certs, which covers the way they are set up really nicely for self signed certs for use as a web service.

The main thing wrong with nxlog was as b0ti pointed out I was trying to ship in binary when that will only work when shipping to nxlog server. I also noticed in the docs that the default for AllowUntrusted is false, so I just had to delete it once I was happy ssl was working.

<Output sslout>
    Module          om_ssl
    Host            lumberjack.domain.com
    Port            5001
    CAFile          %CERTDIR%\nxlog-ca.crt
    OutputType      LineBased
</Output>

Creating the CA key, and secure it as this needs to be kept secret (cd to /etc/pki/tls):

certtool --generate-privkey --bits 2048 --outfile private/nxlog-ca.key
chown logstash:logstash private/nxlog-ca.key
chmod 600 private/nxlog-ca.key

And then Self Signed CA Cert, which will need to be transferred to your clients:

certtool --generate-self-signed --load-privkey private/nxlog-ca.key --bits 2048 --template nxlog-ca-rules.cnf --outfile certs/nxlog-ca.crt

The cnf file is standard only with this option modified:

# Whether this is a CA certificate or not
ca

The logstash input method:

input {
  tcp {
    port => 5001
    type => "nxlogs"
    ssl_cacert => "/etc/pki/tls/certs/nxlog-ca.crt"
    ssl_cert => "/etc/pki/tls/certs/nxlog.crt"
    ssl_key => "/etc/pki/tls/private/nxlog.key"
    ssl_enable => true
    format => 'json'
  }
}

Generate the private key:

certtool --generate-privkey --bits 2048 --outfile private/nxlog.key
chown logstash:logstash private private/nxlog.key
chmod 600 private/nxlog.key

Generate the CSR (Certificate Signing Request):

certtool --generate-request --bits 2048 --load-privkey private/nxlog.key --outfile private/nxlog.csr

Sign the Cert with the CA private key

certtool --generate-certificate --bits 2048 --load-request private/nxlog.csr --outfile certs/nxlog.crt --load-ca-certificate certs/nxlog-ca.crt --load-ca-privkey private/nxlog-ca.key --template nxlog-rules.cnf

Again the only important part over the standard inputs for the cnf file will be:

# Whether this certificate will be used to encrypt data (needed
# in TLS RSA ciphersuites). Note that it is preferred to use different
# keys for encryption and signing.
encryption_key

# Whether this certificate will be used for a TLS client
tls_www_client

I've tested this and it works well, I just need to get the filters set up now

Solution 2

The binary data format is nxlog specific, you should only use it if you send to nxlog.

OutputType      Binary

If this doesn't help, check the logstash logs since it's the remote end (logstash) which closes the connection.