How to authenticate Logstash output to a secure Elasticsearch URL (version 5.6.5)

Question

I am using Logstash and Elasticsearch versions 5.6.5. So far used elasticsearch output with HTTP protocol and no authentication. Now Elasticsearch is being secured using basic authentication (user/password) and CA certified HTTPS URL. I don't have any control over the elasticsearch server. I just use it to output from Logstash.

Now when I try to configure the HTTPS URL of elasticsearch with basic authentication, it fails to create the pipeline.

Output Configuration

output { 
 elasticsearch {
   hosts => ["https://myeslasticsearch.server.io"]
   user => "esusername"
   password => "espassword"
   ssl => true
 }
}

Errors

 1. Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x50aa9200
 2. Pipeline aborted due to error {:exception=>#<URI::InvalidComponentError: bad component(expected user component):

How to fix this? I notice that there is a field called cacert which requires some PEM file. But I am not sure what to put there since the Elasticsearch server is using a CA certified SSL not a self-signed one.

Addtional question: I don't have any xpack installed. Is 'xpack' required to be purchased for HTTPS output to Elasticsearch from Logstash?

Solution

I found the root cause of the issue. There were three things to fix:

1. The logstash version I tested with was wrong 5.5.0. I downloaded the correct version to match with Elasticsearch Version 5.6.5.

2. The host I used was running on 443 port. When I didn't specify the port as below logstash appended 9200 with it, due to which the connection failed.

   hosts => ['https://my.es.server.com']

   Below configuration corrected the port used by logstash.

   hosts => ['https://my.es.server.com:443']

3. I was missing proxy connection settings.

   proxy => 'http://my.proxy.com:80'

Overall settings that worked.

output {
    elasticsearch {
       hosts => ['https://my.es.server.com:443']
       user => 'esusername'
       password => 'espassword'
       proxy => 'http://my.proxy:80'
       index => "my-index-%{+YYYY.MM.dd}"
    }
}

No need for 'ssl' field.

Also NO need for 'xpack' installation for this requirement.

Comments

[Annie]

What is that proxy for?

[Loganathan]

That is the network proxy through which logstash connects to elasticsearch, it is optional.