Filebeat can't connect to logstash on another server

Question

Filebeat (11.11.11.11) can't connect to logstash (22.22.22.22) on another server (connection reset by peer). But filebeat services from other servers can do it.

Also I can connect from this server(11.11.11.11) using telnet to this port (telnet 22.22.22.22 5044).

I can see "failed":34816,"total":34816 in filebeat logs:

root@stage /var/log/filebeat # tail filebeat
2018-05-09T08:30:48.298Z    INFO    [monitoring]    log/log.go:124  Non-zero metrics in the last 30s    {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2390,"time":2392},"total":{"ticks":539690,"time":539696,"value":539690},"user":{"ticks":537300,"time":537304}},"info":{"ephemeral_id":"e0b9a5bb-4446-4ee1-a6f4-e2b0ccfb1677","uptime":{"ms":2340029}},"memstats":{"gc_next":205064832,"memory_alloc":146205312,"memory_total":29895393656}},"filebeat":{"harvester":{"open_files":39,"running":48}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":17,"failed":34816,"total":34816},"write":{"bytes":17029928,"errors":17}},"pipeline":{"clients":3,"events":{"active":4119,"retry":69632}}},"registrar":{"states":{"current":139}},"system":{"load":{"1":0.24,"15":0.43,"5":0.31,"norm":{"1":0.03,"15":0.0538,"5":0.0388}}}}}}
2018-05-09T08:30:48.391Z    ERROR   pipeline/output.go:92   Failed to publish events: write tcp 11.11.11.11:34626->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:49.943Z    ERROR   logstash/async.go:235   Failed to publish events caused by: write tcp 11.11.11.11:34632->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:50.943Z    ERROR   pipeline/output.go:92   Failed to publish events: write tcp 11.11.11.11:34632->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:51.383Z    ERROR   logstash/async.go:235   Failed to publish events caused by: write tcp 11.11.11.11:34634->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:52.383Z    ERROR   pipeline/output.go:92   Failed to publish events: write tcp 11.11.11.11:34634->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:53.942Z    ERROR   logstash/async.go:235   Failed to publish events caused by: write tcp 11.11.11.11:34636->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:54.942Z    ERROR   pipeline/output.go:92   Failed to publish events: write tcp 11.11.11.11:34636->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:55.407Z    ERROR   logstash/async.go:235   Failed to publish events caused by: write tcp 11.11.11.11:34654->22.22.22.22:5044: write: connection reset by peer
2018-05-09T08:30:56.407Z    ERROR   pipeline/output.go:92   Failed to publish events: write tcp 11.11.11.11:34654->22.22.22.22:5044: write: connection reset by peer

filebeat is already the newest version (6.2.4).
logstash is already the newest version (1:6.2.4-1).

How can it be solved or investigated?

Update:
If I switch off the logstash, the error changes:

ERROR   pipeline/output.go:74   Failed to connect:
dial tcp 22.22.22.22:5044: getsockopt: connection refused

Solution

not sure about this but for sure you can enable debug for having more information:

https://www.elastic.co/guide/en/beats/filebeat/current/enable-filebeat-debugging.html

i would also check the logstash and filebeat configurations:

Logstash:

in this case is receiving and should have an input configuration like the one below

input {
  beats {
    port => "5044"
    ssl  => false
  }
}

be sure logstash service has the permission to open a listen socket on the machine. check it with a

netstat -l

Filebeat:

check if filebeat is pointing to the correct logstash port

 output:
  logstash:
    hosts: ["22.22.22.22:5044"]

Comments

[Dmitry]

Thank you, ssl => false became necessary in the last update.