logstash + kibana + elasticsearch web issue

Question

I have 2 elasticsearch nodes in an AWS EC2 environment and I'd like to use logstash + kibana on a separate logging server. Logstash works as an indexer and is able to get the logs from the ES nodes (I'm still playing with how/what I need to log) The issue is with Kibana.

I'd like to be able to use Kibana on the logstash server to monitor the logs on my ES nodes. For example: ES nodes: 10.110.65.91/92 Logstash/kibana 10.110.65.93

I'd like to be able to hit [http://10.110.65.93/kibana-3.0.0milestone5/#/dashboard/file/default.json] and load Kibana. Right now it does not load using .93; though it does work on .91. The kibana config for .91 has elasticsearch: "http://"+window.location.hostname+":9200", which I know is not recommended but if I change it to the IP it does not load. Kibana config for .93 is elasticsearch: "http://"+10.110.65.91+":9200",

Security groups are open.

Am I missing something?

Solution

Suppose I put the following in config.js:

elasticsearch: "http://"+127.0.0.1+":9200",

I then load Kibana's dashboard, which comes up blank. I go to check my browser's JavaScript console, and find two errors:

Uncaught SyntaxError: Unexpected number (in config.js)
Uncaught ReferenceError: config is not defined (in app.js)

In a nutshell, Kibana fails to render because you gave it a config file that won't compile.

Suppose I try this, instead:

elasticsearch: "http://127.0.0.1:9200",

This compiles! Kibana renders! But then throws up a red error message:

Could not contact Elasticsearch at http://127.0.0.1:9200. Please ensure that Elasticsearch is reachable from your system.

Reason being: Kibana is resolving the IP address on the client machine, not the server; since my workstation isn't an ElasticSearch node, my browser cannot find an ES cluster at 127.0.0.1:9200.

Kibana doesn't know about Logstash, and doesn't care, just as long as the rendered page can connect to an ES cluster at the address given. For your setup, that seems to be http://10.110.65.91:9200 or http://10.110.65.92:9200. You could set up a proxy or firewall around those, whether for security, load balancing, or whatever.