Kibana 3 Milestone 4 and Graphite Integration

Question

I am having difficulties understanding integration of Graphite and Kibana 3 to monitor logs and system vitals. I am referring to figure in Log management system described here.

1. Considering the new features in Kibana 3 Milestone 4, can we collect system vitals and store it directly into elastic search instead of graphite and use a single kibana dashboard (What could be right choice to implement in a distributed system where emphasis is on performance and low memory foot print)?
2. Why must we use StatsD and graphite, when count and simple statistics are now supported by kibana - Elasticsearch combination?
3. In case, we decide to use both graphite and kibana, How do we integrate it into a single Dashboard?
4. Is there a tutorial to integrate Dashboards (kibana and graphitos/graph explorer/orion/pencil)?

Thanks in advance.

Solution

Why statsd-graphite:

1. Statsd and Graphite can help you visualize anything, not just logs and system vitals. It is very straightforward with the statsd-graphite stack, to measure say- number of users that hovered on the left bottom of your site for more than 10 seconds.

2. Because there is no in-between logging involved, the scalability that graphite provides is unparalleled from an IO point of view. Also consider the fact that statsd talks UDP, so collecting 300K metrics per minute is a breeze.

3. You don't have to log something in order to see it.

Integration:

As clearly shown in the architectural diagram you shared, you can filter the stats that you want to visualize, have them forwarded to statsd. This is in parallel with kibana visualizing directly from logstash-elasticsearch. Going redundant with data is a easier approach if you want to view both Graphite and Kibana data over Graphite, since the webapp would not query elasticsearch directly.

Vimeo's Graph Explorer is something you might want to look into. It queries elasticsearch.

---

Updates:

Not that Logstash catn't do it, but it isn't 'designed' for that role, whereas statsd et al, are.

I have been wondering if we have a simpler query language.

The inherent scheme of organization in graphite is tree-like and hence the searches do-not/ can-not yeild results from a different subtree. This makes it not-so-suitable for cross-dimensional searches. GE is the simplest, given you want the power.

Graph Explorer's flow-

Graph Explorer addresses this by adding tags to the metrics and integrating it with elasticsearch. So what GE actually does is that-

1. One time- It connects to your Graphite front-end, makes API calls to retrieve all metrics.

2. It then 'converts' the old style proto 1 metrics (A.B.C) into tag-based proto 2 metrics (host=A.app=B.username=C).

3. This is then exported to ES which maintains an index.

4. When you query GE front-end, it connects to ES to understand what you want.

5. GE then queries the Graphite-API, and delivers the results in GE front-end.

Moreover, does graph explorer assume we are using diamond for collection?

No.

How does it compare to pencil, orion and graphiti?

These are on-surface optimizations to visualization. They-

1. change the look and feel of the graphs.

2. make querying the API easier.

3. allow a better monitoring flow.

They DO NOT change the way you store or search the information. GE, embeds itself 'deeper' into the metric data and hence has a real edge over how you query metrics. (Cross dimensional search)

Heads up-

GE's metric-importing plug-in is far from perfect. It successfully imported 300 out of my 1000 metrics. It is also heavier to render, and the front-end eats more NW (because of the hoverable, zoomable features).

Update-

Grafana is out.

Comments

[Vamsi Krishna]

Hi, Thanks for the reply, I have checked graph explorer. I have been wondering if we have a simpler query language. Moreover, does graph explorer assume we are using diamond for collection? How does it compare to pencil, orion and graphiti? Could you kindly share your experience with dashboards.

[Vamsi Krishna]

Moreover, Logstash could also listen to UDP sockets, parse and send it over.

[erbdex]

stackoverflow.com/questions/18651477/…

[Vamsi Krishna]

Thanks again, as you mentioned regarding importing only 30% of metrics and regarding the heavy front end [checked the vimeo video tutorial] (vimeo.com/67080202). Did you write your own custom plugin? How did you solve your problem of visualizing 1000 metrics?

[erbdex]

Didn't actually. At a 10s granularity, viewing graphs of my ~50 servers for past 7 days meant 30MB of traffic EVERY CALL. This was highly unacceptable. i stick to graphite dashboards.

[Vamsi Krishna]

Assuming that graph explorer does not work in case of scaling up, and I would want to implement a single dashboard, where I could visualize my logs [nginx,apache] {maps of IP addresses, pie charts regarding logs etc using Kibana3} and performance metrics of Host/Cpu/ sys health, counters etc for servers/switches/routers how would i proceed in case of distributed system assuming I am going to use statsD and Graphite combo for metrices?

[Vamsi Krishna]

Currently, I installed Logstash, elastic search and Kibana on VM1, and StatsD and graphite on VM2.

[erbdex]

Send everything to Graphite. For this, send all logs to logstash and then you have logstash-graphite plugins that will forward the data to graphite. Use collectd to send the host metrics to graphite. This way, all your data is on Graphite.

[Vamsi Krishna]

Hi, Incase say we are monitoring HTTP 2xx, 4xx and 5xx messages from webserver. Due to some reason, say we doubt that a malicious code ran on our servers and causing service to be not available, How would we do a drill down to the root cause? In this case: on graphite dashboard, we notice a huge amount of HTTP 401 spiking at 3 AM. how do we drill down to root cause of problem with available metrices?

[erbdex]

In your metric syntax, you can incorporate something like- HTTP.ERR.5XX.THREADS.count HTTP.ERR.5XX.CONNECTIONS.count etc. Basically, you can send anything that your logs can filter out, at the filtering step and accordingly send the metrics so that you can localize the issue.

[erbdex]

If you have any further questions, perhaps you can ask them separately. Let's not clutter the information here.

[erbdex]

Did i answer the question you asked?

[Vamsi Krishna]

Thanks, yes have answered my question.

[Vamsi Krishna]

Hi, how do we create default scripted dashboards in Graph Explorer, i.e where we have default set of graphs.. from set of sources. Assume that we use Diamond for collecting metrics from different servers.

[erbdex]

What's your email id?

[erbdex]

Now delete the comment. You don't want spam. :)

[Vamsi Krishna]

thanks :), got the graph explorer dashboard working. I was just surprised that it does not recognize my diamond metrics, it uses default plugin. I must say the example in graphexplorer/assets/embedded_example.html is a simple example of a default dashabord.. but its too rudimentary.