unable to configure grafana with graphite

Question

I am using Nginx to serve both graphite and grafana (they are all running on the same server - not my desktop). I am able to access graphite via Nginx. However, grafana cannot seem to connect to graphite (error: Graphite HTTP Request Error). I have copied the nginx config below for grafana - any ideas on fixing this will be appreciated. The request URL that fails in the browser is this (accessible if I access it directly in the browser):

**http://xxx.xxx.xxx.xxx:8080/render**

Nginx default

server { 
        listen 85;  ##listen [::]:85; #ipv6only=on;
        server_name grafana;
        root /home/xxxx/grafana-1.5.3/;
        index index.html index.htm;
        ##logging per server
        access_log /var/log/nginx/grafana/access.log;
        error_log /var/log/nginx/grafana/error.log;

       location / {
       ##  root /home/xxxx/grafana-1.5.3/;
       }
}

config.js URL for graphite (in grafana)

graphiteUrl: "http://xxx.xxx.xxx.xxx:8080"

Edit Graphite does not need authentication for access from grafana. Also, I am using grafana v1.5.3

Solution 1

Try to access graphite through nginx (same origin). Just add new location

location /render {
                proxy_pass      http://xxx.xxx.xxx.xxx:8080/render;
}

then in your grafana config file change graphite url

Solution 2

Try to run browser whith "disable-web-security" flag.

Solution 3

I was able to solve this problem by changing the requests to GET instead of POST. See this issue for more information. https://github.com/grafana/grafana/issues/345

My data sources ended up looking like

datasources: {
  graphite: {
    type: 'graphite',
    url: window.location.protocol+"//"+window.location.hostname+":"+window.location.port+"/_graphite",
    default: true,
    render_method: 'GET'  
  },
},

I still haven't figured out how to get my graphite install to accept POST requests. Even when querying directly so I can be sure CORS isn't an issue.

Solution 4

I think you need to enable CORS in nginx configuration for graphite.Take a look at : http://enable-cors.org/server_nginx.html . Here's the configuration I made using this link:

(In my case, grafana is exposed on port 8100, and graphite on port 8090; adapt accordingly (8100 -> 85 , 8090 -> 8080) ).

upstream django {
    # Distribute requests to servers based on client IP. This keeps load
    # balancing fair but consistent per-client. In this instance we're
    # only using one uWGSI worker anyway.
    ip_hash;
    server unix:/tmp/uwsgi.sock;
}

server {
   listen      yourServerIp:8090;
   server_name yourServerName.com;
   access_log      /var/log/nginx/graphite_access.log;
   error_log       /var/log/nginx/graphite_error.log;
   charset     utf-8;


   # Django admin media.
   location /media/admin/ {
      alias /usr/lib/python2.7/dist-packages/django/contrib/admin/media/;
   }

   # Static media.
   location /content/ {
      alias /opt/graphite/webapp/content/;
   }

   # Send all non-media requests to the Django server.
   location / {

    # CORS (for grafana)

    if ($http_origin ~* "^http://yourServerName.com:8100$") {
     set $cors "true";
    }

    if ($request_method = 'OPTIONS') {
     set $cors "${cors}options";  
    }

    if ($request_method = 'GET') {
     set $cors "${cors}get";  
    }

    if ($request_method = 'POST') {
     set $cors "${cors}post";
    }

    if ($cors = "trueoptions") {
     add_header 'Access-Control-Allow-Origin' "$http_origin";
     add_header 'Access-Control-Allow-Credentials' 'true';
     add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
     add_header 'Access-Control-Allow-Headers' 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
     add_header 'Access-Control-Max-Age' 1728000;
     add_header 'Content-Type' 'text/plain charset=UTF-8';
     add_header 'Content-Length' 0;

     return 204;
    }

    if ($cors = "truepost") {
     add_header 'Access-Control-Allow-Origin' "$http_origin";
     add_header 'Access-Control-Allow-Credentials' 'true';
     add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
     add_header 'Access-Control-Allow-Headers' 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    }

    if ($cors = "trueget") {
     add_header 'Access-Control-Allow-Origin' "$http_origin";
     add_header 'Access-Control-Allow-Credentials' 'true';
     add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
     add_header 'Access-Control-Allow-Headers' 'DNT,X-Mx-ReqToken,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
    }

     uwsgi_pass  django;
     include     uwsgi_params;
   }
}

Note that the interesting part for you is what's below # CORS , the django stuff might be useless for you.

To ensure it's a CORS issue, you want to inspect HTTP headers sent by your browser; if there's a Origin header, it means you have to use CORS.

Solution 5

I'm not sure if the OP has resolved their problem yet or not, but here's what worked for me:

I put graphite and grafana on the same site. Graphite exists at the root and grafana within /grafana/

This solves the cross site scripting problem without having to setup CORS: graphite and grafana are on the same site.

nginx site configuration:

upstream graphite {
    server unix:///var/uwsgi/graphite-web.sock;
}

server {
    listen 8080;
    listen [::]:8080 ipv6only=on;

    root /usr/share/graphite-web/;

    server_name localhost;

    location /static {
            alias /usr/share/graphite-web/static;
    }

    location /grafana {
            alias /usr/share/graphite-web/grafana;
    }


    location / {
            uwsgi_pass graphite;
            include /etc/nginx/uwsgi_params;
    }
}

To access grafana, I therefore go to:

http://192.168.1.1:8080/grafana

(192.168.1.1 is my server's ip address)

View more solutions

Comments

[ali haider]

I added '/render' in the server portion of the nginx default conf file for grafana - the get request still fails.

[ali haider]

I will most certainly do when I get back onto that installation - I was not able to look into the installation earlier. Much appreciated

[ali haider]

Based on my setup, just using the root and index options under the location setting do not work. I get a HTTP request error frmo app.js (in the render call). xxx.xxx.xxx.xxx:8080/render (8080 is where grafite is running). Do I need to change any config in grafana or graphite to make this work?

[ali haider]

The post call to the graphite URL (ON PORT 8080) from app.js (render call) fails.

[ali haider]

The issue (I think) is that I have a domain name for this server which I use to access the applications on it (including graphite and grafana). When I try directly with the URL, my existing setup works (I will try the CORS option mentioned by Colin when I get the chance). Once again, thanks for sharing

[foz]

In the current version of phantomjs, I had to use --web-security=false to make this work.

[Ryne Everett]

I tried this (without the elasticsearch block) and get "Error Could not load dashboards/default.json. Please make sure it exists". I'm not exactly sure which directory your root is pointing to and wonder if that could be the issue.