Varnish "FetchError no backend connection" error

linux nginx varnish

9,943

I had a similar issue when trying to test Varnish locally with different backends. Using 127.0.0.1:8080 worked fine but changing the port to 8081 gave me a 503, even though that backend worked perfectly for me outside Varnish.

The problem was caused by SELinux not allowing the connection. I found that out by tailing the audit log and provoking the 503 from Varnish:

$ sudo tail -f /var/log/audit/audit.log type=AVC msg=audit(1539253067.438:1379): avc: denied { name_connect } for pid=10154 comm="varnishd" dest=8081 scontext=system_u:system_r:varnishd_t:s0 tcontext=system_u:object_r:transproxy_port_t:s0 tclass=tcp_socket

To see ports allowed by SELinux from Varnish you can use this command:

$ sudo semanage port -l | grep http_cache_port_t http_cache_port_t tcp 8080, 8118, 8123, 10001-10010

To fix the issue you can either use another port, e.g. 8118 or ask SELinux to allow connections to 8081 from Varnish.

The command to add the port is - the -a flag is to add the port:

semanage port -a -t http_cache_port_t -p tcp 8081

If you get a message telling you that the port is already defined then you need to modify rather than add the port:

ValueError: Port tcp/8081 already defined

The command swaps the -a flag for -m - 'modify':

semanage port -m -t http_cache_port_t -p tcp 8081

9,943

anon-123

Updated on September 18, 2022

Comments

anon-123 over 1 year

Varnishlog:

    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1340829925 1.0
   12 SessionOpen  c 79.124.74.11 3063 :80
   12 SessionClose c EOF
   12 StatSess     c 79.124.74.11 3063 0 1 0 0 0 0 0 0
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1340829928 1.0
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1340829931 1.0
   12 SessionOpen  c 108.62.115.226 46211 :80
   12 ReqStart     c 108.62.115.226 46211 467185881
   12 RxRequest    c GET
   12 RxURL        c /
   12 RxProtocol   c HTTP/1.0
   12 RxHeader     c User-Agent: Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
   12 RxHeader     c Host: www.mysite.com
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /
   12 Hash         c www.mysite.com
   12 VCL_return   c hash
   12 VCL_call     c miss fetch
   12 FetchError   c no backend connection
   12 VCL_call     c error deliver
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 503
   12 TxResponse   c Service Unavailable
   12 TxHeader     c Server: Varnish
   12 TxHeader     c Content-Type: text/html; charset=utf-8
   12 TxHeader     c Retry-After: 5
   12 TxHeader     c Content-Length: 418
   12 TxHeader     c Accept-Ranges: bytes
   12 TxHeader     c Date: Wed, 27 Jun 2012 20:45:31 GMT
   12 TxHeader     c X-Varnish: 467185881
   12 TxHeader     c Age: 1
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: close
   12 Length       c 418
   12 ReqEnd       c 467185881 1340829931.192433119 1340829931.891024113 0.000051022 0.698516846 0.000074035
   12 SessionClose c error
   12 StatSess     c 108.62.115.226 46211 1 1 1 0 0 0 256 418
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1340829934 1.0
    0 CLI          - Rd ping
    0 CLI          - Wr 200 19 PONG 1340829937 1.0

netstat -tlnp
 Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      3086/nginx      
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1915/varnishd   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1279/sshd       
tcp        0      0 127.0.0.2:25            0.0.0.0:*               LISTEN      3195/sendmail: MTA:
tcp        0      0 127.0.0.2:6082          0.0.0.0:*               LISTEN      1914/varnishd   
tcp        0      0 127.0.0.2:9000          0.0.0.0:*               LISTEN      1317/php-fpm.conf)
tcp        0      0 127.0.0.2:3306          0.0.0.0:*               LISTEN      1192/mysqld     
tcp        0      0 127.0.0.2:587           0.0.0.0:*               LISTEN      3195/sendmail: MTA:
tcp        0      0 127.0.0.2:11211         0.0.0.0:*               LISTEN      3072/memcached  
tcp6       0      0 :::8080                 :::*                    LISTEN      3086/nginx      
tcp6       0      0 :::80                   :::*                    LISTEN      1915/varnishd   
tcp6       0      0 :::22                   :::*                    LISTEN      1279/sshd

/etc/nginx/site-enabled/default

server {
    listen   8080; ## listen for ipv4; this line is default and implied
    listen   [::]:8080 default ipv6only=on; ## listen for ipv6

    root /usr/share/nginx/www;
    index index.html index.htm index.php;

    # Make site accessible from http://localhost/
    server_name localhost;

    location / {
        # First attempt to serve request as file, then
        # as directory, then fall back to index.html
        try_files $uri $uri/ /index.html;
    }

    location /doc {
        root /usr/share;
        autoindex on;
        allow 127.0.0.2;
        deny all;
    }

    location /images {
        root /usr/share;
        autoindex off;
    }

    #error_page 404 /404.html;

    # redirect server error pages to the static page /50x.html
    #
    #error_page 500 502 503 504 /50x.html;
    #location = /50x.html {
    #   root /usr/share/nginx/www;
    #}

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #   proxy_pass http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    location ~ \.php$ {
        fastcgi_pass 127.0.0.2:9000;
        fastcgi_index index.php;
        include fastcgi_params;
    }

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #   deny all;
    #}
}

/etc/nginx/sites-enabled/www.mysite.com.vhost

server {
       listen 8080;
       server_name www.mysite.com mysite.com.net;
       root /var/www/www.mysite.com/web;
       if ($http_host != "www.mysite.com") {
                 rewrite ^ http://www.mysite.com$request_uri permanent;
       }
       index index.php index.html;
       location = /favicon.ico {
                log_not_found off;
                access_log off;
       }
       location = /robots.txt {
                allow all;
                log_not_found off;
                access_log off;
       }
       # Deny all attempts to access hidden files such as .htaccess, .htpasswd, .DS_Store (Mac).
       location ~ /\. {
                deny all;
                access_log off;
                log_not_found off;
       }
       location / {
                try_files $uri $uri/ /index.php?$args;
       }
       # Add trailing slash to */wp-admin requests.
       rewrite /wp-admin$ $scheme://$host$uri/ permanent;
       location ~*  \.(jpg|jpeg|png|gif|css|js|ico)$ {
                expires max;
                log_not_found off;
       }
       location ~ \.php$ {
                try_files $uri =404;
                include /etc/nginx/fastcgi_params;
                fastcgi_pass 127.0.0.2:9000;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
       }

include /var/www/www.mysite.com/web/nginx.conf;
       location ~ /nginx.conf {
                deny all;
                access_log off;
                log_not_found off;
       }

}

/etc/varnish/default.vcl

# This is a basic VCL configuration file for varnish.  See the vcl(7)
# man page for details on VCL syntax and semantics.
# 
# Default backend definition.  Set this to point to your content
# server.
# 
backend default {
    .host = "127.0.0.2";
    .port = "8080";
    # .connect_timeout = 600s;
    #.first_byte_timeout = 600s;
    # .between_bytes_timeout = 600s;
    # .max_connections = 800;

Note: uncommenting the last four options at default.vcl made no difference.

cat /etc/default/varnish

# Configuration file for varnish
#
# /etc/init.d/varnish expects the variables $DAEMON_OPTS, $NFILES and $MEMLOCK
# to be set from this shell script fragment.
#

# Should we start varnishd at boot?  Set to "yes" to enable.
START=yes

# Maximum number of open files (for ulimit -n)
NFILES=131072

# Maximum locked memory size (for ulimit -l)
# Used for locking the shared memory log in memory.  If you increase log size,
# you need to increase this number as well
MEMLOCK=82000

# Default varnish instance name is the local nodename.  Can be overridden with
# the -n switch, to have more instances on a single server.
INSTANCE=$(uname -n)

# This file contains 4 alternatives, please use only one.

## Alternative 1, Minimal configuration, no VCL
#
# Listen on port 6081, administration on localhost:6082, and forward to
# content server on localhost:8080.  Use a 1GB fixed-size cache file.
#
# DAEMON_OPTS="-a :6081 \
#              -T localhost:6082 \
#        -b localhost:8080 \
#        -u varnish -g varnish \
#            -S /etc/varnish/secret \
#        -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"


## Alternative 2, Configuration with VCL
#
# Listen on port 6081, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request.  Use a 1GB
# fixed-size cache file.
#
DAEMON_OPTS="-a :80 \
             -T 127.0.0.2:6082 \
             -f /etc/varnish/default.vcl \
             -S /etc/varnish/secret \
             -s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

If you need any other info let me know. I am all out of clue as to whats the problem.

curl header gives me this:

curl -v -I -H "Testing: Test header so you see this works" http://www.mysite.com:8080
* About to connect() to www.mysite.com port 8080 (#0)
*   Trying 176.31.158.78... connected
* Connected to www.mysite.com (176.31.158.78) port 8080 (#0)
> HEAD / HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: www.mysite.com:8080
> Accept: */*
> Testing: Test header so you see this works
> 
< HTTP/1.1 301 Moved Permanently
HTTP/1.1 301 Moved Permanently
< Server: nginx/1.0.5
Server: nginx/1.0.5
< Date: Thu, 28 Jun 2012 11:01:23 GMT
Date: Thu, 28 Jun 2012 11:01:23 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 184
Content-Length: 184
< Connection: keep-alive
Connection: keep-alive
< Location: http://www.mysite.com/
Location: http://www.mysite.com/

< 
* Connection #0 to host www.mysite.com left intact
* Closing connection #0

Pax almost 12 years

Try enabling backend polling. varnish-cache.org/trac/wiki/BackendPolling Does nginx return your content if you hit it directly on port 8080 with curl or wget?
cyberx86 almost 12 years

The last time I saw this, the problem was SELinux. As a test, try to turn it off: echo 0 >/selinux/enforce. If you confirm that it is the problem, you can set an exception with audit2allow.
Skamasle over 10 years

There are something on nginx error_log ? maybe a 503 error ? and in your mysite.com.vhost try to remove lines 5, 6 and 7 and then try again to access.
Grizly over 10 years

Does it work with malloc instead of file? You might also need the closing } in your vcl file.
Kojo about 7 years

I posted an answer even if your post is very old, because I had the same error and searched a while before I realized this was caused by non www to www redirections