111: Connection refused nginx proxy for Docker containers

Question

• CentOS 7

I have a simple Nginx proxy Docker container listening on port 80. Here is the Dockerfile:

FROM centos:7
MAINTAINER Brian Ogden

# Not currently being used but may come in handy
ARG ENVIRONMENT

RUN yum -y update && \
    yum clean all && \
    yum -y install http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm \
    yum -y makecache && \
    yum -y install nginx-1.12.0 wget

# Cleanup some default NGINX configuration files we don’t need
RUN rm -f /etc/nginx/conf.d/default.conf


COPY /conf/proxy.conf /etc/nginx/conf.d/proxy.conf
COPY /conf/nginx.conf /etc/nginx/nginx.conf


CMD ["nginx"]

And for this Nginx Proxy here is my nginx.conf:

daemon off;
user  nginx;
worker_processes  2;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
    use epoll;
    accept_mutex off;
}


http {
    include       /etc/nginx/mime.types;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    client_max_body_size 300m;
    client_body_buffer_size 300k;
    large_client_header_buffers 8 64k;

    gzip  on;
    gzip_http_version 1.0;
    gzip_comp_level 6;
    gzip_min_length 0;
    gzip_buffers 16 8k;
    gzip_proxied any;
    gzip_types text/plain text/css text/xml text/javascript application/xml application/xml+rss application/javascript application/json;
    gzip_disable "MSIE [1-6]\.";
    gzip_vary on;

    include /etc/nginx/conf.d/*.conf;
}

And here is my proxy configuration:

upstream accountstaging {
    server 127.0.0.1:5023;
}

server {

    listen 80;
    server_name account.staging.mysite.com;

    location / {
        proxy_pass         http://accountstaging;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
    }
}

My proxy configuration is listening on port 80 and trying to request requests from account.staging.mysite.com to a Docker container running on the same Docker host as the Ngnix proxy listening on port 5023.

Here is my docker-compose.yml for my Nginx proxy:

version: '3'
services:
  reverseproxy:
    build: 
      context: ./
      dockerfile: docker/Dockerfile
    image: tsl.devops.reverseproxy.image
    container_name: tsl.devops.reverseproxy.container
    ports:
      - "80:80"

Here is the docker-compose.yml for this Docker container listening on port 5023: version: '3'

services:
  apistaging:
    build: 
      context: ./
      dockerfile: docker/staging/Dockerfile
    image: tsl.api.example.image
    container_name: tsl.api.example.container
    ports:
      - "127.0.0.1:5023:80"

The Dockerfile does not really matter much to my question but here it is anyways:

FROM tsl.devops.dotnetcore.base.image:2
MAINTAINER Brian Ogden

WORKDIR /app
COPY ./src/Tsl.Example/bin/Release/netcoreapp2.0/publish .

ENTRYPOINT ["dotnet", "Tsl.Example.dll"]

I followed this example to setup my proxy.

I have previously asked a related question on Stackexchange forums here and here. This question I have refined and simplified the scenario to a simply proxy forwarding a request to one Docker container listening on port 5023.

Since my base image is CentOS I have followed this here to make sure SELinux is allowing forward to port 5023

Solution

Thanks to this question and answer here, I was able realize that I had two issues going on:

1. the containers have different default Docker networks because I am using two different docker-compose.yml files, I had envisioned my Ngnix proxy working independently from any of my API containers entirely, including the docker-compose, more on that issue below
2. the second issue is simply when I tried to proxy to 127.0.0.1:5023 that is localhost inside the Ngnix container, not the network outside of the Nginx proxy container

So the different default networks being created by docker-compose for my Nginx proxy docker container and my api docker container are because I amusing two different docker-compose.yml files. This is because I have Jenkins builds for many API microservices so the have independant docker-compose files and I needed a Nginx proxy to forward requests on port 80 to each microservice.

To test this out, created a docker-compose.yml for both containers, the API and the Nginx proxy:

version: '3'

services:
  reverseproxy:
    build: 
      context: ./
      dockerfile: docker/nginxproxy/docker/Dockerfile
    image: tsl.devops.reverseproxy.image
    container_name: tsl.devops.reverseproxy.container
    ports:
      - "80:80"
  apistaging:
    build: 
      context: ./
      dockerfile: docker/staging/Dockerfile
    image: tsl.api.example.image
    container_name: tsl.api.example.container
    ports:
      - "5023:5023"
    environment: 
      ASPNETCORE_URLS: http://+:5023

Yes there was still an issue, the proxy pass to http//:127.0.0.1:5023, that forward remains in the Nginx Docker container and never finds the API running on the Docker host, I simply needed to use the docker-compose.yml service name to get to it:

upstream accountstaging {
    server apistaging:5023;
}

server {

    listen 80;
    server_name account.staging.mysite.com;

    location / {
        proxy_pass         http://accountstaging;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
    }
}

Comments

[Bugbeeb]

omg thank you! In you apistaging service I don't think you should expose port 5000 to host, so just put ports: - "5000"