Why does nginx return 404 on my Kubernetes Ingress route?

Question

I have two nginx web servers set up. One is set up on AWS Elastic Beanstalk, the other on Kubernetes using the stable/nginx-ingress helm chart.

The Elastic Beanstalk webserver forwards traffic from all subroutes of my domain to the Kubernetes nginx webserver. I can verify these are being forwarded correctly by checking the logs from the Kubernetes nginx. I use an Ingress resource to make sure this traffic is being forwarded to the right Kubernetes service.

Here is the problem: One of the two routes, the main / route, is forwarded to the correct service and returns a 200. The other route, /eks-test, is supposed to route to the same service, but returns a 404. How is this possible?

Specs:

The nginx on Kubernetes is running nginx 0.25.1.

Nginx logs:

172.16.10.103 - [172.16.10.103] - - [12/Sep/2019:08:05:09 +0000] "GET / HTTP/1.0" 200 8 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 703 0.004 [default-eks-test-repo-80] [] 172.16.10.100:8080 8 0.004 200 90dfa37364a5c43e57f12c5fb1a2d86f
172.16.40.108 - [172.16.40.108] - - [12/Sep/2019:08:05:12 +0000] "GET /eks-test HTTP/1.0" 404 9 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" 730 0.002 [default-eks-test-repo2-80] [] 172.16.43.125:8080 9 0.004 404 ef1c81bba75dff2bdd2376799aa93c56

First nginx config (Elastic Beanstalk):

server {
    listen 80;

    server_name my.domain.com;

    location / {

        proxy_pass http://internal.my.domain.lan/;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

Kubernetes Ingress resource:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: internal
  name: eks-test
  namespace: default
spec:
  rules:
    - host: my.domain.com
      http:
        paths:
          - path: /
            backend:
              serviceName: eks-test-repo
              servicePort: 80
          - path: /eks-test
            backend:
              serviceName: eks-test-repo
              servicePort: 80

Kubernetes Service:

kind: Service
apiVersion: v1
metadata:
  name: eks-test-repo
  namespace: default
  labels:
    name: eks-test-repo
spec
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: eks-test-repo
  type: ClusterIP

Nginx helm chart values (ones that are not default):

controller.ingressClass: internal

Solution

The Nginx Ingress does not strip the path of your request.

When you request /eks-test on Ingress, the request is forwarded to your service including the path, ending in your container as "GET /eks-test HTTP/1.0". Your container then returns 404, as it does not have the /eks-test route.

If you want to strip the path from the request, you can configure some rewrite rules (e.g.: setting the nginx.ingress.kubernetes.io/rewrite-target: / annotation).

Comments

[Lars S]

Thanks! Adding ingress.kubernetes.io/rewrite-target: / to my annotations (without the nginx prefix) did the job!

[Eduardo Baitello]

Good to know @LarsS. The new nginx. annotation prefix was added to version 0.9.0 of Nginx Ingress. You must be using an older version (or enforcing the old prefix with the flag --annotations-prefix). If you are using such an old version, consider updating for better performance/security.

[Lars S]

Thanks for the clarification @Eduardo. I am on version 0.25.1 and I was indeed using the --annotations-prefix flag. Your answer should help most people.