nginx-ingress config map snippets being ignored by the nginx.conf

Question

I have a kubernetes cluster, where I have deployed an nginx ingress controller using the helm nginx-ingress chart.

I need to add some custom config to the nginx.conf file that is generated in the nginx-controller-pod, and I am seeing an issue where if I add a one line option such as proxy-buffer-size: "512k" I can see this reflected in the nginx.conf file and everything works as expected.

However, if I attempt to add a snippet to accomplish the same thing:

location-snippet: |
  proxy_buffer_size "512k";

It is as though this is ignored by the nginx.conf file and the proxy_buffer_size setting remains at it's default value.

I need to be able to add http-snippet, server-snippet and location-snippet overrides but whether I try to add them to the ConfigMap or as an annotation in the Ingress.yaml file they are always ignored.

My Ingress yaml file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/ssl-redirect: "true" 
    ingress.kubernetes.io/secure-backends: "true"    
    ingress.kubernetes.io/force-ssl-redirect: "true"

    ingress.kubernetes.io/location-snippet: |
       proxy_buffer_size 512k;     --This does not update the nginx.conf
spec:
  tls:
  - hosts:
    - my.app.co.uk
    secretName: tls-secret

  rules:
  - host: my.app.co.uk
    http:
      paths:
      - path: /
        backend:
          serviceName: myappweb-service
          servicePort: 80

My nginx config map:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: nginx-ingress
    chart: nginx-ingress-0.28.3
    component: controller
    heritage: Tiller
    release: nginx-ingress
  name: nginx-ingress-controller
  namespace: default
data:
  proxy-buffer-size: "512k" -- this works and updates the nginx.conf

  location-snippet: |
    proxy_buffers 4 512k; -- this does not update the nginx.conf

  server-snippet: |       -- this does not update the nginx.conf
    location /messagehub {
      proxy_set_header Upgrade $http_upgrade;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header Connection "upgrade";
      proxy_cache_bypass $http_upgrade;
   }

Solution 1

It turns out that my problem was due to the content of the snippet that I was applying. Every time you run kubectl apply -f myconfigmap.yaml, a validation is run against the changes that you are trying to apply to the nginx.conf. When this validation fails, it fails silently and there is nothing to alert you to this in the terminal.

In fact, you still get the configmap/nginx-ingress-controller configured message.

For example, when I add this to the config map, it updates the nginx.conf as expected:

http-snippet: |
  sendfile on;

However, when I add this, nothing changes:

http-snippet: |
  sendfile on;
  tcp_nopush on;

The reason being that this has failed validation, but the only way to find that out is to look at the logs of the nginx ingress controller pod. In this instance I see:

Error: exit status 1
2018/10/16 07:45:49 [emerg] 470#470: "tcp_nopush" directive is duplicate in 
/tmp/nginx-cfg468835321:245
nginx: [emerg] "tcp_nopush" directive is duplicate in /tmp/nginx-cfg468835321:245
nginx: configuration file /tmp/nginx-cfg468835321 test failed

So I was duplicating a directive that already existed.

Solution 2

If you'd like to modify your Kubernetes Ingress the annotation options are these:

• nginx.ingress.kubernetes.io/configuration-snippet for an nginx location block snippet
• nginx.ingress.kubernetes.io/server-snippet for a snippet in the nginx config service block

Looks like you are using nginx.org/location-snippets: for that case.

There's also a YAML invalid syntax on nginx config example and also you should use plurals as in server-snippets according to this example. There's a typo in the docs as of this writing. Opened this ticket to follow up.

It should be something like this:

  server-snippets: |
    location /messagehub {
      proxy_set_header Upgrade $http_upgrade;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header Connection "upgrade";
      proxy_cache_bypass $http_upgrade;
      }

instead of this:

  server-snippet: |
    location /messagehub {
      proxy_set_header Upgrade $http_upgrade;
      proxy_http_version 1.1;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header Connection "upgrade";
      proxy_cache_bypass $http_upgrade;
    }

Notice the indentation of the last curly brace.

Solution 3

I spent a day untill got that for ingress-nginx (from Kubernetes) there are *-snippet, but for nginx-ingress (from NGINX) that *-snippets with s.

Look here:

• ingress-nginx: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/#main-snippet
• nginx-ingress: https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#snippets-and-custom-templates