SSL passthrough not being configured for ingress-nginx backend

Question

I am running the nginx ingress controller in minikube via helm and I can see SSL passthrough is enabled in the controller by looking at the logs of the nginx ingress controller pod.

helm upgrade ingress stable/nginx-ingress --install --namespace kube-system --set "controller.extraArgs.annotations-prefix=nginx.ingress.kubernetes.io" --set "controller.extraArgs.enable-ssl-passthrough=" --set controller.hostNetwork=true

Internally I have a HTTPS REST API service exposed on port 19000. I want mutual TLS between a client and the service running inside k8s so I am trying to configure my ingress with SSL passthrough enabled but when I set the nginx.ingress.kubernetes.io/ssl-passthrough annotation to "true" on my ingress, the backend still shows sslPassthrough set to false and when I send requests to the service, nginx is stripping the TLS certificate from my requests.

Is there some configuration I am missing to enable SSL passthrough on the backend?

$ kubectl ingress-nginx --deployment ingress-nginx-ingress-controller -n kube-system backends
[
  {
    "name": "default-tlsapi-service-19000",
    "service": {
      "metadata": {
        "creationTimestamp": null
      },
      "spec": {
        "ports": [
          {
            "protocol": "TCP",
            "port": 19000,
            "targetPort": 19000,
            "nodePort": 30000
          }
        ],
        "selector": {
          "app": "tlsapi"
        },
        "clusterIP": "10.96.218.188",
        "type": "NodePort",
        "sessionAffinity": "None",
        "externalTrafficPolicy": "Cluster"
      },
      "status": {
        "loadBalancer": {}
      }
    },
    "port": 19000,
    "sslPassthrough": false,
    "endpoints": [
      {
        "address": "172.17.0.7",
        "port": "19000"
      }
    ],
    "sessionAffinityConfig": {
      "name": "",
      "mode": "",
      "cookieSessionAffinity": {
        "name": ""
      }
    },
    "upstreamHashByConfig": {
      "upstream-hash-by-subset-size": 3
    },
    "noServer": false,
    "trafficShapingPolicy": {
      "weight": 0,
      "header": "",
      "headerValue": "",
      "cookie": ""
    }
  }

ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: tlsapi-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
spec:
  rules:
  - host: tlsapi.k8s
  - http:
      paths:
      - path: /
        backend:
          serviceName: tlsapi-service
          servicePort: 19000

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: tlsapi-service
spec:
  type: NodePort
  selector:
    app: tlsapi
  ports:
    - port: 19000
      targetPort: 19000
      nodePort: 30000

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: tlsapi-deployment
spec:
  selector:
    matchLabels:
      app: tlsapi
  replicas: 1
  template:
    metadata:
      name: tlsapi-pod
      labels:
        app: tlsapi
    spec:
      containers:
      - name: tlsapi-container
        image: tlsapi:latest
        imagePullPolicy: IfNotPresent

Solution 1

Turns out the problem was with my ingress definition. I needed to remove the - from in front of the http key:

working ingress.yaml snippet

spec:
  rules:
  - host: tlsapi.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: tlsapi-service
          servicePort: 19000

not working ingress.yaml snippet

spec:
  rules:
  - host: tlsapi.k8s
  - http:
      paths:
      - path: /
        backend:
          serviceName: tlsapi-service
          servicePort: 19000

The single character difference is much more obvious if you convert the yaml to json!

With the character removed, SSL passthrough works as expected

Solution 2

You need to start the nginx ingress controller with flag --enable-ssl-passthrough

You can provide this flag in the args section of nginx ingress controller deployment yaml

spec:
  # wait up to five minutes for the drain of connections
  terminationGracePeriodSeconds: 300
  serviceAccountName: nginx-ingress-serviceaccount
  containers:
    - name: nginx-ingress-controller
      image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:master
      args:
        - --enable-ssl-passthrough

You can confirm that the argument took effect by looking at nginx ingress controller logs and searching for Starting TLS proxy for SSL Passthrough.

Comments

[mc2020]

Hi, that flag has already been provided. And I can confirm that nginx is running with SSL passthrough enabled by looking at the logs and seeing the line: nginx.go:750] Starting TLS proxy for SSL Passthrough

[Arghya Sadhu]

Can you give more details on where did you get that command that you are running to check the backends?

[mc2020]

It is from the ingress-nginx kubectl plugin, see: kubernetes.github.io/ingress-nginx/kubectl-plugin/#backends Is there another way of seeing/changing the backend configuration? perhaps by exec'ing into the running nginx-controller container