GKE ingress unable to connect to healthy service

Question

I've been following the tutorial here: https://medium.com/@nithinmallya4/deploying-a-rails-application-to-google-container-engine-with-kubernetes-b08b2de353fc

With their code base: https://github.com/nmallya/gkedemoapp

You can see how this deployment is put together here, between the "FIRST TIME DEPLOYMENT" comments: https://github.com/nmallya/gkedemoapp/blob/master/gcloud_deployment.sh

I have created the SSL cert, the web service is up and healthy, but I can't create the ingress as described. The status for the ingress is "All backend services are in UNHEALTHY state".

The YAML for the ingress is as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gke-ingress 
  annotations:
    kubernetes.io/ingress.class: "gce"
    # kubernetes.io/ingress.allow-http: "false"
    # ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
  # This assumes tls-secret exists.
  - secretName: gkecert
  backend:
    serviceName: web
    servicePort: 443

I imagine I'll need to provide more information but I'm not sure what/where to troubleshoot next.

Solution

It turns out my services were not in a healthy state. Being "READY" doesn't mean that they're responding with 200 on the root path (/). I used kubectl get pods to list my pods, then inspected the logs.

NAME                            READY     STATUS    RESTARTS   AGE
cable-76bfb45bb-rkz8f           2/2       Running   0          24m
rails-67d569746d-54hs8          2/2       Running   0          24m
elasticsearch-0                 1/1       Running   0          16m
redis-master-945b795f6-ldg5x    1/1       Running   0          14h
sidekiq-597c876bc4-zg24k        2/2       Running   0          24m

I could then inspect specific logs with kubectl logs rails-67d569746d-54hs8 rails where the second rails is the name of the container within the pod. You should see the logs for the health check there, which should indicate what's wrong.

If you only see the start up of the server and no health check logs, it's most likely your pod is not reachable. Check your services and the ports on them. As long as they're reachable and responding with 200 on the root path (and not redirecting), the health check should pass and the ingress should allow connection.

Comments

[Patrick W]

the error message implies your backend (pods) are failing their health checks. You need to curl the HC path to see if you get a 200 response there. Can you provide us the YAML of your service so we can tell which target port the HC is querying?

[Archonic]

My impression that the pods were healthy was from the READY 2/2 statement in kubectl get pods. They were in fact not healthy and checking the logs for the web service in those pods revealed why. I'll post an answer on how I found the issue shortly.