Kubernetes Ingress (GCE) keeps returning 502 error
Solution 1
Your backend k8s-be-32396--5fc40252fadea594
is showing as "UNHEALTHY"
.
Ingress will not forward traffic if the backend is UNHEALTHY, this will result in the 502 error you are seeing.
It will be being marked as UNHEALTHY becuase it is not passing it's health check, you can check the health check setting for k8s-be-32396--5fc40252fadea594 to see if they are appropriate for your pod, it may be polling an URI or port that is not returning a 200 response. You can find these setting under Compute Engine > Health Checks.
If they are correct then there are many steps between your browser and the container that could be passing traffic incorrectly, you could try kubectl exec -it PODID -- bash
(or ash if you are using Alpine) and then try curl-ing localhost to see if the container is responding as expected, if it is and the health checks are also configured correctly then this would narrow down the issue to likely be with your service, you could then try changing the service from a NodePort type to a LoadBalancer and see if hitting the service IP directly from your browser works.
Solution 2
I was having the same issue. It turns out I had to wait a few minutes before ingress to validate the service health. If someone is going to the same and done all the steps like readinessProbe
and linvenessProbe
, just ensure your ingress is pointing to a service that is either a NodePort
, and wait a few minutes until the yellow warning icon turns into a green one. Also, check the log on StackDriver to get a better idea of what's going on. My readinessProbe
and livenessProbe
is on /login
, for the gce
class. So I don't think it has to be on /healthz
.
Solution 3
Issue is indeed a health check and seemed "random" for my apps where I used name-based virtual hosts to reverse proxy requests from ingress via domains to two separate backend services. Both were secured using Lets Encrypt and kube-lego
. My solution was to standardize the path for health checks for all services sharing an ingress, and declare the readinessProbe
and livenessProbe
configs in my deployment.yml
file.
I faced this issue with Google cloud cluster node version 1.7.8
and found this issue that closely-resembled what I experienced:
* https://github.com/jetstack/kube-lego/issues/27
I'm using gce
and kube-lego
and my backend service health checks were on /
and kube-lego
is on /healthz
. It appears differing paths for health checks with gce ingress
might be the cause so it may be worth updating backend services to match the /healthz
pattern so all use same (or as one commenter in Github issue stated they updated kube-lego to pass on /
).
Solution 4
I had the same issue. I turned out that the pod itself was running ok, which I tested via port-forwarding and accessing the health-check URL.
Port-Forward can be activated in console as follows:
$ kubectl port-forward <pod-name> local-port:pod-port
So if the pod is running ok and ingress still shows unhealthy state there might be an issue with your service configuration. In my case my app-selector was incorrect, causing the selection of a non existent pod. Interestingly this isn't showed as an errors or alerts in google console.
Definition of the pods:
#pod-definition.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: <pod-name>
namespace: <namespace>
spec:
selector:
matchLabels:
app: **<pod-name>**
template:
metadata:
labels:
app: <pod-name>
spec:
#spec-definition follows
#service.yaml
apiVersion: v1
kind: Service
metadata:
name: <name-of-service-here>
namespace: <namespace>
spec:
type: NodePort
selector:
app: **<pod-name>**
ports:
- protocol: TCP
port: 8080
targetPort: 8080
name: <port-name-here>
Solution 5
I had the same problem, and it persisted after I enabled livenessProbe
as well readinessPorbe
.
It turned this was to do with basic auth. I've added basic auth to livenessProbe
and the readinessPorbe
, but turns out the GCE HTTP(S) load balancer doesn't have a configuration option for that.
There seem to be a few another kind of issue with too, e.g. setting container port to 8080 and service port to 80 didn't work with GKE ingress controller (yet I wouldn't clearly indicate what the problem was). And broadly, it looks to me like there is very little visibility and running your own ingress container is a better option with respect to visibility.
I picked Traefik for my project, it worked out of the box, and I'd like to enable Let's Encrypt integration. The only change I had to make to Traefik manifests was about tweaking the service object to disabling access to the UI from outside of the cluster and expose my app with through external load balancer (GCE TCP LB). Also, Traefik is more native to Kubernetes. I tried Heptio Contour, but something didn't work out of the box (will give it a go next time when the new version comes out).
Moon
I am hands-on software architect / engineer and effective manager with over 11 years of experience in software craftsmanship and cloud infrastructure. I have a degree in Computer Science. Key Skills Technical Leadership Team Building / Mentoring Software Architecture & Engineering Business Management Systems Implementation & Optimization Software Engineering Project Management Cloud Based Enterprise Solutions IT Infrastructure Management Quality Assurance Personal Favorites Kubernetes, Docker, Python, Golang (Google Go), RabbitMQ, RethinkDB, Google Cloud Platform, Webpack, React, ES6, Atom, Erlang/Elixir (currently learning), gRPC Background An entrepreneur at heart, I have dedicated my career to developing efficient and scalable solutions to problems faced by many organizations and governments across the world. I believe that educating and empowering individuals and teams to use cutting edge technical inventions help improve efficiency, reduce redundancy, and ultimately push the upward boundaries of potential that is the underlying key to success in any venture. I have an insatiable attitude towards learning and creating solutions that are aesthetically pleasing yet architecturally efficient and scalable. I have a solid background in microservices based enterprise applications development for systems related to business automation, such as Project Portfolio Management (PPM), Enterprise Resource Planning (ERP), Enterprise Content Management (ECM), and Customer Relationship Management (CRM). I am a balanced advocate and practitioner of design patterns with expertise in architecture and engineering of enterprise grade, scalable, real-time solution infrastructure on Microsoft Azure and Amazon Web Services (AWS). I have extensive experience managing teams of in-house and off-shore Software and QA Engineers. Over past few years, I have help build amazing products used and loved by many organizations and Governments worldwide.
Updated on June 19, 2022Comments
-
Moon almost 2 years
I am trying to setup an Ingress in GCE Kubernetes. But when I visit the IP address and path combination defined in the Ingress, I keep getting the following 502 error:
Here is what I get when I run:
kubectl describe ing --namespace dpl-staging
Name: dpl-identity Namespace: dpl-staging Address: 35.186.221.153 Default backend: default-http-backend:80 (10.0.8.5:8080) TLS: dpl-identity terminates Rules: Host Path Backends ---- ---- -------- * /api/identity/* dpl-identity:4000 (<none>) Annotations: https-forwarding-rule: k8s-fws-dpl-staging-dpl-identity--5fc40252fadea594 https-target-proxy: k8s-tps-dpl-staging-dpl-identity--5fc40252fadea594 url-map: k8s-um-dpl-staging-dpl-identity--5fc40252fadea594 backends: {"k8s-be-31962--5fc40252fadea594":"HEALTHY","k8s-be-32396--5fc40252fadea594":"UNHEALTHY"} Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 15m 15m 1 {loadbalancer-controller } Normal ADD dpl-staging/dpl-identity 15m 15m 1 {loadbalancer-controller } Normal CREATE ip: 35.186.221.153 15m 6m 4 {loadbalancer-controller } Normal Service no user specified default backend, using system default
I think the problem is
dpl-identity:4000 (<none>)
. Shouldn't I see the IP address of thedpl-identity
service instead of<none>
?Here is my service description:
kubectl describe svc --namespace dpl-staging
Name: dpl-identity Namespace: dpl-staging Labels: app=dpl-identity Selector: app=dpl-identity Type: NodePort IP: 10.3.254.194 Port: http 4000/TCP NodePort: http 32396/TCP Endpoints: 10.0.2.29:8000,10.0.2.30:8000 Session Affinity: None No events.
Also, here is the result of executing:
kubectl describe ep -n dpl-staging dpl-identity
Name: dpl-identity Namespace: dpl-staging Labels: app=dpl-identity Subsets: Addresses: 10.0.2.29,10.0.2.30 NotReadyAddresses: <none> Ports: Name Port Protocol ---- ---- -------- http 8000 TCP No events.
Here is my deployment.yaml:
apiVersion: v1 kind: Secret metadata: namespace: dpl-staging name: dpl-identity type: Opaque data: tls.key: <base64 key> tls.crt: <base64 crt> --- apiVersion: v1 kind: Service metadata: namespace: dpl-staging name: dpl-identity labels: app: dpl-identity spec: type: NodePort ports: - port: 4000 targetPort: 8000 protocol: TCP name: http selector: app: dpl-identity --- apiVersion: extensions/v1beta1 kind: Ingress metadata: namespace: dpl-staging name: dpl-identity labels: app: dpl-identity annotations: kubernetes.io/ingress.allow-http: "false" spec: tls: - secretName: dpl-identity rules: - http: paths: - path: /api/identity/* backend: serviceName: dpl-identity servicePort: 4000 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: dpl-staging name: dpl-identity kind: Ingress metadata: namespace: dpl-staging name: dpl-identity labels: app: dpl-identity annotations: kubernetes.io/ingress.allow-http: "false" spec: tls: - secretName: dpl-identity rules: - http: paths: - path: /api/identity/* backend: serviceName: dpl-identity servicePort: 4000 --- apiVersion: extensions/v1beta1 kind: Deployment metadata: namespace: dpl-staging name: dpl-identity labels: app: dpl-identity spec: replicas: 2 strategy: type: RollingUpdate template: metadata: labels: app: dpl-identity spec: containers: - image: gcr.io/munpat-container-engine/dpl/identity:0.4.9 name: dpl-identity ports: - containerPort: 8000 name: http volumeMounts: - name: dpl-identity mountPath: /data volumes: - name: dpl-identity secret: secretName: dpl-identity