Readiness probe failed: Get http://10.32.1.71:80/setting s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Question

The configuration I have is Jenkins on Kubernetes and the project is written in PHP.

The issue here is that the pod is attached to an ingress(than on a loadBalancer using GCE) and when the pod is unhealthy it won't add it.

The first time I load the project from 0 it works after I update it, it fails since it's unhealthy.

When I describe the pod I get the following warning:

Readiness probe failed: Get http://10.32.1.71:80/setting s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

My production configuration:

# Configuration for the SQL connection 
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: wobbl-main-backend-production
spec:
  replicas: 1
  template:
    metadata:
      name: backend
      labels:
        app: wobbl-main
        role: backend
        env: production
    spec:
      containers:
        - name: backend
          image: gcr.io/cloud-solutions-images/wobbl-mobile-backend:1.0.0
          resources:
            limits:
              memory: "500Mi"
              cpu: "100m"
          imagePullPolicy: Always
          readinessProbe:
            httpGet: # make an HTTP request
              port: 80 # port to use
              path: /settings # endpoint to hit
              scheme: HTTP # or HTTPS
            initialDelaySeconds: 3 # how long to wait before checking
            periodSeconds: 5 # how long to wait between checks
            successThreshold: 1 # how many successes to hit before accepting
            failureThreshold: 2 # how many failures to accept before failing
            timeoutSeconds: 10 # how long to wait for a response
          ports:
          - name: backend
            containerPort: 80

Any hints on how to solve this.

Solution 1

The error message implies your HTTP request is not successful. The readiness probe needs to succeed for the pod to be added as an endpoint for the service exposing it.

1) kubectl get po -o wide

This is so you can get the pod's cluster IP

2) kubectl exec -t [another_pod] -- curl -I [pod's cluster IP]

If you get a 200 response, you know the path is configured properly and the readiness probe should be passing. If you get anything other than a 200 response, this is why the readiness probe fails and you need to check your image.

Solution 2

The workaround from github.com/kubernetes/kubernetes/issues/89898

You might want to change httpGet to the exec / command / curl:

      ...
      readinessProbe:
        exec:
          command:
          - "curl"
          - "--fail"
          - "-o"
          - "/dev/null"
          - "http://localhost/settings"
        initialDelaySeconds: 3
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 2
        timeoutSeconds: 10

Comments

[DaAmidza]

The second command is working only when i put my namespace.Does that infect the readiness probe from being executed?

[Patrick W]

no the namespace is required because you are targeting a pod that is not in the default namespace. As long as you are getting a 200 response from that command (with the namespace if needed) then the path in your target pod should be set correctly and your probe should reflect it

[DaAmidza]

After I changed the namespace from production to default(without specifying any namespace) it works and the error didn't occur.Any idea?

[Patrick W]

specifying the namespace is required for ant kubectl command when the target resource is not in the default namespace. It should not affect the response code from the curl though, only whether the command is executed or not

[Patrick Barattin]

Hi, I have exactly the same problem. All the pod are in the default namespace. If I run the second command and I get a 200 response. But Readiness probe failed: Get http://10.48.8.29:80/wp-login.php: net/http: request canceled (Client.Timeout exceeded while awaiting headers) Any Ideas ?

[Patrick W]

What is the full command you ran? Make sure when you run the second command you are using the same path as the probe (add /wp-login.php)