Kubernetes HTTP liveness probe fails with "connection refused" even though URL works without it
Solution 1
For anyone interested I've managed to solve this issue.
I was getting a 301 redirect response from Wordpress due to Wordpress forcing my domain name example.com. Solved this issue by disabling Wordpress canonical redirection feature for the specific request http://POD_IP:8080/index.php.
Here's how:
Added the Pod IP address as an environment variable:
- name: K8S_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
Created a Wordpress plugin with a custom redirect_canonical filter that prevents Wordpress from redirecting http://POD_IP:8080/index.php:
<?php
/**
* Plugin Name: Kubernetes Liveness Probe Exception
*/
add_filter('redirect_canonical', function($redirect_url, $requested_url) {
$K8S_POD_IP = getenv('K8S_POD_IP');
$LIVENESS_URL = "http://" . $K8S_POD_IP . ":8080/index.php";
if ($requested_url == $LIVENESS_URL) {
return $requested_url;
}
return $redirect_url;
}, 10, 2);
Solution 2
Just to give another way - wordpress will try to redirect because you are missing the X-Forwarded http headers that you should have if you are connecting to wordpress via a proxy.
Something like this works without the need for custom php:
livenessProbe:
initialDelaySeconds: 10
httpGet:
path: /
port: 8080
httpHeaders:
- name: X-Forwarded-Proto
value: https
- name: X-Forwarded-Host
value: www.your-wordpress-domain-here.com
- name: Host
value: www.your-wordpress-domain-here.com
timeoutSeconds: 10
periodSeconds: 15
failureThreshold: 5
Solution 3
10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
You're getting a 301 redirect response from Apache. You need to be getting a 2xx to be considered a success.
To check what path it's redirecting you to try curl --location --verbose http://url/index.php
If you can't figure out a way around Apache or Wordpress's redirection, you could consider a tcpSocket probe rather than httpGet
iamcryptoki
Updated on June 05, 2022Comments
-
iamcryptoki almost 2 years
ENVIRONMENT:
Kubernetes version: v1.16.3 OS: CentOS 7 Kernel: Linux k8s02-master01 3.10.0-1062.4.3.el7.x86_64 #1 SMP Wed Nov 13 23:58:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
WHAT HAPPENED:
I have a Wordpress Deployment running a container built from a custom Apache/Wordpress image. The image exposes port 8080 instead of 80 (Dockerfile below). The Pod is exposed to the world through Traefik reverse proxy. Everything works fine without any liveness or readiness checks. Pod gets ready and Wordpress is accessible from https://www.example.com/.
I tried adding liveness and readiness probes and they both repeatedly fail with "connection refused". When I remove both probes and reapply the Deployment, it works again. It works until the probe hits the failure threshhold, at which point the container goes into an endless restart loop and becomes unaccessible.
POD EVENTS:
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned development/blog-wordpress-5dbcd9c7c7-kdgpc to gg-k8s02-worker02 Normal Killing 16m (x2 over 17m) kubelet, gg-k8s02-worker02 Container blog-wordpress failed liveness probe, will be restarted Normal Created 16m (x3 over 18m) kubelet, gg-k8s02-worker02 Created container blog-wordpress Normal Started 16m (x3 over 18m) kubelet, gg-k8s02-worker02 Started container blog-wordpress Normal Pulled 13m (x5 over 18m) kubelet, gg-k8s02-worker02 Container image "wordpress-test:test12" already present on machine Warning Unhealthy 8m17s (x35 over 18m) kubelet, gg-k8s02-worker02 Liveness probe failed: Get http://10.244.3.83/: dial tcp 10.244.3.83:80: connect: connection refused Warning BackOff 3m27s (x27 over 11m) kubelet, gg-k8s02-worker02 Back-off restarting failed container
POD LOGS:
WordPress not found in /var/www/html - copying now... WARNING: /var/www/html is not empty! (copying anyhow) Complete! WordPress has been successfully copied to /var/www/html AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.244.3.83. Set the 'ServerName' directive globally to suppress this message [Wed Dec 11 06:39:07.502247 2019] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.11 configured -- resuming normal operations [Wed Dec 11 06:39:07.502323 2019] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND' 10.244.3.1 - - [11/Dec/2019:06:39:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16" 10.244.3.1 - - [11/Dec/2019:06:39:33 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16" 10.244.3.1 - - [11/Dec/2019:06:39:48 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16" 10.244.3.1 - - [11/Dec/2019:06:40:03 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16" 10.244.3.1 - - [11/Dec/2019:06:40:18 +0000] "GET /index.php HTTP/1.1" 301 264 "-" "kube-probe/1.16"
DOCKERFILE ("wordpress-test:test12"):
FROM wordpress:5.2.4-apache RUN sed -i 's/Listen 80/Listen 8080/g' /etc/apache2/ports.conf; RUN sed -i 's/:80/:8080/g' /etc/apache2/sites-enabled/000-default.conf; # RUN sed -i 's/#ServerName www.example.com/ServerName localhost/g' /etc/apache2/sites-enabled/000-default.conf; EXPOSE 8080 CMD ["apache2-foreground"]
DEPLOYMENT:
apiVersion: apps/v1 kind: Deployment metadata: name: blog-wordpress namespace: development labels: app: blog spec: selector: matchLabels: app: blog tier: wordpress replicas: 4 strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 2 template: metadata: labels: app: blog tier: wordpress spec: volumes: - name: blog-wordpress persistentVolumeClaim: claimName: blog-wordpress containers: - name: blog-wordpress # image: wordpress:5.2.4-apache image: wordpress-test:test12 securityContext: runAsUser: 65534 allowPrivilegeEscalation: false capabilities: add: - "NET_ADMIN" - "NET_BIND_SERVICE" - "SYS_TIME" resources: requests: cpu: "250m" memory: "64Mi" limits: cpu: "500m" memory: "128Mi" ports: - name: liveness-port containerPort: 8080 readinessProbe: initialDelaySeconds: 15 httpGet: path: /index.php port: 8080 timeoutSeconds: 15 periodSeconds: 15 failureThreshold: 5 livenessProbe: initialDelaySeconds: 10 httpGet: path: /index.php port: 8080 timeoutSeconds: 10 periodSeconds: 15 failureThreshold: 5 env: # Database - name: WORDPRESS_DB_HOST value: blog-mysql - name: WORDPRESS_DB_NAME value: wordpress - name: WORDPRESS_DB_USER valueFrom: secretKeyRef: name: blog-mysql key: username - name: WORDPRESS_DB_PASSWORD valueFrom: secretKeyRef: name: blog-mysql key: password - name: WORDPRESS_TABLE_PREFIX value: wp_ - name: WORDPRESS_AUTH_KEY valueFrom: secretKeyRef: name: blog-wordpress key: auth-key - name: WORDPRESS_SECURE_AUTH_KEY valueFrom: secretKeyRef: name: blog-wordpress key: secure-auth-key - name: WORDPRESS_LOGGED_IN_KEY valueFrom: secretKeyRef: name: blog-wordpress key: logged-in-key - name: WORDPRESS_NONCE_KEY valueFrom: secretKeyRef: name: blog-wordpress key: nonce-key - name: WORDPRESS_AUTH_SALT valueFrom: secretKeyRef: name: blog-wordpress key: auth-salt - name: WORDPRESS_SECURE_AUTH_SALT valueFrom: secretKeyRef: name: blog-wordpress key: secure-auth-salt - name: WORDPRESS_LOGGED_IN_SALT valueFrom: secretKeyRef: name: blog-wordpress key: logged-in-salt - name: WORDPRESS_NONCE_SALT valueFrom: secretKeyRef: name: blog-wordpress key: nonce-salt - name: WORDPRESS_CONFIG_EXTRA value: | define('WPLANG', 'fr_FR'); define('WP_CACHE', false); define('WP_MEMORY_LIMIT', '64M'); volumeMounts: - name: blog-wordpress mountPath: "/var/www/html/wp-content"
DEPLOYMENT SERVICE:
apiVersion: v1 kind: Service metadata: name: blog-wordpress namespace: development labels: app: blog spec: ports: - protocol: TCP port: 80 targetPort: 8080 selector: app: blog tier: wordpress type: ClusterIP
TRAEFIK INGRESSROUTE:
## # HTTP ## apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: blog namespace: development spec: entryPoints: - http routes: - match: Host(`example.com`) kind: Rule services: - name: blog-wordpress port: 80 middlewares: - name: redirect-to-https namespace: kube-system --- ## # HTTPS ## apiVersion: traefik.containo.us/v1alpha1 kind: IngressRoute metadata: name: blog-https namespace: development spec: entryPoints: - https routes: - match: Host(`example.com`) && PathPrefix(`/`) kind: Rule services: - name: blog-wordpress port: 80 tls: certResolver: letsencrypt
Thank you!
-
acid_fuji over 4 yearsFrom your
Pod
event i can see that the probe is being checked on port 80 (dial tcp 10.244.3.83:80: connect: connection refused
) while in deployment it`s shows 8080. Can you verify that?
-
-
iamcryptoki over 4 yearsThanks! Curl returned the following header:
X-Redirect-By: WordPress
. Wordpress redirects the probe request. -
iamcryptoki over 4 yearsIs the tcpSocket probe revelant for a Wordpress deployment? @hipyhop
-
iamcryptoki over 4 yearsWP still redirects 8080 to 80.
-
Farcaller over 4 yearsIn general WP is very nitpicky about the serving port and it's always better to match the ports and serve it from
80
. You can hack around that with something like$_SERVER['PORT'] = '80'
in your wp-config.php to force WP to think it's served from the canonical port. -
Michael Mayo over 2 yearsWorked like a charm! This should really be the accepted answer IMO.