Where are the Google Health check logs

Question

I am running a GCE instance and on it I have a Kubernetes cluster running. An HTTP Load Balancer is also configured to probe the health of the a Kubernetes service running on this instance. But I constantly get this error:

Error: Server Error

The server encountered a temporary error and could not complete your request. Please try again in 30 seconds.

So I wonder if the health check associated with the load balancer is running/configured correctly.

Where can I find or how can I enable the log files of such a health checker?

The instance is running Debian.

Solution 1

I was troubleshooting a similar problem, the environment was including HTTPS load balancer and Managed Instance group

Searched the available documentation no trace to access that logs, finally raised a support ticket with Google Cloud team, they replied that Health checks logs are not available for public access. Google team have their internal tools to verify health check failure logs and suggested their recommendation accordingly

We have the option to check health checks using gcloud command as follows but do not serve the log requirements we expect

$ gcloud alpha compute http-health-checks 

$ gcloud beta compute http-health-checks

Solution 2

Health check logs are now available (in Beta, at the time of writing) via Cloud Logging.

(Note: Legacy Health checks are not supported)

In Cloud Logging (formerly known as Stackdriver) use a filter such as this:

logName="projects/<project-id>/logs/compute.googleapis.com%2Fhealthchecks"

Check (no pun intended) out the Healthcheck Logging Documentation for more information.

1. Take you to the Cloud Logging, Log Viewer
2. Pre-populate the filter to show all recent health check logs:

https://cloud.google.com/console/logs/query?query=logName:%22logs%2Fcompute.googleapis.com%252Fhealthchecks%22

Comments

[Carlos]

Were you able to solve this issue? If so please consider posting an answer. One thing to check is that the appropriate firewall rules are setup properly. The HTTP log viewer might also unveil additional information. This blog might also help.

[ctrlbrk]

A year later, still no way to find out why Google says health of a node is bad?

[internetdotcom]

Two more years and, to confirm, there's still no way. There are logs in Stackdriver/Cloud Logging, but all they show is when the health check is added, updated, or removed, not when it's run and certainly not its result.

[internetdotcom]

This no longer works, at least not in August 2021. No logs appear when using that logName filter and the only logs that appear when selecting GCE Health Check in Basic mode describe the health check being added, updated, or removed.