Getting "x509: certificate signed by unknown authority" in GKE on pulling image (a private registry) when a pod is created

Question

I generated a CA certificate, then issued a certificate based on it for a private registry, that located in the same GKE cluster. Put the server certificates to the private registry and the CA certificate to all GKE nodes and run:

update-ca-certificates && systemctl restart docker

Images are building and putting into the private registry without problems. When a pod tries to pull the an image from the repository I get an error:

x509: certificate signed by unknown authority

Also I tried to put the CA certificate to the docker certs.d directory (10.3.240.100:3000 — the IP address of the private registry) and restart the docker on each node of the GKE cluster, but it doesn't help too:

/etc/docker/certs.d/10.3.240.100:3000/ca.cert

How to solve this problem? Am I understand correctly that the GKE nodes' docker is responsible for pulling images when creating a pod?

Solution

I found a solution. The problem is actual for Kubernetes version 1.19+ and COS/Ubuntu images based on containerd for GKE nodes. Before the 1.19 version Kubernetes used to use Docker for building images, but now it uses containerd. More details could be found in the official Google Cloud documentation.

You need to create and put an CA certificate to each GKE node. Here you can find an answer how to do it correctly https://stackoverflow.com/a/67724696/3319341.

But for containerd solution you should replace command

nsenter --target 1 --mount systemctl restart docker

by

nsenter --target 1 --mount systemctl restart containerd

A more detailed answer: https://stackoverflow.com/a/67990395/3319341

Comments

[Max Koshel]

Sorry, but your answer is useless. I've already done it, as I wrote in the topic

[Max Koshel]

Thanks. Yes, it' a correct solution if a cluster is based on Docker. But as it turns out, mine is based on containerd. I found a solution and add my own answer to this question.