ImagePullBackOff error Google Kubernetes Engine

Question

I know a lot of people already had similar question, i read a few of them, but found nothing what actualy helped me so far.

I have a gitlab with private repo enabled, I also use Google Kubernetes Engine. I have a few Docker container in my private repo, and I want to deploy one of them to the Kubernetes Engine.

I have created a secret with kubectl create secret generic db-user-pass --from-file=./username.txt --from-file=./password.txt I also tried kubectl create secret docker-registry name --docker-server=registry.xy.z --docker-username=google --docker-password=xyz [email protected] Then I created my Deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: backend-test
labels:
app: 13371337
spec:
replicas: 1
template:
metadata:
labels:
app: 13371337
spec:
  containers:
  - name: backend
    image: registry.xy.z/group/project/backend:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 8080
  imagePullSecrets:
  - name: db-user-pass or name

Any ideas how to get it running?

Solution 1

Using kubectl create secret docker-registry name is a right way to provide credentials of private docker registry.

imagePullSecrets options looking good too, if you specify there a name of your docker-registry secret.

So, from Kubernetes path everything looking good.

Try to check events of the pod which will be created by Deployment, just find you pod by kubectl get pods and call kubectl describe pod $name_of_your_pod, you will see an actual reason why it cannot pull an image.

Also, if your depository is insecure or has self-signed certificate, try to follow that guide to allow docker daemon pulling image from there, that is an often reason of image pull failures.

Solution 2

In order to create a secret you can use the following command: (notice I gave it a name)

kubectl create secret docker-registry my_registry \
--docker-server=registry.xy.z \
--docker-username=google \
--docker-password=xyz \
[email protected]

or using yaml:

apiVersion: v1
kind: Secret
metadata:
  name: my_registry
type: Opaque
data:
  docker-server: registry.xy.z
  docker-username: google
  docker-password: xyz
  docker-email: [email protected]

and your deployment need to use the secret name:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: backend-test
labels:
app: 13371337
spec:
replicas: 1
template:
metadata:
labels:
app: 13371337
spec:
  containers:
  - name: backend
    image: registry.xy.z/group/project/backend:latest
    imagePullPolicy: Always
    ports:
    - containerPort: 8080
  imagePullSecrets:
  - name: my_registry

Notice: you must create the secret per namespace.

Comments

[eragon-2006]

certificate is from let´s encrypt, inside event i see: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedSync 2m (x568 over 2h) kubelet, gke-kubernetes-cluster-default-pool Error syncing pod

[Anton Kostenko]

It is not looking as a full output. Can you try to attach a full output of the command to somewhere like pastebin etc? Because message "Error syncing pod" mean just "we have some errors".

[eragon-2006]

this is pretty much what I did

[Ami Hollander]

besides that you named it as name which probably its reserved keyword, and your imagePullSecrets was incorrect

[Anton Kostenko]

Ok, thanks, I checked it. Can you please delete that pod (replicaset will create a new one), wait 1-2 minutes and send me a describe of that pod? And can you please tell me the version of the Kubernetes and which type installation it is (minukube etc)?

[eragon-2006]

name was just an example and my imagepullsecret is set to the correct word. sorry for my bad explanation at the beginning.

[eragon-2006]

pastebin.com/7CfY5GzV it is on google kubernetes engine in google cloud. Version is 1.8.7-gke.1

[Anton Kostenko]

Thanks, now info is full. Strange, but I don't see a pull secret here. Maybe it was missed somewhere. Can you please call kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "YOU_SECRET_NAME"}]}', where is YOUR_SECRET_NAME is a name of docker-registry secret (you created it earlier, as I understood) and remove current pod. That command will attack your secret to your default service account and K8s will always use that secret by default without any extra settings.

[eragon-2006]

did it, but getting still the same message with pod describe from the pod

[Anton Kostenko]

So, here I can see only 3 options - you have no secret in namespace of your pod (your namespace is 'default'), or it has a wrong type or name, or credentials is wrong. Try to carefully check it's type, name and existing by kubectl get secrets and kubectl describe secret $secret_name, it must has type 'kubernetes.io/dockercfg' and a correct name. And of course, try to login manually by username and password which you used for secret using docker login and try to pull your image by docker with that credentials.

[eragon-2006]

that is my secret: Name: name Namespace: default Labels: <none> Annotations: <none> Type: kubernetes.io/dockercfg Data ==== .dockercfg: 142 bytes Pulling with that credentials is possibly on my local machine.

[Anton Kostenko]

Secret looking good. Can you try to SSH to one of your GKE nodes (you can do it from Cloud Console) and check that credential again from there? Also, please check that the 'patch' command was exactly the same: kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "name"}]}'

[eragon-2006]

patch was the same and docker pull is working from cloud console

[Anton Kostenko]

Sorry, but I have no idea what to do next:) It has work if all credentials, names, namespaces etc. is correct. Only thing I can suggest - remove all resources (deployment, secret etc) and recreate it again checking each step.

[eragon-2006]

It works now, I deleted everything and did it again. Thank you!