Cannot connect to a mongodb service in a Kubernetes cluster

Question

I have a Kubernetes cluster on Google Cloud, I have a database service, which is running in front of a mongodb deployment. I also have a series of microservices, which are attempting to connect to that datastore.

However, they can't seem to find the host.

apiVersion: v1
kind: Service
metadata:
 labels:
   name: mongo
 name: mongo
spec:
 ports:
 - port: 27017
   targetPort: 27017
 selector:
   name: mongo

Here's my mongo deployment...

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mongo-deployment
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: mongo
    spec:
      containers:
      - image: mongo:latest
        name: mongo
        ports:
        - name: mongo
          containerPort: 27017
          hostPort: 27017
        volumeMounts:
          - name: mongo-persistent-storage
            mountPath: /data/db
      volumes:
        - name: mongo-persistent-storage
          gcePersistentDisk:
            pdName: mongo-disk
            fsType: ext4

And an example of one of my services...

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bandzest-artists
spec:
  replicas: 1
  template:
    metadata:
      labels:
        name: bandzest-artists
    spec:
      containers:
      - name: artists-container
        image: gcr.io/<omitted>/artists:41040e8
        ports: 
        - containerPort: 7000
        imagePullPolicy: Always
        env:
        - name: DB_HOST
          value: mongo
        - name: AWS_BUCKET_NAME
          value: <omitted>
        - name: AWS_ACCESS_KEY_ID
          value: <omitted>
        - name: AWS_SECRET_KEY
          value: <omitted> 

Solution

First, check that the service is created

kubectl describe svc mongo

You should see it show that it is both created and routing to your pod's IP. If you're wondering what your pod's IP is you can check it out via

kubectl get po | grep mongo

Which should return something like: mongo-deployment-<guid>-<guid>, then do

kubectl describe po mongo-deployment-<guid>-<guid>

You should make sure the pod is started correctly and says Running not something like ImagePullBackoff. It looks like you're mounting a volume from a gcePersistentDisk. If you're seeing your pod just hanging out in the ContainerCreating state it's very likely you're not mounting the disk correctly. Make sure you create the disk before you try and mount it as a volume.

If it looks like your service is routing correctly, then you can check the logs of your pod to make sure it started mongo correctly:

kubectl logs mongo-deployment-<guid>-<guid>

If it looks like the pod and logs are correct, you can exec into the pod and make sure mongo is actually starting and working: kubectl exec -it mongo-deployment-<guid>-<guid> sh

Which should get you into the container (Pod) and then you can try something like this to see if your DB is running.