Kubernetes + Minikube - How to see all stdout output?

Question

I'm running a Ruby app on Kubernetes with Minikube.

However, whenever I look at the logs I don't see the output I would have seen in my terminal when running the app locally.

I presume it's because it only shows stderr?

What can I do to see all types of console logs (e.g. from puts or raise)?

On looking around is this something to do with it being in detached mode - see the Python related issue: Logs in Kubernetes Pod not showing up

Thanks.

=

As requested - here is the deployment.yaml

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: sample
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: sample
    spec:
      containers:
        - name: sample
          image: someregistry
          imagePullPolicy: Always
          command: ["/bin/sh","-c"]
          args: ["bundle exec rake sample:default --trace"]
          envFrom:
          - configMapRef:
              name: sample
          - secretRef:
              name: sample
          ports:
            - containerPort: 3000
      imagePullSecrets:
        - name: regsecret

Solution 1

As shown in this article, kubectl logs pod apod should show you stdout and stderr for a pod deployed in a minikube.

By default in Kubernetes, Docker is configured to write a container's stdout and stderr to a file under /var/log/containers on the host system

Kubernetes adds:

There are two types of system components: those that run in a container and those that do not run in a container.
For example:

• The Kubernetes scheduler and kube-proxy run in a container.
• The kubelet and container runtime, for example Docker, do not run in containers.

And:

• On machines with systemd, the kubelet and container runtime write to journald.
• If systemd is not present, they write to .log files in the /var/log directory.

Similarly to the container logs, system component logs in the /var/log directory should be rotated.
In Kubernetes clusters brought up by the kube-up.sh script, those logs are configured to be rotated by the logrotate tool daily or once the size exceeds 100MB.

Solution 2

I presume it's because it only shows stderr?

Not really, unless something specific is disabled in your container or pod spec. I assume you are using Docker so then the default it's to output stdout and stderr and that's what you see when you do a kubectl logs <pod-name>

What can I do to see all types of console logs (e.g. from puts or raise)?

You should see them in the container logs. It would help to post your pod or deployment definition.

Comments

[userMod2]

I tried kubectl logs mypodname-7f74cb6f54-hr67v -n the-namespace and it shows me the same log as in Minikube i.e. not including stdout

[VonC]

@userMod2 is there any permission issue, as in github.com/moby/moby/issues/19616#issuecomment-174492543? Can you try a docker exec to test it out?

[userMod2]

added the deployment.yaml above

[userMod2]

Also when you say container logs - I presume pod logs is the same? i.e. kubectl logs mypodname-7f74cb6f54-hr67v -n my-namespace

[userMod2]

No permissions issue.

[VonC]

@userMod2 Would a kubectl describe pod aPod -n anamespace returns any clues?