How to tail all logs in a kubernetes cluster

Question

I tried this command:

kubectl logs --tail

I got this error/help output:

Error: flag needs an argument: --tail


Aliases:
logs, log

Examples:
  # Return snapshot logs from pod nginx with only one container
  kubectl logs nginx

  # Return snapshot logs for the pods defined by label app=nginx
  kubectl logs -lapp=nginx

  # Return snapshot of previous terminated ruby container logs from pod web-1
  kubectl logs -p -c ruby web-1

  # Begin streaming the logs of the ruby container in pod web-1
  kubectl logs -f -c ruby web-1

  # Display only the most recent 20 lines of output in pod nginx
  kubectl logs --tail=20 nginx

  # Show all logs from pod nginx written in the last hour
  kubectl logs --since=1h nginx

  # Return snapshot logs from first container of a job named hello
  kubectl logs job/hello

  # Return snapshot logs from container nginx-1 of a deployment named nginx
  kubectl logs deployment/nginx -c nginx-1

ummm I just want to see all the logs, isn't this a common thing to want to do? How can I tail all the logs for a cluster?

Solution 1

If you don't mind using a third party tool, kail does exactly what you're describing.

Streams logs from all containers of all matched pods. [...] With no arguments, kail matches all pods in the cluster.

Solution 2

kail from the top answer is Linux and macOS only, but Stern also works on Windows.

It can do pod matching based on e.g. a regex match for the name, and then can follow the logs.

To follow ALL pods without printing any prior logs from the default namespace you would run e.g.:

stern ".*" --tail 0

For absolutely everything, incl. internal stuff happening in kube-system namespace:

stern ".*" --all-namespaces --tail 0

Alternatively you could e.g. follow all login-.* containers and get some context with

stern "login-.*" --tail 25

Solution 3

I would recommend using a nice bash script named kubetail.

You can just download the bash script and add it to in your project and run for example:

$ ./some-tools-directory/kubetail.sh --selector app=user --since 10m

To see all pods with the label app=user.

Notice the nice display of colors per pod:

---

(*) Run ./tools/kubetail.sh -h to see some nice execution options.

kubetail.sh <search term> [-h] [-c] [-n] [-t] [-l] [-d] [-p] [-s] [-b] [-k] [-v] [-r] [-i] -- tail multiple Kubernetes pod logs at the same time

where:
    -h, --help              Show this help text
    -c, --container         The name of the container to tail in the pod (if multiple containers are defined in the pod).
                            Defaults to all containers in the pod. Can be used multiple times.
    -t, --context           The k8s context. ex. int1-context. Relies on ~/.kube/config for the contexts.
    -l, --selector          Label selector. If used the pod name is ignored.
    -n, --namespace         The Kubernetes namespace where the pods are located (defaults to "default")
    -f, --follow            Specify if the logs should be streamed. (true|false) Defaults to true.
    -d, --dry-run           Print the names of the matched pods and containers, then exit.
    -p, --previous          Return logs for the previous instances of the pods, if available. (true|false) Defaults to false.
    -s, --since             Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to 10s.
    -b, --line-buffered     This flags indicates to use line-buffered. Defaults to false.
    -e, --regex             The type of name matching to use (regex|substring)
    -j, --jq                If your output is json - use this jq-selector to parse it.
                            example: --jq ".logger + \" \" + .message"
    -k, --colored-output    Use colored output (pod|line|false).
                            pod = only color pod name, line = color entire line, false = don't use any colors.
                            Defaults to line.
    -z, --skip-colors       Comma-separated list of colors to not use in output
                            If you have green foreground on black, this will skip dark grey and some greens -z 2,8,10
                            Defaults to: 7,8
        --timestamps        Show timestamps for each log line
        --tail              Lines of recent log file to display. Defaults to -1, showing all log lines.
    -v, --version           Prints the kubetail version
    -r, --cluster           The name of the kubeconfig cluster to use.
    -i, --show-color-index  Show the color index before the pod name prefix that is shown before each log line.
                                                Normally only the pod name is added as a prefix before each line, for example "[app-5b7ff6cbcd-bjv8n]",
                                                but if "show-color-index" is true then color index is added as well: "[1:app-5b7ff6cbcd-bjv8n]".
                            This is useful if you have color blindness or if you want to know which colors to exclude (see "--skip-colors").
                                Defaults to false.

examples:
    kubetail.sh my-pod-v1
    kubetail.sh my-pod-v1 -c my-container
    kubetail.sh my-pod-v1 -t int1-context -c my-container
    kubetail.sh '(service|consumer|thing)' -e regex
    kubetail.sh -l service=my-service
    kubetail.sh --selector service=my-service --since 10m
    kubetail.sh --tail 1

Solution 4

The only thing you can do is to get logs of multiple pods using label selectors like this:

kubectl logs -f -l app=nginx -l app=php 

For getting all logs of the entire cluster you have to setup centralized log collection like Elasticsearch, Fluentd and Kibana. Simplest way to do it is installation using Helm charts like described here: https://linux-admin.tech/kubernetes/logging/2018/10/24/elk-stack-installation.html

Solution 5

I have hardly ever seen anyone pulling all logs from entire clusters, because you usually either need logs to manually search for certain issues or follow (-f) a routine, or collect audit information, or stream all logs to a log sink to have them prepared for monitoring (e.g. prometheus).

However, if there's a need to fetch all logs, using the --tail option is not what you're looking for (tail only shows the last number of lines of a certain log source and avoids spilling the entire log history of a single log source to your terminal).

For kubernetes, you can write a simple script in a language of your choice (bash, Python, whatever) to kubectl get all --show-all --all-namespaces and iterate over the pods to run kubectl -n <namespace> logs <pod>; but be aware that there might be multiple containers in a pod with individual logs each, and also logs on the cluster nodes themselves, state changes in the deployments, extra meta information that changes, volume provisioning, and heaps more.

That's probably the reason why it's quite uncommon to pull all logs from an entire cluster and thus there's no easy (shortcut) way to do so.

View more solutions

Comments

[Alexander Mills]

Everything from the cluster, I mean everything

[OneCricketeer]

I find the "common" thing for such large scale logging is to setup something like Filebeat+Elasticsearch or Splunk, rather than use CLI. elastic.co/blog/…

[Janne Enberg]

kail is unfortunately not the most cross-platform option out there and only works for macOS and Linux, the question does not specify an OS so it could well be Windows.