Change log level on runtime for containers

Question

im using logrus for logging for out applications which run on K8S we have env variable which we can set the log-level and change it when we restart out application our applications is running with docker containers on k8s Now we want to change the log-level on runtime, i.e. don’t restart the container and change it when it’s running and with this we can change it from error to debug, I think this is legitimic request but didn’t find any reference or any open source which doing this, any idea?

package logs

import (
    "fmt"
    "os"

    "github.com/sirupsen/logrus"
)

const (
    AppLogLevel = “APP_LOG_LEVEL"
    DefLvl = "info"
)


var Logger *logrus.Logger


func NewLogger() *logrus.Logger {

    var level logrus.Level
    lvl := getLogLevel()
    // In case level doesn't set will not print any message
    level = logLevel(lvl)
    logger := &logrus.Logger{
        Out:   os.Stdout,
        Level: level,
    }
    Logger = logger
    return Logger
}

// use from env
func getLogLevel() string {
    lvl, _ := os.LookupEnv(AppLogLevel)
    if lvl != "" {
        return lvl
    }
    return DefLvl
}

func logLevel(lvl string) logrus.Level {

    switch lvl {
    case "debug":
        // Used for tracing
        return logrus.DebugLevel
    case "info":
        return logrus.InfoLevel
    case "error":
        return logrus.ErrorLevel
    case "fatal":
        return logrus.FatalLevel
    default:
        panic(fmt.Sprintf("the specified %s log level is not supported", lvl))
    }
}

I know how to change the log level but I need a way to infuance the logger to change the level

Solution

As a general Un*x statement, you cannot change an environment variable in a process after it has started. (You can setenv(3) your own environment, and you can specify a new process's environment when you execve(2) it, but once it's started, you can't change it again.)

This restriction carries through to higher levels. If you've docker run a container, its -e option to set an environment variable is one of the things you have to delete and recreate a container to change. The env: is one of the many immutable parts of a Kubernetes Pod specification; you also can't change it without deleting and recreating the pod.

If you've deployed the pod via a Deployment (and you really should), you can change the environment variable setting in the Deployment spec (edit the YAML file in source control and kubectl apply -f it, or directly kubectl edit). This will cause Kubernetes to start new pods with the new log value and shut down old ones, in that order, doing a zero-downtime update. Deleting and recreating pods like this is totally normal and happens whenever you want to, for example, change the image inside the deployment to have today's build.

If your application is capable of noticing changes to config files it's loaded (and it would have to be specially coded to do that) one other path that could work for you is to mount a ConfigMap into a container; if you change the ConfigMap contents, the files the container sees will change but it will not restart. I wouldn't go out of my way to write this just to avoid restarting a pod, though.

Comments

[JME]

Thanks, in this way I was able to change the env, but the log-level doesnt changed...

[Shahaf Shavit]

Seems like the log level will be updated only when you initialise a new logger. So you are dependant on restarting the container. Maybe add a mechanism that gets the log level from the env var for each log.

[JME]

Thanks, why we shouldn't be this in prod? sometimes when you re-start a pod you dont see the error in the new pod instance ...

[Todai]

It's just not good practice to have to restart pods in a production environment to change the log-level. However, your prod environment should be monitoring / logging the bare minimum for you to be able to at least get an idea of what's wrong. But maybe that's not a realistic scenario?

[Todai]

Bottom line is, you shouldn't have to remote onto a server to change an application's log-level. The log-level should be something static for each environment, so that you know it's not been tampered with. Especially in a clustered environment where logs have a tendency to disappear due to pods dying. How else are you going to have any form of confidence in what has broken?

[JME]

if customer is reporting a bug I can ask him to run some script/command to change the log level and in this way I've precise info of what happining in the pod, otherwise restart the pod will not reflect this info as it was before the restard due to many things such is user interacation with the app etc

[Todai]

Why would you ever ask a customer to run a script to find out about information in your production environment? If that's something that you need to debug an issue on your end, you have bigger problems than you are letting on.. :/