Organizing Environment Variables Golang

Question

In Node.js I use the nconf module to house environment variables like S3 keys, GCM keys, etc for each of my projects.

I haven't been able to find a similar solution in Go.

What are the generally accepted tools to help manage environment variables for each Go project?

Thanks in advance.

Solution 1

I did some reading on this a while back when I was getting started with Go. According to this link, http://peter.bourgon.org/go-in-production/, they recommend using CLI flags (parameters) instead of environment vars - they even convert environment vars to flags to their CLI apps.

It took some getting used to; but, I really do see the advantages of going pure CLI flags between development, staging and production environments - having specific scripts for each environment.

For example, here's a little web app I wrote recently:

// global flags
var isdebug bool
var port int
var cert string
var key string
var dbdsn string
var dbmaxidle int
var dbmaxopen int
var imguri string

// init is the entry point for the entire web application.
func init() {

    log.Println("Starting wwwgo ...")

    // setup the flags
    //flag.StringVar(&host, "host", "", "Specify a host to redirect to. Use this to redirect all traffic to a single url.")
    flag.IntVar(&port, "port", 8080, "Specify the port to listen to.")
    flag.BoolVar(&isdebug, "isdebug", false, "Set to true to run the app in debug mode.  In debug, it may panic on some errors.")
    flag.StringVar(&cert, "cert", "", "Enables listening on 443 with -cert and -key files specified.  This must be a full path to the certificate .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.")
    flag.StringVar(&key, "key", "", "Enables listening on 443 with -cert and -key files specified.  This must be a full path to the key .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.")
    flag.StringVar(&dbdsn, "dbdsn", "root:root@tcp(localhost:3306)/dev_db?timeout=5s&tls=false&autocommit=true", "Specifies the MySql DSN connection.")
    flag.IntVar(&dbmaxidle, "dbmaxidle", 0, "Sets the database/sql MaxIdleConns.")
    flag.IntVar(&dbmaxopen, "dbmaxopen", 500, "Sets the database/sql MaxOpenConns.")
    flag.StringVar(&imguri, "imguri", "/cdn/uploads/", "Set this to the full base uri of all images, for example on a remote CDN server or local relative virtual directory.")
    flag.Parse()

    // log our flags
    if isdebug != false {
        log.Println("DEBUG mode enabled")
    }
    if cert != "" && key != "" {
        log.Println("Attempting SSL binding with supplied cert and key.")
    }
    if dbdsn != "" {
        log.Printf("Using default dbdsn: %s", dbdsn)
    }

    ...
}

This really becomes nice in staging/production environments.

A quick ./wwwgo -h for "what the heck was that parameter?" gives you full documentation:

admin@dev01:~/code/frontend/src/wwwgo [master]$ ./wwwgo -h
Usage of ./wwwgo:
  -cert="": Enables listening on 443 with -cert and -key files specified.  This must be a full path to the certificate .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.
  -dbdsn="root:root@tcp(localhost:3306)/dev_db?timeout=5s&tls=false&autocommit=true": Specifies the MySql DSN connection.
  -dbmaxidle=0: Sets the database/sql MaxIdleConns.
  -dbmaxopen=500: Sets the database/sql MaxOpenConns.
  -imguri="/cdn/uploads/": Set this to the full base uri of all images, for example on a remote CDN server or local relative virtual directory.
  -isdebug=false: Set to true to run the app in debug mode.  In debug, it may panic on some errors.
  -key="": Enables listening on 443 with -cert and -key files specified.  This must be a full path to the key .pem file. See http://golang.org/pkg/net/http/#ListenAndServeTLS for more information.
  -port=8080: Specify the port to listen to.

Very nice to have many options at CLI, and no documentation required - it's built into the flags package.

You can clearly see the defaults immediately.

With this type of documentation, I tend to setup all the defaults for common "development environments" that the team uses. We all have root/root access to our local databases. We all are using port 8080 for this particular web app during development, etc. That way, you just have to run:

go build
./wwwgo

And the app runs with all defaults - defaults that are documented. In production, just override the defaults. The built-in flag parsers will panic the application if any parameters are in the wrong format, which is also very nice.

Solution 2

I would strongly recommend using github.com/namsral/flag instead. It's like the built in flag except you can also supply the parameters via environment variables.

For example, suppose you have this code:

package main

import "fmt"
import "github.com/namsral/flag"

func main() {
    var port = 3000
    flag.IntVar(&port, "port", port, "Port number")
    flag.Parse()
    fmt.Println("You seem to prefer", port)
}

Then you can supply the values with either a command line option or an environment variable:

:~/dev/GO$ go run dummy.go
You seem to prefer 3000
:~/dev/GO$ go run dummy.go -port=1234
You seem to prefer 1234
:~/dev/GO$ PORT=4321 go run dummy.go
You seem to prefer 4321
:~/dev/GO$ PORT=4321 go run dummy.go -port=5555
You seem to prefer 5555

This might matter when it's hard to supply command line args. For example, if you use gin to automatically restart a server you have no way to supply command line arguments since gin is just calling go run on the main code without any arguments passed along.

Solution 3

We have used this for a large scale microservice application. This doesn't involve use of any third party libraries, just plain go lang using reflection. It is very simple to use and follows DRY principle. Adding new environment variables and configuring defaults is quite easy. You can also set loading from a config file as a fallback by changing just 2 lines of code. Checkout github page for more info.

package config

import (
    "fmt"
    "os"
    "reflect"
)

/* Tag names to load configuration from environment variable */
const (
    ENV     = "env"
    DEFAULT = "default"
)

type Configuration struct {
    Port        string `env:"port" default:"3009"`
    MongoURL    string `env:"MongoUrl" default:"mongodb://localhost:27017/test"`
    UserService string `env:"UserService" default:"http://localhost:3005"`
    AuthService string `env:"AuthService" default:"http://localhost:3050"`
    Debug       string `env:"Debug" default:"true"`
}

/* Non-exported instance to avoid accidental overwrite */
var serviceConfig Configuration

func setConfig() {
    // ValueOf returns a Value representing the run-time data
    v := reflect.ValueOf(serviceConfig)
    for i := 0; i < v.NumField(); i++ {
        // Get the field tag value
        tag := v.Type().Field(i).Tag.Get(ENV)
        defaultTag := v.Type().Field(i).Tag.Get(DEFAULT)

        // Skip if tag is not defined or ignored
        if tag == "" || tag == "-" {
            continue
        }
        a := reflect.Indirect(reflect.ValueOf(serviceConfig))
        EnvVar, Info := loadFromEnv(tag, defaultTag)
        if Info != "" {
            fmt.Println("Missing environment configuration for '" + a.Type().Field(i).Name + "', Loading default setting!")
        }
        /* Set the value in the environment variable to the respective struct field */
        reflect.ValueOf(&serviceConfig).Elem().Field(i).SetString(EnvVar)
    }
}

func loadFromEnv(tag string, defaultTag string) (string, string) {
    /* Check if the tag is defined in the environment or else replace with default value */
    envVar := os.Getenv(tag)
    if envVar == "" {
        envVar = defaultTag
        /* '1' is used to indicate that default value is being loaded */
        return envVar, "1"
    }
    return envVar, ""
}

/*GetConfiguration :Exported function to return a copy of the configuration instance */
func GetConfiguration() Configuration {
    return serviceConfig
}

func init() {
    setConfig()
    fmt.Printf("Service configuration : %+v\n ", serviceConfig)
}

Solution 4

Well I prefer go-arg for setting environment variables. It is easy to use and has nice features.

For example:

package configs

import (
    "fmt"
    "github.com/alexflint/go-arg"
)

type config struct {
     DBHost string `arg:"env:DBHost, -D, --dbhost" help:"Host of the database" placeholder:"DBHost"`
}

var Config config

func main(){

  arg.MustParse(&Config)
  fmt.Println(Config.DBHost)

}

With this library either you can take the variable from your env or you can pass through args.

export DBHost=127.0.0.1

or

go run ./main.go --dbhost=127.0.0.1

View more solutions

Comments

[Peter Bengtsson]

I rewrote my hackish messing with environment variables to use flags but then suddenly I can no longer use gin :(

[Roy Lee]

@eduncan911 do you have sample scripts for each environment?

[eduncan911]

@Roylee sample scripts for each environment? if u are asking if I keep scripts checked into the repo as samples for staging/prod, the answer is not really. it's up to each environment to set the correct parameters and track them accordingly in each system (e.g. Ansible, Chef/Knife, Salt/Pillard, etc)

[Roy Lee]

@eduncan911 ah I see, so you should never really check-in your scripts. Sorry for the vague question, let me give you the correct context, I was actually referring to this: "But even so, we use a start script to convert environment variables to flags" -- (Peter Bourgon, Configuration). What does he meant by that? And what would a sample script of that look like?

[eduncan911]

Yep, hence the whole "self-documentation" aspect - it's documented with the myapp -h run, which prints the entire params list and its current Default. Now with that said, yes I do check in some skeleton scripts - usually in the form of a Makefile or Dockerfile that passes some defaults in for specific environments. But even then, I still fallback on Sensible Defaults in the app itself. Each custom environment will need to set it, or get set dynamically. Think at scale, where you have 100s of instances running - u want to pass in those parameters dynamically as new ones are spun up.

[eduncan911]

@Roylee to answer what Peter meant is most likely a one-off script per application that reads X, Y, Z environment vars and passes them as parameters to the Go app. That way, you can enforce the "all apps use CLI parameters, not environment" across the dev and deployment environments, while still supporting older legacy stuff that may be set in Environment vars (until they are updated, or where it isn't possible). Doing it this way keeps from having to code two things in your app: reading from CLI and ENVs. Instead, 1 way coded in apps - with some wrapper script going from ENV -> CLI params.

[Roy Lee]

@eduncan911 Thanks! That's really detailed. :) How would that apply to docker?

[eduncan911]

@Roylee you know u are speaking my language... Go, Docker. :) With Dockerfile (or Compose), it's a different ballgame: you typically do not pass vars via command line but instead with ENVs. I have a project right now of where I am using a service discovery tool that has a built-in key-value store, called Consul. I am attempting to use Consul to manage all the configuration for the services, with a local process that retrieves the values and creates the local config vars - and even calls some Go binary within the container.