How can I log in golang to a file with log rotation?

Question

I am trying to write a web application which will run on a remote server. I need to log to capture errors/debug/audit. I find that multiple logging packages are available for golang including the standard "log" package. However, I need to fulfill three requirements:

1. The log files need to be rotated
2. It applies to the included packages which use "log"
3. It needs to be cross-platform. Dev environment is Linux and needs to be deployed on Windows.

Solution 1

Though @Crast has given a very good answer, I want to also bring to the notice - lumberjack logger by Nate Finch which I ended up using.

Here is how to use it:

1. First, clone the lumberjack repository OR get it somehow.
2. Run the go install command on the folder.
3. Now import go's "log" package and "lumberjack package".

import ( "log" "github.com/natefinch/lumberjack" )

4. Now use it in your code like this:

Outside of main, declare your log variable.

var errLog *log.Logger

Inside main:

e, err := os.OpenFile("./foo.log", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0666)

if err != nil {
    fmt.Printf("error opening file: %v", err)
    os.Exit(1)
}
errLog = log.New(e, "", log.Ldate|log.Ltime)
errLog.SetOutput(&lumberjack.Logger{
    Filename:   "./foo.log",
    MaxSize:    1,  // megabytes after which new file is created
    MaxBackups: 3,  // number of backups
    MaxAge:     28, //days
})

Now as soon as the file size get 1MB, a new file is created to keep the previous logs with the current timestamps, and the new logs will continue to log into foo.log file. Also, I have created the file using os.OpenFile but you may not need it as lumberjack internally does it, but I preferred it that way. Thanks, hope it helps. Once again thanks to @Crast and NateFinch.

Solution 2

The best way to fulfill all your three requirements instead of creating an alternate logger struct, if you were satisfied using the base-level log.Log, is instead to set the output of the logger to your own io.Writer instance.

So basically what I'm going to do here is show an example where I create my own io.Writer:

import (
    "os"
    "sync"
    "time"
)

type RotateWriter struct {
    lock     sync.Mutex
    filename string // should be set to the actual filename
    fp       *os.File
}

// Make a new RotateWriter. Return nil if error occurs during setup.
func New(filename string) *RotateWriter {
    w := &RotateWriter{filename: filename}
    err := w.Rotate()
    if err != nil {
        return nil
    }
    return w
}

// Write satisfies the io.Writer interface.
func (w *RotateWriter) Write(output []byte) (int, error) {
    w.lock.Lock()
    defer w.lock.Unlock()
    return w.fp.Write(output)
}

// Perform the actual act of rotating and reopening file.
func (w *RotateWriter) Rotate() (err error) {
    w.lock.Lock()
    defer w.lock.Unlock()

    // Close existing file if open
    if w.fp != nil {
        err = w.fp.Close()
        w.fp = nil
        if err != nil {
            return
        }
    }
    // Rename dest file if it already exists
    _, err = os.Stat(w.filename)
    if err == nil {
        err = os.Rename(w.filename, w.filename+"."+time.Now().Format(time.RFC3339))
        if err != nil {
            return
        }
    }

    // Create a file.
    w.fp, err = os.Create(w.filename)
    return
}

You then create a RotateWriter and use log.SetOutput to set this writer (if other packages are using the standard logger instance) or alternately create your own instances using log.New to pass around.

I haven't solved the situation of when to call Rotate, I'll leave that to you to decide. It'd be fairly simple to trigger it based on time, or alternately do so after some amount of writes or some amount of bytes.

Solution 3

Here is a light-weighted logging package that supports log rotation and auto purging

https://github.com/antigloss/go/tree/master/logger

// logger.Init must be called first to setup logger
logger.Init("./log", // specify the directory to save the logfiles
            400, // maximum logfiles allowed under the specified log directory
            20, // number of logfiles to delete when number of logfiles exceeds the configured limit
            100, // maximum size of a logfile in MB
            false) // whether logs with Trace level are written down
logger.Info("Failed to find player! uid=%d plid=%d cmd=%s xxx=%d", 1234, 678942, "getplayer", 102020101)
logger.Warn("Failed to parse protocol! uid=%d plid=%d cmd=%s", 1234, 678942, "getplayer")

Solution 4

With logrus and lumberjack plugin for logrus:

package mypackage

import (
"io"
"os"
"path/filepath"
"time"

log "github.com/sirupsen/logrus"
"gopkg.in/natefinch/lumberjack.v2"
)

func SetupLogger() {

    lumberjackLogger := &lumberjack.Logger{
        // Log file abbsolute path, os agnostic
        Filename:   filepath.ToSlash("/path/to/log/file"), 
        MaxSize:    5, // MB
        MaxBackups: 10,
        MaxAge:     30,   // days
        Compress:   true, // disabled by default
    }

    // Fork writing into two outputs
    multiWriter := io.MultiWriter(os.Stderr, lumberjackLogger)

    logFormatter := new(log.TextFormatter)
    logFormatter.TimestampFormat = time.RFC1123Z // or RFC3339
    logFormatter.FullTimestamp = true

   log.SetFormatter(logFormatter)
   log.SetLevel(log.InfoLevel)
   log.SetOutput(multiWriter)
}

Use this function at early stage of your program (maybe inside init).

Import log "github.com/sirupsen/logrus" in any file and log with:

log.Info("some message")

Solution 5

We can also use a lib https://github.com/lestrrat/go-file-rotatelogs to achieve the same. It provides option to set

Max Age
Log Rotation Time

It can be hooked to any kind of logger too.

Source: https://golangbyexample.com/go-logger-rotation/

View more solutions

Comments

[klobucar]

Can you clarify what rotation is a little more? Is it on program start, or when the files reach a specific size, etc?

[elithrar]

For posterity: I would highly recommend lumberjack (github.com/natefinch/lumberjack) - which hooks into the standard library's log package via log.SetOutput and handles log rotation, max sizes and retaining backups.

[Dave C]

I'd probably just us sync.Mutex rather than sync.RWMutex since (A) the log package already handles serializing writes so supporting multiple callers of RotateWriter.Write doesn't get you anything and (B) unless perhaps you use os.O_APPEND, doing concurrent writes to a file is undefined/bad.

[Nilambar Sharma]

Link only answers are not quite recommended. Please provide some examples with explanation.

[lethalman]

@Dave Even with O_APPEND writes are not guaranteed to be atomic unless the written data is under PIPE_BUF.

[f0ster]

when renaming your file, you can't concatenate a string with a rune, so you would have to use "." (to build without error)

[jame2981]

log.Logger.Writer use any std logger.

[greenthunder]

I've tried your answer above for log rolling but it didn't work. The log file stopped at specified MaxSize and stopped logging without rolling it. What am I missing?

[kinshuk4]

@greenthunder did you set MaxBackups more than 1?

[Sharon Katz]

Works great for me. I tested it with several threads running stress logging and it looks good.

[farhany]

Works perfectly. Thanks for mentioning that package. Though I learned something important from the @Crast by trying out his code. Nothing beats experimenting with actual code.

[Muhammad Hewedy]

A simple tee writer won't harm (to run inside ide) gist.github.com/mhewedy/63d243e8ef84ac23c740aabba9ac3f32

[SomeGuyWhoCodes]

this is not working for me on Mac, is it only for linux?

[kinshuk4]

@SomeGuyWhoCodes It should work for all OSes. Try setting gopath

[Ravi]

@antigloss Though I like your library, it is under LGPL-3.0. Might not be suitable for closed source commercial deployments.