Golang read request body multiple times

Question

I am writing my own logginMiddleware. Basically, I need to log body of the request and the response. The problem that I faced is that when I read body, it becomes empty and I cannot read it twice. I understand that it happens because it is of type ReadCloser. Is there a way to rewind body to the beginning?

Solution 1

Inspecting and mocking request body

When you first read the body, you have to store it so once you're done with it, you can set a new io.ReadCloser as the request body constructed from the original data. So when you advance in the chain, the next handler can read the same body.

One option is to read the whole body using ioutil.ReadAll(), which gives you the body as a byte slice.

You may use bytes.NewBuffer() to obtain an io.Reader from a byte slice.

The last missing piece is to make the io.Reader an io.ReadCloser, because bytes.Buffer does not have a Close() method. For this you may use ioutil.NopCloser() which wraps an io.Reader, and returns an io.ReadCloser, whose added Close() method will be a no-op (does nothing).

Note that you may even modify the contents of the byte slice you use to create the "new" body. You have full control over it.

^{Care must be taken though, as there might be other HTTP fields like content-length and checksums which may become invalid if you modify only the data. If subsequent handlers check those, you would also need to modify those too!}

Inspecting / modifying response body

If you also want to read the response body, then you have to wrap the http.ResponseWriter you get, and pass the wrapper on the chain. This wrapper may cache the data sent out, which you can inspect either after, on on-the-fly (as the subsequent handlers write to it).

Here's a simple ResponseWriter wrapper, which just caches the data, so it'll be available after the subsequent handler returns:

type MyResponseWriter struct {
    http.ResponseWriter
    buf *bytes.Buffer
}

func (mrw *MyResponseWriter) Write(p []byte) (int, error) {
    return mrw.buf.Write(p)
}

Note that MyResponseWriter.Write() just writes the data to a buffer. You may also choose to inspect it on-the-fly (in the Write() method) and write the data immediately to the wrapped / embedded ResponseWriter. You may even modify the data. You have full control.

^{Care must be taken again though, as the subsequent handlers may also send HTTP response headers related to the response data –such as length or checksums– which may also become invalid if you alter the response data.}

Full example

Putting the pieces together, here's a full working example:

func loginmw(handler http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        body, err := ioutil.ReadAll(r.Body)
        if err != nil {
            log.Printf("Error reading body: %v", err)
            http.Error(w, "can't read body", http.StatusBadRequest)
            return
        }

        // Work / inspect body. You may even modify it!

        // And now set a new body, which will simulate the same data we read:
        r.Body = ioutil.NopCloser(bytes.NewBuffer(body))

        // Create a response wrapper:
        mrw := &MyResponseWriter{
            ResponseWriter: w,
            buf:            &bytes.Buffer{},
        }

        // Call next handler, passing the response wrapper:
        handler.ServeHTTP(mrw, r)

        // Now inspect response, and finally send it out:
        // (You can also modify it before sending it out!)
        if _, err := io.Copy(w, mrw.buf); err != nil {
            log.Printf("Failed to send out response: %v", err)
        }
    })
}

Solution 2

I could use the GetBody from Request package.

Look this comment in source code from request.go in net/http:

GetBody defines an optional func to return a new copy of Body. It is used for client requests when a redirect requires reading the body more than once. Use of GetBody still requires setting Body. For server requests it is unused."

GetBody func() (io.ReadCloser, error)

This way you can get the body request without make it empty.

Sample:

getBody := request.GetBody
copyBody, err := getBody()
if err != nil {
    // Do something return err
}
http.DefaultClient.Do(request)

Comments

[seanlook]

** For server requests it is unused **, you can get the body from server side for copy purpose, or a panic runtime error: invalid memory address or nil pointer dereference will happen

[Yuri Giovani]

Hi @seanlook, my mistake. You need to check the error returned by getBody() getBody := request.GetBody copyBody, err := getBody() if err != nil { // Do something return err } http.DefaultClient.Do(request)

[Jeff]

wouldn't you still want to close the initial body before reassigning it?

[icza]

@Jeff The request body does not need to be closed by the handler, it is closed by the server. See Where to put “defer req.Body.Close()”?

[sztanpet]

You should limit the amount you read from the body, otherwise it's just a DoS vector. Please use golang.org/pkg/net/http/#MaxBytesReader @icza

[icza]

@sztanpet Yes, there are even more considerations, the answer is just the theory how inspecting the request and response bodies can be achieved. Protecting against malicious large request should be handled at the top level, not in every intermediate handler.

[zakaria amine]

ioutil.ReadAll is not recommended for http request body handling, especially for high loads servers. See: haisum.github.io/2017/09/11/golang-ioutil-readall

[EdwinCab]

with golang 1.4 the func GetBody returns a nil, then copyBody fires an error

[overexchange]

@icza I did not get this in your answer: "You may use bytes.NewBuffer() to obtain an io.Reader from a byte slice." Because... bytes.NewBuffer(buf) creates and initializes a new Buffer using buf([]byte) as its initial contents... Are you saying that, bytes.Buffer is io.Reader & io.Writer?

[icza]

@overexchange Yes, bytes.Buffer is both io.Reader and io.Writer. If you pass a slice to it, you can use it to read from that slice.

[latitov]

AFAIK, GetBody() isn't a function to use, but rather a function to define, isn't it? Well, according to docs, it's an optional field in the Request struct, which may be filled with a user code. And only then used. Not the other way around.