Reading body of http.Request without modifying request state?

Question

I have a type implementing the http.Handler interface where, in its ServeHTTP method, incoming HTTP requests are inspected, some action is taken, and then the requests are forwarded to a reverse proxy handler (httputil.NewSingleHostReverseProxy).

This works fine, so long as I'm only inspecting the basic request properties, such as the URL or headers.

When I want to inspect the body of an incoming POST request, e.g. by calling req.ParseForm() and then using the req.Form property, I run into an error once the request is passed onto the reverse proxy:

http: proxy error: http: Request.ContentLength=687 with Body length 0

I imagine this happens because looking at the body of the HTTP request causes the req.Body.Reader stream to be drained, meaning it cannot be read again by the proxy handler.

I've been playing with things like io.Copy() and bufio.Peek(), but I'm not really getting anywhere.

Is there a way to peek at the HTTP request body (and use the built-in parsing of req.ParseForm etc.), while leaving the original request object in its original state, so that it can be passed to the reverse proxy?

Solution

Try reading into a buffer and then using the buffer to back two new readers, one for you to use, and one for subsequent consumers to use. For example, imagine that we want to modify the following code:

doStuff(r.Body) // r is an http.Request

We could do:

buf, _ := ioutil.ReadAll(r.Body)
rdr1 := ioutil.NopCloser(bytes.NewBuffer(buf))
rdr2 := ioutil.NopCloser(bytes.NewBuffer(buf))

doStuff(rdr1)
r.Body = rdr2 // OK since rdr2 implements the io.ReadCloser interface

// Now the program can continue oblivious to the fact that
// r.Body was ever touched.

Note that *bytes.Buffer does not have a Close() error method, so it doesn't implement the io.ReadCloser interface. Thus, we have to wrap our *bytes.Buffer values in calls to ioutil.NopCloser.

Comments

[nemo]

Have you tried replacing the Body with a seekable reader like bytes.Reader? You can set Body to every value that implements ReadCloser.

[Volker]

Untested (but worth a try): Make a deep copy of the Request as needed, i.e. all but not Body and .*Form. Then use io.TeeReader to clone Request.Body, wrap it in an NopCloser and stuff into the deepcopy of the request. Pass the deep copy to the reverse proxy.

[joshlf]

Yep, np! And good catch on the Close signature. I guess reading the documentation is important after all ;)

[Lars Wiegman]

You can get a io.ReadCloser from a buffer using the io.NopCloser function, e.g. io.NopCloser(buf). See golang.org/src/pkg/io/ioutil/ioutil.go?s=3623:3664#L111

[danmux]

perhaps a TeeReader would be usefull

[Joppe]

This works nicely, but only for bounded bodies that you can hold in memory or on disk.

[joshlf]

@Joppe, that's a good point. You'd need something much more subtle and complicated if you wanted to gracefully handle unbounded or too-large-for-memory requests.

[mattes]

GetBody() is only used for client requests. For server requests it is unused.

[mattes]

drainBody in golang.org/src/net/http/httputil/dump.go#L26 offers a similar solution, but taking http.NoBody into account.