How to read response body of ReverseProxy

Question

package main

import (
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    target := &url.URL{Scheme: "http", Host: "www.google.com"}
    proxy := httputil.NewSingleHostReverseProxy(target)

    http.Handle("/google", proxy)
    http.ListenAndServe(":8099", nil)
}

Reverse Proxy is works. How can I get the response body?

Solution 1

httputil.ReverseProxy has a Transport field. You can use it to modify the response. For example:

type transport struct {
    http.RoundTripper
}

func (t *transport) RoundTrip(req *http.Request) (resp *http.Response, err error) {
    resp, err = t.RoundTripper.RoundTrip(req)
    if err != nil {
        return nil, err
    }
    b, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return nil, err
    }
    err = resp.Body.Close()
    if err != nil {
        return nil, err
    }
    b = bytes.Replace(b, []byte("server"), []byte("schmerver"), -1)
    body := ioutil.NopCloser(bytes.NewReader(b))
    resp.Body = body
    resp.ContentLength = int64(len(b))
    resp.Header.Set("Content-Length", strconv.Itoa(len(b)))
    return resp, nil
}

// ...
proxy := httputil.NewSingleHostReverseProxy(target)
proxy.Transport = &transport{http.DefaultTransport}

Playground example of the whole thing: http://play.golang.org/p/b0S5CbCMrI.

Solution 2

now httputil/reverseproxy, support than, see source

 type ReverseProxy struct {
        ...

        // ModifyResponse is an optional function that
        // modifies the Response from the backend
        // If it returns an error, the proxy returns a StatusBadGateway error.
        ModifyResponse func(*http.Response) error
    }



func rewriteBody(resp *http.Response) (err error) {
    b, err := ioutil.ReadAll(resp.Body) //Read html
    if err != nil {
        return  err
    }
    err = resp.Body.Close()
    if err != nil {
        return err
    }
    b = bytes.Replace(b, []byte("server"), []byte("schmerver"), -1) // replace html
    body := ioutil.NopCloser(bytes.NewReader(b))
    resp.Body = body
    resp.ContentLength = int64(len(b))
    resp.Header.Set("Content-Length", strconv.Itoa(len(b)))
    return nil
}

// ...
target, _ := url.Parse("http://example.com")
proxy := httputil.NewSingleHostReverseProxy(target)
proxy.ModifyResponse = rewriteBody

Solution 3

I don't know best solution. But you can do something like this:

package main

import (
    "fmt"
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    target := &url.URL{Scheme: "http", Host: "www.google.com"}
    proxy := httputil.NewSingleHostReverseProxy(target)

    http.Handle("/google", CustomHandler(proxy))
    http.ListenAndServe(":8099", nil)
}

func CustomHandler(h http.Handler) http.HandlerFunc {
    return func(res http.ResponseWriter, req *http.Request) {
        h.ServeHTTP(NewCustomWriter(res), req)
    }
}

type customWriter struct {
    http.ResponseWriter
}

func NewCustomWriter(w http.ResponseWriter) *customWriter {
    return &customWriter{w}
}

func (c *customWriter) Header() http.Header {
    return c.ResponseWriter.Header()
}

func (c *customWriter) Write(data []byte) (int, error) {
    fmt.Println(string(data)) //get response here
    return c.ResponseWriter.Write(data)
}

func (c *customWriter) WriteHeader(i int) {
    c.ResponseWriter.WriteHeader(i)
}

Comments

[JimB]

small nit, b can be wrapped with bytes.NewReader, which is a bit smaller structure.

[JimB]

This only manages to print out the body one chunk at a time. You would still need to copy it into a another io.Writer to get the entire stream; luckily there's already something for that: io.TeeReader. (or make a new RoundTripper if you want the response before it's being written out)

[RoninDev]

Yes, of course. I just show the idea. I personally think that @Ainar-G solution is better than mine.

[tomasz]

I don't think OP's question is clear enough to discard this answer, in particular it doesn't say the body should be written to the string as a whole; this example, although a little bit too bloated, shows how to capture the response stream. Good solution for getting hold of response body on the fly and performing, for example, some encoding/decoding.

[Brent Bradburn]

Shouldn't this use defer for resp.Body.Close()?

[craftizmv]

No as, you will end up in an unhandled err. This call returns an error also if Close() fails.

[SVUser]

Unfortunately, it seems that there is no way to access both req and res in a single function when using ModifyResponse, so if the modification logic is based on req dynamically, the Transport mod is the only approach. Why wouldn't they add req to this function? What am I missing?

[Bravo Delta]

@SVUser - Probably a bit late, but you can access the request off the response like such: resp.Request