golang get massive read tcp ip:port i/o timeout in ubuntu 14.04 LTS

Question

I wrote a golang program which run well in the past several months in ubuntu 12.04 LTS until I upgraded it to 14.04 LTS

My program is focused on sending HTTP requests which send about 2-10 HTTP requests per second. The HTTP request address vary.

When the problem occurs, first, some of the requests shows read tcp [ip]:[port]: i/o timeout, then after several minutes all requests show read tcp [ip]:[port]: i/o timeout, not any request can be sent.

I restart the program, everything become right again.

All of our servers(2 server) have such problem after upgraded from 12.04 to 14.04

I create new goroutine for every request

the problem does not occur in the same interval, sometimes it won't occur one or two day, sometimes It occur twice in an hour

Bellow is my code requesting HTTP Address:

t := &http.Transport{
    Dial:            timeoutDial(data.Timeout),
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}
//req := s.ParseReq(data)
req := data.convert2Request()
if req == nil {
    return
}

var resp *http.Response
if data.Redirect {
    c := &http.Client{
        Transport: t,
    }
    resp, err = c.Do(req)
} else {
    resp, err = t.RoundTrip(req)
}

data.updateTry()

r := s.ParseResp(data, resp, err)

updateTry:

func (d *SendData) updateTry() {
    d.Try++
    d.LastSend = time.Now()
}

timeoutDial:

func timeoutDial(timeout int) func(netw, addr string) (net.Conn, error) {
    if timeout <= 0 {
        timeout = 10
    }
    return func(netw, addr string) (net.Conn, error) {
        deadline := time.Now().Add(time.Duration(timeout) * time.Second)
        c, err := net.DialTimeout(netw, addr, time.Second*time.Duration(timeout+5))
        if err != nil {
            return nil, err
        }
        c.SetDeadline(deadline)
        return c, nil
    }
}

and My dealing with response is:

func (s *Sender) ParseResp(data SendData, resp *http.Response, err error) (r Resp) {
    r = Resp{URL: data.URL}
    if err != nil {
        r.Err = err.Error()
    } else {
        r.HttpCode = resp.StatusCode
        r.Header = resp.Header
        r.URL = resp.Request.URL.String()
        defer resp.Body.Close()
        // we just read part of response and log it.
        reader := bufio.NewReader(resp.Body)
        buf := make([]byte, bytes.MinRead) // 512 byte
        for len(r.Body) < 1024 {           // max 1k
            var n int
            if n, _ = reader.Read(buf); n == 0 {
                break
            }
            r.Body += string(buf[:n])
        }
    }
    return
}

I also found setting in /etc/sysctl.conf which can make the problem happen less frequently:

net.core.somaxconn = 65535
net.netfilter.nf_conntrack_max = 655350
net.netfilter.nf_conntrack_tcp_timeout_established = 1200

I need help for solving this problem.

It seems like this but I don't see any solution https://bugs.launchpad.net/juju-core/+bug/1307434

Solution

To more explicitly state what Not_a_Golfer and OneOfOne have said, when you're done with the response, you need to close the connection which has been left open (through the Body field which is an io.ReadCloser). So basically, one simple though would be to change the code pertaining to making an http request to:

var resp *http.Response
if data.Redirect {
    c := &http.Client{
        Transport: t,
    }
    resp, err = c.Do(req)
} else {
    resp, err = t.RoundTrip(req)
}
if err == nil {
    defer resp.Body.Close() // we need to close the connection
}