Go, tcp too many open files debug

Question

Here's a straightforward Go http (tcp) connection test script

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    for i := 0; i < 2000; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()
            resp, err := http.Get(ts.URL)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%s", i, greeting)
        }(i)
    }
    wg.Wait()
}

And If I run this in Ubuntu I get:

panic: Get http://127.0.0.1:33202: dial tcp 127.0.0.1:33202: too many open files

Other posts say to make sure Close the connection, which I am doing it all here. And others say to increase the limit of maximum connection with ulimit or try sudo sysctl -w fs.inotify.max_user_watches=100000 but still does not work.

How do I run millions of tcp connection goroutines in a single server? It crashes only with 2,000 connections.

Thanks,

Solution 1

I think you need to change your max file descriptors. I have run into the same problem on one of my development VMs before and needed to change the file descriptors max, not anything with inotify settings.

FWIW, your program runs fine on my VM.

·> ulimit -n
120000

But after I run

·> ulimit -n 500
·> ulimit -n
500

I get:

panic: Get http://127.0.0.1:51227: dial tcp 127.0.0.1:51227: socket: too many open files

** Don't fall into the trap that Praveen did **

Note ulimit != ulimit -n.

➜  cmd git:(wip-poop) ✗ ulimit -a
-t: cpu time (seconds)              unlimited
-f: file size (blocks)              unlimited
-d: data seg size (kbytes)          unlimited
-s: stack size (kbytes)             8192
-c: core file size (blocks)         0
-v: address space (kbytes)          unlimited
-l: locked-in-memory size (kbytes)  unlimited
-u: processes                       1418
-n: file descriptors                4864

Solution 2

Go’s http package doesn’t specify request timeouts by default. You should always include a timeout in your service. What if a client doesn't close their session? Your process will keep alive old sessions hitting ulimits. A bad actor could intentionally open thousands of sessions, DOSing your server. Heavy load services should adjust ulimits as well but timeouts for backstop.

Ensure you specify a timeout:

http.DefaultClient.Timeout = time.Minute * 10

You can validate before and after by monitoring files opened by your process:

lsof -p [PID_ID]

Solution 3

If you want to run millions of go routines that open/read/close a socket, well you better up your ulimit, or open/read/close the socket and pass the value read in to the go-routine, but I would use a buffered channel to control how many file descriptors you want to be able to open.

const (
    // this is where you can specify how many maxFileDescriptors
    // you want to allow open
    maxFileDescriptors = 100
)

func main() {
    ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintln(w, "Hello, client")
    }))
    defer ts.Close()
    var wg sync.WaitGroup
    maxChan := make(chan bool, maxFileDescriptors)
    for i := 0; i < 1000; i++ {
        maxChan <- true
        wg.Add(1)
        go func(url string, i int, maxChan chan bool, wg *sync.WaitGroup) {
            defer wg.Done()
            defer func(maxChan chan bool) { <-maxChan }(maxChan)
            resp, err := http.Get(url)
            if err != nil {
                panic(err)
            }
            greeting, err := ioutil.ReadAll(resp.Body)
            if err != nil {
                panic(err)
            }
            err = resp.Body.Close()
            if err != nil {
                panic(err)
            }
            fmt.Printf("%d: %s", i, string(greeting))
        }(ts.URL, i, maxChan, &wg)
    }
    wg.Wait()
}

Solution 4

HTTP/1.1 uses persistent connections by default:
A significant difference between HTTP/1.1 and earlier versions of HTTP is that persistent connections are the default behavior of any HTTP connection.
http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html
The solution was to inform the server that the client wants to close the connection after the transaction is complete. This can be done by setting the Connection header,
req.Header.Set("Connection", "close") or by setting the Close property to true on the http.Request:
req.Close = true After doing that, the “too many open files” issue went away as the program was no longer keeping HTTP connections open and thus not using up file descriptors.

I solved this by adding req.Close = true and req.Header.Set("Connection", "close"). I think it's better than changing ulimit.

source: http://craigwickesser.com/2015/01/golang-http-to-many-open-files/

Solution 5

I had to also to manually set the close connection header to avoid the file descriptor issue:

r, _ := http.NewRequest(http.MethodDelete, url, nil)
r.Close = true
res, err := c.Do(r)
res.Body.Close();

Without r.Close = true and res.Body.Close() I hit the file descriptor limit. With both I could fire off as many as I needed.

View more solutions

Comments

[Popmedic]

If you are increasing your ulimit, you will still run out of file descriptors eventually. Try using a buffered channel and limit the number of go-routines that open a file to below the ulimit. 1000 open files at a time should be more then enough...

[Daniel]

Excellent, this is exactly the practical wisdom I am seeking.

[elijahcarrel]

I believe this has a (very unlikely but possible) race condition bug— you should wg.Add(1) outside of and just before go func(...) { ... }. Otherwise, if the first n goroutines finish before the n + 1th goroutine begins (which is, again, extremely unlikely), then the wg.Wait() command might actually terminate before the n+1th and remaining goroutines get run. (They will still get run, but the downstream code won't have waited for it.)