How long is a TCP local socket address that has been bound unavailable after closing?

Question

On Linux (my live servers are on RHEL 5.5 - the LXR links below are to the kernel version in that), man 7 ip says:

A TCP local socket address that has been bound is unavailable for some time after closing, unless the SO_REUSEADDR flag has been set.

I am not using SO_REUSEADDR. How long is "some time"? How can i found out how long it is, and how can i change it?

I've been googling around this, and have found a few morsels of information, none of which really explain this from an application programmer's perspective. To wit:

• TCP_TIMEWAIT_LEN in net/tcp.h is "how long to wait to destroy TIME-WAIT state", and is fixed at "about 60 seconds"
• /proc/sys/net/ipv4/tcp_fin_timeout is "Time to hold socket in state FIN-WAIT-2, if it was closed by our side", and "Default value is 60sec"

Where i stumble is in bridging the gap between the kernel's model of the TCP lifecycle, and the programmer's model of ports being unavailable, that is, in understanding how these states relate to the "some time".

Solution

I believe that the idea of the socket being unavailable to a program is to allow any TCP data segments still in transit to arrive, and get discarded by the kernel. That is, it's possible for an application to call close(2) on a socket, but routing delays or mishaps to control packets or what have you can allow the other side of a TCP connection to send data for a while. The application has indicated it no longer wants to deal with TCP data segments, so the kernel should just discard them as they come in.

I hacked out a little program in C that you can compile and use to see how long the timeout is:

#include <stdio.h>        /* fprintf() */
#include <string.h>       /* strerror() */
#include <errno.h>        /* errno */
#include <stdlib.h>       /* strtol() */
#include <signal.h>       /* signal() */
#include <sys/time.h>     /* struct timeval */
#include <unistd.h>       /* read(), write(), close(), gettimeofday() */
#include <sys/types.h>    /* socket() */
#include <sys/socket.h>   /* socket-related stuff */
#include <netinet/in.h>
#include <arpa/inet.h>    /* inet_ntoa() */
float elapsed_time(struct timeval before, struct timeval after);
int
main(int ac, char **av)
{
        int opt;
        int listen_fd = -1;
        unsigned short port = 0;
        struct sockaddr_in  serv_addr;
        struct timeval before_bind;
        struct timeval after_bind;

        while (-1 != (opt = getopt(ac, av, "p:"))) {
                switch (opt) {
                case 'p':
                        port = (unsigned short)atoi(optarg);
                        break;
                }
        }

        if (0 == port) {
                fprintf(stderr, "Need a port to listen on\n");
                return 2;
        }

        if (0 > (listen_fd = socket(AF_INET, SOCK_STREAM, 0))) {
                fprintf(stderr, "Opening socket: %s\n", strerror(errno));
                return 1;
        }

        memset(&serv_addr, '\0', sizeof(serv_addr));
        serv_addr.sin_family      = AF_INET;
        serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
        serv_addr.sin_port        = htons(port);

        gettimeofday(&before_bind, NULL);
        while (0 > bind(listen_fd, (struct sockaddr *)&serv_addr, sizeof(serv_addr))) {
                fprintf(stderr, "binding socket to port %d: %s\n",
                        ntohs(serv_addr.sin_port),
                        strerror(errno));

                sleep(1);
        }
        gettimeofday(&after_bind, NULL);
        printf("bind took %.5f seconds\n", elapsed_time(before_bind, after_bind));

        printf("# Listening on port %d\n", ntohs(serv_addr.sin_port));
        if (0 > listen(listen_fd, 100)) {
                fprintf(stderr, "listen() on fd %d: %s\n",
                        listen_fd,
                        strerror(errno));
                return 1;
        }

        {
                struct sockaddr_in  cli_addr;
                struct timeval before;
                int newfd;
                socklen_t clilen;

                clilen = sizeof(cli_addr);

                if (0 > (newfd = accept(listen_fd, (struct sockaddr *)&cli_addr, &clilen))) {
                        fprintf(stderr, "accept() on fd %d: %s\n", listen_fd, strerror(errno));
                        exit(2);
                }
                gettimeofday(&before, NULL);
                printf("At %ld.%06ld\tconnected to: %s\n",
                        before.tv_sec, before.tv_usec,
                        inet_ntoa(cli_addr.sin_addr)
                );
                fflush(stdout);

                while (close(newfd) == EINTR) ;
        }

        if (0 > close(listen_fd))
                fprintf(stderr, "Closing socket: %s\n", strerror(errno));

        return 0;
}
float
elapsed_time(struct timeval before, struct timeval after)
{
        float r = 0.0;

        if (before.tv_usec > after.tv_usec) {
                after.tv_usec += 1000000;
                --after.tv_sec;
        }

        r = (float)(after.tv_sec - before.tv_sec)
                + (1.0E-6)*(float)(after.tv_usec - before.tv_usec);

        return r;
}

I tried this program on 3 different machines, and I get a variable time, between 55 and 59 seconds, when the kernel refuses to allow a non-root user to reopen a socket. I compiled the above code to an executable named "opener", and ran it like this:

./opener -p 7896; ./opener -p 7896

I opened another window and did this:

telnet otherhost 7896

That causes the first instance of "opener" to accept a connection, then close it. The second instance of "opener" tries to bind(2) to the TCP port 7896 every second. "opener" reports 55 to 59 seconds of delay.

Googling around, I find that people recommend doing this:

echo 30 > /proc/sys/net/ipv4/tcp_fin_timeout

to reduce that interval. It didn't work for me. Of the 4 linux machines I had access to, two had 30 and two had 60. I also set that value as low as 10. No difference to the "opener" program.

Doing this:

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle

did change things. The second "opener" only took about 3 seconds to get its new socket.

Comments

[Admin]

@Caleb: Concerning the tags, bind is a system call too! Try man 2 bind if you don't believe me. Admittedly, it's probably not the first thing unix people think of when someone says "bind", so fair enough.

[Admin]

I was well aware of the alternate uses of bind, but the tag here is specifically applied to the DNS server. We don't have tags for every possible system call.

[Tom Anderson]

I understand (roughly) what the purpose of the period of unavailability is. What i would like to know is exactly how long that period is on Linux, and how it can be changed. The problem with a number from a Wikipedia page about TCP is that it is necessarily a generalised value, and not something that is definitely true of my specific platform.

[Philippe Gachoud]

your speculations were interesting! just flag them as that with a title instead of removing them, it gives ways to search for the reason why!