Socket programming for multi-clients with 'select()' in C

Question

This is a question about socket programming for multi-client.

While I was thinking how to make my single client and server program to multi clients,I encountered how to implement this. But even if I was searching for everything, kind of confusion exists.

1. I was thinking to implement with select(), because it is less heavy than fork. but I have much global variables not to be shared, so I hadn`t considered thread to use.

2. and so to use select(), I could have the general knowledge about FD_functions to utilize, but here I have my question, because generally in the examples on websites, it only shows multi-client server program...

Since I use sequential recv() and send() in client and also in server program that work really well when it`s single client and server, but I have no idea about how it must be changed for multi cilent. Does the client also must be unblocking? What are all requirements for select()?

The things I did on my server program to be multi-client

1) I set my socket option for reuse address, with SO_REUSEADDR

2) and set my server as non-blocking mode with O_NONBLOCK using fctl().

3) and put the timeout argument as zero.

and proper use of FD_functions after above.

But when I run my client program one and many more, from the second client, client program blocks, not getting accepted by server.

I guess the reason is because I put my server program`s main function part into the 'recv was >0 ' case.

for example with my server code,

(I`m using temp and read as fd_set, and read as master in this case)

int main(void)
{

  int conn_sock, listen_sock;
  struct sockaddr_in s_addr, c_addr;
  int rq, ack;
  char path[100];
  int pre, change, c;
  int conn, page_num, x;
  int c_len = sizeof(c_addr);
  int fd;
  int flags;
  int opt = 1;
  int nbytes;
  fd_set read, temp;

  if ((listen_sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0)
  {
    perror("socket error!");
    return 1;
  }

  memset(&s_addr, 0, sizeof(s_addr));

  s_addr.sin_family = AF_INET;
  s_addr.sin_addr.s_addr = htonl(INADDR_ANY);
  s_addr.sin_port = htons(3500);
  if (setsockopt(listen_sock, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(int)) == -1)
  {

    perror("Server-setsockopt() error ");
    exit(1);

  }
  flags = fcntl(listen_sock, F_GETFL, 0);
  fcntl(listen_sock, F_SETFL, flags | O_NONBLOCK);

  //fcntl(listen_sock, F_SETOWN, getpid());

  bind(listen_sock, (struct sockaddr*) &s_addr, sizeof(s_addr));

  listen(listen_sock, 8);

  FD_ZERO(&read);
  FD_ZERO(&temp);

  FD_SET(listen_sock, &read);
  while (1)
  {

    temp = read;

    if (select(FD_SETSIZE, &temp, (fd_set *) 0, (fd_set *) 0,
        (struct timeval *) 0) < 1)
    {
      perror("select error:");
      exit(1);
    }

    for (fd = 0; fd < FD_SETSIZE; fd++)
    {
      //CHECK all file descriptors

      if (FD_ISSET(fd, &temp))
      {

        if (fd == listen_sock)
        {

          conn_sock = accept(listen_sock, (struct sockaddr *) &c_addr, &c_len);
          FD_SET(conn_sock, &read);
          printf("new client got session: %d\n", conn_sock);

        }
        else
        {

          nbytes = recv(fd, &conn, 4, 0);
          if (nbytes <= 0)
          {
            close(fd);
            FD_CLR(fd, &read);
          }
          else
          {

            if (conn == Session_Rq)
            {

              ack = Session_Ack;
              send(fd, &ack, sizeof(ack), 0);

              root_setting();

              c = 0;
              while (1)
              {
                c++;
                printf("in while loop\n");
                recv(fd, &page_num, 4, 0);

                if (c > 1)
                {
                  change = compare_with_pre_page(pre, page_num);

                  if (change == 1)
                  {
                    page_stack[stack_count] = page_num;
                    stack_count++;
                  }
                  else
                  {
                    printf("same as before page\n");
                  }
                } //end of if
                else if (c == 1)
                {
                  page_stack[stack_count] = page_num;
                  stack_count++;
                }

                printf("stack count:%d\n", stack_count);
                printf("in page stack: <");

                for (x = 0; x < stack_count; x++)
                {
                  printf(" %d ", page_stack[x]);
                }

                printf(">\n");

                rq_handler(fd);

                if (logged_in == 1)
                {
                  printf("You are logged in state now, user: %s\n",
                      curr_user.ID);
                }
                else
                {
                  printf("not logged in.\n");
                  c = 0;
                }

                pre = page_num;
              } //end of while
            } //end of if
          }
        } //end of else
      } //end of fd_isset
    } //end of for loop  
  } //end of outermost while
}

if needed for code explanation : What I was about to work of this code was, to make kind of web pages to implement 'browser' for server. I wanted to make every client get session for server to get login-page or so.

But the execution result is, as I told above. Why is that?

1. the socket in the client program must be non-blocking mode too to be used with non-blocking Server program to use select()?

2. Or should I use fork or thread to make multi client and manage with select? The reason I say this is, after I considered a lot about this problem, 'select()' seems only proper for multi client chatting program... that many 'forked' or 'threaded' clients can pend to, in such as chat room. how do you think?... Is select also possible or proper thing to use for normal multi-client program?

If there something I missed to let my multi client program work fine, please give me some knowledge of yours or some requirements for the proper use of select. I didn`t know multi-client communication was not this much easy before :) I also considered to use epoll but I think I need to understand first about select well.

Thanks for reading.

Solution

Besides the fact you want to go from single-client to multi-client, it's not very clear what's blocking you here.

Are you sure you fully understood how does select is supposed to work ? The manual (man 2 select on Linux) may be helpful, as it provides a simple example. You can also check Wikipedia.

To answer your questions :

1. First of all, are you sure you need non-blocking mode for your sockets ? Unless you have a good reason to do so, blocking sockets are also fine for multi-client networking.

2. Usually, there are basically two ways to deal with multi-clients in C: fork, or select. The two aren't really used altogether (or I don't know how :-) ). Models using lightweight threads are essentially asynchronous programming (did I mention it also depends on what you mean by 'asynchronous' ?) and may be a bit overkill for what you seem to do (a good example in C++ is Boost.Asio).

As you probably already know, the main problem when dealing with more than one client is that I/O operations, like a read, are blocking, not letting us know when there's a new client, or when a client has said something.

The fork way is pretty straighforward : the server socket (the one which accepts the connections) is in the main process, and each time it accepts a new client, it forks a whole new process just to monitor this new client : this new process will be dedicated to it. Since there's one process per client, we don't care if i/o operations are blocking or not.

The select way allows us to monitor multiple clients in one same process : it is a multiplexer telling us when something happens on the sockets we give it. The base idea, on the server side, is first to put the server socket on the read_fds FD_SET of the select. Each time select returns, you need to do a special check for it : if the server socket is set in the read_fds set (using FD_ISSET(...)), it means you have a new client connecting : you can then call accept on your server socket to create the connection. Then you have to put all your clients sockets in the fd_sets you give to select in order to monitor any change on it (e.g., incoming messages).

I'm not really sure of what you don't understand about select, so that's for the big explaination. But long story short, select is a clean and neat way to do single-threaded, synchronous networking, and it can absolutely manage multiple clients at the same time without using any fork or threads. Be aware though that if you absolutely want to deal with non-blocking sockets with select, you have to deal extra error conditions that wouldn't be in a blocking way (the Wikipedia example shows it well as they have to check if errno isn't EWOULDBLOCK). But that's another story.

EDIT : Okay, with a little more code it's easier to know what's wrong.

1. select's first parameter should be nfds+1, i.e. "the highest-numbered file descriptor in any of the three sets, plus 1" (cf. manual), not FD_SETSIZE, which is the maximum size of an FD_SET. Usually it is the last accept-ed client socket (or the server socket at beginning) who has it.
2. 1. You shouldn't do the "CHECK all file descriptors" for loop like that. FD_SETSIZE, e.g. on my machine, equal to 1024. That means once select returns, even if you have just one client you would be passing in the loop 1024 times ! You can set fd to 0 (like in the Wikipedia example), but since 0 is stdin, 1 stdout and 2 stderr, unless you're monitoring one of those, you can directly set it to your server socket's fd (since it is probably the first of the monitored sockets, given socket numbers always increase), and iterate until it is equal to "nfds" (the currently highest fd).
3. Not sure that it is mandatory, but before each call to select, you should clear (with FD_ZERO for example) and re-populate your read fd_set with all the sockets you want to monitor (i.e. your server socket and all your clients sockets). Once again, inspire yourself of the Wikipedia example.

Comments

[kfsone]

You might want to look at libevent.

[johimi]

Please post all of your server loop (especially the select and other FD_ calls)

[Swen]

I hadn`t put code before, because it may not helpful to understand... I put it and added what I was about to work out of the code.

[Swen]

oh god...thanks. I thought I need to use non blocking just because recv and send blocks because they do not let the rest of the code go, just serving one client. uhh.. then select is checking all the time with time check for the one process`s fd_set? It is kind of mystery for me that how even without multitasking like thread or fork, it acts as if it is.

[j3141592653589793238]

That's actually a really good explanation of how to use select in this case.