multithreaded epoll

Question

I am creating a multithreaded server using epoll (edge-triggered) and non-blocking sockets. Currently I'm creating an event loop on the main thread and waiting for notifications and it works correctly
I have to choose between two approaches to make it multithreaded:

1. Create an event loop for each thread and add the server socket's file descriptor to look for notifications on each thread. (is that possible? I mean: is epoll thread-safe?)
2. Create a single event loop and wait for notifications. Whenever a notification is received, spawn a thread to handle it.

If I use the first method, is there a chance for multiple threads to get notified with the same event? how can I handle this situation?

What could be the best approach? Thank you.

Solution 1

I think option 1 is more popular since the primary purpose of non-blocking IO is to avoid the overhead of create & destroy threads.

take the popular web server nginx as an example, it create multiple processes (not threads) to handle incoming events on a handle, and the process the events in the subprocess. all of them share the same listening socket. it's quite similar to option 1.

Solution 2

I'm also writing a server using epoll, and I have considered the same model as you attached.

It is possible to use option 1, but it may cause "thundering herd" effect, you can read the source of nginx to find the solution. As for option 2, I deem that it is better to use thread pool instead of spawning a new thread each time.

And you can also the following model：

Main thread/process: accept incoming connection with blocking IO, and send the fd to the other threads using BlockingList or to the other processes using PIPE.

Sub threads/process: create an instance of epoll respectively, and add the incoming fd to the epoll, then processing them with non-blocking IO.