Linux and I/O completion ports?

Question

Using winsock, you can configure sockets or seperate I/O operations to "overlap". This means that calls to perform I/O are returned immediately, while the actual operations are completed asynchronously by separate worker threads.

Winsock also provides "completion ports". From what I understand, a completion port acts as a multiplexer of handles (sockets). A handle can be demultiplexed if it isn't in the middle of an I/O operation, i.e. if all its I/O operations are completed.

So, on to my question... does linux support completion ports or even asynchronous I/O for sockets?

Solution 1

So, on to my question... does linux support completion ports or even asynchronous I/O for sockets?

With regard to sockets, in 5.3 and later kernels, Linux has something analogous to completion ports in the shape of io_uring (for files/block devices io_uring support appeared in the 5.1 kernel).

Solution 2

If you're looking for something exactly like IOCP, you won't find it, because it doesn't exist.

Windows uses a notify on completion model (hence I/O Completion Ports). You start some operation asynchronously, and receive a notification when that operation has completed.

Linux applications (and most other Unix-alikes) generally use a notify on ready model. You receive a notification that the socket can be read from or written to without blocking. Then, you do the I/O operation, which will not block.

With this model, you don't need asynchronous I/O. The data is immediately copied into / out of the socket buffer.

The programming model for this is kind of tricky, which is why there are abstraction libraries like libevent. It provides a simpler programming model, and abstracts away the implementation differences between the supported operating systems.

There is a notify on ready model in Windows as well (select or WSAWaitForMultipleEvents), which you may have looked at before. It can't scale to large numbers of sockets, so it's not suitable for high-performance network applications.

Don't let that put you off - Windows and Linux are completely different operating systems. Something that doesn't scale well on one system may work very well on the other. This approach actually works very well on Linux, with performance comparable to IOCP on Windows.

Solution 3

IOCP is pronounced "asynchronous I/O" on various UNIX platforms:

• POSIX AIO is the standard
• Kernel AIO, epoll and io_uring seem to be a Linux-specific implementations
• Kqueue is the *BSD and Mac OSX implementation
• Message Passing Interface (MPI) is an option for high-performance computing
• obligatory Boost reference - Boost.Asio

Solution 4

Use boost::asio. Hands down. It has a mild learning curve, but it's cross-platform, and automatically uses the best available method for the system you're compiling on. There's simply no reason not to.

I know that this isn't quite an answer to your question, but it's the best advice I could give.

View more solutions

Comments

[someguy]

Thanks. Any chance you could provide me with some sample codes? I've looked into POSIX AIO before, but I couldn't find anything related to sockets.

[michelson]

I haven't used POSIX AIO before. I just know that it has been discussed in various forums. Personally, I run BSD-based systems so I have used Kqueue for this. I would look into using Boost or some MPI implementation for concurrent I/O if that is what you are after.

[someguy]

I've looked into POSIX AIO and Kernel AIO, and both seem to imply that there is no true support for sockets, which is a shame. I'm going to assume that all the documents I've read are outdated, as Java 7 is apparently going to support asynchronous I/O, so linux has to have it, or will have it, right? Maybe I'll try and have a look at what Sun/Oracle have done so far.

[michelson]

It does look like POSIX AIO has gotten short shrift. You might want to read some of references on "Fast UNIX Servers" (dank.qemfd.net/dankwiki/index.php/Fast_UNIX_Servers). There seems to be a lot of good information there.

[someguy]

I don't think libevent supports asynchronous I/O, and it even says so on the linked page that it's for non-blocking sockets. I guess I'll look into it deeper later. 0MQ is also interesting, but I don't think it's what I'm looking for. It seems to be a message-passing library (similar to erlang actors?)

[jweyrich]

@someguy asynchronous I/O comprehends/includes non-blocking sockets.

[Gabe]

Any synchronous I/O will block so long as you're writing more data than the buffer can hold. And if the data you're writing has been paged out, your process will block on the page fault.

[Ben Voigt]

@BlackAura: Wouldn't WSAAsyncSelect support a large number of sockets on Windows with the notify-on-ready model? Admittedly I've not used it with a large number of sockets, but since the socket handle is passed as an event parameter, and there's no bitmask or array of sockets to query, it seems like it ought to scale up magnificently.

[someguy]

@Ben Voigt The problem with WSAAsyncSelect is that it's built around the window procedure API. Its message loop is slow compared to the other polling functions. Having to handle thousands of sockets can significantly degrade the performance of your program. By the way, what do you by "there's no bitmask"? You still have to pass flags.

[Ben Voigt]

@someguy: By "no bitmask" I mean no use of FD_SET, as is done in select. Also, WSAAsyncSelect requires using the thread message queue, not the window procedure API. It's perfectly possible to catch those messages in the message loop and not pass them through DispatchMessage and the window procedure code.

[someguy]

@Ben Voigt Oh, yeah, you're right about the messages being queued. However, you still have to poll using GetMessage. My point is that it's slow compared to other polling functions. I don't know too much about it, but I've been told that WSAAsyncSelect doesn't scale very well.

[Vladislav Vaintroub]

@BlackAura, the models are not completely different. Asynchronous WSARecv() with 0 byte buffer is readiness notification, just like epoll_wait(). With high confidence (and depending on how big are the buffers you receive) the following WSARecv with > 0 size won't block, since the bytes are already cached by Winsock.

[unixman83]

"asynchronous I/O" is buggy and poorly supported on most Unices! Beware

[unixman83]

+1 I hate ASIO because of its extreme use of namespaces, but I have to agree that it's the highest quality free library that fully supports BOTH Windows AND Linux. Libevent on the other hand has shoddy, unfinished support for Windows.

[someguy]

@Vladislav Vaintroub: WSARecv() returns 0 if the I/O operation has been scheduled; it doesn't mean the socket is ready for that operation. Also, any value other than 0 means there was an error.

[Vladislav Vaintroub]

@someguy, GetQueuedCompletionStatus returns when WSARecv() with 0 bytes is completed, which then means "ready for operation". I understand it as - there is at least one byte on the connection that can be read without blocking

[Vladislav Vaintroub]

... or there is an error. GetQueuedCompletionStatus() is comparable to epoll_wait() , while asynchronous 0 bytes WSARecv() is comparable to epoll_ctl() with EPOLL_CTL_MOD|EPOLLIN|EPOLLET|EPOLLONESHOT flags

[someguy]

@Vladislav Vaintroub: WSARecv() schedules the I/O operation to be performed. When the operation is completed, GetQueuedCompletionStatus() will return. See the differences between readiness events and completion events.

[Vladislav Vaintroub]

I do not think I ever denied it. You need to understand that WSARecv() 0 bytes operation is equivalent to "start watching for readiness". GetQueuedCompletionStatus in this case says "yes,something can be read from the socket". It is edge-triggered readiness notification, fully equivalent to epoll_ctl/epoll_wait example above. And epoll does not do completion notification

[t0mm13b]

Might be more useful to quote and summarize on that page from the linky supplied as link rot will happen and users will click on it to find 404 at some point.

[Scott 混合理论]

i was told completionPorts(IOCP) are much more faster than epoll/select, but linux now has io_uring witch is the linux version of IOCP.