Difference between POSIX AIO and libaio on Linux?

Question

What I seem to understand:

POSIX AIO APIs are prototyped in <aio.h> and you link your program with librt(-lrt), while the libaio APIs in <libaio.h> and your program is linked with libaio (-laio).

What I can't figure out:

1.Does the kernel handle the either of these methods differently?

2.Is the O_DIRECT flag mandatory for using either of them?

As mentioned in this post, libaio works fine without O_DIRECT when using libaio.Okay,understood but:

According to R.Love's Linux System Programming book, Linux supports aio (which I assume is POSIX AIO) on regular files only if opened with O_DIRECT.But a small program that I wrote (using aio.h,linked with -lrt) that calls aio_write on a file opened without the O_DIRECT flag works without issues.

Solution

On linux, the two AIO implementations are fundamentally different.

The POSIX AIO is a user-level implementation that performs normal blocking I/O in multiple threads, hence giving the illusion that the I/Os are asynchronous. The main reason to do this is that:

1. it works with any filesystem
2. it works (essentially) on any operating system (keep in mind that gnu's libc is portable)
3. it works on files with buffering enabled (i.e. no O_DIRECT flag set)

The main drawback is that your queue depth (i.e. the number of outstanding operations you can have in practice) is limited by the number of threads you choose to have, which also means that a slow operation on one disk may block an operation going to a different disk. It also affects which I/Os (or how many) is seen by the kernel and the disk scheduler as well.

The kernel AIO (i.e. io_submit() et.al.) is kernel support for asynchronous I/O operations, where the io requests are actually queued up in the kernel, sorted by whatever disk scheduler you have, presumably some of them are forwarded (in somewhat optimal order one would hope) to the actual disk as asynchronous operations (using TCQ or NCQ). The main restriction with this approach is that not all filesystems work that well or at all with async I/O (and may fall back to blocking semantics), files have to be opened with O_DIRECT which comes with a whole lot of other restrictions on the I/O requests. If you fail to open your files with O_DIRECT, it may still "work", as in you get the right data back, but it probably isn't done asynchronously, but is falling back to blocking semantics.

Also keep in mind that io_submit() can actually block on the disk under certain circumstances.

Comments

[itisravi]

Thanks for the reply.So for POSIX AIO, O_DIRECT is not mandatory, but for kerenel AIO,it is(to make sure no fallback happens)?This seems to contradict what is mentioned in the book.In the kernel code,VFS has these aio_read/write functions which are called(?) by aio_read/write system calls (which is POXIS AIO and not kernel AIO).

[Arvid]

There are no aio_* syscalls (linux.die.net/man/2/syscalls). The aio_* functions you see in vfs are probably part of kernel aio. the user level aio_* functions don't map 1:1 to system calls.

[ARH]

@Arvid Would you please elaborate in what kind of circumstances io_submit() is blocking ? (you mention sometime it may block disk)

[Arvid]

My understanding is that the block device layer is essentially guaranteed to not block. If you're only concerned about accessing raw disks, you're fine. However, filesystems are not implemented with the assumption that someone may want to use them asynchronously. I tested ext4, reiser and xfs. iirc, ext4 was the one that blocked the most, but it was also the fastest to complete my benchmark. searching the linux-aio mailing list will give you many hits. Here's my: article.gmane.org/gmane.linux.kernel.aio.general/3024/…

[Anon]

There's a non-exhaustive list of things that make io_submit() block over on stackoverflow.com/a/46377629/2732969 ...