Which is better for local IPC, POSIX message queues (mqueues) or Unix domain (local) sockets?

Question

Is it better to use POSIX message queues or Unix domain sockets for local IPC communication?

I have worked with Unix sockets between machines (not domain) and I remember that making and breaking the connection would cause sockets to linger awhile before they finally went away. Moreover, if you wanted a "reliable" exchange you either had to use TCP or design the application to return an ACK. I'm not certain if this also applies to Unix domain sockets though.

In my current project we need local IPC. My first reaction was to use POSIX MQueues, since I've used them before for local messaging. However, a co-worker is suggesting Unix domain sockets instead.

Is one better than the other, or is it a matter of programming familiarity? Or perhaps its depends upon the application being created?

At a big picture the application we are working on follows a client/server model. The clients send messages to the server to "do something". However, the client doesn't wait for an "its done" response -- although they do want to know if their request has been received or not.

The basic logic for the send side is:

connect to server
send request
note if the send worked or not
disconnect from server

There can be hundreds of clients to the one server.

We're are executing on an SMP system (4-8 cores) running the Linux OS.

Thanks in advance.

Solution 1

UNIX domain sockets do not have to "linger" in a TIME_WAIT-like status, since that wait time is used in case there are stray packets from the connection still wandering around the Internet. The concern doesn't apply locally.

UNIX domain sockets can be either SOCK_STREAM (like TCP) or SOCK_DGRAM (like UDP), with the added guarantee that UNIX domain datagram sockets are reliable and don't re-order datagrams.

You will still need some kind of ACK (you do even with TCP) if you want to be certain that your other application has read the message you sent; after all, even if the send() succeeded it may have crashed before it had a chance to process the message. (This applies to message queues too - to be totally sure that a message won't be lost, the recieving application must write the request to a journal, flush that to disk and then send back an acknowledgement).

I agree that the choice is essentially a matter of programming familiarity.

Solution 2

Is one better than the other, or is it a matter of programming familiarity? Or perhaps its depends upon the application being created?

SysV message queues compared to UNIX domain datagram sockets have the major differences I'm aware of:

• You can poll() socket, but you can't message queue.

• Message queue is global and might (and usually do) require some administrative involvement: cleaning old hanging SysV resources is one of the many sysadmin daily routines. While semantics of UNIX domain is much simpler and applications can generally maintain it completely internally without sysadmin involvement.

• (?) Message queue is persistent, it might retain messages from the old sessions. (Can't recall that bit precisely, but IIRC that was happening to me more than once).

• Looking at the man msgrcv I do not see analogue of socket's MSG_PEEK. Rarely needed, but at times comes quite handy.

• Most of the time, users prefer to use in the configuration symbolic names, not the numeric key id. Lack of symbolic keys IMO is quite grave oversight on part of the SysV interface designers.

As with all SysV resources, their management is the major PITA. If you let system decide message queue ID, then you have to take care of properly sharing it with other applications. (And you also has to tell admins somehow that the ID has to be removed eventually). If you allow to configure the key for message queue, then you might run into trivial problems that the id is already used by some application or it is a remnant of the previous run. (Seeing servers being reboot only because they run out of SysV resources is pretty common.)

All in all, I avoid SysV resources when possible: lack of poll() support under most common circumstances is a deal breaker.

However, the client doesn't wait for an "its done" response -- although they do want to know if their request has been received or not.

That is a common dilemma of transactional processing. General response is (as in RDBMSs) you can't and after communication interruption (crash or whatever) the application has to check itself whether the request was already processed or not.

From that I can tell that probably TCP would be a better choice. Client sends request and declares it as finished only when it gets positive response from the server. Server unless it is capable of sending the response to client, has to rollback the transaction.

Solution 3

I'd suggest looking at DBus for such an application, if only for their data marshalling and RPC like interface (both synchronous and asynchronous). It uses domain sockets natively and works really welll after an initial learning curve.

Comments

[nos]

Though, on Linux a posix message queue descriptor is actually a file descriptor and it supports select/poll. I'm not sure about sysv message queues.

[Dummy00001]

@nos: reference? aren't you mixing it with AIX? POSIX doesn't describe that and the old Linux reference I know also says no.

[nos]

man mq_overview, "On Linux, a message queue descriptor is actually a file descriptor, and can be monitored using select(2), poll(2), or epoll(7). This is not portable." , as mentioned this is posix message queues, I'm not sure if sysv message quees work that way, if they do it's changed in the recent years.

[Dummy00001]

@nos: thanks. That explains why there are two interfaces. The mq_* functions is the new POSIX interface: "First released in Issue 5. Included for alignment with the POSIX Realtime Extension." And yet the POSIX doesn't define the use of poll() with the "message queue descriptors" ... :(