Good alternative to shared memory IPC for Java/C++ apps on Linux

Question

I'm currently using shared memory for IPC between Java and C++ apps, but looking for a more convenient alternative.

Can someone advise a better method with same performance and speed?

Thanks!

Solution 1

It depends on how you plan to have your apps interact. In the POSIX environment, you have pipes, shared memory, sockets, semaphores and message queues. See this question: Comparing unix linux IPC for more information.

What is the interaction model for your processes (i.e. client/server, producer-consumer, etc)?

From personal experience, I would suggest your best bet would be pipes (since they are just files to read and write bytes) or sockets (since both languages support them).

Solution 2

As mikelong said, this depends a lot on what you are doing. AFAIK, none of the IPC methods have native Java bindings, so you're probably going to have to use JNI and make bindings yourself, so all the different methods are roughly equally as hard. If you are doing message passing though, I highly suggest using message queues. They are very easy to use (once you have the bindings), and have good performance. If you need to "share" some resource, then you probably want to stick with shared memory.

As it sounds like you are having some sort client/server thing, I would say use either message queues, unix domain sockets, or named pipes. They all involve copying data in the kernel, so they are not quite as fast as shared memory, but they are still very fast. If you have message-like data (individual small packets), go with message queues. That is probably the cleanest solution. If you have more of a stream of data, use pipes or sockets. Sockets have the advantage that you can easily make it network transparent (like X11) later on if you want, but they are slightly harder to work with than pipes. The performance is probably very similar.

Comments

[SyRenity]

I updated the description due to "non-clear" score, hope it more clear now.

[SyRenity]

Only looking to move some binary data - but as fast as possible.

[SyRenity]

I'm actually using JNA for IPC and it works fine - but again, the general inconvenience of shared memory (segments leftover, etc...) are a killer.

[SyRenity]

Main consumer and several dumb (but highly efficient) producers.

[Aiden Bell]

@SyRenity , pretty fast. Depends on how much you lock and what locking you use really. Also +1, named pipes are cool.

[Duck]

Does Java support Unix domain sockets? You may need an add-on lib for that. If your data consists of a few fixed structures go with message queues. They are fast and easy to use. If the data is more variable and stream-like then go with named pipes.

[SyRenity]

Final question - what is faster? :) Named pipes or sockets?

[SyRenity]

Are named-pipes have same or very closed speed to shared memory?

[hieubk]

I would do a benchmark to test this on your platform. The performance will vary with the kernel implementation.

[Duck]

named pipes (fifos) are going to be slower than shared memory but that doesn't mean they aren't within tolerable limits for your app. Since you have multiple writers and one reader you are probably doing a lot of locking now. If your data is less than PIPE_BUF (usually around 4k) then your writes from various processes will be atomic and managed by the OS so that complexity is removed from your pgms. Alternatively you can have each writer use its own FIFO and multiplex your reader with something like select().

[SyRenity]

Actually I use a circular buffer in shared memory, which allows me to not require any locking at all. AFAIK this possible only in shared memory - but perhaps mmap can do it it? Dunno about pipes or sockets at this department.

[Duck]

CORBA would be overkill in this situation IMO. Its easy to forget how much work goes into setting up the ORBs, learning CORBA, IDL, etc. It's a daunting prospect for someone who just wants to pass some data on the same machine, much of it unstructured which is a bit of a PITA in CORBA.

[Nathaniel Sharp]

@Duck in my experience it's easier to learn how to use CORBA than to do cross language/platform communication manually correctly. Just to get all the marhalling/demarshalling right can be a nightmare. Not to mention all the nice services (naming, event, ...) that one can use out of the box.

[MK.]

Are you serious?? COBRA? In 2012? I'm sure it's a nice technology and the fact that it's dead is unfortunate and sad, but it's still dead.