Sharing data across processes on linux

Question

In my application, I have a process which forks off a child, say child1, and this child process writes a huge binary file on the disk and exits. The parent process then forks off another child process, child2, which reads in this huge file to do further processing.

The file dumping and re-loading is making my application slow and I'm thinking of possible ways of avoiding disk I/O completely. Possible ways I have identified are ram-disk or tmpfs. Can I somehow implement ram-disk or tmpfs from within my application? Or is there any other way by which I can avoid disk I/O completely and send data across processes reliably.

Solution 1

If the two sub-processes do not run at the same time pipes or sockets won't work for you – their buffers would be too small for the 'huge binary file' and the first process will block waiting for anything for reading the data.

In such case you rather need some kind of shared memory. You can use the SysV IPC shared memory API, POSIX shared memory API (which internally uses tmpfs on recent Linux) or use files on a tmpfs (usually mounted on /dev/shm, sometimes on /tmp) file system directly.

Solution 2

Create an anonymous shared memory region before forking and then all children can use it after the fork:

char *shared = mmap(0,size,PROT_READ|PROT_WRITE,MAP_SHARED|MAP_ANONYMOUS,-1,0);

Be aware that you'll need some synchronization mechanism when sharing memory. One way to accomplish this is to put a mutex or semaphore inside the shared memory region.

Solution 3

A named pipe is exactly what you want. You can write data into it and read data from it like it was a file, but there's no need to store it on disk.

Solution 4

You can use pipes, sockets, and take advantage of sendfile() or splice() features of Linux kernel (they can avoid data copying).

View more solutions

Comments

[Jacek Konieczny]

Why named pipes, when the processes are forked from a single process and can share the file descriptors directly?

[user900563]

Yes, the sub-process do not run at the same time but the parent process in active till the last child exits. I have been considering using files on tmpfs file system directly but don't know how to implement them from inside a program. Any pointers?

[user900563]

Actually the 2 child processes don't exist at the same time. The first one finishes work, dumps the file and exits, after which the next child is forked which loads this file and does rest of the processing. Can I set up a socket between parent-child1 and then parent-child2?

[John Zwinck]

How about just letting them run at the same time? It might drastically reduce the total running time as a side benefit.

[Sebi]

Welcome on SO. Please explain how to solve the problem and give some example of it. You can read this: stackoverflow.com/help/how-to-answer