Sharing memory across multiple computers?

Question

I'd like to share certain memory areas around multiple computers, that is, for a C/C++ project. When something on computer B accesses some memory area which is currently on computer A, that has to be locked on A and sent to B. I'm fine when its only linux compitable.

Thanks in advance :D

Solution 1

You cannot do this for a simple C/C++ project.

Common computer hardware does not have the physical properties that support this directly: Memory on one system cannot be read by another system.

In order to make it appear to C/C++ programs on different machines that they are sharing memory, you have to write software that provides this function. Typically, you would need to do something like this:

1. Allocate some pages in the virtual memory address space (of each process).
2. Mark those pages read-only.
3. Set a handler to receive the exception that occurs when the process attempts to write to the read-only memory. (This handler might be in the operating system, as some sort of kernel extension, or it might be a signal handler in your process.)
4. When the exception is received, determine what the process was attempting to write to memory. Write that to the page (perhaps by writing it through a separate mapping in virtual memory to the same physical memory, with this extra mapping marked writeable).
5. Send a message by network communications to the other machine telling it that memory has changed.
6. Resume execution in the process after the instruction that wrote to memory.

Additionally, you need to determine what to do about memory coherence: If two processes write to the same address in memory at nearly the same time, what happens? If process A writes to location X and then reads location Y while, at nearly the same time, process B writes to location Y and reads X, what do they see? Is it okay if the two processes see data that cannot possibly be the result of a single time sequence of writes to memory?

On top of all that, this is hugely expensive in time: Stores to memory that require exception handling and network operations take many thousands, likely hundreds of thousands, times as long as normal stores to memory. Your processes will execute excruciatingly slowly whenever they write to this shared memory.

Solution 2

There are software solutions, as noted in the comments. These use the paging hardware in the processors on a node to detect access, and use your local network fabric to disseminate the changes to the memory. One hardware alternative is reflective memory - you can read more about it here:

https://en.wikipedia.org/wiki/Reflective_memory http://www.ecrin.com/embedded/downloads/reflectiveMemory.pdf

Old page was broken

http://www.dolphinics.com/solutions/embedded-system-reflective-memory.html

Reflective memory provides low latency (about one microsecond per hop) in either a ring or tree configuration.

Comments

[RageD]

You should first explore a topic called Distributed Shared Memory (i.e. google.com/…)

[Nikolai Fetissov]

Read up on RDMA: en.wikipedia.org/wiki/Remote_direct_memory_access

[imacake]

This is exactly what i want to do. Do you know anything more about writing my own kernel extension? Also, I want to send code through the network, too, so that code and data being operated on will be on the same computer, and that'll be quick. the results will be shared only. Its about the possibilities. The problem you see in syncronisation is weird to me. just lock the page before sending it?

[Eric Postpischil]

These are complex issues and require a great deal of study and work. I can only recommend learning about each piece individually first. How do you change pages on your system to read-only? How do you create maps with two different sets of virtual addresses to the same memory? How do you write a signal handler? How do you decode the instruction that caused an exception? How do you send messages over the network?

[Eric Postpischil]

Simply locking the page is likely not a solution to the synchronization issue. If you lock the page and send an update to the other system, what happens if the other system sends an update at the same time? Do you reject it? Then what is the other system supposed to do, since it already applied its update locally and released the process to continue executing?

[imacake]

Do you know any system like that for linux which does the job? I've already looked at tldp.org/LDP/lkmpg/2.6/html/lkmpg.html here and I think it wouldn't be that much of a hard job, it'd just take time(too much time).

[Eric Postpischil]

I cannot help further due to the large amount of work required and believe that, in the absence of compelling reason, dissuading you from pursuing this is the best course of action. Any practical problem likely has a better solution.

[Ryan Brown]

It's a C++ compiler problem. The CPU accesses memory remotely in exactly the same way but C++ is built for that, whereas it's not built for other remote communications.