Why are the IPC-Mechanism between threads?

Question

I have a doubt that, threads are share all the segments of process, except the stack. So, to communicate between threads, let say i want to pass a word "Hello" from one thread to another, what is the need of IPC mechanisms (such as message queues).

Solution 1

There are all sorts of ways of communicating data between threads in a process and also to threads in other processes. You just pick one that suits your needs.

Shared Memory

Threads in a single process can access all the memory in a process, though as you say they have their own stacks. This means that it is fairly straight forward for one thread to share a piece of memory with another thread. Normally you would use a semaphore of some sort to ensure that they don't both try to access the memory at the same time (which would normally lead to very confusing program behaviour...).

The code is fast and efficient because the data is never copied, but it can be very difficult to get complex programs right.

Message Queues, Pipes, etc.

However, something like a message queue is a way of sending data from memory used by one thread to memory used by another.

There's a lot of advantages because it's often far easier to write a complex program by using memory queues than by sharing memory. However, a memory queue is inherently less efficient. This is because the message queue implementation in the operating system has to copy the data (often twice).

Threads vs Processes

With the shared memory approach it is natural to have all the threads in a single process, whereas with message queues it begins to be more natural to place threads in separate processes. It doesn't really matter though. Most operating systems allow processes to expose some of their memory to other processes (/dev/shm on Linux), and named semaphores can be picked up by any process.

So architecturally speaking the question of "all threads in one process" versus "all threads in their own process" doesn't really matter. You might need to bear in mind the thread vs process context switch time of the operating system, because that has an impact on efficiency too.

Scalability

If you choose an IPC mechanism like a pipe then there is an opportunity for scalability. Pipes in particular behave very much like sockets. So, if your program is running too slow on your computer it isn't that much work to turn the pipes into sockets and place the threads in processes spread across two or more computers. So using pipes means that it's not that difficult to turn your application into multi-computer distributed application.

Of course, a socket is a lot slower, so you have to consider how much data per second you'd actually need to send from one thread to another before settling on that.

CPU Memory Architectures

Remember that I said that memory queues are less efficient than shared memory? Well, that's not so clear cut these days. Intel's current architecture, and AMD's in particular, means that copying data isn't so very different to just reading it.

Look at Intel's QPI link between chips and think about what's actually going on at the level of the electronics. Assume that you have two filled CPU sockets in your machine, and that your application has two threads sharing a memory buffer.

For a thread on one chip to access data that's resident in memory of the other chip it has to read that data across the QPI link.

Now, imagine that those threads used a message queue instead. The OS has to copy the data from one thread to another. That means reading the data across the QPI link. It's roughly a similar amount of activity on the QPI link.

AMD's architecture makes that more blatant; their equivalent (and superior...) of QPI is called Hypertransport, and that runs around inside AMD's chips.

It's not quite that straight forward. Cache's make things a lot more complicated. But for some programs with a lot of threads spread across two or more chips sharing a lot of data the QPI has the potential to slow everything up. Intel of course know that and design it so that normally QPI isn't a bottleneck. But if your program is not what Intel had in mind when they chose QPI's parameters then performance might be less than desired.

Effectively it is possible in some circumstances and with a lot of care to use pipes, message queues, etc. and end up with better performance than a program using shared memory.

However, I would still choose pipes and message queues for the simplicity and reliability reasons.

Solution 2

You really should take several hours to learn more by reading a good Pthreads Tutorial. You probably should understand more the basic IPC (Inter-Process Communication) machinery e.g. by reading Advanced Linux Programming

With a C++ compiler supporting the latest C++11 standard you could use its C++ thread library (often built above pthreads). You want GCC 4.8 or better or Clang 3.3 or better for this (use -std=c++11 flag to g++ or clang++ compiler).

All the threads of some given process are sharing the same address space (and often use hardware assisted cache coherence) thru some memory model.

^{I suggest you to learn more about proc(5). The command cat /proc/1234/maps shows you the address space of process of pid 1234.}

In particular, the stack is not really an isolated segment of each thread: in other words, a thread can access some data on some other thread's stack. For example, you can have a thread with a local int x; variable on its stack writing the address &x; somewhere (e.g. in a global variable g with g = &x; C statement) and have another thread dereference that pointer thru *g (but that is a poor programming style). All the stacks of threads of a given process are still in the same address space.

However, what you really need is to synchronize threads. A major purpose of pthreads is to give you several means for synchronization (mutexes and condition variables, barriers, semaphores, etc...). Recent language standards also have atomicity constructs like std::atomic of C++11 or stdatomic.h of C11. Beware of race conditions and deadlocks! Debugging multi-threaded (or other parallel) programs is painful because of possible heisenbugs. You probably should favor a functional programming style.

Multi-threaded programming is hard, because parallel programming is difficult.

You can use IPC mechanism between threads, for instance you could have a pipe(2) and have one thread writing it and another reading it (perhaps multiplexing with poll(2) ...). But people often prefer using pthread mechanisms (e.g. have a global data link some linked list and serially access it by locking mutexes) when coding threads. You'll then need to signal with a condition variable the transition from empty to non-empty and wait for that condition when taking an element from an (empty) linked list. This is a producer-consumer situation. You could have a shared queue, or a list etc... for strings like your "Hello World" but you still need to synchronize threads.

Comments

[Todd Li]

Do you realize IPC stands for "inter-process communication"? To communicate between threads, you simply put the word in an in-memory object that both threads know..

[dar7yl]

You need IPC mechanisms to synchronize between the tasks/processes/threads. For instance, you need to avoid "race" conditions where one thread is trying to read the data before the other thread produces it.

[Stabledog]

You don't need IPC mechanisms to synchronize threads -- thread synchronization and IPC are two separate concepts.