Condition Variable in Shared Memory - is this code POSIX-conformant?

11,076

Solution 1

I can easily see how PTHREAD_PROCESS_SHARED can be tricky to implement on the OS-level (e.g. MacOS doesn't, except for rwlocks it seems). But just from reading the standard, you seem to have a case.

For completeness, you might want to assert on sysconf(_SC_THREAD_PROCESS_SHARED) and the return value of the *_setpshared() function calls— maybe there's another "surprise" waiting for you (but I can see from the comments that you already checked that SHARED is actually supported).

@JesperE: you might want to refer to the API docs at the OpenGroup instead of the HP docs.

Solution 2

The pthread_mutexattr_setpshared function may be used to allow a pthread mutex in shared memory to be accessed by any thread which has access to that memory, even threads in different processes. According to this link, pthread_mutex_setpshared conforms to POSIX P1003.1c. (Same thing goes for the condition variable, see pthread_condattr_setpshared.)

Related question: pthread condition variables on Linux, odd behaviour

Solution 3

May be there is some pointers in pthread_cond_t (without pshared), so you must place it into the same addresses in both threads/processes. With same-ordered mmaps you may get a equal addresses for both processes.

In glibc the pointer in cond_t was to thread descriptor of thread, owned mutex/cond.

You can control addresses with non-NULL first parameter to mmap.

Share:
11,076

Related videos on Youtube

django11
Author by

django11

I'm a Senior Software Engineer, currently based near Newcastle (England) and working remotely for a company based in Fife (Scotland). I've been programming professionally since 1997 in the usual variety of languages (C, C++, C#, Java, Python, VB, HTML/CSS, etc) working on everything from low-level bit-bashing device drivers and microprocessor-based firmware for safety-critical medical devices, up to full Windows applications and enterprise systems. I hate cucumbers. If you're interested then here is my linkedin.com profile. SOreadytohelp

Updated on June 04, 2022

Comments

  • django11
    django11 over 1 year

    Does the POSIX standard allow a named shared memory block to contain a mutex and condition variable?

    We've been trying to use a mutex and condition variable to synchronise access to named shared memory by two processes on a LynuxWorks LynxOS-SE system (POSIX-conformant).

    One shared memory block is called "/sync" and contains the mutex and condition variable, the other is "/data" and contains the actual data we are syncing access to.

    We're seeing failures from pthread_cond_signal() if both processes don't perform the mmap() calls in exactly the same order, or if one process mmaps in some other piece of shared memory before it mmaps the "/sync" memory.

    This example code is about as short as I can make it:

    #include <sys/types.h>
    #include <sys/stat.h>
    #include <sys/mman.h>
    #include <sys/file.h>
    #include <stdlib.h>
    #include <pthread.h>
    #include <errno.h>
    #include <iostream>
    #include <string>
    using namespace std;
    
    static const string shm_name_sync("/sync");
    static const string shm_name_data("/data");
    
    struct shared_memory_sync
    {
      pthread_mutex_t mutex;
      pthread_cond_t condition;
    };
    
    struct shared_memory_data
    {
      int a;
      int b;
    };
    
    
    //Create 2 shared memory objects
    // - sync contains 2 shared synchronisation objects (mutex and condition)
    // - data not important 
    void create()
    {
      // Create and map 'sync' shared memory
      int fd_sync = shm_open(shm_name_sync.c_str(), O_CREAT|O_RDWR, S_IRUSR|S_IWUSR);
      ftruncate(fd_sync, sizeof(shared_memory_sync));
      void* addr_sync = mmap(0, sizeof(shared_memory_sync), PROT_READ|PROT_WRITE, MAP_SHARED, fd_sync, 0);
      shared_memory_sync* p_sync = static_cast<shared_memory_sync*> (addr_sync);
    
        // init the cond and mutex
      pthread_condattr_t cond_attr;
        pthread_condattr_init(&cond_attr);
        pthread_condattr_setpshared(&cond_attr, PTHREAD_PROCESS_SHARED);
        pthread_cond_init(&(p_sync->condition), &cond_attr);
        pthread_condattr_destroy(&cond_attr);
    
        pthread_mutexattr_t m_attr;
        pthread_mutexattr_init(&m_attr);
        pthread_mutexattr_setpshared(&m_attr, PTHREAD_PROCESS_SHARED);
        pthread_mutex_init(&(p_sync->mutex), &m_attr);
        pthread_mutexattr_destroy(&m_attr);
    
      // Create the 'data' shared memory   
      int fd_data = shm_open(shm_name_data.c_str(), O_CREAT|O_RDWR, S_IRUSR|S_IWUSR);
      ftruncate(fd_data, sizeof(shared_memory_data));
      
      void* addr_data = mmap(0, sizeof(shared_memory_data), PROT_READ|PROT_WRITE, MAP_SHARED, fd_data, 0);
      shared_memory_data* p_data = static_cast<shared_memory_data*> (addr_data);
    
      // Run the second process while it sleeps here.
      sleep(10);
    
      int res = pthread_cond_signal(&(p_sync->condition));
      assert(res==0);  // <--- !!!THIS ASSERT WILL FAIL ON LYNXOS!!!
    
      munmap(addr_sync, sizeof(shared_memory_sync));
      shm_unlink(shm_name_sync.c_str());
      munmap(addr_data, sizeof(shared_memory_data));
      shm_unlink(shm_name_data.c_str());
    }
    
    //Open the same 2 shared memory objects but in reverse order
    // - data
    // - sync 
    void open()
    {
      sleep(2);
      int fd_data = shm_open(shm_name_data.c_str(), O_RDWR, S_IRUSR|S_IWUSR);
      void* addr_data = mmap(0, sizeof(shared_memory_data), PROT_READ|PROT_WRITE, MAP_SHARED, fd_data, 0);
      shared_memory_data* p_data = static_cast<shared_memory_data*> (addr_data);
    
      int fd_sync = shm_open(shm_name_sync.c_str(), O_RDWR, S_IRUSR|S_IWUSR);
      void* addr_sync = mmap(0, sizeof(shared_memory_sync), PROT_READ|PROT_WRITE, MAP_SHARED, fd_sync, 0);
      shared_memory_sync* p_sync = static_cast<shared_memory_sync*> (addr_sync);
    
      // Wait on the condvar
      pthread_mutex_lock(&(p_sync->mutex));
      pthread_cond_wait(&(p_sync->condition), &(p_sync->mutex));
      pthread_mutex_unlock(&(p_sync->mutex));
      
      munmap(addr_sync, sizeof(shared_memory_sync));
      munmap(addr_data, sizeof(shared_memory_data));
    }
    
    int main(int argc, char** argv) 
    {
      if(argc>1)
      {
        open(); 
      }
      else
      {
        create();
      }
    
        return (0);
    }
    

    Run this program with no args, then another copy with args, and the first one will fail at the assert checking the pthread_cond_signal(). But change the order of the open() function to mmap() the "/sync" memory before the "/data" and it will all work fine.

    This seems like a major bug in LynxOS to me, but LynuxWorks claim that using mutex and condition variable within named shared memory in this way is not covered by the POSIX standard, so they are not interested.

    Can anyone determine if this code does actually violate POSIX?
    Or does anyone have any convincing documentation that it is POSIX compliant?

    Edit: we know that PTHREAD_PROCESS_SHARED is POSIX and is supported by LynxOS. The area of contention is whether mutexes and semaphores can be used within named shared memory (as we have done) or if POSIX only allows them to be used when one process creates and mmaps the shared memory and then forks the second process.

    • SF.
      SF. over 6 years
      I wonder how they exactly imagine sharing the same variable using PTHREAD_PROCESS_SHARED between two processes (which clearly implies some mechanism of sharing the variable between processes is supposed to exist.) And AFAIK no standard forbids placing mutexes and semaphores wherever you desire, so "not covered" means "should behave as usual."
  • django11
    django11 over 13 years
    Thanks @JesperE, I understand that pthread_mutex_setpshared and PTHREAD_PROCESS_SHARED are POSIX. I don't think LynuxWorks are denying that. I think the dispute is more about the way we are creating the shared memory that the mutex and condvar are in e.g. each process accessing via named shared memory, rather than just creating it in one process and then forking to create the other.
  • JesperE
    JesperE over 13 years
    Sorry, I read the question a little sloppily. The man-page says that "this option permits a mutex to be operated upon by any thread that has access to the memory where the mutex is allocated." It sounds to me that how the memory is shared is up to the user, and any other calls to mmap() should not affect the mutex/condition variable semantics. Why should the sharing of the data-area affect the mutex at all? What are LynuxWorks claiming the standard says? Are they refering to any place in the standard, or are they just handwaving?
  • django11
    django11 over 13 years
    Yeah LynuxWorks just seem to be handwaving and saying that if it isn't explicitly specified in POSIX then they don't support it. I agree with you: POSIX explicitly allows mutex and condvars to appear in shared memory (which LynxOS supports) - but I don't see anything in POSIX that limits how that shared memory is acquired by the processes that share it.
  • django11
    django11 over 13 years
    Thanks @vs: Yeah I clipped out the various other asserts and error handling to prevent the code for the sake of brevity, but rest assured that all the various calls do return success until the pthread_cond_signal() indicated. pthread_*_setpshared() is definitely supported and is explicitly mentioned in the LynxOS training materials (but they only provide examples where a process creates shared memory then fork()s, rather than two processes using named shared memory).
  • django11
    django11 over 13 years
    Yeah a pointer to the underlying mutex was our conclusion too, but since we've indicated that the condition variable will be shared between processes it seems like a bit of an implementation flaw if you also need to specify an mmap address (which is only supposed to be a hint anyway!) The only reliable way to use mmap address hints is to use the address returned by mmap() in the first process as the hint for the other process, which of course requires interprocess communication! :)
  • django11
    django11 over 13 years
    No real answer has been offered yet, so I'm going to award you the bounty as yours is the only one that has dealt with whether this is POSIX compliant or not.
  • osgx
    osgx over 13 years
    @GrahamS, process shared flag on cond_t will not change the actual internals of the structure. And if in implementation we have some pointers inside it, which will be used (hmm... e.g. pointer to cond_t itself?), even in process-shared environment there will be requirement of the equal addresses. the reliability of fixed mmap addresses depends on platform. It you know platform or have some configs (file-based pseudo IPC), you can select some mmap address.
  • Madars Vi
    Madars Vi almost 4 years
    See issue with MacOS openradar.appspot.com/19600706 problem is that cond variable and mutex must live in the same memory addresses. Thus for different processes addresses might be randomized....
  • Harry
    Harry about 2 years
    @GrahamS I'm also facing a similar kind of issue. Can you share the solution you've done to this problem. This is my post stackoverflow.com/questions/69605275/…