How to determine MPI rank/process number local to a socket/node

Question

Say, I run a parallel program using MPI. Execution command

mpirun -n 8 -npernode 2 <prg>

launches 8 processes in total. That is 2 processes per node and 4 nodes in total. (OpenMPI 1.5). Where a node comprises 1 CPU (dual core) and network interconnect between nodes is InfiniBand.

Now, the rank number (or process number) can be determined with

int myrank;
MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

This returns a number between 0 and 7.

But, How can I determine the node number (in this case a number between 0 and 3) and the process number within a node (number between 0 and 1)?

Solution 1

It depends on the MPI implementation - and there is no standard for this particular problem.

Open MPI has some environment variables that can help. OMPI_COMM_WORLD_LOCAL_RANK will give you the local rank within a node - ie. this is the process number which you are looking for. A call to getenv will therefore answer your problem - but this is not portable to other MPI implementations.

See this for the (short) list of variables in OpenMPI.

I don't know of a corresponding "node number".

Solution 2

I believe you can achieve that with MPI-3 in this manner:

MPI_Comm shmcomm;
MPI_Comm_split_type(MPI_COMM_WORLD, MPI_COMM_TYPE_SHARED, 0,
                    MPI_INFO_NULL, &shmcomm);
int shmrank;
MPI_Comm_rank(shmcomm, &shmrank);

Solution 3

This exact problem is discussed on Markus Wittmann's Blog, MPI Node-Local Rank determination.

There, three strategies are suggested:

1. A naive, portable solution employs MPI_Get_processor_name or gethostname to create an unique identifier for the node and performs an MPI_Alltoall on it. [...]
2. [Method 2] relies on MPI_Comm_split, which provides an easy way to split a communicator into subgroups (sub-communicators). [...]
3. Shared memory can be utilized, if available. [...]

For some working code (presumably LGPL licensed?), Wittmann links to MpiNodeRank.cpp from the APSM library.

Comments

[cxwangyi]

Thanks for the answer. The pasted link is broken. I found this link that works open-mpi.org/faq/?category=running#mpi-environmental-variabl‌​es

[Victor Eijkhout]

This should not be the accepted answer. Using MPI_Comm_split_type you can analyze the distribution over nodes without a problem. Maybe that didn't exist when this answer was written, but right now this answer is insufficient.

[Victor Eijkhout]

Strategy 2 is elegant and portable.

[Mark Gates]

@aleixrocks That link describing MPI_Comm_split_type is failing. Here's an updated one, which just strips off ?Aspx...