Thank you, Kenneth.
Here is a simple C++ equivalent of what i am doing:
server.cpp:
=============================
#include <iostream>
#include <mpi/mpi.h>
#include <stdlib.h>
using namespace std;
// The only argument must be the number of processes in communicator we expect to build.
int main(int argc, char** argv)
{
int np = atoi(argv[1]);
int ac = 0;
MPI_Init(&ac, &argv);
char portName[MPI_MAX_PORT_NAME];
MPI_Open_port(MPI_INFO_NULL, portName);
cout << portName << "\n";
MPI_Comm intercomm, intracomm = MPI_COMM_SELF;
// Build an intracom dynamically until n processes are reached.
for (int i = 1; i < np; i++) {
MPI_Comm_accept(portName, MPI_INFO_NULL, 0, intracomm, &intercomm);
cout << "Accepted.\n";
MPI_Intercomm_merge(intercomm, false, &intracomm);
cout << "Merged to an intracom.\n";
MPI_Comm_free(&intercomm);
}
// Intracomm contains the one we can now use with n-grid.
MPI_Comm_free(&intracomm);
MPI_Close_port(portName);
MPI_Finalize();
}
client.cpp:
===============================
#include <iostream>
#include <mpi/mpi.h>
#include <stdlib.h>
using namespace std;
// This expects intracom size and the port name to connect to.
// When using with shell, use single quotas to avoid shell substitution.
int main(int argc, char** argv)
{
int ac = 0;
MPI_Init(&ac, &argv);
int np = atoi(argv[1]);
char* portName = argv[2];
cout << "Connecting to " << portName << "\n";
MPI_Comm intercomm, intracomm;
MPI_Comm_connect(portName, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm);
cout << "Connected.\n";
MPI_Intercomm_merge(intercomm, true, &intracomm);
cout << "Merged.\n";
MPI_Comm_free(&intercomm);
int i;
MPI_Comm_size(intracomm, &i);
// Build an intracom dynamically until n processes are reached.
for (; i < np; i++) {
MPI_Comm_accept(portName, MPI_INFO_NULL, 0, intracomm, &intercomm);
cout << "Accepted.\n";
MPI_Intercomm_merge(intercomm, false, &intracomm);
cout << "Merged to an intracom.\n";
MPI_Comm_free(&intercomm);
}
// Intracomm contains the one we can now use with n-grid.
MPI_Comm_free(&intracomm);
MPI_Finalize();
}
============================
Run example on one machine, intracom size=2 (in this case I have run 3.3a)
dmitriy@Intel-Kubu:~/projects/mpitests$ mpic++ server.cpp -o server
dmitriy@Intel-Kubu:~/projects/mpitests$ mpic++ client.cpp -o client
dmitriy@Intel-Kubu:~/projects/mpitests$
dmitriy@Intel-Kubu:~/projects/mpitests$
dmitriy@Intel-Kubu:~/projects/mpitests$ mpiexec ./server 2
tag#0$description#Intel-Kubu$port#39210$ifname#127.0.1.1$
Accepted.
Merged to an intracom.
dmitriy@Intel-Kubu:~/projects/mpitests$
(in another shell)
dmitriy@Intel-Kubu:~/projects/mpitests$ mpiexec ./client 2 'tag#0$description#Intel-Kubu$port#39210$ifname#127.0.1.1$'
Connecting to tag#0$description#Intel-Kubu$port#39210$ifname#127.0.1.1$
Connected.
Merged.
dmitriy@Intel-Kubu:~/projects/mpitests$
First parameter is eventual size of intracom we are trying to build dynamically, and client also needs to know the port reported by server process. So there's therfore 1 server and (n-1) clients that connect to form the intracom.
Now, if i do that for 192 processes on a 192 core cluster (occasionally slightly overloaded in terms of cpu load), I more often get a lock-up than not. The incidence is more frequent for 3.2 than 3.3a2. This cluster has mellanox infiniband.
3.2 usually locks up on merge intercom call; and 3.3a2 locked up at least once on 2 clients connected and waiting on merge intercom at the same time (but my understanding only one client should be connected at a time, even if there massive connects pending from other clients).
Hope this gives a little more material.
-Dmitriy