It has taken a while to condense this down to something small enough because the whole thing is embedded within a large and reasonably complex software infrastructure of which parts were not trivial to strip away. I have tested a few further cases including using an MPI_Improbe/MPI_Imreceive combination in the MPISpinner() without success. The code I am sending reliably crashes on the MPI_Comm_connect at the end, failing within MPI_Comm_accept with an error message essentially similar to the one I have in the previous message. It seems clear the bug is related to some sort of message race, because depending upon the exact sequence (within the actual application we can have numerous possible scenarios before the final ending MPI_Comm_connect), the application may exit cleanly, give an error similar to the above, or give a variety of other, different errors. It's timing/state dependent in some non-obvious way. I'm running MPICH 3.2.1 with, in this case, a fairly standard set of options: --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --enable-shared --prefix=/usr --enable-fortran=all --disable-rpath --disable-wrapper-rpath --sysconfdir=/etc/mpich --libdir=/usr/lib/x86_64-linux-gnu --includedir=/usr/include/mpich --docdir=/usr/share/doc/mpich --with-hwloc-prefix=system --enable-checkpointing --with-hydra-ckpointlib=blcr CPPFLAGS= CFLAGS= CXXFLAGS= FFLAGS= FCFLAGS= I also have a version with the performance-monitoring features enabled that I use from time to time, which also displays similar behaviour, although the exact output changes (presumably because differences in the performance-monitoring code change the execution timings or sequence). On Thu, Sep 27, 2018 at 2:39 PM Amer, Abdelhalim <[email protected]> wrote:
Hi,
Can you pack the smallest test example in a single file so that we compile and run it? Also please give us more information about the MPICH version you are using and how it was built (run the `mpichversion` binary to get this information). It would be better to use the latest version (3.2.1) before reporting back. All of this is to help us reproduce your problem or simply solving it by upgrading to a newer version of MPICH.
Halim www.mcs.anl.gov/~aamer
On 9/21/18 10:39 AM, Alexander Rast wrote:
All,
I'm running an MPI_Comm_accept in a separate thread whose purpose is to allow connections from other (hitherto unknown) MPI universes. A well-known issue of such configurations is that MPI_Comm_accept being blocking, in order for the application to exit, a connection MUST be made even if no actual other universes attempted to connect. In this situation (see Gropp, et al. "Using Advanced MPI", section 6.5) it seems the 'customary' solution is to have the local universe connect to itself and then shut down.
However, I'm getting the following error on exit:
Fatal error in PMPI_Comm_accept: Message truncated, error stack: PMPI_Comm_accept(129).............: MPI_Comm_accept(port="tag#0$description#aesop.cl.cam.ac.uk <http://aesop.cl.cam.ac.uk>$port#43337$ifname#128.232.98.176$", MPI_INFO_NULL, root=6, MPI_COMM_WORLD, newcomm=0x1473f24) failed MPID_Comm_accept(153).............: MPIDI_Comm_accept(1005)...........: MPIR_Bcast_intra(1249)............: MPIR_SMP_Bcast(1088)..............: MPIR_Bcast_binomial(239)..........: MPIDI_CH3U_Receive_data_found(131): Message from rank 0 and tag 2 truncated; 260 bytes received but buffer size is 12
You typically get several of these messages, the number seems to vary from trial to trial, I'm guessing it comes from different MPI processes (although there is no fixed relationship between number of processes started and number of error messages)
Does anyone have any suggestions on what types of problems might cause this error? (I'm not expecting you to identify and debug the problem specifically, unless, perhaps, this error is indicative of some particular mistake, just would like any hints on where to look)
If it helps, here are the 3 main relevant functions involved - there is, of course, a lot more going on in the application besides these, but the error is occurring at shutdown and by this point the rest of the application is quiescent. Also note that the Connect routine is NOT called at shutdown; it's invoked when a 'real' universe wants to connect.
//------------------------------------------------------------------------------
void* CommonBase::Accept(void* par) /* Blocking routine to connect to another MPI universe by publishing a
port.
This operates in a separate thread to avoid blocking the whole process. */ { CommonBase* parent=static_cast<CommonBase*>(par);
while (parent->AcceptConns.load(std::memory_order_relaxed)) { // run the blocking accept itself. if
(MPI_Comm_accept(parent->MPIPort.load(std::memory_order_seq_cst),MPI_INFO_NULL,parent->Lrank.load(std::memory_order_relaxed),MPI_COMM_WORLD,parent->Tcomm.load(std::memory_order_seq_cst)))
{ printf("Error: attempt to connect to another MPI universe failed\n"); parent->AcceptConns.store(false,std::memory_order_relaxed); break; } // Now trigger the Connect process in the main thread to complete the setup PMsg_p Creq; string N(""); // zero-length string indicates a server-side connection Creq.Put(0,&N); Creq.Key(Q::SYST,Q::CONN); Creq.Src(0); Creq.comm = MPI_COMM_SELF; Creq.Send(0); while (*(parent->Tcomm.load(std::memory_order_seq_cst)) != MPI_COMM_NULL); // block until connect has succeeded } pthread_exit(par); return par; }
//------------------------------------------------------------------------------
unsigned CommonBase::Connect(string svc) // connects this process' MPI universe to a remote universe that has published // a name to access it by. { int error = MPI_SUCCESS; // a server has its port already so can just open a comm if (svc=="") Comms.push_back(*Tcomm.load(std::memory_order_seq_cst)); else // clients need to look up the service name { MPI_Comm newcomm; char port[MPI_MAX_PORT_NAME]; // Get the published port for the service name asked for. // Exit if we don't get a port, probably because the remote universe isn't // initialised yet (we can always retry). if (error = MPI_Lookup_name(svc.c_str(),MPI_INFO_NULL,port)) return error; // now try to establish the connection itself. Again, we can always retry. if (error = MPI_Comm_connect(port,MPI_INFO_NULL,0,MPI_COMM_WORLD,&newcomm)) return error; Comms.push_back(newcomm); // as long as we succeeded, add to the list of comms } int rUsize; MPI_Comm_remote_size(Comms.back(), &rUsize); Usize.push_back(rUsize); // record the size of the remote universe FnMapx.push_back(new FnMap_t); // give the new comm some function tables to use pPmap.push_back(new ProcMap(this)); // and a new processor map for the remote group PMsg_p prMsg; SendPMap(Comms.back(), &prMsg); // Send our process data to the remote group int fIdx=FnMapx.size()-1; // populate the new function table with the global functions (*FnMapx[fIdx])[Msg_p::KEY(Q::EXIT )] =
&CommonBase::OnExit;
(*FnMapx[fIdx])[Msg_p::KEY(Q::PMAP )] = &CommonBase::OnPmap; (*FnMapx[fIdx])[Msg_p::KEY(Q::SYST,Q::PING,Q::ACK )] = &CommonBase::OnSystPingAck; (*FnMapx[fIdx])[Msg_p::KEY(Q::SYST,Q::PING,Q::REQ )] = &CommonBase::OnSystPingReq; (*FnMapx[fIdx])[Msg_p::KEY(Q::TEST,Q::FLOO )] = &CommonBase::OnTestFloo; if (svc=="") *Tcomm.load(std::memory_order_seq_cst) = MPI_COMM_NULL; // release any Accept comm. return error; }
//------------------------------------------------------------------------------
unsigned CommonBase::OnExit(PMsg_p * Z,unsigned cIdx) // Do not post anything further here - the LogServer may have already
gone
{ AcceptConns.store(false,std::memory_order_relaxed); // stop accepting connections if (acpt_running) { printf("(%s)::CommonBase::OnExit closing down Accept MPI request\n",Sderived.c_str()); fflush(stdout); // have to close the Accept thread via a matching MPI_Comm_connect // because the MPI interface has made no provision for a nonblocking accept, thus // otherwise the Accept thread will block forever waiting for a message that will // never come because we are shutting down. See Gropp, et al. "Using Advanced MPI" MPI_Comm dcomm;
MPI_Comm_connect(MPIPort.load(std::memory_order_seq_cst),MPI_INFO_NULL,Lrank.load(std::memory_order_relaxed),MPI_COMM_WORLD,&dcomm);
pthread_join(MPI_accept,NULL); acpt_running = false; } if (Urank == Lrank.load(std::memory_order_relaxed)) {
MPI_Unpublish_name(MPISvc,MPI_INFO_NULL,MPIPort.load(std::memory_order_seq_cst));
MPI_Close_port(MPIPort.load(std::memory_order_seq_cst)); } return 1; }
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss
_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss