September 2018 - discuss - mailman3.cels.anl.gov

Getting an error in MPI_Comm_accept - ideas?
by Alexander Rast 21 Sep '18

21 Sep '18

All, I'm running an MPI_Comm_accept in a separate thread whose purpose is to allow connections from other (hitherto unknown) MPI universes. A well-known issue of such configurations is that MPI_Comm_accept being blocking, in order for the application to exit, a connection MUST be made even if no actual other universes attempted to connect. In this situation (see Gropp, et al. "Using Advanced MPI", section 6.5) it seems the 'customary' solution is to have the local universe connect to itself and then shut down. However, I'm getting the following error on exit: Fatal error in PMPI_Comm_accept: Message truncated, error stack: PMPI_Comm_accept(129).............: MPI_Comm_accept(port="tag#0$description# aesop.cl.cam.ac.uk$port#43337$ifname#128.232.98.176$", MPI_INFO_NULL, root=6, MPI_COMM_WORLD, newcomm=0x1473f24) failed MPID_Comm_accept(153).............: MPIDI_Comm_accept(1005)...........: MPIR_Bcast_intra(1249)............: MPIR_SMP_Bcast(1088)..............: MPIR_Bcast_binomial(239)..........: MPIDI_CH3U_Receive_data_found(131): Message from rank 0 and tag 2 truncated; 260 bytes received but buffer size is 12 You typically get several of these messages, the number seems to vary from trial to trial, I'm guessing it comes from different MPI processes (although there is no fixed relationship between number of processes started and number of error messages) Does anyone have any suggestions on what types of problems might cause this error? (I'm not expecting you to identify and debug the problem specifically, unless, perhaps, this error is indicative of some particular mistake, just would like any hints on where to look) If it helps, here are the 3 main relevant functions involved - there is, of course, a lot more going on in the application besides these, but the error is occurring at shutdown and by this point the rest of the application is quiescent. Also note that the Connect routine is NOT called at shutdown; it's invoked when a 'real' universe wants to connect. //------------------------------------------------------------------------------ void* CommonBase::Accept(void* par) /* Blocking routine to connect to another MPI universe by publishing a port. This operates in a separate thread to avoid blocking the whole process. */ { CommonBase* parent=static_cast<CommonBase*>(par); while (parent->AcceptConns.load(std::memory_order_relaxed)) { // run the blocking accept itself. if (MPI_Comm_accept(parent->MPIPort.load(std::memory_order_seq_cst),MPI_INFO_NULL,parent->Lrank.load(std::memory_order_relaxed),MPI_COMM_WORLD,parent->Tcomm.load(std::memory_order_seq_cst))) { printf("Error: attempt to connect to another MPI universe failed\n"); parent->AcceptConns.store(false,std::memory_order_relaxed); break; } // Now trigger the Connect process in the main thread to complete the setup PMsg_p Creq; string N(""); // zero-length string indicates a server-side connection Creq.Put(0,&N); Creq.Key(Q::SYST,Q::CONN); Creq.Src(0); Creq.comm = MPI_COMM_SELF; Creq.Send(0); while (*(parent->Tcomm.load(std::memory_order_seq_cst)) != MPI_COMM_NULL); // block until connect has succeeded } pthread_exit(par); return par; } //------------------------------------------------------------------------------ unsigned CommonBase::Connect(string svc) // connects this process' MPI universe to a remote universe that has published // a name to access it by. { int error = MPI_SUCCESS; // a server has its port already so can just open a comm if (svc=="") Comms.push_back(*Tcomm.load(std::memory_order_seq_cst)); else // clients need to look up the service name { MPI_Comm newcomm; char port[MPI_MAX_PORT_NAME]; // Get the published port for the service name asked for. // Exit if we don't get a port, probably because the remote universe isn't // initialised yet (we can always retry). if (error = MPI_Lookup_name(svc.c_str(),MPI_INFO_NULL,port)) return error; // now try to establish the connection itself. Again, we can always retry. if (error = MPI_Comm_connect(port,MPI_INFO_NULL,0,MPI_COMM_WORLD,&newcomm)) return error; Comms.push_back(newcomm); // as long as we succeeded, add to the list of comms } int rUsize; MPI_Comm_remote_size(Comms.back(), &rUsize); Usize.push_back(rUsize); // record the size of the remote universe FnMapx.push_back(new FnMap_t); // give the new comm some function tables to use pPmap.push_back(new ProcMap(this)); // and a new processor map for the remote group PMsg_p prMsg; SendPMap(Comms.back(), &prMsg); // Send our process data to the remote group int fIdx=FnMapx.size()-1; // populate the new function table with the global functions (*FnMapx[fIdx])[Msg_p::KEY(Q::EXIT )] = &CommonBase::OnExit; (*FnMapx[fIdx])[Msg_p::KEY(Q::PMAP )] = &CommonBase::OnPmap; (*FnMapx[fIdx])[Msg_p::KEY(Q::SYST,Q::PING,Q::ACK )] = &CommonBase::OnSystPingAck; (*FnMapx[fIdx])[Msg_p::KEY(Q::SYST,Q::PING,Q::REQ )] = &CommonBase::OnSystPingReq; (*FnMapx[fIdx])[Msg_p::KEY(Q::TEST,Q::FLOO )] = &CommonBase::OnTestFloo; if (svc=="") *Tcomm.load(std::memory_order_seq_cst) = MPI_COMM_NULL; // release any Accept comm. return error; } //------------------------------------------------------------------------------ unsigned CommonBase::OnExit(PMsg_p * Z,unsigned cIdx) // Do not post anything further here - the LogServer may have already gone { AcceptConns.store(false,std::memory_order_relaxed); // stop accepting connections if (acpt_running) { printf("(%s)::CommonBase::OnExit closing down Accept MPI request\n",Sderived.c_str()); fflush(stdout); // have to close the Accept thread via a matching MPI_Comm_connect // because the MPI interface has made no provision for a nonblocking accept, thus // otherwise the Accept thread will block forever waiting for a message that will // never come because we are shutting down. See Gropp, et al. "Using Advanced MPI" MPI_Comm dcomm; MPI_Comm_connect(MPIPort.load(std::memory_order_seq_cst),MPI_INFO_NULL,Lrank.load(std::memory_order_relaxed),MPI_COMM_WORLD,&dcomm); pthread_join(MPI_accept,NULL); acpt_running = false; } if (Urank == Lrank.load(std::memory_order_relaxed)) { MPI_Unpublish_name(MPISvc,MPI_INFO_NULL,MPIPort.load(std::memory_order_seq_cst)); MPI_Close_port(MPIPort.load(std::memory_order_seq_cst)); } return 1; } _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_f08 and Totalview
by John DelSignore 17 Sep '18

17 Sep '18

_______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpi_f08 and Totalview
by Lorenzo Zanon 17 Sep '18

17 Sep '18

Hello, Regarding totalview, I read the follwoing warning in my c.txt file: configure: WARNING: Some versions of gcc do not include debugging information within the executable. Totalview requires this information to detect an MPICH code. If you have trouble, try linking with the additional option +noobjdebug on all link lines (consider adding it to LDFLAGS) How should I modify my configure line? For now, it is: ../mpich-3.2.1/configure F77=gfortran FC=gfortran -prefix=/data/homes/zanon/software/mpich/mpich-install --enable-debuginfo --enable-g=dbg,mem,log --with-pm=hydra:gforker:remshell 2>&1 | tee c.txt Thanks! Lorenzo -------- Forwarded Message -------- Subject: mpi_f08 and Totalview Date: Mon, 17 Sep 2018 15:25:45 +0200 From: Lorenzo Zanon <zanon(a)mechbau.uni-stuttgart.de> To: discuss(a)mpich.org Hello, After the mpich installation I have these two issues: - I cannot use the module mpi_f08 in a Fortran program, only the old mpi module. The compilation error is: Fatal Error: Can't open module file ‘mpi_f08.mod’ for reading at (1): No such file or directory compilation terminated. - I cannot debug with Totalview. Totalview freezes when starting a debug session with "go" and a breakpoint, without error message. Maybe there was something wrong in my debug configuration options? Attached are my c.txt and m.txt files. Thanks for helping! Best, Lorenzo -- ============================ Lorenzo Zanon Institute of Applied Mechanics Continuum Mechanics and Mechanobiology University of Stuttgart Pfaffenwaldring 5a 70569 Stuttgart Germany zanon(a)mechbau.uni-stuttgart.de +49 (0)711 685 60 829 +49 (0)1577 2790439 ============================ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_f08 and Totalview
by Zaak Beekman 17 Sep '18

17 Sep '18

Hi Lorenzo, My understanding is that you must have ISO_Fortran_binding support in your Fortran compiler in order to use the mpi_f08 capabilities of MPICH. Currently, GFortran (as of 8.2) does not support ISO_Fortran_binding natively. However, you may try downloading and installing ISO_Fortran_binding from: https://github.com/sourceryinstitute/iso_fortran_binding. Currently, the tests make use of C VLAs, so if you compile with a C compiler other than GCC the tests may fail to build. However, the runtime *should* (AFAIK) still work, as I don't believe we used VLAs or any other (breaking) GNU extensions in the source itself. If you install ISO_Fortran_binding onto your default system search path before configuring MPICH, then I think MPICH will find it and enable the f08 module. I have no idea what is wrong with TotalView. Good luck! Zaak Izaak "Zaak" Beekman ------------------------------------------------------------------------------- HPC Scientist ibeekman(a)paratools.com ParaTools Inc. <http://www.paratools.com/> 1509 16th St, NW Washington, DC 20036 mobile: (917) 797-3239 ------------------------------------------------------------------------------- On Mon, Sep 17, 2018 at 9:25 AM <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. mpi_f08 and Totalview (Lorenzo Zanon) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 17 Sep 2018 15:25:45 +0200 > From: Lorenzo Zanon <zanon(a)mechbau.uni-stuttgart.de> > To: discuss(a)mpich.org > Subject: [mpich-discuss] mpi_f08 and Totalview > Message-ID: > <10a0c010-7304-9bac-eb75-6429c111526c(a)mechbau.uni-stuttgart.de> > Content-Type: text/plain; charset="utf-8"; Format="flowed" > > Hello, > > After the mpich installation I have these two issues: > > - I cannot use the module mpi_f08 in a Fortran program, only the old mpi > module. The compilation error is: > > Fatal Error: Can't open module file ?mpi_f08.mod? for reading at (1): No > such file or directory > compilation terminated. > > - I cannot debug with Totalview. Totalview freezes when starting a debug > session with "go" and a breakpoint, without error message. Maybe there > was something wrong in my debug configuration options? > > Attached are my c.txt and m.txt files. > > Thanks for helping! > Best, > Lorenzo > > -- > ============================ > Lorenzo Zanon > > Institute of Applied Mechanics > Continuum Mechanics and Mechanobiology > University of Stuttgart > > Pfaffenwaldring 5a > 70569 Stuttgart > Germany > > zanon(a)mechbau.uni-stuttgart.de > +49 (0)711 685 60 829 <+49%20711%2068560829> > +49 (0)1577 2790439 <+49%201577%202790439> > ============================ > > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpi_f08 and Totalview
by Lorenzo Zanon 17 Sep '18

17 Sep '18

Hello, After the mpich installation I have these two issues: - I cannot use the module mpi_f08 in a Fortran program, only the old mpi module. The compilation error is: Fatal Error: Can't open module file ‘mpi_f08.mod’ for reading at (1): No such file or directory compilation terminated. - I cannot debug with Totalview. Totalview freezes when starting a debug session with "go" and a breakpoint, without error message. Maybe there was something wrong in my debug configuration options? Attached are my c.txt and m.txt files. Thanks for helping! Best, Lorenzo -- ============================ Lorenzo Zanon Institute of Applied Mechanics Continuum Mechanics and Mechanobiology University of Stuttgart Pfaffenwaldring 5a 70569 Stuttgart Germany zanon(a)mechbau.uni-stuttgart.de +49 (0)711 685 60 829 +49 (0)1577 2790439 ============================ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Travis CI unit tests failing with hwloc "operating system" error
by Jeff Hammond 13 Sep '18

13 Sep '18

I just posted to the hwloc list but perhaps this is better solved via some mpirun option. I am running ARMCI-MPI over MPICH in a Travis CI Linux instance and topology is causing it to fail. I do not care about topology in a virtualized environment. How do I fix this? **************************************************************************** * hwloc 2.0.0rc2-git has encountered what looks like an error from the operating system. * * Group0 (cpuset 0x00001111,0x11111111) intersects with L3 (cpuset 0x00001000,0x02100002) without inclusion! * Error occurred in topology.c line 1384 * * The following FAQ entry in the hwloc documentation may help: * What should I do when hwloc reports "operating system" warnings? * Otherwise please report this error message to the hwloc user's mailing list * along with the files generated by the hwloc-gather-topology script. **************************************************************************** https://travis-ci.org/jeffhammond/armci-mpi/jobs/425342479 has all of the details. Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Problem setting up MPI cluster
by Ken Raffenetti 11 Sep '18

11 Sep '18

Hi, This type of heterogeneous configuration (arm + x86) is not supported with MPICH today. Ken On 9/10/18 9:03 AM, Matthew Overlin wrote: > Hello, all. I have been using MPI locally for awhile, but I am now > tryiong to set up an MPI cluster: my ubuntu desktop computer & a > raspberry pi with ubuntu mate. > > I am following the steps here > <https://help.ubuntu.com/community/MpichCluster>. > That is, on the client (rpi) side, I mount a drive on the master (desktop). > To test the set-up, I issue an mpirun command to get the hostname from > each IP address: > $ *mpirun -np 2 -hosts XX.XX.X.60, XX.XX.X.136 hostname* > userName-Z97X-UD3H-BK > [mpiexec@userName-Z97X-UD3H-BK] control_cb > (pm/pmiserv/pmiserv_cb.c:208): assert (!closed) failed > [mpiexec@userName-Z97X-UD3H-BK] HYDT_dmxu_poll_wait_for_event > (tools/demux/demux_poll.c:77): callback returned error status > [mpiexec@userName-Z97X-UD3H-BK] HYD_pmci_wait_for_completion > (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event > [mpiexec@userName-Z97X-UD3H-BK] main (ui/mpich/mpiexec.c:340): process > manager error waiting for completion > > I get a hostname from one IP address, but not the other IP address. > Would someone happen to know what is going on here? I'm not having much > luck with this error message, or from other people's advice on > forums/blogs/etc. I have mpich on both the desktop and raspberry pi. > Ideally, I want to have point-to-point communication example between the > two, some kind of inter-computer ("client/server") communication. > > Thanks! > Matt Overlin > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Problem setting up MPI cluster
by Matthew Overlin 10 Sep '18

10 Sep '18

Hello, all. I have been using MPI locally for awhile, but I am now tryiong to set up an MPI cluster: my ubuntu desktop computer & a raspberry pi with ubuntu mate. I am following the steps here <https://help.ubuntu.com/community/MpichCluster>. That is, on the client (rpi) side, I mount a drive on the master (desktop). To test the set-up, I issue an mpirun command to get the hostname from each IP address: $ *mpirun -np 2 -hosts XX.XX.X.60, XX.XX.X.136 hostname* userName-Z97X-UD3H-BK [mpiexec@userName-Z97X-UD3H-BK] control_cb (pm/pmiserv/pmiserv_cb.c:208): assert (!closed) failed [mpiexec@userName-Z97X-UD3H-BK] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec@userName-Z97X-UD3H-BK] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:198): error waiting for event [mpiexec@userName-Z97X-UD3H-BK] main (ui/mpich/mpiexec.c:340): process manager error waiting for completion I get a hostname from one IP address, but not the other IP address. Would someone happen to know what is going on here? I'm not having much luck with this error message, or from other people's advice on forums/blogs/etc. I have mpich on both the desktop and raspberry pi. Ideally, I want to have point-to-point communication example between the two, some kind of inter-computer ("client/server") communication. Thanks! Matt Overlin _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] using mpi4py in a Singluarity container run at a large computing center with Slurm installed
by Heather Kelly 01 Sep '18

01 Sep '18

Hi Martin, Thank you so much for taking the time to look at this so carefully - especially on a Friday before a holiday weekend! You reproduced the behavior I am seeing precisely, where things run fine on an interactive node. Just hacking something to work intra-node would be fine as a start. This gives me a much better understanding so I can play around more. I'll report back if I manage to make some progress or can come up with an intelligent question :) Take care, Heather On Fri, Aug 31, 2018 at 7:13 PM Martin Cuma <martin.cuma(a)utah.edu> wrote: > Hi Heather, > > this is a nicely complex problem that I can't say I know a solution of, > but, let me say what I know and perhaps it'll shed some light on the > problem. > > To answer your question on how mpirun interacts with srun (or SLURM in > general), most MPIs (or better to say, PMIs that MPI uses for process > launch) these days have SLURM support so when built they can leverage > SLURM. Or the SLURM is set up to facilitate the remote node connection > (e.g. by hijacking ssh through its own PMI - I don't know this just > guessing). So, for the MPI distros that I tried (MPICH and derivatives - > Intel MPI, MVAPICH2; and OpenMPI), mpirun at some point calls srun, no > matter if it was built with SLURM support explicitly or not. Which would > explain the srun error you are getting. > > Now, what I think is happening in your case is that you are calling the > mpirun (or its equivalent inside mpi4py) from INSIDE of the container, > where there's no srun. Notice that most MPI container examples, including > the very well written ANL page, instruct you to use mpirun (or aprun in > Cray's case) OUTSIDE of the container (the host), and launch N instances > of the container through the mpirun. > > I reproduced your problem on our system in the following way: > 1. Build a Singularity container with local MPI installation, e.g. > https://github.com/CHPC-UofU/Singularity-ubuntu-mpi > 2. shell into the container and build some mpi program (e.g. I have the > cpi.c example from mpich - mpicc cpi.c -o cpi). > 3. This then runs OK in the container on an interactive node (= outside > SLURM job - mpirun does not use SLURM's PMI = does not use srun). > 4. Launch the job, then shell into the container, and try to run mpirun > -np 2 ./cpi - I get the same error you get, since > $ which srun > $ > > Now, I can try to set the path to the SLURM binaries > $ export PATH="/uufs/notchpeak.peaks/sys/installdir/slurm/std/bin:$PATH" > $ which srun > /uufs/notchpeak.peaks/sys/installdir/slurm/std/bin/srun > but then get another error: > $ mpirun -np 2 ./cpi.c > srun: error: Invalid user for SlurmUser slurm, ignored > srun: fatal: Unable to process configuration file > so the environment needs some more changes to get the srun to work > correctly from inside the container. Though I think this would still only > be hackable for an intra-node MPI launch, as inter-node you'll rely on > SLURM that would have to get accessed from outside of the container. > > So, bottom line, launching mpirun from the host is preferable. > > I am not sure how you can launch the mpi4py from the host, since I don't > use mpi4py, but, in theory it should not be any different than launching > MPI binaries. Though I figure modifying your launch scripts around this > may be complicated. > > BTW, I have had a reasonable success with mixing ABI compatible MPIs > (MPICH, MVAPICH2, Intel MPI) in and out of the container. It often works > but sometimes it does not. > > HTH, > MC > > -- > Martin Cuma > Center for High Performance Computing > Department of Geology and Geophysics > University of Utah > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0