September 2017 - discuss - mailman3.cels.anl.gov

Problem in configure of MPICH3.2
by Xu ZHANG 13 Sep '17

13 Sep '17

Hello, I face some problem when "./configure" the mpich3.2. How can I solve the problem? Thank you. Details in c.txt is attached. ------------------ Best Regards! Sincerely, Dr. Xu ZHANG（张旭） Haixi Institutes, Chinese Academy of Sciences Haixi Hi-Tech Park, Shangjie, Minhou, Fuzhou, Fujian 350108, P. R. China E-Mail: wyszhangxu(a)foxmail.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Rsend advantage
by Dorier, Matthieu 06 Sep '17

06 Sep '17

Thanks, Jeff! Matthieu ________________________________ From: Jeff Hammond <jeff.science(a)gmail.com> Sent: Wednesday, September 6, 2017 10:33:35 PM To: MPICH Subject: Re: [mpich-discuss] MPI_Rsend advantage I skimmed Ch3 and there is no difference for rendezvous. For eager, the Rsend doesn't bother with the unnecessary MPI_THREAD_MULTIPLE deadlock check and the short eager optimization (payload in header). Thus, I would expect no benefit due to Rsend and potentially a performance loss for small messages, if the short eager path is worthwhile. In Ch4, the code paths for Send and Rsend are identical in src/mpid/ch4/src/ch4_send.h. Because most applications and benchmarks use the normal Send-Recv (including nonblocking) variants, this is where the optimization work happens. Unless you use an implementation that documents an optimized implementation of Rsend, I would not bother using it. Jeff On Wed, Sep 6, 2017 at 12:58 PM, Dorier, Matthieu <mdorier(a)anl.gov<mailto:[email protected]>> wrote: > > Let's say process A MPI_Sends a message to process B, as a result process B posts an MPI_Irecv then MPI_Sends a message to A to notify it that the receive was posted, following which A calls MPI_Rsend. > > > It's a dumb example, but just to say that in some scenarios it's possible to know in advance that an MPI_Recv or an MPI_Irecv was posted. > > > No my question is, what does Mpich optimizes within MPI_Rsend; what is done differently from MPI_Send? > > > Thanks, > > > Matthieu > > ________________________________ > From: Atchley, Scott <atchleyes(a)ornl.gov<mailto:[email protected]>> > Sent: Wednesday, September 6, 2017 7:41:54 PM > To: discuss(a)mpich.org<mailto:[email protected]> > Subject: Re: [mpich-discuss] MPI_Rsend advantage > > If you can guarantee that it is posted, then you can guarantee that it will match. If you cannot guarantee that it is posted and will match, then it may be placed in the unexpected buffer and will need to be copied into the final location once the receive is posted. > > My question is, “How do you know it has been posted?” > > > > On Sep 6, 2017, at 6:21 AM, Dorier, Matthieu <mdorier(a)anl.gov<mailto:[email protected]>> wrote: > > > > Hi, > > > > Out of curiosity, what gain is there to using MPI_Rsend instead of MPI_Send when I know that the receiver has posted a receive? Do I avoid a rendez-vous protocol? Do I avoid a buffer copy somewhere? > > > > Thanks, > > > > Matthieu > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Rsend advantage
by Jeff Hammond 06 Sep '17

06 Sep '17

I skimmed Ch3 and there is no difference for rendezvous. For eager, the Rsend doesn't bother with the unnecessary MPI_THREAD_MULTIPLE deadlock check and the short eager optimization (payload in header). Thus, I would expect no benefit due to Rsend and potentially a performance loss for small messages, if the short eager path is worthwhile. In Ch4, the code paths for Send and Rsend are identical in src/mpid/ch4/src/ch4_send.h. Because most applications and benchmarks use the normal Send-Recv (including nonblocking) variants, this is where the optimization work happens. Unless you use an implementation that documents an optimized implementation of Rsend, I would not bother using it. Jeff On Wed, Sep 6, 2017 at 12:58 PM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: > > Let's say process A MPI_Sends a message to process B, as a result process B posts an MPI_Irecv then MPI_Sends a message to A to notify it that the receive was posted, following which A calls MPI_Rsend. > > > It's a dumb example, but just to say that in some scenarios it's possible to know in advance that an MPI_Recv or an MPI_Irecv was posted. > > > No my question is, what does Mpich optimizes within MPI_Rsend; what is done differently from MPI_Send? > > > Thanks, > > > Matthieu > > ________________________________ > From: Atchley, Scott <atchleyes(a)ornl.gov> > Sent: Wednesday, September 6, 2017 7:41:54 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] MPI_Rsend advantage > > If you can guarantee that it is posted, then you can guarantee that it will match. If you cannot guarantee that it is posted and will match, then it may be placed in the unexpected buffer and will need to be copied into the final location once the receive is posted. > > My question is, “How do you know it has been posted?” > > > > On Sep 6, 2017, at 6:21 AM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: > > > > Hi, > > > > Out of curiosity, what gain is there to using MPI_Rsend instead of MPI_Send when I know that the receiver has posted a receive? Do I avoid a rendez-vous protocol? Do I avoid a buffer copy somewhere? > > > > Thanks, > > > > Matthieu > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Rsend advantage
by Dorier, Matthieu 06 Sep '17

06 Sep '17

Let's say process A MPI_Sends a message to process B, as a result process B posts an MPI_Irecv then MPI_Sends a message to A to notify it that the receive was posted, following which A calls MPI_Rsend. It's a dumb example, but just to say that in some scenarios it's possible to know in advance that an MPI_Recv or an MPI_Irecv was posted. No my question is, what does Mpich optimizes within MPI_Rsend; what is done differently from MPI_Send? Thanks, Matthieu ________________________________ From: Atchley, Scott <atchleyes(a)ornl.gov> Sent: Wednesday, September 6, 2017 7:41:54 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] MPI_Rsend advantage If you can guarantee that it is posted, then you can guarantee that it will match. If you cannot guarantee that it is posted and will match, then it may be placed in the unexpected buffer and will need to be copied into the final location once the receive is posted. My question is, “How do you know it has been posted?” > On Sep 6, 2017, at 6:21 AM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: > > Hi, > > Out of curiosity, what gain is there to using MPI_Rsend instead of MPI_Send when I know that the receiver has posted a receive? Do I avoid a rendez-vous protocol? Do I avoid a buffer copy somewhere? > > Thanks, > > Matthieu > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Rsend advantage
by Atchley, Scott 06 Sep '17

06 Sep '17

If you can guarantee that it is posted, then you can guarantee that it will match. If you cannot guarantee that it is posted and will match, then it may be placed in the unexpected buffer and will need to be copied into the final location once the receive is posted. My question is, “How do you know it has been posted?” > On Sep 6, 2017, at 6:21 AM, Dorier, Matthieu <mdorier(a)anl.gov> wrote: > > Hi, > > Out of curiosity, what gain is there to using MPI_Rsend instead of MPI_Send when I know that the receiver has posted a receive? Do I avoid a rendez-vous protocol? Do I avoid a buffer copy somewhere? > > Thanks, > > Matthieu > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Code works with -ppn, fails without using MPICH 3.2
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 06 Sep '17

06 Sep '17

On 09/06/2017 11:02 AM, Kenneth Raffenetti wrote: > On 09/05/2017 10:28 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND > APPLICATIONS INC] wrote: >> An update! >> >> I started reading all the MPICH wiki pages I could find and thought I >> should try -hosts or -f, and that *does* work: >> >>> (1024) $ mpirun -f machinefile -np 96 ./GEOSgcm.x >>> srun.slurm: cluster configuration lacks support for cpu binding >>> >>> In MAPL_Shmem: >>> NumCores per Node varies from 12 to 28 >>> NumNodes in use = 4 >>> Total PEs = 96 >>> >> >> So, I guess the answer is that MPICH 3.2 can't quite decode the SLURM >> environment to figure out a machinefile, so I need to make one myself. > > Can you check the config.log in src/pm/hydra and see if slurm was > detected? If not, you can specify --with-slurm=<path/to/install>. Hydra > should be able to detect and understand the slurm host list from the job > environment. I tried compiling three different ways: If I just add --with-slurm, nothing happens. Configure seems not to react and I get the same mpirun behavior. If I add --with-slurm and --with-pmi=slurm, configure dies saying: configure: error: The PM chosen (hydra) requires the PMI implementation simple but slurm was selected as the PMI implementation. If I do --with-slurm --with-pmi=slurm --with-pm=none, then I don't get mpiexec.hydra built. So srun is all that can work. > >> >> Would this be the best way to do this, or is there a way to >> build/configure MPICH to better support this? >> >> Next up: trying to figure out how to get Inifiniband supported as I >> think I'm using TCP: > > If you are using Mellanox infiniband, try --with-device=ch3:nemesis:mxm > --with-mxm=<path/to/install>. The MXM library is part of MOFED, or can > be downloaded separately from the Mellanox website. Now this I was able to do, and I could see from the configure that it found it and did something different. The problem? If I run a simple Hello World, the code locks up after printing "Hello world from..." It's like it never finalizes, or gets stuck somewhere. Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] Code works with -ppn, fails without using MPICH 3.2
by Kenneth Raffenetti 06 Sep '17

06 Sep '17

On 09/05/2017 10:28 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote: > An update! > > I started reading all the MPICH wiki pages I could find and thought I > should try -hosts or -f, and that *does* work: > >> (1024) $ mpirun -f machinefile -np 96 ./GEOSgcm.x >> srun.slurm: cluster configuration lacks support for cpu binding >> >> In MAPL_Shmem: >> NumCores per Node varies from 12 to 28 >> NumNodes in use = 4 >> Total PEs = 96 >> > > So, I guess the answer is that MPICH 3.2 can't quite decode the SLURM > environment to figure out a machinefile, so I need to make one myself. Can you check the config.log in src/pm/hydra and see if slurm was detected? If not, you can specify --with-slurm=<path/to/install>. Hydra should be able to detect and understand the slurm host list from the job environment. > > Would this be the best way to do this, or is there a way to > build/configure MPICH to better support this? > > Next up: trying to figure out how to get Inifiniband supported as I > think I'm using TCP: If you are using Mellanox infiniband, try --with-device=ch3:nemesis:mxm --with-mxm=<path/to/install>. The MXM library is part of MOFED, or can be downloaded separately from the Mellanox website. Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_comm_spawn - process not destroyed
by Stanislav Simko 06 Sep '17

06 Sep '17

Hi Min, I will have a look into it when I get a bit of time, but it's strange, because I still see it. I've tested similar c++ code as well, and the same problem is there - I see 3 processes running during the computation in main program, after the workers should be finished. Also, I have tested on Debian 8.7 in VirtualBox with mpich from distro repositories v3.1-5+b2 with the same outcome. I'll try to get back with new info asap. If interested, here's the c++ code with removed calculation from worker but added to the main program:mpi.cpp:#include <mpi.h>#include <math.h>#include <iostream> int main(int argc, char *argv[]){ MPI_Comm everyone; MPI_Init(&argc, &argv); double c = 0.; std::cout << "starting worker\n"; for(int i=0;i<2;i++){ MPI_Comm_spawn("./worker",MPI_ARGV_NULL,1,MPI_INFO _NULL,0,MPI_COMM_SELF, &everyone,MPI_ERRCODES_IGNORE); MPI_Barrier(everyone); } std::cout << "parent rolling again & doing stuff" << "\n"; for(int i=0;i<50000000;i++){ c = i*i+1- (i*10) + sin(pow(i,i%8)); } std::cout << "parent finished stuff: " << c << "\n"; MPI_Finalize(); return 0;} worker.cpp:#include <mpi.h>#include <iostream>using namespace std; int main(int argc, char *argv[]){ double c; MPI_Init(&argc, &argv); MPI_Comm parent; MPI_Comm_get_parent(&parent); cout << "worker inside worker finished stuff: " << c << "\n"; if(parent != MPI_COMM_NULL){ MPI_Barrier(parent); } MPI_Finalize(); return 0;} best,stanislav. On Tue, 2017-09-05 at 16:27 -1000, Min Si wrote: > Hi Stanislav, > > > > I apologize for the late update. After a while I finally find > chance > to try this test. I could not reproduce this problem on my side. > > > > Here is my environment: > > - MPICH version: version 3.3a2 (using the same configure options > as > shown in your config.log) > > - mpi4py: version 2.0.0 > > - platform: a Fedora25 VM and a Centos7 VM > > > > If you are still facing this problem, please try the following > steps > to narrow down: > > 1. Try remove the computation in child.py > > 2. Try MPICH test suite under <your MPICH build > directory>/test/mpi/spawn/ > > make > > make testing V=1 > > > > Regards, > > Min > > > > On 8/13/17 11:07 AM, Stanislav Simko > wrote: > > > > > > > Hi Min, > > attached are configure and make logs for the build that I did > > for myself on our local cluster with Intel compilers. I.e., > > build is definitely not guaranteed to be perfect/optimal. > > But > > please keep in mind that I get the same behaviour with > > standard > > MPICH package in Fedora distribution - I think that > > configuration options should be available online, maybe > > prepared > > by someone from MPICH? > > > > > > > > Also, I tested with simple c++ hello world like program and > > it's the same. > > > > > > > > Thank you. > > best, > > stanislav. > > > > > > > > > > > > > > On Sun, 2017-08-13 at 19:58 +0100, Min Si wrote: > > > > > Hi Stanislav, > > > > > > > > > > > > This seems interesting. Could you please also attach the > > > MPICH > > > config.log ? You can find under the directory where you > > > build > > > MPICH. I will look into this problem then and keep you > > > updated. > > > > > > > > > > > > Min > > > > > > > > > > > > On 8/11/17 1:31 PM, Stanislav Simko > > > wrote: > > > > > > > > > > > > > Dear all, > > > > > > > > > > > > > > > > I'm just trying some very basic stuff with > > > > MPI_COMM_SPAWN > > > > in python (i.e. I use mpi4py package), but I see > > > > behaviour > > > > that I do not understand - the child process gets > > > > spawned, > > > > does its stuff and then "should" finish. I see > > > > though, that > > > > the process created for the child stays alive. I > > > > see this > > > > only with the MPICH, OPENMPI does what I would > > > > (naively) > > > > expect. In this way I can end up with N "ghost" > > > > process, > > > > after calling SPAWN N-times. My minimal working > > > > example is > > > > following: > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________ > > > > parent.py > > > > > > > > > > > > > > > > from __future__ import print_function > > > > from mpi4py import MPI > > > > comm = MPI.COMM_WORLD > > > > spawned = > > > > MPI.COMM_SELF.Spawn(sys.executable,args=['child.py' > > > > ],maxprocs=1) > > > > print("parent process is waiting for child") > > > > spawned.Barrier() > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________ > > > > child.py > > > > > > > > > > > > > > > > from __future__ import print_function > > > > from mpi4py import MPI > > > > parent = MPI.Comm.Get_parent() > > > > # just do some stupid stuff that takes a bit of time > > > > for i in range(5000000): > > > > a = i*i+1-(i*10) + math.sin(math.pow(i,i%8)) > > > > parent.Barrier() > > > > > > > > > > > > > > > > ______________________________________________ > > > > > > > > > > > > > > > > I run with e.g.: > > > > mpirun -n 1 python mpi.py > > > > > > > > > > > > > > > > Do I miss something with SPAWN method? > > > > (I tested on two independent systems, our local > > > > cluster > > > > with mpich v3.0.4, and my laptop - fedora 26, mpich > > > > v3.2.8 > > > > from repositories) > > > > > > > > > > > > > > > > thank you very much for suggestions. > > > > > > > > > > > > > > > > Regards, > > > > stanislav. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > discuss mailing list discuss(a)mpich.org > > > > To manage subscription options or unsubscribe: > > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI_Rsend advantage
by Dorier, Matthieu 06 Sep '17

06 Sep '17

Hi, Out of curiosity, what gain is there to using MPI_Rsend instead of MPI_Send when I know that the receiver has posted a receive? Do I avoid a rendez-vous protocol? Do I avoid a buffer copy somewhere? Thanks, Matthieu _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpi_comm_spawn - process not destroyed
by Min Si 05 Sep '17

05 Sep '17

Hi Stanislav, I apologize for the late update. After a while I finally find chance to try this test. I could not reproduce this problem on my side. Here is my environment: - MPICH version: version 3.3a2 (using the same configure options as shown in your config.log) - mpi4py: version 2.0.0 - platform: a Fedora25 VM and a Centos7 VM If you are still facing this problem, please try the following steps to narrow down: 1. Try remove the computation in child.py 2. Try MPICH test suite under <your MPICH build directory>/test/mpi/spawn/ make make testing V=1 Regards, Min On 8/13/17 11:07 AM, Stanislav Simko wrote: > Hi Min, > attached are configure and make logs for the build that I did for > myself on our local cluster with Intel compilers. I.e., build is > definitely not guaranteed to be perfect/optimal. But please keep in > mind that I get the same behaviour with standard MPICH package in > Fedora distribution - I think that configuration options should be > available online, maybe prepared by someone from MPICH? > > Also, I tested with simple c++ hello world like program and it's the same. > > Thank you. > best, > stanislav. > > > On Sun, 2017-08-13 at 19:58 +0100, Min Si wrote: >> Hi Stanislav, >> >> This seems interesting. Could you please also attach the MPICH >> config.log ? You can find under the directory where you build MPICH. >> I will look into this problem then and keep you updated. >> >> Min >> >> On 8/11/17 1:31 PM, Stanislav Simko wrote: >>> Dear all, >>> >>> I'm just trying some very basic stuff with MPI_COMM_SPAWN in python >>> (i.e. I use mpi4py package), but I see behaviour that I do not >>> understand - the child process gets spawned, does its stuff and then >>> "should" finish. I see though, that the process created for the >>> child stays alive. I see this only with the MPICH, OPENMPI does what >>> I would (naively) expect. In this way I can end up with N "ghost" >>> process, after calling SPAWN N-times. My minimal working example is >>> following: >>> >>> >>> ______________________________________________ >>> parent.py >>> >>> from __future__ import print_function >>> from mpi4py import MPI >>> comm = MPI.COMM_WORLD >>> spawned = >>> MPI.COMM_SELF.Spawn(sys.executable,args=['child.py'],maxprocs=1) >>> print("parent process is waiting for child") >>> spawned.Barrier() >>> >>> >>> ______________________________________________ >>> child.py >>> >>> from __future__ import print_function >>> from mpi4py import MPI >>> parent = MPI.Comm.Get_parent() >>> # just do some stupid stuff that takes a bit of time >>> for i in range(5000000): >>> a = i*i+1-(i*10) + math.sin(math.pow(i,i%8)) >>> parent.Barrier() >>> >>> ______________________________________________ >>> >>> I run with e.g.: >>> mpirun -n 1 python mpi.py >>> >>> Do I miss something with SPAWN method? >>> (I tested on two independent systems, our local cluster with mpich >>> v3.0.4, and my laptop - fedora 26, mpich v3.2.8 from repositories) >>> >>> thank you very much for suggestions. >>> >>> Regards, >>> stanislav. >>> >>> >>> _______________________________________________ >>> discuss mailing listdiscuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0