January 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] halo update performance comparison
by Kokron, Daniel S. (GSFC-606.2)[Computer Sciences Corporation] 13 Jan '14

13 Jan '14

II should be able to package up my test case. Please let me know if I can help Daniel Kokron NASA Ames (ARC-TN) SciCon group 301-286-3959 ________________________________________ From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Rajeev Thakur [thakur(a)mcs.anl.gov] Sent: Monday, January 13, 2014 4:45 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] halo update performance comparison > Is there room for improvement? Yes, we are looking into it. Rajeev On Dec 13, 2013, at 2:31 PM, "Kokron, Daniel S. (GSFC-610.1)[Computer Sciences Corporation]" <daniel.s.kokron(a)nasa.gov> wrote: > All, > > I have been interested in MPI datatypes since reading about them and playing with the benchmark found at http://unixer.de/research/datatypes/ddtbench/ > > I am working with a weather model that spends considerable time doing halo (aka ghost cell) updates. The existing code uses irecv/isend. Since most of the time spent doing halo updates is actually spent packing and unpacking the MPI buffers, I was hoping the use of datatypes would improve performance. Unfortunately, the datatype version is actually quite a bit slower than the standard non-blocking pt2pt code. I found this to be that case for both mpich-3.1rc1 and a recent version of a vendor MPI implementation (not MPICH based). I have not run this comparison using OpenMPI. > > Is data type performance an active area of development? Is there room for improvement? > > Daniel Kokron > NASA Ames (ARC-TN) > SciCon group > 301-286-3959 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] halo update performance comparison
by Rajeev Thakur 13 Jan '14

13 Jan '14

> Is there room for improvement? Yes, we are looking into it. Rajeev On Dec 13, 2013, at 2:31 PM, "Kokron, Daniel S. (GSFC-610.1)[Computer Sciences Corporation]" <daniel.s.kokron(a)nasa.gov> wrote: > All, > > I have been interested in MPI datatypes since reading about them and playing with the benchmark found at http://unixer.de/research/datatypes/ddtbench/ > > I am working with a weather model that spends considerable time doing halo (aka ghost cell) updates. The existing code uses irecv/isend. Since most of the time spent doing halo updates is actually spent packing and unpacking the MPI buffers, I was hoping the use of datatypes would improve performance. Unfortunately, the datatype version is actually quite a bit slower than the standard non-blocking pt2pt code. I found this to be that case for both mpich-3.1rc1 and a recent version of a vendor MPI implementation (not MPICH based). I have not run this comparison using OpenMPI. > > Is data type performance an active area of development? Is there room for improvement? > > Daniel Kokron > NASA Ames (ARC-TN) > SciCon group > 301-286-3959 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1rc2 build error on Darwin.
by Jeff Hammond 13 Jan '14

13 Jan '14

I always build MPICH from git by hand. MacPorts broke my toolchain such that I could not compile MPICH or MPI-free Fortran applications. I believe GCC and/or binutils stuff was improperly told to generate/work with 32bit objects. Replacing MacPorts with Homebrew solved my problems. I suppose I could have reinstalled MacPorts (since it worked fine for me for over a year before self-destructing) but an expert advised me to switch. Jeff Sent from my iPhone > On Jan 13, 2014, at 12:39 AM, "Eric A. Borisch" <eborisch(a)gmail.com> wrote: > > He's building with icc/ifort, so it's not being done via MacPorts. I don't think he even needs MP or Homebrew installed to build MPICH, just Intel's tools (and Xcode). > > Building with 'make V=1' (or configuring with --disable-silent-rules) will provide more insight into what is going on. If you do have MP or Homebrew installed, you can rule out most interaction by removing their locations (/opt/local/bin, or /usr/local/bin, respectively) from PATH. > > Jeff, I'm sorry to year you were unhappy with the (MP) tools. FWIW, it's been reworked to have a standardized MPICH version (with GCC's Fortan and the system's default C compiler) that is installed for use by any (macports-built) package that needs it. > > Thanks, > - Eric > >> On Sunday, January 12, 2014, Jeff Hammond wrote: >> Might be the MacPorts garbage that bit me a while back too. If GCC fails, I suggest wiping MacPorts and installing Homebrew and the associated toolchain. >> >> Jeff >> >> Sent from my iPhone >> >> > On Jan 12, 2014, at 7:58 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> > >> > It sounds like a 32-bit, 64-bit incompatibility between some installed compilers/linker. Are you able to build with gcc? >> > >> > >> >> On Jan 10, 2014, at 3:09 PM, "Salmon, Rene" <salmr0(a)bp.com> wrote: >> >> >> >> >> >> Hi, >> >> >> >> Just trying to build MPICH 3.1rc2 on a Mac. The build seems to work but fails while trying to compile the examples. This is using the intel compilers. >> >> >> >> ifort,icc version 14.0.0.074 >> >> >> >> Here is the config line. >> >> ./configure CC=icc CXX=icpc F77=ifort FC=ifort --enable-fc >> >> >> >> >> >> error message. >> >> >> >> make[4]: Nothing to be done for `all'. >> >> make[4]: Nothing to be done for `all-am'. >> >> Making all in . >> >> Making all in . >> >> Making all in examples >> >> CCLD cpi >> >> Undefined symbols for architecture x86_64: >> >> "_MPI_Bcast", referenced from: >> >> _main in cpi.o >> >> "_MPI_Comm_rank", referenced from: >> >> _main in cpi.o >> >> "_MPI_Comm_size", referenced from: >> >> _main in cpi.o >> >> "_MPI_Finalize", referenced from: >> >> _main in cpi.o >> >> "_MPI_Get_processor_name", referenced from: >> >> _main in cpi.o >> >> "_MPI_Init", referenced from: >> >> _main in cpi.o >> >> "_MPI_Reduce", referenced from: >> >> _main in cpi.o >> >> "_MPI_Wtime", referenced from: >> >> _main in cpi.o >> >> ld: symbol(s) not found for architecture x86_64 >> >> make[2]: *** [cpi] Error 1 >> >> make[1]: *** [all-recursive] Error 1 >> >> make: *** [all] Error 2 >> >> >> >> Thank you. >> >> >> >> Rene >> >> _______________________________________________ >> >> discuss mailing list discuss(a)mpich.org >> >> To manage subscription options or unsubscribe: >> >> https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI fail on 20 processes, but not on 10.
by Anatoly G 13 Jan '14

13 Jan '14

Thank you very much. I tried to clean functions which use MPI: 1) mainInit 2) Rcv_WaitAny 3) SndSyncSlave I added input data verification - which shows that messages are come as expected. In current configuration I still see that network measurements become low and ends with fail. But if I comment out two lines (74,75): MPI_Recv(RcvBufs[slaveIdx], BUF_SZ, MPI::CHAR, slaveRank, TAG1, MPI_COMM_WORLD, &status); validateInput(fpLog, RcvBufs, SlavesRcvIters, slaveIdx, myStatistics, &SlavesFinished); in Rcv_WaitAny function (validate input needs for verification only no MPI inside), I see stable & full network rate, no degradation, no failure. I suppose that using MPI_Recv together with MPI_Irecv & MPI_Waitany causes this degradation (unexpected messages) and failure. If I set instead of these two lines "sleep for 0.5 second", I see low but stable network rate, no failure. (Senders are flow controlled by master). Execution command: mpiexec.hydra -genvall -f MpiConfigMachines1.txt -launcher=rsh -n 10 mpi_rcv_any_multithread 100000 1000000 out Is my scenario legal? If it is legal, why I see degradation? Attached out_r0.log for master and out_r2.log for slave. I got error in shell: Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 640: pkt->type >= 0 && pkt->type < MPIDI_NEM_PKT_END internal ABORT - process 0 Please help. Regards, Anatoly. On Mon, Jan 6, 2014 at 7:23 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > There should be automatic flow-control within the network layer to deal > with this, unless the receives are not posted causing unexpected messages. > Looking through your code, that seems to be at least one of the problems. > You are posting one Irecv for each slave, and waiting for all of them to > finish, while the slaves are sending out many messages. So you can get > many messages from one slave before getting the first message from all the > slaves. This creates unexpected messages. > > Btw, your program can be significantly simplified to a bare-bones 100-line > source code, which can make debugging easier. I’d recommend doing that. > > — Pavan > > On Jan 6, 2014, at 11:14 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > > > Are you blasting the server (master) with messages from N clients > > (slaves)? At some point, that will overwhelm the communication > > buffers and fail. Can you turn off eager using the documented > > environment variable? Rendezvous-only should be much slower but not > > fail. Then you can eliminate the pathological usage in your > > application. > > > > Jeff > > > > On Sun, Jan 5, 2014 at 10:38 PM, Anatoly G <anatolyrishon(a)gmail.com> > wrote: > >> Hi. > >> I have created an application. This application fails on MPI error. > >> Assertion failed in file > src/mpid/ch3/channels/nemesis/src/ch3_progress.c at > >> line 640: pkt->type >= 0 && pkt->type < MPIDI_NEM_PKT_END > >> internal ABORT - process 0 > >> > >> Scenario: > >> Master receives messages from slaves. > >> Each slave sends data using MPI_Send. > >> Master receives using MPI_Irecv and MPI_Recv. > >> > >> There are another errors in out*.log files. > >> Application doesn't fail with 10 processes, but fails with 20. > >> > >> execute command: > >> mpiexec.hydra -genvall -f MpiConfigMachines1.txt -launcher=rsh -n 20 > >> > /home/anatol-g/Grape/release_under_constr_MPI_tests_quantum/bin/linux64/rhe6/g++4.4.6/debug/mpi_rcv_any_multithread > >> 100000 1000000 -1 -1 1 out > >> > >> Please help, > >> > >> Regards, > >> Anatoly. > >> > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > > > -- > > Jeff Hammond > > jeff.science(a)gmail.com > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

MPICH 3.1rc2 build error on Darwin.
by Eric A. Borisch 13 Jan '14

13 Jan '14

He's building with icc/ifort, so it's not being done via MacPorts. I don't think he even needs MP or Homebrew installed to build MPICH, just Intel's tools (and Xcode). Building with 'make V=1' (or configuring with --disable-silent-rules) will provide more insight into what is going on. If you do have MP or Homebrew installed, you can rule out most interaction by removing their locations (/opt/local/bin, or /usr/local/bin, respectively) from PATH. Jeff, I'm sorry to year you were unhappy with the (MP) tools. FWIW, it's been reworked to have a standardized MPICH version (with GCC's Fortan and the system's default C compiler) that is installed for use by any (macports-built) package that needs it. Thanks, - Eric On Sunday, January 12, 2014, Jeff Hammond wrote: > Might be the MacPorts garbage that bit me a while back too. If GCC fails, > I suggest wiping MacPorts and installing Homebrew and the associated > toolchain. > > Jeff > > Sent from my iPhone > > > On Jan 12, 2014, at 7:58 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > > > > It sounds like a 32-bit, 64-bit incompatibility between some installed > compilers/linker. Are you able to build with gcc? > > > > > >> On Jan 10, 2014, at 3:09 PM, "Salmon, Rene" <salmr0(a)bp.com> wrote: > >> > >> > >> Hi, > >> > >> Just trying to build MPICH 3.1rc2 on a Mac. The build seems to work but > fails while trying to compile the examples. This is using the intel > compilers. > >> > >> ifort,icc version 14.0.0.074 > >> > >> Here is the config line. > >> ./configure CC=icc CXX=icpc F77=ifort FC=ifort --enable-fc > >> > >> > >> error message. > >> > >> make[4]: Nothing to be done for `all'. > >> make[4]: Nothing to be done for `all-am'. > >> Making all in . > >> Making all in . > >> Making all in examples > >> CCLD cpi > >> Undefined symbols for architecture x86_64: > >> "_MPI_Bcast", referenced from: > >> _main in cpi.o > >> "_MPI_Comm_rank", referenced from: > >> _main in cpi.o > >> "_MPI_Comm_size", referenced from: > >> _main in cpi.o > >> "_MPI_Finalize", referenced from: > >> _main in cpi.o > >> "_MPI_Get_processor_name", referenced from: > >> _main in cpi.o > >> "_MPI_Init", referenced from: > >> _main in cpi.o > >> "_MPI_Reduce", referenced from: > >> _main in cpi.o > >> "_MPI_Wtime", referenced from: > >> _main in cpi.o > >> ld: symbol(s) not found for architecture x86_64 > >> make[2]: *** [cpi] Error 1 > >> make[1]: *** [all-recursive] Error 1 > >> make: *** [all] Error 2 > >> > >> Thank you. > >> > >> Rene > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH 3.1rc2 build error on Darwin.
by Jeff Hammond 12 Jan '14

12 Jan '14

Might be the MacPorts garbage that bit me a while back too. If GCC fails, I suggest wiping MacPorts and installing Homebrew and the associated toolchain. Jeff Sent from my iPhone > On Jan 12, 2014, at 7:58 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > > It sounds like a 32-bit, 64-bit incompatibility between some installed compilers/linker. Are you able to build with gcc? > > >> On Jan 10, 2014, at 3:09 PM, "Salmon, Rene" <salmr0(a)bp.com> wrote: >> >> >> Hi, >> >> Just trying to build MPICH 3.1rc2 on a Mac. The build seems to work but fails while trying to compile the examples. This is using the intel compilers. >> >> ifort,icc version 14.0.0.074 >> >> Here is the config line. >> ./configure CC=icc CXX=icpc F77=ifort FC=ifort --enable-fc >> >> >> error message. >> >> make[4]: Nothing to be done for `all'. >> make[4]: Nothing to be done for `all-am'. >> Making all in . >> Making all in . >> Making all in examples >> CCLD cpi >> Undefined symbols for architecture x86_64: >> "_MPI_Bcast", referenced from: >> _main in cpi.o >> "_MPI_Comm_rank", referenced from: >> _main in cpi.o >> "_MPI_Comm_size", referenced from: >> _main in cpi.o >> "_MPI_Finalize", referenced from: >> _main in cpi.o >> "_MPI_Get_processor_name", referenced from: >> _main in cpi.o >> "_MPI_Init", referenced from: >> _main in cpi.o >> "_MPI_Reduce", referenced from: >> _main in cpi.o >> "_MPI_Wtime", referenced from: >> _main in cpi.o >> ld: symbol(s) not found for architecture x86_64 >> make[2]: *** [cpi] Error 1 >> make[1]: *** [all-recursive] Error 1 >> make: *** [all] Error 2 >> >> Thank you. >> >> Rene >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1rc2 build error on Darwin.
by Rajeev Thakur 12 Jan '14

12 Jan '14

It sounds like a 32-bit, 64-bit incompatibility between some installed compilers/linker. Are you able to build with gcc? On Jan 10, 2014, at 3:09 PM, "Salmon, Rene" <salmr0(a)bp.com> wrote: > > Hi, > > Just trying to build MPICH 3.1rc2 on a Mac. The build seems to work but fails while trying to compile the examples. This is using the intel compilers. > > ifort,icc version 14.0.0.074 > > Here is the config line. > ./configure CC=icc CXX=icpc F77=ifort FC=ifort --enable-fc > > > error message. > > make[4]: Nothing to be done for `all'. > make[4]: Nothing to be done for `all-am'. > Making all in . > Making all in . > Making all in examples > CCLD cpi > Undefined symbols for architecture x86_64: > "_MPI_Bcast", referenced from: > _main in cpi.o > "_MPI_Comm_rank", referenced from: > _main in cpi.o > "_MPI_Comm_size", referenced from: > _main in cpi.o > "_MPI_Finalize", referenced from: > _main in cpi.o > "_MPI_Get_processor_name", referenced from: > _main in cpi.o > "_MPI_Init", referenced from: > _main in cpi.o > "_MPI_Reduce", referenced from: > _main in cpi.o > "_MPI_Wtime", referenced from: > _main in cpi.o > ld: symbol(s) not found for architecture x86_64 > make[2]: *** [cpi] Error 1 > make[1]: *** [all-recursive] Error 1 > make: *** [all] Error 2 > > Thank you. > > Rene > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Windows Server 2012 Essentials
by "Antonio J. Peña" 11 Jan '14

11 Jan '14

Hi Bill, Please refer to our downloads section: http://www.mpich.org/downloads/ There is a link to Microsoft MPI, an MPICH derivative. Antonio On 01/10/2014 09:50 PM, mcdougal wrote: > Hello, > Is there a version of this package that will work on Win Server 2012 > (which by default has .net4.0 installed)? > Thanks, Bill > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peï¿½a Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya

1 0

Windows Server 2012 Essentials
by mcdougal 10 Jan '14

10 Jan '14

Hello, Is there a version of this package that will work on Win Server 2012 (which by default has .net4.0 installed)? Thanks, Bill

1 0

MPICH 3.1rc2 build error on Darwin.
by Salmon, Rene 10 Jan '14

10 Jan '14

Hi, Just trying to build MPICH 3.1rc2 on a Mac. The build seems to work but fails while trying to compile the examples. This is using the intel compilers. ifort,icc version 14.0.0.074 Here is the config line. ./configure CC=icc CXX=icpc F77=ifort FC=ifort --enable-fc error message. make[4]: Nothing to be done for `all'. make[4]: Nothing to be done for `all-am'. Making all in . Making all in . Making all in examples CCLD cpi Undefined symbols for architecture x86_64: "_MPI_Bcast", referenced from: _main in cpi.o "_MPI_Comm_rank", referenced from: _main in cpi.o "_MPI_Comm_size", referenced from: _main in cpi.o "_MPI_Finalize", referenced from: _main in cpi.o "_MPI_Get_processor_name", referenced from: _main in cpi.o "_MPI_Init", referenced from: _main in cpi.o "_MPI_Reduce", referenced from: _main in cpi.o "_MPI_Wtime", referenced from: _main in cpi.o ld: symbol(s) not found for architecture x86_64 make[2]: *** [cpi] Error 1 make[1]: *** [all-recursive] Error 1 make: *** [all] Error 2 Thank you. Rene

1 0