November 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Predefined datatype implementation
by Jed Brown 20 Nov '13

20 Nov '13

Jeff Hammond <jeff.science(a)gmail.com> writes: >>> FWIW, even if MPI_VERSION and MPI_SUBVERSION are set correctly, >>> there’s no guarantee that these types are available. Types are only >>> defined based on what compiler you use. For example, if you use a C89 >>> only compiler, these types will not be available. > > Wouldn't the C89-compiled MPICH need to support these datatypes to > provide interoperability with C99 code in exactly the same manner as > it provides Fortran and C++ types for interoperability with code > written in those languages? It is ambiguous in what context the word > "use" is used here so maybe I am misinterpreting your statement. In that case, you should configure MPICH with C99 support (or at least do not use -std=c89). > Filesystem performance is a red herring. Don't compile on the > gold-plated filesystem. Use /tmp or /scratch, as suggested by > https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq. Every > Linux login node has /tmp. 1. System headers and libraries are still often on a remote filesystem. The workflow is a lot more complicated if you mirror that locally. 2. /tmp isn't shared between login nodes, let alone compute nodes, so you have to copy back to global storage. With multiple login nodes, when you background/nohup a compilation task and log out or are disconnected, you have to remember the login node number to get back to the result. The cognitive load of doing all compilation on /tmp, especially with multiple packages, is not trivial. 3. Try teaching this to a novice. The fact is that if HPC is going to grow its user base, it has to be easier to use. If you want to take the elitist stance that expensive machines should only be used by experts, stop giving INCITE awards to people that don't know how to use compilers, debuggers, or the shell. Until then, hiccups along the way land in our lap, and a complicated workflow only serves to intimidate and to create more opportunity for human error, which again comes back in the form of support mail.

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Rajeev Thakur 20 Nov '13

20 Nov '13

You have to build MPICH with the compilers you plan to use it with. Rajeev On Nov 19, 2013, at 4:29 PM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >>> FWIW, even if MPI_VERSION and MPI_SUBVERSION are set correctly, >>> there’s no guarantee that these types are available. Types are only >>> defined based on what compiler you use. For example, if you use a C89 >>> only compiler, these types will not be available. > > Wouldn't the C89-compiled MPICH need to support these datatypes to > provide interoperability with C99 code in exactly the same manner as > it provides Fortran and C++ types for interoperability with code > written in those languages? It is ambiguous in what context the word > "use" is used here so maybe I am misinterpreting your statement. > >> Though with MPICH, that is decided at configure time. If you later add >> -std=c89, the header does not avoid defining the types. Note that some >> applications need to build different modules with different language >> standards. >> >>> configure is the only reliable check for these. >> >> That's what we have always done, but it ends up being quite a lot of >> configure tests and some people complain because configure takes 100x >> longer on a multi-million dollar file system than it takes on a laptop, >> so I was double-checking that they were all necessary. > > Filesystem performance is a red herring. Don't compile on the > gold-plated filesystem. Use /tmp or /scratch, as suggested by > https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq. Every > Linux login node has /tmp. > > Jeff > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Jeff Hammond 19 Nov '13

19 Nov '13

>> FWIW, even if MPI_VERSION and MPI_SUBVERSION are set correctly, >> there’s no guarantee that these types are available. Types are only >> defined based on what compiler you use. For example, if you use a C89 >> only compiler, these types will not be available. Wouldn't the C89-compiled MPICH need to support these datatypes to provide interoperability with C99 code in exactly the same manner as it provides Fortran and C++ types for interoperability with code written in those languages? It is ambiguous in what context the word "use" is used here so maybe I am misinterpreting your statement. > Though with MPICH, that is decided at configure time. If you later add > -std=c89, the header does not avoid defining the types. Note that some > applications need to build different modules with different language > standards. > >> configure is the only reliable check for these. > > That's what we have always done, but it ends up being quite a lot of > configure tests and some people complain because configure takes 100x > longer on a multi-million dollar file system than it takes on a laptop, > so I was double-checking that they were all necessary. Filesystem performance is a red herring. Don't compile on the gold-plated filesystem. Use /tmp or /scratch, as suggested by https://www.alcf.anl.gov/user-guides/compiling-and-linking-faq. Every Linux login node has /tmp. Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Jed Brown 19 Nov '13

19 Nov '13

Pavan Balaji <balaji(a)mcs.anl.gov> writes: > FWIW, even if MPI_VERSION and MPI_SUBVERSION are set correctly, > there’s no guarantee that these types are available. Types are only > defined based on what compiler you use. For example, if you use a C89 > only compiler, these types will not be available. Though with MPICH, that is decided at configure time. If you later add -std=c89, the header does not avoid defining the types. Note that some applications need to build different modules with different language standards. > configure is the only reliable check for these. That's what we have always done, but it ends up being quite a lot of configure tests and some people complain because configure takes 100x longer on a multi-million dollar file system than it takes on a laptop, so I was double-checking that they were all necessary.

1 0

MPICH 3.1rc2 packages for Ubuntu, Debian and Slackware (x86 and amd64)
by Marcelo Soares Souza 19 Nov '13

19 Nov '13

mpich-3.1rc2 packages * Ubuntu 13.10 (Saucy Salamander) - amd64 http://www.cebacad.net/files/mpich/ubuntu/mpich-3.1rc2/mpich_3.1rc2-1ubuntu… - x86 http://www.cebacad.net/files/mpich/ubuntu/mpich-3.1rc2/mpich_3.1rc2-1ubuntu… * Debian 7.2.0 (Wheezy) - amd64 http://www.cebacad.net/files/mpich/debian/mpich-3.1rc2/mpich_3.1rc2-1debian… - x86 http://www.cebacad.net/files/mpich/debian/mpich-3.1rc2/mpich_3.1rc2-1debian… * Slackware 14.1 - amd64 http://www.cebacad.net/files/mpich/slackware/mpich-3.1rc2/mpich-3.1rc2-x86_… - x86 http://www.cebacad.net/files/mpich/slackware/mpich-3.1rc2/mpich-3.1rc2-i486… Enjoy! ;) -- Marcelo Soares Souza https://marcelo.juntadados.org Diaspora* https://diaspora.juntadados.org/u/marcelo Friendica http://friendica.ambientedigital.org/profile/marcelo

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Pavan Balaji 19 Nov '13

19 Nov '13

FWIW, even if MPI_VERSION and MPI_SUBVERSION are set correctly, there’s no guarantee that these types are available. Types are only defined based on what compiler you use. For example, if you use a C89 only compiler, these types will not be available. configure is the only reliable check for these. — Pavan On Nov 19, 2013, at 10:39 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: >> Since some vendors have been less than precise about setting MPI_VERSION >> and MPI_SUBVERSION, configure tests are the only reliable way to >> determine availability of such types. > > Correct. It seems you answered your own questions :-) > > Jeff > > > -- > Jeff Hammond > jeff.science(a)gmail.com > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Predefined datatype implementation
by Jeff Hammond 19 Nov '13

19 Nov '13

> Since some vendors have been less than precise about setting MPI_VERSION > and MPI_SUBVERSION, configure tests are the only reliable way to > determine availability of such types. Correct. It seems you answered your own questions :-) Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Predefined datatype implementation
by Jed Brown 19 Nov '13

19 Nov '13

My reading of section 2.5.4 (Named Constants) is that predefined MPI datatypes need not be implemented as compile-time constants/macros, so one cannot do #if defined(MPI_INT64_T) or use in a switch/case statement. Perhaps this makes sense if the Forum some day decides to standardize the ABI, yet all implementations I know of use macros for these types. Is there some other language in the standard that effectively requires them to be macros already? Since some vendors have been less than precise about setting MPI_VERSION and MPI_SUBVERSION, configure tests are the only reliable way to determine availability of such types.

1 0

Re: [mpich-discuss] double free error after MPI run finished
by Antonio J. Peña 19 Nov '13

19 Nov '13

Thank you very much for reporting. We're glad you have confirmed the fix. Best, Antonio On Tuesday, November 19, 2013 09:36:37 AM John Bray wrote: > I can confirm that 3.1rc2 fixes my problem > > Thanks for your help. > > John > > On 9 November 2013 08:19, John Bray <jbray(a)allinea.com> wrote: > > Happy to validate any fix when its ready, not only for this test case, > > but with our full DDT test suite. > > > > John > > > > On 8 November 2013 19:44, Antonio J. Peña <apenya(a)mcs.anl.gov> wrote: > >> Hi John, > >> > >> You seem to describe a known bug: > >> > >> https://trac.mpich.org/projects/mpich/ticket/1932 > >> > >> We've addressed it and the fix is waiting for review before going to the > >> master branch. It would be nice if you could validate it once it's in > >> master. May I contact you then to see if that addresses your problem? > >> > >> Antonio > >> > >> On Friday, November 08, 2013 05:43:29 PM John Bray wrote: > >>> I've got some sample code that I'm using to test our debugger DDT. It > >>> works on openmpi and intelmpi, but with all the MPICH series from > >>> MPICH2 onwards to 3.1rc1, I get an error after MPI_FINALIZE. Our > >>> memory debugging library claims its at > >>> > >>> SendqFreePool (dbginit.c:386) trying to free previously freed pointer > >>> > >>> with a stacktrrace > >>> auditmpi > >>> > >>> pmpi_finalize__ > >>> > >>> PMPI__finalize > >>> > >>> MPIR_Call_finalize_callbacks > >>> > >>> sendqFreePool (dbginit.c) > >>> > >>> Run from a raw mpirun command outside DDT > >>> > >>> *** glibc detected *** ./auditmpibad_f_debug.exe: double free or > >>> corruption (fasttop): 0x1032a5c8 *** > >>> *** glibc detected *** ./auditmpibad_f_debug.exe: double free or > >>> corruption (fasttop): 0x104da5c8 *** > >>> ======= Backtrace: ========= > >>> ======= Backtrace: ========= > >>> /lib/libc.so.6(+0xfe750e4)[0xf78e50e4] > >>> /home/jbray/prog/mpich/mpich-3.1rc1/rhel-6-ppc64_ibm/install/lib/libmpic > >>> h.so .11(+0x1a8b1c)[0xfec8b1c] /lib/libc.so.6(+0xfe750e4)[0xf76c50e4] > >>> /home/jbray/prog/mpich/mpich-3.1rc1/rhel-6-ppc64_ibm/install/lib/libmpic > >>> h.so .11(+0x1a8b1c)[0xfec8b1c] ... and so on > >>> > >>> I've configured with > >>> > >>> ./configure --prefix=$PWD/install --enable-shared --enable-fast=all > >>> --enable-debuginfo > >>> > >>> and compile with > >>> > >>> mpif90 -o auditmpibad_f_debug.exe auditmpibad.F90 -O0 -g > >>> > >>> Its works with the ALLTOALLV only called once. > >>> > >>> Is this a bug or a misunderstanding on my part. The code is > >>> > >>> program auditmpi > >>> > >>> use mpi, only : & > >>> > >>> MPI_COMM_WORLD, & > >>> MPI_REAL, & > >>> MPI_STATUS_SIZE, & > >>> MPI_BSEND_OVERHEAD, & > >>> MPI_SUM, & > >>> MPI_UNDEFINED > >>> > >>> implicit none > >>> > >>> integer, parameter :: repeats=2 > >>> > >>> integer :: rank, nproc,ierr,rpt > >>> real :: input(100000),output(100000) > >>> integer :: status (MPI_STATUS_SIZE) > >>> integer :: statuses (MPI_STATUS_SIZE,repeats) > >>> integer :: request > >>> logical :: odd > >>> integer, allocatable :: recvcounts(:),sendcounts(:),displs(:) > >>> > >>> call MPI_INIT(ierr) > >>> > >>> call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr) > >>> call MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr) > >>> > >>> if (rank == 0) print *,"auditmpi start" > >>> > >>> if (mod(rank,2) == 0) then > >>> > >>> odd=.false. > >>> > >>> else > >>> > >>> odd=.true. > >>> > >>> end if > >>> > >>> allocate (recvcounts(nproc)) > >>> recvcounts(:) = 100000/nproc > >>> > >>> allocate (sendcounts(nproc)) > >>> sendcounts(:) = 100000/nproc > >>> > >>> allocate (displs(nproc)) > >>> displs(:)=0 ! don't care if we overlap > >>> > >>> if ( .NOT. odd) then > >>> > >>> call MPI_IRECV(output,size(output),MPI_REAL,rank+1,1,MPI_COMM_WORLD, > >>> > >>> request, ierr) > >>> > >>> end if > >>> call MPI_BARRIER(MPI_COMM_WORLD,ierr) > >>> if (odd) then > >>> > >>> call > >>> > >>> MPI_IRSEND(input,size(input),MPI_REAL,rank-1,1,MPI_COMM_WORLD,request,ie > >>> rr) > >>> end if > >>> > >>> call MPI_WAIT(request,status,ierr) > >>> > >>> do rpt=1,2 > >>> > >>> call > >>> > >>> MPI_ALLTOALLV(input,sendcounts,displs,MPI_REAL,output,recvcounts,displs, > >>> MPI > >>> _REAL,MPI_COMM_WORLD,ierr) end do > >>> > >>> call MPI_BARRIER(MPI_COMM_WORLD,ierr) > >>> > >>> deallocate (recvcounts) > >>> deallocate (sendcounts) > >>> deallocate (displs) > >>> > >>> if (rank == 0) print *,"auditmpi finished" > >>> > >>> call MPI_FINALIZE(ierr) > >>> > >>> end program auditmpi > >>> > >>> John > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] double free error after MPI run finished
by John Bray 19 Nov '13

19 Nov '13

I can confirm that 3.1rc2 fixes my problem Thanks for your help. John On 9 November 2013 08:19, John Bray <jbray(a)allinea.com> wrote: > Happy to validate any fix when its ready, not only for this test case, > but with our full DDT test suite. > > John > > On 8 November 2013 19:44, Antonio J. Peña <apenya(a)mcs.anl.gov> wrote: >> >> Hi John, >> >> You seem to describe a known bug: >> >> https://trac.mpich.org/projects/mpich/ticket/1932 >> >> We've addressed it and the fix is waiting for review before going to the master >> branch. It would be nice if you could validate it once it's in master. May I >> contact you then to see if that addresses your problem? >> >> Antonio >> >> >> On Friday, November 08, 2013 05:43:29 PM John Bray wrote: >>> I've got some sample code that I'm using to test our debugger DDT. It >>> works on openmpi and intelmpi, but with all the MPICH series from >>> MPICH2 onwards to 3.1rc1, I get an error after MPI_FINALIZE. Our >>> memory debugging library claims its at >>> >>> SendqFreePool (dbginit.c:386) trying to free previously freed pointer >>> >>> with a stacktrrace >>> auditmpi >>> pmpi_finalize__ >>> PMPI__finalize >>> MPIR_Call_finalize_callbacks >>> sendqFreePool (dbginit.c) >>> >>> Run from a raw mpirun command outside DDT >>> >>> *** glibc detected *** ./auditmpibad_f_debug.exe: double free or >>> corruption (fasttop): 0x1032a5c8 *** >>> *** glibc detected *** ./auditmpibad_f_debug.exe: double free or >>> corruption (fasttop): 0x104da5c8 *** >>> ======= Backtrace: ========= >>> ======= Backtrace: ========= >>> /lib/libc.so.6(+0xfe750e4)[0xf78e50e4] >>> /home/jbray/prog/mpich/mpich-3.1rc1/rhel-6-ppc64_ibm/install/lib/libmpich.so >>> .11(+0x1a8b1c)[0xfec8b1c] /lib/libc.so.6(+0xfe750e4)[0xf76c50e4] >>> /home/jbray/prog/mpich/mpich-3.1rc1/rhel-6-ppc64_ibm/install/lib/libmpich.so >>> .11(+0x1a8b1c)[0xfec8b1c] ... and so on >>> >>> I've configured with >>> >>> ./configure --prefix=$PWD/install --enable-shared --enable-fast=all >>> --enable-debuginfo >>> >>> and compile with >>> >>> mpif90 -o auditmpibad_f_debug.exe auditmpibad.F90 -O0 -g >>> >>> Its works with the ALLTOALLV only called once. >>> >>> Is this a bug or a misunderstanding on my part. The code is >>> >>> program auditmpi >>> >>> use mpi, only : & >>> MPI_COMM_WORLD, & >>> MPI_REAL, & >>> MPI_STATUS_SIZE, & >>> MPI_BSEND_OVERHEAD, & >>> MPI_SUM, & >>> MPI_UNDEFINED >>> >>> implicit none >>> >>> integer, parameter :: repeats=2 >>> >>> integer :: rank, nproc,ierr,rpt >>> real :: input(100000),output(100000) >>> integer :: status (MPI_STATUS_SIZE) >>> integer :: statuses (MPI_STATUS_SIZE,repeats) >>> integer :: request >>> logical :: odd >>> integer, allocatable :: recvcounts(:),sendcounts(:),displs(:) >>> >>> call MPI_INIT(ierr) >>> >>> call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr) >>> call MPI_COMM_SIZE(MPI_COMM_WORLD, nproc, ierr) >>> >>> if (rank == 0) print *,"auditmpi start" >>> >>> if (mod(rank,2) == 0) then >>> odd=.false. >>> else >>> odd=.true. >>> end if >>> >>> allocate (recvcounts(nproc)) >>> recvcounts(:) = 100000/nproc >>> >>> allocate (sendcounts(nproc)) >>> sendcounts(:) = 100000/nproc >>> >>> allocate (displs(nproc)) >>> displs(:)=0 ! don't care if we overlap >>> >>> if ( .NOT. odd) then >>> call MPI_IRECV(output,size(output),MPI_REAL,rank+1,1,MPI_COMM_WORLD, >>> request, ierr) >>> end if >>> call MPI_BARRIER(MPI_COMM_WORLD,ierr) >>> if (odd) then >>> call >>> MPI_IRSEND(input,size(input),MPI_REAL,rank-1,1,MPI_COMM_WORLD,request,ierr) >>> end if >>> call MPI_WAIT(request,status,ierr) >>> >>> do rpt=1,2 >>> call >>> MPI_ALLTOALLV(input,sendcounts,displs,MPI_REAL,output,recvcounts,displs,MPI >>> _REAL,MPI_COMM_WORLD,ierr) end do >>> >>> call MPI_BARRIER(MPI_COMM_WORLD,ierr) >>> >>> deallocate (recvcounts) >>> deallocate (sendcounts) >>> deallocate (displs) >>> >>> if (rank == 0) print *,"auditmpi finished" >>> >>> call MPI_FINALIZE(ierr) >>> >>> end program auditmpi >>> >>> John >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss

1 0