July 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fwd: [petsc-dev] MPICH from --download-mpich reports inconsistent allocs/frees with valgrind
by Raffenetti, Kenneth J. 25 Jul '19

25 Jul '19

Hi Patrick, Thanks for the report. I am able to reproduce similar behavior on my machine. I suspect our valgrind client request integration needs fixing. I have created an issue on github https://github.com/pmodels/mpich/issues/3945. Ken On 7/23/19 8:19 AM, Patrick Sanan via discuss wrote: > I'm trying to track down the cause of an inconsistency in valgrind's > heap allocation analysis. > > This arose when using the MPICH downloaded by PETSc. To reproduce, I run > the following and note that the number of allocations and frees are > different (yet all blocks are reported free). > > printf "#include<mpi.h>\nint main(int a,char > **b){MPI_Init(&a,&b);MPI_Finalize();}" > t.c && > $PETSC_DIR/$PETSC_ARCH/bin/mpicc t.c && valgrind ./a.out > > ==8578== Memcheck, a memory error detector > ==8578== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. > ==8578== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info > ==8578== Command: ./a.out > ==8578== > ==8578== > ==8578== HEAP SUMMARY: > ==8578== in use at exit: 0 bytes in 0 blocks > ==8578== total heap usage: *1,979 allocs, 1,974 frees*, 4,720,483 > bytes allocated > ==8578== > ==8578== All heap blocks were freed -- no leaks are possible > ==8578== > ==8578== For counts of detected and suppressed errors, rerun with: -v > ==8578== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) > > The above was generated on Ubuntu 18.04.2 LTS, using MPICH 3.3b1 as > configured in the attached config.log (which was done automatically for > me by PETSc master). > > ---------- Forwarded message --------- > Von: *Smith, Barry F.* <bsmith(a)mcs.anl.gov <mailto:[email protected]>> > Date: Mo., 22. Juli 2019 um 20:24 Uhr > Subject: Re: [petsc-dev] MPICH from --download-mpich reports > inconsistent allocs/frees with valgrind > To: petsc-dev <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> > Cc: Patrick Sanan <patrick.sanan(a)gmail.com > <mailto:[email protected]>>, Balay, Satish <balay(a)mcs.anl.gov > <mailto:[email protected]>> > > > > Bug report to MPICH. > > > On Jul 22, 2019, at 1:22 PM, Balay, Satish via petsc-dev > <petsc-dev(a)mcs.anl.gov <mailto:[email protected]>> wrote: > > > > Hm - I don't think we were monitoring the leaks via valgrind that > closely. > > > > Looking at my old mpich install - I don't see a problem - so likely > > its an issue with newer versions of mpich. > > > > Satish > > > > ------- > > balay@sb /home/balay/tmp > > $ mpichversion > > MPICH Version: 3.3 > > MPICH Release date: Wed Nov 21 11:32:40 CST 2018 > > MPICH Device: ch3:sock > > MPICH configure: --prefix=/home/balay/soft/mpich-3.3 > MAKE=/usr/bin/gmake --libdir=/home/balay/soft/mpich-3.3/lib CC=gcc > CFLAGS=-fPIC -g -O AR=/usr/bin/ar ARFLAGS=cr CXX=g++ CXXFLAGS=-g -O > -fPIC F77=gfortran FFLAGS=-fPIC -g -O FC=gfortran FCFLAGS=-fPIC -g -O > --enable-shared --with-device=ch3:sock --with-pm=hydra --enable-fast=no > --enable-error-messages=all --enable-g=meminit > > MPICH CC: gcc -fPIC -g -O -O0 > > MPICH CXX: g++ -g -O -fPIC -O0 > > MPICH F77: gfortran -fPIC -g -O -O0 > > MPICH FC: gfortran -fPIC -g -O -O0 > > MPICH Custom Information: > > balay@sb /home/balay/tmp > > $ printf "#include<mpi.h>\nint main(int > a,char**b){MPI_Init(&a,&b);MPI_Finalize();}" > t.c && mpicc t.c && > valgrind ./a.out > > ==9024== Memcheck, a memory error detector > > ==9024== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. > > ==9024== Using Valgrind-3.15.0 and LibVEX; rerun with -h for > copyright info > > ==9024== Command: ./a.out > > ==9024== > > ==9024== > > ==9024== HEAP SUMMARY: > > ==9024== in use at exit: 0 bytes in 0 blocks > > ==9024== total heap usage: 1,886 allocs, 1,886 frees, 4,884,751 > bytes allocated > > ==9024== > > ==9024== All heap blocks were freed -- no leaks are possible > > ==9024== > > ==9024== For lists of detected and suppressed errors, rerun with: -s > > ==9024== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) > > balay@sb /home/balay/tmp > > > > > > > > On Mon, 22 Jul 2019, Patrick Sanan via petsc-dev wrote: > > > >> It was pointed out to me that valgrind memcheck reports inconsistent > heap > >> usage information when running PETSc examples. All blocks are reported > >> freed, yet the number of allocations and frees are different. My > guess as > >> to what's going on is that this is an MPICH issue, as I can > reproduce the > >> behavior with a minimal MPI program. > >> > >>> From the PETSc perspective, is this a known issue? I'm wondering > if this > >> inconsistency was always there, whether it's worth looking into > more, etc. > >> > >> Here's a 1-liner to reproduce, using a PETSc master build with > >> --download-mpich (though note that this doesn't use anything from PETSc > >> except the MPICH it builds for you). > >> > >> printf "#include<mpi.h>\nint main(int a,char > >> **b){MPI_Init(&a,&b);MPI_Finalize();}" > t.c && > >> $PETSC_DIR/$PETSC_ARCH/bin/mpicc t.c && valgrind ./a.out > >> > >> ==14242== Memcheck, a memory error detector > >> ==14242== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward > et al. > >> ==14242== Using Valgrind-3.13.0 and LibVEX; rerun with -h for > copyright info > >> ==14242== Command: ./a.out > >> ==14242== > >> ==14242== > >> ==14242== HEAP SUMMARY: > >> ==14242== in use at exit: 0 bytes in 0 blocks > >> ==14242== total heap usage: *1,979 allocs, 1,974 frees*, 4,720,483 > bytes > >> allocated > >> ==14242== > >> ==14242== All heap blocks were freed -- no leaks are possible > >> ==14242== > >> ==14242== For counts of detected and suppressed errors, rerun with: -v > >> ==14242== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) > >> > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Fwd: [petsc-dev] MPICH from --download-mpich reports inconsistent allocs/frees with valgrind
by Patrick Sanan 23 Jul '19

23 Jul '19

I'm trying to track down the cause of an inconsistency in valgrind's heap allocation analysis. This arose when using the MPICH downloaded by PETSc. To reproduce, I run the following and note that the number of allocations and frees are different (yet all blocks are reported free). printf "#include<mpi.h>\nint main(int a,char **b){MPI_Init(&a,&b);MPI_Finalize();}" > t.c && $PETSC_DIR/$PETSC_ARCH/bin/mpicc t.c && valgrind ./a.out ==8578== Memcheck, a memory error detector ==8578== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==8578== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==8578== Command: ./a.out ==8578== ==8578== ==8578== HEAP SUMMARY: ==8578== in use at exit: 0 bytes in 0 blocks ==8578== total heap usage: *1,979 allocs, 1,974 frees*, 4,720,483 bytes allocated ==8578== ==8578== All heap blocks were freed -- no leaks are possible ==8578== ==8578== For counts of detected and suppressed errors, rerun with: -v ==8578== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) The above was generated on Ubuntu 18.04.2 LTS, using MPICH 3.3b1 as configured in the attached config.log (which was done automatically for me by PETSc master). ---------- Forwarded message --------- Von: Smith, Barry F. <bsmith(a)mcs.anl.gov> Date: Mo., 22. Juli 2019 um 20:24 Uhr Subject: Re: [petsc-dev] MPICH from --download-mpich reports inconsistent allocs/frees with valgrind To: petsc-dev <petsc-dev(a)mcs.anl.gov> Cc: Patrick Sanan <patrick.sanan(a)gmail.com>, Balay, Satish < balay(a)mcs.anl.gov> Bug report to MPICH. > On Jul 22, 2019, at 1:22 PM, Balay, Satish via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > > Hm - I don't think we were monitoring the leaks via valgrind that closely. > > Looking at my old mpich install - I don't see a problem - so likely > its an issue with newer versions of mpich. > > Satish > > ------- > balay@sb /home/balay/tmp > $ mpichversion > MPICH Version: 3.3 > MPICH Release date: Wed Nov 21 11:32:40 CST 2018 > MPICH Device: ch3:sock > MPICH configure: --prefix=/home/balay/soft/mpich-3.3 MAKE=/usr/bin/gmake --libdir=/home/balay/soft/mpich-3.3/lib CC=gcc CFLAGS=-fPIC -g -O AR=/usr/bin/ar ARFLAGS=cr CXX=g++ CXXFLAGS=-g -O -fPIC F77=gfortran FFLAGS=-fPIC -g -O FC=gfortran FCFLAGS=-fPIC -g -O --enable-shared --with-device=ch3:sock --with-pm=hydra --enable-fast=no --enable-error-messages=all --enable-g=meminit > MPICH CC: gcc -fPIC -g -O -O0 > MPICH CXX: g++ -g -O -fPIC -O0 > MPICH F77: gfortran -fPIC -g -O -O0 > MPICH FC: gfortran -fPIC -g -O -O0 > MPICH Custom Information: > balay@sb /home/balay/tmp > $ printf "#include<mpi.h>\nint main(int a,char**b){MPI_Init(&a,&b);MPI_Finalize();}" > t.c && mpicc t.c && valgrind ./a.out > ==9024== Memcheck, a memory error detector > ==9024== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. > ==9024== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info > ==9024== Command: ./a.out > ==9024== > ==9024== > ==9024== HEAP SUMMARY: > ==9024== in use at exit: 0 bytes in 0 blocks > ==9024== total heap usage: 1,886 allocs, 1,886 frees, 4,884,751 bytes allocated > ==9024== > ==9024== All heap blocks were freed -- no leaks are possible > ==9024== > ==9024== For lists of detected and suppressed errors, rerun with: -s > ==9024== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) > balay@sb /home/balay/tmp > > > > On Mon, 22 Jul 2019, Patrick Sanan via petsc-dev wrote: > >> It was pointed out to me that valgrind memcheck reports inconsistent heap >> usage information when running PETSc examples. All blocks are reported >> freed, yet the number of allocations and frees are different. My guess as >> to what's going on is that this is an MPICH issue, as I can reproduce the >> behavior with a minimal MPI program. >> >>> From the PETSc perspective, is this a known issue? I'm wondering if this >> inconsistency was always there, whether it's worth looking into more, etc. >> >> Here's a 1-liner to reproduce, using a PETSc master build with >> --download-mpich (though note that this doesn't use anything from PETSc >> except the MPICH it builds for you). >> >> printf "#include<mpi.h>\nint main(int a,char >> **b){MPI_Init(&a,&b);MPI_Finalize();}" > t.c && >> $PETSC_DIR/$PETSC_ARCH/bin/mpicc t.c && valgrind ./a.out >> >> ==14242== Memcheck, a memory error detector >> ==14242== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. >> ==14242== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info >> ==14242== Command: ./a.out >> ==14242== >> ==14242== >> ==14242== HEAP SUMMARY: >> ==14242== in use at exit: 0 bytes in 0 blocks >> ==14242== total heap usage: *1,979 allocs, 1,974 frees*, 4,720,483 bytes >> allocated >> ==14242== >> ==14242== All heap blocks were freed -- no leaks are possible >> ==14242== >> ==14242== For counts of detected and suppressed errors, rerun with: -v >> ==14242== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0) >> >

1 0

stdin behavior with mpich / mpiexec 3.x
by Kehoe, Devon 16 Jul '19

16 Jul '19

Hello, We are in the process of transitioning an application from mpich2 1.5 to mpich 3.2.1 . I am seeing different behavior between these two versions in how stdin is handled for processes other than the rank=0 process - I will refer to these as "slave processes". I am trying to understand if this behavior is expected, and if so, if there is some way (switch etc.) to revert behavior to match that of mpich 2.x. The behavior I see in mpich2 is that for these slave processes, if a read occurs in stdin, an "EOF" is returned. For mpich 3.x, when one of these processes reads stdin, it appears to just block on the read. So, it appears as if the stdin was previously (for mpich2) closed, but is now left open, though not really receiving stdin. I have tried this with mpich 3.2.1 and mpich 3.3.1, and see the same behavior. Below I include a simple test I ran to reproduce the behavior. I understand that these slave processes should not be reading stdin at all. However, we have a source code that is used in a variety of scenarios, not always as a parallel application, and it can be problematic to just eliminate all cases where this can occur. We handled the EOF condition gracefully, but not the blocking behavior. So I first just want to understand whether this new behavior in mpich 3.x (blocking on stdin read) is a bug or is expected behavior with mpich 3.x. (And also, just out of curiousity: why did it change between mpich2 and mpich 3.x?) My guess is this behavior is expected or an artifact of some other change, and the solution is to avoid stdin reads for all processes besides rank=0. But I would like to confirm this if possible. I have read discussions about a couple stdin issues that seem related, but I can't quite map them to my particular situation, perhaps because I don't understand the full context of these discussions (e.g., "slurm"): https://lists.mpich.org/mailman/htdig/discuss/2015-July/001104.html https://github.com/pmodels/mpich/issues/1782 Below is a simple C program I launched from mpiexec to test this, along with the mpiexec command I use to launch two processes running the test program. Note that it makes no calls to the MPI API, it just tries to read a character from stdin from each process. When I run with mpich2 1.5, what I see from the slave/child process is that reading stdin (getc call) immediately returns EOF. With mpich 3.2.1 or 3.3.1, it appears to block on the call to getc. (And then a subsequent attempt to enter a character, after the first process exits, causes an mpiexec failure, perhaps because the first process has exited.) Here is the mpiexec commands used to run the test (test program executable is: <MPICH_PATH>/mpich2-1.5/linux/bin/mpiexec -n 1 read_stdin : -n 1 read_stdin -child <MPICH_PATH>/mpich-3.3.1/linux/bin/mpiexec -n 1 read_stdin : -n 1 read_stdin -child Test program "read_stdin.c" : /***** BEGIN TEST PROGRAM *****/ #include <stdio.h> #include <sys/time.h> #include <sys/types.h> #include <unistd.h> /* Simple program that reads line from stdin - for testing mpiexec and stdin redirect (for flps/mc2 issue) */ main(int argc, char *argv[]) { int stat = 0; pid_t pid = getpid(); if ((argc > 1) && (!strcmp(argv[1], "-child"))) { fprintf(stdout, "pid=%d : child process\n", pid); fflush(stdout); } else { fprintf(stdout, "pid=%d : master process\n", pid); fflush(stdout); } fprintf(stdout, "pid=%d : Enter char: \n", pid); fflush(stdout); stat = getc(stdin); if (stat == EOF) { fprintf(stdout, "pid=%d : getc returned EOF\n", pid); fflush(stdout); } else { fprintf(stdout, "pid=%d : getc returned %c\n", pid, (char) stat); fflush(stdout); } fprintf(stdout, "pid=%d : exiting\n", pid); fflush(stdout); } /***** END TEST PROGRAM *****/ Thanks in advance for any information about this. Devon Kehoe

1 0

mpix_comm_shrink hangs for more than 32 ranks
by Atis Degro 12 Jul '19

12 Jul '19

Dear MPICH community, I am working on fault tolerant code and using the mpix_comm_shrink call to exclude the failed processes from the communicator. Everything seems to be working fine if the size of the communicator is 32 or smaller. If I increase the size however, the mpix_comm_shrink call fails (hangs). Is this a known issue and is there any workaround to enable this call for larger communicators? Thank you! Atis

1 0