- devel - mailman3.cels.anl.gov

Re: [mpich-devel] MPI_Win leaking memory
by Zhao, Xin 05 Jun '15

05 Jun '15

Hi Lisandro, Thanks. I will look into this. Xin ________________________________________ From: Lisandro Dalcin [dalcinl(a)gmail.com] Sent: Friday, June 05, 2015 10:02 AM To: mpich-devel Subject: Re: [mpich-devel] MPI_Win leaking memory On 5 June 2015 at 15:01, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Huh. I wonder why our testing is not catching this. Can you send us the exact test you used? > This is the minimal Python code able to reproduce the leak, translation to C is obvious. I would not be surprised if none of your tests have such a code path. from mpi4py import MPI win = MPI.Win.Create(MPI.BOTTOM, 1, MPI.INFO_NULL, MPI.COMM_SELF) info = win.Get_info() win.Set_info(info) info.Free() win.Free() -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459 _______________________________________________ To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/devel

1 0

Re: [mpich-devel] MPI_Win leaking memory
by Lisandro Dalcin 05 Jun '15

05 Jun '15

On 5 June 2015 at 15:01, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Huh. I wonder why our testing is not catching this. Can you send us the exact test you used? > This is the minimal Python code able to reproduce the leak, translation to C is obvious. I would not be surprised if none of your tests have such a code path. from mpi4py import MPI win = MPI.Win.Create(MPI.BOTTOM, 1, MPI.INFO_NULL, MPI.COMM_SELF) info = win.Get_info() win.Set_info(info) info.Free() win.Free() -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] MPI_Win leaking memory
by William Gropp 05 Jun '15

05 Jun '15

That’s a good question. If the coverage testing hadn’t been broken, we could check to see if there is RMA code that our tests aren’t executing. Bill On Jun 5, 2015, at 7:01 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Huh. I wonder why our testing is not catching this. Can you send us the exact test you used? > > -- Pavan > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > > > > > On 6/5/15, 3:42 AM, "Lisandro Dalcin" <dalcinl(a)gmail.com> wrote: > >> I configured 3.2b3 with --enable-g=all --enable-error-messages=all >> --enable-error-checking=all >> >> Then I get: >> >> $ python test/runtests.py -q >> ---------------------------------------------------------------------- >> Ran 1018 tests in 7.253s >> >> OK >> [0] 20480 at [0x0000000002b3ed58], src/mpid/ch3/src/mpid_rma.c[382] >> [0] 20480 at [0x0000000002b38118], src/mpid/ch3/src/mpid_rma.c[382] >> [0] 20480 at [0x0000000002b70ad8], src/mpid/ch3/src/mpid_rma.c[382] >> [0] 20480 at [0x0000000002b6ba28], src/mpid/ch3/src/mpid_rma.c[382] >> [0] 20480 at [0x0000000002b60978], src/mpid/ch3/src/mpid_rma.c[382] >> [0] 20480 at [0x0000000002b5b8c8], src/mpid/ch3/src/mpid_rma.c[382] >> [0] 20480 at [0x0000000002b56818], src/mpid/ch3/src/mpid_rma.c[382] >> [0] 20480 at [0x0000000002b4c1c8], src/mpid/ch3/src/mpid_rma.c[382] >> >> So it seems MPI_Win's are leaking memory [ (*win_ptr)->lock_entry_pool_start ] >> >> -- >> Lisandro Dalcin >> ============ >> Research Scientist >> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) >> Numerical Porous Media Center (NumPor) >> King Abdullah University of Science and Technology (KAUST) >> http://numpor.kaust.edu.sa/ >> >> 4700 King Abdullah University of Science and Technology >> al-Khawarizmi Bldg (Bldg 1), Office # 4332 >> Thuwal 23955-6900, Kingdom of Saudi Arabia >> http://www.kaust.edu.sa >> >> Office Phone: +966 12 808-0459 >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0

Re: [mpich-devel] MPI_Win leaking memory
by Balaji, Pavan 05 Jun '15

05 Jun '15

Huh. I wonder why our testing is not catching this. Can you send us the exact test you used? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji On 6/5/15, 3:42 AM, "Lisandro Dalcin" <dalcinl(a)gmail.com> wrote: >I configured 3.2b3 with --enable-g=all --enable-error-messages=all >--enable-error-checking=all > >Then I get: > >$ python test/runtests.py -q >---------------------------------------------------------------------- >Ran 1018 tests in 7.253s > >OK >[0] 20480 at [0x0000000002b3ed58], src/mpid/ch3/src/mpid_rma.c[382] >[0] 20480 at [0x0000000002b38118], src/mpid/ch3/src/mpid_rma.c[382] >[0] 20480 at [0x0000000002b70ad8], src/mpid/ch3/src/mpid_rma.c[382] >[0] 20480 at [0x0000000002b6ba28], src/mpid/ch3/src/mpid_rma.c[382] >[0] 20480 at [0x0000000002b60978], src/mpid/ch3/src/mpid_rma.c[382] >[0] 20480 at [0x0000000002b5b8c8], src/mpid/ch3/src/mpid_rma.c[382] >[0] 20480 at [0x0000000002b56818], src/mpid/ch3/src/mpid_rma.c[382] >[0] 20480 at [0x0000000002b4c1c8], src/mpid/ch3/src/mpid_rma.c[382] > >So it seems MPI_Win's are leaking memory [ (*win_ptr)->lock_entry_pool_start ] > >-- >Lisandro Dalcin >============ >Research Scientist >Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) >Numerical Porous Media Center (NumPor) >King Abdullah University of Science and Technology (KAUST) >http://numpor.kaust.edu.sa/ > >4700 King Abdullah University of Science and Technology >al-Khawarizmi Bldg (Bldg 1), Office # 4332 >Thuwal 23955-6900, Kingdom of Saudi Arabia >http://www.kaust.edu.sa > >Office Phone: +966 12 808-0459 >_______________________________________________ >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/devel

1 0

MPI_Win leaking memory
by Lisandro Dalcin 05 Jun '15

05 Jun '15

I configured 3.2b3 with --enable-g=all --enable-error-messages=all --enable-error-checking=all Then I get: $ python test/runtests.py -q ---------------------------------------------------------------------- Ran 1018 tests in 7.253s OK [0] 20480 at [0x0000000002b3ed58], src/mpid/ch3/src/mpid_rma.c[382] [0] 20480 at [0x0000000002b38118], src/mpid/ch3/src/mpid_rma.c[382] [0] 20480 at [0x0000000002b70ad8], src/mpid/ch3/src/mpid_rma.c[382] [0] 20480 at [0x0000000002b6ba28], src/mpid/ch3/src/mpid_rma.c[382] [0] 20480 at [0x0000000002b60978], src/mpid/ch3/src/mpid_rma.c[382] [0] 20480 at [0x0000000002b5b8c8], src/mpid/ch3/src/mpid_rma.c[382] [0] 20480 at [0x0000000002b56818], src/mpid/ch3/src/mpid_rma.c[382] [0] 20480 at [0x0000000002b4c1c8], src/mpid/ch3/src/mpid_rma.c[382] So it seems MPI_Win's are leaking memory [ (*win_ptr)->lock_entry_pool_start ] -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] Issue with too many calls to MPIR_Add_finalize
by Lisandro Dalcin 05 Jun '15

05 Jun '15

On 5 June 2015 at 01:58, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: >> That's a good fix. My only concern is if we hit the limit not because it is too small, but because something is going wrong. > > I would also be a bit concerned about that. So, I did the following test: I applied my patch to 3.2b3, then modified mpi4py's test/runtests.py slightly to run the full testsuite three successive times. Using gdb the same way (see the ticket for details) I can confirm MPIR_Add_finalize() is being invoked only during the first round, and exactly 33 times ;-). So now I'm somewhat confident my patch is OK and nothing else is going wrong. About using a linked list, not sure whether it is worth the effort. I would just improve the error message with some directions to increase the array size, so if this issue ever triggers again, MPICH developers would figure out immediately what's going on. -- Lisandro Dalcin ============ Research Scientist Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) Numerical Porous Media Center (NumPor) King Abdullah University of Science and Technology (KAUST) http://numpor.kaust.edu.sa/ 4700 King Abdullah University of Science and Technology al-Khawarizmi Bldg (Bldg 1), Office # 4332 Thuwal 23955-6900, Kingdom of Saudi Arabia http://www.kaust.edu.sa Office Phone: +966 12 808-0459

1 0

Re: [mpich-devel] Issue with too many calls to MPIR_Add_finalize
by Dave Goodell (dgoodell) 04 Jun '15

04 Jun '15

On Jun 4, 2015, at 4:00 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > On 06/04/2015 02:26 AM, Lisandro Dalcin wrote: >> >> >> Although I've not tested, this fix should be enough: > > That's a good fix. My only concern is if we hit the limit not because it is too small, but because something is going wrong. I would also be a bit concerned about that. > Hey, we doubled the value back in 2008. Maybe we just double it every 6.5 years and we're fine for a few decades. A linked-list-based stack of malloc-ed nodes is a pretty easy fix for this too, instead of having some arbitrarily sized array. But that's assuming that Rob's earlier concern is not actually an issue. -Dave

1 0

Re: [mpich-devel] Issue with too many calls to MPIR_Add_finalize
by Rob Latham 04 Jun '15

04 Jun '15

On 06/04/2015 02:26 AM, Lisandro Dalcin wrote: > On 3 June 2015 at 21:57, Rob Latham <robl(a)mcs.anl.gov> wrote: >> >> >> On 05/07/2015 07:45 AM, Lisandro Dalcin wrote: >>> >>> I'm running the testsuite of my Python wrappers with MPICH 3.2b2. The >>> runs are aborting this way: >> >> >> Sorry no one got back to you yet. I opened a ticket to track this issue >> >> https://trac.mpich.org/projects/mpich/ticket/2272 >> > > Although I've not tested, this fix should be enough: That's a good fix. My only concern is if we hit the limit not because it is too small, but because something is going wrong. Hey, we doubled the value back in 2008. Maybe we just double it every 6.5 years and we're fine for a few decades. ==rob > > diff --git a/src/mpi/init/finalize.c b/src/mpi/init/finalize.c > index f71668b..afebd77 100644 > --- a/src/mpi/init/finalize.c > +++ b/src/mpi/init/finalize.c > @@ -65,7 +65,7 @@ typedef struct Finalize_func_t { > } Finalize_func_t; > /* When full debugging is enabled, each MPI handle type has a finalize handler > installed to detect unfreed handles. */ > -#define MAX_FINALIZE_FUNC 32 > +#define MAX_FINALIZE_FUNC 64 > static Finalize_func_t fstack[MAX_FINALIZE_FUNC]; > static int fstack_sp = 0; > static int fstack_max_priority = 0; > > > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

lots of windows -> lots of comms -> fail
by Jeff Hammond 04 Jun '15

04 Jun '15

Why does MPICH need to dupe the comm for every window? I guess performing a collective in MPI_WIN_FENCE is one reason. Does MPI_WIN_FREE need an internal comm? Are there others? If I were to provide an info saying that I only use passive target, do you still need to dupe the comm? If there is any way to avoid one comm per window, I would like to do that. And yes, I am indeed asserting that 2048 windows is not enough. Thanks, Jeff jrhammon-mac01:ticket459 jrhammon$ mpiexec -n 2 ./test_win_sync.x 10000 Fatal error in MPI_Win_allocate_shared: Other MPI error, error stack: MPI_Win_allocate_shared(162).........: MPI_Win_allocate_shared(size=4, MPI_INFO_NULL, comm=0x84000004, baseptr=0x7fff53c163c0, win=0x7fff53c0a780) failed MPID_Win_allocate_shared(248)........: win_init(294)........................: MPIR_Comm_dup_impl(57)...............: MPIR_Comm_copy(1773).................: MPIR_Get_contextid(829)..............: MPIR_Get_contextid_sparse_group(1203): Too many communicators (0/2048 free on this process; ignore_id=0) Fatal error in MPI_Win_allocate_shared: Other MPI error, error stack: MPI_Win_allocate_shared(162).........: MPI_Win_allocate_shared(size=4, MPI_INFO_NULL, comm=0x84000002, baseptr=0x7fff572853c0, win=0x7fff57279780) failed MPID_Win_allocate_shared(248)........: win_init(294)........................: MPIR_Comm_dup_impl(57)...............: MPIR_Comm_copy(1773).................: MPIR_Get_contextid(829)..............: MPIR_Get_contextid_sparse_group(1203): Too many communicators (0/2048 free on this process; ignore_id=0) -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] MPICH hangs in MPI_Waitall when MPI_Cancel is used
by Balaji, Pavan 04 Jun '15

04 Jun '15

Actually, we just discovered that the portals4 implementation is incorrect as well. I'll try to write a test to demonstrate it. -- Pavan On 6/4/15, 1:11 PM, "Halim Amer" <aamer(a)anl.gov> wrote: >That's right, but more importantly, cancelling sends is still not >supported by the MXM and OFI netmods. Intel and Mellanox are working on >it (tickets 2266 and 2270). It works fine so far with the TCP and >Portals4 netmods though. > >--Halim > > >On 6/4/15 1:00 PM, Rob Latham wrote: >> >> >> On 06/04/2015 12:19 PM, Jeff Hammond wrote: >>> Thanks for pointing that out. It runs correctly now. Sorry for the >>> stupid question. >> >> it just so happens, Jeff, that they've been spending a lot of time >> debugging cancel send operations for all our various devices and so >> "cancel semantics" are (moreso than usual) quite warm in the cache. >> >> ==rob >> >>> On Thu, Jun 4, 2015 at 11:49 AM, Halim Amer <aamer(a)anl.gov> wrote: >>>> Hi Jeff, >>>> >>>> I don't think it is a correct program. If the send is correctly canceled >>>> then the origin has to satisfy the destination with another send. The >>>> hang >>>> is an expected result. >>>> >>>> This is what the standard says (P102): >>>> >>>> "...or that the send is successfully cancelled, in which case no part >>>> of the >>>> message was received at the destination. Then, any matching receive >>>> has to >>>> be satisfied by another send." >>>> >>>> --Halim >>>> >>>> Abdelhalim Amer (Halim) >>>> Postdoctoral Appointee >>>> MCS Division >>>> Argonne National Laboratory >>>> >>>> >>>> On 6/4/15 9:21 AM, Jeff Hammond wrote: >>>>> >>>>> I can't tell for sure if this is a correct program, but multiple >>>>> members of the MPI Forum suggested it is. >>>>> >>>>> If it is a correct program, it appears to expose a bug in MPICH, >>>>> because the MPI_Waitall hangs. >>>>> >>>>> Thanks, >>>>> >>>>> Jeff >>>>> >>>>> $ mpicc -g -Wall -std=c99 cancel-sucks.c && mpiexec -n 4 ./a.out >>>>> >>>>> $ mpichversion >>>>> MPICH Version: 3.2b1 >>>>> MPICH Release date: unreleased development copy >>>>> MPICH Device: ch3:nemesis >>>>> MPICH configure: CC=gcc-4.9 CXX=g++-4.9 FC=gfortran-4.9 >>>>> F77=gfortran-4.9 --enable-cxx --enable-fortran >>>>> --enable-threads=runtime --enable-g=dbg --with-pm=hydra >>>>> --prefix=/opt/mpich/dev/gcc/default --enable-wrapper-rpath >>>>> --enable-static --enable-shared >>>>> MPICH CC: gcc-4.9 -g -O2 >>>>> MPICH CXX: g++-4.9 -g -O2 >>>>> MPICH F77: gfortran-4.9 -g -O2 >>>>> MPICH FC: gfortran-4.9 -g -O2 >>>>> >>>>> >>>>> #include <stdio.h> >>>>> #include <stdlib.h> >>>>> #include <mpi.h> >>>>> >>>>> const int n=1000; >>>>> >>>>> int main(void) >>>>> { >>>>> MPI_Init(NULL,NULL); >>>>> >>>>> int size, rank; >>>>> MPI_Comm_size(MPI_COMM_WORLD, &size); >>>>> MPI_Comm_rank(MPI_COMM_WORLD, &rank); >>>>> if (size<2) { >>>>> printf("You must use 2 or more processes!\n"); >>>>> MPI_Finalize(); >>>>> exit(1); >>>>> } >>>>> >>>>> MPI_Request reqs[2*n]; >>>>> >>>>> int target = (rank+1)%size; >>>>> for (int i=0; i<n; i++) { >>>>> >>>>> MPI_Issend(NULL,0,MPI_BYTE,target,0,MPI_COMM_WORLD,&(reqs[i])); >>>>> } >>>>> >>>>> srand((unsigned)(rank+MPI_Wtime())); >>>>> int slot = rand()%n; >>>>> printf("Cancelling send %d.\n", slot); >>>>> MPI_Cancel(&reqs[slot]); >>>>> >>>>> #if 1 >>>>> MPI_Barrier(MPI_COMM_WORLD); >>>>> #endif >>>>> >>>>> int origin = (rank==0) ? (size-1) : (rank-1); >>>>> for (int i=0; i<n; i++) { >>>>> >>>>> MPI_Irecv(NULL,0,MPI_BYTE,origin,0,MPI_COMM_WORLD,&(reqs[n+i])); >>>>> } >>>>> >>>>> MPI_Status stats[2*n]; >>>>> MPI_Waitall(2*n,reqs,stats); >>>>> >>>>> for (int i=0; i<n; i++) { >>>>> int flag; >>>>> MPI_Test_cancelled(&(stats[i]),&flag); >>>>> if (flag) { >>>>> printf("Status %d indicates cancel was successful.\n", >>>>> i); >>>>> } >>>>> } >>>>> >>>>> MPI_Finalize(); >>>>> return 0; >>>>> } >>>>> >>>>> >>>> _______________________________________________ >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/devel >>> >>> >>> >> >_______________________________________________ >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/devel

1 0