May 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI-IO bug
by Wei-keng Liao 02 May '14

02 May '14

Hi, Rob, Now we are all clear about the lower bound for indexed data type. I wonder if the correct solution (at least for calculating n_filetypes) should change from n_filetypes = (offset - flat_file->indices[0]) / filetype_extent; to n_filetypes = (offset - filetype_lb) / filetype_extent; Wei-keng On May 1, 2014, at 6:03 PM, Wei-keng Liao wrote: > > The flattened filetype offset-length lists are: (i.e. flat_file->indices[] and flat_file->blocklens[]) > rank0: (0, 0) (4, 0) (8, 0) (12, 0) (16, 8) (28, 4) > rank1: (0, 16) (20, 0) (24, 4) (32, 4) > > What should their type extents be? > > MPI_Type_extent() returns values below: > rank 0: filetype_extent == 16 LB == 16 > rank 1: filetype_extent == 36 LB == 0 > > Is this correct? Shouldn't rank 0's extent be 32 and its LB be 0? > > If filetype_extent 16 is correct, then the way of calculating n_filetypes (below) is wrong. > n_filetypes = (offset - flat_file->indices[0]) / filetype_extent; > > The divisor should be the "range extent" of flat_file, i.e. > (flat_file->indices[flat_file->count-1] + flat_file->blocklens[flat_file->count-1]) - flat_file->indices[0] > > > Wei-keng > > On May 1, 2014, at 3:22 PM, Rob Latham wrote: > >> >> >> On 05/01/2014 12:02 PM, Wei-keng Liao wrote: >>> They are reasonable doubts. >>> I tested this patch against test/mpi/io/resized.c and src/mpi/romio/test/hindexed.c >>> >>> Could you run tests (both MPI and ROMIO) and see if the patch failed for any tests? >>> At least we will have an idea if it breaks anything first. >> >> OK, happy to do so. We've groused about the sparseness of the ROMIO test coverage for 12 years, but it's enough to catch a few things. >> >> This change introduces no new test failures, but I think precisely zero tests tile the file view. >> >> I still want to better understand how this works >> >> the indexed datatype looks like this (view in fixed-with font please) >> >> rank 0: |----11-1-| >> rank 1: |1111--1-1| >> >> ADIOI_Calc_my_off_len() should return a list of 3 things for rank 0 and a list of 3 things for rank 1 . For this test, it does the right thing for rank 1, returning 3 offset-length tuples: (0,16) (24,4) (32,4) >> >> but rank 0 returns the entirely incorrect >> (16,12) >> >> instead of >> (0,0) (16, 8) (28, 4) >> >> My concern is mostly with tiling, or in the context of Calc_my_off_len, how n_filetypes is computed (there's also filetype_extent, but ROMIO calls the MPI library for that, so I have high confidence it is correct) >> >> Fun finding: if I alter your test case to test the effects of tiling a file view -- by resizing the indexed type -- ROMIO handles that just fine without your change. >> >> So actually the problem with Calc_my_off_len seems to be that the indexed type puts the underlying file offset at '16', and ROMIO thinks that means the datatype was tiled once -- it seems to be ignoring the lower bound marker. >> >> It's just a small tweak to your approach but I'd like to make it a bit more explicit that the flattened representation uses a zero-length item to indicate the LB of the type. What do you think of the attached patch? >> >> I'm still not convinced this properly handles tiling but we fix a known issue and can deal with the tiling-when-not-resized issue later. >> >> ==rob >> >> >>> >>> Wei-keng >>> >>> On May 1, 2014, at 10:53 AM, Rob Latham wrote: >>> >>>> >>>> >>>> On 04/09/2014 02:52 PM, Wei-keng Liao wrote: >>>> >>>>> >>>>> The patch below can fix this problem. Hope it does not break other tests. >>>> >>>> I'm uncertain about this patch... >>>> >>>>> Index: adio/common/ad_read_coll.c >>>> >>>> it should have been in the write path, correct >>>> ? >>>>> @@ -368,13 +368,16 @@ >>>>> #endif >>>>> if (file_ptr_type == ADIO_INDIVIDUAL) { >>>>> /* Wei-keng reworked type processing to be a bit more >>>>> efficient */ >>>>> + for (i=0; i<flat_file->count; i++) /* skip blocklens[] == 0 */ >>>>> + if (flat_file->blocklens[i] > 0) break; >>>>> + >>>> >>>> the flat_file->blocklens[] array might contain a UB and LB marker. The marker will be stored in the flattened representation with a zero-length element at a particular offset. So if you resize a type, your flattened representation would be like this: >>>> (offset: 50, length: 0), >>>> (offset: 100, length: 1024), >>>> (offset: 200, length: 0) >>>> >>>> furthermore, ADIOI_Optimize_flatten will coalesce multiple zero-length elements into one, but will leave the first and last elements alone. >>>> >>>>> offset = fd->fp_ind - disp; >>>>> - n_filetypes = (offset - flat_file->indices[0]) / >>>>> filetype_extent; >>>>> + n_filetypes = (offset - flat_file->indices[i]) / >>>>> filetype_extent; >>>> >>>> doesn't this mess up tiling of the file view? >>>> >>>> ==rob >>>> >>>>> offset -= (ADIO_Offset)n_filetypes * filetype_extent; >>>>> /* now offset is local to this extent */ >>>>> >>>>> /* find the block where offset is located, skip >>>>> blocklens[i]==0 */ >>>>> - for (i=0; i<flat_file->count; i++) { >>>>> + for (; i<flat_file->count; i++) { >>>>> ADIO_Offset dist; >>>>> if (flat_file->blocklens[i] == 0) continue; >>>>> dist = flat_file->indices[i] + flat_file->blocklens[i] >>>>> - offset; >>>>> >>>>> Wei-keng >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> discuss mailing list discuss(a)mpich.org >>>>> To manage subscription options or unsubscribe: >>>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> -- >>>> Rob Latham >>>> Mathematics and Computer Science Division >>>> Argonne National Lab, IL USA >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> -- >> Rob Latham >> Mathematics and Computer Science Division >> Argonne National Lab, IL USA >> <0001-Better-deal-with-file-view-types-with-lb.patch>_______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI-IO bug
by Wei-keng Liao 01 May '14

01 May '14

The flattened filetype offset-length lists are: (i.e. flat_file->indices[] and flat_file->blocklens[]) rank0: (0, 0) (4, 0) (8, 0) (12, 0) (16, 8) (28, 4) rank1: (0, 16) (20, 0) (24, 4) (32, 4) What should their type extents be? MPI_Type_extent() returns values below: rank 0: filetype_extent == 16 LB == 16 rank 1: filetype_extent == 36 LB == 0 Is this correct? Shouldn't rank 0's extent be 32 and its LB be 0? If filetype_extent 16 is correct, then the way of calculating n_filetypes (below) is wrong. n_filetypes = (offset - flat_file->indices[0]) / filetype_extent; The divisor should be the "range extent" of flat_file, i.e. (flat_file->indices[flat_file->count-1] + flat_file->blocklens[flat_file->count-1]) - flat_file->indices[0] Wei-keng On May 1, 2014, at 3:22 PM, Rob Latham wrote: > > > On 05/01/2014 12:02 PM, Wei-keng Liao wrote: >> They are reasonable doubts. >> I tested this patch against test/mpi/io/resized.c and src/mpi/romio/test/hindexed.c >> >> Could you run tests (both MPI and ROMIO) and see if the patch failed for any tests? >> At least we will have an idea if it breaks anything first. > > OK, happy to do so. We've groused about the sparseness of the ROMIO test coverage for 12 years, but it's enough to catch a few things. > > This change introduces no new test failures, but I think precisely zero tests tile the file view. > > I still want to better understand how this works > > the indexed datatype looks like this (view in fixed-with font please) > > rank 0: |----11-1-| > rank 1: |1111--1-1| > > ADIOI_Calc_my_off_len() should return a list of 3 things for rank 0 and a list of 3 things for rank 1 . For this test, it does the right thing for rank 1, returning 3 offset-length tuples: (0,16) (24,4) (32,4) > > but rank 0 returns the entirely incorrect > (16,12) > > instead of > (0,0) (16, 8) (28, 4) > > My concern is mostly with tiling, or in the context of Calc_my_off_len, how n_filetypes is computed (there's also filetype_extent, but ROMIO calls the MPI library for that, so I have high confidence it is correct) > > Fun finding: if I alter your test case to test the effects of tiling a file view -- by resizing the indexed type -- ROMIO handles that just fine without your change. > > So actually the problem with Calc_my_off_len seems to be that the indexed type puts the underlying file offset at '16', and ROMIO thinks that means the datatype was tiled once -- it seems to be ignoring the lower bound marker. > > It's just a small tweak to your approach but I'd like to make it a bit more explicit that the flattened representation uses a zero-length item to indicate the LB of the type. What do you think of the attached patch? > > I'm still not convinced this properly handles tiling but we fix a known issue and can deal with the tiling-when-not-resized issue later. > > ==rob > > >> >> Wei-keng >> >> On May 1, 2014, at 10:53 AM, Rob Latham wrote: >> >>> >>> >>> On 04/09/2014 02:52 PM, Wei-keng Liao wrote: >>> >>>> >>>> The patch below can fix this problem. Hope it does not break other tests. >>> >>> I'm uncertain about this patch... >>> >>>> Index: adio/common/ad_read_coll.c >>> >>> it should have been in the write path, correct >>> ? >>>> @@ -368,13 +368,16 @@ >>>> #endif >>>> if (file_ptr_type == ADIO_INDIVIDUAL) { >>>> /* Wei-keng reworked type processing to be a bit more >>>> efficient */ >>>> + for (i=0; i<flat_file->count; i++) /* skip blocklens[] == 0 */ >>>> + if (flat_file->blocklens[i] > 0) break; >>>> + >>> >>> the flat_file->blocklens[] array might contain a UB and LB marker. The marker will be stored in the flattened representation with a zero-length element at a particular offset. So if you resize a type, your flattened representation would be like this: >>> (offset: 50, length: 0), >>> (offset: 100, length: 1024), >>> (offset: 200, length: 0) >>> >>> furthermore, ADIOI_Optimize_flatten will coalesce multiple zero-length elements into one, but will leave the first and last elements alone. >>> >>>> offset = fd->fp_ind - disp; >>>> - n_filetypes = (offset - flat_file->indices[0]) / >>>> filetype_extent; >>>> + n_filetypes = (offset - flat_file->indices[i]) / >>>> filetype_extent; >>> >>> doesn't this mess up tiling of the file view? >>> >>> ==rob >>> >>>> offset -= (ADIO_Offset)n_filetypes * filetype_extent; >>>> /* now offset is local to this extent */ >>>> >>>> /* find the block where offset is located, skip >>>> blocklens[i]==0 */ >>>> - for (i=0; i<flat_file->count; i++) { >>>> + for (; i<flat_file->count; i++) { >>>> ADIO_Offset dist; >>>> if (flat_file->blocklens[i] == 0) continue; >>>> dist = flat_file->indices[i] + flat_file->blocklens[i] >>>> - offset; >>>> >>>> Wei-keng >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> -- >>> Rob Latham >>> Mathematics and Computer Science Division >>> Argonne National Lab, IL USA >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > <0001-Better-deal-with-file-view-types-with-lb.patch>_______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI-IO bug
by Rob Latham 01 May '14

01 May '14

On 05/01/2014 12:02 PM, Wei-keng Liao wrote: > They are reasonable doubts. > I tested this patch against test/mpi/io/resized.c and src/mpi/romio/test/hindexed.c > > Could you run tests (both MPI and ROMIO) and see if the patch failed for any tests? > At least we will have an idea if it breaks anything first. OK, happy to do so. We've groused about the sparseness of the ROMIO test coverage for 12 years, but it's enough to catch a few things. This change introduces no new test failures, but I think precisely zero tests tile the file view. I still want to better understand how this works the indexed datatype looks like this (view in fixed-with font please) rank 0: |----11-1-| rank 1: |1111--1-1| ADIOI_Calc_my_off_len() should return a list of 3 things for rank 0 and a list of 3 things for rank 1 . For this test, it does the right thing for rank 1, returning 3 offset-length tuples: (0,16) (24,4) (32,4) but rank 0 returns the entirely incorrect (16,12) instead of (0,0) (16, 8) (28, 4) My concern is mostly with tiling, or in the context of Calc_my_off_len, how n_filetypes is computed (there's also filetype_extent, but ROMIO calls the MPI library for that, so I have high confidence it is correct) Fun finding: if I alter your test case to test the effects of tiling a file view -- by resizing the indexed type -- ROMIO handles that just fine without your change. So actually the problem with Calc_my_off_len seems to be that the indexed type puts the underlying file offset at '16', and ROMIO thinks that means the datatype was tiled once -- it seems to be ignoring the lower bound marker. It's just a small tweak to your approach but I'd like to make it a bit more explicit that the flattened representation uses a zero-length item to indicate the LB of the type. What do you think of the attached patch? I'm still not convinced this properly handles tiling but we fix a known issue and can deal with the tiling-when-not-resized issue later. ==rob > > Wei-keng > > On May 1, 2014, at 10:53 AM, Rob Latham wrote: > >> >> >> On 04/09/2014 02:52 PM, Wei-keng Liao wrote: >> >>> >>> The patch below can fix this problem. Hope it does not break other tests. >> >> I'm uncertain about this patch... >> >>> Index: adio/common/ad_read_coll.c >> >> it should have been in the write path, correct >> ? >>> @@ -368,13 +368,16 @@ >>> #endif >>> if (file_ptr_type == ADIO_INDIVIDUAL) { >>> /* Wei-keng reworked type processing to be a bit more >>> efficient */ >>> + for (i=0; i<flat_file->count; i++) /* skip blocklens[] == 0 */ >>> + if (flat_file->blocklens[i] > 0) break; >>> + >> >> the flat_file->blocklens[] array might contain a UB and LB marker. The marker will be stored in the flattened representation with a zero-length element at a particular offset. So if you resize a type, your flattened representation would be like this: >> (offset: 50, length: 0), >> (offset: 100, length: 1024), >> (offset: 200, length: 0) >> >> furthermore, ADIOI_Optimize_flatten will coalesce multiple zero-length elements into one, but will leave the first and last elements alone. >> >>> offset = fd->fp_ind - disp; >>> - n_filetypes = (offset - flat_file->indices[0]) / >>> filetype_extent; >>> + n_filetypes = (offset - flat_file->indices[i]) / >>> filetype_extent; >> >> doesn't this mess up tiling of the file view? >> >> ==rob >> >>> offset -= (ADIO_Offset)n_filetypes * filetype_extent; >>> /* now offset is local to this extent */ >>> >>> /* find the block where offset is located, skip >>> blocklens[i]==0 */ >>> - for (i=0; i<flat_file->count; i++) { >>> + for (; i<flat_file->count; i++) { >>> ADIO_Offset dist; >>> if (flat_file->blocklens[i] == 0) continue; >>> dist = flat_file->indices[i] + flat_file->blocklens[i] >>> - offset; >>> >>> Wei-keng >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Rob Latham >> Mathematics and Computer Science Division >> Argonne National Lab, IL USA >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] MPI-IO bug
by Wei-keng Liao 01 May '14

01 May '14

They are reasonable doubts. I tested this patch against test/mpi/io/resized.c and src/mpi/romio/test/hindexed.c Could you run tests (both MPI and ROMIO) and see if the patch failed for any tests? At least we will have an idea if it breaks anything first. Wei-keng On May 1, 2014, at 10:53 AM, Rob Latham wrote: > > > On 04/09/2014 02:52 PM, Wei-keng Liao wrote: > >> >> The patch below can fix this problem. Hope it does not break other tests. > > I'm uncertain about this patch... > >> Index: adio/common/ad_read_coll.c > > it should have been in the write path, correct > ? >> @@ -368,13 +368,16 @@ >> #endif >> if (file_ptr_type == ADIO_INDIVIDUAL) { >> /* Wei-keng reworked type processing to be a bit more >> efficient */ >> + for (i=0; i<flat_file->count; i++) /* skip blocklens[] == 0 */ >> + if (flat_file->blocklens[i] > 0) break; >> + > > the flat_file->blocklens[] array might contain a UB and LB marker. The marker will be stored in the flattened representation with a zero-length element at a particular offset. So if you resize a type, your flattened representation would be like this: > (offset: 50, length: 0), > (offset: 100, length: 1024), > (offset: 200, length: 0) > > furthermore, ADIOI_Optimize_flatten will coalesce multiple zero-length elements into one, but will leave the first and last elements alone. > >> offset = fd->fp_ind - disp; >> - n_filetypes = (offset - flat_file->indices[0]) / >> filetype_extent; >> + n_filetypes = (offset - flat_file->indices[i]) / >> filetype_extent; > > doesn't this mess up tiling of the file view? > > ==rob > >> offset -= (ADIO_Offset)n_filetypes * filetype_extent; >> /* now offset is local to this extent */ >> >> /* find the block where offset is located, skip >> blocklens[i]==0 */ >> - for (i=0; i<flat_file->count; i++) { >> + for (; i<flat_file->count; i++) { >> ADIO_Offset dist; >> if (flat_file->blocklens[i] == 0) continue; >> dist = flat_file->indices[i] + flat_file->blocklens[i] >> - offset; >> >> Wei-keng >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI-IO bug
by Rob Latham 01 May '14

01 May '14

On 05/01/2014 10:53 AM, Rob Latham wrote: > > > On 04/09/2014 02:52 PM, Wei-keng Liao wrote: > >> >> The patch below can fix this problem. Hope it does not break other tests. > > I'm uncertain about this patch... > >> Index: adio/common/ad_read_coll.c > > it should have been in the write path, correct incorrect. this change is to ADIOI_Calc_my_offset_len, common to both read and write paths. I'm still concerned about tiling and proper recording of UB and LB, though... still digging. ==rob > ? >> @@ -368,13 +368,16 @@ >> #endif >> if (file_ptr_type == ADIO_INDIVIDUAL) { >> /* Wei-keng reworked type processing to be a bit more >> efficient */ >> + for (i=0; i<flat_file->count; i++) /* skip blocklens[] == >> 0 */ >> + if (flat_file->blocklens[i] > 0) break; >> + > > the flat_file->blocklens[] array might contain a UB and LB marker. The > marker will be stored in the flattened representation with a zero-length > element at a particular offset. So if you resize a type, your flattened > representation would be like this: > (offset: 50, length: 0), > (offset: 100, length: 1024), > (offset: 200, length: 0) > > furthermore, ADIOI_Optimize_flatten will coalesce multiple zero-length > elements into one, but will leave the first and last elements alone. > >> offset = fd->fp_ind - disp; >> - n_filetypes = (offset - flat_file->indices[0]) / >> filetype_extent; >> + n_filetypes = (offset - flat_file->indices[i]) / >> filetype_extent; > > doesn't this mess up tiling of the file view? > > ==rob > >> offset -= (ADIO_Offset)n_filetypes * filetype_extent; >> /* now offset is local to this extent */ >> >> /* find the block where offset is located, skip >> blocklens[i]==0 */ >> - for (i=0; i<flat_file->count; i++) { >> + for (; i<flat_file->count; i++) { >> ADIO_Offset dist; >> if (flat_file->blocklens[i] == 0) continue; >> dist = flat_file->indices[i] + flat_file->blocklens[i] >> - offset; >> >> Wei-keng >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] MPI-IO bug
by Rob Latham 01 May '14

01 May '14

On 04/09/2014 02:52 PM, Wei-keng Liao wrote: > > The patch below can fix this problem. Hope it does not break other tests. I'm uncertain about this patch... > Index: adio/common/ad_read_coll.c it should have been in the write path, correct ? > @@ -368,13 +368,16 @@ > #endif > if (file_ptr_type == ADIO_INDIVIDUAL) { > /* Wei-keng reworked type processing to be a bit more > efficient */ > + for (i=0; i<flat_file->count; i++) /* skip blocklens[] == 0 */ > + if (flat_file->blocklens[i] > 0) break; > + the flat_file->blocklens[] array might contain a UB and LB marker. The marker will be stored in the flattened representation with a zero-length element at a particular offset. So if you resize a type, your flattened representation would be like this: (offset: 50, length: 0), (offset: 100, length: 1024), (offset: 200, length: 0) furthermore, ADIOI_Optimize_flatten will coalesce multiple zero-length elements into one, but will leave the first and last elements alone. > offset = fd->fp_ind - disp; > - n_filetypes = (offset - flat_file->indices[0]) / > filetype_extent; > + n_filetypes = (offset - flat_file->indices[i]) / > filetype_extent; doesn't this mess up tiling of the file view? ==rob > offset -= (ADIO_Offset)n_filetypes * filetype_extent; > /* now offset is local to this extent */ > > /* find the block where offset is located, skip > blocklens[i]==0 */ > - for (i=0; i<flat_file->count; i++) { > + for (; i<flat_file->count; i++) { > ADIO_Offset dist; > if (flat_file->blocklens[i] == 0) continue; > dist = flat_file->indices[i] + flat_file->blocklens[i] > - offset; > > Wei-keng > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] Memory leak comparing communicators
by VandeVondele Joost 01 May '14

01 May '14

Dear rob, so.. no leaks but a user error. Thanks for pointing this out. I'll have a look at --enable-g=all as well. Joost ________________________________________ From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of Rob Latham [robl(a)mcs.anl.gov] Sent: Thursday, May 01, 2014 5:29 PM To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Memory leak comparing communicators On 05/01/2014 09:29 AM, VandeVondele Joost wrote: > actually, the issue seems to be with MPI_Comm_group since translate_ranks can be just commented out. your test case is missing two calls to MPI_Group_free. Building against an mpich compiled with with --enable-g=all makes this instantly clear, so thanks for the test case. ==rob > > Joost > > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of "Antonio J. Peña" [apenya(a)mcs.anl.gov] > Sent: Thursday, May 01, 2014 4:08 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Memory leak comparing communicators > > Thank you Joost. We'll check this issue. I'll keep you posted. > > > On 05/01/2014 09:02 AM, VandeVondele Joost wrote: >> Hi Antonio, >> >> I finally reproduced this in a small program. The culprit actually seems to be MPI_Group_translate_ranks, but the leak shows up with a trace to MPI_Comm_compare: >> >> Direct leak of 3072 byte(s) in 96 object(s) allocated from: >> #0 0x7fdab2e213a8 in __interceptor_malloc ../../../../gcc/libsanitizer/lsan/lsan_interceptors.cc:66 >> #1 0x7fdab3efaf6b in MPIR_Group_create /data/vjoost/mpich-3.1/src/mpi/group/grouputil.c:77 >> #2 0x7fdab3fc8314 in MPIR_Comm_group_impl /data/vjoost/mpich-3.1/src/mpi/comm/comm_group.c:44 >> #3 0x7fdab3fc40d3 in PMPI_Comm_compare /data/vjoost/mpich-3.1/src/mpi/comm/comm_compare.c:122 >> #4 0x4009de in main /data/vjoost/mpich-3.1/debug/2d.c:23 >> >>> cat 2d.c >> #include <stdio.h> >> #include <mpi.h> >> >> main(int argc, char **argv) // needs 2 ranks // >> { >> MPI_Comm comm_cart, dup_comm_world; >> int result,reorder; >> int dim[2], period[2], rin[2], rout[2]; >> int i,j,k; >> int g1,g2; >> >> MPI_Init (&argc, &argv); >> >> for(i=0;i<100;i++) >> { >> >> MPI_Comm_dup( MPI_COMM_WORLD, &dup_comm_world); >> dim[0]=2; dim[1]=1; >> reorder=0; >> period[0]=1; period[1]=1; >> rin[0]=0; rin[1]=1; >> MPI_Cart_create(dup_comm_world, 2, dim, period, reorder, &comm_cart); >> MPI_Comm_compare(dup_comm_world, comm_cart, &result ); >> MPI_Comm_group(dup_comm_world,&g1); >> MPI_Comm_group(comm_cart,&g2); >> MPI_Group_translate_ranks(g1, 2, rin, g2, rout); >> MPI_Comm_free(&comm_cart); >> MPI_Comm_free(&dup_comm_world); >> >> } >> >> MPI_Finalize(); >> >> } >> >> Thanks, >> >> Joost >> >> >> >> >> ________________________________________ >> From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of "Antonio J. Peña" [apenya(a)mcs.anl.gov] >> Sent: Monday, April 28, 2014 6:26 PM >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] Memory leak comparing communicators >> >> Hi Joost, >> >> Can you share the smallest fragment of code with which we can reproduce >> this? >> >> Antonio >> >> >> On 04/28/2014 11:25 AM, VandeVondele Joost wrote: >>> Hi, >>> >>> using mpich 3.1 I notice that my application leaks memory. Compiling it with '-O1 -g -fno-omit-frame-pointer -fsanitize=leak' using gcc 4.9 it appears the sources is comparing communicators, for example: >>> >>> [0] Direct leak of 38400 byte(s) in 600 object(s) allocated from: >>> [0] #0 0x7f84298d53a8 in __interceptor_malloc ../../../../gcc/libsanitizer/lsan/lsan_interceptors.cc:66 >>> [0] #1 0x7f842aabaf6b in MPIR_Group_create /data/vjoost/mpich-3.1/src/mpi/group/grouputil.c:77 >>> [0] #2 0x7f842ab88314 in MPIR_Comm_group_impl /data/vjoost/mpich-3.1/src/mpi/comm/comm_group.c:44 >>> [0] #3 0x7f842ab840d3 in PMPI_Comm_compare /data/vjoost/mpich-3.1/src/mpi/comm/comm_compare.c:122 >>> [0] #4 0x7f842aac8e84 in pmpi_comm_compare_ /data/vjoost/mpich-3.1/src/binding/f77/comm_comparef.c:267 >>> [0] #5 0x1bfd447 in __message_passing_MOD_mp_comm_compare /data/vjoost/clean/cp2k/cp2k/src/common/message_passing.F:1084 >>> >>> I'm not seeing other leaks, so is guess something particular is going on here. Any suggestions on a possible fix ? >>> >>> Thanks, >>> >>> Joost >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Antonio J. Peña >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847 >> apenya(a)mcs.anl.gov >> www.mcs.anl.gov/~apenya >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Memory leak comparing communicators
by Rob Latham 01 May '14

01 May '14

On 05/01/2014 09:29 AM, VandeVondele Joost wrote: > actually, the issue seems to be with MPI_Comm_group since translate_ranks can be just commented out. your test case is missing two calls to MPI_Group_free. Building against an mpich compiled with with --enable-g=all makes this instantly clear, so thanks for the test case. ==rob > > Joost > > ________________________________________ > From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of "Antonio J. Peï¿½a" [apenya(a)mcs.anl.gov] > Sent: Thursday, May 01, 2014 4:08 PM > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Memory leak comparing communicators > > Thank you Joost. We'll check this issue. I'll keep you posted. > > > On 05/01/2014 09:02 AM, VandeVondele Joost wrote: >> Hi Antonio, >> >> I finally reproduced this in a small program. The culprit actually seems to be MPI_Group_translate_ranks, but the leak shows up with a trace to MPI_Comm_compare: >> >> Direct leak of 3072 byte(s) in 96 object(s) allocated from: >> #0 0x7fdab2e213a8 in __interceptor_malloc ../../../../gcc/libsanitizer/lsan/lsan_interceptors.cc:66 >> #1 0x7fdab3efaf6b in MPIR_Group_create /data/vjoost/mpich-3.1/src/mpi/group/grouputil.c:77 >> #2 0x7fdab3fc8314 in MPIR_Comm_group_impl /data/vjoost/mpich-3.1/src/mpi/comm/comm_group.c:44 >> #3 0x7fdab3fc40d3 in PMPI_Comm_compare /data/vjoost/mpich-3.1/src/mpi/comm/comm_compare.c:122 >> #4 0x4009de in main /data/vjoost/mpich-3.1/debug/2d.c:23 >> >>> cat 2d.c >> #include <stdio.h> >> #include <mpi.h> >> >> main(int argc, char **argv) // needs 2 ranks // >> { >> MPI_Comm comm_cart, dup_comm_world; >> int result,reorder; >> int dim[2], period[2], rin[2], rout[2]; >> int i,j,k; >> int g1,g2; >> >> MPI_Init (&argc, &argv); >> >> for(i=0;i<100;i++) >> { >> >> MPI_Comm_dup( MPI_COMM_WORLD, &dup_comm_world); >> dim[0]=2; dim[1]=1; >> reorder=0; >> period[0]=1; period[1]=1; >> rin[0]=0; rin[1]=1; >> MPI_Cart_create(dup_comm_world, 2, dim, period, reorder, &comm_cart); >> MPI_Comm_compare(dup_comm_world, comm_cart, &result ); >> MPI_Comm_group(dup_comm_world,&g1); >> MPI_Comm_group(comm_cart,&g2); >> MPI_Group_translate_ranks(g1, 2, rin, g2, rout); >> MPI_Comm_free(&comm_cart); >> MPI_Comm_free(&dup_comm_world); >> >> } >> >> MPI_Finalize(); >> >> } >> >> Thanks, >> >> Joost >> >> >> >> >> ________________________________________ >> From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf of "Antonio J. Peï¿½a" [apenya(a)mcs.anl.gov] >> Sent: Monday, April 28, 2014 6:26 PM >> To: discuss(a)mpich.org >> Subject: Re: [mpich-discuss] Memory leak comparing communicators >> >> Hi Joost, >> >> Can you share the smallest fragment of code with which we can reproduce >> this? >> >> Antonio >> >> >> On 04/28/2014 11:25 AM, VandeVondele Joost wrote: >>> Hi, >>> >>> using mpich 3.1 I notice that my application leaks memory. Compiling it with '-O1 -g -fno-omit-frame-pointer -fsanitize=leak' using gcc 4.9 it appears the sources is comparing communicators, for example: >>> >>> [0] Direct leak of 38400 byte(s) in 600 object(s) allocated from: >>> [0] #0 0x7f84298d53a8 in __interceptor_malloc ../../../../gcc/libsanitizer/lsan/lsan_interceptors.cc:66 >>> [0] #1 0x7f842aabaf6b in MPIR_Group_create /data/vjoost/mpich-3.1/src/mpi/group/grouputil.c:77 >>> [0] #2 0x7f842ab88314 in MPIR_Comm_group_impl /data/vjoost/mpich-3.1/src/mpi/comm/comm_group.c:44 >>> [0] #3 0x7f842ab840d3 in PMPI_Comm_compare /data/vjoost/mpich-3.1/src/mpi/comm/comm_compare.c:122 >>> [0] #4 0x7f842aac8e84 in pmpi_comm_compare_ /data/vjoost/mpich-3.1/src/binding/f77/comm_comparef.c:267 >>> [0] #5 0x1bfd447 in __message_passing_MOD_mp_comm_compare /data/vjoost/clean/cp2k/cp2k/src/common/message_passing.F:1084 >>> >>> I'm not seeing other leaks, so is guess something particular is going on here. Any suggestions on a possible fix ? >>> >>> Thanks, >>> >>> Joost >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> -- >> Antonio J. Peï¿½a >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847 >> apenya(a)mcs.anl.gov >> www.mcs.anl.gov/~apenya >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Antonio J. Peï¿½a > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] Memory leak comparing communicators
by "Antonio J. Peña" 01 May '14

01 May '14

Created ticket 2080. On 05/01/2014 09:18 AM, Rob Latham wrote: > > > On 05/01/2014 09:08 AM, "Antonio J. Peï¿½a" wrote: >> >> Thank you Joost. We'll check this issue. I'll keep you posted. > > what's the trac ticket for this issue? > > ==rob > >> >> >> On 05/01/2014 09:02 AM, VandeVondele Joost wrote: >>> Hi Antonio, >>> >>> I finally reproduced this in a small program. The culprit actually >>> seems to be MPI_Group_translate_ranks, but the leak shows up with a >>> trace to MPI_Comm_compare: >>> >>> Direct leak of 3072 byte(s) in 96 object(s) allocated from: >>> #0 0x7fdab2e213a8 in __interceptor_malloc >>> ../../../../gcc/libsanitizer/lsan/lsan_interceptors.cc:66 >>> #1 0x7fdab3efaf6b in MPIR_Group_create >>> /data/vjoost/mpich-3.1/src/mpi/group/grouputil.c:77 >>> #2 0x7fdab3fc8314 in MPIR_Comm_group_impl >>> /data/vjoost/mpich-3.1/src/mpi/comm/comm_group.c:44 >>> #3 0x7fdab3fc40d3 in PMPI_Comm_compare >>> /data/vjoost/mpich-3.1/src/mpi/comm/comm_compare.c:122 >>> #4 0x4009de in main /data/vjoost/mpich-3.1/debug/2d.c:23 >>> >>>> cat 2d.c >>> #include <stdio.h> >>> #include <mpi.h> >>> >>> main(int argc, char **argv) // needs 2 ranks // >>> { >>> MPI_Comm comm_cart, dup_comm_world; >>> int result,reorder; >>> int dim[2], period[2], rin[2], rout[2]; >>> int i,j,k; >>> int g1,g2; >>> >>> MPI_Init (&argc, &argv); >>> >>> for(i=0;i<100;i++) >>> { >>> >>> MPI_Comm_dup( MPI_COMM_WORLD, &dup_comm_world); >>> dim[0]=2; dim[1]=1; >>> reorder=0; >>> period[0]=1; period[1]=1; >>> rin[0]=0; rin[1]=1; >>> MPI_Cart_create(dup_comm_world, 2, dim, period, reorder, >>> &comm_cart); >>> MPI_Comm_compare(dup_comm_world, comm_cart, &result ); >>> MPI_Comm_group(dup_comm_world,&g1); >>> MPI_Comm_group(comm_cart,&g2); >>> MPI_Group_translate_ranks(g1, 2, rin, g2, rout); >>> MPI_Comm_free(&comm_cart); >>> MPI_Comm_free(&dup_comm_world); >>> >>> } >>> >>> MPI_Finalize(); >>> >>> } >>> >>> Thanks, >>> >>> Joost >>> >>> >>> >>> >>> ________________________________________ >>> From: discuss-bounces(a)mpich.org [discuss-bounces(a)mpich.org] on behalf >>> of "Antonio J. Peï¿½a" [apenya(a)mcs.anl.gov] >>> Sent: Monday, April 28, 2014 6:26 PM >>> To: discuss(a)mpich.org >>> Subject: Re: [mpich-discuss] Memory leak comparing communicators >>> >>> Hi Joost, >>> >>> Can you share the smallest fragment of code with which we can reproduce >>> this? >>> >>> Antonio >>> >>> >>> On 04/28/2014 11:25 AM, VandeVondele Joost wrote: >>>> Hi, >>>> >>>> using mpich 3.1 I notice that my application leaks memory. Compiling >>>> it with '-O1 -g -fno-omit-frame-pointer -fsanitize=leak' using gcc >>>> 4.9 it appears the sources is comparing communicators, for example: >>>> >>>> [0] Direct leak of 38400 byte(s) in 600 object(s) allocated from: >>>> [0] #0 0x7f84298d53a8 in __interceptor_malloc >>>> ../../../../gcc/libsanitizer/lsan/lsan_interceptors.cc:66 >>>> [0] #1 0x7f842aabaf6b in MPIR_Group_create >>>> /data/vjoost/mpich-3.1/src/mpi/group/grouputil.c:77 >>>> [0] #2 0x7f842ab88314 in MPIR_Comm_group_impl >>>> /data/vjoost/mpich-3.1/src/mpi/comm/comm_group.c:44 >>>> [0] #3 0x7f842ab840d3 in PMPI_Comm_compare >>>> /data/vjoost/mpich-3.1/src/mpi/comm/comm_compare.c:122 >>>> [0] #4 0x7f842aac8e84 in pmpi_comm_compare_ >>>> /data/vjoost/mpich-3.1/src/binding/f77/comm_comparef.c:267 >>>> [0] #5 0x1bfd447 in __message_passing_MOD_mp_comm_compare >>>> /data/vjoost/clean/cp2k/cp2k/src/common/message_passing.F:1084 >>>> >>>> I'm not seeing other leaks, so is guess something particular is going >>>> on here. Any suggestions on a possible fix ? >>>> >>>> Thanks, >>>> >>>> Joost >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> -- >>> Antonio J. Peï¿½a >>> Postdoctoral Appointee >>> Mathematics and Computer Science Division >>> Argonne National Laboratory >>> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >>> Argonne, IL 60439-4847 >>> apenya(a)mcs.anl.gov >>> www.mcs.anl.gov/~apenya >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> > -- Antonio J. Peï¿½a Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya

1 0

Re: [mpich-discuss] Getting an undeclared error on compile, with mpicc
by mark 01 May '14

01 May '14

Well, it was the declaration of the mpi.h . It wasn't there. Thanks for the instant support Ken, i appreciate it. Στις 01/05/2014 05:56 μμ, ο/η Kenneth Raffenetti έγραψε: > How was MPICH installed? Are you including the right mpi.h header in > your program? Have you tried compiling and running the cpi.c example > that comes with MPICH? > > If you send us the smallest possible program that can reproduce this > error, it would help us get to the bottom of the issue. > > Ken > > On 05/01/2014 09:49 AM, mark wrote: >> Hello. >> I am getting an error that i wasn't getting before. >> >> ------------------------------------------------------------------------ >> test.c: In function 'main': >> test.c:63: error: 'MPI_COMM_WORLD' undeclared (first use in this >> function) >> test.c:63: error: (Each undeclared identifier is reported only once >> test.c:63: error: for each function it appears in.) >> test.c:111: error: 'MPI_INT' undeclared (first use in this function) >> ------------------------------------------------------------------------ >> line 63 : MPI_Comm_size(MPI_COMM_WORLD, &numnodes); >> line 111:MPI_Scatter( array, PARTS, MPI_INT, my_array, PARTS, MPI_INT, >> master, MPI_COMM_WORLD); >> >> >> I am under a 2.6.23-gentoo-r5 cluster with gcc version 4.2.2 (Gentoo >> 4.2.2 p1.0) >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0