- devel - mailman3.cels.anl.gov

odd timings in type create/free (related to handle pool?)
by Jeff Hammond 03 Oct '14

03 Oct '14

I wanted to time how long it took to create a datatype. Obviously, timing a series of calls is the normal way to get reasonable data. However, I find that my test shows that the time per call is in some way proportional to the number of calls in the series, even when I reuse the same handle. I previously timed on a vector of handles and saw the same result. I can only assume this is related to how MPICH does handle allocation internally. Can you confirm? Is there any way to get MPICH to garbage collect the internal handle pool so that the time per call goes back down again? An increase from 4 us to 112 us per call is pretty substantial if I have a library that is going to use a lot of derived datatypes and has no reasonable way to cache them. Thanks, Jeff OUTPUT create, commit (free?) 100 Type_contig_x in 0.000393 s (3.926220 us per call) create, commit (free?) 1000 Type_contig_x in 0.006105 s (6.105444 us per call) create, commit (free?) 10000 Type_contig_x in 0.082496 s (8.249623 us per call) create, commit (free?) 25000 Type_contig_x in 0.341852 s (13.674085 us per call) create, commit (free?) 50000 Type_contig_x in 1.280882 s (25.617630 us per call) create, commit (free?) 100000 Type_contig_x in 4.565911 s (45.659108 us per call) create, commit (free?) 250000 Type_contig_x in 27.989672 s (111.958686 us per call) SOURCE #include <stdint.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <limits.h> #include <math.h> #include <mpi.h> /* there is a reason for this silliness */ static volatile int bigmpi_int_max = 3; int MPIX_Type_contiguous_x(MPI_Count count, MPI_Datatype oldtype, MPI_Datatype * newtype) { MPI_Count c = count/bigmpi_int_max; MPI_Count r = count%bigmpi_int_max; MPI_Datatype chunks; MPI_Type_vector(c, bigmpi_int_max, bigmpi_int_max, oldtype, &chunks); MPI_Datatype remainder; MPI_Type_contiguous(r, oldtype, &remainder); MPI_Aint lb /* unused */, extent; MPI_Type_get_extent(oldtype, &lb, &extent); MPI_Aint remdisp = (MPI_Aint)c*bigmpi_int_max*extent; int blocklengths[2] = {1,1}; MPI_Aint displacements[2] = {0,remdisp}; MPI_Datatype types[2] = {chunks,remainder}; MPI_Type_create_struct(2, blocklengths, displacements, types, newtype); MPI_Type_free(&chunks); MPI_Type_free(&remainder); return MPI_SUCCESS; } int main(int argc, char* argv[]) { int rank=0, size=1; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); int n = (argc>1) ? atoi(argv[1]) : 10000; //MPI_Datatype * dtout = malloc(n*sizeof(MPI_Datatype)); MPI_Datatype dtout; double t0 = MPI_Wtime(); for (int i=0; i<n; i++) { //MPIX_Type_contiguous_x((MPI_Count)i, MPI_DOUBLE, &(dtout[i])); //MPI_Type_commit(&(dtout[i])); MPIX_Type_contiguous_x((MPI_Count)i, MPI_DOUBLE, &dtout); MPI_Type_commit(&dtout); MPI_Type_free(&dtout); } double t1 = MPI_Wtime(); double dt = t1-t0; printf("create, commit (free?) %d Type_contig_x in %lf s (%lf us per call)\n", n, dt, 1.e6*dt/(double)n); //for (int i=0; i<n; i++) { // MPI_Type_free(&(dtout[i])); //} //free(dtout); MPI_Finalize(); return 0; } -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] [mpich-commits] [mpich] MPICH primary repository branch, master, updated. v3.2a1-34-geda105b
by Dave Goodell (dgoodell) 02 Oct '14

02 Oct '14

On Oct 2, 2014, at 7:36 AM, William Gropp <wgropp(a)illinois.edu> wrote: > There are two points here: > > [... C89 / compiler warning screed here ...] I'm not going to waste our time debating this with you, as we both know we are highly unlikely to change the other's views on this issue. I think that achieving zero warnings on common compilers in strict compilation mode is important and more valuable than certain other things, and you don't. It's largely a value judgement either direction, which doesn't make either of us right in any absolute sense. > 2) Language version choices and coding standards. The real point here is that coding standards are important and should be followed by anyone working on the code. Changes to the coding standard need to be made collectively and explicitly; individuals must not decide that they know better. On this specific question, it may very well be time to move to C99, but that must be a deliberate, collective choice. And the issue is not whether such compilers exist, but whether our user community uses and trusts them. At this point, the answer to that question is probably yes as well, but that’s the correct question to ask (and yes, a deliberate and explicit decision could be made to require a C99 compiler and exclude any users that insist on sticking with C89). > > Junchao made the right call. I was not asking Junchao to go off and make a bunch of cowboy changes, but for him to push for modernization in a collective decision instead of accepting the C89 coding standards as dogma. I just said it in a terse, semi-inflammatory style that could be misunderstood. Perhaps it's time to take my iconoclastic views and go back to full-on lurking on this list. -Dave

1 0

Re: [mpich-devel] [mpich-commits] [mpich] MPICH primary repository branch, master, updated. v3.2a1-34-geda105b
by William Gropp 02 Oct '14

02 Oct '14

There are two points here: 1) Should the code compile “cleanly” with every compiler and many choices of warning options? This sounds like a good idea on the surface, but in fact it is not. Lots of compilers generate too many false positives, and suppressing them often requires adding unnecessary code and distorting the rest of the code. That is the case here (the same compiler that can’t figure out that the array isn’t referenced before use is going to have to perform the initialization, adding unnecessary overhead and thus latency). Rather, the code should compile cleanly with some compiler and set of warnings that is accurate and effective. The fact that some compiler has poor accuracy in its warnings should not be a reason to harm the code just to keep that compiler quiet. Note that this is more important for performance sensitive code (particularly for latency) such as MPICH; however, even for non-performance-sensitive codes, making the compiler happy can distort the code, making it harder, not easier, to maintain. 2) Language version choices and coding standards. The real point here is that coding standards are important and should be followed by anyone working on the code. Changes to the coding standard need to be made collectively and explicitly; individuals must not decide that they know better. On this specific question, it may very well be time to move to C99, but that must be a deliberate, collective choice. And the issue is not whether such compilers exist, but whether our user community uses and trusts them. At this point, the answer to that question is probably yes as well, but that’s the correct question to ask (and yes, a deliberate and explicit decision could be made to require a C99 compiler and exclude any users that insist on sticking with C89). Junchao made the right call. Bill On Oct 1, 2014, at 11:15 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: > On Oct 1, 2014, at 10:20 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > >> On Wed, Oct 1, 2014 at 5:40 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> >> wrote: >> >>> Begin forwarded message: >>> >>>> - Log ----------------------------------------------------------------- >>>> >>> http://git.mpich.org/mpich.git/commitdiff/eda105b029572ebf7f4008f6de2e1afb9… >>>> >>>> commit eda105b029572ebf7f4008f6de2e1afb9ccf6fc2 >>>> Author: Junchao Zhang <jczhang(a)mcs.anl.gov> >>>> Date: Tue Sep 16 15:58:55 2014 -0500 >>>> >>>> Remove unnessary array init in MPIU_CHKLMEM_DECL >>>> >>>> Initializing mpiu_chklmem_stk_sp_ to 0 should already have the effect >>> to >>>> initialize mpiu_chklmem_stk_[n_]. >>>> >>>> Signed-off-by: Antonio J. Pena <apenya(a)mcs.anl.gov> >>>> >>>> diff --git a/src/include/mpimem.h b/src/include/mpimem.h >>>> index a217e8b..e1478a5 100644 >>>> --- a/src/include/mpimem.h >>>> +++ b/src/include/mpimem.h >>>> @@ -317,7 +317,7 @@ extern char *strdup( const char * ); >>>> }} >>>> #else >>>> #define MPIU_CHKLMEM_DECL(n_) \ >>>> - void *(mpiu_chklmem_stk_[n_]) = {0};\ >>>> + void *(mpiu_chklmem_stk_[n_]); \ >>>> int mpiu_chklmem_stk_sp_=0;\ >>>> MPIU_AssertDeclValue(const int mpiu_chklmem_stk_sz_,n_) >>> >>> You might want to revert this. My recollection is that not all compilers >>> are smart enough to see that mpiu_chklmem_stk_ is not used before being >>> defined, which can lead to a *lot* of warnings throughout the MPICH >>> codebase. >>> >> >> Dave, I don't quite understand your concern. I removed the array >> initialization code since MPIU_CHKLMEM_FREEALL() can only free pointers set >> before. The number of pointers is mpiu_chklmem_stk_sp_, which is >> initialized to 0. > > Look, I don't have time (or frankly, that much interest) to run down an example compiler where this will happen, but I definitely made this change in the past because we were encountering the warning: http://git.mpich.org/mpich.git/commitdiff/70f4742 > > I wish I had made a better commit message at the time :-/ > > I have a suspicion that GCC ~4.1 with some set of (non-?)optimization flags would issue the "may be used uninitialized" warning, but I really don't recall for sure. I don't disagree with your analysis that the value cannot actually be used uninitialized, I'm just disputing whether the compiler will issue a warning about it. > > Actually, it looks like I'm not the only person who has experienced this, Toni fixed an issue like this much more recently than 2009: http://git.mpich.org/mpich.git/commitdiff/d8b8eb6 > >>> Also, what's the sudden interest in restoring C89 support again? The >>> whole world really does support enough of C99 now... Seems like a big >>> waste of time and an unnecessary annoyance to developers who might want to >>> use useful C99 features (e.g., named structure initializers). >>> >>> >> See https://trac.mpich.org/projects/mpich/ticket/2167 > > > That's not really an answer. It's just a rehash of the endless discussion that MPICH development has had for the past 7+ years. > > Open MPI and many other projects now require nominal C99 compliance for any recent release. All the work that you did to run around and fix the "--enable-strict=c89" mode was just a huge waste of time. The correct fix is to delete "--enable-strict=c89" and embrace the "future" of 6+ years ago. > > -Dave > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0

Re: [mpich-devel] [mpich-commits] [mpich] MPICH primary repository branch, master, updated. v3.2a1-34-geda105b
by Dave Goodell (dgoodell) 01 Oct '14

01 Oct '14

On Oct 1, 2014, at 10:20 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > On Wed, Oct 1, 2014 at 5:40 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> > wrote: > >> Begin forwarded message: >> >>> - Log ----------------------------------------------------------------- >>> >> http://git.mpich.org/mpich.git/commitdiff/eda105b029572ebf7f4008f6de2e1afb9… >>> >>> commit eda105b029572ebf7f4008f6de2e1afb9ccf6fc2 >>> Author: Junchao Zhang <jczhang(a)mcs.anl.gov> >>> Date: Tue Sep 16 15:58:55 2014 -0500 >>> >>> Remove unnessary array init in MPIU_CHKLMEM_DECL >>> >>> Initializing mpiu_chklmem_stk_sp_ to 0 should already have the effect >> to >>> initialize mpiu_chklmem_stk_[n_]. >>> >>> Signed-off-by: Antonio J. Pena <apenya(a)mcs.anl.gov> >>> >>> diff --git a/src/include/mpimem.h b/src/include/mpimem.h >>> index a217e8b..e1478a5 100644 >>> --- a/src/include/mpimem.h >>> +++ b/src/include/mpimem.h >>> @@ -317,7 +317,7 @@ extern char *strdup( const char * ); >>> }} >>> #else >>> #define MPIU_CHKLMEM_DECL(n_) \ >>> - void *(mpiu_chklmem_stk_[n_]) = {0};\ >>> + void *(mpiu_chklmem_stk_[n_]); \ >>> int mpiu_chklmem_stk_sp_=0;\ >>> MPIU_AssertDeclValue(const int mpiu_chklmem_stk_sz_,n_) >> >> You might want to revert this. My recollection is that not all compilers >> are smart enough to see that mpiu_chklmem_stk_ is not used before being >> defined, which can lead to a *lot* of warnings throughout the MPICH >> codebase. >> > > Dave, I don't quite understand your concern. I removed the array > initialization code since MPIU_CHKLMEM_FREEALL() can only free pointers set > before. The number of pointers is mpiu_chklmem_stk_sp_, which is > initialized to 0. Look, I don't have time (or frankly, that much interest) to run down an example compiler where this will happen, but I definitely made this change in the past because we were encountering the warning: http://git.mpich.org/mpich.git/commitdiff/70f4742 I wish I had made a better commit message at the time :-/ I have a suspicion that GCC ~4.1 with some set of (non-?)optimization flags would issue the "may be used uninitialized" warning, but I really don't recall for sure. I don't disagree with your analysis that the value cannot actually be used uninitialized, I'm just disputing whether the compiler will issue a warning about it. Actually, it looks like I'm not the only person who has experienced this, Toni fixed an issue like this much more recently than 2009: http://git.mpich.org/mpich.git/commitdiff/d8b8eb6 >> Also, what's the sudden interest in restoring C89 support again? The >> whole world really does support enough of C99 now... Seems like a big >> waste of time and an unnecessary annoyance to developers who might want to >> use useful C99 features (e.g., named structure initializers). >> >> > See https://trac.mpich.org/projects/mpich/ticket/2167 That's not really an answer. It's just a rehash of the endless discussion that MPICH development has had for the past 7+ years. Open MPI and many other projects now require nominal C99 compliance for any recent release. All the work that you did to run around and fix the "--enable-strict=c89" mode was just a huge waste of time. The correct fix is to delete "--enable-strict=c89" and embrace the "future" of 6+ years ago. -Dave

1 0

Re: [mpich-devel] Fwd: [mpich-commits] [mpich] MPICH primary repository branch, master, updated. v3.2a1-34-geda105b
by Junchao Zhang 01 Oct '14

01 Oct '14

On Wed, Oct 1, 2014 at 5:40 PM, Dave Goodell (dgoodell) <dgoodell(a)cisco.com> wrote: > Begin forwarded message: > > > From: Service Account <noreply(a)mpich.org> > > Subject: [mpich-commits] [mpich] MPICH primary repository branch, > master, updated. v3.2a1-34-geda105b > > Date: October 1, 2014 5:29:25 PM CDT > > To: <all-commits(a)mpich.org>, <commits(a)mpich.org> > > Reply-To: <commits(a)mpich.org> > > > > This is an automated email from the git hooks/post-receive script. It was > > generated because a ref change was pushed to the repository containing > > the project "MPICH primary repository". > > > > The branch, master has been updated > > via eda105b029572ebf7f4008f6de2e1afb9ccf6fc2 (commit) > > via 98114df4c0504568ab188ea98a9d1269d087bd82 (commit) > > via ad25799865749680338c05264c8007242f01326a (commit) > > from 6b8c426d90b28bb0ed6ff6ade6ac03a4e4b96634 (commit) > > > > Those revisions listed above that are new to this repository have > > not appeared on any other notification email; so we list those > > revisions in full, below. > > > > - Log ----------------------------------------------------------------- > > > http://git.mpich.org/mpich.git/commitdiff/eda105b029572ebf7f4008f6de2e1afb9… > > > > commit eda105b029572ebf7f4008f6de2e1afb9ccf6fc2 > > Author: Junchao Zhang <jczhang(a)mcs.anl.gov> > > Date: Tue Sep 16 15:58:55 2014 -0500 > > > > Remove unnessary array init in MPIU_CHKLMEM_DECL > > > > Initializing mpiu_chklmem_stk_sp_ to 0 should already have the effect > to > > initialize mpiu_chklmem_stk_[n_]. > > > > Signed-off-by: Antonio J. Pena <apenya(a)mcs.anl.gov> > > > > diff --git a/src/include/mpimem.h b/src/include/mpimem.h > > index a217e8b..e1478a5 100644 > > --- a/src/include/mpimem.h > > +++ b/src/include/mpimem.h > > @@ -317,7 +317,7 @@ extern char *strdup( const char * ); > > }} > > #else > > #define MPIU_CHKLMEM_DECL(n_) \ > > - void *(mpiu_chklmem_stk_[n_]) = {0};\ > > + void *(mpiu_chklmem_stk_[n_]); \ > > int mpiu_chklmem_stk_sp_=0;\ > > MPIU_AssertDeclValue(const int mpiu_chklmem_stk_sz_,n_) > > You might want to revert this. My recollection is that not all compilers > are smart enough to see that mpiu_chklmem_stk_ is not used before being > defined, which can lead to a *lot* of warnings throughout the MPICH > codebase. > Dave, I don't quite understand your concern. I removed the array initialization code since MPIU_CHKLMEM_FREEALL() can only free pointers set before. The number of pointers is mpiu_chklmem_stk_sp_, which is initialized to 0. > > Also, what's the sudden interest in restoring C89 support again? The > whole world really does support enough of C99 now... Seems like a big > waste of time and an unnecessary annoyance to developers who might want to > use useful C99 features (e.g., named structure initializers). > > See https://trac.mpich.org/projects/mpich/ticket/2167 > -Dave > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Fwd: [mpich-commits] [mpich] MPICH primary repository branch, master, updated. v3.2a1-34-geda105b
by Dave Goodell (dgoodell) 01 Oct '14

01 Oct '14

Begin forwarded message: > From: Service Account <noreply(a)mpich.org> > Subject: [mpich-commits] [mpich] MPICH primary repository branch, master, updated. v3.2a1-34-geda105b > Date: October 1, 2014 5:29:25 PM CDT > To: <all-commits(a)mpich.org>, <commits(a)mpich.org> > Reply-To: <commits(a)mpich.org> > > This is an automated email from the git hooks/post-receive script. It was > generated because a ref change was pushed to the repository containing > the project "MPICH primary repository". > > The branch, master has been updated > via eda105b029572ebf7f4008f6de2e1afb9ccf6fc2 (commit) > via 98114df4c0504568ab188ea98a9d1269d087bd82 (commit) > via ad25799865749680338c05264c8007242f01326a (commit) > from 6b8c426d90b28bb0ed6ff6ade6ac03a4e4b96634 (commit) > > Those revisions listed above that are new to this repository have > not appeared on any other notification email; so we list those > revisions in full, below. > > - Log ----------------------------------------------------------------- > http://git.mpich.org/mpich.git/commitdiff/eda105b029572ebf7f4008f6de2e1afb9… > > commit eda105b029572ebf7f4008f6de2e1afb9ccf6fc2 > Author: Junchao Zhang <jczhang(a)mcs.anl.gov> > Date: Tue Sep 16 15:58:55 2014 -0500 > > Remove unnessary array init in MPIU_CHKLMEM_DECL > > Initializing mpiu_chklmem_stk_sp_ to 0 should already have the effect to > initialize mpiu_chklmem_stk_[n_]. > > Signed-off-by: Antonio J. Pena <apenya(a)mcs.anl.gov> > > diff --git a/src/include/mpimem.h b/src/include/mpimem.h > index a217e8b..e1478a5 100644 > --- a/src/include/mpimem.h > +++ b/src/include/mpimem.h > @@ -317,7 +317,7 @@ extern char *strdup( const char * ); > }} > #else > #define MPIU_CHKLMEM_DECL(n_) \ > - void *(mpiu_chklmem_stk_[n_]) = {0};\ > + void *(mpiu_chklmem_stk_[n_]); \ > int mpiu_chklmem_stk_sp_=0;\ > MPIU_AssertDeclValue(const int mpiu_chklmem_stk_sz_,n_) You might want to revert this. My recollection is that not all compilers are smart enough to see that mpiu_chklmem_stk_ is not used before being defined, which can lead to a *lot* of warnings throughout the MPICH codebase. Also, what's the sudden interest in restoring C89 support again? The whole world really does support enough of C99 now... Seems like a big waste of time and an unnecessary annoyance to developers who might want to use useful C99 features (e.g., named structure initializers). -Dave

1 0

lockcontention3.c bug
by Archer, Charles J 24 Sep '14

24 Sep '14

I think I found a bug in lockcontention3.c: Line 126, MPI_Win_create creates a window of size RMA_SIZE (2048 bytes). Then, in case 9 of RMATest (line 249): MPI_Put( source, longcount, MPI_INT, master, OFFSET_1, longcount, MPI_INT, win ); longcount is 512 OFFSET_1 is 7 longcount * sizeof(int) + OFFSET_1 is > 2048, and we have a buffer overrun. I think on some platforms this test case will still work since the memory we are talking about is less than a page. I didn’t check the other tests for the same problem yet. Here’s a fix: diff --git a/test/mpi/rma/lockcontention3.c b/test/mpi/rma/lockcontention3.c index d70e26c..f42ff62 100644 --- a/test/mpi/rma/lockcontention3.c +++ b/test/mpi/rma/lockcontention3.c @@ -123,7 +123,7 @@ int main( int argc, char *argv[] ) srcbuf = malloc(RMA_SIZE*sizeof(*srcbuf)); assert(srcbuf); - MPI_Win_create( rmabuffer, bufsize, sizeof(int), MPI_INFO_NULL, + MPI_Win_create( rmabuffer, bufsize*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &win ); /* Run a sequence of tests */

1 0

Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers
by Junchao Zhang 22 Sep '14

22 Sep '14

Forward new messages from IBM: * If you set PAMID_COLLECTIVES_MEMORY_OPTIMIZED=1 then the pami optimized collectives are disabled for that comm and the geometry / collective network is not created at comm creation time if the block is irregular, so the comm dup/split performance is good. So for now the user can go ahead and use it, and if there are problems with it we'll need to decide then if we will support it or look at what code changes would be necessary to address the underlying issue. * This whole 'power-of-2' thing was basically too loose a definition for an irregular block, for Blue Gene /Q at the moment there is no crystal-clear formal definition, but a tighter one would be a block that is fully populated with every node in all 5 dimensions, with the same number of MPI ranks on each node --- there are still some exceptions and caveats beyond this that can only be answered by looking at the code, but this is the tightest and simplest definition I can come up with at the moment. On Mon, Sep 22, 2014 at 10:51 AM, Sam Williams <swwilliams(a)lbl.gov> wrote: > All of Mira (48k) is technically not a power of two. > > I tried the pami option at one point. I don't recall it giving better > performance > > - Sam > > On Sep 22, 2014, at 8:39 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > > Sam, > I had some updates from IBM last week. They reproduced the problem and > found it only happens when the number of MPI ranks is non-power-of-2. > Their advice is that since the IBM BG/Q optimized collectives themselves > are mostly designed only to be helpful for blocks with power-of-2 > geometries, you can try in your program to see if subsequent collective > calls with PAMID_COLLECTIVES=1 are actually faster than PAMID_COLLECTIVES=0 > on comms with a non-power-of-2 geometry. If the answer is no, then you > can just run with PAMID_COLLECTIVES=0 and avoid the dup/split performance > issue. Otherwise, IBM may prioritize this ticket. > > Thanks. > --Junchao Zhang > > On Thu, Jul 3, 2014 at 4:41 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > >> Hi, Sam, >> I wrote micro-benchmarks for MPI_Comm_split/dup. My profiling results >> suggested the problem lies in a IBM PAMI library call, >> PAMI_Geometry_create_taskrange(). Unfortunately, I don't have access to >> the PAMI source code and don't know why. I reported it to IBM and hope IBM >> will fix it. >> Alternatively, you can set an environment variable PAMID_COLLECTIVES=0 >> to disables pami collectives. My tests showed it at least fixed the >> scalability problem of Comm_split and Comm_dup. >> Also through profiling, I found the qsort() called in MPICH code is >> actually using the merge sort algorithm in Mira's libc library. >> >> >> >> --Junchao Zhang >> >> >> On Sat, May 17, 2014 at 9:06 AM, Sam Williams <swwilliams(a)lbl.gov> wrote: >> >>> I've been conducting scaling experiments on the Mira (Blue Gene/Q) and K >>> (Sparc) supercomputers. I've noticed that the time required for >>> MPI_Comm_split and MPI_Comm_dup can grow quickly with scale (~P^2). As >>> such, its performance eventually becomes a bottleneck. That is, although >>> the benefit of using a subcommunicator is huge (multigrid solves are >>> weak-scalable), the penalty of creating one (multigrid build time) is also >>> huge. >>> >>> For example, when scaling from 1 to 46K nodes (= cubes of integers) on >>> Mira, the time (in seconds) required to build a MG solver (including a >>> subcommunicator) scales as >>> 222335.output: Total time in MGBuild 0.056704 >>> 222336.output: Total time in MGBuild 0.060834 >>> 222348.output: Total time in MGBuild 0.064782 >>> 222349.output: Total time in MGBuild 0.090229 >>> 222350.output: Total time in MGBuild 0.075280 >>> 222351.output: Total time in MGBuild 0.091852 >>> 222352.output: Total time in MGBuild 0.137299 >>> 222411.output: Total time in MGBuild 0.301552 >>> 222413.output: Total time in MGBuild 0.606444 >>> 222415.output: Total time in MGBuild 0.745272 >>> 222417.output: Total time in MGBuild 0.779757 >>> 222418.output: Total time in MGBuild 4.671838 >>> 222419.output: Total time in MGBuild 15.123162 >>> 222420.output: Total time in MGBuild 33.875626 >>> 222421.output: Total time in MGBuild 49.494547 >>> 222422.output: Total time in MGBuild 151.329026 >>> >>> If I disable the call to MPI_Comm_Split, my time scales as >>> 224982.output: Total time in MGBuild 0.050143 >>> 224983.output: Total time in MGBuild 0.052607 >>> 224988.output: Total time in MGBuild 0.050697 >>> 224989.output: Total time in MGBuild 0.078343 >>> 224990.output: Total time in MGBuild 0.054634 >>> 224991.output: Total time in MGBuild 0.052158 >>> 224992.output: Total time in MGBuild 0.060286 >>> 225008.output: Total time in MGBuild 0.062925 >>> 225009.output: Total time in MGBuild 0.097357 >>> 225010.output: Total time in MGBuild 0.061807 >>> 225011.output: Total time in MGBuild 0.076617 >>> 225012.output: Total time in MGBuild 0.099683 >>> 225013.output: Total time in MGBuild 0.125580 >>> 225014.output: Total time in MGBuild 0.190711 >>> 225016.output: Total time in MGBuild 0.218329 >>> 225017.output: Total time in MGBuild 0.282081 >>> >>> Although I didn't directly measure it, this suggests the time for >>> MPI_Comm_Split is growing roughly quadratically with process concurrency. >>> >>> >>> >>> >>> I see the same effect on the K machine (8...64K nodes) where the code >>> uses comm_split/dup in conjunction: >>> run00008_7_1.sh.o2412931: Total time in MGBuild 0.026458 seconds >>> run00064_7_1.sh.o2415876: Total time in MGBuild 0.039121 seconds >>> run00512_7_1.sh.o2415877: Total time in MGBuild 0.086800 seconds >>> run01000_7_1.sh.o2414496: Total time in MGBuild 0.129764 seconds >>> run01728_7_1.sh.o2415878: Total time in MGBuild 0.224576 seconds >>> run04096_7_1.sh.o2415880: Total time in MGBuild 0.738979 seconds >>> run08000_7_1.sh.o2414504: Total time in MGBuild 2.123800 seconds >>> run13824_7_1.sh.o2415881: Total time in MGBuild 6.276573 seconds >>> run21952_7_1.sh.o2415882: Total time in MGBuild 13.634200 seconds >>> run32768_7_1.sh.o2415884: Total time in MGBuild 36.508670 seconds >>> run46656_7_1.sh.o2415874: Total time in MGBuild 58.668228 seconds >>> run64000_7_1.sh.o2415875: Total time in MGBuild 117.322217 seconds >>> >>> >>> A glance at the implementation on Mira (I don't know if the >>> implementation on K is stock) suggests it should be using qsort to sort >>> based on keys. Unfortunately, qsort is not performance robust like >>> heap/merge sort. If one were to be productive and call comm_split like... >>> MPI_Comm_split(...,mycolor,myrank,...) >>> then one runs the risk that the keys are presorted. This hits the worst >>> case computational complexity for qsort... O(P^2). Demanding programmers >>> avoid sending sorted keys seems unreasonable. >>> >>> >>> I should note, I see a similar lack of scaling with MPI_Comm_dup on the >>> K machine. Unfortunately, my BGQ data used an earlier version of the code >>> that did not use comm_dup. As such, I can’t definitively say that it is a >>> problem on that machine as well. >>> >>> Thus, I'm asking for scalable implementations of comm_split/dup using >>> merge/heap sort whose worst case complexity is still PlogP to be >>> prioritized in the next update. >>> >>> >>> thanks >>> _______________________________________________ >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/devel >>> >> >> >

1 0

Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers
by Michael Blocksome 22 Sep '14

22 Sep '14

Thanks Bob! Here's the code snippet in question: $ git blame src/mpid/pamid/src/comm/mpid_comm.c ... 77999d6f (Bob Cernohous 2013-03-18 14:51:05 -0500 291) if((MPIDI_Process.optimized.memory & MPID_OPT_LVL_IRREG) && (comm->local_size & (comm->local_size-1))) 53f6e934 (Haizhu Liu 2012-11-07 20:56:27 -0500 292) { 858da8da (Bob Cernohous 2013-02-14 13:37:36 -0600 293) /* Don't create irregular geometries. Fallback to MPICH only collectives */ 858da8da (Bob Cernohous 2013-02-14 13:37:36 -0600 294) geom_init = 0; 224dfb1b (Bob Cernohous 2013-03-06 13:14:47 -0600 295) comm->mpid.geometry = PAMI_GEOMETRY_NULL; 63577b28 (Bob Cernohous 2013-02-07 10:21:02 -0600 296) } ... The environment variable "PAMID_COLLECTIVES_MEMORY_OPTIMIZED=1" will enable this code. Here is the documentation: $ git blame src/mpid/pamid/src/mpidi_env.c ... 63577b28 (Bob Cernohous 2013-02-07 10:21:02 -0600 111) * - PAMID_COLLECTIVES_MEMORY_OPTIMIZED - Controls whether collectives are 63577b28 (Bob Cernohous 2013-02-07 10:21:02 -0600 112) * optimized to reduce memory usage. This may disable some PAMI collectives. 63577b28 (Bob Cernohous 2013-02-07 10:21:02 -0600 113) * Possible values: 63577b28 (Bob Cernohous 2013-02-07 10:21:02 -0600 114) * - 0 - Collectives are not memory optimized. 77999d6f (Bob Cernohous 2013-03-18 14:51:05 -0500 115) * - n - Collectives are memory optimized. Levels are bitwise values : 77999d6f (Bob Cernohous 2013-03-18 14:51:05 -0500 116) * MPID_OPT_LVL_IRREG = 1, Do not optimize irregular communicators 77999d6f (Bob Cernohous 2013-03-18 14:51:05 -0500 117) * MPID_OPT_LVL_NONCONTIG = 2, Disable some non-contig collectives ... Michael Blocksome Parallel Environment MPI Middleware POWER, x86, and Blue Gene HPC Messaging blocksom(a)us.ibm.com From: Bob Cernohous <bcernohous(a)cray.com> To: "devel(a)mpich.org" <devel(a)mpich.org> Date: 09/22/2014 01:18 PM Subject: Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers Sent by: devel-bounces(a)mpich.org I thought there was a "memory" optimization that disabled PAMI on irregular communicators. However I don't know the current state of that code. if(MPIDI_Process.optimized.memory && (comm->local_size & (comm->local_size-1))) { /* Don't create irregular geometries. Fallback to MPICH only collectives */ geom_init = 0; comm->mpid.geometry = NULL; } > -----Original Message----- > From: devel-bounces(a)mpich.org [mailto:[email protected]] On > Behalf Of Jeff Hammond > Sent: Monday, September 22, 2014 12:16 PM > To: devel(a)mpich.org > Subject: Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K > supercomputers > > PAMID_COLLECTIVES=0 is really bad for performance. IBM should figure out a > way to disable it on a per-communicator basis when MPI_COMM_SPLIT is > going to have issues. I recall they allow one to "unoptimize" a communicator > but I thought that was only possible after it was created. > > Jeff > > On Mon, Sep 22, 2014 at 8:39 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > > Sam, > > I had some updates from IBM last week. They reproduced the problem > > and found it only happens when the number of MPI ranks is > > non-power-of-2. Their advice is that since the IBM BG/Q optimized > > collectives themselves are mostly designed only to be helpful for > > blocks with power-of-2 geometries, you can try in your program to see > > if subsequent collective calls with > > PAMID_COLLECTIVES=1 are actually faster than PAMID_COLLECTIVES=0 on > > comms with a non-power-of-2 geometry. If the answer is no, then you > > can just run with PAMID_COLLECTIVES=0 and avoid the dup/split > performance issue. > > Otherwise, IBM may prioritize this ticket. > > > > Thanks. > > --Junchao Zhang > > > > On Thu, Jul 3, 2014 at 4:41 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > >> > >> Hi, Sam, > >> I wrote micro-benchmarks for MPI_Comm_split/dup. My profiling > >> results suggested the problem lies in a IBM PAMI library call, > >> PAMI_Geometry_create_taskrange(). Unfortunately, I don't have access > >> to the PAMI source code and don't know why. I reported it to IBM and > >> hope IBM will fix it. > >> Alternatively, you can set an environment variable > >> PAMID_COLLECTIVES=0 to disables pami collectives. My tests showed it > >> at least fixed the scalability problem of Comm_split and Comm_dup. > >> Also through profiling, I found the qsort() called in MPICH code is > >> actually using the merge sort algorithm in Mira's libc library. > >> > >> > >> > >> --Junchao Zhang > >> > >> > >> On Sat, May 17, 2014 at 9:06 AM, Sam Williams <swwilliams(a)lbl.gov> > wrote: > >>> > >>> I've been conducting scaling experiments on the Mira (Blue Gene/Q) > >>> and K > >>> (Sparc) supercomputers. I've noticed that the time required for > >>> MPI_Comm_split and MPI_Comm_dup can grow quickly with scale > (~P^2). > >>> As such, its performance eventually becomes a bottleneck. That is, > >>> although the benefit of using a subcommunicator is huge (multigrid > >>> solves are weak-scalable), the penalty of creating one (multigrid > >>> build time) is also huge. > >>> > >>> For example, when scaling from 1 to 46K nodes (= cubes of integers) > >>> on Mira, the time (in seconds) required to build a MG solver > >>> (including a > >>> subcommunicator) scales as > >>> 222335.output: Total time in MGBuild 0.056704 > >>> 222336.output: Total time in MGBuild 0.060834 > >>> 222348.output: Total time in MGBuild 0.064782 > >>> 222349.output: Total time in MGBuild 0.090229 > >>> 222350.output: Total time in MGBuild 0.075280 > >>> 222351.output: Total time in MGBuild 0.091852 > >>> 222352.output: Total time in MGBuild 0.137299 > >>> 222411.output: Total time in MGBuild 0.301552 > >>> 222413.output: Total time in MGBuild 0.606444 > >>> 222415.output: Total time in MGBuild 0.745272 > >>> 222417.output: Total time in MGBuild 0.779757 > >>> 222418.output: Total time in MGBuild 4.671838 > >>> 222419.output: Total time in MGBuild 15.123162 > >>> 222420.output: Total time in MGBuild 33.875626 > >>> 222421.output: Total time in MGBuild 49.494547 > >>> 222422.output: Total time in MGBuild 151.329026 > >>> > >>> If I disable the call to MPI_Comm_Split, my time scales as > >>> 224982.output: Total time in MGBuild 0.050143 > >>> 224983.output: Total time in MGBuild 0.052607 > >>> 224988.output: Total time in MGBuild 0.050697 > >>> 224989.output: Total time in MGBuild 0.078343 > >>> 224990.output: Total time in MGBuild 0.054634 > >>> 224991.output: Total time in MGBuild 0.052158 > >>> 224992.output: Total time in MGBuild 0.060286 > >>> 225008.output: Total time in MGBuild 0.062925 > >>> 225009.output: Total time in MGBuild 0.097357 > >>> 225010.output: Total time in MGBuild 0.061807 > >>> 225011.output: Total time in MGBuild 0.076617 > >>> 225012.output: Total time in MGBuild 0.099683 > >>> 225013.output: Total time in MGBuild 0.125580 > >>> 225014.output: Total time in MGBuild 0.190711 > >>> 225016.output: Total time in MGBuild 0.218329 > >>> 225017.output: Total time in MGBuild 0.282081 > >>> > >>> Although I didn't directly measure it, this suggests the time for > >>> MPI_Comm_Split is growing roughly quadratically with process > concurrency. > >>> > >>> > >>> > >>> > >>> I see the same effect on the K machine (8...64K nodes) where the > >>> code uses comm_split/dup in conjunction: > >>> run00008_7_1.sh.o2412931: Total time in MGBuild 0.026458 > seconds > >>> run00064_7_1.sh.o2415876: Total time in MGBuild 0.039121 > seconds > >>> run00512_7_1.sh.o2415877: Total time in MGBuild 0.086800 > seconds > >>> run01000_7_1.sh.o2414496: Total time in MGBuild 0.129764 > seconds > >>> run01728_7_1.sh.o2415878: Total time in MGBuild 0.224576 > seconds > >>> run04096_7_1.sh.o2415880: Total time in MGBuild 0.738979 > seconds > >>> run08000_7_1.sh.o2414504: Total time in MGBuild 2.123800 > seconds > >>> run13824_7_1.sh.o2415881: Total time in MGBuild 6.276573 > seconds > >>> run21952_7_1.sh.o2415882: Total time in MGBuild 13.634200 > seconds > >>> run32768_7_1.sh.o2415884: Total time in MGBuild 36.508670 > seconds > >>> run46656_7_1.sh.o2415874: Total time in MGBuild 58.668228 > seconds > >>> run64000_7_1.sh.o2415875: Total time in MGBuild 117.322217 > seconds > >>> > >>> > >>> A glance at the implementation on Mira (I don't know if the > >>> implementation on K is stock) suggests it should be using qsort to > >>> sort based on keys. Unfortunately, qsort is not performance robust > >>> like heap/merge sort. If one were to be productive and call comm_split > like... > >>> MPI_Comm_split(...,mycolor,myrank,...) > >>> then one runs the risk that the keys are presorted. This hits the > >>> worst case computational complexity for qsort... O(P^2). Demanding > >>> programmers avoid sending sorted keys seems unreasonable. > >>> > >>> > >>> I should note, I see a similar lack of scaling with MPI_Comm_dup on > >>> the K machine. Unfortunately, my BGQ data used an earlier version > >>> of the code that did not use comm_dup. As such, I can’t > >>> definitively say that it is a problem on that machine as well. > >>> > >>> Thus, I'm asking for scalable implementations of comm_split/dup > >>> using merge/heap sort whose worst case complexity is still PlogP to > >>> be prioritized in the next update. > >>> > >>> > >>> thanks > >>> _______________________________________________ > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/devel > >> > >> > > > > > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel _______________________________________________ To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/devel

1 0

Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers
by Bob Cernohous 22 Sep '14

22 Sep '14

I thought there was a "memory" optimization that disabled PAMI on irregular communicators. However I don't know the current state of that code. if(MPIDI_Process.optimized.memory && (comm->local_size & (comm->local_size-1))) { /* Don't create irregular geometries. Fallback to MPICH only collectives */ geom_init = 0; comm->mpid.geometry = NULL; } > -----Original Message----- > From: devel-bounces(a)mpich.org [mailto:[email protected]] On > Behalf Of Jeff Hammond > Sent: Monday, September 22, 2014 12:16 PM > To: devel(a)mpich.org > Subject: Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K > supercomputers > > PAMID_COLLECTIVES=0 is really bad for performance. IBM should figure out a > way to disable it on a per-communicator basis when MPI_COMM_SPLIT is > going to have issues. I recall they allow one to "unoptimize" a communicator > but I thought that was only possible after it was created. > > Jeff > > On Mon, Sep 22, 2014 at 8:39 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > > Sam, > > I had some updates from IBM last week. They reproduced the problem > > and found it only happens when the number of MPI ranks is > > non-power-of-2. Their advice is that since the IBM BG/Q optimized > > collectives themselves are mostly designed only to be helpful for > > blocks with power-of-2 geometries, you can try in your program to see > > if subsequent collective calls with > > PAMID_COLLECTIVES=1 are actually faster than PAMID_COLLECTIVES=0 on > > comms with a non-power-of-2 geometry. If the answer is no, then you > > can just run with PAMID_COLLECTIVES=0 and avoid the dup/split > performance issue. > > Otherwise, IBM may prioritize this ticket. > > > > Thanks. > > --Junchao Zhang > > > > On Thu, Jul 3, 2014 at 4:41 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> > wrote: > >> > >> Hi, Sam, > >> I wrote micro-benchmarks for MPI_Comm_split/dup. My profiling > >> results suggested the problem lies in a IBM PAMI library call, > >> PAMI_Geometry_create_taskrange(). Unfortunately, I don't have access > >> to the PAMI source code and don't know why. I reported it to IBM and > >> hope IBM will fix it. > >> Alternatively, you can set an environment variable > >> PAMID_COLLECTIVES=0 to disables pami collectives. My tests showed it > >> at least fixed the scalability problem of Comm_split and Comm_dup. > >> Also through profiling, I found the qsort() called in MPICH code is > >> actually using the merge sort algorithm in Mira's libc library. > >> > >> > >> > >> --Junchao Zhang > >> > >> > >> On Sat, May 17, 2014 at 9:06 AM, Sam Williams <swwilliams(a)lbl.gov> > wrote: > >>> > >>> I've been conducting scaling experiments on the Mira (Blue Gene/Q) > >>> and K > >>> (Sparc) supercomputers. I've noticed that the time required for > >>> MPI_Comm_split and MPI_Comm_dup can grow quickly with scale > (~P^2). > >>> As such, its performance eventually becomes a bottleneck. That is, > >>> although the benefit of using a subcommunicator is huge (multigrid > >>> solves are weak-scalable), the penalty of creating one (multigrid > >>> build time) is also huge. > >>> > >>> For example, when scaling from 1 to 46K nodes (= cubes of integers) > >>> on Mira, the time (in seconds) required to build a MG solver > >>> (including a > >>> subcommunicator) scales as > >>> 222335.output: Total time in MGBuild 0.056704 > >>> 222336.output: Total time in MGBuild 0.060834 > >>> 222348.output: Total time in MGBuild 0.064782 > >>> 222349.output: Total time in MGBuild 0.090229 > >>> 222350.output: Total time in MGBuild 0.075280 > >>> 222351.output: Total time in MGBuild 0.091852 > >>> 222352.output: Total time in MGBuild 0.137299 > >>> 222411.output: Total time in MGBuild 0.301552 > >>> 222413.output: Total time in MGBuild 0.606444 > >>> 222415.output: Total time in MGBuild 0.745272 > >>> 222417.output: Total time in MGBuild 0.779757 > >>> 222418.output: Total time in MGBuild 4.671838 > >>> 222419.output: Total time in MGBuild 15.123162 > >>> 222420.output: Total time in MGBuild 33.875626 > >>> 222421.output: Total time in MGBuild 49.494547 > >>> 222422.output: Total time in MGBuild 151.329026 > >>> > >>> If I disable the call to MPI_Comm_Split, my time scales as > >>> 224982.output: Total time in MGBuild 0.050143 > >>> 224983.output: Total time in MGBuild 0.052607 > >>> 224988.output: Total time in MGBuild 0.050697 > >>> 224989.output: Total time in MGBuild 0.078343 > >>> 224990.output: Total time in MGBuild 0.054634 > >>> 224991.output: Total time in MGBuild 0.052158 > >>> 224992.output: Total time in MGBuild 0.060286 > >>> 225008.output: Total time in MGBuild 0.062925 > >>> 225009.output: Total time in MGBuild 0.097357 > >>> 225010.output: Total time in MGBuild 0.061807 > >>> 225011.output: Total time in MGBuild 0.076617 > >>> 225012.output: Total time in MGBuild 0.099683 > >>> 225013.output: Total time in MGBuild 0.125580 > >>> 225014.output: Total time in MGBuild 0.190711 > >>> 225016.output: Total time in MGBuild 0.218329 > >>> 225017.output: Total time in MGBuild 0.282081 > >>> > >>> Although I didn't directly measure it, this suggests the time for > >>> MPI_Comm_Split is growing roughly quadratically with process > concurrency. > >>> > >>> > >>> > >>> > >>> I see the same effect on the K machine (8...64K nodes) where the > >>> code uses comm_split/dup in conjunction: > >>> run00008_7_1.sh.o2412931: Total time in MGBuild 0.026458 > seconds > >>> run00064_7_1.sh.o2415876: Total time in MGBuild 0.039121 > seconds > >>> run00512_7_1.sh.o2415877: Total time in MGBuild 0.086800 > seconds > >>> run01000_7_1.sh.o2414496: Total time in MGBuild 0.129764 > seconds > >>> run01728_7_1.sh.o2415878: Total time in MGBuild 0.224576 > seconds > >>> run04096_7_1.sh.o2415880: Total time in MGBuild 0.738979 > seconds > >>> run08000_7_1.sh.o2414504: Total time in MGBuild 2.123800 > seconds > >>> run13824_7_1.sh.o2415881: Total time in MGBuild 6.276573 > seconds > >>> run21952_7_1.sh.o2415882: Total time in MGBuild 13.634200 > seconds > >>> run32768_7_1.sh.o2415884: Total time in MGBuild 36.508670 > seconds > >>> run46656_7_1.sh.o2415874: Total time in MGBuild 58.668228 > seconds > >>> run64000_7_1.sh.o2415875: Total time in MGBuild 117.322217 > seconds > >>> > >>> > >>> A glance at the implementation on Mira (I don't know if the > >>> implementation on K is stock) suggests it should be using qsort to > >>> sort based on keys. Unfortunately, qsort is not performance robust > >>> like heap/merge sort. If one were to be productive and call comm_split > like... > >>> MPI_Comm_split(...,mycolor,myrank,...) > >>> then one runs the risk that the keys are presorted. This hits the > >>> worst case computational complexity for qsort... O(P^2). Demanding > >>> programmers avoid sending sorted keys seems unreasonable. > >>> > >>> > >>> I should note, I see a similar lack of scaling with MPI_Comm_dup on > >>> the K machine. Unfortunately, my BGQ data used an earlier version > >>> of the code that did not use comm_dup. As such, I can’t > >>> definitively say that it is a problem on that machine as well. > >>> > >>> Thus, I'm asking for scalable implementations of comm_split/dup > >>> using merge/heap sort whose worst case complexity is still PlogP to > >>> be prioritized in the next update. > >>> > >>> > >>> thanks > >>> _______________________________________________ > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/devel > >> > >> > > > > > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0