July 2014 - devel - mailman3.cels.anl.gov

MPI_Reduce_scatter_block() with zero-size, NULL buffers.
by Lisandro Dalcin 06 Jul '14

06 Jul '14

This is from a build of master updated today and configured with --enable-g=all. AFAICT, previous versions of MPICH did not generate this error. Was this an intentional change? Traceback (most recent call last): File "/home/dalcinl/Devel/mpi4py-dev/test/test_msgzero.py", line 36, in testReductions comm.Reduce_scatter_block(self.null_b, self.null_b) File "Comm.pyx", line 714, in mpi4py.MPI.Comm.Reduce_scatter_block (src/mpi4py.MPI.c:93219) Exception: Invalid buffer pointer, error stack: PMPI_Reduce_scatter_block(1182): MPI_Reduce_scatter_block(sbuf=(nil), rbuf=(nil), recvcount=0, MPI_INT, MPI_SUM, MPI_COMM_SELF) failed PMPI_Reduce_scatter_block(1142): Buffers must not be aliased -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] mpich-3.1.1: MPI_Status_c2f
by Balaji, Pavan 05 Jul '14

05 Jul '14

It was not intentional. https://trac.mpich.org/projects/mpich/ticket/2122 I have a patch pending review, which should go in either this weekend or early next week. We’ll push out a 3.1.2 release with a few fixes, including this one, soon. — Pavan On Jul 5, 2014, at 2:29 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > Was this change intentional? This way, code linked with "mpicc" is > failing because of the missing symbol. IMHO, MPI_Status_c2f should be > defined in libmpi.dylib. Not checked on Linux, but looking at the > Homebrew formula, I cannot find anything obvious to indicate this is a > brew and/or OS X issue: > https://github.com/Homebrew/homebrew/blob/master/Library/Formula/mpich2.rb > > $ nm /usr/local/Cellar/mpich2/3.1.1/lib/libmpi.dylib | grep MPI_Status_c2f > > $ nm /usr/local/Cellar/mpich2/3.1.1/lib/libmpifort.dylib | grep MPI_Status_c2f > > 0000000000008d60 T _MPI_Status_c2f > > > -- > Lisandro Dalcin > --------------- > CIMEC (UNL/CONICET) > Predio CONICET-Santa Fe > Colectora RN 168 Km 472, Paraje El Pozo > 3000 Santa Fe, Argentina > Tel: +54-342-4511594 (ext 1016) > Tel/Fax: +54-342-4511169 > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0

mpich-3.1.1: MPI_Status_c2f
by Lisandro Dalcin 05 Jul '14

05 Jul '14

Was this change intentional? This way, code linked with "mpicc" is failing because of the missing symbol. IMHO, MPI_Status_c2f should be defined in libmpi.dylib. Not checked on Linux, but looking at the Homebrew formula, I cannot find anything obvious to indicate this is a brew and/or OS X issue: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/mpich2.rb $ nm /usr/local/Cellar/mpich2/3.1.1/lib/libmpi.dylib | grep MPI_Status_c2f $ nm /usr/local/Cellar/mpich2/3.1.1/lib/libmpifort.dylib | grep MPI_Status_c2f 0000000000008d60 T _MPI_Status_c2f -- Lisandro Dalcin --------------- CIMEC (UNL/CONICET) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1016) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers
by Jeff Hammond 03 Jul '14

03 Jul '14

PAMI is open-source via https://repo.anl-external.org/repos/bgq-driver/. I believe ALCF has already reported this bug but you can contact support(a)alcf.anl.gov for an update. Best, Jeff On Thu, Jul 3, 2014 at 2:41 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > Hi, Sam, > I wrote micro-benchmarks for MPI_Comm_split/dup. My profiling results > suggested the problem lies in a IBM PAMI library call, > PAMI_Geometry_create_taskrange(). Unfortunately, I don't have access to the > PAMI source code and don't know why. I reported it to IBM and hope IBM will > fix it. > Alternatively, you can set an environment variable PAMID_COLLECTIVES=0 to > disables pami collectives. My tests showed it at least fixed the scalability > problem of Comm_split and Comm_dup. > Also through profiling, I found the qsort() called in MPICH code is > actually using the merge sort algorithm in Mira's libc library. > > > > --Junchao Zhang > > > On Sat, May 17, 2014 at 9:06 AM, Sam Williams <swwilliams(a)lbl.gov> wrote: >> >> I've been conducting scaling experiments on the Mira (Blue Gene/Q) and K >> (Sparc) supercomputers. I've noticed that the time required for >> MPI_Comm_split and MPI_Comm_dup can grow quickly with scale (~P^2). As >> such, its performance eventually becomes a bottleneck. That is, although >> the benefit of using a subcommunicator is huge (multigrid solves are >> weak-scalable), the penalty of creating one (multigrid build time) is also >> huge. >> >> For example, when scaling from 1 to 46K nodes (= cubes of integers) on >> Mira, the time (in seconds) required to build a MG solver (including a >> subcommunicator) scales as >> 222335.output: Total time in MGBuild 0.056704 >> 222336.output: Total time in MGBuild 0.060834 >> 222348.output: Total time in MGBuild 0.064782 >> 222349.output: Total time in MGBuild 0.090229 >> 222350.output: Total time in MGBuild 0.075280 >> 222351.output: Total time in MGBuild 0.091852 >> 222352.output: Total time in MGBuild 0.137299 >> 222411.output: Total time in MGBuild 0.301552 >> 222413.output: Total time in MGBuild 0.606444 >> 222415.output: Total time in MGBuild 0.745272 >> 222417.output: Total time in MGBuild 0.779757 >> 222418.output: Total time in MGBuild 4.671838 >> 222419.output: Total time in MGBuild 15.123162 >> 222420.output: Total time in MGBuild 33.875626 >> 222421.output: Total time in MGBuild 49.494547 >> 222422.output: Total time in MGBuild 151.329026 >> >> If I disable the call to MPI_Comm_Split, my time scales as >> 224982.output: Total time in MGBuild 0.050143 >> 224983.output: Total time in MGBuild 0.052607 >> 224988.output: Total time in MGBuild 0.050697 >> 224989.output: Total time in MGBuild 0.078343 >> 224990.output: Total time in MGBuild 0.054634 >> 224991.output: Total time in MGBuild 0.052158 >> 224992.output: Total time in MGBuild 0.060286 >> 225008.output: Total time in MGBuild 0.062925 >> 225009.output: Total time in MGBuild 0.097357 >> 225010.output: Total time in MGBuild 0.061807 >> 225011.output: Total time in MGBuild 0.076617 >> 225012.output: Total time in MGBuild 0.099683 >> 225013.output: Total time in MGBuild 0.125580 >> 225014.output: Total time in MGBuild 0.190711 >> 225016.output: Total time in MGBuild 0.218329 >> 225017.output: Total time in MGBuild 0.282081 >> >> Although I didn't directly measure it, this suggests the time for >> MPI_Comm_Split is growing roughly quadratically with process concurrency. >> >> >> >> >> I see the same effect on the K machine (8...64K nodes) where the code uses >> comm_split/dup in conjunction: >> run00008_7_1.sh.o2412931: Total time in MGBuild 0.026458 seconds >> run00064_7_1.sh.o2415876: Total time in MGBuild 0.039121 seconds >> run00512_7_1.sh.o2415877: Total time in MGBuild 0.086800 seconds >> run01000_7_1.sh.o2414496: Total time in MGBuild 0.129764 seconds >> run01728_7_1.sh.o2415878: Total time in MGBuild 0.224576 seconds >> run04096_7_1.sh.o2415880: Total time in MGBuild 0.738979 seconds >> run08000_7_1.sh.o2414504: Total time in MGBuild 2.123800 seconds >> run13824_7_1.sh.o2415881: Total time in MGBuild 6.276573 seconds >> run21952_7_1.sh.o2415882: Total time in MGBuild 13.634200 seconds >> run32768_7_1.sh.o2415884: Total time in MGBuild 36.508670 seconds >> run46656_7_1.sh.o2415874: Total time in MGBuild 58.668228 seconds >> run64000_7_1.sh.o2415875: Total time in MGBuild 117.322217 seconds >> >> >> A glance at the implementation on Mira (I don't know if the implementation >> on K is stock) suggests it should be using qsort to sort based on keys. >> Unfortunately, qsort is not performance robust like heap/merge sort. If one >> were to be productive and call comm_split like... >> MPI_Comm_split(...,mycolor,myrank,...) >> then one runs the risk that the keys are presorted. This hits the worst >> case computational complexity for qsort... O(P^2). Demanding programmers >> avoid sending sorted keys seems unreasonable. >> >> >> I should note, I see a similar lack of scaling with MPI_Comm_dup on the K >> machine. Unfortunately, my BGQ data used an earlier version of the code >> that did not use comm_dup. As such, I can’t definitively say that it is a >> problem on that machine as well. >> >> Thus, I'm asking for scalable implementations of comm_split/dup using >> merge/heap sort whose worst case complexity is still PlogP to be prioritized >> in the next update. >> >> >> thanks >> _______________________________________________ >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/devel > > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] MPI_Comm_Split/Dup scalability on BGQ and K supercomputers
by Junchao Zhang 03 Jul '14

03 Jul '14

Hi, Sam, I wrote micro-benchmarks for MPI_Comm_split/dup. My profiling results suggested the problem lies in a IBM PAMI library call, PAMI_Geometry_create_taskrange(). Unfortunately, I don't have access to the PAMI source code and don't know why. I reported it to IBM and hope IBM will fix it. Alternatively, you can set an environment variable PAMID_COLLECTIVES=0 to disables pami collectives. My tests showed it at least fixed the scalability problem of Comm_split and Comm_dup. Also through profiling, I found the qsort() called in MPICH code is actually using the merge sort algorithm in Mira's libc library. --Junchao Zhang On Sat, May 17, 2014 at 9:06 AM, Sam Williams <swwilliams(a)lbl.gov> wrote: > I've been conducting scaling experiments on the Mira (Blue Gene/Q) and K > (Sparc) supercomputers. I've noticed that the time required for > MPI_Comm_split and MPI_Comm_dup can grow quickly with scale (~P^2). As > such, its performance eventually becomes a bottleneck. That is, although > the benefit of using a subcommunicator is huge (multigrid solves are > weak-scalable), the penalty of creating one (multigrid build time) is also > huge. > > For example, when scaling from 1 to 46K nodes (= cubes of integers) on > Mira, the time (in seconds) required to build a MG solver (including a > subcommunicator) scales as > 222335.output: Total time in MGBuild 0.056704 > 222336.output: Total time in MGBuild 0.060834 > 222348.output: Total time in MGBuild 0.064782 > 222349.output: Total time in MGBuild 0.090229 > 222350.output: Total time in MGBuild 0.075280 > 222351.output: Total time in MGBuild 0.091852 > 222352.output: Total time in MGBuild 0.137299 > 222411.output: Total time in MGBuild 0.301552 > 222413.output: Total time in MGBuild 0.606444 > 222415.output: Total time in MGBuild 0.745272 > 222417.output: Total time in MGBuild 0.779757 > 222418.output: Total time in MGBuild 4.671838 > 222419.output: Total time in MGBuild 15.123162 > 222420.output: Total time in MGBuild 33.875626 > 222421.output: Total time in MGBuild 49.494547 > 222422.output: Total time in MGBuild 151.329026 > > If I disable the call to MPI_Comm_Split, my time scales as > 224982.output: Total time in MGBuild 0.050143 > 224983.output: Total time in MGBuild 0.052607 > 224988.output: Total time in MGBuild 0.050697 > 224989.output: Total time in MGBuild 0.078343 > 224990.output: Total time in MGBuild 0.054634 > 224991.output: Total time in MGBuild 0.052158 > 224992.output: Total time in MGBuild 0.060286 > 225008.output: Total time in MGBuild 0.062925 > 225009.output: Total time in MGBuild 0.097357 > 225010.output: Total time in MGBuild 0.061807 > 225011.output: Total time in MGBuild 0.076617 > 225012.output: Total time in MGBuild 0.099683 > 225013.output: Total time in MGBuild 0.125580 > 225014.output: Total time in MGBuild 0.190711 > 225016.output: Total time in MGBuild 0.218329 > 225017.output: Total time in MGBuild 0.282081 > > Although I didn't directly measure it, this suggests the time for > MPI_Comm_Split is growing roughly quadratically with process concurrency. > > > > > I see the same effect on the K machine (8...64K nodes) where the code uses > comm_split/dup in conjunction: > run00008_7_1.sh.o2412931: Total time in MGBuild 0.026458 seconds > run00064_7_1.sh.o2415876: Total time in MGBuild 0.039121 seconds > run00512_7_1.sh.o2415877: Total time in MGBuild 0.086800 seconds > run01000_7_1.sh.o2414496: Total time in MGBuild 0.129764 seconds > run01728_7_1.sh.o2415878: Total time in MGBuild 0.224576 seconds > run04096_7_1.sh.o2415880: Total time in MGBuild 0.738979 seconds > run08000_7_1.sh.o2414504: Total time in MGBuild 2.123800 seconds > run13824_7_1.sh.o2415881: Total time in MGBuild 6.276573 seconds > run21952_7_1.sh.o2415882: Total time in MGBuild 13.634200 seconds > run32768_7_1.sh.o2415884: Total time in MGBuild 36.508670 seconds > run46656_7_1.sh.o2415874: Total time in MGBuild 58.668228 seconds > run64000_7_1.sh.o2415875: Total time in MGBuild 117.322217 seconds > > > A glance at the implementation on Mira (I don't know if the implementation > on K is stock) suggests it should be using qsort to sort based on keys. > Unfortunately, qsort is not performance robust like heap/merge sort. If > one were to be productive and call comm_split like... > MPI_Comm_split(...,mycolor,myrank,...) > then one runs the risk that the keys are presorted. This hits the worst > case computational complexity for qsort... O(P^2). Demanding programmers > avoid sending sorted keys seems unreasonable. > > > I should note, I see a similar lack of scaling with MPI_Comm_dup on the K > machine. Unfortunately, my BGQ data used an earlier version of the code > that did not use comm_dup. As such, I can’t definitively say that it is a > problem on that machine as well. > > Thus, I'm asking for scalable implementations of comm_split/dup using > merge/heap sort whose worst case complexity is still PlogP to be > prioritized in the next update. > > > thanks > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0