- discuss - mailman3.cels.anl.gov

low performance in an asynchronous, mixed MPI/pthreads app
by Geoffrey Irving 06 Jan '13

06 Jan '13

Hello, With help from Jeff Hammond and Jed Brown, I am trying to optimize and scale a solver for the board game pentago on the Cray XE6 machine Hopper and NERSC. The code mixes MPI and pthreads, and has an asynchronous message passing structure. We're seeing much lower performance than expected, and are trying to understand why. Here is a visualization of a 96 core test run using 16 ranks with 6 threads per core (4 Hopper nodes): http://naml.us/random/pentago/history-random17.png Each rank has 1 dedicated communication thread doing almost all of the MPI calls, and 5 worker threads doing computation. White (of which there is too much) indicates an idle thread. The color says what's happening on each thread, and the lines indicate information flow: ranks request input data from other ranks, respond to input requests with data, compute once data arrives, send output to other ranks, and process output for final storage. The compute is yellow and the processing is interleaved red/blue. The weirdness is as follows: 1. All messages are sent using MPI_Isend, but Isend sometimes takes a very long time to return even for tiny (8 byte) messages. For example, the light blue "wakeup" messages are sent from a worker thread to the master communication thread to indicate that computation is complete, breaking the communication thread out of an MPI_Waitsome call (using MPI_THREAD_MULTIPLE). Some of the input request Isends also take a long time, and these are also 8 bytes. What would cause MPI_Isend to take a long time to return, both for small messages (wakeups and input requests) and for large messages (input responses and output sends)? 2. There's a common pattern in the visualization where a bunch of output processing happens immediately *before* computation. Whenever this happens the output and compute are unrelated in terms of dataflow. This is probably the same problem as (1). What seems to be happening is that an MPI_Isend on the communication thread is taking a long time to return, prevent the communication thread from noticing that other messages have arrived (the communication will call MPI_Waitsome once the Isend returns. If both output messages from processing and input messages for compute stack up, the code processes the output first in order to deallocate certain memory as soon possible. 3. Stalled wakeup Isends seem to complete at the same time as input response Isends on the same rank. Is the MPI_THREAD_MULTIPLE support problematic? 4. The message latency is quite high. For example, one of the input responses takes over .3 s in the above image. Each such response message is about 92K. It's possible that line is actually more than one input response from the same rank (I do not merge them currently), but in that case I'd expect them to complete at different times. Unfortunately, my current visualization doesn't do a good job of visualizing instantaneous bandwidth: I have the data but don't know quite what to measure. Bandwidth averaged over 10s intervals is shown here: http://naml.us/random/pentago/bandwidth-10s-async-random17-nonblock.svg I've also tried asynchronous progress mode (6 threads per rank, 1 progress thread for MPI, 1 communication thread for me, and 4 worker threads). This gave a 12% speedup but didn't change the picture qualitatively (apologies for the mismatched colors and different horizontal scale): http://naml.us/random/pentago/history-async-random17.png Some things that might contribute to the problem: 1. Since the 92K input responses are sent in response to 8 byte request messages, I can prepost all matching Irecvs. However, output messages arrive unexpectedly, so the best I can do is post a few wildcard Irecvs (MPI_ANY_SOURCE, MPI_ANY_TAG on the output message communicator). Jeff thought wildcard Irecvs might not be good enough to hit the fast path. Moreover, output messages are not compressed, so most are 262K. I didn't compress them historically because compression isn't all that fast (I'm using Google's Snappy plus domain specific preconditioning), but that may be a good experiment to run. 2. I don't know how to visualize instantaneous bandwidth, even though I have start and end times for all large messages (start and end of Isend, Irecv request completion time). It's possible the problem is bad network load balancing. I'm happy to whip up a plot if anyone has suggestions as to what to draw. 3. Waking a communication thread out of an MPI_Waitsome is somewhat of a hack. It's the only time the worker threads touch MPI; otherwise I could use MPI_THREAD_FUNNELED. However, the only alternative I know is for the communication thread to poll on MPI_Testsome and check for thread completion in between. Might that be better (it's an easy experiment to run)? Finally, a few details about the computation structure follow. I've tried to make the above independent, so feel free to stop here. Thanks for any thoughts! Geoffrey --------------------------------------- Values of pentago positions with n stones depend only values of positions with n+1 stones, so the computation is embarrassingly parallel within each such "level". However, it is also memory constrained: holding level 24 and 25 in memory at the same time (one input level and one output level) requires about 80 TB. The state space of each level is divided into "blocks", and the space of computation required to go from level n to n+1 is divided into "lines". Each line has a set of input blocks at level n and a set of output blocks at level n+1. Most blocks have size 64 * 8**4 = 262144 bytes uncompressed. We partition all blocks and lines amongst the ranks, then operate as follows: 1. When a rank has enough memory to allocate a new line, it posts Irecvs and Isends 8 byte request message to all ranks which own an input block. 2. When a rank receives a request, it replies using an Isend from a flat buffer (doing O(1) work otherwise). These messages are compressed, and take up around 92K. 3. Once all input blocks arrive, we compute the line's output blocks. Each block at level n+1 is computed from several (4) lines, so if necessary we send the output blocks to the rank which owns them. These messages are uncompressed, and are usually 262K. Code is here for reference: https://github.com/girving/pentago

1 0

Re: [mpich-discuss] mpich 3.0.1 won't build shared libs without fortran?
by Dave Goodell 03 Jan '13

03 Jan '13

On Jan 3, 2013, at 8:55 AM CST, Eric A. Borisch wrote: > I am having difficulty getting mpich 3.0.1 to build shared libs when I'm not using fortran; it still runs checks (against false) for the fortran compilers (even though I am requesting --disable-fc --disable-f77) and associated tools and predictably fails those tests, needlessly disabling the "build shared libs" var. > > Any thoughts? I've put the config.log [1] up on pastebin rather than attaching the half-meg file. You can see things go south in lines 498-579. I can reproduce this locally. We were trying to be clever in [2], but it seems to have backfired into preventing shared linking in this case. It looks like the right way to communicate this sort of thing to libtool is to set these variables to "no" instead. I've committed a fix along these lines in [3]. Thanks for reporting this. -Dave [2] http://trac.mpich.org/projects/mpich/changeset/10622 [3] http://trac.mpich.org/projects/mpich/changeset/10820

1 0

Re: [mpich-discuss] MPICH2 version number
by Jayesh Krishna 03 Jan '13

03 Jan '13

Hi, Did you try checking information about MPICH2 from the Control Panel (Add Remove Programs - Select MPICH2 - Click on "Click here for Support Information")? Regards, Jayesh ----- Original Message ----- From: "Pavan Balaji" <balaji(a)mcs.anl.gov> To: discuss(a)mpich.org Sent: Thursday, January 3, 2013 12:01:57 AM Subject: Re: [mpich-discuss] MPICH2 version number On 01/02/2013 01:22 PM US Central Time, Guanhua Zhang wrote: > Once I installed MPICH2 under Windows. How can I know which version > installed? I tried "mpiexec -version". It did not work. Did you try mpich2version? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

mpich 3.0.1 won't build shared libs without fortran?
by Eric A. Borisch 03 Jan '13

03 Jan '13

I am having difficulty getting mpich 3.0.1 to build shared libs when I'm not using fortran; it still runs checks (against false) for the fortran compilers (even though I am requesting --disable-fc --disable-f77) and associated tools and predictably fails those tests, needlessly disabling the "build shared libs" var. Any thoughts? I've put the config.log [1] up on pastebin rather than attaching the half-meg file. You can see things go south in lines 498-579. Hopefully I've just missed something obvious... (This was building with clang; similar happens with llvm-gcc.) Thanks, Eric [1] http://pastebin.com/snr3VGdf

1 0

Re: [mpich-discuss] Hard-coded path to mpiexec.hydra.conf (1.4, 1.5)
by Pavan Balaji 03 Jan '13

03 Jan '13

On 01/02/2013 01:25 PM US Central Time, Yauheni Zelenko wrote: > I think this could be solution, but as it looks like also source code > and object files should be shipped to customers. Or my be I didn't > notice configure option for intermediate build output :-) > > Solution with environment variable will decrease size of shipped > package, since only final binaries and support files will be shipped. > It may be more preferable solution when software installation script > is not trivial. Ok, I do see some value in that. I've created a ticket for it: https://trac.mpich.org/projects/mpich/ticket/1777 -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] MPICH2 version number
by Pavan Balaji 03 Jan '13

03 Jan '13

On 01/02/2013 01:22 PM US Central Time, Guanhua Zhang wrote: > Once I installed MPICH2 under Windows. How can I know which version > installed? I tried "mpiexec -version". It did not work. Did you try mpich2version? -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Extensible Clang type annotation
by Jed Brown 02 Jan '13

02 Jan '13

On Wed, Jan 2, 2013 at 1:23 PM, Dmitri Gribenko <gribozavr(a)gmail.com> wrote: > Unfortunately we can not, because user-defined datatypes are > syntactically different from builtin datatypes in MPICH. MPI_INT is a > macro, so what Clang sees is essentially a magic number. With > user-defined datatypes there's an identifier (MyEnumDatatype) that > actually shows up in the AST. There's a separate quirk in Clang for > handling these magic numbers and corresponding 'static const' > variables (not needed for OpenMPI, btw). > Okay, makes sense. > > so that the symbols are guaranteed to only show up once in > > executables (rather than once per compilation unit)? > > A smart enough compiler (and Clang is smart enough) will eliminate > those even at -O0. > Okay, gcc also removes them with optimization. Thanks, Dmitri.

1 0

Re: [mpich-discuss] Hard-coded path to mpiexec.hydra.conf (1.4, 1.5)
by Yauheni Zelenko 02 Jan '13

02 Jan '13

Hi, Pavan! I think this could be solution, but as it looks like also source code and object files should be shipped to customers. Or my be I didn't notice configure option for intermediate build output :-) Solution with environment variable will decrease size of shipped package, since only final binaries and support files will be shipped. It may be more preferable solution when software installation script is not trivial. Eugene. ________________________________________ From: Pavan Balaji [balaji(a)mcs.anl.gov] Sent: Sunday, December 23, 2012 7:00 PM To: mpich-discuss(a)mcs.anl.gov Cc: Yauheni Zelenko Subject: Re: [mpich-discuss] Hard-coded path to mpiexec.hydra.conf (1.4, 1.5) Sorry about the delay in getting around to this. We were tied up with the mpich-3.0 release. I'm still missing the point of this environment variable. Once you have a pre-built mpich installation, you can install mpich in any location of your choosing using: % make install prefix=<foobar> This overrides the --prefix passed to configure. In this case, you should ship the prebuilt mpich to your customers and run the above "make install" command to place it in the right location. Some of our other vendor partners follow the same model as well. -- Pavan On 07/12/2012 12:46 PM US Central Time, Yauheni Zelenko wrote: > Hi, Pavan! > > Thank you for help! > > However will be good idea to disable check of file in install location at all. > > Or there should be way to specify install location in run-time with environment variable or command-line option (OpenMPI uses OPAL_PREFIX and --prefix correspondingly). > > Other files affected are mpi compilation scripts (mpicc, etc), MPE scripts, libtool generated files, packages, makefiles. > > Easy way to check - to compile MPICH2 in your home directory, install it and grep installed files for your login name. > > Eugene. > ________________________________________ > From: Pavan Balaji [balaji(a)mcs.anl.gov] > Sent: Thursday, July 12, 2012 10:26 AM > To: mpich-discuss(a)mcs.anl.gov > Cc: Yauheni Zelenko > Subject: Re: [mpich-discuss] Hard-coded path to mpiexec.hydra.conf (1.4, 1.5) > > Thanks. I've fixed this in r10004. Now the priority order for the > config file: > > 1. Command line > > 2. Environment variable > > 3. Home directory > > 4. Install directory > > -- Pavan > > On 07/11/2012 06:37 PM, Yauheni Zelenko wrote: >> Hi! >> >> Hydra always try to open mpiexec.hydra.conf file in original install path location in HYD_uii_mpx_get_parameters() if such file is not found in user's home directory. >> >> We ship pre-built MPICH2 with our application to customers, where original install path is invalid. >> >> I think will be good idea to rely on environment variable instead of hard-coded path. >> >> Eugene. >> _______________________________________________ >> mpich-discuss mailing list mpich-discuss(a)mcs.anl.gov >> To manage subscription options or unsubscribe: >> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > > _______________________________________________ > mpich-discuss mailing list mpich-discuss(a)mcs.anl.gov > To manage subscription options or unsubscribe: > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Extensible Clang type annotation
by Dmitri Gribenko 02 Jan '13

02 Jan '13

On Wed, Jan 2, 2013 at 9:01 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Wed, Jan 2, 2013 at 12:09 PM, Dmitri Gribenko <gribozavr(a)gmail.com> > wrote: >> >> Sorry if what I wrote was unclear. Here's a complete example: >> >> $ cat t.cc >> #include <mpi.h> >> >> enum MyEnum { Foo }; >> MPI_Datatype MyEnumDatatype MPICH_ATTR_TYPE_TAG(MyEnum); > > > Oh, fantastic. I didn't realize this could be applied to uninitialized > variables (and somehow didn't recognize that you were saying as much in the > first email). In light of this, can mpi.h change all the static declarations > to extern Unfortunately we can not, because user-defined datatypes are syntactically different from builtin datatypes in MPICH. MPI_INT is a macro, so what Clang sees is essentially a magic number. With user-defined datatypes there's an identifier (MyEnumDatatype) that actually shows up in the AST. There's a separate quirk in Clang for handling these magic numbers and corresponding 'static const' variables (not needed for OpenMPI, btw). > so that the symbols are guaranteed to only show up once in > executables (rather than once per compilation unit)? A smart enough compiler (and Clang is smart enough) will eliminate those even at -O0. Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr(a)gmail.com>*/

1 0

Re: [mpich-discuss] MPICH2 version number
by Guanhua Zhang 02 Jan '13

02 Jan '13

Hi, Once I installed MPICH2 under Windows. How can I know which version installed? I tried "mpiexec -version". It did not work. Thanks, Guanhua ----- Original Message ----- From: "Pavan Balaji" <balaji(a)mcs.anl.gov> To: <discuss(a)mpich.org> Cc: "Átila Gracco" <gracco.atila(a)gmail.com> Sent: Sunday, December 23, 2012 7:17 PM Subject: Re: [mpich-discuss] Trouble importing mpich2 library This looks like a problem with Maple Grid Computing. Please contact them for instructions. -- Pavan On 10/05/2012 03:23 PM US Central Time, Átila Gracco wrote: > Greetings, > > I'm trying to install a program, Maple Grid Computing 16 , that requires > that i install the mpich2 library, after i do that and go to the program > instalation screen, at some point during instalation it asks me to put > the Name of MPI shared lib, and the root location of it, i'm using > Ubuntu 12.04 - 64 bits. I almost forgot to say, i'm kind of newbie in > linux. > > -- > Att, > *Átila Gracco* > Acadêmico do 8º período | Engenharia de Computação UFG > +55 (62) 8126-1016 > Skype: atilagracco > > > _______________________________________________ > mpich-discuss mailing list mpich-discuss(a)mcs.anl.gov > To manage subscription options or unsubscribe: > https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0