January 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fatal error in PMPI_Reduce
by Pavan Balaji 11 Jan '13

11 Jan '13

Michael, Did you try just using mpiexec? mpiexec -n 32 /usr/local/apps/cxxcpi -- Pavan On 01/11/2013 03:31 PM US Central Time, Michael Colonno wrote: > Hi All ~ > > > > I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on > a CentOS 6.3 x64 system using SLURM as the process manager. My configure > was simply: > > > > ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 > > > > No errors during build or install. When I compile and run the example > program cxxcpi I get (truncated): > > > > $ srun -n32 /usr/local/apps/cxxcpi > > Fatal error in PMPI_Reduce: A process has failed, error stack: > > PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, > rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, > MPI_COMM_WORLD) failed > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(779)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(144).......: > > MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 > > MPIR_Reduce_intra(799)..........: > > MPIR_Reduce_impl(1029)..........: > > MPIR_Reduce_intra(835)..........: > > MPIR_Reduce_binomial(206).......: Failure during collective > > srun: error: task 0: Exited with exit code 1 > > > > This error is experienced with many of my MPI programs. A > different application yields: > > > > PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, > root=0, MPI_COMM_WORLD) failed > > MPIR_Bcast_impl(1369).: > > MPIR_Bcast_intra(1160): > > MPIR_SMP_Bcast(1077)..: Failure during collective > > > > Can anyone point me in the right direction? > > > > Thanks, > > ~Mike C. > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Fatal error in PMPI_Reduce
by Michael Colonno 11 Jan '13

11 Jan '13

Hi All ~ I've compiled MPICH2 3.0 with the Intel compiler (v. 13) on a CentOS 6.3 x64 system using SLURM as the process manager. My configure was simply: ./configure --with-pmi=slurm --with-pm=no --prefix=/usr/local/apps/MPICH2 No errors during build or install. When I compile and run the example program cxxcpi I get (truncated): $ srun -n32 /usr/local/apps/cxxcpi Fatal error in PMPI_Reduce: A process has failed, error stack: PMPI_Reduce(1217)...............: MPI_Reduce(sbuf=0x7fff4ad18120, rbuf=0x7fff4ad18128, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(779)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(144).......: MPIDI_CH3U_Recvq_FDU_or_AEP(612): Communication error with rank 16 MPIR_Reduce_intra(799)..........: MPIR_Reduce_impl(1029)..........: MPIR_Reduce_intra(835)..........: MPIR_Reduce_binomial(206).......: Failure during collective srun: error: task 0: Exited with exit code 1 This error is experienced with many of my MPI programs. A different application yields: PMPI_Bcast(1525)......: MPI_Bcast(buf=0x7fff545be5fc, count=1, MPI_INT, root=0, MPI_COMM_WORLD) failed MPIR_Bcast_impl(1369).: MPIR_Bcast_intra(1160): MPIR_SMP_Bcast(1077)..: Failure during collective Can anyone point me in the right direction? Thanks, ~Mike C.

1 0

Re: [mpich-discuss] start MPD daemons on 2 different subnets
by Dave Goodell 11 Jan '13

11 Jan '13

Use hydra instead of MPD: http://wiki.mpich.org/mpich/index.php/FAQ#Q:_I_don.27t_like_.3CWHATEVER.3E_… -Dave On Jan 11, 2013, at 12:55 PM CST, Na Zhang wrote: > Dear developers, > > We want to start MPD daemons on 2 different subnets: for example, > subnet #1 192.168.0.0/24 and subnet #2: 192.168.1.0/24 > > The two subnets are connected via switches and they can talk to each > other. Next, we'd start MPD daemons on two nodes: > > Node #1 (in subnet #1): (hostname:node_001) IP=192.168.0.1 > Node #2 (in subnet #2): (hostname:node_002) IP=192.168.1.1 > > We used the following commands: > > on Node #1: mpd --ifhn=192.168.0.1 --daemon > (daemon is successfully started on node_001) > > on Node #2: mpd -h node_001 -p <node1's_port_number> > --ifhn=192.168.1.1 --daemon > (daemon cannot be started on Node #2. there is no error message, we > use "mpdtrace" to check on Node #1, it shows no daemon started on Node > #2 ) > > Node #2 cannot join the ring that is generated by node #1. > > How should we do? > > Thank you in advance. > > Sincerely, > Na Zhang > > -- > Na Zhang, Ph.D. Candidate > Dept. of Applied Mathematics and Statistics > Stony Brook University > Phone: 631-838-3205 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

start MPD daemons on 2 different subnets
by Na Zhang 11 Jan '13

11 Jan '13

Dear developers, We want to start MPD daemons on 2 different subnets: for example, subnet #1 192.168.0.0/24 and subnet #2: 192.168.1.0/24 The two subnets are connected via switches and they can talk to each other. Next, we'd start MPD daemons on two nodes: Node #1 (in subnet #1): (hostname:node_001) IP=192.168.0.1 Node #2 (in subnet #2): (hostname:node_002) IP=192.168.1.1 We used the following commands: on Node #1: mpd --ifhn=192.168.0.1 --daemon (daemon is successfully started on node_001) on Node #2: mpd -h node_001 -p <node1's_port_number> --ifhn=192.168.1.1 --daemon (daemon cannot be started on Node #2. there is no error message, we use "mpdtrace" to check on Node #1, it shows no daemon started on Node #2 ) Node #2 cannot join the ring that is generated by node #1. How should we do? Thank you in advance. Sincerely, Na Zhang -- Na Zhang, Ph.D. Candidate Dept. of Applied Mathematics and Statistics Stony Brook University Phone: 631-838-3205

1 0

Re: [mpich-discuss] [PATCH] Use attribute layout_compatible for pair types
by Dmitri Gribenko 09 Jan '13

09 Jan '13

On Wed, Jan 9, 2013 at 10:00 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Jan 9, 2013 12:56 PM, "Dmitri Gribenko" <gribozavr(a)gmail.com> wrote: >> I don't think GCC isn't smart enough to do the same. Do you compile >> with optimization? > > Dmitri, as discussed in the other thread, it's smart enough, but only when > optimization is turned on. There's no reason to needlessly make debug builds > heavier than necessary. This is not a big deal either way. Oh, now I see -- in debug builds it still emits these. Thank you for fixing! Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr(a)gmail.com>*/

1 0

Re: [mpich-discuss] [PATCH] Use attribute layout_compatible for pair types
by Jed Brown 09 Jan '13

09 Jan '13

On Jan 9, 2013 12:56 PM, "Dmitri Gribenko" <gribozavr(a)gmail.com> wrote: > On Wed, Jan 9, 2013 at 8:19 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > > Both implemented and pushed as d440abb and ac15f7a. Thanks. > > > > -Dave > > > > On Jan 1, 2013, at 11:14 PM CST, Jed Brown wrote: > > > >> In addition, I suggest guarding these definitions. Leaving these in > increases the total number of symbols in an example executable linking > PETSc by a factor of 2. (They're all read-only, but they're still there.) > Clang is smart enough to remove these, presumably because it understands > the special attributes. > > No, LLVM removes these not because of the attributes, but because > these are unused. And when they are used, most of the time they don't > have their address taken, so their value is propagated to the point > where they are read and the constants again become unused. > > I don't think GCC isn't smart enough to do the same. Do you compile > with optimization? > Dmitri, as discussed in the other thread, it's smart enough, but only when optimization is turned on. There's no reason to needlessly make debug builds heavier than necessary. This is not a big deal either way.

1 0

Re: [mpich-discuss] [PATCH] Use attribute layout_compatible for pair types
by Dmitri Gribenko 09 Jan '13

09 Jan '13

On Wed, Jan 9, 2013 at 8:19 PM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: > Both implemented and pushed as d440abb and ac15f7a. Thanks. > > -Dave > > On Jan 1, 2013, at 11:14 PM CST, Jed Brown wrote: > >> In addition, I suggest guarding these definitions. Leaving these in increases the total number of symbols in an example executable linking PETSc by a factor of 2. (They're all read-only, but they're still there.) Clang is smart enough to remove these, presumably because it understands the special attributes. No, LLVM removes these not because of the attributes, but because these are unused. And when they are used, most of the time they don't have their address taken, so their value is propagated to the point where they are read and the constants again become unused. I don't think GCC isn't smart enough to do the same. Do you compile with optimization? Dmitri -- main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if (j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr(a)gmail.com>*/

1 0

Re: [mpich-discuss] [PATCH] Use attribute layout_compatible for pair types
by Dave Goodell 09 Jan '13

09 Jan '13

Both implemented and pushed as d440abb and ac15f7a. Thanks. -Dave On Jan 1, 2013, at 11:14 PM CST, Jed Brown wrote: > In addition, I suggest guarding these definitions. Leaving these in increases the total number of symbols in an example executable linking PETSc by a factor of 2. (They're all read-only, but they're still there.) Clang is smart enough to remove these, presumably because it understands the special attributes. > > static const MPI_Datatype mpich_mpi_float_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_float_int) = MPI_FLOAT_INT; > static const MPI_Datatype mpich_mpi_double_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_double_int) = MPI_DOUBLE_INT; > static const MPI_Datatype mpich_mpi_long_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_long_int) = MPI_LONG_INT; > static const MPI_Datatype mpich_mpi_short_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_short_int) = MPI_SHORT_INT; > static const MPI_Datatype mpich_mpi_2int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_2int) = MPI_2INT; > static const MPI_Datatype mpich_mpi_long_double_int MPICH_ATTR_TYPE_TAG_LAYOUT_COMPATIBLE(struct mpich_struct_mpi_long_double_int) = MPI_LONG_DOUBLE_INT; > > > > > On Tue, Jan 1, 2013 at 11:48 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: >> --- >> src/include/mpi.h.in | 12 ++++++------ >> 1 file changed, 6 insertions(+), 6 deletions(-) >>

1 0

Re: [mpich-discuss] low performance in an asynchronous, mixed MPI/pthreads app
by Geoffrey Irving 08 Jan '13

08 Jan '13

On Mon, Jan 7, 2013 at 11:09 PM, Geoffrey Irving <irving(a)naml.us> wrote: > On Mon, Jan 7, 2013 at 11:39 AM, Darius Buntinas <buntinas(a)mcs.anl.gov> wrote: >> Hi Geoff, >> >> Is it possible for you to try this with fewer worker threads per node (leaving some idle cores on each node)? > > Yes, this should have been an easy thing to try, but it crashed due to > passing a 0 pointer to MPI. That may indicate that I ran out of > memory and an unchecked malloc failed. I'll track that down and > retry. Here are results: 1 progress thread, 1 comm thread, 4 workers: 169.9694 s 1 progress thread, 1 comm thread, 3 workers: 160.2020 s This is using MPI_THREAD_MULTIPLE. Geoffrey

1 0

Re: [mpich-discuss] low performance in an asynchronous, mixed MPI/pthreads app
by Geoffrey Irving 08 Jan '13

08 Jan '13

On Mon, Jan 7, 2013 at 10:44 PM, Geoffrey Irving <irving(a)naml.us> wrote: > On Mon, Jan 7, 2013 at 10:53 AM, Dave Goodell <goodell(a)mcs.anl.gov> wrote: >> On Jan 6, 2013, at 8:57 PM CST, Geoffrey Irving wrote: >> >>> We're seeing much lower >>> performance than expected, and are trying to understand why. Here is >>> a visualization of a 96 core test run using 16 ranks with 6 threads >>> per core (4 Hopper nodes): >>> >>> http://naml.us/random/pentago/history-random17.png >>> >>> 1. All messages are sent using MPI_Isend, but Isend sometimes takes a >>> very long time to return even for tiny (8 byte) messages. For >>> example, the light blue "wakeup" messages are sent from a worker >>> thread to the master communication thread to indicate that computation >>> is complete, breaking the communication thread out of an MPI_Waitsome >>> call (using MPI_THREAD_MULTIPLE). Some of the input request Isends >>> also take a long time, and these are also 8 bytes. What would cause >>> MPI_Isend to take a long time to return, both for small messages >>> (wakeups and input requests) and for large messages (input responses >>> and output sends)? >> >> One possibility is because MPICH-nemesis (on which Cray's MPI for XE6 is based) does not have any fine-grained parallelism in the library. So in order for an Isend-ing thread to properly enter the MPI library, it must queue on a mutex until the thread which is currently occupying the progress engine decides to yield. In nemesis by default this is 10 iterations of {polling shared memory then the network module}, although Cray may have decided to tweak this number (I have not seen their source). So overly long netmod polling times could contribute to this delay. >> >> This design is simplistic, and we aren't especially happy with it, but so far we've found it to be very difficult to retrofit finer-grained parallelism into the existing nemesis codebase. My personal opinion is that a real fix will require a substantial rewrite of a large part of the lower level code. >> >> If I'm understanding your questions/explanations correctly, then this is likely related to issues 2 & 3 as well, possibly 4. >> >> […] >>> Some things that might contribute to the problem: >>> >>> 1. Since the 92K input responses are sent in response to 8 byte >>> request messages, I can prepost all matching Irecvs. However, output >>> messages arrive unexpectedly, so the best I can do is post a few >>> wildcard Irecvs (MPI_ANY_SOURCE, MPI_ANY_TAG on the output message >>> communicator). Jeff thought wildcard Irecvs might not be good enough >>> to hit the fast path. Moreover, output messages are not compressed, >>> so most are 262K. I didn't compress them historically because >>> compression isn't all that fast (I'm using Google's Snappy plus domain >>> specific preconditioning), but that may be a good experiment to run. >> >> Preposted any_source+any_tag irecvs should be on the fast-enough path. The size is large enough that it might be tipping you over into a rendezvous instead of eager protocol, but only Cray can answer that for sure. I probably wouldn't pursue compression as a first next step. > > And you're correct: at least in the current setup output compression > slows things down, probably in part because it results in more > intrarank messages (I had already started implementing this experiment > before I got your email). > >>> 2. I don't know how to visualize instantaneous bandwidth, even though >>> I have start and end times for all large messages (start and end of >>> Isend, Irecv request completion time). It's possible the problem is >>> bad network load balancing. I'm happy to whip up a plot if anyone has >>> suggestions as to what to draw. >> >> I'm not sure what the goal is here. 262 KiB messages don't strike me as large enough to worry much about bandwidth on a modern Cray machine. Latencies seem like a more useful metric still, esp. if you aren't seeing much variation as a function of message size. >> >>> 3. Waking a communication thread out of an MPI_Waitsome is somewhat of >>> a hack. It's the only time the worker threads touch MPI; otherwise I >>> could use MPI_THREAD_FUNNELED. However, the only alternative I know >>> is for the communication thread to poll on MPI_Testsome and check for >>> thread completion in between. Might that be better (it's an easy >>> experiment to run)? >> >> Frankly, it might, because of the aforementioned design. What non-MPI mechanism for inter-thread signaling are you planning on using? > > All the thread signalling is via pthread spinlocks and the occasional > lockless spin when I can get away with it. In the MPI_Testsome case > the communication thread would check a flag for inter-thread messages > without grabbing a lock, then grab a spinlock and recheck. > > I'll implement this tomorrow and see how it fares. Polling on MPI_Testsome and spinlocks and using MPI_THREAD_FUNNELED does significantly better than bare MPI_THREAD_MULTIPLE, and a little better than asynchronous progress mode: funneled, 1 comm thread and 5 workers: 157.0734 s multiple, 1 comm thread and 5 workers: 208.9402 s asynchronous progress, 1 progress thread, 1 comm thread, 4 workers: 169.9694 s Here are a few history trace screenshots (zooming in). The worker thread stalls is wakeup are gone of course, but otherwise it is similar. http://naml.us/random/pentago/history-funnel-random17.png http://naml.us/random/pentago/history-funnel-random17-1.png http://naml.us/random/pentago/history-funnel-random17-2.png I'm going to try on BlueGene next to see if the performance is similar. Geoffrey

1 0