November 2012 - discuss - mailman3.cels.anl.gov

Some problems with mpich.org website.
by Phil Scadden 13 Nov '12

13 Nov '12

Firstly, the website lacks any kind of webmaster link with which to report problems. Subscribing to this list seems the only way. Bug reports - the link is to https://trac.mpich.org/mpich/newticket This appears to be dead - (not found in DNS). The same problem applies to all links under "release Information". Downloads. Page http://www.mpich.org/downloads/ The links http://www.mpich.org/downloads/tarballs/1.4.1p1/mpich2-1.4.1p1-win-ia32.msi http://www.mpich.org/downloads/tarballs/1.4.1p1/mpich2-1.4.1p1-win-x86-64.m… do not work. I hope these can be fixed. Notice: This email and any attachments are confidential. If received in error please destroy and immediately notify us. Do not copy or disclose the contents.

1 0

Re: [mpich-discuss] Support for MIC in mpich2-1.5
by John Fettig 13 Nov '12

13 Nov '12

On Mon, Nov 5, 2012 at 9:37 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 11/05/12 13:12, John Fettig wrote: > >> I believe I have a working build, I'll append my cross file to the end >> of this email if anybody else wants to try it. >> > > Thanks! > > > I have a followup question: is there any support for launching jobs >> that use both the MIC and the host CPU? >> > > Yes. Once you have setup MPICH on both the host and MIC, you can launch > jobs across them. > > If you didn't pass any configure option, it'll use TCP/IP, which is very > slow. If you configure with --with-device=ch3:nemesis:scif, it'll use the > SCIF protocol, which is much faster. > I compiled examples/hellow.c for both the MIC and the host CPU, and copied it to the card. This seems to work: $ mpiexec -hosts 172.31.1.1:1,172.31.1.254:1 -n 1 ./hellow.mic : -n 1 ./hellow Hello world from process 1 of 2 Hello world from process 0 of 2 However, if I try to run more processes it crashes: $ mpiexec -hosts 172.31.1.1:3,172.31.1.254:3 -n 3 ./hellow.mic : -n 3 ./hellow Hello world from process 4 of 6 Hello world from process 0 of 6 Hello world from process 3 of 6 Hello world from process 1 of 6 0: 3: 00000033: 00000042: readv err 0 Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(293).................: MPI_Finalize failed MPI_Finalize(213).................: MPID_Finalize(117)................: MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close MPIDI_CH3I_Progress(367)..........: MPID_nem_mpich2_blocking_recv(904): state_commrdy_handler(175)........: state_commrdy_handler(138)........: MPID_nem_scif_recv_handler(115)...: Communication error with rank 3 MPID_nem_scif_recv_handler(35)....: scif_scif_read failed (scif_scif_read failed with error 'Success') 1: 3: 00000033: 00000042: readv err 0 Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(293).................: MPI_Finalize failed MPI_Finalize(213).................: MPID_Finalize(117)................: MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close MPIDI_CH3I_Progress(367)..........: MPID_nem_mpich2_blocking_recv(904): state_commrdy_handler(175)........: state_commrdy_handler(138)........: MPID_nem_scif_recv_handler(115)...: Communication error with rank 3 MPID_nem_scif_recv_handler(35)....: scif_scif_read failed (scif_scif_read failed with error 'Success') Hello world from process 5 of 6 Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(293).................: MPI_Finalize failed MPI_Finalize(213).................: MPID_Finalize(117)................: MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close MPIDI_CH3I_Progress(367)..........: MPID_nem_mpich2_blocking_recv(904): state_commrdy_handler(184)........: poll of socket fds failed Fatal error in MPI_Finalize: Other MPI error, error stack: MPI_Finalize(293).................: MPI_Finalize failed MPI_Finalize(213).................: MPID_Finalize(117)................: MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close MPIDI_CH3I_Progress(367)..........: MPID_nem_mpich2_blocking_recv(904): state_commrdy_handler(184)........: poll of socket fds failed =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:883): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:210): demux engine error waiting for event [mpiexec@host] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@host] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@host] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:216): launcher returned error waiting for completion [mpiexec@host] main (./ui/mpich/mpiexec.c:325): process manager error waiting for completion Any ideas? John

1 0

Re: [mpich-discuss] determinism in collectives
by Dave Goodell 13 Nov '12

13 Nov '12

Oops :( There's nothing new in my previous post relative to the other comments in this thread, please ignore me. I haven't updated my mail filters for the lists.mcs.anl.gov --> mpich.org switch yet, so I didn't see the rest of the thread after Nick's original email. -Dave On Nov 13, 2012, at 12:17 PM CST, Dave Goodell wrote: > On Nov 12, 2012, at 9:00 PM CST, Nichols A. Romero wrote: > >> Are collective operations, such as sums, using MPI_Reduce deterministic? In other words, if I run MPI_Reduce N times on P tasks, will the sums always occur in the same order. Thus it would be reproducible to the expected precision. > > In practice, this usually holds, and it's even recommended by the MPI standard (see MPI-3, sec. 5.9, p.175, l.9-13). However, you may not be able to capture all inputs to a collective routine, and it's entirely reasonable for future versions of MPICH (or any MPI library) to attempt to use adaptive methods for selecting collective algorithms. Such adaptive behavior could easily disturb the commutivity/associativity of your reduction collective.

1 0

Re: [mpich-discuss] determinism in collectives
by Dave Goodell 13 Nov '12

13 Nov '12

On Nov 12, 2012, at 9:00 PM CST, Nichols A. Romero wrote: > Are collective operations, such as sums, using MPI_Reduce deterministic? In other words, if I run MPI_Reduce N times on P tasks, will the sums always occur in the same order. Thus it would be reproducible to the expected precision. In practice, this usually holds, and it's even recommended by the MPI standard (see MPI-3, sec. 5.9, p.175, l.9-13). However, you may not be able to capture all inputs to a collective routine, and it's entirely reasonable for future versions of MPICH (or any MPI library) to attempt to use adaptive methods for selecting collective algorithms. Such adaptive behavior could easily disturb the commutivity/associativity of your reduction collective. -Dave

1 0

Announcing the availability of MPICH 3.0rc1
by Pavan Balaji 13 Nov '12

13 Nov '12

The MPICH team is pleased to announce the availability of the first release candidate (rc1) in the new major release series (mpich-3.0). http://www.mpich.org/ MPICH-3.0rc1 is the first preview of a major feature release. The primary focus of this release is to provide full support for the MPI-3 standard. But other features including support for ARM v7 native atomics are also included. This is a feature-preview release. MPICH2 developers, testers and other advanced users are encouraged to try this release out and let us know if you encounter any problems. This release is not intended for production systems at this time; such systems should continue to use the latest release from the 1.5.x branch (currently 1.5). The major additions in the 3.0rc1 release are highlighted below. Regards, The MPICH Team =============================================================================== Changes in 3.0rc1 =============================================================================== # MPI-3: All MPI-3 features are now implemented and the MPI_VERSION bumped up to 3.0. # OVERALL: Added support for ARM-v7 native atomics # MPE: MPE is now separated out of MPICH and can be downloaded/used as a separate package. # Several other minor bug fixes, memory leak fixes, and code cleanup. A full list of changes is available using: svn log -r10344:HEAD https://svn.mcs.anl.gov/repos/mpi/mpich2/tags/release/mpich-3.0rc1 ... or at the following link: https://trac.mcs.anl.gov/projects/mpich2/log/mpich2/tags/release/mpich-3.0r… -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] New communicator and MPI_Reduce...
by Pavan Balaji 13 Nov '12

13 Nov '12

On 11/13/2012 08:31 AM US Central Time, Alejandro Allievi wrote: > However, when I want to apply the reduction operation using the new > communicator as > > MPI_Reduce(&test_newcomm, &sum_newcomm, 1, MPI_INT, MPI_SUM, 0, > first_row_comm); > > It compiles fine but throws an execution error: > > Fatal error in PMPI_Reduce: Invalid communicator, error stack: > PMPI_Reduce(1270): MPI_Reduce(sbuf=000000000022FE38, > rbuf=000000000022FE30, count=1, MPI_INT, MPI_SUM, root=0, > MPI_COMM_NULL) failed > PMPI_Reduce(1160): Null communicator > > But communicator " first_row_comm " is properly create. Is this a bug > or I'm doing something wrong with the way I'm passing the > new communicator argument in MPI_Reduce, or...?? It looks like you didn't create first_row_comm correctly. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

New communicator and MPI_Reduce...
by Alejandro Allievi 13 Nov '12

13 Nov '12

Good morning, I've created a new communicator "first_row_comm" which is the 1st row of a mesh of processes contained in MPI_COMM_WORLD. Everything works fine in the reduction operation involving the processes in the new communicator if I use: MPI_Reduce(&test_newcomm, &sum_newcomm, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); However, when I want to apply the reduction operation using the new communicator as MPI_Reduce(&test_newcomm, &sum_newcomm, 1, MPI_INT, MPI_SUM, 0, first_row_comm); It compiles fine but throws an execution error: Fatal error in PMPI_Reduce: Invalid communicator, error stack: PMPI_Reduce(1270): MPI_Reduce(sbuf=000000000022FE38, rbuf=000000000022FE30, count=1, MPI_INT, MPI_SUM, root=0, MPI_COMM_NULL) failed PMPI_Reduce(1160): Null communicator But communicator " first_row_comm " is properly create. Is this a bug or I'm doing something wrong with the way I'm passing the new communicator argument in MPI_Reduce, or...?? Thanks for any hints!! Alejandro

1 0

Re: [mpich-discuss] determinism in collectives
by Nichols A. Romero 12 Nov '12

12 Nov '12

What about rectangular communicators? Presumably MPI_SUM is optimized even for the MPI_DOUBLE_COMPLEX on a rectangular communicator on BG/Q. ----- Original Message ----- > From: "Jeff Hammond" <jhammond(a)alcf.anl.gov> > To: "Pavan Balaji" <balaji(a)mcs.anl.gov> > Cc: discuss(a)mpich.org > Sent: Monday, November 12, 2012 9:47:29 PM > Subject: Re: [mpich-discuss] determinism in collectives > Yes, what I was saying before is that we know that MPI on Blue Gene/Q > drops into the MPICH2 algorithm for MPI_Reduce on for > MPI_DOUBLE_COMPLEX with MPI_SUM on irregular communicators and that > MPIC_Send and MPIC_Recv do not change their behavior on this machine, > so whatever is true of stock MPICH2 1.4 is true for MPI on Blue > Gene/Q. > > This confirms what we suspected, which is that BGQ-MPI has > deterministic reductions in this case because MPICH2 does. > > Jeff > > On Mon, Nov 12, 2012 at 9:43 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > > > > The default machine-independent implementation of MPICH is > > deterministic. However, each device can (and most times does) > > implement > > its own version of MPI_Reduce. We have no control on them. > > > > -- Pavan > > > > On 11/12/2012 09:40 PM US Central Time, Jeff Hammond wrote: > >> We are primarily interested in MPICH2, as the motivation for this > >> question relates to a particular supercomputer that uses MPICH2 > >> and, > >> in particular, the MPICH2 implementation of MPI_Reduce for > >> (comm,type,op) of interest to us. > >> > >> Jeff > >> > >> On Mon, Nov 12, 2012 at 9:35 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > >> wrote: > >>> > >>> The MPI standard does not guarantee this. For example, an > >>> implementation could use topology-aware reductions or an > >>> MPI_ANY_SOURCE > >>> while waiting for messages from peers making the order in which > >>> the > >>> reduction operations are applied different for each run. > >>> > >>> -- Pavan > >>> > >>> On 11/12/2012 09:00 PM US Central Time, Nichols A. Romero wrote: > >>>> Hi, > >>>> > >>>> I apologize if this has been asked a trillion times, if its > >>>> documented somewhere just point it out to me online. > >>>> > >>>> Are collective operations, such as sums, using MPI_Reduce > >>>> deterministic? In other words, if I run MPI_Reduce N times on P > >>>> tasks, will the sums always occur in the same order. Thus it > >>>> would be reproducible to the expected precision. > >>>> > >>> > >>> -- > >>> Pavan Balaji > >>> http://www.mcs.anl.gov/~balaji > >>> _______________________________________________ > >>> discuss mailing list discuss(a)mpich.org > >>> To manage subscription options or unsubscribe: > >>> https://lists.mpich.org/mailman/listinfo/discuss > >> > >> > >> > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > > > -- > Jeff Hammond > Argonne Leadership Computing Facility > University of Chicago Computation Institute > jhammond(a)alcf.anl.gov / (630) 252-5381 > http://www.linkedin.com/in/jeffhammond > https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Nichols A. Romero, Ph.D. Argonne Leadership Computing Facility Argonne National Laboratory Building 240 Room 2-127 9700 South Cass Avenue Argonne, IL 60490 (630) 252-3441

1 0

Re: [mpich-discuss] determinism in collectives
by Jeff Hammond 12 Nov '12

12 Nov '12

Yes, what I was saying before is that we know that MPI on Blue Gene/Q drops into the MPICH2 algorithm for MPI_Reduce on for MPI_DOUBLE_COMPLEX with MPI_SUM on irregular communicators and that MPIC_Send and MPIC_Recv do not change their behavior on this machine, so whatever is true of stock MPICH2 1.4 is true for MPI on Blue Gene/Q. This confirms what we suspected, which is that BGQ-MPI has deterministic reductions in this case because MPICH2 does. Jeff On Mon, Nov 12, 2012 at 9:43 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > The default machine-independent implementation of MPICH is > deterministic. However, each device can (and most times does) implement > its own version of MPI_Reduce. We have no control on them. > > -- Pavan > > On 11/12/2012 09:40 PM US Central Time, Jeff Hammond wrote: >> We are primarily interested in MPICH2, as the motivation for this >> question relates to a particular supercomputer that uses MPICH2 and, >> in particular, the MPICH2 implementation of MPI_Reduce for >> (comm,type,op) of interest to us. >> >> Jeff >> >> On Mon, Nov 12, 2012 at 9:35 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >>> >>> The MPI standard does not guarantee this. For example, an >>> implementation could use topology-aware reductions or an MPI_ANY_SOURCE >>> while waiting for messages from peers making the order in which the >>> reduction operations are applied different for each run. >>> >>> -- Pavan >>> >>> On 11/12/2012 09:00 PM US Central Time, Nichols A. Romero wrote: >>>> Hi, >>>> >>>> I apologize if this has been asked a trillion times, if its documented somewhere just point it out to me online. >>>> >>>> Are collective operations, such as sums, using MPI_Reduce deterministic? In other words, if I run MPI_Reduce N times on P tasks, will the sums always occur in the same order. Thus it would be reproducible to the expected precision. >>>> >>> >>> -- >>> Pavan Balaji >>> http://www.mcs.anl.gov/~balaji >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond

1 0

Re: [mpich-discuss] determinism in collectives
by Pavan Balaji 12 Nov '12

12 Nov '12

The default machine-independent implementation of MPICH is deterministic. However, each device can (and most times does) implement its own version of MPI_Reduce. We have no control on them. -- Pavan On 11/12/2012 09:40 PM US Central Time, Jeff Hammond wrote: > We are primarily interested in MPICH2, as the motivation for this > question relates to a particular supercomputer that uses MPICH2 and, > in particular, the MPICH2 implementation of MPI_Reduce for > (comm,type,op) of interest to us. > > Jeff > > On Mon, Nov 12, 2012 at 9:35 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: >> >> The MPI standard does not guarantee this. For example, an >> implementation could use topology-aware reductions or an MPI_ANY_SOURCE >> while waiting for messages from peers making the order in which the >> reduction operations are applied different for each run. >> >> -- Pavan >> >> On 11/12/2012 09:00 PM US Central Time, Nichols A. Romero wrote: >>> Hi, >>> >>> I apologize if this has been asked a trillion times, if its documented somewhere just point it out to me online. >>> >>> Are collective operations, such as sums, using MPI_Reduce deterministic? In other words, if I run MPI_Reduce N times on P tasks, will the sums always occur in the same order. Thus it would be reproducible to the expected precision. >>> >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0