- discuss - mailman3.cels.anl.gov

Fwd: Problem in Building MPICH
by Ali Behjatian 12 Apr '13

12 Apr '13

Hi, I tried to follow the instructions in the MPICH2 Installer's Guide but I received an error after running "make 2>&1 | tee m.txt". I appreciate if you help me to fix this problem. Thanks Ali

1 0

Re: [mpich-discuss] Merging stdout from multiple ranks
by Pavan Balaji 12 Apr '13

12 Apr '13

On 04/12/2013 11:48 AM US Central Time, Rob Latham wrote: > On Tue, Apr 09, 2013 at 09:57:12AM -0500, Pavan Balaji wrote: >> Not when everything is dumped to the same output from mpiexec (e.g., >> stdout), since that'll require an arbitrary amount of buffering at >> mpiexec to do that. You could redirect the output to separate per-rank >> files. mpiexec -help should give you information on that. > > Oh man how did I not know that? What is the correct > '-outfile-pattern' to use? the -help doesn't tell you anything > about the "pattern language" You can get more information on that: % mpiexec -outfile-pattern -help -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] Merging stdout from multiple ranks
by Rob Latham 12 Apr '13

12 Apr '13

On Tue, Apr 09, 2013 at 09:57:12AM -0500, Pavan Balaji wrote: > Not when everything is dumped to the same output from mpiexec (e.g., > stdout), since that'll require an arbitrary amount of buffering at > mpiexec to do that. You could redirect the output to separate per-rank > files. mpiexec -help should give you information on that. Oh man how did I not know that? What is the correct '-outfile-pattern' to use? the -help doesn't tell you anything about the "pattern language" ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

Re: [mpich-discuss] mpich-3.0.3 ARMCI compile warnings
by Jim Dinan 09 Apr '13

09 Apr '13

Hi Jeff, Thanks for the bug report. I created a ticket for this; should be an easy fix. https://trac.mpich.org/projects/mpich/ticket/1807 ~Jim. On 04/09/2013 06:09 PM, Daily, Jeff A wrote: > On NERSCï¿½s hopper I get the following warnings when compiling the ARMCI > sources using the default module environment. > > > > CC parmci.lo > > PGC-W-0119-void function ARMCI_Barrier cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 52) > > PGC-W-0119-void function ARMCI_Fence cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 57) > > PGC-W-0119-void function ARMCI_AllFence cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 62) > > PGC-W-0119-void function ARMCI_Access_begin cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 67) > > PGC-W-0119-void function ARMCI_Access_end cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 72) > > PGC-W-0119-void function ARMCI_Lock cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 262) > > PGC-W-0119-void function ARMCI_Unlock cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 267) > > PGC-W-0119-void function armci_msg_barrier cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 277) > > PGC-W-0119-void function armci_msg_group_barrier cannot return value > (../mpich-3.0.3/src/armci/src/parmci.c: 282) > > PGC/x86-64 Linux 12.9-0: compilation completed with warnings > > > > Looks like the when the profiling interface was generated, all functions > were assumed to return a value. I hope itï¿½s an easy fix. >

1 0

mpich-3.0.3 ARMCI compile warnings
by Daily, Jeff A 09 Apr '13

09 Apr '13

MPICH Team, On NERSC's hopper I get the following warnings when compiling the ARMCI sources using the default module environment. CC parmci.lo PGC-W-0119-void function ARMCI_Barrier cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 52) PGC-W-0119-void function ARMCI_Fence cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 57) PGC-W-0119-void function ARMCI_AllFence cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 62) PGC-W-0119-void function ARMCI_Access_begin cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 67) PGC-W-0119-void function ARMCI_Access_end cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 72) PGC-W-0119-void function ARMCI_Lock cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 262) PGC-W-0119-void function ARMCI_Unlock cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 267) PGC-W-0119-void function armci_msg_barrier cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 277) PGC-W-0119-void function armci_msg_group_barrier cannot return value (../mpich-3.0.3/src/armci/src/parmci.c: 282) PGC/x86-64 Linux 12.9-0: compilation completed with warnings Looks like the when the profiling interface was generated, all functions were assumed to return a value. I hope it's an easy fix. __________________________________________________ Jeff Daily Scientist Data Intensive Scientific Computing Pacific Northwest National Laboratory 902 Battelle Boulevard P.O. Box 999, MSIN K7-90 Richland, WA 99352 USA Tel: 509-372-6548 Fax: 509-372-4720 jeff.daily(a)pnnl.gov www.pnnl.gov

1 0

Re: [mpich-discuss] MPICH (3.0.3) w/ SCIF
by Pavan Balaji 09 Apr '13

09 Apr '13

Hi Eric, On 04/09/2013 02:39 PM US Central Time, Eric A. Borisch wrote: > Any thoughts? Or is this something to take up with Intel? Compiler is > latest (13.1.1) icc; latest MPSS (2-2.1.5889-14); Centos 6.4. Much of this code has been contributed by Intel, and they'd be the best ones to answer this question. I did notice some other errors as well, so we'll be discussing this with them as well, though we don't have an ETA on the fixes. Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

MPICH (3.0.3) w/ SCIF
by Eric A. Borisch 09 Apr '13

09 Apr '13

I'm trying to get MPICH (3.0.3) and SCIF working (on an Intel PHI card.) I'm using the tests from osu_benchmarks(from mvapich2 tarball) as a set of sanity checks, and I'm running into some unexpected errors. One example: running osu_mbw_mr works sometimes, and then fail on the next try. The printout from two successive runs as well as the hosts file are below. I especially like the "scif_scif_read failed with error 'Success'" message. :) Any thoughts? Or is this something to take up with Intel? Compiler is latest (13.1.1) icc; latest MPSS (2-2.1.5889-14); Centos 6.4. Thanks, Eric This particular test should be setting up four pairs of processes, each with one element on the host, and one on the PHI, and try to communicate between the two (node 0<->4, ...) Things seem to be more stable with one or two pairs of processes, but that's not really the desired use case... [eborisch@rt5 osu_benchmarks]$ mpiexec -map-by rr -n 4 native/osu_mbw_mr : -n 4 mic/osu_mbw_mr # OSU MPI Multiple Bandwidth / Message Rate Test # [ pairs: 4 ] [ window size: 64 ] # Size MB/s Messages/s 1 0.23 230138.66 2 0.46 229579.06 4 0.92 231201.97 8 1.85 231515.16 16 3.63 226781.39 32 6.92 216285.18 64 12.65 197678.16 128 25.21 196946.20 256 50.46 197106.54 512 86.11 168184.28 1024 132.69 129577.13 2048 180.60 88183.67 4096 179.81 43898.89 8192 358.07 43710.21 16384 696.33 42500.74 32768 1364.41 41638.46 65536 2737.42 41769.74 131072 4657.86 35536.68 262144 6160.59 23500.77 524288 6584.39 12558.73 1048576 6690.91 6380.95 2097152 6782.58 3234.18 4194304 6789.25 1618.68 ( Note: Seems ~reasonable for pushing data one direction over 16x PCIe 2.0) [eborisch@rt5 osu_benchmarks]$ mpiexec -map-by rr -n 4 native/osu_mbw_mr : -n 4 mic/osu_mbw_mr # OSU MPI Multiple Bandwidth / Message Rate Test # [ pairs: 4 ] [ window size: 64 ] # Size MB/s Messages/s 0: 5: 00000051: 00000060: readv err 0 0: 5: 00000052: 00000060: readv err 0 0: 5: 00000053: 00000060: readv err 0 Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(426)................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(283)...........: MPIR_Barrier_or_coll_fn(121).....: MPIR_Barrier_intra(83)...........: MPIC_Sendrecv(209)...............: MPIC_Wait(563)...................: MPIDI_CH3I_Progress(367).........: MPID_nem_mpich_blocking_recv(894): state_commrdy_handler(175).......: state_commrdy_handler(138).......: MPID_nem_scif_recv_handler(115)..: Communication error with rank 5 MPID_nem_scif_recv_handler(35)...: scif_scif_read failed (scif_scif_read failed with error 'Success') MPIR_Barrier_intra(83)...........: MPIC_Sendrecv(209)...............: MPIC_Wait(563)...................: MPIDI_CH3I_Progress(367).........: MPID_nem_mpich_blocking_recv(894): state_commrdy_handler(175).......: state_commrdy_handler(138).......: MPID_nem_scif_recv_handler(115)..: Communication error with rank 5 MPID_nem_scif_recv_handler(35)...: scif_scif_read failed (scif_scif_read failed with error 'Success') MPIR_Barrier_impl(294)...........: MPIR_Barrier_or_coll_fn(121).....: MPIR_Barrier_intra(83)...........: MPIC_Sendrecv(209)...............: MPIC_Wait(563)...................: MPIDI_CH3I_Progress(367).........: MPID_nem_mpich_blocking_recv(894): state_commrdy_handler(175).......: state_commrdy_handler(138).......: MPID_nem_scif_recv_handler(115)..: Communication error with rank 5 MPID_nem_scif_recv_handler(35)...: scif_scif_read failed (scif_scif_read failed with error 'Success') MPIR_Barrier_impl(308)...........: MPIR_Bcast_impl(1369)............: MPIR_Bcast_intra(1199)...........: MPIR_Bcast_binomial(220).........: Failure during collective =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYDU_sock_write (./utils/sock/sock.c:291): write error (Broken pipe) [proxy:0:[email protected]] stdoe_cb (./pm/pmiserv/pmip_cb.c:63): sock write error [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@rt5] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@rt5] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@rt5] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec@rt5] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Here's the host file (unchanged between runs): host:4 mic0:4 binding=user:4,8,12,16

1 0

Re: [mpich-discuss] Merging stdout from multiple ranks
by Matthieu Dorier 09 Apr '13

09 Apr '13

That would be fine too, thanks! Matthieu ----- Mail original ----- > De: "Pavan Balaji" <balaji(a)mcs.anl.gov> > À: discuss(a)mpich.org > Cc: "Matthieu Dorier" <matthieu.dorier(a)irisa.fr> > Envoyé: Mardi 9 Avril 2013 16:57:12 > Objet: Re: [mpich-discuss] Merging stdout from multiple ranks > > > On 04/09/2013 09:50 AM US Central Time, Matthieu Dorier wrote: > > Is there any option in Mpich to make sure a line sent to stdout > > from one > > rank is not broken by a line sent by another rank when mpirun > > merges the > > output of all ranks? > > Not when everything is dumped to the same output from mpiexec (e.g., > stdout), since that'll require an arbitrary amount of buffering at > mpiexec to do that. You could redirect the output to separate > per-rank > files. mpiexec -help should give you information on that. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji >

1 0

Re: [mpich-discuss] Merging stdout from multiple ranks
by Pavan Balaji 09 Apr '13

09 Apr '13

On 04/09/2013 09:50 AM US Central Time, Matthieu Dorier wrote: > Is there any option in Mpich to make sure a line sent to stdout from one > rank is not broken by a line sent by another rank when mpirun merges the > output of all ranks? Not when everything is dumped to the same output from mpiexec (e.g., stdout), since that'll require an arbitrary amount of buffering at mpiexec to do that. You could redirect the output to separate per-rank files. mpiexec -help should give you information on that. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Merging stdout from multiple ranks
by Matthieu Dorier 09 Apr '13

09 Apr '13

Hi, Is there any option in Mpich to make sure a line sent to stdout from one rank is not broken by a line sent by another rank when mpirun merges the output of all ranks? Thanks! Matthieu Dorier PhD student at ENS Cachan Brittany and IRISA http://people.irisa.fr/Matthieu.Dorier

1 0