April 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH (3.0.3) w/ SCIF
by Pavan Balaji 09 Apr '13

09 Apr '13

Hi Eric, On 04/09/2013 02:39 PM US Central Time, Eric A. Borisch wrote: > Any thoughts? Or is this something to take up with Intel? Compiler is > latest (13.1.1) icc; latest MPSS (2-2.1.5889-14); Centos 6.4. Much of this code has been contributed by Intel, and they'd be the best ones to answer this question. I did notice some other errors as well, so we'll be discussing this with them as well, though we don't have an ETA on the fixes. Thanks, -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

MPICH (3.0.3) w/ SCIF
by Eric A. Borisch 09 Apr '13

09 Apr '13

I'm trying to get MPICH (3.0.3) and SCIF working (on an Intel PHI card.) I'm using the tests from osu_benchmarks(from mvapich2 tarball) as a set of sanity checks, and I'm running into some unexpected errors. One example: running osu_mbw_mr works sometimes, and then fail on the next try. The printout from two successive runs as well as the hosts file are below. I especially like the "scif_scif_read failed with error 'Success'" message. :) Any thoughts? Or is this something to take up with Intel? Compiler is latest (13.1.1) icc; latest MPSS (2-2.1.5889-14); Centos 6.4. Thanks, Eric This particular test should be setting up four pairs of processes, each with one element on the host, and one on the PHI, and try to communicate between the two (node 0<->4, ...) Things seem to be more stable with one or two pairs of processes, but that's not really the desired use case... [eborisch@rt5 osu_benchmarks]$ mpiexec -map-by rr -n 4 native/osu_mbw_mr : -n 4 mic/osu_mbw_mr # OSU MPI Multiple Bandwidth / Message Rate Test # [ pairs: 4 ] [ window size: 64 ] # Size MB/s Messages/s 1 0.23 230138.66 2 0.46 229579.06 4 0.92 231201.97 8 1.85 231515.16 16 3.63 226781.39 32 6.92 216285.18 64 12.65 197678.16 128 25.21 196946.20 256 50.46 197106.54 512 86.11 168184.28 1024 132.69 129577.13 2048 180.60 88183.67 4096 179.81 43898.89 8192 358.07 43710.21 16384 696.33 42500.74 32768 1364.41 41638.46 65536 2737.42 41769.74 131072 4657.86 35536.68 262144 6160.59 23500.77 524288 6584.39 12558.73 1048576 6690.91 6380.95 2097152 6782.58 3234.18 4194304 6789.25 1618.68 ( Note: Seems ~reasonable for pushing data one direction over 16x PCIe 2.0) [eborisch@rt5 osu_benchmarks]$ mpiexec -map-by rr -n 4 native/osu_mbw_mr : -n 4 mic/osu_mbw_mr # OSU MPI Multiple Bandwidth / Message Rate Test # [ pairs: 4 ] [ window size: 64 ] # Size MB/s Messages/s 0: 5: 00000051: 00000060: readv err 0 0: 5: 00000052: 00000060: readv err 0 0: 5: 00000053: 00000060: readv err 0 Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(426)................: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(283)...........: MPIR_Barrier_or_coll_fn(121).....: MPIR_Barrier_intra(83)...........: MPIC_Sendrecv(209)...............: MPIC_Wait(563)...................: MPIDI_CH3I_Progress(367).........: MPID_nem_mpich_blocking_recv(894): state_commrdy_handler(175).......: state_commrdy_handler(138).......: MPID_nem_scif_recv_handler(115)..: Communication error with rank 5 MPID_nem_scif_recv_handler(35)...: scif_scif_read failed (scif_scif_read failed with error 'Success') MPIR_Barrier_intra(83)...........: MPIC_Sendrecv(209)...............: MPIC_Wait(563)...................: MPIDI_CH3I_Progress(367).........: MPID_nem_mpich_blocking_recv(894): state_commrdy_handler(175).......: state_commrdy_handler(138).......: MPID_nem_scif_recv_handler(115)..: Communication error with rank 5 MPID_nem_scif_recv_handler(35)...: scif_scif_read failed (scif_scif_read failed with error 'Success') MPIR_Barrier_impl(294)...........: MPIR_Barrier_or_coll_fn(121).....: MPIR_Barrier_intra(83)...........: MPIC_Sendrecv(209)...............: MPIC_Wait(563)...................: MPIDI_CH3I_Progress(367).........: MPID_nem_mpich_blocking_recv(894): state_commrdy_handler(175).......: state_commrdy_handler(138).......: MPID_nem_scif_recv_handler(115)..: Communication error with rank 5 MPID_nem_scif_recv_handler(35)...: scif_scif_read failed (scif_scif_read failed with error 'Success') MPIR_Barrier_impl(308)...........: MPIR_Bcast_impl(1369)............: MPIR_Bcast_intra(1199)...........: MPIR_Bcast_binomial(220).........: Failure during collective =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:[email protected]] HYDU_sock_write (./utils/sock/sock.c:291): write error (Broken pipe) [proxy:0:[email protected]] stdoe_cb (./pm/pmiserv/pmip_cb.c:63): sock write error [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@rt5] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@rt5] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@rt5] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec@rt5] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Here's the host file (unchanged between runs): host:4 mic0:4 binding=user:4,8,12,16

1 0

Re: [mpich-discuss] Merging stdout from multiple ranks
by Matthieu Dorier 09 Apr '13

09 Apr '13

That would be fine too, thanks! Matthieu ----- Mail original ----- > De: "Pavan Balaji" <balaji(a)mcs.anl.gov> > À: discuss(a)mpich.org > Cc: "Matthieu Dorier" <matthieu.dorier(a)irisa.fr> > Envoyé: Mardi 9 Avril 2013 16:57:12 > Objet: Re: [mpich-discuss] Merging stdout from multiple ranks > > > On 04/09/2013 09:50 AM US Central Time, Matthieu Dorier wrote: > > Is there any option in Mpich to make sure a line sent to stdout > > from one > > rank is not broken by a line sent by another rank when mpirun > > merges the > > output of all ranks? > > Not when everything is dumped to the same output from mpiexec (e.g., > stdout), since that'll require an arbitrary amount of buffering at > mpiexec to do that. You could redirect the output to separate > per-rank > files. mpiexec -help should give you information on that. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji >

1 0

Re: [mpich-discuss] Merging stdout from multiple ranks
by Pavan Balaji 09 Apr '13

09 Apr '13

On 04/09/2013 09:50 AM US Central Time, Matthieu Dorier wrote: > Is there any option in Mpich to make sure a line sent to stdout from one > rank is not broken by a line sent by another rank when mpirun merges the > output of all ranks? Not when everything is dumped to the same output from mpiexec (e.g., stdout), since that'll require an arbitrary amount of buffering at mpiexec to do that. You could redirect the output to separate per-rank files. mpiexec -help should give you information on that. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Merging stdout from multiple ranks
by Matthieu Dorier 09 Apr '13

09 Apr '13

Hi, Is there any option in Mpich to make sure a line sent to stdout from one rank is not broken by a line sent by another rank when mpirun merges the output of all ranks? Thanks! Matthieu Dorier PhD student at ENS Cachan Brittany and IRISA http://people.irisa.fr/Matthieu.Dorier

1 0

Re: [mpich-discuss] command.so: wmpiconfig.exe program....
by Jayesh Krishna 08 Apr '13

08 Apr '13

Hi, The wmpiconfig program was designed to search for PCs, with MPICH2 installed, in a workgroup. However there are some bugs in the code and we don't have any developer cycles to fix it. We don't have any specific developer documents pertaining to the wmpiconfig utility. I would recommend taking a look at the MPICH2 source (/src/pm/smpd/wmpiconfig/*). Regards, Jayesh ----- Original Message ----- From: "seungkyun lim" <seungkyun9021(a)gmail.com> To: discuss(a)mpich.org Sent: Monday, April 8, 2013 5:27:10 AM Subject: [mpich-discuss] command.so: wmpiconfig.exe program.... [Specification of our environment] OS: Windows 7 (64bit) Two workstations: Intel CPU (64bit) Programming environment: Visual studio 2010 (64bit) hi. I use MPICH2 64 bit in windows 64bit. and I've already finished sample test for communicate and Master PC and slave PCs. now, I need command which can search slave PCs. It can communicate with saster PC from CMD program Is there any command of CMDprogram to search a slaves PCs in workgroup from master PC? lastly, I'd appreciate if you let me know some materials to make wmpiconfig.exe thank you ~ -- 임승균 올림 연락처 : 010-7979-4456 E-Mail: seungkyun9021(a)gmail.com , tmdrbs9021(a)naver.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Issue related to MPI_Comm_Spawn_Multiple
by Dave Goodell 08 Apr '13

08 Apr '13

On Apr 8, 2013, at 4:47 AM CDT, Mahesh Doijade <maheshdoijade(a)gmail.com> wrote: > Is there any upper limit for command line arguments passed to MPI_Comm_Spawn_Multiple. There is no intentional, specific upper limit, though there will probably be two limits in practice: * an OS limit on number of arguments and/or total size (in bytes) of arguments: http://www.in-ulm.de/~mascheck/various/argmax/ * some limitations in our internal PMI protocols, which hopefully are at least being error checked right now The first limit will probably result in an understandable error message. I have no idea what will happen if you hit the second limit. -Dave

1 0

command.so: wmpiconfig.exe program....
by seungkyun lim 08 Apr '13

08 Apr '13

[Specification of our environment] OS: Windows 7 (64bit)Two workstations: Intel CPU (64bit) Programming environment: Visual studio 2010 (64bit) hi.**** I use MPICH2 64 bit in windows 64bit.**** and I've already finished sample test for communicate and Master PC and slave PCs.**** now, I need command which can search slave PCs. It can communicate with saster PC from CMD program**** Is there any command of CMDprogram to search a slaves PCs in workgroup from master PC?**** lastly, I'd appreciate if you let me know some materials to make wmpiconfig.exe**** thank you ~ -- 임승균 올림 연락처 : 010-7979-4456 E-Mail: seungkyun9021(a)gmail.com, *tmdrbs9021(a)naver.com*

1 0

Issue related to MPI_Comm_Spawn_Multiple
by Mahesh Doijade 08 Apr '13

08 Apr '13

Is there any upper limit for command line arguments passed to MPI_Comm_Spawn_Multiple. -- Regards, -- Mahesh Doijade

1 0

Re: [mpich-discuss] run more then 1 process
by Pavan Balaji 08 Apr '13

08 Apr '13

Please keep discuss(a)mpich.org cc'ed on your emails. Also, you are using mpiexec (hydra) incorrectly. mpiexec -help will give you more information on how to use it. Also see: http://wiki.mpich.org/mpich/index.php/Using_the_Hydra_Process_Manager -- Pavan On 04/08/2013 12:14 AM US Central Time, Иващук Роман wrote: > Good afternoon. Here are some mistakes with hydra and smpd. > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0