On Mon, Nov 5, 2012 at 9:37 PM, Pavan Balaji <balaji@mcs.anl.gov> wrote:

On 11/05/12 13:12, John Fettig wrote:

I believe I have a working build, I'll append my cross file to the end
of this email if anybody else wants to try it.

Thanks!

I have a followup question: is there any support for launching jobs
that use both the MIC and the host CPU?

Yes. Once you have setup MPICH on both the host and MIC, you can launch jobs across them.

If you didn't pass any configure option, it'll use TCP/IP, which is very slow. If you configure with --with-device=ch3:nemesis:scif, it'll use the SCIF protocol, which is much faster.

I compiled examples/hellow.c for both the MIC and the host CPU, and copied it to the card. This seems to work:

$ mpiexec -hosts 172.31.1.1:1,172.31.1.254:1 -n 1 ./hellow.mic : -n 1 ./hellow
Hello world from process 1 of 2
Hello world from process 0 of 2

However, if I try to run more processes it crashes:

$ mpiexec -hosts 172.31.1.1:3,172.31.1.254:3 -n 3 ./hellow.mic : -n 3 ./hellow
Hello world from process 4 of 6
Hello world from process 0 of 6
Hello world from process 3 of 6
Hello world from process 1 of 6
0: 3: 00000033: 00000042: readv err 0
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(293).................: MPI_Finalize failed
MPI_Finalize(213).................:
MPID_Finalize(117)................:
MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3I_Progress(367)..........:
MPID_nem_mpich2_blocking_recv(904):
state_commrdy_handler(175)........:
state_commrdy_handler(138)........:
MPID_nem_scif_recv_handler(115)...: Communication error with rank 3
MPID_nem_scif_recv_handler(35)....: scif_scif_read failed (scif_scif_read failed with error 'Success')
1: 3: 00000033: 00000042: readv err 0
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(293).................: MPI_Finalize failed
MPI_Finalize(213).................:
MPID_Finalize(117)................:
MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3I_Progress(367)..........:
MPID_nem_mpich2_blocking_recv(904):
state_commrdy_handler(175)........:
state_commrdy_handler(138)........:
MPID_nem_scif_recv_handler(115)...: Communication error with rank 3
MPID_nem_scif_recv_handler(35)....: scif_scif_read failed (scif_scif_read failed with error 'Success')
Hello world from process 5 of 6
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(293).................: MPI_Finalize failed
MPI_Finalize(213).................:
MPID_Finalize(117)................:
MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3I_Progress(367)..........:
MPID_nem_mpich2_blocking_recv(904):
state_commrdy_handler(184)........: poll of socket fds failed
Fatal error in MPI_Finalize: Other MPI error, error stack:
MPI_Finalize(293).................: MPI_Finalize failed
MPI_Finalize(213).................:
MPID_Finalize(117)................:
MPIDI_CH3U_VC_WaitForClose(385)...: an error occurred while the device was waiting for all open connections to close
MPIDI_CH3I_Progress(367)..........:
MPID_nem_mpich2_blocking_recv(904):
state_commrdy_handler(184)........: poll of socket fds failed

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 1
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
[proxy:0:0@mic0.local] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:883): assert (!closed) failed
[proxy:0:0@mic0.local] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status
[proxy:0:0@mic0.local] main (./pm/pmiserv/pmip.c:210): demux engine error waiting for event
[mpiexec@host] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting
[mpiexec@host] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion
[mpiexec@host] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:216): launcher returned error waiting for completion
[mpiexec@host] main (./ui/mpich/mpiexec.c:325): process manager error waiting for completion

Any ideas?

John