Hi, Abhishek, As mentioned in previous email, it looks like the problem lies in the application. Is it possible that you can provide us with a minimum example that fails? So that we can looked at the code and reproduce the problem on our machines. Thanks, — Huiwei On Sep 17, 2014, at 3:29 PM, Abhishek Bhat <[email protected]> wrote:

Sangmin,

Fatal error in MPI_Recv: A process has failed, error stack: MPI_Recv(187).............: MPI_Recv(buf=0x7fff21bc04b0, count=644490, MPI_REAL, src=1, tag=14131, MPI_COMM_WORLD, status=0x7fff227c47a0) failed dequeue_and_set_error(865): Communication error with rank 1

=================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 4183 RUNNING AT dfw-camx = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:4@dfw-camx-n4] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:4@dfw-camx-n4] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:4@dfw-camx-n4] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:0:2@dfw-camx-n2] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:2@dfw-camx-n2] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:2@dfw-camx-n2] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:0:3@dfw-camx-n3] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:3@dfw-camx-n3] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:3@dfw-camx-n3] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:0:6@dfw-camx-n6] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:6@dfw-camx-n6] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:6@dfw-camx-n6] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:0:5@dfw-camx-n5] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:5@dfw-camx-n5] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:5@dfw-camx-n5] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [proxy:0:7@dfw-camx-n7] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:885): assert (!closed) failed [proxy:0:7@dfw-camx-n7] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status [proxy:0:7@dfw-camx-n7] main (pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@dfw-camx] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@dfw-camx] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@dfw-camx] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:218): launcher returned error waiting for completion [mpiexec@dfw-camx] main (ui/mpich/mpiexec.c:344): process manager error waiting for completion

This is what I got from the exitallcodes.

Anything helpful??

Thank you very much for all help Abhishek

…………………………………………………………………………………………………. Abhishek Bhat, PhD, EPI, Senior Consultant

From: Seo, Sangmin [mailto:[email protected]] Sent: Wednesday, September 17, 2014 1:17 PM To: Abhishek Bhat Subject: Re: [mpich-discuss] Error Running MPICH for Photochemical Modeling

On Sep 17, 2014, at 1:08 PM, Abhishek Bhat <[email protected]> wrote:

Sangmin,

What should be the correct syntax for print all exitcodes – If I use

if( ! { mpiexec -machinefile nodes -np $NUMPROCS -print-all-exitcodes $EXEC -mpich-dbg=file -mpich-dbg-class=all -mpich-dbg-level=verbose } )

This is correct. And, the output will be shown on your terminal, not in file, like: [mpiexec@host] Exit codes: [host] 0,0

I am getting error saying “-print-all-exitcodes” is not a valid local parameters

Which version of MPICH are you using? Can you let me know the result of the following? $ mpiexec -info

— Sangmin

_________________________________________________________________________

The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer. _________________________________________________________________________ _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss