One of the proxy processes died suddenly causing everything else to fall apart. Try using the latest version of hydra (3.0.1), and also try running the proxy processes with a debugger (set the environment variable HYDRA_USE_DDD=1). -- Pavan On 08/20/2012 03:46 AM US Central Time, Yann RADENAC wrote:

Hi,

I'm developing MPI support for XtreemOS (www.xtreemos.eu) so that an MPI program is managed as a single XtreemOS job. To manage all processes as a single XtreemOS job, I've developed the program xos-createProcess that plays the role of the launcher (replacing ssh/rsh) to start a process on a remote machine that is part of the ones reserved for the current job.

I'm running a simple hello world MPI program where each processes sends a string to the process 0 that itself prints them on standard output.

When using MPICH2 with ssh, this program works perfectly on several machines.

When using MPICH2 with my launcher xos-createProcess, it works with an MPI program of 2 processes on 2 different machines.

However I cannot pass through the following error that happens when running an MPI program of 3 processes on 3 different machines (or any n processes on n different machines with n >= 3). Everything terminates almost immediately with these error messages:

Process 0 ends with error code 7 and its standard error output is :

[[email protected]] cmd_response (./pm/pmiserv/pmiserv_pmi_v1.c:29): assert (!closed) failed [[email protected]] fn_barrier_in (./pm/pmiserv/pmiserv_pmi_v1.c:70): error writing PMI line [[email protected]] handle_pmi_cmd (./pm/pmiserv/pmiserv_cb.c:44): PMI handler returned error [[email protected]] control_cb (./pm/pmiserv/pmiserv_cb.c:289): unable to process PMI command [[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [[email protected]] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:181): error waiting for event [[email protected]] main (./ui/mpich/mpiexec.c:405): process manager error waiting for completion

On *only* one of the other processes, the standard error output is:

[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:[email protected]] main (./pm/pmiserv/pmip.c:226): demux engine error waiting for event

The run command is:

-bash -c '(mpiexec -launcher-exec /usr/bin/xos-createProcess -np 3 -host `xreservation -a $XOS_RSVID` ./mpi/hello_world_MPI < /dev/null > mpiexec.out) >& mpiexec.err'

Can anyone explain me what this error means ?

I'm using MPICH2 1.4.1p1

Thanks for your help.

-- Pavan Balaji http://www.mcs.anl.gov/~balaji