June 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpich hangs
by Pavan Balaji 27 Jun '13

27 Jun '13

% sudo /usr/sbin/ufw status On 06/27/2013 09:52 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > Yes thats what I did before and found out this to be exact: > "sudo: ufw: command not found" > > > On Fri, Jun 28, 2013 at 11:50 AM, Pavan Balaji <balaji(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > > IIRC, Ubuntu has a firewall setup by default. Try this: > > % sudo ufw status > > -- Pavan > > > On 06/27/2013 09:48 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > > > As I already read some discussion on the mpich thread and I came > to know > these questions before: > > * *Can you ssh from "node01" to "node02"?* > > Yes I have passwordless ssh from each node to other (and > vice versa) > * *Make sure the firewalls are turned off on all machines.* > > I am using Ubuntu Linaro, 12.04, and I have no firewall > installed on > all the nodes. > > > Any other workaround :( > > On Fri, Jun 28, 2013 at 11:43 AM, Pavan Balaji > <balaji(a)mcs.anl.gov <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > > On 06/27/2013 09:41 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > > MPID_nem_tcp_connpoll(1826): Communication error with > rank 0: > Connection > refused > > > Sounds like a firewall problem. > > http://wiki.mpich.org/mpich/____index.php/Frequently_Asked_____Questions#Q:… > <http://wiki.mpich.org/mpich/__index.php/Frequently_Asked___Questions#Q:_My_…> > > > <http://wiki.mpich.org/mpich/__index.php/Frequently_Asked___Questions#Q:_My_… > <http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_…>> > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] mpich hangs
by Syed. Jahanzeb Maqbool Hashmi 27 Jun '13

27 Jun '13

Yes thats what I did before and found out this to be exact: "sudo: ufw: command not found" On Fri, Jun 28, 2013 at 11:50 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > IIRC, Ubuntu has a firewall setup by default. Try this: > > % sudo ufw status > > -- Pavan > > > On 06/27/2013 09:48 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > >> >> As I already read some discussion on the mpich thread and I came to know >> these questions before: >> >> * *Can you ssh from "node01" to "node02"?* >> >> Yes I have passwordless ssh from each node to other (and vice versa) >> * *Make sure the firewalls are turned off on all machines.* >> >> I am using Ubuntu Linaro, 12.04, and I have no firewall installed on >> all the nodes. >> >> >> Any other workaround :( >> >> On Fri, Jun 28, 2013 at 11:43 AM, Pavan Balaji <balaji(a)mcs.anl.gov >> <mailto:[email protected]>> wrote: >> >> >> On 06/27/2013 09:41 PM, Syed. Jahanzeb Maqbool Hashmi wrote: >> >> MPID_nem_tcp_connpoll(1826): Communication error with rank 0: >> Connection >> refused >> >> >> Sounds like a firewall problem. >> >> http://wiki.mpich.org/mpich/__**index.php/Frequently_Asked___** >> Questions#Q:_My_MPI_program___**aborts_with_an_error_saying___** >> it_cannot_communicate_with___**other_processes<http://wiki.mpich.org/mpich/__index.php/Frequently_Asked___Questions#Q:_My_…> >> >> <http://wiki.mpich.org/mpich/**index.php/Frequently_Asked_** >> Questions#Q:_My_MPI_program_**aborts_with_an_error_saying_** >> it_cannot_communicate_with_**other_processes<http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_…> >> > >> >> >> -- Pavan >> >> -- >> Pavan Balaji >> http://www.mcs.anl.gov/~balaji >> >> >> > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji >

1 0

Re: [mpich-discuss] mpich hangs
by Pavan Balaji 27 Jun '13

27 Jun '13

IIRC, Ubuntu has a firewall setup by default. Try this: % sudo ufw status -- Pavan On 06/27/2013 09:48 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > > As I already read some discussion on the mpich thread and I came to know > these questions before: > > * *Can you ssh from "node01" to "node02"?* > Yes I have passwordless ssh from each node to other (and vice versa) > * *Make sure the firewalls are turned off on all machines.* > I am using Ubuntu Linaro, 12.04, and I have no firewall installed on > all the nodes. > > > Any other workaround :( > > On Fri, Jun 28, 2013 at 11:43 AM, Pavan Balaji <balaji(a)mcs.anl.gov > <mailto:[email protected]>> wrote: > > > On 06/27/2013 09:41 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > > MPID_nem_tcp_connpoll(1826): Communication error with rank 0: > Connection > refused > > > Sounds like a firewall problem. > > http://wiki.mpich.org/mpich/__index.php/Frequently_Asked___Questions#Q:_My_… > <http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_…> > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] mpich hangs
by Syed. Jahanzeb Maqbool Hashmi 27 Jun '13

27 Jun '13

As I already read some discussion on the mpich thread and I came to know these questions before: - *Can you ssh from "node01" to "node02"?* Yes I have passwordless ssh from each node to other (and vice versa) - *Make sure the firewalls are turned off on all machines.* I am using Ubuntu Linaro, 12.04, and I have no firewall installed on all the nodes. Any other workaround :( On Fri, Jun 28, 2013 at 11:43 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 06/27/2013 09:41 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > >> MPID_nem_tcp_connpoll(1826): Communication error with rank 0: Connection >> refused >> > > Sounds like a firewall problem. > > http://wiki.mpich.org/mpich/**index.php/Frequently_Asked_** > Questions#Q:_My_MPI_program_**aborts_with_an_error_saying_** > it_cannot_communicate_with_**other_processes<http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_…> > > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji >

1 0

Re: [mpich-discuss] mpich hangs
by Pavan Balaji 27 Jun '13

27 Jun '13

On 06/27/2013 09:41 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > MPID_nem_tcp_connpoll(1826): Communication error with rank 0: Connection > refused Sounds like a firewall problem. http://wiki.mpich.org/mpich/index.php/Frequently_Asked_Questions#Q:_My_MPI_… -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] mpich hangs
by Syed. Jahanzeb Maqbool Hashmi 27 Jun '13

27 Jun '13

sorry for giving such little information. ok here is the output after a long hang (which sometimes comes out) ================START OF OUTPUT===================== linaro@weiser1:/mnt/nfs/jahanzeb/bench/hpl/hpl-2.1/bin/armv7-a$ mpirun -np 8 -machinefile machines ./xhp l Fatal error in MPI_Send: A process has failed, error stack: MPI_Send(171)..............: MPI_Send(buf=0xbe84fc50, count=1, MPI_INT, dest=0, tag=9001, MPI_COMM_WORLD ) failed MPID_nem_tcp_connpoll(1826): Communication error with rank 0: Connection refused =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = EXIT CODE: 1 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== [proxy:0:0@weiser1] HYD_pmcd_pmip_control_cmd_cb (./pm/pmiserv/pmip_cb.c:886): assert (!closed) failed [proxy:0:0@weiser1] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [proxy:0:0@weiser1] main (./pm/pmiserv/pmip.c:206): demux engine error waiting for event [mpiexec@weiser1] HYDT_bscu_wait_for_completion (./tools/bootstrap/utils/bscu_wait.c:76): one of the processes terminated badly; aborting [mpiexec@weiser1] HYDT_bsci_wait_for_completion (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for completion [mpiexec@weiser1] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:217): launcher returned error waiting for completion [mpiexec@weiser1] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion ================ENDOF OUTPUT===================== On Fri, Jun 28, 2013 at 11:39 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > On 06/27/2013 09:36 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > >> I am trying to run HPL on a cluster of nodes. The problem I am facing is >> with mpich, as I have successfully configured mpich. The program runs on >> single node without passing -machinefile argument. But as long as I >> execute of multiple nodes (-machinefile nodes) then the program hangs on >> indefinitely right after issuing the command. >> > > Given how little information you have provided, here's the only response I > can give: > > You are doing something wrong. > > -- Pavan > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji >

1 0

Re: [mpich-discuss] mpich hangs
by Pavan Balaji 27 Jun '13

27 Jun '13

On 06/27/2013 09:36 PM, Syed. Jahanzeb Maqbool Hashmi wrote: > I am trying to run HPL on a cluster of nodes. The problem I am facing is > with mpich, as I have successfully configured mpich. The program runs on > single node without passing -machinefile argument. But as long as I > execute of multiple nodes (-machinefile nodes) then the program hangs on > indefinitely right after issuing the command. Given how little information you have provided, here's the only response I can give: You are doing something wrong. -- Pavan -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

mpich hangs
by Syed. Jahanzeb Maqbool Hashmi 27 Jun '13

27 Jun '13

Hello I am trying to run HPL on a cluster of nodes. The problem I am facing is with mpich, as I have successfully configured mpich. The program runs on single node without passing -machinefile argument. But as long as I execute of multiple nodes (-machinefile nodes) then the program hangs on indefinitely right after issuing the command. Any help?

1 0

Process-to-core binding in MPI_Comm_spawn
by Gonzalo Martín Cruz 27 Jun '13

27 Jun '13

Hi all, I am working with MPI_Comm_spawn to launch dynamic processes in my MPI application and I would like to use the process-to-core binding allocation strategy in processes spawned dynamically. I already know that it is possible to set the "host" key in the MPI_Info parameter of the MPI_Comm_spawn call to spawn the process in a specific host. However, I would like to know how to bound a dynamic process to a specific processor core. I tried passing to MPI_Comm_spawn the "host" info key as "compute-node-X binding:user=4", but it does not work and the process goes to any core randomly. Thank you very much, Regards, Gonzalo.

1 0

Re: [mpich-discuss] Non-blocking Collectives
by Jiri Simsa 27 Jun '13

27 Jun '13

Hi Pavan, To rephrase, I am interested in understanding when would MPI_Wait() block indefinitely, waiting for other process to make progress. I believe that your response answers my question. Thanks again. --Jiri On Wed, Jun 26, 2013 at 4:48 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Hi Jiri, > > > On 06/26/2013 03:08 PM, Jiri Simsa wrote: > >> Thank you for your quick answer. I am trying to understand the blocking >> behavior of MPI_Wait in the case of non-blocking collectives. Is it safe >> to assume that, for a non-blocking collective, MPI_Wait is guaranteed to >> return once all other processes call the corresponding completion >> operation (e.g. MPI_Wait or MPI_Test)? >> > > I'm not sure I understand your question. Are you asking if MPI_WAIT in a > process is guaranteed to return after some finite amount of time after > every other process has called MPI_WAIT? Then, yes. > > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji >

1 0