May 2015 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Killing mpiexec.hydra process
by Kenneth Raffenetti 11 May '15

11 May '15

Sounds good. If you still see issues using 3.1.4, I can investigate. Ken On 05/11/2015 12:27 PM, Cayanan, Michael D (398G) wrote: > Hi Ken, > > Looks like we are running 1.5: > > $ ./mpich2version > MPICH2 Version: 1.5 > > I guess I should update the software to the 3.1 series to see how this > behavior has changed. > > Thanks, > Mike > > > > On 5/11/15 10:19 AM, "Kenneth Raffenetti" <raffenet(a)mcs.anl.gov> wrote: > >> Which version of Hydra is this? There was a bug fix that should have >> improved child process cleanup in the 3.1 series. >> >> Ken >> >> On 05/11/2015 12:04 PM, Cayanan, Michael D (398G) wrote: >>> Hi, >>> >>> I'm running the mpiexec.hydra command within a script and there are >>> cases where I need to kill this process. >>> >>> Our application is designed to issue a "pkill -P <process id>" command, >>> where the "process id" is the id of the script. Nominally, the pkill >>> command is designed to kill the parent and all of its children. However, >>> in this case, it turns out that the script and the parent mpiexec.hydra >>> processes are killed, but the children processes that are spawned by the >>> mpiexec.hydra processes are left running. This is because >>> all of those children are parents of the mpiexec.hydra process. >>> >>> Example: >>> >>> Process 1: <script command to launch the mpiexec.hydra command> >>> Process 2 (Parent 1): mpiexec.hydra command >>> Process 3 (Parent 2): Child mpiexec.hydra process >>> Process 4 (Parent 2): Child mpiexec.hydra process >>> Process 5 (Parent 2): Child mpiexec.hydra process >>> >>> >>> %> pkill -P 1 >>> >>> Processes 1 and 2 are killed, but Processes 3, 4, and 5 are left >>> running. >>> >>> I've done some online research and it looks like there is a utility >>> called "mpicleanup". Has anybody used this tool and is it intended to >>> kill an mpiexec.hydra process and all its children in a single command? >>> If not, what is the best way to kill the mpiexec.hydra process? >>> >>> Thanks in advance for your help, >>> Mike >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Killing mpiexec.hydra process
by Cayanan, Michael D (398G) 11 May '15

11 May '15

Hi Ken, Looks like we are running 1.5: $ ./mpich2version MPICH2 Version: 1.5 I guess I should update the software to the 3.1 series to see how this behavior has changed. Thanks, Mike On 5/11/15 10:19 AM, "Kenneth Raffenetti" <raffenet(a)mcs.anl.gov> wrote: >Which version of Hydra is this? There was a bug fix that should have >improved child process cleanup in the 3.1 series. > >Ken > >On 05/11/2015 12:04 PM, Cayanan, Michael D (398G) wrote: >> Hi, >> >> I'm running the mpiexec.hydra command within a script and there are >> cases where I need to kill this process. >> >> Our application is designed to issue a "pkill -P <process id>" command, >> where the "process id" is the id of the script. Nominally, the pkill >> command is designed to kill the parent and all of its children. However, >> in this case, it turns out that the script and the parent mpiexec.hydra >> processes are killed, but the children processes that are spawned by the >> mpiexec.hydra processes are left running. This is because >> all of those children are parents of the mpiexec.hydra process. >> >> Example: >> >> Process 1: <script command to launch the mpiexec.hydra command> >> Process 2 (Parent 1): mpiexec.hydra command >> Process 3 (Parent 2): Child mpiexec.hydra process >> Process 4 (Parent 2): Child mpiexec.hydra process >> Process 5 (Parent 2): Child mpiexec.hydra process >> >> >> %> pkill -P 1 >> >> Processes 1 and 2 are killed, but Processes 3, 4, and 5 are left >>running. >> >> I've done some online research and it looks like there is a utility >> called "mpicleanup". Has anybody used this tool and is it intended to >> kill an mpiexec.hydra process and all its children in a single command? >> If not, what is the best way to kill the mpiexec.hydra process? >> >> Thanks in advance for your help, >> Mike >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >_______________________________________________ >discuss mailing list discuss(a)mpich.org >To manage subscription options or unsubscribe: >https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] single vs multiple communicator
by Pavan Balaji 11 May '15

11 May '15

The second option will be slower. Apart from the fact that the MPI implementation needs to manage a large number of communicators, in most implementations this option will disable shared memory communication between the slave and master. -- Pavan > On May 11, 2015, at 7:17 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > > Hi MPICH team, > > I have an application which is mostly using master-slave kind of architecture. One master and <n> number of slaves. The slaves do not communicate between themselves and only sends/receives messages to/from master. > > I have two ways I can setup the application > 1> Static : Master and slaves are all in the same MPI World > 2> Dynamic : Each slave is connected to master using a MPI_Comm and the communicator is created using MPI_Comm_connect/MPI_Comm_accept. Essentially master has <n> communicators. > > The way master works is : it uses MPI_WaitAny kind of method to check messages from slave. > > Could you please tell me whether <2> can be significantly slow with respect to <1> if all the slaves send lots of small messages? > Is there some kind of buffering overhead in <2> for <n> communicators and which can be solved using some configuration. > > Thanks, > Hirak > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Killing mpiexec.hydra process
by Kenneth Raffenetti 11 May '15

11 May '15

Which version of Hydra is this? There was a bug fix that should have improved child process cleanup in the 3.1 series. Ken On 05/11/2015 12:04 PM, Cayanan, Michael D (398G) wrote: > Hi, > > I'm running the mpiexec.hydra command within a script and there are > cases where I need to kill this process. > > Our application is designed to issue a "pkill -P <process id>" command, > where the "process id" is the id of the script. Nominally, the pkill > command is designed to kill the parent and all of its children. However, > in this case, it turns out that the script and the parent mpiexec.hydra > processes are killed, but the children processes that are spawned by the > mpiexec.hydra processes are left running. This is because > all of those children are parents of the mpiexec.hydra process. > > Example: > > Process 1: <script command to launch the mpiexec.hydra command> > Process 2 (Parent 1): mpiexec.hydra command > Process 3 (Parent 2): Child mpiexec.hydra process > Process 4 (Parent 2): Child mpiexec.hydra process > Process 5 (Parent 2): Child mpiexec.hydra process > > > %> pkill -P 1 > > Processes 1 and 2 are killed, but Processes 3, 4, and 5 are left running. > > I've done some online research and it looks like there is a utility > called "mpicleanup". Has anybody used this tool and is it intended to > kill an mpiexec.hydra process and all its children in a single command? > If not, what is the best way to kill the mpiexec.hydra process? > > Thanks in advance for your help, > Mike > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Killing mpiexec.hydra process
by Cayanan, Michael D (398G) 11 May '15

11 May '15

Hi, I'm running the mpiexec.hydra command within a script and there are cases where I need to kill this process. Our application is designed to issue a "pkill -P <process id>" command, where the "process id" is the id of the script. Nominally, the pkill command is designed to kill the parent and all of its children. However, in this case, it turns out that the script and the parent mpiexec.hydra processes are killed, but the children processes that are spawned by the mpiexec.hydra processes are left running. This is because all of those children are parents of the mpiexec.hydra process. Example: Process 1: <script command to launch the mpiexec.hydra command> Process 2 (Parent 1): mpiexec.hydra command Process 3 (Parent 2): Child mpiexec.hydra process Process 4 (Parent 2): Child mpiexec.hydra process Process 5 (Parent 2): Child mpiexec.hydra process %> pkill -P 1 Processes 1 and 2 are killed, but Processes 3, 4, and 5 are left running. I've done some online research and it looks like there is a utility called "mpicleanup". Has anybody used this tool and is it intended to kill an mpiexec.hydra process and all its children in a single command? If not, what is the best way to kill the mpiexec.hydra process? Thanks in advance for your help, Mike _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

single vs multiple communicator
by Roy, Hirak 11 May '15

11 May '15

Hi MPICH team, I have an application which is mostly using master-slave kind of architecture. One master and <n> number of slaves. The slaves do not communicate between themselves and only sends/receives messages to/from master. I have two ways I can setup the application 1> Static : Master and slaves are all in the same MPI World 2> Dynamic : Each slave is connected to master using a MPI_Comm and the communicator is created using MPI_Comm_connect/MPI_Comm_accept. Essentially master has <n> communicators. The way master works is : it uses MPI_WaitAny kind of method to check messages from slave. Could you please tell me whether <2> can be significantly slow with respect to <1> if all the slaves send lots of small messages? Is there some kind of buffering overhead in <2> for <n> communicators and which can be solved using some configuration. Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Allreduce is slow for 7 processes or more
by Ken Raffenetti 05 May '15

05 May '15

On 05/05/2015 05:03 AM, David Froger wrote: > I did not tried mpich3.2b2 yet (will do tomorrow), but I've ask another > question here, just on the time to execute MPI_Barrier: > http://stackoverflow.com/questions/30039802/what-control-mpi-barrier-time-t… > > Seems that that was why `bench_mpi.cxx` was slow. The underlying issue is that recent MPICH causes processes to busy poll for incoming messages. In over-subscription scenarios, this can have significant performance implications. There is a ongoing work to address the busy polling issue for MPICH 3.3. In the interim, our advice is to not oversubscribe processes for the available hardware resources. Ken _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] libmpich.so.10: cannot open shared object file
by Jorge Pachon 04 May '15

04 May '15

Thanks, it worked On Mon, May 4, 2015 at 4:11 PM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > > On Mon, May 4, 2015 at 1:50 PM, Jorge Pachon <jpachon3(a)gmail.com> wrote: > >> Hi Ken, >> >> I did some test and MPICH seems to be working OK, at least I ran the CPI >> example and worked fine. >> >> However, I have problems running an 'n' process CPI job on multiple >> nodes. It seems I have not created a machinefile yet to succeed. Is this >> step absolutely necessary? >> > > Yes, if you manage the cluster yourself. > >> >> Moreover, my problem running WRF still exists, because the libmpich.so.10 >> file is not successfully created when installing MPICH 3.0.4. >> > > Looked at your c.txt, and found "checking whether to build shared > libraries... no" > Could you add --enable-shared=yes to configure when building mpich. That > is the default but I don't know why it was not enabled in your build. > > >> I'm attaching all the configure.log files from my installation as well as >> the output from mpiexce -info >> >> Your help is greatly appreciated. >> >> Jorge >> >> >> >> On Tue, Apr 28, 2015 at 9:30 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov >> > wrote: >> >>> Can you confirm your MPICH installation working correctly? Have you >>> tried compiling and running the cpi.c program from the examples directory >>> from the tarball? >>> >>> The MPICH 3.0.4 installation should contain a libmpich.so.10 file, so it >>> will be helpful to know things are installed and working well. >>> >>> Ken >>> >>> >>> On 04/28/2015 04:58 PM, Jorge Pachon wrote: >>> >>>> Hi all, >>>> >>>> I´m trying to run WRF for which I´m using mpich libraries. I recently >>>> installed mpich-3.0.4 and mpich-3.1.4 (with ifort) hoping to install the >>>> libmpich.so.10 file, but neither of them have it ,and I always got this >>>> error >>>> >>>> ./real.exe: error while loading shared libraries: libmpich.so.10: cannot >>>> open shared object file: No such file or directory >>>> >>>> I searched in the mpich folders: /usr/local/lib and >>>> /usr/local/bin/mpich, but cannot find the file >>>> >>>> any help will be appreciated >>>> >>>> Jorge >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] libmpich.so.10: cannot open shared object file
by Junchao Zhang 04 May '15

04 May '15

On Mon, May 4, 2015 at 1:50 PM, Jorge Pachon <jpachon3(a)gmail.com> wrote: > Hi Ken, > > I did some test and MPICH seems to be working OK, at least I ran the CPI > example and worked fine. > > However, I have problems running an 'n' process CPI job on multiple nodes. > It seems I have not created a machinefile yet to succeed. Is this step > absolutely necessary? > Yes, if you manage the cluster yourself. > > Moreover, my problem running WRF still exists, because the libmpich.so.10 > file is not successfully created when installing MPICH 3.0.4. > Looked at your c.txt, and found "checking whether to build shared libraries... no" Could you add --enable-shared=yes to configure when building mpich. That is the default but I don't know why it was not enabled in your build. > I'm attaching all the configure.log files from my installation as well as > the output from mpiexce -info > > Your help is greatly appreciated. > > Jorge > > > > On Tue, Apr 28, 2015 at 9:30 PM, Kenneth Raffenetti <raffenet(a)mcs.anl.gov> > wrote: > >> Can you confirm your MPICH installation working correctly? Have you tried >> compiling and running the cpi.c program from the examples directory from >> the tarball? >> >> The MPICH 3.0.4 installation should contain a libmpich.so.10 file, so it >> will be helpful to know things are installed and working well. >> >> Ken >> >> >> On 04/28/2015 04:58 PM, Jorge Pachon wrote: >> >>> Hi all, >>> >>> I´m trying to run WRF for which I´m using mpich libraries. I recently >>> installed mpich-3.0.4 and mpich-3.1.4 (with ifort) hoping to install the >>> libmpich.so.10 file, but neither of them have it ,and I always got this >>> error >>> >>> ./real.exe: error while loading shared libraries: libmpich.so.10: cannot >>> open shared object file: No such file or directory >>> >>> I searched in the mpich folders: /usr/local/lib and >>> /usr/local/bin/mpich, but cannot find the file >>> >>> any help will be appreciated >>> >>> Jorge >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI_Allreduce is slow for 7 processes or more
by David Froger 04 May '15

04 May '15

Quoting Junchao Zhang (2015-05-04 18:28:12) > Could you try mpich3.2b2? I tested your code with it on my laptop. My timing > is > > np time(s) > 2 102.57 > 4 51.285 > 8 25.6425 > 16 12.8212 I did not tried mpich3.2b2 yet (will do tomorrow), but I've ask another question here, just on the time to execute MPI_Barrier: http://stackoverflow.com/questions/30039802/what-control-mpi-barrier-time-t… Seems that that was why `bench_mpi.cxx` was slow. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0