December 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] A good starting point for programming and understanding MPI, and parallel programming in general.
by mark 11 Dec '13

11 Dec '13

Thanks for the pointers, they are always welcomed. Speaking about MPI and MPICH, if i try openmpi, will it be difficult to transcribe my projects and learning to mpich? Are they different or completely the same? In my opinion is like using a different IDE for the same language, so the same, but i am asked to follow the mpich as for my mpi and parallel programming studies. ???? 11/12/2013 05:52 ??, ?/? Brock Palen ??????: > I'm a big fan of CI tutor for getting started: > > http://www.citutor.org/ > > Brock Palen > www.umich.edu/~brockp > > > sing Your Own Computer to Run CI-Tutor MPI Exercises > > In order to use CI-Tutor courses and exercises involving MPI, you > don't have to have access to XSEDE resources: you can run MPI on your > own machine. Simply install your favorite implementation of MPI, if > one is not already pre-installed. > > To find out whether MPI is already installed, run a query. For > instance, on distributions that support rpm packaging: > > [arnoldg@lgheron ~]$ rpm -q openmpi > openmpi-1.3.1-1.fc11.x86_64 > > OpenMPI (http:\\www.open-mpi.org) is a popular MPI implementation > easily available for most Linux distributions. You must have root > privileges in order to install OpenMPI on your system. The website > includes tarballs and source code that you can download and use to > build OpenMPI manually, or (if you are a relatively new Linux user) > you can use the software management package that is included with your > distribution. Below, we include example instructions for downloading > and installing OpenMPI using the software management packages for two > commonly-used Linux distributions, Fedora and Ubuntu. > > For example, to download and install OpenMPI on a machine running > Fedora (or another RedHat-based distribution), you should be able > simply to run the yum package manager: > > # yum install openmpi openmpi-devel openmpi-libs > > On an Ubuntu or other Debian-based distribution, you use the Synaptic > or Aptitude package manager, either with the graphical user interface > provided, or with the command line, e.g.: > > # apt-get install openmpi openmpi-dev openmpi-devel > CAEN Advanced Computing > XSEDE Campus Champion > brockp(a)umich.edu > (734)936-1985 > > > > On Dec 11, 2013, at 10:50 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > >> Hello, Mark, >> Yes, you can install the latest mpich (http://www.mpich.org/downloads/) and run it on your laptop. >> Do a google search, there are many MPI tutorials. I used this one when I started to learn MPI, https://computing.llnl.gov/tutorials/mpi/ >> >> --Junchao Zhang >> >> >> On Wed, Dec 11, 2013 at 8:54 AM, mark <dimitsas.markos(a)gmail.com> wrote: >> Hello from Greece! >> Being new in this field, what would be a good starting point to master MPI and parallel programming in general? First of i don't have a cluster, so all the computations are going to happen to a modern day linux laptop. >> However if i get familiar with MPI, i will have the chance to work with a cluster remotely on some text retrieval algorithms (yea right :P) for my bachelor thesis. >> So, looking for starting points and speed learning about MPI (and parallel programming) in general. >> Any tips and advices are welcome.Thank you :) >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] A good starting point for programming and understanding MPI, and parallel programming in general.
by Brock Palen 11 Dec '13

11 Dec '13

I'm a big fan of CI tutor for getting started: http://www.citutor.org/ Brock Palen www.umich.edu/~brockp CAEN Advanced Computing XSEDE Campus Champion brockp(a)umich.edu (734)936-1985 On Dec 11, 2013, at 10:50 AM, Junchao Zhang <jczhang(a)mcs.anl.gov> wrote: > Hello, Mark, > Yes, you can install the latest mpich (http://www.mpich.org/downloads/) and run it on your laptop. > Do a google search, there are many MPI tutorials. I used this one when I started to learn MPI, https://computing.llnl.gov/tutorials/mpi/ > > --Junchao Zhang > > > On Wed, Dec 11, 2013 at 8:54 AM, mark <dimitsas.markos(a)gmail.com> wrote: > Hello from Greece! > Being new in this field, what would be a good starting point to master MPI and parallel programming in general? First of i don't have a cluster, so all the computations are going to happen to a modern day linux laptop. > However if i get familiar with MPI, i will have the chance to work with a cluster remotely on some text retrieval algorithms (yea right :P) for my bachelor thesis. > So, looking for starting points and speed learning about MPI (and parallel programming) in general. > Any tips and advices are welcome.Thank you :) > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] A good starting point for programming and understanding MPI, and parallel programming in general.
by Junchao Zhang 11 Dec '13

11 Dec '13

Hello, Mark, Yes, you can install the latest mpich (http://www.mpich.org/downloads/) and run it on your laptop. Do a google search, there are many MPI tutorials. I used this one when I started to learn MPI, https://computing.llnl.gov/tutorials/mpi/ --Junchao Zhang On Wed, Dec 11, 2013 at 8:54 AM, mark <dimitsas.markos(a)gmail.com> wrote: > Hello from Greece! > Being new in this field, what would be a good starting point to master MPI > and parallel programming in general? First of i don't have a cluster, so > all the computations are going to happen to a modern day linux laptop. > However if i get familiar with MPI, i will have the chance to work with a > cluster remotely on some text retrieval algorithms (yea right :P) for my > bachelor thesis. > So, looking for starting points and speed learning about MPI (and parallel > programming) in general. > Any tips and advices are welcome.Thank you :) > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

A good starting point for programming and understanding MPI, and parallel programming in general.
by mark 11 Dec '13

11 Dec '13

Hello from Greece! Being new in this field, what would be a good starting point to master MPI and parallel programming in general? First of i don't have a cluster, so all the computations are going to happen to a modern day linux laptop. However if i get familiar with MPI, i will have the chance to work with a cluster remotely on some text retrieval algorithms (yea right :P) for my bachelor thesis. So, looking for starting points and speed learning about MPI (and parallel programming) in general. Any tips and advices are welcome.Thank you :)

1 0

Re: [mpich-discuss] how to kill all tasks of a job
by Pavan Balaji 11 Dec '13

11 Dec '13

Kathrin, We, unfortunately, have no developer bandwidth to support windows. Perhaps you could consider using MS-MPI, the MPI implementation provided by Microsoft? That’s well supported by Microsoft. Regards, — Pavan On Dec 11, 2013, at 8:40 AM, Kathrin Burckhardt <burckhardt(a)itis.ethz.ch> wrote: > Hi Antonio, > > I found out that we use the unofficial binary package 1.3.2p1 for Windows and can't expect any help from Microsoft. > May it be possible to transmit my question to Jayesh Krishna, the author of this package? > > Thanks again, > Kathrin > > > ----- Original Message ----- >> From: "Antonio J. Peña" <apenya(a)mcs.anl.gov> >> To: discuss(a)mpich.org >> Sent: Tuesday, December 10, 2013 5:25:01 PM >> Subject: Re: [mpich-discuss] how to kill all tasks of a job >> >> >> Sorry, MPICH doesn't support Windows since a few versions ago. In our >> Downloads section you can find a link for an MPICH derivative from >> Microsoft. >> >> Antonio >> >> >> On 12/10/2013 02:37 AM, Kathrin Burckhardt wrote: >>> Hi, >>> >>> I'm using MPICH2 under Windows and would like to kill on the console all >>> tasks of an executable started at a certain time instance with >>> >>> mpiexec -n 8 executable >>> >>> What do you suggest? >>> >>> Thanks in advance for your help, >>> >>> Kathrin >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> >> -- >> Antonio J. Peña >> Postdoctoral Appointee >> Mathematics and Computer Science Division >> Argonne National Laboratory >> 9700 South Cass Avenue, Bldg. 240, Of. 3148 >> Argonne, IL 60439-4847 >> apenya(a)mcs.anl.gov >> www.mcs.anl.gov/~apenya >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] how to kill all tasks of a job
by Kathrin Burckhardt 11 Dec '13

11 Dec '13

Hi Antonio, I found out that we use the unofficial binary package 1.3.2p1 for Windows and can't expect any help from Microsoft. May it be possible to transmit my question to Jayesh Krishna, the author of this package? Thanks again, Kathrin ----- Original Message ----- > From: "Antonio J. Peña" <apenya(a)mcs.anl.gov> > To: discuss(a)mpich.org > Sent: Tuesday, December 10, 2013 5:25:01 PM > Subject: Re: [mpich-discuss] how to kill all tasks of a job > > > Sorry, MPICH doesn't support Windows since a few versions ago. In our > Downloads section you can find a link for an MPICH derivative from > Microsoft. > > Antonio > > > On 12/10/2013 02:37 AM, Kathrin Burckhardt wrote: > > Hi, > > > > I'm using MPICH2 under Windows and would like to kill on the console all > > tasks of an executable started at a certain time instance with > > > > mpiexec -n 8 executable > > > > What do you suggest? > > > > Thanks in advance for your help, > > > > Kathrin > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Antonio J. Peña > Postdoctoral Appointee > Mathematics and Computer Science Division > Argonne National Laboratory > 9700 South Cass Avenue, Bldg. 240, Of. 3148 > Argonne, IL 60439-4847 > apenya(a)mcs.anl.gov > www.mcs.anl.gov/~apenya > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] how to kill all tasks of a job
by "Antonio J. Peña" 10 Dec '13

10 Dec '13

Sorry, MPICH doesn't support Windows since a few versions ago. In our Downloads section you can find a link for an MPICH derivative from Microsoft. Antonio On 12/10/2013 02:37 AM, Kathrin Burckhardt wrote: > Hi, > > I'm using MPICH2 under Windows and would like to kill on the console all tasks of an executable started at a certain time instance with > > mpiexec -n 8 executable > > What do you suggest? > > Thanks in advance for your help, > > Kathrin > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peï¿½a Postdoctoral Appointee Mathematics and Computer Science Division Argonne National Laboratory 9700 South Cass Avenue, Bldg. 240, Of. 3148 Argonne, IL 60439-4847 apenya(a)mcs.anl.gov www.mcs.anl.gov/~apenya

1 0

how to kill all tasks of a job
by Kathrin Burckhardt 10 Dec '13

10 Dec '13

Hi, I'm using MPICH2 under Windows and would like to kill on the console all tasks of an executable started at a certain time instance with mpiexec -n 8 executable What do you suggest? Thanks in advance for your help, Kathrin

1 0

Re: [mpich-discuss] MPI memory allocation.
by Anatoly G 09 Dec '13

09 Dec '13

Yes, I understand that. I"ll try to make my stand alone test closer to real application. Thank you. On Mon, Dec 9, 2013 at 9:31 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > It sounds like MPICH is working correctly. Without a test case, it’s > unfortunately quite hard for us to even know what to look for. It’s also > possible that there’s a bug in your code which might be causing some bad > behavior. > > — Pavan > > On Dec 9, 2013, at 1:27 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > > Yes, I"m actually need Fault tolerance, and it was the main reason for > choosing MPICH2. I use fault tolerance for unpredictable bugs in the > future. My system should survive partially. But in the regular case I just > need full performance. I"m suspect that I don't use MPI correctly, but on > slow rate everything works fine. The fail caused by increasing rate of > MPI_Isend or increasing data buffer size. I didn't find yet any strong > dependence, only main stream. > > > > Unfortunately I have a complex system which has a number of threads in > each process. Part of the threads use different communicators. > > > > I try to simulate the same MPI behavior in simple stand alone test, but > stand alone test works fine. It shows a full network performance, when I > slow down master (on stand alone test), all slaves are stopped too and are > waiting for master to continue. Can I open any MPICH log to send you the > results? > > > > > > On Mon, Dec 9, 2013 at 8:10 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > > > Do you actually need Fault Tolerance (one of your previous emails seemed > to indicate that)? > > > > It sounds like there a bug in either your application or in the MPICH > stack and you are trying to trace that down, and don’t really care about > fault tolerance. Is that a correct assessment? > > > > Do you have a simplified program that reproduces this error, that we can > try? > > > > — Pavan > > > > On Dec 9, 2013, at 11:44 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > > > > No. Hardware is Ok. Master process allocates memory (check with > MemoryScape doesn't show any sufficient memory allocation in my code). Then > network traffic become low, and then Master process crashes w/o saving core > file. I have unlimited size of core files. The same fail (w/o core) I see > when I call MPI_Abort, but I don't call it. > > > > > > > > > On Mon, Dec 9, 2013 at 7:28 PM, Wesley Bland <wbland(a)mcs.anl.gov> > wrote: > > > Are you actually seeing hardware failure or is your code just > crashing? It's odd that one specific process would fail so often in the > same way if it were a hardware problem. > > > > > > Thanks, > > > Wesley > > > > > > On Dec 9, 2013, at 11:15 AM, Anatoly G <anatolyrishon(a)gmail.com> > wrote: > > > > > >> One more interesting fact. Each time I have a failure, the fails only > master process, but slaves are still exists together with mpiexec.hydra. I > thought that slaves should fail too, but slaves are live. > > >> > > >> > > >> On Mon, Dec 9, 2013 at 10:30 AM, Anatoly G <anatolyrishon(a)gmail.com> > wrote: > > >> I configure MPICH-3.1rc2 build w/o "so" files. But instead of MPICH2 > & MPICH-3.0.4 I get so files. What should I change in configure line to > link MPI with my application statically. > > >> > > >> > > >> > > >> On Mon, Dec 9, 2013 at 9:47 AM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > > >> > > >> Can you try mpich-3.1rc2? There were several fixes for this in this > version and it’ll be good to try that out before we go digging too far into > this. > > >> > > >> — Pavan > > >> > > >> On Dec 9, 2013, at 1:46 AM, Anatoly G <anatolyrishon(a)gmail.com> > wrote: > > >> > > >> > With MPICH - 3.0.4 the situation repeated. It looks like MPI > allocates memory for messages. > > >> > Can you please advice about scenario when MPI or may be TCP under > MPI allocates memory due to high transfer rate? > > >> > > > >> > > > >> > On Mon, Dec 9, 2013 at 9:32 AM, Anatoly G <anatolyrishon(a)gmail.com> > wrote: > > >> > Thank you very much. > > >> > Issend - is not so good, It can't support me Fault tolerance. If > slave process fails, the master stall. > > >> > I tried mpich-3.0.4 with hydra-3.0.4 but my program which uses MPI > Fault tolerance doesn't recognize failure of slave process, but recognizes > failure with MPICH2. May be you can suggest solution? > > >> > I tried to use hydra from MPICH2 but link my program with MPICH3. > This combination recognizes failures, but I"m not sure that such > combination is stable enough. > > >> > Can you please advice? > > >> > Anatoly. > > >> > > > >> > > > >> > > > >> > On Sat, Dec 7, 2013 at 5:20 PM, Pavan Balaji <balaji(a)mcs.anl.gov> > wrote: > > >> > > > >> > As much as I hate saying this — some people find it easier to think > of it as “MPICH3”. > > >> > > > >> > — Pavan > > >> > > > >> > On Dec 7, 2013, at 7:37 AM, Wesley Bland <wbland(a)mcs.anl.gov> > wrote: > > >> > > > >> > > MPICH is just the new version of MPICH2. We renamed it when we > went past version 3.0. > > >> > > > > >> > > On Dec 7, 2013, at 3:55 AM, Anatoly G <anatolyrishon(a)gmail.com> > wrote: > > >> > > > > >> > >> Ok. I"ll try both Issend, and next step to upgrade MPICH to > 3.0.4. > > >> > >> I thought before that MPICH & MPICH2 are two different branches, > when MPICH2 partially supports Fault tolerance, but MPICH not. Now I > understand, that I was wrong and MPICH2 is just main version of MPICH. > > >> > >> > > >> > >> Thank you very much, > > >> > >> Anatoly. > > >> > >> > > >> > >> > > >> > >> > > >> > >> On Thu, Dec 5, 2013 at 11:01 PM, Rajeev Thakur < > thakur(a)mcs.anl.gov> wrote: > > >> > >> The master is receiving too many incoming messages than it can > match quickly enough with Irecvs. Try using MPI_Issend instead of MPI_Isend. > > >> > >> > > >> > >> Rajeev > > >> > >> > > >> > >> On Dec 5, 2013, at 2:58 AM, Anatoly G <anatolyrishon(a)gmail.com> > wrote: > > >> > >> > > >> > >> > Hello. > > >> > >> > I"m using MPICH2 1.5. > > >> > >> > My system contains master and 16 slaves. > > >> > >> > System uses number of communicators. > > >> > >> > The single communicator used for below scenario: > > >> > >> > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend > and waits using MPI_Wait. > > >> > >> > Master starts with MPI_Irecv to each slave > > >> > >> > Then in endless loop: > > >> > >> > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. > > >> > >> > > > >> > >> > Another communicator used for broadcast communication > (commands between master + slaves), > > >> > >> > but it's not used in parallel with previous communicator, > > >> > >> > only before or after data transfer. > > >> > >> > > > >> > >> > The system executed on two computers linked by 1Gbit/s > Ethernet. > > >> > >> > Master executed on first computer, all slaves on other one. > > >> > >> > Network traffic is ~800Mbit/s. > > >> > >> > > > >> > >> > After 1-2 minutes of execution, master process starts to > increase it's memory allocation and network traffic becomes low. > > >> > >> > This memory allocation & network traffic slow down continues > until fail of MPI, > > >> > >> > without core file save. > > >> > >> > My program doesn't allocate memory. Can you please explain > this behaviour. > > >> > >> > How can I cause MPI to stop sending slaves if Master can't > serve such traffic, instead of memory allocation and fail? > > >> > >> > > > >> > >> > > > >> > >> > Thank you, > > >> > >> > Anatoly. > > >> > >> > > > >> > >> > P.S. > > >> > >> > On my stand alone test, I simulate similar behaviour, but with > single thread on each process (master & hosts). > > >> > >> > When I start stand alone test, master stops slaves until it > completes accumulated data processing and MPI doesn't increase memory > allocation. > > >> > >> > When Master is free slaves continue to send data. > > >> > >> > _______________________________________________ > > >> > >> > discuss mailing list discuss(a)mpich.org > > >> > >> > To manage subscription options or unsubscribe: > > >> > >> > https://lists.mpich.org/mailman/listinfo/discuss > > >> > >> > > >> > >> _______________________________________________ > > >> > >> discuss mailing list discuss(a)mpich.org > > >> > >> To manage subscription options or unsubscribe: > > >> > >> https://lists.mpich.org/mailman/listinfo/discuss > > >> > >> > > >> > >> _______________________________________________ > > >> > >> discuss mailing list discuss(a)mpich.org > > >> > >> To manage subscription options or unsubscribe: > > >> > >> https://lists.mpich.org/mailman/listinfo/discuss > > >> > > _______________________________________________ > > >> > > discuss mailing list discuss(a)mpich.org > > >> > > To manage subscription options or unsubscribe: > > >> > > https://lists.mpich.org/mailman/listinfo/discuss > > >> > > > >> > -- > > >> > Pavan Balaji > > >> > http://www.mcs.anl.gov/~balaji > > >> > > > >> > _______________________________________________ > > >> > discuss mailing list discuss(a)mpich.org > > >> > To manage subscription options or unsubscribe: > > >> > https://lists.mpich.org/mailman/listinfo/discuss > > >> > > > >> > > > >> > _______________________________________________ > > >> > discuss mailing list discuss(a)mpich.org > > >> > To manage subscription options or unsubscribe: > > >> > https://lists.mpich.org/mailman/listinfo/discuss > > >> > > >> -- > > >> Pavan Balaji > > >> http://www.mcs.anl.gov/~balaji > > >> > > >> _______________________________________________ > > >> discuss mailing list discuss(a)mpich.org > > >> To manage subscription options or unsubscribe: > > >> https://lists.mpich.org/mailman/listinfo/discuss > > >> > > >> > > >> _______________________________________________ > > >> discuss mailing list discuss(a)mpich.org > > >> To manage subscription options or unsubscribe: > > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPI memory allocation.
by Pavan Balaji 09 Dec '13

09 Dec '13

It sounds like MPICH is working correctly. Without a test case, it’s unfortunately quite hard for us to even know what to look for. It’s also possible that there’s a bug in your code which might be causing some bad behavior. — Pavan On Dec 9, 2013, at 1:27 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > Yes, I"m actually need Fault tolerance, and it was the main reason for choosing MPICH2. I use fault tolerance for unpredictable bugs in the future. My system should survive partially. But in the regular case I just need full performance. I"m suspect that I don't use MPI correctly, but on slow rate everything works fine. The fail caused by increasing rate of MPI_Isend or increasing data buffer size. I didn't find yet any strong dependence, only main stream. > > Unfortunately I have a complex system which has a number of threads in each process. Part of the threads use different communicators. > > I try to simulate the same MPI behavior in simple stand alone test, but stand alone test works fine. It shows a full network performance, when I slow down master (on stand alone test), all slaves are stopped too and are waiting for master to continue. Can I open any MPICH log to send you the results? > > > On Mon, Dec 9, 2013 at 8:10 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > > Do you actually need Fault Tolerance (one of your previous emails seemed to indicate that)? > > It sounds like there a bug in either your application or in the MPICH stack and you are trying to trace that down, and don’t really care about fault tolerance. Is that a correct assessment? > > Do you have a simplified program that reproduces this error, that we can try? > > — Pavan > > On Dec 9, 2013, at 11:44 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > > No. Hardware is Ok. Master process allocates memory (check with MemoryScape doesn't show any sufficient memory allocation in my code). Then network traffic become low, and then Master process crashes w/o saving core file. I have unlimited size of core files. The same fail (w/o core) I see when I call MPI_Abort, but I don't call it. > > > > > > On Mon, Dec 9, 2013 at 7:28 PM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > > Are you actually seeing hardware failure or is your code just crashing? It's odd that one specific process would fail so often in the same way if it were a hardware problem. > > > > Thanks, > > Wesley > > > > On Dec 9, 2013, at 11:15 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > > >> One more interesting fact. Each time I have a failure, the fails only master process, but slaves are still exists together with mpiexec.hydra. I thought that slaves should fail too, but slaves are live. > >> > >> > >> On Mon, Dec 9, 2013 at 10:30 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > >> I configure MPICH-3.1rc2 build w/o "so" files. But instead of MPICH2 & MPICH-3.0.4 I get so files. What should I change in configure line to link MPI with my application statically. > >> > >> > >> > >> On Mon, Dec 9, 2013 at 9:47 AM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> > >> Can you try mpich-3.1rc2? There were several fixes for this in this version and it’ll be good to try that out before we go digging too far into this. > >> > >> — Pavan > >> > >> On Dec 9, 2013, at 1:46 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > >> > >> > With MPICH - 3.0.4 the situation repeated. It looks like MPI allocates memory for messages. > >> > Can you please advice about scenario when MPI or may be TCP under MPI allocates memory due to high transfer rate? > >> > > >> > > >> > On Mon, Dec 9, 2013 at 9:32 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > >> > Thank you very much. > >> > Issend - is not so good, It can't support me Fault tolerance. If slave process fails, the master stall. > >> > I tried mpich-3.0.4 with hydra-3.0.4 but my program which uses MPI Fault tolerance doesn't recognize failure of slave process, but recognizes failure with MPICH2. May be you can suggest solution? > >> > I tried to use hydra from MPICH2 but link my program with MPICH3. This combination recognizes failures, but I"m not sure that such combination is stable enough. > >> > Can you please advice? > >> > Anatoly. > >> > > >> > > >> > > >> > On Sat, Dec 7, 2013 at 5:20 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > >> > > >> > As much as I hate saying this — some people find it easier to think of it as “MPICH3”. > >> > > >> > — Pavan > >> > > >> > On Dec 7, 2013, at 7:37 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > >> > > >> > > MPICH is just the new version of MPICH2. We renamed it when we went past version 3.0. > >> > > > >> > > On Dec 7, 2013, at 3:55 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > >> > > > >> > >> Ok. I"ll try both Issend, and next step to upgrade MPICH to 3.0.4. > >> > >> I thought before that MPICH & MPICH2 are two different branches, when MPICH2 partially supports Fault tolerance, but MPICH not. Now I understand, that I was wrong and MPICH2 is just main version of MPICH. > >> > >> > >> > >> Thank you very much, > >> > >> Anatoly. > >> > >> > >> > >> > >> > >> > >> > >> On Thu, Dec 5, 2013 at 11:01 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > >> > >> The master is receiving too many incoming messages than it can match quickly enough with Irecvs. Try using MPI_Issend instead of MPI_Isend. > >> > >> > >> > >> Rajeev > >> > >> > >> > >> On Dec 5, 2013, at 2:58 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > >> > >> > >> > >> > Hello. > >> > >> > I"m using MPICH2 1.5. > >> > >> > My system contains master and 16 slaves. > >> > >> > System uses number of communicators. > >> > >> > The single communicator used for below scenario: > >> > >> > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend and waits using MPI_Wait. > >> > >> > Master starts with MPI_Irecv to each slave > >> > >> > Then in endless loop: > >> > >> > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. > >> > >> > > >> > >> > Another communicator used for broadcast communication (commands between master + slaves), > >> > >> > but it's not used in parallel with previous communicator, > >> > >> > only before or after data transfer. > >> > >> > > >> > >> > The system executed on two computers linked by 1Gbit/s Ethernet. > >> > >> > Master executed on first computer, all slaves on other one. > >> > >> > Network traffic is ~800Mbit/s. > >> > >> > > >> > >> > After 1-2 minutes of execution, master process starts to increase it's memory allocation and network traffic becomes low. > >> > >> > This memory allocation & network traffic slow down continues until fail of MPI, > >> > >> > without core file save. > >> > >> > My program doesn't allocate memory. Can you please explain this behaviour. > >> > >> > How can I cause MPI to stop sending slaves if Master can't serve such traffic, instead of memory allocation and fail? > >> > >> > > >> > >> > > >> > >> > Thank you, > >> > >> > Anatoly. > >> > >> > > >> > >> > P.S. > >> > >> > On my stand alone test, I simulate similar behaviour, but with single thread on each process (master & hosts). > >> > >> > When I start stand alone test, master stops slaves until it completes accumulated data processing and MPI doesn't increase memory allocation. > >> > >> > When Master is free slaves continue to send data. > >> > >> > _______________________________________________ > >> > >> > discuss mailing list discuss(a)mpich.org > >> > >> > To manage subscription options or unsubscribe: > >> > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > >> > >> > >> _______________________________________________ > >> > >> discuss mailing list discuss(a)mpich.org > >> > >> To manage subscription options or unsubscribe: > >> > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > >> > >> > >> _______________________________________________ > >> > >> discuss mailing list discuss(a)mpich.org > >> > >> To manage subscription options or unsubscribe: > >> > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > > _______________________________________________ > >> > > discuss mailing list discuss(a)mpich.org > >> > > To manage subscription options or unsubscribe: > >> > > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > -- > >> > Pavan Balaji > >> > http://www.mcs.anl.gov/~balaji > >> > > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > > >> > > >> > _______________________________________________ > >> > discuss mailing list discuss(a)mpich.org > >> > To manage subscription options or unsubscribe: > >> > https://lists.mpich.org/mailman/listinfo/discuss > >> > >> -- > >> Pavan Balaji > >> http://www.mcs.anl.gov/~balaji > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > >> > >> > >> _______________________________________________ > >> discuss mailing list discuss(a)mpich.org > >> To manage subscription options or unsubscribe: > >> https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0