December 2013 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Fatal error in PMPI_Isend: Internal MPI error!, error stack
by Pavan Balaji 08 Dec '13

08 Dec '13

Hello, Can you provide a simple program that reproduces this error? Also, please make sure you are using the latest version of mpich — either 3.0.4 or 3.1rc2. Thanks, — Pavan On Dec 8, 2013, at 10:50 PM, 박보국 <limited107(a)gmail.com> wrote: > Hi, > I'm trying to run Titan2D on a linux. I found a similar problem on a different system. ( Ubuntu PC / CentOS Cluster ) > Small operations, the error does not occur. However, In the case of long simulation error occur. > > The error message is shown below. > Fatal error in PMPI_Isend: Internal MPI error!, error stack: > PMPI_Isend(148): MPI_Isend(buf=0x244c53c0, count=45, dtype=USER<struct>, dest=13, tag=22676, MPI_COMM_WORLD, request=0x227ab074) failed > (unknown)(): Internal MPI error! > Fatal error in PMPI_Test: Other MPI error, error stack: > PMPI_Test(168)............: MPI_Test(request=0x147a6068, flag=0x7fff9bd5098c, status=0x7fff9bd50960) failed > MPIR_Test_impl(63)........: > dequeue_and_set_error(596): Communication error with rank 2 > Fatal error in PMPI_Test: Other MPI error, error stack: > PMPI_Test(168)............: MPI_Test(request=0x2deaf058, flag=0x7fff88bd375c, status=0x7fff88bd3730) failed > MPIR_Test_impl(63)........: > dequeue_and_set_error(596): Communication error with rank 2 > Fatal error in PMPI_Barrier: Other MPI error, error stack: > PMPI_Barrier(425)...........: MPI_Barrier(MPI_COMM_WORLD) failed > MPIR_Barrier_impl(292)......: > MPIR_Barrier_or_coll_fn(121): > MPIR_Barrier_intra(83)......: > dequeue_and_set_error(596)..: Communication error with rank 10 > Fatal error in PMPI_Barrier: Other MPI error, error stack: > PMPI_Barrier(425)...........: MPI_Barrier(MPI_COMM_WORLD) failed > MPIR_Barrier_impl(306)......: > MPIR_Bcast_impl(1321).......: > MPIR_Bcast_intra(1155)......: > MPIR_Bcast_binomial(213)....: Failure during collective > MPIR_Barrier_impl(292)......: > MPIR_Barrier_or_coll_fn(121): > MPIR_Barrier_intra(83)......: > dequeue_and_set_error(596)..: Communication error with rank 0 > Fatal error in PMPI_Isend: Other MPI error, error stack: > PMPI_Isend(148)..........: MPI_Isend(buf=0x282dbdc0, count=260, dtype=USER<struct>, dest=10, tag=22683, MPI_COMM_WORLD, request=0x26b3a9b8) failed > MPID_nem_lmt_RndvSend(81): > MPIDI_CH3_RndvSend(63)...: failure occurred while attempting to send RTS packet > MPIDI_CH3_iStartMsg(36)..: Communication error with rank 10 > Fatal error in PMPI_Test: Other MPI error, error stack: > PMPI_Test(168)..................: MPI_Test(request=0xa66b46c, flag=0x7fff849b92dc, status=0x7fff849b92b0) failed > MPIR_Test_impl(63)..............: > MPIDI_CH3U_Recvq_FDU_or_AEP(380): Communication error with rank 3 > Thanks, > Park Bokuk > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Fatal error in PMPI_Isend: Internal MPI error!, error stack
by 박보국 08 Dec '13

08 Dec '13

Hi, I'm trying to run Titan2D on a linux. I found a similar problem on a different system. ( Ubuntu PC / CentOS Cluster ) Small operations, the error does not occur. However, In the case of long simulation error occur. The error message is shown below. Fatal error in PMPI_Isend: Internal MPI error!, error stack: PMPI_Isend(148): MPI_Isend(buf=0x244c53c0, count=45, dtype=USER<struct>, dest=13, tag=22676, MPI_COMM_WORLD, request=0x227ab074) failed (unknown)(): Internal MPI error! Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(168)............: MPI_Test(request=0x147a6068, flag=0x7fff9bd5098c, status=0x7fff9bd50960) failed MPIR_Test_impl(63)........: dequeue_and_set_error(596): Communication error with rank 2 Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(168)............: MPI_Test(request=0x2deaf058, flag=0x7fff88bd375c, status=0x7fff88bd3730) failed MPIR_Test_impl(63)........: dequeue_and_set_error(596): Communication error with rank 2 Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(425)...........: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(292)......: MPIR_Barrier_or_coll_fn(121): MPIR_Barrier_intra(83)......: dequeue_and_set_error(596)..: Communication error with rank 10 Fatal error in PMPI_Barrier: Other MPI error, error stack: PMPI_Barrier(425)...........: MPI_Barrier(MPI_COMM_WORLD) failed MPIR_Barrier_impl(306)......: MPIR_Bcast_impl(1321).......: MPIR_Bcast_intra(1155)......: MPIR_Bcast_binomial(213)....: Failure during collective MPIR_Barrier_impl(292)......: MPIR_Barrier_or_coll_fn(121): MPIR_Barrier_intra(83)......: dequeue_and_set_error(596)..: Communication error with rank 0 Fatal error in PMPI_Isend: Other MPI error, error stack: PMPI_Isend(148)..........: MPI_Isend(buf=0x282dbdc0, count=260, dtype=USER<struct>, dest=10, tag=22683, MPI_COMM_WORLD, request=0x26b3a9b8) failed MPID_nem_lmt_RndvSend(81): MPIDI_CH3_RndvSend(63)...: failure occurred while attempting to send RTS packet MPIDI_CH3_iStartMsg(36)..: Communication error with rank 10 Fatal error in PMPI_Test: Other MPI error, error stack: PMPI_Test(168)..................: MPI_Test(request=0xa66b46c, flag=0x7fff849b92dc, status=0x7fff849b92b0) failed MPIR_Test_impl(63)..............: MPIDI_CH3U_Recvq_FDU_or_AEP(380): Communication error with rank 3 Thanks, Park Bokuk

1 0

Re: [mpich-discuss] mpifort wrapper compiler?
by Jeff Hammond 07 Dec '13

07 Dec '13

Users really shouldn't read this email. And yes Dave Goodell, I know I'm tilting at windmills :-) >>> Is this MPICH, or IBM MPI? >> >> That dichotomy is dated, but not particularly relevant for this discussion. > > Ah -- forgive my ignorance; I thought/assumed IBM periodically seeded from upstream MPICH and was an actual separate MPI implementation distribution. My bad. IBM's implementation of MPI based upon PAMI, which ships on Blue Gene, PERCS and InfiniBand systems, is downstream from MPICH. Whether or not one can build the MPICH master on Blue Gene/Q on a given day is a function of planetary alignment. I have certainly done it, albeit not recently. I know IBM supports other MPI implementations, but I really can't comment on the details. The MPI formerly known as Platform is not dead as I understand it. >>> When I install MPICH 3.x on my Linux commodity cluster, I only see mpif77 and mpif90. Does MPICH do something different when installed on a BG system? >> >> IBM deploys MPICH for BG with a bunch of wrappers, one for each bgxl* >> compiler (including an _r variant for each as well). They are named >> mpixl* to prevent confusion with the GCC wrappers, at least when some >> variants of MPI are used. See >> https://wiki.alcf.anl.gov/parts/index.php/Mira_MPI_Documentation#Explanatio… >> for details. If you have further questions, request an ALCF account >> and you can dig around yourself :-) > > Good enough. > > But my original point remains: those wrappers are not in MPICH itself -- they come from IBM. I honestly don't know exactly where the generation of mpixl{c,cxx,f77,f90,f95,f2003,f2008}{,_r} happens w.r.t. MPICH's build system. It might be part of the IBM toolchain packaging. >> MPICH does not have these, but I don't see why it would today given >> that it only supports the F77 and F90 interfaces. > > I'm going to be pedantic again, because this is a critical point: there are no such things as "F77" and "F90" interfaces for MPI. > > There is mpif.h, the mpi module, and the mpi_f08 module. Labeling them as F77 and F90 is incorrect. The interfaces _are in the spirit_ of F77 and F90. It is stupid to use mpif.h in F90+ because modules are infinitely better that common blocks. Fortran common blocks are a crime against humanity that are only acceptable in F77 codes. That mpif.h relies upon common blocks for MPI_BOTTOM, for instance, makes them a uniquely F77-like design. The use of a module is something that is strictly impossible in an F77 code and thus has to be considered an F90+ interface. As F90 is the first language for which the MPI module can be used, I think it is reasonable to call it the F90 interface. I accept that people will use the mpi module rather than the mpi_f08 module in F08 codes because of inertia, but that doesn't mean that the mpi_f08 module shouldn't be called the F08 interface to MPI. After all, it's in the name. > 1st case in point: mpif.h never worked for Fortran 77. Yes, and as Nick McLaren points out, all forms of asynchronous communication in MPI (e.g. MPI_Isend and MPI_Put) are incompatible with every specification of Fortran, including 2008, unless we include TS 29113, right? I suppose MPI works if all buffers in Fortran are in common blocks or modules. Other than 11.7.4, does the MPI Standard say that is required? I guess all the Fortran codes calling MPI_IRECV with automatic buffers are not strictly valid, regardless of whether they 'include mpif.h' or 'use mpi'. Pedantic Squyres must admit that there is no valid Fortran interface to MPI, just ones that happen to work due to the absence of sufficiently optimizing compilers :-D > 2nd case in point: you can use mpif.h with any version of Fortran that supports MPI (e.g., 90, 95, 03, 08), and you can use the mpi module with any version of Fortran that supports MPI and modules (e.g., 90, 95, 03, 08). The mpi_f08 module works with F08 and later, obviously (it uses some newer features that debuted in Fortran 2008). > Isn't every C89 program also a C++11 program? So I can go around saying that MPICH is written in C++11? I don't consider arguments based upon the backwards compatibility of languages to be particularly useful. > My point: They are not F77 and F90 interfaces, since they can be used with many different Fortran versions. Sure, and we need to stop talking about how we deleted "the" C++ interface since we really only deleted the _newer_ C++ interface and made sure the older C89-based C++ interface works with C++ types. > Also, MPI states that you should be able to use all 3 interfaces in a single compilation unit. It also defines MPI_Alltoallw. > So other than the backwards compatibility issue, a single "mpifort" wrapper that makes all 3 interfaces available makes perfect sense. > > I don't think there can be any expectation that "mpif77" provides a true, honest, and only Fortran 77 compiler (because MPI never worked with Fortran 77). How about we have mpifisffc (MPI FIxed Source Form Fortran Compiler) and mpifrsffc (MPI FRee Source Form Fortran Compiler) then? > "mpif90" might have a reasonable expectation that it only provides Fortran 90 syntax, and potentially errors out if you use F03/F08 syntax. But that's kinda weak. I believe that F90 compilers are allowed to define array layout in a nonstandard way whereas they removed this in F03 (because it was a horrible idea in the first place). In theory, forcing the user to use an F08 compiler on F90 code can reduce performance. If Fortran 2008 deprecates any features of older versions of Fortran, it may be a problem for users to force everyone to use the newest compiler. The only natural conclusion at this point is that it is critical to deprecate both mpif.h and the 'mpi' module, since the 'mpi_f08' module is clearly the only sane way to use MPI from Fortran. Best, Jeff -- Jeff Hammond jeff.science(a)gmail.com

1 0

Re: [mpich-discuss] MPI memory allocation.
by Pavan Balaji 07 Dec '13

07 Dec '13

As much as I hate saying this — some people find it easier to think of it as “MPICH3”. — Pavan On Dec 7, 2013, at 7:37 AM, Wesley Bland <wbland(a)mcs.anl.gov> wrote: > MPICH is just the new version of MPICH2. We renamed it when we went past version 3.0. > > On Dec 7, 2013, at 3:55 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > >> Ok. I"ll try both Issend, and next step to upgrade MPICH to 3.0.4. >> I thought before that MPICH & MPICH2 are two different branches, when MPICH2 partially supports Fault tolerance, but MPICH not. Now I understand, that I was wrong and MPICH2 is just main version of MPICH. >> >> Thank you very much, >> Anatoly. >> >> >> >> On Thu, Dec 5, 2013 at 11:01 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> The master is receiving too many incoming messages than it can match quickly enough with Irecvs. Try using MPI_Issend instead of MPI_Isend. >> >> Rajeev >> >> On Dec 5, 2013, at 2:58 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> >> > Hello. >> > I"m using MPICH2 1.5. >> > My system contains master and 16 slaves. >> > System uses number of communicators. >> > The single communicator used for below scenario: >> > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend and waits using MPI_Wait. >> > Master starts with MPI_Irecv to each slave >> > Then in endless loop: >> > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. >> > >> > Another communicator used for broadcast communication (commands between master + slaves), >> > but it's not used in parallel with previous communicator, >> > only before or after data transfer. >> > >> > The system executed on two computers linked by 1Gbit/s Ethernet. >> > Master executed on first computer, all slaves on other one. >> > Network traffic is ~800Mbit/s. >> > >> > After 1-2 minutes of execution, master process starts to increase it's memory allocation and network traffic becomes low. >> > This memory allocation & network traffic slow down continues until fail of MPI, >> > without core file save. >> > My program doesn't allocate memory. Can you please explain this behaviour. >> > How can I cause MPI to stop sending slaves if Master can't serve such traffic, instead of memory allocation and fail? >> > >> > >> > Thank you, >> > Anatoly. >> > >> > P.S. >> > On my stand alone test, I simulate similar behaviour, but with single thread on each process (master & hosts). >> > When I start stand alone test, master stops slaves until it completes accumulated data processing and MPI doesn't increase memory allocation. >> > When Master is free slaves continue to send data. >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0

Re: [mpich-discuss] MPI memory allocation.
by Wesley Bland 07 Dec '13

07 Dec '13

MPICH is just the new version of MPICH2. We renamed it when we went past version 3.0. > On Dec 7, 2013, at 3:55 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > Ok. I"ll try both Issend, and next step to upgrade MPICH to 3.0.4. > I thought before that MPICH & MPICH2 are two different branches, when MPICH2 partially supports Fault tolerance, but MPICH not. Now I understand, that I was wrong and MPICH2 is just main version of MPICH. > > Thank you very much, > Anatoly. > > > >> On Thu, Dec 5, 2013 at 11:01 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: >> The master is receiving too many incoming messages than it can match quickly enough with Irecvs. Try using MPI_Issend instead of MPI_Isend. >> >> Rajeev >> >> On Dec 5, 2013, at 2:58 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> >> > Hello. >> > I"m using MPICH2 1.5. >> > My system contains master and 16 slaves. >> > System uses number of communicators. >> > The single communicator used for below scenario: >> > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend and waits using MPI_Wait. >> > Master starts with MPI_Irecv to each slave >> > Then in endless loop: >> > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. >> > >> > Another communicator used for broadcast communication (commands between master + slaves), >> > but it's not used in parallel with previous communicator, >> > only before or after data transfer. >> > >> > The system executed on two computers linked by 1Gbit/s Ethernet. >> > Master executed on first computer, all slaves on other one. >> > Network traffic is ~800Mbit/s. >> > >> > After 1-2 minutes of execution, master process starts to increase it's memory allocation and network traffic becomes low. >> > This memory allocation & network traffic slow down continues until fail of MPI, >> > without core file save. >> > My program doesn't allocate memory. Can you please explain this behaviour. >> > How can I cause MPI to stop sending slaves if Master can't serve such traffic, instead of memory allocation and fail? >> > >> > >> > Thank you, >> > Anatoly. >> > >> > P.S. >> > On my stand alone test, I simulate similar behaviour, but with single thread on each process (master & hosts). >> > When I start stand alone test, master stops slaves until it completes accumulated data processing and MPI doesn't increase memory allocation. >> > When Master is free slaves continue to send data. >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI memory allocation.
by Anatoly G 07 Dec '13

07 Dec '13

Ok. I"ll try both Issend, and next step to upgrade MPICH to 3.0.4. I thought before that MPICH & MPICH2 are two different branches, when MPICH2 partially supports Fault tolerance, but MPICH not. Now I understand, that I was wrong and MPICH2 is just main version of MPICH. Thank you very much, Anatoly. On Thu, Dec 5, 2013 at 11:01 PM, Rajeev Thakur <thakur(a)mcs.anl.gov> wrote: > The master is receiving too many incoming messages than it can match > quickly enough with Irecvs. Try using MPI_Issend instead of MPI_Isend. > > Rajeev > > On Dec 5, 2013, at 2:58 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > > Hello. > > I"m using MPICH2 1.5. > > My system contains master and 16 slaves. > > System uses number of communicators. > > The single communicator used for below scenario: > > Each slave sends non-stop 2Kbyte data buffer using MPI_Isend and waits > using MPI_Wait. > > Master starts with MPI_Irecv to each slave > > Then in endless loop: > > MPI_Waitany and MPI_Irecv on rank returned by MPI_Waitany. > > > > Another communicator used for broadcast communication (commands between > master + slaves), > > but it's not used in parallel with previous communicator, > > only before or after data transfer. > > > > The system executed on two computers linked by 1Gbit/s Ethernet. > > Master executed on first computer, all slaves on other one. > > Network traffic is ~800Mbit/s. > > > > After 1-2 minutes of execution, master process starts to increase it's > memory allocation and network traffic becomes low. > > This memory allocation & network traffic slow down continues until fail > of MPI, > > without core file save. > > My program doesn't allocate memory. Can you please explain this > behaviour. > > How can I cause MPI to stop sending slaves if Master can't serve such > traffic, instead of memory allocation and fail? > > > > > > Thank you, > > Anatoly. > > > > P.S. > > On my stand alone test, I simulate similar behaviour, but with single > thread on each process (master & hosts). > > When I start stand alone test, master stops slaves until it completes > accumulated data processing and MPI doesn't increase memory allocation. > > When Master is free slaves continue to send data. > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Mpich + Siesta erro
by Julio Henrique 06 Dec '13

06 Dec '13

Ok Jeff. Thank you. julio. CC: discuss(a)mpich.org From: jeff.science(a)gmail.com Date: Fri, 6 Dec 2013 19:09:36 -0600 To: discuss(a)mpich.org Subject: Re: [mpich-discuss] Mpich + Siesta erro I bet you (1) Siesta doesn't check MPI return codes and (2) Siesta has no way to handle node failure. I bet it can't even handle malloc returning NULL. If a node fails more than once a month, the hardware is bad and you should buy new stuff. Jeff Sent from my iPhone On Dec 6, 2013, at 7:04 PM, Julio Henrique <juliohenrique(a)msn.com> wrote: Okay Pavan. I'll try that. Then I'll return the result. Thank's. Julio. > From: balaji(a)mcs.anl.gov > Date: Fri, 6 Dec 2013 19:01:08 -0600 > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Mpich + Siesta erro > > Hi Julio, > > There are two steps needed for this: > > 1. You need to tell your MPI application to return errors instead of aborting. > > 2. Tell the process manager to not clean up your remaining processes when one of the processes dies. > > Details on both these steps are listed in the "Fault Tolerance” section of the MPICH README. Please try it out and let us know how it goes. > > — Pavan > > On Dec 6, 2013, at 6:54 PM, Julio Henrique <juliohenrique(a)msn.com> wrote: > > > > > I am using mpich-3.0.4 on cluster with 7 nodes running the latest version of siesta. My problem is that when a one node goes down, the siesta and mpich stops running and giveserror. > > How do I get when a node falls, siesta and mpich continue to run? > > Thank's. > > Julio. > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Mpich + Siesta erro
by Jeff Hammond 06 Dec '13

06 Dec '13

I bet you (1) Siesta doesn't check MPI return codes and (2) Siesta has no way to handle node failure. I bet it can't even handle malloc returning NULL. If a node fails more than once a month, the hardware is bad and you should buy new stuff. Jeff Sent from my iPhone > On Dec 6, 2013, at 7:04 PM, Julio Henrique <juliohenrique(a)msn.com> wrote: > > > Okay Pavan. I'll try that. Then I'll return the result. > Thank's. > Julio. > > > > > > > > > From: balaji(a)mcs.anl.gov > > Date: Fri, 6 Dec 2013 19:01:08 -0600 > > To: discuss(a)mpich.org > > Subject: Re: [mpich-discuss] Mpich + Siesta erro > > > > Hi Julio, > > > > There are two steps needed for this: > > > > 1. You need to tell your MPI application to return errors instead of aborting. > > > > 2. Tell the process manager to not clean up your remaining processes when one of the processes dies. > > > > Details on both these steps are listed in the "Fault Tolerance” section of the MPICH README. Please try it out and let us know how it goes. > > > > — Pavan > > > > On Dec 6, 2013, at 6:54 PM, Julio Henrique <juliohenrique(a)msn.com> wrote: > > > > > > > > I am using mpich-3.0.4 on cluster with 7 nodes running the latest version of siesta. My problem is that when a one node goes down, the siesta and mpich stops running and giveserror. > > > How do I get when a node falls, siesta and mpich continue to run? > > > Thank's. > > > Julio. > > > > > > > > > > > > _______________________________________________ > > > discuss mailing list discuss(a)mpich.org > > > To manage subscription options or unsubscribe: > > > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > > Pavan Balaji > > http://www.mcs.anl.gov/~balaji > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Mpich + Siesta erro
by Julio Henrique 06 Dec '13

06 Dec '13

Okay Pavan. I'll try that. Then I'll return the result. Thank's. Julio. > From: balaji(a)mcs.anl.gov > Date: Fri, 6 Dec 2013 19:01:08 -0600 > To: discuss(a)mpich.org > Subject: Re: [mpich-discuss] Mpich + Siesta erro > > Hi Julio, > > There are two steps needed for this: > > 1. You need to tell your MPI application to return errors instead of aborting. > > 2. Tell the process manager to not clean up your remaining processes when one of the processes dies. > > Details on both these steps are listed in the "Fault Tolerance” section of the MPICH README. Please try it out and let us know how it goes. > > — Pavan > > On Dec 6, 2013, at 6:54 PM, Julio Henrique <juliohenrique(a)msn.com> wrote: > > > > > I am using mpich-3.0.4 on cluster with 7 nodes running the latest version of siesta. My problem is that when a one node goes down, the siesta and mpich stops running and giveserror. > > How do I get when a node falls, siesta and mpich continue to run? > > Thank's. > > Julio. > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Mpich + Siesta erro
by Pavan Balaji 06 Dec '13

06 Dec '13

Also, there have been several fixes for this in the recently released 3.1rc2. I’d recommend trying that out instead of mpich-3.0.4. Regards, — Pavan On Dec 6, 2013, at 7:01 PM, Pavan Balaji <balaji(a)mcs.anl.gov> wrote: > Hi Julio, > > There are two steps needed for this: > > 1. You need to tell your MPI application to return errors instead of aborting. > > 2. Tell the process manager to not clean up your remaining processes when one of the processes dies. > > Details on both these steps are listed in the "Fault Tolerance” section of the MPICH README. Please try it out and let us know how it goes. > > — Pavan > > On Dec 6, 2013, at 6:54 PM, Julio Henrique <juliohenrique(a)msn.com> wrote: > >> >> I am using mpich-3.0.4 on cluster with 7 nodes running the latest version of siesta. My problem is that when a one node goes down, the siesta and mpich stops running and giveserror. >> How do I get when a node falls, siesta and mpich continue to run? >> Thank's. >> Julio. >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji http://www.mcs.anl.gov/~balaji

1 0