January 2016 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Unexpected "Bad termination"
by Balaji, Pavan 29 Jan '16

29 Jan '16

Luiz, The exit code (11) that was displayed indicates segmentation fault, though a human-readable string saying so would have been better. Can you try attaching a debugger? If you have ddd installed, you can do this: % mpiexec -np <num_procs> ddd ./your_application <your_application_parameters> This will fire up a bunch of ddd windows, one for each process. -- Pavan > On Jan 29, 2016, at 12:00 PM, Luiz Carlos da Costa Junior <lccostajr(a)gmail.com> wrote: > > Pavan, I am not suspicious of MPICH itself, it is very stable and we have been using it for almost a decade. > I was thinking of some kind of bad interaction with our SO, like Daniel said. > > My intention here is only to try to get some help from the experience you have. > As I commented, it isn't easy to reproduce the problem and when it happens there are no clues about it. > > I forgot to mention that the problem happens at the very beginning of execution of my application (I am not sure since there is no output from it on screen, not even the "logo" with program version). > In case of a segmentation fault as Halim said, am I supposed to get any output related to it? If yes, is there something I can do to activate or redirect it? > > Thanks again for your help. > > Best regards, > Luiz > > On 29 January 2016 at 15:26, Daniel Ibanez <dan.a.ibanez(a)gmail.com> wrote: > You're right, Pavan, I got mixed up. > I think it is the "assert (!closed) failed" that confuses me, > the other messages are clear enough. > > On Fri, Jan 29, 2016 at 12:09 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Every time the application (not MPICH) has an error, the same error message is shown. > > I'm going to go and delete that error message now since users keep thinking this is a problem in MPICH, while MPICH is only trying to be helpful and telling them that "this is not MPICH's fault. your application is broken." Maybe I should just say that in the error message. > > -- Pavan > > > On Jan 29, 2016, at 10:49 AM, Daniel Ibanez <dan.a.ibanez(a)gmail.com> wrote: > > > > Hello all, > > > > Sorry to intrude on someone else's question, > > but we also see this error message on our Linux cluster, > > with code that we're very confident is free of MPI bugs > > (has run on most of Mira, etc...). > > > > The MPICH build being used was compiled on our > > workstations, and runs well on the workstations > > (up to the 16 ranks their CPUs handle). > > However, using that MPICH install on the cluster > > causes this for small (2-4) rank operations. > > > > I'm saying this to suggest that it has to do with interactions > > of MPICH and the architecture/OS, maybe also with > > MPICH build configuration. > > > > I'll try to come up with a reproducing test program today. > > > > On Fri, Jan 29, 2016 at 11:42 AM, Halim Amer <aamer(a)anl.gov> wrote: > > Luiz, > > > > You are experiencing a segmentation fault. > > We don't have enough information to pinpoint the source of the problem, however. We usually require a small piece of code that reproduces the bug to help debugging. > > > > --Halim > > > > www.mcs.anl.gov/~aamer > > > > > > On 1/29/16 10:27 AM, Luiz Carlos da Costa Junior wrote: > > Dear all, > > > > We have been using MPICH with our software and performing execution in > > Amazon AWS Linux servers for a long time. > > We use to have in production environment MPICH version 1.4.1p1 (which - > > I know - is very old), but it has been very very stable in the latest years. > > However, recently we have been facing a "Bad termination" problem once > > in a while, so we decided to investigate this issue. > > In principle, we don't have a apparent reason to believe that the > > problem lies on our code, since there was no changes that explain this > > behavior. > > The other point is that it occurs in a intermittent fashion, if we run > > the program again it doesn't happen, so it has been difficult to > > debug/trace the source of the problem. > > > > Our first step, then, was to update the MPI version to the latest > > version 3.2. > > However, we faced the same problem (output below): > > > > ===================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = EXIT CODE: 11 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > ===================================================================================== > > [proxy:0:0@ip-10-137-129-86] HYD_pmcd_pmip_control_cmd_cb > > (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed > > [proxy:0:0@ip-10-137-129-86] HYDT_dmxu_poll_wait_for_event > > (./tools/demux/demux_poll.c:77): callback returned error status > > [mpiexec@ip-10-137-129-86] HYDT_bscu_wait_for_completion > > (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes > > terminated badly; aborting > > [mpiexec@ip-10-137-129-86] HYDT_bsci_wait_for_completion > > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error > > waiting for completion > > [mpiexec@ip-10-137-129-86] HYD_pmci_wait_for_completion > > (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting > > for completion > > [mpiexec@ip-10-137-129-86] main (./ui/mpich/mpiexec.c:405): process > > manager error waiting for completion > > > > > > Do you have any clue about what might have been causing this problem? > > Any suggestion at this point would be highly appreciated. > > > > Best regards, > > Luiz > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Unexpected "Bad termination"
by Luiz Carlos da Costa Junior 29 Jan '16

29 Jan '16

Pavan, I am not suspicious of MPICH itself, it is very stable and we have been using it for almost a decade. I was thinking of some kind of bad interaction with our SO, like Daniel said. My intention here is only to try to get some help from the experience you have. As I commented, it isn't easy to reproduce the problem and when it happens there are no clues about it. I forgot to mention that the problem happens at the very beginning of execution of my application (I am not sure since there is no output from it on screen, not even the "logo" with program version). In case of a segmentation fault as Halim said, am I supposed to get any output related to it? If yes, is there something I can do to activate or redirect it? Thanks again for your help. Best regards, Luiz On 29 January 2016 at 15:26, Daniel Ibanez <dan.a.ibanez(a)gmail.com> wrote: > You're right, Pavan, I got mixed up. > I think it is the "assert (!closed) failed" that confuses me, > the other messages are clear enough. > > On Fri, Jan 29, 2016 at 12:09 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > >> >> Every time the application (not MPICH) has an error, the same error >> message is shown. >> >> I'm going to go and delete that error message now since users keep >> thinking this is a problem in MPICH, while MPICH is only trying to be >> helpful and telling them that "this is not MPICH's fault. your application >> is broken." Maybe I should just say that in the error message. >> >> -- Pavan >> >> > On Jan 29, 2016, at 10:49 AM, Daniel Ibanez <dan.a.ibanez(a)gmail.com> >> wrote: >> > >> > Hello all, >> > >> > Sorry to intrude on someone else's question, >> > but we also see this error message on our Linux cluster, >> > with code that we're very confident is free of MPI bugs >> > (has run on most of Mira, etc...). >> > >> > The MPICH build being used was compiled on our >> > workstations, and runs well on the workstations >> > (up to the 16 ranks their CPUs handle). >> > However, using that MPICH install on the cluster >> > causes this for small (2-4) rank operations. >> > >> > I'm saying this to suggest that it has to do with interactions >> > of MPICH and the architecture/OS, maybe also with >> > MPICH build configuration. >> > >> > I'll try to come up with a reproducing test program today. >> > >> > On Fri, Jan 29, 2016 at 11:42 AM, Halim Amer <aamer(a)anl.gov> wrote: >> > Luiz, >> > >> > You are experiencing a segmentation fault. >> > We don't have enough information to pinpoint the source of the problem, >> however. We usually require a small piece of code that reproduces the bug >> to help debugging. >> > >> > --Halim >> > >> > www.mcs.anl.gov/~aamer >> > >> > >> > On 1/29/16 10:27 AM, Luiz Carlos da Costa Junior wrote: >> > Dear all, >> > >> > We have been using MPICH with our software and performing execution in >> > Amazon AWS Linux servers for a long time. >> > We use to have in production environment MPICH version 1.4.1p1 (which - >> > I know - is very old), but it has been very very stable in the latest >> years. >> > However, recently we have been facing a "Bad termination" problem once >> > in a while, so we decided to investigate this issue. >> > In principle, we don't have a apparent reason to believe that the >> > problem lies on our code, since there was no changes that explain this >> > behavior. >> > The other point is that it occurs in a intermittent fashion, if we run >> > the program again it doesn't happen, so it has been difficult to >> > debug/trace the source of the problem. >> > >> > Our first step, then, was to update the MPI version to the latest >> > version 3.2. >> > However, we faced the same problem (output below): >> > >> > >> ===================================================================================== >> > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> > = EXIT CODE: 11 >> > = CLEANING UP REMAINING PROCESSES >> > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> > >> ===================================================================================== >> > [proxy:0:0@ip-10-137-129-86] HYD_pmcd_pmip_control_cmd_cb >> > (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed >> > [proxy:0:0@ip-10-137-129-86] HYDT_dmxu_poll_wait_for_event >> > (./tools/demux/demux_poll.c:77): callback returned error status >> > [mpiexec@ip-10-137-129-86] HYDT_bscu_wait_for_completion >> > (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes >> > terminated badly; aborting >> > [mpiexec@ip-10-137-129-86] HYDT_bsci_wait_for_completion >> > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error >> > waiting for completion >> > [mpiexec@ip-10-137-129-86] HYD_pmci_wait_for_completion >> > (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting >> > for completion >> > [mpiexec@ip-10-137-129-86] main (./ui/mpich/mpiexec.c:405): process >> > manager error waiting for completion >> > >> > >> > Do you have any clue about what might have been causing this problem? >> > Any suggestion at this point would be highly appreciated. >> > >> > Best regards, >> > Luiz >> > >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Unexpected "Bad termination"
by Daniel Ibanez 29 Jan '16

29 Jan '16

You're right, Pavan, I got mixed up. I think it is the "assert (!closed) failed" that confuses me, the other messages are clear enough. On Fri, Jan 29, 2016 at 12:09 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Every time the application (not MPICH) has an error, the same error > message is shown. > > I'm going to go and delete that error message now since users keep > thinking this is a problem in MPICH, while MPICH is only trying to be > helpful and telling them that "this is not MPICH's fault. your application > is broken." Maybe I should just say that in the error message. > > -- Pavan > > > On Jan 29, 2016, at 10:49 AM, Daniel Ibanez <dan.a.ibanez(a)gmail.com> > wrote: > > > > Hello all, > > > > Sorry to intrude on someone else's question, > > but we also see this error message on our Linux cluster, > > with code that we're very confident is free of MPI bugs > > (has run on most of Mira, etc...). > > > > The MPICH build being used was compiled on our > > workstations, and runs well on the workstations > > (up to the 16 ranks their CPUs handle). > > However, using that MPICH install on the cluster > > causes this for small (2-4) rank operations. > > > > I'm saying this to suggest that it has to do with interactions > > of MPICH and the architecture/OS, maybe also with > > MPICH build configuration. > > > > I'll try to come up with a reproducing test program today. > > > > On Fri, Jan 29, 2016 at 11:42 AM, Halim Amer <aamer(a)anl.gov> wrote: > > Luiz, > > > > You are experiencing a segmentation fault. > > We don't have enough information to pinpoint the source of the problem, > however. We usually require a small piece of code that reproduces the bug > to help debugging. > > > > --Halim > > > > www.mcs.anl.gov/~aamer > > > > > > On 1/29/16 10:27 AM, Luiz Carlos da Costa Junior wrote: > > Dear all, > > > > We have been using MPICH with our software and performing execution in > > Amazon AWS Linux servers for a long time. > > We use to have in production environment MPICH version 1.4.1p1 (which - > > I know - is very old), but it has been very very stable in the latest > years. > > However, recently we have been facing a "Bad termination" problem once > > in a while, so we decided to investigate this issue. > > In principle, we don't have a apparent reason to believe that the > > problem lies on our code, since there was no changes that explain this > > behavior. > > The other point is that it occurs in a intermittent fashion, if we run > > the program again it doesn't happen, so it has been difficult to > > debug/trace the source of the problem. > > > > Our first step, then, was to update the MPI version to the latest > > version 3.2. > > However, we faced the same problem (output below): > > > > > ===================================================================================== > > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > > = EXIT CODE: 11 > > = CLEANING UP REMAINING PROCESSES > > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > > ===================================================================================== > > [proxy:0:0@ip-10-137-129-86] HYD_pmcd_pmip_control_cmd_cb > > (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed > > [proxy:0:0@ip-10-137-129-86] HYDT_dmxu_poll_wait_for_event > > (./tools/demux/demux_poll.c:77): callback returned error status > > [mpiexec@ip-10-137-129-86] HYDT_bscu_wait_for_completion > > (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes > > terminated badly; aborting > > [mpiexec@ip-10-137-129-86] HYDT_bsci_wait_for_completion > > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error > > waiting for completion > > [mpiexec@ip-10-137-129-86] HYD_pmci_wait_for_completion > > (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting > > for completion > > [mpiexec@ip-10-137-129-86] main (./ui/mpich/mpiexec.c:405): process > > manager error waiting for completion > > > > > > Do you have any clue about what might have been causing this problem? > > Any suggestion at this point would be highly appreciated. > > > > Best regards, > > Luiz > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Unexpected "Bad termination"
by Balaji, Pavan 29 Jan '16

29 Jan '16

Every time the application (not MPICH) has an error, the same error message is shown. I'm going to go and delete that error message now since users keep thinking this is a problem in MPICH, while MPICH is only trying to be helpful and telling them that "this is not MPICH's fault. your application is broken." Maybe I should just say that in the error message. -- Pavan > On Jan 29, 2016, at 10:49 AM, Daniel Ibanez <dan.a.ibanez(a)gmail.com> wrote: > > Hello all, > > Sorry to intrude on someone else's question, > but we also see this error message on our Linux cluster, > with code that we're very confident is free of MPI bugs > (has run on most of Mira, etc...). > > The MPICH build being used was compiled on our > workstations, and runs well on the workstations > (up to the 16 ranks their CPUs handle). > However, using that MPICH install on the cluster > causes this for small (2-4) rank operations. > > I'm saying this to suggest that it has to do with interactions > of MPICH and the architecture/OS, maybe also with > MPICH build configuration. > > I'll try to come up with a reproducing test program today. > > On Fri, Jan 29, 2016 at 11:42 AM, Halim Amer <aamer(a)anl.gov> wrote: > Luiz, > > You are experiencing a segmentation fault. > We don't have enough information to pinpoint the source of the problem, however. We usually require a small piece of code that reproduces the bug to help debugging. > > --Halim > > www.mcs.anl.gov/~aamer > > > On 1/29/16 10:27 AM, Luiz Carlos da Costa Junior wrote: > Dear all, > > We have been using MPICH with our software and performing execution in > Amazon AWS Linux servers for a long time. > We use to have in production environment MPICH version 1.4.1p1 (which - > I know - is very old), but it has been very very stable in the latest years. > However, recently we have been facing a "Bad termination" problem once > in a while, so we decided to investigate this issue. > In principle, we don't have a apparent reason to believe that the > problem lies on our code, since there was no changes that explain this > behavior. > The other point is that it occurs in a intermittent fashion, if we run > the program again it doesn't happen, so it has been difficult to > debug/trace the source of the problem. > > Our first step, then, was to update the MPI version to the latest > version 3.2. > However, we faced the same problem (output below): > > ===================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 11 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > ===================================================================================== > [proxy:0:0@ip-10-137-129-86] HYD_pmcd_pmip_control_cmd_cb > (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed > [proxy:0:0@ip-10-137-129-86] HYDT_dmxu_poll_wait_for_event > (./tools/demux/demux_poll.c:77): callback returned error status > [mpiexec@ip-10-137-129-86] HYDT_bscu_wait_for_completion > (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes > terminated badly; aborting > [mpiexec@ip-10-137-129-86] HYDT_bsci_wait_for_completion > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error > waiting for completion > [mpiexec@ip-10-137-129-86] HYD_pmci_wait_for_completion > (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting > for completion > [mpiexec@ip-10-137-129-86] main (./ui/mpich/mpiexec.c:405): process > manager error waiting for completion > > > Do you have any clue about what might have been causing this problem? > Any suggestion at this point would be highly appreciated. > > Best regards, > Luiz > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Unexpected "Bad termination"
by Daniel Ibanez 29 Jan '16

29 Jan '16

Hello all, Sorry to intrude on someone else's question, but we also see this error message on our Linux cluster, with code that we're very confident is free of MPI bugs (has run on most of Mira, etc...). The MPICH build being used was compiled on our workstations, and runs well on the workstations (up to the 16 ranks their CPUs handle). However, using that MPICH install on the cluster causes this for small (2-4) rank operations. I'm saying this to suggest that it has to do with interactions of MPICH and the architecture/OS, maybe also with MPICH build configuration. I'll try to come up with a reproducing test program today. On Fri, Jan 29, 2016 at 11:42 AM, Halim Amer <aamer(a)anl.gov> wrote: > Luiz, > > You are experiencing a segmentation fault. > We don't have enough information to pinpoint the source of the problem, > however. We usually require a small piece of code that reproduces the bug > to help debugging. > > --Halim > > www.mcs.anl.gov/~aamer > > > On 1/29/16 10:27 AM, Luiz Carlos da Costa Junior wrote: > >> Dear all, >> >> We have been using MPICH with our software and performing execution in >> Amazon AWS Linux servers for a long time. >> We use to have in production environment MPICH version 1.4.1p1 (which - >> I know - is very old), but it has been very very stable in the latest >> years. >> However, recently we have been facing a "Bad termination" problem once >> in a while, so we decided to investigate this issue. >> In principle, we don't have a apparent reason to believe that the >> problem lies on our code, since there was no changes that explain this >> behavior. >> The other point is that it occurs in a intermittent fashion, if we run >> the program again it doesn't happen, so it has been difficult to >> debug/trace the source of the problem. >> >> Our first step, then, was to update the MPI version to the latest >> version 3.2. >> However, we faced the same problem (output below): >> >> >> ===================================================================================== >> = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES >> = EXIT CODE: 11 >> = CLEANING UP REMAINING PROCESSES >> = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES >> >> ===================================================================================== >> [proxy:0:0@ip-10-137-129-86] HYD_pmcd_pmip_control_cmd_cb >> (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed >> [proxy:0:0@ip-10-137-129-86] HYDT_dmxu_poll_wait_for_event >> (./tools/demux/demux_poll.c:77): callback returned error status >> [mpiexec@ip-10-137-129-86] HYDT_bscu_wait_for_completion >> (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes >> terminated badly; aborting >> [mpiexec@ip-10-137-129-86] HYDT_bsci_wait_for_completion >> (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error >> waiting for completion >> [mpiexec@ip-10-137-129-86] HYD_pmci_wait_for_completion >> (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting >> for completion >> [mpiexec@ip-10-137-129-86] main (./ui/mpich/mpiexec.c:405): process >> manager error waiting for completion >> >> >> Do you have any clue about what might have been causing this problem? >> Any suggestion at this point would be highly appreciated. >> >> Best regards, >> Luiz >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Unexpected "Bad termination"
by Halim Amer 29 Jan '16

29 Jan '16

Luiz, You are experiencing a segmentation fault. We don't have enough information to pinpoint the source of the problem, however. We usually require a small piece of code that reproduces the bug to help debugging. --Halim www.mcs.anl.gov/~aamer On 1/29/16 10:27 AM, Luiz Carlos da Costa Junior wrote: > Dear all, > > We have been using MPICH with our software and performing execution in > Amazon AWS Linux servers for a long time. > We use to have in production environment MPICH version 1.4.1p1 (which - > I know - is very old), but it has been very very stable in the latest years. > However, recently we have been facing a "Bad termination" problem once > in a while, so we decided to investigate this issue. > In principle, we don't have a apparent reason to believe that the > problem lies on our code, since there was no changes that explain this > behavior. > The other point is that it occurs in a intermittent fashion, if we run > the program again it doesn't happen, so it has been difficult to > debug/trace the source of the problem. > > Our first step, then, was to update the MPI version to the latest > version 3.2. > However, we faced the same problem (output below): > > ===================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 11 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > ===================================================================================== > [proxy:0:0@ip-10-137-129-86] HYD_pmcd_pmip_control_cmd_cb > (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed > [proxy:0:0@ip-10-137-129-86] HYDT_dmxu_poll_wait_for_event > (./tools/demux/demux_poll.c:77): callback returned error status > [mpiexec@ip-10-137-129-86] HYDT_bscu_wait_for_completion > (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes > terminated badly; aborting > [mpiexec@ip-10-137-129-86] HYDT_bsci_wait_for_completion > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error > waiting for completion > [mpiexec@ip-10-137-129-86] HYD_pmci_wait_for_completion > (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting > for completion > [mpiexec@ip-10-137-129-86] main (./ui/mpich/mpiexec.c:405): process > manager error waiting for completion > > > Do you have any clue about what might have been causing this problem? > Any suggestion at this point would be highly appreciated. > > Best regards, > Luiz > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Unexpected "Bad termination"
by Luiz Carlos da Costa Junior 29 Jan '16

29 Jan '16

Dear all, We have been using MPICH with our software and performing execution in Amazon AWS Linux servers for a long time. We use to have in production environment MPICH version 1.4.1p1 (which - I know - is very old), but it has been very very stable in the latest years. However, recently we have been facing a "Bad termination" problem once in a while, so we decided to investigate this issue. In principle, we don't have a apparent reason to believe that the problem lies on our code, since there was no changes that explain this behavior. The other point is that it occurs in a intermittent fashion, if we run the program again it doesn't happen, so it has been difficult to debug/trace the source of the problem. Our first step, then, was to update the MPI version to the latest version 3.2. However, we faced the same problem (output below): ===================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = EXIT CODE: 11 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > > ===================================================================================== > [proxy:0:0@ip-10-137-129-86] HYD_pmcd_pmip_control_cmd_cb > (./pm/pmiserv/pmip_cb.c:928): assert (!closed) failed > [proxy:0:0@ip-10-137-129-86] HYDT_dmxu_poll_wait_for_event > (./tools/demux/demux_poll.c:77): callback returned error status > [mpiexec@ip-10-137-129-86] HYDT_bscu_wait_for_completion > (./tools/bootstrap/utils/bscu_wait.c:70): one of the processes terminated > badly; aborting > [mpiexec@ip-10-137-129-86] HYDT_bsci_wait_for_completion > (./tools/bootstrap/src/bsci_wait.c:23): launcher returned error waiting for > completion > [mpiexec@ip-10-137-129-86] HYD_pmci_wait_for_completion > (./pm/pmiserv/pmiserv_pmci.c:191): launcher returned error waiting for > completion > [mpiexec@ip-10-137-129-86] main (./ui/mpich/mpiexec.c:405): process > manager error waiting for completion Do you have any clue about what might have been causing this problem? Any suggestion at this point would be highly appreciated. Best regards, Luiz _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Crash on one node cause all programm termination.
by Antonio J. Peña 29 Jan '16

29 Jan '16

Hi Andris, Unfortunately right now the MPI standard leaves the behavior upon failures pretty open to implementations. With the current implementation, you can pass the flag "-disable-auto-cleanup" to mpiexec (Hydra) to prevent it from killing the non-failing processes. But, bear in mind that you shouldn't use those communicators that involved the failing process anymore. So, what you say is somehow possible if you use MPI_Comm_spawn to create the slaves and set the error handler MPI_ERRORS_RETURN in the intracommunicator. Again, this is something not defined in the standard, and hence there's no guarantee it'll work in all conditions and/or MPI implementations. On the other hand, you can search and take a look at the ULFM proposal. I hope this helps. Best, Antonio On 29/01/16 14:29, Andris wrote: > Hello! > I'm a newbie in MPI world. > I have some question about program execution. > I build some program (for example, exp1) using mpicc and run it to > multiple hosts. > mpiexec -f my_hosts -n 8 ./exp1 > 4 exp1 are running on host A (rank 0-3) and 4 - on host B (rank 4-7). > If one of them crashes, all other are terminating too. mpiexec print: > =================================================================================== > = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES > = PID 5774 RUNNING AT slave1 > = EXIT CODE: 11 > = CLEANING UP REMAINING PROCESSES > = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES > =================================================================================== > > If I understand right, I can handle in my code only MPI function errors. > In my project I need if one process is terminated, all other processes > will stay running. For example, if slave node lose power, processes on > master node stay running. Master node will know that processes on > slave node are terminated and after some time master node will rerun > these processes on slave node. > Is it possible? If yes, how? > > Big thanks! > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Antonio J. Peï¿½a, PhD Senior Researcher Barcelona Supercomputing Center http://www.bsc.es/about-bsc/staff-directory/pena-antonio WARNING / LEGAL TEXT: This message is intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. http://www.bsc.es/disclaimer _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Crash on one node cause all programm termination.
by Andris 29 Jan '16

29 Jan '16

_______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] mpich startup without networking
by Orion Poplawski 22 Jan '16

22 Jan '16

On 01/22/2016 10:17 AM, Balaji, Pavan wrote: > > That's gross. I think nemesis has the same problem that I just fixed in Hydra. > > The correct solution is to move this check for localhost/local IP to MPL and use it in both places. I'm in the middle of a different code hack right now and can't context switch to this, unfortunately. But given that you have a workaround for now, I've created a ticket for it and will deal with this later. > > https://trac.mpich.org/projects/mpich/ticket/2329 > > -- Pavan Great, thanks again for the quick help. -- Orion Poplawski Technical Manager 303-415-9701 x222 NWRA, Boulder/CoRA Office FAX: 303-415-9702 3380 Mitchell Lane orion(a)nwra.com Boulder, CO 80301 http://www.nwra.com _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0