October 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPICH 3.1 fails doesn't create core file.
by Lu, Huiwei 12 Oct '14

12 Oct '14

You may add printf to your application to report the application progress. So when it exits (or disappears), it will print the last message that can help you locate the error. — Huiwei On Oct 12, 2014, at 10:42 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > I'm not sure about my current failure. > I execute my application (MPI processes). Each process executes endless loop. After a couple of hours Master process fails (disappears - I can't see it using 'ps' command). There is no core file. I'm not sure about the reason of failure. > It looks strange that single process (15 processes executed) fails w/o core file dropping. > I suspect that I'm using MPICH in wrong way, which causes failure. > But like I said before I'm not sure about failure. > > Regards, > Anatoly. > > On Mon, Oct 13, 2014 at 6:21 AM, Wesley Bland <wbland(a)anl.gov> wrote: > Calling MPI_Abort is not an error that would cause the OS to dump a core. For that, you'll need to do something within your application that would normally cause it to dump a core. MPICH itself won't do it for you. You can add a divide by zero error to your own code and you'll get a core file of your app. What specifically within MPICH are you trying to debug? > > Thanks, > Wesley > > > > On Oct 12, 2014, at 10:09 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > >> Thank you. >> My OS (Kubuntu 14.04). >> ulimit is already set to unlimited. >> When I generate exception (abort() function or divide by zero) I always get a core file, >> but when MPICH fails (for example with MPI_Abort) no core file created. >> May be I need to change any other settings? >> >> >> Regards, >> Anatoly. >> >> On Sun, Oct 12, 2014 at 4:47 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: >> The core file is created by the OS, not MPICH. What’s the output of ‘ulimit -c’ on your machine? Could you set ‘ulimit -c unlimited’ and try again? >> >> — >> Huiwei >> >> On Oct 12, 2014, at 7:35 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> >> > Dear MPICH. >> > I'm using MPICH3.1 >> > After application run of 4-5 hours Master process fails, but it doesn't creates a core file. >> > When I used MPI_Abort (in the past) it not creates core file too. >> > Can I make MPICH create core in any fail (of course if it's MPI fail). >> > >> > Regards, >> > Anatoly. >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1 fails doesn't create core file.
by Anatoly G 12 Oct '14

12 Oct '14

I'm not sure about my current failure. I execute my application (MPI processes). Each process executes endless loop. After a couple of hours Master process fails (disappears - I can't see it using 'ps' command). There is no core file. I'm not sure about the reason of failure. It looks strange that single process (15 processes executed) fails w/o core file dropping. I suspect that I'm using MPICH in wrong way, which causes failure. But like I said before I'm not sure about failure. Regards, Anatoly. On Mon, Oct 13, 2014 at 6:21 AM, Wesley Bland <wbland(a)anl.gov> wrote: > Calling MPI_Abort is not an error that would cause the OS to dump a core. > For that, you'll need to do something within your application that would > normally cause it to dump a core. MPICH itself won't do it for you. You can > add a divide by zero error to your own code and you'll get a core file of > your app. What specifically within MPICH are you trying to debug? > > Thanks, > Wesley > > > > On Oct 12, 2014, at 10:09 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > Thank you. > My OS (Kubuntu 14.04). > ulimit is already set to unlimited. > When I generate exception (abort() function or divide by zero) I always > get a core file, > but when MPICH fails (for example with MPI_Abort) no core file created. > May be I need to change any other settings? > > > Regards, > Anatoly. > > On Sun, Oct 12, 2014 at 4:47 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > >> The core file is created by the OS, not MPICH. What’s the output of >> ‘ulimit -c’ on your machine? Could you set ‘ulimit -c unlimited’ and try >> again? >> >> — >> Huiwei >> >> On Oct 12, 2014, at 7:35 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> >> > Dear MPICH. >> > I'm using MPICH3.1 >> > After application run of 4-5 hours Master process fails, but it doesn't >> creates a core file. >> > When I used MPI_Abort (in the past) it not creates core file too. >> > Can I make MPICH create core in any fail (of course if it's MPI fail). >> > >> > Regards, >> > Anatoly. >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1 fails doesn't create core file.
by Wesley Bland 12 Oct '14

12 Oct '14

Calling MPI_Abort is not an error that would cause the OS to dump a core. For that, you'll need to do something within your application that would normally cause it to dump a core. MPICH itself won't do it for you. You can add a divide by zero error to your own code and you'll get a core file of your app. What specifically within MPICH are you trying to debug? Thanks, Wesley > On Oct 12, 2014, at 10:09 PM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > Thank you. > My OS (Kubuntu 14.04). > ulimit is already set to unlimited. > When I generate exception (abort() function or divide by zero) I always get a core file, > but when MPICH fails (for example with MPI_Abort) no core file created. > May be I need to change any other settings? > > > Regards, > Anatoly. > >> On Sun, Oct 12, 2014 at 4:47 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: >> The core file is created by the OS, not MPICH. What’s the output of ‘ulimit -c’ on your machine? Could you set ‘ulimit -c unlimited’ and try again? >> >> — >> Huiwei >> >> On Oct 12, 2014, at 7:35 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: >> >> > Dear MPICH. >> > I'm using MPICH3.1 >> > After application run of 4-5 hours Master process fails, but it doesn't creates a core file. >> > When I used MPI_Abort (in the past) it not creates core file too. >> > Can I make MPICH create core in any fail (of course if it's MPI fail). >> > >> > Regards, >> > Anatoly. >> > >> > _______________________________________________ >> > discuss mailing list discuss(a)mpich.org >> > To manage subscription options or unsubscribe: >> > https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1 fails doesn't create core file.
by Anatoly G 12 Oct '14

12 Oct '14

Thank you. My OS (Kubuntu 14.04). ulimit is already set to unlimited. When I generate exception (abort() function or divide by zero) I always get a core file, but when MPICH fails (for example with MPI_Abort) no core file created. May be I need to change any other settings? Regards, Anatoly. On Sun, Oct 12, 2014 at 4:47 PM, Lu, Huiwei <huiweilu(a)mcs.anl.gov> wrote: > The core file is created by the OS, not MPICH. What’s the output of > ‘ulimit -c’ on your machine? Could you set ‘ulimit -c unlimited’ and try > again? > > — > Huiwei > > On Oct 12, 2014, at 7:35 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > > > Dear MPICH. > > I'm using MPICH3.1 > > After application run of 4-5 hours Master process fails, but it doesn't > creates a core file. > > When I used MPI_Abort (in the past) it not creates core file too. > > Can I make MPICH create core in any fail (of course if it's MPI fail). > > > > Regards, > > Anatoly. > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.1 fails doesn't create core file.
by Lu, Huiwei 12 Oct '14

12 Oct '14

The core file is created by the OS, not MPICH. What’s the output of ‘ulimit -c’ on your machine? Could you set ‘ulimit -c unlimited’ and try again? — Huiwei On Oct 12, 2014, at 7:35 AM, Anatoly G <anatolyrishon(a)gmail.com> wrote: > Dear MPICH. > I'm using MPICH3.1 > After application run of 4-5 hours Master process fails, but it doesn't creates a core file. > When I used MPI_Abort (in the past) it not creates core file too. > Can I make MPICH create core in any fail (of course if it's MPI fail). > > Regards, > Anatoly. > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH 3.1 fails doesn't create core file.
by Anatoly G 12 Oct '14

12 Oct '14

Dear MPICH. I'm using MPICH3.1 After application run of 4-5 hours Master process fails, but it doesn't creates a core file. When I used MPI_Abort (in the past) it not creates core file too. Can I make MPICH create core in any fail (of course if it's MPI fail). Regards, Anatoly. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Wesley Bland 09 Oct '14

09 Oct '14

You will only receive the notification if you use a communicating call. That call doesn't do any communication so it won't return the error. > On Oct 9, 2014, at 11:37 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > > Hi Pavan and Wesley, > > Thanks Pavan for the information. > > Wesley, > > If I try MPI_Comm_remote_size periodically, then should I get Error-code, when the client on the other side of the intercom is killed ? ( I have already used MPI_ERRORS_RETURN). > > Thanks, > Hirak > > MPICH should detect and notify you about process failures without you having to install your own signal handler. You’ll be notified via the return code on your MPI call. You can also use an MPI Errhandler to catch this notification. So once you’ve set up your intercommunicator between your two processes, you should be able to change the default errhandler from MPI_ERRORS_ABORT to your own custom error handler (or just use MPI_ERRORS_RETURN and check the return codes). > > If you’re asking about the implementation of the proposed fault tolerance features for MPI-4 in MPICH, that’s a work in progress. We hope to have something for the MPICH 3.2 release cycle, but it’s not guaranteed and it will still be a very experimental feature given that the MPI Forum has not yet actually adopted the fault tolerance proposal. > > Thanks, > Wesley > > > On Oct 9, 2014, at 11:12 AM, Balaji, Pavan <balaji at anl.gov> wrote: > > > > > > I’ll let Wesley answer the FT notification part. > > > > MPI-4 is a major standard release of MPI. That’ll take a few years. We are currently working on the MPI-3.1 release. (hopefully you know the difference between MPI and MPICH, otherwise it’ll take many emails to explain that part :-) ). > > > > — Pavan > > > > On Oct 9, 2014, at 11:09 AM, Roy, Hirak <Hirak_Roy at mentor.com> wrote: > > > >> Hi Pavan, > >> > >> Just wondering with the current release whether we have any way to notify the server that client is terminated unexpectedly! > >> Another point: When do we expect to have MPI-4 release out? > >> > >> Thanks, > >> Hirak > >> > >> > >> Please don’t rely on this feature. We are preparing for MPI-4 Fault Tolerance and are in the process of reworking a bunch of this stuff. This might or might not exist in the future if you are planning to use this for production code. > >> > >> — Pavan > >> > >> On Oct 9, 2014, at 10:57 AM, Roy, Hirak <Hirak_Roy at mentor.com> wrote: > >> > >>> > >>> Hi Sangmin, > >>> > >>> The readme of mpich says the following : > >>> > >>> FAILURE NOTIFICATION: THIS IS AN UNSUPPORTED FEATURE AND WILL > >>> ALMOST CERTAINLY CHANGE IN THE FUTURE! > >>> > >>> In the current release, hydra notifies the MPICH library of failed > >>> processes by sending a SIGUSR1 signal. The application can catch > >>> this signal to be notified of failed processes. If the application > >>> replaces the library's signal handler with its own, the application > >>> must be sure to call the library's handler from it's own > >>> handler. Note that you cannot call any MPI function from inside a > >>> signal handler. > >>> > >>> If this is true, should not I expect SIGUSR1? > >>> > >>> > >>> Thanks, > >>> Hirak > >>> First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. > >>> > >>> — Sangmin > >>> > >>> > >>> On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy at mentor.com<mailto:Hirak_Roy at mentor.com>> wrote: > >>> > >>> Hi , > >>> > >>> I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) > >>> 1> mpiexec –disable-auto-cleanup –n 1 ./server > >>> 2> mpiexec –disable-auto-cleanup –n 1 ./client > >>> > >>> The server opens a port and does MPI_Comm_accept. > >>> The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. > >>> I don’t do MPI_Comm_merge. > >>> > >>> I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). > >>> > >>>>> signal (SIGUSR1, mysignalhandler); > >>> > >>> Now suppose, the ‘client’ process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process ‘server’. > >>> However, I don’t get any signal in ‘server’ process. > >>> Am I doing something wrong? > >>> I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. > >>> > >>> Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? > >>> > >>> > >>> Thanks, > >>> Hirak > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Roy, Hirak 09 Oct '14

09 Oct '14

Hi Pavan and Wesley, Thanks Pavan for the information. Wesley, If I try MPI_Comm_remote_size periodically, then should I get Error-code, when the client on the other side of the intercom is killed ? ( I have already used MPI_ERRORS_RETURN). Thanks, Hirak ________________________________ MPICH should detect and notify you about process failures without you having to install your own signal handler. You'll be notified via the return code on your MPI call. You can also use an MPI Errhandler to catch this notification. So once you've set up your intercommunicator between your two processes, you should be able to change the default errhandler from MPI_ERRORS_ABORT to your own custom error handler (or just use MPI_ERRORS_RETURN and check the return codes). If you're asking about the implementation of the proposed fault tolerance features for MPI-4 in MPICH, that's a work in progress. We hope to have something for the MPICH 3.2 release cycle, but it's not guaranteed and it will still be a very experimental feature given that the MPI Forum has not yet actually adopted the fault tolerance proposal. Thanks, Wesley > On Oct 9, 2014, at 11:12 AM, Balaji, Pavan <balaji at anl.gov<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > > > I'll let Wesley answer the FT notification part. > > MPI-4 is a major standard release of MPI. That'll take a few years. We are currently working on the MPI-3.1 release. (hopefully you know the difference between MPI and MPICH, otherwise it'll take many emails to explain that part :-) ). > > - Pavan > > On Oct 9, 2014, at 11:09 AM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > >> Hi Pavan, >> >> Just wondering with the current release whether we have any way to notify the server that client is terminated unexpectedly! >> Another point: When do we expect to have MPI-4 release out? >> >> Thanks, >> Hirak >> >> >> Please don't rely on this feature. We are preparing for MPI-4 Fault Tolerance and are in the process of reworking a bunch of this stuff. This might or might not exist in the future if you are planning to use this for production code. >> >> - Pavan >> >> On Oct 9, 2014, at 10:57 AM, Roy, Hirak <Hirak_Roy at mentor.com> wrote: >> >>> >>> Hi Sangmin, >>> >>> The readme of mpich says the following : >>> >>> FAILURE NOTIFICATION: THIS IS AN UNSUPPORTED FEATURE AND WILL >>> ALMOST CERTAINLY CHANGE IN THE FUTURE! >>> >>> In the current release, hydra notifies the MPICH library of failed >>> processes by sending a SIGUSR1 signal. The application can catch >>> this signal to be notified of failed processes. If the application >>> replaces the library's signal handler with its own, the application >>> must be sure to call the library's handler from it's own >>> handler. Note that you cannot call any MPI function from inside a >>> signal handler. >>> >>> If this is true, should not I expect SIGUSR1? >>> >>> >>> Thanks, >>> Hirak >>> First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. >>> >>> - Sangmin >>> >>> >>> On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy at mentor.com<mailto:Hirak_Roy at mentor.com>> wrote: >>> >>> Hi , >>> >>> I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) >>> 1> mpiexec -disable-auto-cleanup -n 1 ./server >>> 2> mpiexec -disable-auto-cleanup -n 1 ./client >>> >>> The server opens a port and does MPI_Comm_accept. >>> The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. >>> I don't do MPI_Comm_merge. >>> >>> I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). >>> >>>>> signal (SIGUSR1, mysignalhandler); >>> >>> Now suppose, the 'client' process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process 'server'. >>> However, I don't get any signal in 'server' process. >>> Am I doing something wrong? >>> I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. >>> >>> Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? >>> >>> >>> Thanks, >>> Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Roy, Hirak 09 Oct '14

09 Oct '14

Thanks Sangmin for your suggestion. I have taken a similar approach. I have notified one of the connected clients to reconnect and terminate the blocking accept. Regards, Hirak Hirak, I would correct my previous response. We discussed this issue inside the MPICH team, and we concluded this is a gray area of the standard (it's not clear when the functions are used with threads). Thus, we are not sure if your use case is right or wrong because you were using different communicators. BTW, you may solve this issue by spawning a new process that connects back. Thanks, Sangmin On Oct 9, 2014, at 7:27 AM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss><mailto:Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>>> wrote: Thanks Sangmin. I would change the strategy to terminate the accept. Regards, Hirak Hi Hirak, I checked your code. I think your code is incorrect because you are trying to establish communication within a single process. You are using two threads and two duplicated communicators, but they are associated with a single process. MPI_Comm_accept() and MPI_Comm_connect() are used to establish communication between "two sets of MPI processes" (i.e., at least two different processes are needed). Please refer to 10.4 Establishing Communication (p. 387) in MPI Standard 3.0. - Sangmin From: Roy, Hirak Sent: Thursday, October 09, 2014 2:17 AM To: discuss at mpich.org<https://lists.mpich.org/mailman/listinfo/discuss><mailto:discuss at mpich.org<https://lists.mpich.org/mailman/listinfo/discuss>> Subject: Re: How to terminate MPI_Comm_accept Hi Pavan, Here is the code attached. Let me know if you think the code is incorrect. I used MPI-3.0.4 with sock device. Compile command : export MPI_ROOT = /home/hroy/local/mpich-3.0.4/linux_x86_64 #export MPI_ROOT = /home/hroy/local/mpich-3.0.4.nemesis/linux_x86_64 export MPI_BIN = ${MPI_ROOT}/bin export MPI_LIB = ${MPI_ROOT}/lib export INCLUDE_DIR = ${MPI_ROOT}/include compile: g++ -g main.cpp -I ${INCLUDE_DIR} ${MPI_LIB}/libmpich.a ${MPI_LIB}/libmpl.a -lpthread Thanks, Hirak Can you send us a simple program that reproduces the issue? - Pavan On Oct 8, 2014, at 2:54 PM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > Hi Pavan, > > I even tried duping the communicator (MPI_COMM_SELF à DUP1, DUP2) before I do any kind of MPI calls. > Still it does not work. > > Thanks, > Hirak > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Wesley Bland 09 Oct '14

09 Oct '14

MPICH should detect and notify you about process failures without you having to install your own signal handler. You’ll be notified via the return code on your MPI call. You can also use an MPI Errhandler to catch this notification. So once you’ve set up your intercommunicator between your two processes, you should be able to change the default errhandler from MPI_ERRORS_ABORT to your own custom error handler (or just use MPI_ERRORS_RETURN and check the return codes). If you’re asking about the implementation of the proposed fault tolerance features for MPI-4 in MPICH, that’s a work in progress. We hope to have something for the MPICH 3.2 release cycle, but it’s not guaranteed and it will still be a very experimental feature given that the MPI Forum has not yet actually adopted the fault tolerance proposal. Thanks, Wesley > On Oct 9, 2014, at 11:12 AM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > > I’ll let Wesley answer the FT notification part. > > MPI-4 is a major standard release of MPI. That’ll take a few years. We are currently working on the MPI-3.1 release. (hopefully you know the difference between MPI and MPICH, otherwise it’ll take many emails to explain that part :-) ). > > — Pavan > > On Oct 9, 2014, at 11:09 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > >> Hi Pavan, >> >> Just wondering with the current release whether we have any way to notify the server that client is terminated unexpectedly! >> Another point: When do we expect to have MPI-4 release out? >> >> Thanks, >> Hirak >> >> >> Please don’t rely on this feature. We are preparing for MPI-4 Fault Tolerance and are in the process of reworking a bunch of this stuff. This might or might not exist in the future if you are planning to use this for production code. >> >> — Pavan >> >> On Oct 9, 2014, at 10:57 AM, Roy, Hirak <Hirak_Roy at mentor.com> wrote: >> >>> >>> Hi Sangmin, >>> >>> The readme of mpich says the following : >>> >>> FAILURE NOTIFICATION: THIS IS AN UNSUPPORTED FEATURE AND WILL >>> ALMOST CERTAINLY CHANGE IN THE FUTURE! >>> >>> In the current release, hydra notifies the MPICH library of failed >>> processes by sending a SIGUSR1 signal. The application can catch >>> this signal to be notified of failed processes. If the application >>> replaces the library's signal handler with its own, the application >>> must be sure to call the library's handler from it's own >>> handler. Note that you cannot call any MPI function from inside a >>> signal handler. >>> >>> If this is true, should not I expect SIGUSR1? >>> >>> >>> Thanks, >>> Hirak >>> First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. >>> >>> — Sangmin >>> >>> >>> On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy at mentor.com<mailto:Hirak_Roy at mentor.com>> wrote: >>> >>> Hi , >>> >>> I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) >>> 1> mpiexec –disable-auto-cleanup –n 1 ./server >>> 2> mpiexec –disable-auto-cleanup –n 1 ./client >>> >>> The server opens a port and does MPI_Comm_accept. >>> The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. >>> I don’t do MPI_Comm_merge. >>> >>> I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). >>> >>>>> signal (SIGUSR1, mysignalhandler); >>> >>> Now suppose, the ‘client’ process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process ‘server’. >>> However, I don’t get any signal in ‘server’ process. >>> Am I doing something wrong? >>> I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. >>> >>> Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? >>> >>> >>> Thanks, >>> Hirak >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > > -- > Pavan Balaji ✉️ > http://www.mcs.anl.gov/~balaji > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0