October 2014 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Balaji, Pavan 09 Oct '14

09 Oct '14

I’ll let Wesley answer the FT notification part. MPI-4 is a major standard release of MPI. That’ll take a few years. We are currently working on the MPI-3.1 release. (hopefully you know the difference between MPI and MPICH, otherwise it’ll take many emails to explain that part :-) ). — Pavan On Oct 9, 2014, at 11:09 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > Hi Pavan, > > Just wondering with the current release whether we have any way to notify the server that client is terminated unexpectedly! > Another point: When do we expect to have MPI-4 release out? > > Thanks, > Hirak > > > Please don’t rely on this feature. We are preparing for MPI-4 Fault Tolerance and are in the process of reworking a bunch of this stuff. This might or might not exist in the future if you are planning to use this for production code. > > — Pavan > > On Oct 9, 2014, at 10:57 AM, Roy, Hirak <Hirak_Roy at mentor.com> wrote: > > > > > Hi Sangmin, > > > > The readme of mpich says the following : > > > > FAILURE NOTIFICATION: THIS IS AN UNSUPPORTED FEATURE AND WILL > > ALMOST CERTAINLY CHANGE IN THE FUTURE! > > > > In the current release, hydra notifies the MPICH library of failed > > processes by sending a SIGUSR1 signal. The application can catch > > this signal to be notified of failed processes. If the application > > replaces the library's signal handler with its own, the application > > must be sure to call the library's handler from it's own > > handler. Note that you cannot call any MPI function from inside a > > signal handler. > > > > If this is true, should not I expect SIGUSR1? > > > > > > Thanks, > > Hirak > > First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. > > > > — Sangmin > > > > > > On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy at mentor.com<mailto:Hirak_Roy at mentor.com>> wrote: > > > > Hi , > > > > I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) > > 1> mpiexec –disable-auto-cleanup –n 1 ./server > > 2> mpiexec –disable-auto-cleanup –n 1 ./client > > > > The server opens a port and does MPI_Comm_accept. > > The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. > > I don’t do MPI_Comm_merge. > > > > I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). > > > > >> signal (SIGUSR1, mysignalhandler); > > > > Now suppose, the ‘client’ process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process ‘server’. > > However, I don’t get any signal in ‘server’ process. > > Am I doing something wrong? > > I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. > > > > Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? > > > > > > Thanks, > > Hirak > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Roy, Hirak 09 Oct '14

09 Oct '14

Hi Pavan, Just wondering with the current release whether we have any way to notify the server that client is terminated unexpectedly! Another point: When do we expect to have MPI-4 release out? Thanks, Hirak Please don't rely on this feature. We are preparing for MPI-4 Fault Tolerance and are in the process of reworking a bunch of this stuff. This might or might not exist in the future if you are planning to use this for production code. - Pavan On Oct 9, 2014, at 10:57 AM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > > Hi Sangmin, > > The readme of mpich says the following : > > FAILURE NOTIFICATION: THIS IS AN UNSUPPORTED FEATURE AND WILL > ALMOST CERTAINLY CHANGE IN THE FUTURE! > > In the current release, hydra notifies the MPICH library of failed > processes by sending a SIGUSR1 signal. The application can catch > this signal to be notified of failed processes. If the application > replaces the library's signal handler with its own, the application > must be sure to call the library's handler from it's own > handler. Note that you cannot call any MPI function from inside a > signal handler. > > If this is true, should not I expect SIGUSR1? > > > Thanks, > Hirak > First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. > > - Sangmin > > > On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy at mentor.com<mailto:Hirak_Roy at mentor.com>> wrote: > > Hi , > > I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) > 1> mpiexec -disable-auto-cleanup -n 1 ./server > 2> mpiexec -disable-auto-cleanup -n 1 ./client > > The server opens a port and does MPI_Comm_accept. > The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. > I don't do MPI_Comm_merge. > > I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). > > >> signal (SIGUSR1, mysignalhandler); > > Now suppose, the 'client' process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process 'server'. > However, I don't get any signal in 'server' process. > Am I doing something wrong? > I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. > > Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? > > > Thanks, > Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Balaji, Pavan 09 Oct '14

09 Oct '14

Please don’t rely on this feature. We are preparing for MPI-4 Fault Tolerance and are in the process of reworking a bunch of this stuff. This might or might not exist in the future if you are planning to use this for production code. — Pavan On Oct 9, 2014, at 10:57 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > > Hi Sangmin, > > The readme of mpich says the following : > > FAILURE NOTIFICATION: THIS IS AN UNSUPPORTED FEATURE AND WILL > ALMOST CERTAINLY CHANGE IN THE FUTURE! > > In the current release, hydra notifies the MPICH library of failed > processes by sending a SIGUSR1 signal. The application can catch > this signal to be notified of failed processes. If the application > replaces the library's signal handler with its own, the application > must be sure to call the library's handler from it's own > handler. Note that you cannot call any MPI function from inside a > signal handler. > > If this is true, should not I expect SIGUSR1? > > > Thanks, > Hirak > First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. > > — Sangmin > > > On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy at mentor.com<mailto:Hirak_Roy at mentor.com>> wrote: > > Hi , > > I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) > 1> mpiexec –disable-auto-cleanup –n 1 ./server > 2> mpiexec –disable-auto-cleanup –n 1 ./client > > The server opens a port and does MPI_Comm_accept. > The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. > I don’t do MPI_Comm_merge. > > I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). > > >> signal (SIGUSR1, mysignalhandler); > > Now suppose, the ‘client’ process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process ‘server’. > However, I don’t get any signal in ‘server’ process. > Am I doing something wrong? > I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. > > Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? > > > Thanks, > Hirak > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Roy, Hirak 09 Oct '14

09 Oct '14

Hi Sangmin, The readme of mpich says the following : FAILURE NOTIFICATION: THIS IS AN UNSUPPORTED FEATURE AND WILL ALMOST CERTAINLY CHANGE IN THE FUTURE! In the current release, hydra notifies the MPICH library of failed processes by sending a SIGUSR1 signal. The application can catch this signal to be notified of failed processes. If the application replaces the library's signal handler with its own, the application must be sure to call the library's handler from it's own handler. Note that you cannot call any MPI function from inside a signal handler. If this is true, should not I expect SIGUSR1? Thanks, Hirak ________________________________ First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. - Sangmin On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss><mailto:Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>>> wrote: Hi , I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) 1> mpiexec -disable-auto-cleanup -n 1 ./server 2> mpiexec -disable-auto-cleanup -n 1 ./client The server opens a port and does MPI_Comm_accept. The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. I don't do MPI_Comm_merge. I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). >> signal (SIGUSR1, mysignalhandler); Now suppose, the 'client' process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process 'server'. However, I don't get any signal in 'server' process. Am I doing something wrong? I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPI process killed and SIGUSR1
by Seo, Sangmin 09 Oct '14

09 Oct '14

First of all, MPI functions are not signal safe. So, if you try to use signals within your MPI program, things might break. — Sangmin On Oct 9, 2014, at 7:37 AM, Roy, Hirak <Hirak_Roy(a)mentor.com<mailto:[email protected]>> wrote: Hi , I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) 1> mpiexec –disable-auto-cleanup –n 1 ./server 2> mpiexec –disable-auto-cleanup –n 1 ./client The server opens a port and does MPI_Comm_accept. The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. I don’t do MPI_Comm_merge. I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). >> signal (SIGUSR1, mysignalhandler); Now suppose, the ‘client’ process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process ‘server’. However, I don’t get any signal in ‘server’ process. Am I doing something wrong? I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Seo, Sangmin 09 Oct '14

09 Oct '14

Hirak, I would correct my previous response. We discussed this issue inside the MPICH team, and we concluded this is a gray area of the standard (it’s not clear when the functions are used with threads). Thus, we are not sure if your use case is right or wrong because you were using different communicators. BTW, you may solve this issue by spawning a new process that connects back. Thanks, Sangmin On Oct 9, 2014, at 7:27 AM, Roy, Hirak <Hirak_Roy(a)mentor.com<mailto:[email protected]>> wrote: Thanks Sangmin. I would change the strategy to terminate the accept. Regards, Hirak Hi Hirak, I checked your code. I think your code is incorrect because you are trying to establish communication within a single process. You are using two threads and two duplicated communicators, but they are associated with a single process. MPI_Comm_accept() and MPI_Comm_connect() are used to establish communication between “two sets of MPI processes” (i.e., at least two different processes are needed). Please refer to 10.4 Establishing Communication (p. 387) in MPI Standard 3.0. — Sangmin From: Roy, Hirak Sent: Thursday, October 09, 2014 2:17 AM To: discuss(a)mpich.org<mailto:[email protected]> Subject: Re: How to terminate MPI_Comm_accept Hi Pavan, Here is the code attached. Let me know if you think the code is incorrect. I used MPI-3.0.4 with sock device. Compile command : export MPI_ROOT = /home/hroy/local/mpich-3.0.4/linux_x86_64 #export MPI_ROOT = /home/hroy/local/mpich-3.0.4.nemesis/linux_x86_64 export MPI_BIN = ${MPI_ROOT}/bin export MPI_LIB = ${MPI_ROOT}/lib export INCLUDE_DIR = ${MPI_ROOT}/include compile: g++ -g main.cpp -I ${INCLUDE_DIR} ${MPI_LIB}/libmpich.a ${MPI_LIB}/libmpl.a -lpthread Thanks, Hirak Can you send us a simple program that reproduces the issue? — Pavan On Oct 8, 2014, at 2:54 PM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > Hi Pavan, > > I even tried duping the communicator (MPI_COMM_SELF à DUP1, DUP2) before I do any kind of MPI calls. > Still it does not work. > > Thanks, > Hirak > _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPI process killed and SIGUSR1
by Roy, Hirak 09 Oct '14

09 Oct '14

Hi , I have two MPI processes (server and client) launched independently by two different mpiexec command. (mpich-3.0.4, sock-device) 1> mpiexec -disable-auto-cleanup -n 1 ./server 2> mpiexec -disable-auto-cleanup -n 1 ./client The server opens a port and does MPI_Comm_accept. The client gets the port information and does MPI_Comm_connect and hence we get a new intercommunicator. I don't do MPI_Comm_merge. I have installed my own signal handler for SIGUSR1 before even I call MPI_Init ( I guess, this will automatically chain the signal handler). >> signal (SIGUSR1, mysignalhandler); Now suppose, the 'client' process gets killed ( I forcefully kill the process by signal 9), I thought I would get SIGUSR1 in the process 'server'. However, I don't get any signal in 'server' process. Am I doing something wrong? I have noticed that if I start 4 client processes with single mpiexec command, and one client gets killed, rest of the 3 clients receive SIGUSR1. Does this mean, SIGUSR1 is not forwarded across processes connected using inter-communicator? Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Roy, Hirak 09 Oct '14

09 Oct '14

Thanks Sangmin. I would change the strategy to terminate the accept. Regards, Hirak Hi Hirak, I checked your code. I think your code is incorrect because you are trying to establish communication within a single process. You are using two threads and two duplicated communicators, but they are associated with a single process. MPI_Comm_accept() and MPI_Comm_connect() are used to establish communication between "two sets of MPI processes" (i.e., at least two different processes are needed). Please refer to 10.4 Establishing Communication (p. 387) in MPI Standard 3.0. - Sangmin From: Roy, Hirak Sent: Thursday, October 09, 2014 2:17 AM To: discuss(a)mpich.org Subject: Re: How to terminate MPI_Comm_accept Hi Pavan, Here is the code attached. Let me know if you think the code is incorrect. I used MPI-3.0.4 with sock device. Compile command : export MPI_ROOT = /home/hroy/local/mpich-3.0.4/linux_x86_64 #export MPI_ROOT = /home/hroy/local/mpich-3.0.4.nemesis/linux_x86_64 export MPI_BIN = ${MPI_ROOT}/bin export MPI_LIB = ${MPI_ROOT}/lib export INCLUDE_DIR = ${MPI_ROOT}/include compile: g++ -g main.cpp -I ${INCLUDE_DIR} ${MPI_LIB}/libmpich.a ${MPI_LIB}/libmpl.a -lpthread Thanks, Hirak Can you send us a simple program that reproduces the issue? - Pavan On Oct 8, 2014, at 2:54 PM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > Hi Pavan, > > I even tried duping the communicator (MPI_COMM_SELF à DUP1, DUP2) before I do any kind of MPI calls. > Still it does not work. > > Thanks, > Hirak > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Seo, Sangmin 09 Oct '14

09 Oct '14

Hi Hirak, I checked your code. I think your code is incorrect because you are trying to establish communication within a single process. You are using two threads and two duplicated communicators, but they are associated with a single process. MPI_Comm_accept() and MPI_Comm_connect() are used to establish communication between “two sets of MPI processes” (i.e., at least two different processes are needed). Please refer to 10.4 Establishing Communication (p. 387) in MPI Standard 3.0. — Sangmin On Oct 8, 2014, at 3:46 PM, Roy, Hirak <Hirak_Roy(a)mentor.com<mailto:[email protected]>> wrote: Hi Pavan, Here is the code attached. Let me know if you think the code is incorrect. I used MPI-3.0.4 with sock device. Compile command : export MPI_ROOT = /home/hroy/local/mpich-3.0.4/linux_x86_64 #export MPI_ROOT = /home/hroy/local/mpich-3.0.4.nemesis/linux_x86_64 export MPI_BIN = ${MPI_ROOT}/bin export MPI_LIB = ${MPI_ROOT}/lib export INCLUDE_DIR = ${MPI_ROOT}/include compile: g++ -g main.cpp -I ${INCLUDE_DIR} ${MPI_LIB}/libmpich.a ${MPI_LIB}/libmpl.a -lpthread Thanks, Hirak Can you send us a simple program that reproduces the issue? — Pavan On Oct 8, 2014, at 2:54 PM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > Hi Pavan, > > I even tried duping the communicator (MPI_COMM_SELF à DUP1, DUP2) before I do any kind of MPI calls. > Still it does not work. > > Thanks, > Hirak > <main.cpp>_______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Roy, Hirak 08 Oct '14

08 Oct '14

Hi Pavan, Here is the code attached. Let me know if you think the code is incorrect. I used MPI-3.0.4 with sock device. Compile command : export MPI_ROOT = /home/hroy/local/mpich-3.0.4/linux_x86_64 #export MPI_ROOT = /home/hroy/local/mpich-3.0.4.nemesis/linux_x86_64 export MPI_BIN = ${MPI_ROOT}/bin export MPI_LIB = ${MPI_ROOT}/lib export INCLUDE_DIR = ${MPI_ROOT}/include compile: g++ -g main.cpp -I ${INCLUDE_DIR} ${MPI_LIB}/libmpich.a ${MPI_LIB}/libmpl.a -lpthread Thanks, Hirak Can you send us a simple program that reproduces the issue? - Pavan On Oct 8, 2014, at 2:54 PM, Roy, Hirak <Hirak_Roy at mentor.com<https://lists.mpich.org/mailman/listinfo/discuss>> wrote: > Hi Pavan, > > I even tried duping the communicator (MPI_COMM_SELF à DUP1, DUP2) before I do any kind of MPI calls. > Still it does not work. > > Thanks, > Hirak > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0