January 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept
by Rupsa Chakraborty 30 Jan '22

30 Jan '22

Hi Hui, Thank you for the clarification. Regards, Rupsa On Sun, Jan 30, 2022 at 11:08 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Rupsa, > > The error message could be better. By "Invalid port", it actually meant > the connection was timed out. So your 2-second time out actually worked. > You should use MPI_Comm_set_errhandler to set error handler to > MPI_ERRORS_RETURN, so that your application can check error and respond > accordingly. The default error handler is to abort. > > As I checked, the current code actually will leak resources when timeout > happens. I just created this issue -- > https://github.com/pmodels/mpich/issues/5815 -- to track it. > > -- > Hui Zhou > <https://github.com/pmodels/mpich/issues/5815> > ch4/ofi: better error handling in MPI_Comm_{accept,connect} · Issue #5815 > · pmodels/mpich <https://github.com/pmodels/mpich/issues/5815> > In MPIDI_OFI_dynamic_send and MPIDI_OFI_dynamic_recv, when error happens, > we should try cancel the send or recv. Otherwise, we may leave or > accumulated unclosed handle in libfabric. Separately, whe... > github.com > > ------------------------------ > *From:* Rupsa Chakraborty <c.rupsa(a)gmail.com> > *Sent:* Saturday, January 29, 2022 1:29 AM > *To:* Zhou, Hui <zhouh(a)anl.gov>; protze(a)itc.rwth-aachen.de < > protze(a)itc.rwth-aachen.de> > *Cc:* discuss(a)mpich.org <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] How to gracefully exit an MPI server > process when it is waiting on MPI_Comm_accept > > Hi Hui, Joachim, > > Thanks for the information. > I am not able to find in the internet a clear example as to how to > populate MPI_Info and use it with MPI_Comm_accept. I tried to do the > following, but this is giving errors as shown below. > Could you provide a small example to set up a timeout with > MPI_Comm_accept? Do I need to populate port info too with MPI_Info? > > MPI_Init(NULL, NULL); > char portName[MPI_MAX_PORT_NAME]; > MPI_Open_port(MPI_INFO_NULL, portName); > > MPI_Info info; > MPI_Info_create( &info ); > MPI_Info_set(info, "timeout", "2"); > > MPI_Comm intercomm; > MPI_Comm_accept(portName, info, 0, MPI_COMM_SELF, > &intercomm); > > Error: > $ ./server > Waiting to accept.... > Abort(671693606) on node 0 (rank 0 in comm 0): Fatal error in > internal_Comm_accept: Invalid port, error stack: > internal_Comm_accept(102)....: > MPI_Comm_accept(port_name=tag#0$connentry#020098ADAC1785B70000000000000000$, > info=0x9c000002, 0, MPI_COMM_SELF, newcomm=0x7f79b17dbcb8) failed > MPID_Comm_accept(442)........: > dynamic_intercomm_create(400): > peer_intercomm_create(334)...: > (unknown)(): Invalid port > > Regards, > Rupsa > > On Fri, Jan 28, 2022 at 8:47 PM Joachim Protze <protze(a)itc.rwth-aachen.de> > wrote: > > Hi, > > since you have a separate thread for the accept, you could just set a > flag at the main thread, open a connection from the main thread to match > the accept. The accept thread sees the flag, immediately closes the > connection and leaves the while loop. > > - Joachim > > Am 28.01.22 um 05:54 schrieb Rupsa Chakraborty via discuss: > > > > Hello, > > > > I am asking this question in this forum as I had already asked this > > question in stack overflow and did not receive any answer/comment: > > > https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… > > < > https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… > > > > > > My concern is as follows: > > I have an MPI server program that calls MPI_Comm_accept, in an infinite > > loop, on a separate thread. The main thread spawns this thread and in > > parallel does some other work. At some point of time the main thread > > decides to exit. It closes the port using MPI_Close_port(portName), > > however the MPI_Comm_accept() is still waiting for connection requests. > > > > > > My question is: > > - How does the main thread exit gracefully? Is there an equivalent of > > socket close in MPI? I am using mpich which I can use from main thread? > > > > - Is there anyway I can set timeout to the MPI_Comm_accept call? > > > > Here is a dummy code that looks similar to my server code: > > > > |// MPI Server Program void accept(std::string portName) { while (true) > > { MPI_Comm intercomm; MPI_Comm_accept(portName.c_str(), MPI_INFO_NULL, > > 0, MPI_COMM_SELF, &intercomm); // handle connection > > MPI_Comm_disconnect(&intercomm); } } int main() { char ch; > > MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; > > MPI_Open_port(MPI_INFO_NULL, portName); > > publishName("nameServerFile.txt", "ocean", std::string(portName)); > > std::thread th(&accept, std::string(portName)); // do something > > MPI_Close_port(portName); std::cout <<"\nClosed port" << std::endl; > > th.join(); // main thread waits infinitely at this join // as child > > thread is still waiting on MPI_Comm_accept MPI_Finalize(); }| > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Dr. rer. nat. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de > >

1 0

Re: [mpich-discuss] How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept
by Zhou, Hui 29 Jan '22

29 Jan '22

Hi Rupsa, The error message could be better. By "Invalid port", it actually meant the connection was timed out. So your 2-second time out actually worked. You should use MPI_Comm_set_errhandler to set error handler to MPI_ERRORS_RETURN, so that your application can check error and respond accordingly. The default error handler is to abort. As I checked, the current code actually will leak resources when timeout happens. I just created this issue -- https://github.com/pmodels/mpich/issues/5815 -- to track it. -- Hui Zhou [https://opengraph.githubassets.com/f9215523be8f812ac3716771553a3d691ec7f493…]<https://github.com/pmodels/mpich/issues/5815> ch4/ofi: better error handling in MPI_Comm_{accept,connect} · Issue #5815 · pmodels/mpich<https://github.com/pmodels/mpich/issues/5815> In MPIDI_OFI_dynamic_send and MPIDI_OFI_dynamic_recv, when error happens, we should try cancel the send or recv. Otherwise, we may leave or accumulated unclosed handle in libfabric. Separately, whe... github.com ________________________________ From: Rupsa Chakraborty <c.rupsa(a)gmail.com> Sent: Saturday, January 29, 2022 1:29 AM To: Zhou, Hui <zhouh(a)anl.gov>; protze(a)itc.rwth-aachen.de <protze(a)itc.rwth-aachen.de> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept Hi Hui, Joachim, Thanks for the information. I am not able to find in the internet a clear example as to how to populate MPI_Info and use it with MPI_Comm_accept. I tried to do the following, but this is giving errors as shown below. Could you provide a small example to set up a timeout with MPI_Comm_accept? Do I need to populate port info too with MPI_Info? MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; MPI_Open_port(MPI_INFO_NULL, portName); MPI_Info info; MPI_Info_create( &info ); MPI_Info_set(info, "timeout", "2"); MPI_Comm intercomm; MPI_Comm_accept(portName, info, 0, MPI_COMM_SELF, &intercomm); Error: $ ./server Waiting to accept.... Abort(671693606) on node 0 (rank 0 in comm 0): Fatal error in internal_Comm_accept: Invalid port, error stack: internal_Comm_accept(102)....: MPI_Comm_accept(port_name=tag#0$connentry#020098ADAC1785B70000000000000000$, info=0x9c000002, 0, MPI_COMM_SELF, newcomm=0x7f79b17dbcb8) failed MPID_Comm_accept(442)........: dynamic_intercomm_create(400): peer_intercomm_create(334)...: (unknown)(): Invalid port Regards, Rupsa On Fri, Jan 28, 2022 at 8:47 PM Joachim Protze <protze(a)itc.rwth-aachen.de<mailto:[email protected]>> wrote: Hi, since you have a separate thread for the accept, you could just set a flag at the main thread, open a connection from the main thread to match the accept. The accept thread sees the flag, immediately closes the connection and leaves the while loop. - Joachim Am 28.01.22 um 05:54 schrieb Rupsa Chakraborty via discuss: > > Hello, > > I am asking this question in this forum as I had already asked this > question in stack overflow and did not receive any answer/comment: > https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… > <https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-…> > > My concern is as follows: > I have an MPI server program that calls MPI_Comm_accept, in an infinite > loop, on a separate thread. The main thread spawns this thread and in > parallel does some other work. At some point of time the main thread > decides to exit. It closes the port using MPI_Close_port(portName), > however the MPI_Comm_accept() is still waiting for connection requests. > > > My question is: > - How does the main thread exit gracefully? Is there an equivalent of > socket close in MPI? I am using mpich which I can use from main thread? > > - Is there anyway I can set timeout to the MPI_Comm_accept call? > > Here is a dummy code that looks similar to my server code: > > |// MPI Server Program void accept(std::string portName) { while (true) > { MPI_Comm intercomm; MPI_Comm_accept(portName.c_str(), MPI_INFO_NULL, > 0, MPI_COMM_SELF, &intercomm); // handle connection > MPI_Comm_disconnect(&intercomm); } } int main() { char ch; > MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; > MPI_Open_port(MPI_INFO_NULL, portName); > publishName("nameServerFile.txt", "ocean", std::string(portName)); > std::thread th(&accept, std::string(portName)); // do something > MPI_Close_port(portName); std::cout <<"\nClosed port" << std::endl; > th.join(); // main thread waits infinitely at this join // as child > thread is still waiting on MPI_Comm_accept MPI_Finalize(); }| > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org<mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de<mailto:[email protected]> www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de>

1 0

Re: [mpich-discuss] How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept
by Rupsa Chakraborty 29 Jan '22

29 Jan '22

Hi Hui, Joachim, Thanks for the information. I am not able to find in the internet a clear example as to how to populate MPI_Info and use it with MPI_Comm_accept. I tried to do the following, but this is giving errors as shown below. Could you provide a small example to set up a timeout with MPI_Comm_accept? Do I need to populate port info too with MPI_Info? MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; MPI_Open_port(MPI_INFO_NULL, portName); MPI_Info info; MPI_Info_create( &info ); MPI_Info_set(info, "timeout", "2"); MPI_Comm intercomm; MPI_Comm_accept(portName, info, 0, MPI_COMM_SELF, &intercomm); Error: $ ./server Waiting to accept.... Abort(671693606) on node 0 (rank 0 in comm 0): Fatal error in internal_Comm_accept: Invalid port, error stack: internal_Comm_accept(102)....: MPI_Comm_accept(port_name=tag#0$connentry#020098ADAC1785B70000000000000000$, info=0x9c000002, 0, MPI_COMM_SELF, newcomm=0x7f79b17dbcb8) failed MPID_Comm_accept(442)........: dynamic_intercomm_create(400): peer_intercomm_create(334)...: (unknown)(): Invalid port Regards, Rupsa On Fri, Jan 28, 2022 at 8:47 PM Joachim Protze <protze(a)itc.rwth-aachen.de> wrote: > Hi, > > since you have a separate thread for the accept, you could just set a > flag at the main thread, open a connection from the main thread to match > the accept. The accept thread sees the flag, immediately closes the > connection and leaves the while loop. > > - Joachim > > Am 28.01.22 um 05:54 schrieb Rupsa Chakraborty via discuss: > > > > Hello, > > > > I am asking this question in this forum as I had already asked this > > question in stack overflow and did not receive any answer/comment: > > > https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… > > < > https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… > > > > > > My concern is as follows: > > I have an MPI server program that calls MPI_Comm_accept, in an infinite > > loop, on a separate thread. The main thread spawns this thread and in > > parallel does some other work. At some point of time the main thread > > decides to exit. It closes the port using MPI_Close_port(portName), > > however the MPI_Comm_accept() is still waiting for connection requests. > > > > > > My question is: > > - How does the main thread exit gracefully? Is there an equivalent of > > socket close in MPI? I am using mpich which I can use from main thread? > > > > - Is there anyway I can set timeout to the MPI_Comm_accept call? > > > > Here is a dummy code that looks similar to my server code: > > > > |// MPI Server Program void accept(std::string portName) { while (true) > > { MPI_Comm intercomm; MPI_Comm_accept(portName.c_str(), MPI_INFO_NULL, > > 0, MPI_COMM_SELF, &intercomm); // handle connection > > MPI_Comm_disconnect(&intercomm); } } int main() { char ch; > > MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; > > MPI_Open_port(MPI_INFO_NULL, portName); > > publishName("nameServerFile.txt", "ocean", std::string(portName)); > > std::thread th(&accept, std::string(portName)); // do something > > MPI_Close_port(portName); std::cout <<"\nClosed port" << std::endl; > > th.join(); // main thread waits infinitely at this join // as child > > thread is still waiting on MPI_Comm_accept MPI_Finalize(); }| > > > > > > > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > > -- > Dr. rer. nat. Joachim Protze > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering > RWTH Aachen University > Seffenter Weg 23 > D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > protze(a)itc.rwth-aachen.de > www.itc.rwth-aachen.de >

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 28 Jan '22

28 Jan '22

On 1/28/22, 2:22 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Dang, I thought that would work :(. Looking at the Slurm documentation https://slurm.schedmd.com/mpi_guide.html#intel_mpiexec_hydra it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? That is the default, so no extra configuration should be needed. One thing I notice in your log output is that the Slurm envvars seems to have changed name from what we have in our source. E.g. SLURM_JOB_NODELIST vs. SLURM_NODELIST. Do your initial processes launch on the right nodes? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Hopefully you don't need to rebuild Slurm to do it. What you could try is configuring the Slurm PMI library when building MPICH. Add "--with-pm=none --with-pmi=slurm --with-slurm=<path/to/install>". Then use srun instead of mpiexec and see how it goes. Ken

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 28 Jan '22

28 Jan '22

Ken, I confirmed that MPI_Comm_spawn fails completely if I build MPICH without the PMI2 option. Looking at the Slurm documentation https://slurm.schedmd.com/mpi_guide.html#intel_mpiexec_hydra it states "All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface." The full quote is below for reference. 1) how do I build MPICH to support hydra's PMI 1.1 interface? 2) Can you offer any guesses on how to build Slurm to do the same? (I realize this isn't a Slurm forum 😊) Thanks, Kurt =========== Full quote =============== MPICH2 with mpiexec Do not add any flags to mpich and build the default (e.g. "./configure -prefix ... ". Do NOT pass the --with-slurm, --with-pmi, --enable-pmiport options). Do not add -lpmi to your application (it will force slurm's pmi 1 interface which doesn't support PMI_Spawn_multiple). Launch the application using salloc to create the job allocation and mpiexec to launch the tasks. A simple example is shown below. salloc -N 2 mpiexec my_application All MPI_comm_spawn work fine now going through hydra's PMI 1.1 interface. -----Original Message----- From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Friday, January 28, 2022 1:43 PM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I have built in the past without the pmi2 option, and MPI_Comm_spawn doesn't work at all. My memory is a little fuzzy, so I'll try that to make sure. In the meantime, can you recommend an option with which to build Slurm that MPICH also supports and has tested? Here is what we have now: $ srun --mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>> Sent: Friday, January 28, 2022 1:35 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I'm just now realizing that we don't test spawn functionality with our pmi2 implementation. Can you try rebuilding without that option and see if it works as expected? Ken On 1/28/22, 1:18 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> wrote: ../mpich-4.0rc3/configure --prefix=/opt/mpich --with-pmi=pmi2/simple --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>> Sent: Friday, January 28, 2022 1:16 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries From what I can tell, the info keyvals are not being sent to the process manager, which would explain what you are seeing. Need to investigate why that is happening next. What is your MPICH ./configure line? That'll help narrow where to look. Ken On 1/28/22, 12:49 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> wrote: Ken, There is a lot of my own output mixed up in the "mpiexec -v" output. I hope you can make sense of this. Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>> Sent: Friday, January 28, 2022 12:36 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries "ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the process manager by the spawn command. Ken On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> wrote: Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://gcc02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.mpi…

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 28 Jan '22

28 Jan '22

I have built in the past without the pmi2 option, and MPI_Comm_spawn doesn't work at all. My memory is a little fuzzy, so I'll try that to make sure. In the meantime, can you recommend an option with which to build Slurm that MPICH also supports and has tested? Here is what we have now: $ srun --mpi=list srun: MPI types are... srun: cray_shasta srun: pmi2 srun: none -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 1:35 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries I'm just now realizing that we don't test spawn functionality with our pmi2 implementation. Can you try rebuilding without that option and see if it works as expected? Ken On 1/28/22, 1:18 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: ../mpich-4.0rc3/configure --prefix=/opt/mpich --with-pmi=pmi2/simple --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 1:16 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries From what I can tell, the info keyvals are not being sent to the process manager, which would explain what you are seeing. Need to investigate why that is happening next. What is your MPICH ./configure line? That'll help narrow where to look. Ken On 1/28/22, 12:49 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, There is a lot of my own output mixed up in the "mpiexec -v" output. I hope you can make sense of this. Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 12:36 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries "ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the process manager by the spawn command. Ken On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 28 Jan '22

28 Jan '22

I'm just now realizing that we don't test spawn functionality with our pmi2 implementation. Can you try rebuilding without that option and see if it works as expected? Ken On 1/28/22, 1:18 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: ../mpich-4.0rc3/configure --prefix=/opt/mpich --with-pmi=pmi2/simple --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 1:16 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries From what I can tell, the info keyvals are not being sent to the process manager, which would explain what you are seeing. Need to investigate why that is happening next. What is your MPICH ./configure line? That'll help narrow where to look. Ken On 1/28/22, 12:49 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, There is a lot of my own output mixed up in the "mpiexec -v" output. I hope you can make sense of this. Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 12:36 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries "ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the process manager by the spawn command. Ken On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 28 Jan '22

28 Jan '22

../mpich-4.0rc3/configure --prefix=/opt/mpich --with-pmi=pmi2/simple --with-device=ch3:nemesis --disable-fortran -enable-debuginfo --enable-g=debug -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 1:16 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries From what I can tell, the info keyvals are not being sent to the process manager, which would explain what you are seeing. Need to investigate why that is happening next. What is your MPICH ./configure line? That'll help narrow where to look. Ken On 1/28/22, 12:49 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, There is a lot of my own output mixed up in the "mpiexec -v" output. I hope you can make sense of this. Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 12:36 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries "ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the process manager by the spawn command. Ken On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 28 Jan '22

28 Jan '22

From what I can tell, the info keyvals are not being sent to the process manager, which would explain what you are seeing. Need to investigate why that is happening next. What is your MPICH ./configure line? That'll help narrow where to look. Ken On 1/28/22, 12:49 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, There is a lot of my own output mixed up in the "mpiexec -v" output. I hope you can make sense of this. Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 12:36 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries "ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the process manager by the spawn command. Ken On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 28 Jan '22

28 Jan '22

Ken, There is a lot of my own output mixed up in the "mpiexec -v" output. I hope you can make sense of this. Thanks, Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 12:36 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov>; discuss(a)mpich.org Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries "ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the process manager by the spawn command. Ken On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0