January 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 28 Jan '22

28 Jan '22

"ip_address" won't be recognized, only "host", "hosts", or "hostfile". Could you run an example using "mpiexec -v" and capture/share the output? That should help tell us if the hostname information is being fed correctly to the process manager by the spawn command. Ken On 1/28/22, 11:35 AM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 28 Jan '22

28 Jan '22

Ken, I'm using sbatch, which calls a bash script that calls mpiexec (4.0rc3). Which host name convention is correct, the short or the long host name? Would the "ip_address" info key work? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, January 28, 2022 10:49 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0

Re: [mpich-discuss] MPI_Comm_spawn crosses node boundaries
by Raffenetti, Ken 28 Jan '22

28 Jan '22

Are you using mpiexec or srun when initially launching your job? Hydra (mpiexec) should support the "host" info key, but I'm not sure if srun will. Ken On 1/28/22, 10:41 AM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I’ve tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = “n001” or even the full name “n001.cluster.pssclabs.com” but that doesn’t prevent the problem. Any suggestions? Thanks, Kurt

1 0

MPI_Comm_spawn crosses node boundaries
by Mccall, Kurt E. (MSFC-EV41) 28 Jan '22

28 Jan '22

Hi, Running MPICH under Slurm, MPI_Comm_spawn unexpectedly creates new processes on any and all of the nodes that Slurm allocates to the job. I would like it to only create new processes locally on the node that called MPI_Comm_spawn. I've tried passing MPI_Comm_spawn an info struct created like this: MPI_Info info; MPI_Info_create(&info); MPI_Info_set(info, "host", host_name); MPI_Info_set(info, "bind_to", "core"); where hostname = "n001" or even the full name "n001.cluster.pssclabs.com" but that doesn't prevent the problem. Any suggestions? Thanks, Kurt

1 0

Re: [mpich-discuss] How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept
by Joachim Protze 28 Jan '22

28 Jan '22

Hi, since you have a separate thread for the accept, you could just set a flag at the main thread, open a connection from the main thread to match the accept. The accept thread sees the flag, immediately closes the connection and leaves the while loop. - Joachim Am 28.01.22 um 05:54 schrieb Rupsa Chakraborty via discuss: > > Hello, > > I am asking this question in this forum as I had already asked this > question in stack overflow and did not receive any answer/comment: > https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… > <https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-…> > > My concern is as follows: > I have an MPI server program that calls MPI_Comm_accept, in an infinite > loop, on a separate thread. The main thread spawns this thread and in > parallel does some other work. At some point of time the main thread > decides to exit. It closes the port using MPI_Close_port(portName), > however the MPI_Comm_accept() is still waiting for connection requests. > > > My question is: > - How does the main thread exit gracefully? Is there an equivalent of > socket close in MPI? I am using mpich which I can use from main thread? > > - Is there anyway I can set timeout to the MPI_Comm_accept call? > > Here is a dummy code that looks similar to my server code: > > |// MPI Server Program void accept(std::string portName) { while (true) > { MPI_Comm intercomm; MPI_Comm_accept(portName.c_str(), MPI_INFO_NULL, > 0, MPI_COMM_SELF, &intercomm); // handle connection > MPI_Comm_disconnect(&intercomm); } } int main() { char ch; > MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; > MPI_Open_port(MPI_INFO_NULL, portName); > publishName("nameServerFile.txt", "ocean", std::string(portName)); > std::thread th(&accept, std::string(portName)); // do something > MPI_Close_port(portName); std::cout <<"\nClosed port" << std::endl; > th.join(); // main thread waits infinitely at this join // as child > thread is still waiting on MPI_Comm_accept MPI_Finalize(); }| > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept
by Zhou, Hui 28 Jan '22

28 Jan '22

Hi Rupsa, With MPICH, you can set info hint "timeout" in seconds. Both MPI_Comm_accept and MPI_Comm_connect accept this info key. -- Hui Zhou ________________________________ From: Rupsa Chakraborty via discuss <discuss(a)mpich.org> Sent: Thursday, January 27, 2022 10:54 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Rupsa Chakraborty <c.rupsa(a)gmail.com> Subject: [mpich-discuss] How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept Hello, I am asking this question in this forum as I had already asked this question in stack overflow and did not receive any answer/comment: https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… My concern is as follows: I have an MPI server program that calls MPI_Comm_accept, in an infinite loop, on a separate thread. The main thread spawns this thread and in parallel does some other work. At some point of time the main thread decides to exit. It closes the port using MPI_Close_port(portName), however the MPI_Comm_accept() is still waiting for connection requests. My question is: - How does the main thread exit gracefully? Is there an equivalent of socket close in MPI? I am using mpich which I can use from main thread? - Is there anyway I can set timeout to the MPI_Comm_accept call? Here is a dummy code that looks similar to my server code: // MPI Server Program void accept(std::string portName) { while (true) { MPI_Comm intercomm; MPI_Comm_accept(portName.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm); // handle connection MPI_Comm_disconnect(&intercomm); } } int main() { char ch; MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; MPI_Open_port(MPI_INFO_NULL, portName); publishName("nameServerFile.txt", "ocean", std::string(portName)); std::thread th(&accept, std::string(portName)); // do something MPI_Close_port(portName); std::cout <<"\nClosed port" << std::endl; th.join(); // main thread waits infinitely at this join // as child thread is still waiting on MPI_Comm_accept MPI_Finalize(); }

1 0

How to gracefully exit an MPI server process when it is waiting on MPI_Comm_accept
by Rupsa Chakraborty 27 Jan '22

27 Jan '22

Hello, I am asking this question in this forum as I had already asked this question in stack overflow and did not receive any answer/comment: https://stackoverflow.com/questions/70745740/how-to-gracefully-exit-an-mpi-… My concern is as follows: I have an MPI server program that calls MPI_Comm_accept, in an infinite loop, on a separate thread. The main thread spawns this thread and in parallel does some other work. At some point of time the main thread decides to exit. It closes the port using MPI_Close_port(portName), however the MPI_Comm_accept() is still waiting for connection requests. My question is: - How does the main thread exit gracefully? Is there an equivalent of socket close in MPI? I am using mpich which I can use from main thread? - Is there anyway I can set timeout to the MPI_Comm_accept call? Here is a dummy code that looks similar to my server code: // MPI Server Program void accept(std::string portName) { while (true) { MPI_Comm intercomm; MPI_Comm_accept(portName.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm); // handle connection MPI_Comm_disconnect(&intercomm); } } int main() { char ch; MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; MPI_Open_port(MPI_INFO_NULL, portName); publishName("nameServerFile.txt", "ocean", std::string(portName)); std::thread th(&accept, std::string(portName)); // do something MPI_Close_port(portName); std::cout <<"\nClosed port" << std::endl; th.join(); // main thread waits infinitely at this join // as child thread is still waiting on MPI_Comm_accept MPI_Finalize(); }

1 0

Re: [mpich-discuss] MPICH Usage Issue: Client process not exiting till server reaches MPI_Finalize() in MPI client server singleton INIT connection
by Zhou, Hui 27 Jan '22

27 Jan '22

Hi Rupsa, Could you try the latest release, mpich-4.0, for your application? This waiting for server finalization behavior may have been fixed. -- Hui Zhou From: Rupsa Chakraborty via discuss <discuss(a)mpich.org> Date: Thursday, January 27, 2022 at 1:42 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Rupsa Chakraborty <c.rupsa(a)gmail.com> Subject: [mpich-discuss] MPICH Usage Issue: Client process not exiting till server reaches MPI_Finalize() in MPI client server singleton INIT connection Hello, I am asking this question in this forum as I had already asked this question in stack overflow and did not receive any answer/comment. I am trying to implement a feature that performs a large computation in parallel over multiple machines. I am trying out zeroMQ as well as MPI for this, and will accept the one which gives better performance and also is better in ease of usage and support. Your help in this is much appreciated. I am implementing a new feature, using MPI client server parallel computation with 'Singleton INIT' mechanism, in an existing large application on Linux. I am using mpich-3.4.1 for this. The main process becomes an mpi server on certain user inputs. Client processes connect to this server and share a large parallel computation between themselves and return the result back to the server. The clients and server processes are independently started (Singleton INIT). All client processes should be able to normally exit after disconnecting from the server, even though the server is still running (doing some other work, but not accepting any more client connections). My questions is: Is it possible to keep the server process running and executing other tasks, while all of its client connections have closed and client processes exited? In my client-server MPI code, I see that for the last client connection to a server (corresponding to the last MPI_Comm_accept() call) the client process always gets stuck at its MPI_Finalize() till the server reaches its own MPI_Finalize(). Thus the last client process does not exit till the server reaches its MPI_Finalize() stage. However, for the previous client connections the client processes are able to normally exit. 'lsof' command shows, at the server side, call to MPI_Comm_accept() releases the file descriptor from the previous connection, which is not getting released for the last connection due to the absence of subsequent MPI_Comm_accept() call, but gets released at the MPI_Finalize() call. What is it that I am missing or doing incorrectly? How to have the last client also exit and not wait at MPI_Finalize() stage in the following code, even though the server is still running? Is there any way I can set a timeout to the MPI_Comm_accept? or interrupt it from a separate thread? My client and server code are as follows: //server.cxx int main() { char ch; MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; MPI_Open_port(MPI_INFO_NULL, portName); publishServerPortNameToFile(portName, /*args*/); MPI_Comm intercomm1; // First connection. Client corresponding to this connection connects and exits successfully. MPI_Comm_accept(portName, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm1); std::cout << "\nConnection1 Accepted."; MPI_Comm intercomm2; // Second connection. Client corresponding to this connection connects, but waits at its // own MPI_Finalize() stage, and exits only after server reaches MPI_finalize(). MPI_Comm_accept(portName, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm2); std::cout << "\nConnection 2 Accepted."; std::cout << "\nBefore disconnect."; MPI_Close_port(portName); MPI_Comm_disconnect(&intercomm1); MPI_Comm_disconnect(&intercomm2); std::cout << "\nBefore Finalize()."; MPI_Finalize(); std::cout << "\nAfter Finalize."; } //client.cxx int main() { MPI_Init(NULL, NULL); std::string portName = getServerPortNameFromFile(/*args*/); MPI_Comm intercomm; MPI_Comm_connect(portName.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm); std::cout << "\nConnected"; MPI_Comm_disconnect(&intercomm); std::cout << "\nBefore Finalize()"; MPI_Finalize(); std::cout << "\nAfter Finalize().\nClient Exited"; } Output after first client connection is made, and the client exits: shell1: $ ./server Connection 1 Accepted < server waiting to accept next connection > shell2: $ ./client // first client requesting connection Connected Before Finalize() After Finalize() Client Exited $ Output after the second client requests connection (at this point the first client had successfully connected and exited): shell1: $ ./server Connection 1 Accepted Connection 2 Accepted. Before disconnect. Press any key // key pressed Before Finalize(). Press any key // key not yet pressed shell2: $ ./client // this is the second client connection to the same server Connected Before Finalize(). < this client process waits here till the server reaches its Finalize() > Regards, Rupsa

1 0

MPICH Usage Issue: Client process not exiting till server reaches MPI_Finalize() in MPI client server singleton INIT connection
by Rupsa Chakraborty 27 Jan '22

27 Jan '22

Hello, I am asking this question in this forum as I had already asked this question in stack overflow and did not receive any answer/comment. I am trying to implement a feature that performs a large computation in parallel over multiple machines. I am trying out zeroMQ as well as MPI for this, and will accept the one which gives better performance and also is better in ease of usage and support. Your help in this is much appreciated. I am implementing a new feature, using MPI client server parallel computation with 'Singleton INIT' mechanism, in an existing large application on Linux. I am using mpich-3.4.1 for this. The main process becomes an mpi server on certain user inputs. Client processes connect to this server and share a large parallel computation between themselves and return the result back to the server. The clients and server processes are independently started (Singleton INIT). All client processes should be able to normally exit after disconnecting from the server, even though the server is still running (doing some other work, but not accepting any more client connections). My questions is: Is it possible to keep the server process running and executing other tasks, while all of its client connections have closed and client processes exited? In my client-server MPI code, I see that for the last client connection to a server (corresponding to the last MPI_Comm_accept() call) the client process always gets stuck at its MPI_Finalize() till the server reaches its own MPI_Finalize(). Thus the last client process does not exit till the server reaches its MPI_Finalize() stage. However, for the previous client connections the client processes are able to normally exit. 'lsof' command shows, at the server side, call to MPI_Comm_accept() releases the file descriptor from the previous connection, which is not getting released for the last connection due to the absence of subsequent MPI_Comm_accept() call, but gets released at the MPI_Finalize() call. What is it that I am missing or doing incorrectly? How to have the last client also exit and not wait at MPI_Finalize() stage in the following code, even though the server is still running? Is there any way I can set a timeout to the MPI_Comm_accept? or interrupt it from a separate thread? My client and server code are as follows: //server.cxx int main() { char ch; MPI_Init(NULL, NULL); char portName[MPI_MAX_PORT_NAME]; MPI_Open_port(MPI_INFO_NULL, portName); publishServerPortNameToFile(portName, /*args*/); MPI_Comm intercomm1; // First connection. Client corresponding to this connection connects and exits successfully. MPI_Comm_accept(portName, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm1); std::cout << "\nConnection1 Accepted."; MPI_Comm intercomm2; // Second connection. Client corresponding to this connection connects, but waits at its // own MPI_Finalize() stage, and exits only after server reaches MPI_finalize(). MPI_Comm_accept(portName, MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm2); std::cout << "\nConnection 2 Accepted."; std::cout << "\nBefore disconnect."; MPI_Close_port(portName); MPI_Comm_disconnect(&intercomm1); MPI_Comm_disconnect(&intercomm2); std::cout << "\nBefore Finalize()."; MPI_Finalize(); std::cout << "\nAfter Finalize."; } //client.cxx int main() { MPI_Init(NULL, NULL); std::string portName = getServerPortNameFromFile(/*args*/); MPI_Comm intercomm; MPI_Comm_connect(portName.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &intercomm); std::cout << "\nConnected"; MPI_Comm_disconnect(&intercomm); std::cout << "\nBefore Finalize()"; MPI_Finalize(); std::cout << "\nAfter Finalize().\nClient Exited"; } Output after first client connection is made, and the client exits: shell1: $ ./server Connection 1 Accepted < server waiting to accept next connection > shell2: $ ./client // first client requesting connection Connected Before Finalize() After Finalize() Client Exited $ Output after the second client requests connection (at this point the first client had successfully connected and exited): shell1: $ ./server Connection 1 Accepted Connection 2 Accepted. Before disconnect. Press any key // key pressed Before Finalize(). Press any key // key not yet pressed shell2: $ ./client // this is the second client connection to the same server Connected Before Finalize(). < this client process waits here till the server reaches its Finalize() > Regards, Rupsa

1 0

Re: [mpich-discuss] ROMIO and asynchronous MPI-IO
by Latham, Robert J. 25 Jan '22

25 Jan '22

On Mon, 2022-01-24 at 14:16 +0100, Ahmad Tarraf via discuss wrote: > It is hard to find any literature or documentation about the > asynchronous implementation ROMIO (or ADIO) use. I have few questions > and It would be great if you can answer them: > 1. How does ROMIO implement asynchronous I/O? I mean do you use > POSIX-AIO, native Linux kernel AIO (io_submit,..), io_uring, or > anything else? If I debug my code at our Clusters with gdb, I can see > that POSIX-AIO is used. But is this on all systems so? ROMIO will support async I/O if two conditions are met: 1: POSIX async I/O routines (aio_write, aio_suspend, etc) 2: "extended generalized requests" -- part of MPICH, but not OpenMPI (Some of the specialized file systems also have async routines: same approach applies though -- rely on the underlying file system or storage library to make progress see below) > 2. When and how does asynchronous I/O progress occur? Do you span > multiple I/O threads for a rank? Is this related to variable > MPICH_ASYNC_PROGRESS? Again with gdb, I can see that several > asynchronous write requests by the same MPI rank are handled > successively by a single thread. The posix aio library will make background progress (typically libc implements aio by spawning a thread -- I think kernel native aio is only used by databases?). Extended generalized requests allow ROMIO to query the state of pending AIO operations. when we designed those generalized request extensions the intent was to allow for "status checking" for those extensions. MPICH_ASYNC_PROGRESS can help speed things along, I think, in that it will kick the MPICH "progress engine" more frequently. Without that set, only TEST and WAIT routines invoke the callbacks that check the state of the async i/o routine. I have not benchmarked this and would love to hear your experiences. > 3. Is there any related recent publication is can use to cite? I > didn't find anything related to the implementation of asynchronous > MPI-IO, except this: > https://web.cels.anl.gov/~thakur/papers/grequest-redesign.pdf. But is > it not outdated? There has not been much recent work on extended generalized requests. You might find _Implementation and Evaluation of MPI Nonblocking Collective I/O_ interesting? https://dl.acm.org/doi/10.1109/CCGrid.2015.81 > Best regards, > Ahmad > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0