October 2014 - discuss - mailman3.cels.anl.gov

Announcing the availability of mpich-3.1.3
by Seo, Sangmin 08 Oct '14

08 Oct '14

The MPICH team is pleased to announce the availability of a new stable release, mpich-3.1.3. This is a stable release that adds several enhancements to Portals4 support, PAMI, RMA, and ROMIO. It also contains a large number of bug fixes. All production environments are encouraged to upgrade to this release. More information about the release can be found on the MPICH website: http://www.mpich.org A list of changes in this release is included at the end of this email. Regards, The MPICH Team =============================================================================== Changes in 3.1.3 =============================================================================== # Several enhancements to Portals4 support. # Several enhancements to PAMI (thanks to IBM for the code contribution). # Several enhancements to the CH3 RMA implementation. # Several enhancements to ROMIO. # Fixed deadlock in multi-threaded MPI_Comm_idup. # Several other minor bug fixes, memory leak fixes, and code cleanup. A full list of changes is available at the following link: http://git.mpich.org/mpich.git/shortlog/v3.1.2..v3.1.3 A full list of bugs that have been fixed is available at the following link: https://trac.mpich.org/projects/mpich/query?status=closed&group=resolution&… _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Lu, Huiwei 08 Oct '14

08 Oct '14

Hi Hirak, I can reproduce your error with the attached program using one process: mpicc -g -o mpi_comm_accept mpi_comm_accept.c -pthread mpiexec -n 1 ./mpi_comm_accept I found MPI_Send and MPI_Recv were not using the same communicator to communicate, that’s why MPI_Recv will never receive the message. So either the communicator creation was wrong or the application was wrong. The standard said MPI_Comm_accept and MPI_Comm_connect are used for “establishing contact between two groups of processes that do not share an existing communicator”. But in this case, thread 1 and 2 do share an existing communicator and want to create a new communicator based on the common communicator. I don’t know if it is allowed. If it is allowed, then MPI_Comm_accept and MPI_Comm_connect should be fixed to support multiple thread case; if it is not allowed, we may need to change the application to use another way to terminate MPI_Comm_accept. Thanks, — Huiwei On Oct 8, 2014, at 12:14 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > Hi Pavan, > > Here is my code for thread2 : > > do { > MPI_Comm newComm ; > MPI_Comm_accept (m_serverPort.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &newComm); > Log ("Accepted a connection"); > int buf = 0 ; > MPI_Status status ; > MPI_Recv(&buf, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, newComm, &status); > > if (status.MPI_TAG == MPI_MSG_TAG_NEW_CONN) { > m_clientComs[m_clientCount] = newComm ; > m_clientCount++; > } else if (status.MPI_TAG == MPI_MSG_TAG_SHUTDOWN) { > Log ("Shutdown"); > //MPI_Comm_disconnect (&newComm); > Log ("Disconnect"); > break; > } else { > Log ("Unmatched Receive"); > } > } while(1) ; > > > Here is my code for thread1 to terminate thread2 : > > MPI_Comm newComm ; > MPI_Comm_connect (m_serverPort.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &newComm); > Log ("Connect to Self"); > int val = 0 ; > MPI_Request req ; > MPI_Send(&val, 1, MPI_INT, 0, MPI_MSG_TAG_SHUTDOWN, newComm); > Log ("Successful"); > //MPI_Status stat ; > //MPI_Wait(&req, &stat); > Log ("Complete"); > > //MPI_Comm_disconnect(&newComm); > > > > > The MPI_Send/Recv waits. > I am using sock channel. > For nemesis, I get the following crash : > Assertion failed in file ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h at line 58: vc_ch->is_local > internal ABORT - process 0 > > I tried non-blocking send and receive followed by wait. However, that also does not solve the problem. > > Thanks, > Hirak > > > > ----- > > Hirak, > > Your approach should work fine. I’m not sure what issue you are facing. I assume thread 1 is doing this: > > while (1) { > MPI_Comm_accept(..); > MPI_Recv(.., tag, ..); > if (tag == REGULAR_CONNECTION) > continue; > else if (tag == TERMINATION) { > MPI_Send(..); > break; > } > } > > In this case, all clients do an MPI_Comm_connect and then send a message with tag = REGULAR_CONNECTION. When thread 2 is done with its work, it’ll do an MPI_Comm_connect and then send a message with tag = TERMINATION, wait for a response from thread 1, and call finalize. > > — Pavan > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Roy, Hirak 08 Oct '14

08 Oct '14

Hi Pavan, Here is my code for thread2 : do { MPI_Comm newComm ; MPI_Comm_accept (m_serverPort.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &newComm); Log ("Accepted a connection"); int buf = 0 ; MPI_Status status ; MPI_Recv(&buf, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG, newComm, &status); if (status.MPI_TAG == MPI_MSG_TAG_NEW_CONN) { m_clientComs[m_clientCount] = newComm ; m_clientCount++; } else if (status.MPI_TAG == MPI_MSG_TAG_SHUTDOWN) { Log ("Shutdown"); //MPI_Comm_disconnect (&newComm); Log ("Disconnect"); break; } else { Log ("Unmatched Receive"); } } while(1) ; Here is my code for thread1 to terminate thread2 : MPI_Comm newComm ; MPI_Comm_connect (m_serverPort.c_str(), MPI_INFO_NULL, 0, MPI_COMM_SELF, &newComm); Log ("Connect to Self"); int val = 0 ; MPI_Request req ; MPI_Send(&val, 1, MPI_INT, 0, MPI_MSG_TAG_SHUTDOWN, newComm); Log ("Successful"); //MPI_Status stat ; //MPI_Wait(&req, &stat); Log ("Complete"); //MPI_Comm_disconnect(&newComm); The MPI_Send/Recv waits. I am using sock channel. For nemesis, I get the following crash : Assertion failed in file ./src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h at line 58: vc_ch->is_local internal ABORT - process 0 I tried non-blocking send and receive followed by wait. However, that also does not solve the problem. Thanks, Hirak ----- Hirak, Your approach should work fine. I'm not sure what issue you are facing. I assume thread 1 is doing this: while (1) { MPI_Comm_accept(..); MPI_Recv(.., tag, ..); if (tag == REGULAR_CONNECTION) continue; else if (tag == TERMINATION) { MPI_Send(..); break; } } In this case, all clients do an MPI_Comm_connect and then send a message with tag = REGULAR_CONNECTION. When thread 2 is done with its work, it'll do an MPI_Comm_connect and then send a message with tag = TERMINATION, wait for a response from thread 1, and call finalize. - Pavan _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Roy, Hirak 07 Oct '14

07 Oct '14

I tried to do : pthread_cancel(thread2). This terminates the blocking accept. However, MPI_Finalize hangs. Regards, Hirak From: Roy, Hirak Sent: Wednesday, October 08, 2014 9:28 AM To: discuss(a)mpich.org Subject: Re: How to terminate MPI_Comm_accept Hi Huiwei, Thanks for your response. I don't think the solution you suggested will work. MPI_Comm_accept is a blocking MPI call and the call does not return until there is a matching MPI_Comm_connect or implementation specified timeout ( the timeout can not be configured and reasonably high). So when Thread2 is waiting on MPI_Comm_accept, it neither can receive any message nor can check any shared flag value. That's why I tried to terminate the accept by connecting from thread1. Please let me know if you have any other suggestion. Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Balaji, Pavan 07 Oct '14

07 Oct '14

Hirak, Your approach should work fine. I’m not sure what issue you are facing. I assume thread 1 is doing this: while (1) { MPI_Comm_accept(..); MPI_Recv(.., tag, ..); if (tag == REGULAR_CONNECTION) continue; else if (tag == TERMINATION) { MPI_Send(..); break; } } In this case, all clients do an MPI_Comm_connect and then send a message with tag = REGULAR_CONNECTION. When thread 2 is done with its work, it’ll do an MPI_Comm_connect and then send a message with tag = TERMINATION, wait for a response from thread 1, and call finalize. — Pavan On Oct 7, 2014, at 10:58 PM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > Hi Huiwei, > > Thanks for your response. > > I don’t think the solution you suggested will work. > MPI_Comm_accept is a blocking MPI call and the call does not return until there is a matching MPI_Comm_connect or implementation specified timeout ( the timeout can not be configured and reasonably high). > So when Thread2 is waiting on MPI_Comm_accept, it neither can receive any message nor can check any shared flag value. > > That’s why I tried to terminate the accept by connecting from thread1. > > Please let me know if you have any other suggestion. > > Thanks, > Hirak > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Pavan Balaji ✉️ http://www.mcs.anl.gov/~balaji _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Roy, Hirak 07 Oct '14

07 Oct '14

Hi Huiwei, Thanks for your response. I don't think the solution you suggested will work. MPI_Comm_accept is a blocking MPI call and the call does not return until there is a matching MPI_Comm_connect or implementation specified timeout ( the timeout can not be configured and reasonably high). So when Thread2 is waiting on MPI_Comm_accept, it neither can receive any message nor can check any shared flag value. That's why I tried to terminate the accept by connecting from thread1. Please let me know if you have any other suggestion. Thanks, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to terminate MPI_Comm_accept
by Lu, Huiwei 07 Oct '14

07 Oct '14

Hi, Hirak, It looks like there is no need for Thread 2 to do an additional MPI_Comm_connect before communicating with Thread 1. Thread 1 and Thread 2 are already in the same communicator, MPI_COMM_SELF. Why not directly send a message from Thread 2 (MPI_Isend, MPI_Wait) to Thread 1 (MPI_Irecv, MPI_Wait) using MPI_COMM_SELF? Moreover, since Thread 1 and 2 share the same address space, they can directly write to each other’s memory, so MPI_Isend/MPI_Irecv can be replaced by shared memory write/read. Let Thread 1 and 2 share one flag=0. After Thread 2 finishes, it can set flag=1. Thread 1 checks the flag in its loop, if flag=1, it can then terminate. Regards, — Huiwei On Oct 7, 2014, at 6:17 AM, Roy, Hirak <Hirak_Roy(a)mentor.com> wrote: > Hi All, > > I am working on a MPI based server-client application. > The number of clients are not fixed. > The clients can come up at any time and will connect to server. > > The server runs two threads. > > Thread 1: opens a port, calls MPI_Comm_accept on a loop on MPI_COMM_SELF. > Thread 2: Distributes works among the clients. > > When the client comes up, it calls MPI_Comm_connect to server. > > The problem I am facing is : how to terminate the blocking MPI_Comm_accept of Therad 1, once Thread2 is done with all the work? > > I tried the following thing which might not be legal to do: > > Thread2 calls MPI_Comm_connect (MPI_COMM_SELF) to Thread1’s port. (created a new inter-communicator newComm) > However, when I tried to send a non-blocking message to close the port from Thread2 (MPI_Isend, MPI_Wait) to Thread1 (MPI_Irecv, MPI_Wait) using newComm, the two threads wait forever. > > Any help regarding the termination of MPI_Comm_accept will be appreciated. > > Regards, > Hirak > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

How to terminate MPI_Comm_accept
by Roy, Hirak 07 Oct '14

07 Oct '14

Hi All, I am working on a MPI based server-client application. The number of clients are not fixed. The clients can come up at any time and will connect to server. The server runs two threads. Thread 1: opens a port, calls MPI_Comm_accept on a loop on MPI_COMM_SELF. Thread 2: Distributes works among the clients. When the client comes up, it calls MPI_Comm_connect to server. The problem I am facing is : how to terminate the blocking MPI_Comm_accept of Therad 1, once Thread2 is done with all the work? I tried the following thing which might not be legal to do: Thread2 calls MPI_Comm_connect (MPI_COMM_SELF) to Thread1's port. (created a new inter-communicator newComm) However, when I tried to send a non-blocking message to close the port from Thread2 (MPI_Isend, MPI_Wait) to Thread1 (MPI_Irecv, MPI_Wait) using newComm, the two threads wait forever. Any help regarding the termination of MPI_Comm_accept will be appreciated. Regards, Hirak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Use other's mpicc to compile code with my modified ROMIO
by Jaln 05 Oct '14

05 Oct '14

Thank you so much, Rob. Best, Jialin On Sun, Oct 5, 2014 at 7:54 PM, Rob Latham <robl(a)mcs.anl.gov> wrote: > > > On 10/05/2014 04:23 PM, Jaln wrote: > > Can you please tell me how to compile the code with a standalone ROMIO? >> For example, with cray’s default mpi compiler, and a modified ROMIO >> source, and a test MPI program, e.g., test.c >> > > It should be as easy as > mpicc test.c /path/to/my/romio/libromio.a -o test > > But even though I've done this many times, I still find myself putting a > "hello from my own romio" inside MPI_File_open just to double check. > > ==rob > > -- > Rob Latham > Mathematics and Computer Science Division > Argonne National Lab, IL USA > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Use other's mpicc to compile code with my modified ROMIO
by Rob Latham 05 Oct '14

05 Oct '14

On 10/05/2014 04:23 PM, Jaln wrote: > Can you please tell me how to compile the code with a standalone ROMIO? > For example, with cray’s default mpi compiler, and a modified ROMIO > source, and a test MPI program, e.g., test.c It should be as easy as mpicc test.c /path/to/my/romio/libromio.a -o test But even though I've done this many times, I still find myself putting a "hello from my own romio" inside MPI_File_open just to double check. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0