September 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] issue/question
by Thakur, Rajeev 15 Sep '23

15 Sep '23

The mailing list for MVAPICH2 is mvapich-discuss(a)lists.osu.edu<mailto:[email protected]> . Rajeev From: "Michael P. Deignan via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, September 15, 2023 at 10:08 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Michael P. Deignan" <michael.p.deignan(a)gmail.com> Subject: [mpich-discuss] issue/question I have a user who is running an mpi model using mvapich2, compiled with the Intel OneAPI compiler (2023.1.0) on a Rocky 8.6 OpenHPC cluster. Bear with me for a minute as I lay the foundation to get to rdma-core. Periodically (randomly after X minutes, where X has been as few as 5 and as much as 30) the model will crash and in the error log there will be thousands of messages: [mv2_mcast_resend_window] Failed to post mcast send errno:0 Basically this message repeats until the disk fills up and the job terminates with an out of disk space message. I tracked the error to the mvapich2 source code at src/mpid/ch3/channels/common/include/ibv_mcast.h: int ret; \ ret = ibv_post_send(_mcast_ctx->ud_ctx->qp, \ &(_v->desc.u.sr), &(_v->desc.y.bad_sr)); \ if (ret) { \ PRINT_ERROR("Failed to post mcast send errno:%d\n", \ errno); \ } \ _mcast_ctx->ud_ctx->send_wqes_avail--; \ while (_mcast_ctx->ud_ctx->send_wqes_avail <= 0) { \ MPIDI_CH3I_Progress(FALSE, NULL); \ } which leads me to the ibv_post_send subroutine in include/infiniband/verbs.h: /** * ibv_post_send - Post a list of work requests to a send queue. * * If IBV_SEND_INLINE flag is set, the data buffers can be reused * immediately after the call returns. */ static inline int ibv_post_send(struct ibv_qp *qp, struct ibv_send_wr *wr, struct ibv_send_wr **bad_wr) { return qp->context->ops.post_send(qp, wr, bad_wr); } For some reason, this function call (ibv_post_send) is returning zero back to the caller, and no error code is being set (errno = 0). As the model does run for a random amount of time, this would seem to suggest some type of hardware problem, but everything we've checked (system, ib card, ib switch, etc.,) doesn't show any errors. Can anyone shed some light on under what circumstances a call to ibv_post_send would, in fact, return zero when previously it didn't? I realize I'm probably not giving much data to really help.. I suppose there could be some type of underlying operating system issue but even there I'm not sure how I would determine where the problem is, since everything else is running fine (and it appears only this one model seems to cause this problem) with nothing abnormal in the event logs. I tried running the model on an older Rocks Centos 7.x cluster and likewise received the same error. Having some basic understanding of what would cause the post send call to return zero and not set an error code might help.

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Joachim Jenke 15 Sep '23

15 Sep '23

Am 15.09.23 um 17:09 schrieb Tony Curtis via discuss: > > >> On Sep 15, 2023, at 11:07 AM, Raffenetti, Ken via discuss >> <discuss(a)mpich.org> wrote: >> >> 1. Is there a way to detect this kind of overload with an MPI call? >> >> If MPI detects an error at runtime, the default behavior is to abort >> the application. If you application does not abort (and you haven't >> changed the default error handler), then no error was detected by MPI. >> > > There’s a tool called MUST that might help > > MUST - RWTH AACHEN UNIVERSITY Lehrstuhl für Informatik 12 - Deutsch > <https://www.i12.rwth-aachen.de/go/id/nrbe> > i12.rwth-aachen.de <https://www.i12.rwth-aachen.de/go/id/nrbe> > apple-touch-icon-180x180.png <https://www.i12.rwth-aachen.de/go/id/nrbe> > > <https://www.i12.rwth-aachen.de/go/id/nrbe> > The current release version can only detect conflicts in buffer usage at MPI API level. That means, it will only detect buffer conflicts for in-flight messages as in: MPI_Irecv(buf, MPI_INT, 10, ..., req1); MPI_Irecv(&buf[9], MPI_INT, 10, ..., req2); MPI_Wait(req1,...); MPI_Wait(req2,...); The upcoming release I was referencing in my other mail would detect conflicting accesses to in-flight buffers as in: MPI_Irecv(buf, MPI_INT, 10, ..., req); buf[5]=5; MPI_Wait(req,...); > > (Not affiliated, just happen to have been looking at it) Happy to see that people look at the tool :D - Joachim > > Tony > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Tony Curtis 15 Sep '23

15 Sep '23

> On Sep 15, 2023, at 11:07 AM, Raffenetti, Ken via discuss <discuss(a)mpich.org> wrote: > > 1. Is there a way to detect this kind of overload with an MPI call? > > If MPI detects an error at runtime, the default behavior is to abort the application. If you application does not abort (and you haven't changed the default error handler), then no error was detected by MPI. > There’s a tool called MUST that might help https://www.i12.rwth-aachen.de/go/id/nrbe (Not affiliated, just happen to have been looking at it) Tony

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Joachim Jenke 15 Sep '23

15 Sep '23

If you run into the same issue with different MPI implementations, this actually sounds like an issue in the application. Either the sender overwrites the buffer between the isend and the wait, or the receiver side overwrites the receive buffer between the recv and your buffer verification. We are wrapping up a MUST release that tackles the detection of this kind of issues. You would need to compile your code with either LLVM or GNU compiler. If you are interested in the details, just let me know. - Joachim Am 15.09.23 um 17:00 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > Yes, I tried it with OpenMPI and the same problem occurred. > > Kurt > > -----Original Message----- > From: Thakur, Rajeev <thakur(a)anl.gov> > Sent: Friday, September 15, 2023 9:59 AM > To: discuss(a)mpich.org > Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages > > Does it happen with other MPI implementations? > > Rajeev > > -----Original Message----- > From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org <mailto:[email protected]>> > Reply-To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> > Date: Friday, September 15, 2023 at 9:43 AM > To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> > Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov <mailto:[email protected]>> > Subject: Re: [mpich-discuss] Buffer corruption due to an excessive number of messages > > > Joachim, > > > Unfortunately, using MPI_Improbe/MPI_Mrecv didn't solve the problem -- I still am receiving buffers with invalid data objects near the ends of the buffers. The problem goes away when I reduce the size of the job (number of nodes), making me think it is the large number of messages that is causing the problem. > > > 1. Is there a way to detect this kind of overload with an MPI call? > 2. Is there an upper bound on the number of messages that can be "in flight"? > 3. Is there a upper bound on message length? > > > Or is there some other possible cause that I haven't thought of? > > > Thanks, > Kurt > > > -----Original Message----- > From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]>> > Sent: Thursday, September 14, 2023 3:10 PM > To: discuss(a)mpich.org <mailto:[email protected]> > Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> > Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages > > > CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. > > > > > > > > > Hi Kurt, > > > just a thought: do you execute single-threaded or multi-threaded? > > > In case of multi-threaded execution, you should look into MPI_Improbe/MPI_Mrecv just to make sure that you really receive the message you probed for. > Even in single-threaded execution you might try whether using these functions instead fixes your issue. > > > Best > Joachim > > > Am 14.09.23 um 22:02 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: >> It seems that when I send a process too many non-blocking messages >> (with >> MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer >> >> with corrupted data for some of the messages. Usually the corrupted >> data objects are at the end of the array that was sent. I checked the >> >> buffers passed to MPI_Isend, and they are uncorrupted. >> >> 1. Is there a way to detect this kind of overload with an MPI call? >> 2. Is there an upper bound on the number of messages that can be "in >> flight"? >> 3. Is there a upper bound on message length? >> >> Thanks, >> >> Kurt >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> To >> manage subscription options or unsubscribe: >> https://list/ >> s.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=05%7C01%7Ckurt.e.mccal >> l%40nasa.gov%7C9e8bfd71610243562ce808dbb55eaace%7C7005d45845be48ae8140 >> d43da96dd17b%7C0%7C0%7C638303190427564481%7CUnknown%7CTWFpbGZsb3d8eyJW >> IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% >> 7C%7C%7C&sdata=yplpEqPO1a8OpC2Nt3z55G2UtfA0lBqV4Zn5OrMRRx0%3D&reserved >> =0 > > > -- > Dr. rer. nat. Joachim Jenke > > > IT Center > Group: High Performance Computing > Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) > Tel: +49 241 80- 24765 > Fax: +49 241 80-624765 > jenke(a)itc.rwth-aachen.de <mailto:[email protected]> > http://www.itc.rwth-aachen.de/ <http://www.itc.rwth-aachen.de/> > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

issue/question
by Michael P. Deignan 15 Sep '23

15 Sep '23

I have a user who is running an mpi model using mvapich2, compiled with the Intel OneAPI compiler (2023.1.0) on a Rocky 8.6 OpenHPC cluster. Bear with me for a minute as I lay the foundation to get to rdma-core. Periodically (randomly after X minutes, where X has been as few as 5 and as much as 30) the model will crash and in the error log there will be thousands of messages: [mv2_mcast_resend_window] Failed to post mcast send errno:0 Basically this message repeats until the disk fills up and the job terminates with an out of disk space message. I tracked the error to the mvapich2 source code at src/mpid/ch3/channels/common/include/ibv_mcast.h: int ret; \ ret = ibv_post_send(_mcast_ctx->ud_ctx->qp, \ &(_v->desc.u.sr), &(_v->desc.y.bad_sr)); \ if (ret) { \ PRINT_ERROR("Failed to post mcast send errno:%d\n", \ errno); \ } \ _mcast_ctx->ud_ctx->send_wqes_avail--; \ while (_mcast_ctx->ud_ctx->send_wqes_avail <= 0) { \ MPIDI_CH3I_Progress(FALSE, NULL); \ } which leads me to the ibv_post_send subroutine in include/infiniband/verbs.h: /** * ibv_post_send - Post a list of work requests to a send queue. * * If IBV_SEND_INLINE flag is set, the data buffers can be reused * immediately after the call returns. */ static inline int ibv_post_send(struct ibv_qp *qp, struct ibv_send_wr *wr, struct ibv_send_wr **bad_wr) { return qp->context->ops.post_send(qp, wr, bad_wr); } For some reason, this function call (ibv_post_send) is returning zero back to the caller, and no error code is being set (errno = 0). As the model does run for a random amount of time, this would seem to suggest some type of hardware problem, but everything we've checked (system, ib card, ib switch, etc.,) doesn't show any errors. Can anyone shed some light on under what circumstances a call to ibv_post_send would, in fact, return zero when previously it didn't? I realize I'm probably not giving much data to really help.. I suppose there could be some type of underlying operating system issue but even there I'm not sure how I would determine where the problem is, since everything else is running fine (and it appears only this one model seems to cause this problem) with nothing abnormal in the event logs. I tried running the model on an older Rocks Centos 7.x cluster and likewise received the same error. Having some basic understanding of what would cause the post send call to return zero and not set an error code might help.

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Raffenetti, Ken 15 Sep '23

15 Sep '23

1. Is there a way to detect this kind of overload with an MPI call? If MPI detects an error at runtime, the default behavior is to abort the application. If you application does not abort (and you haven't changed the default error handler), then no error was detected by MPI. 2. Is there an upper bound on the number of messages that can be "in flight"? There is, but it depends on configuration and things like the number of communicators in use. It is difficult to give an exact answer. However, if you go over the limit, MPI should crash and let you know something went wrong. 3. Is there a upper bound on message length? Not really. We test very large message lengths in our regression tests. Counts of INT_MAX are well tested. Larger counts are supported using the new _c interfaces added in MPI-4. (MPI_Send_c, MPI_Recv_c, etc). Or is there some other possible cause that I haven't thought of? Is it possible the same buffer is posted to different recv operations simultaneously? MPI will not check for this. Concurrent writes to the same buffer could result in corrupted data. Another possibility is message underflow (amount received is less than the size of the posted buffer), meaning the contents of the buffer beyond the received message are not modified by MPI and may be garbage. Ken -----Original Message----- From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org <mailto:[email protected]>> Reply-To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Date: Friday, September 15, 2023 at 10:01 AM To: "Thakur, Rajeev" <thakur(a)anl.gov <mailto:[email protected]>>, "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov <mailto:[email protected]>> Subject: Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Yes, I tried it with OpenMPI and the same problem occurred. Kurt -----Original Message----- From: Thakur, Rajeev <thakur(a)anl.gov <mailto:[email protected]>> Sent: Friday, September 15, 2023 9:59 AM To: discuss(a)mpich.org <mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov <mailto:[email protected]>> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Does it happen with other MPI implementations? Rajeev -----Original Message----- From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Reply-To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Date: Friday, September 15, 2023 at 9:43 AM To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Subject: Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Joachim, Unfortunately, using MPI_Improbe/MPI_Mrecv didn't solve the problem -- I still am receiving buffers with invalid data objects near the ends of the buffers. The problem goes away when I reduce the size of the job (number of nodes), making me think it is the large number of messages that is causing the problem. 1. Is there a way to detect this kind of overload with an MPI call? 2. Is there an upper bound on the number of messages that can be "in flight"? 3. Is there a upper bound on message length? Or is there some other possible cause that I haven't thought of? Thanks, Kurt -----Original Message----- From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Sent: Thursday, September 14, 2023 3:10 PM To: discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Hi Kurt, just a thought: do you execute single-threaded or multi-threaded? In case of multi-threaded execution, you should look into MPI_Improbe/MPI_Mrecv just to make sure that you really receive the message you probed for. Even in single-threaded execution you might try whether using these functions instead fixes your issue. Best Joachim Am 14.09.23 um 22:02 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > It seems that when I send a process too many non-blocking messages > (with > MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer > > with corrupted data for some of the messages. Usually the corrupted > data objects are at the end of the array that was sent. I checked the > > buffers passed to MPI_Isend, and they are uncorrupted. > > 1. Is there a way to detect this kind of overload with an MPI call? > 2. Is there an upper bound on the number of messages that can be "in > flight"? > 3. Is there a upper bound on message length? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> To > manage subscription options or unsubscribe: > https://list/ > s.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=05%7C01%7Ckurt.e.mccal > l%40nasa.gov%7C9e8bfd71610243562ce808dbb55eaace%7C7005d45845be48ae8140 > d43da96dd17b%7C0%7C0%7C638303190427564481%7CUnknown%7CTWFpbGZsb3d8eyJW > IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% > 7C%7C%7C&sdata=yplpEqPO1a8OpC2Nt3z55G2UtfA0lBqV4Zn5OrMRRx0%3D&reserved > =0 -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> http://www.itc.rwth-aachen.de/ <http://www.itc.rwth-aachen.de/> <http://www.itc.rwth-aachen.de/> <http://www.itc.rwth-aachen.de/>> _______________________________________________ discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> <https://lists.mpich.org/mailman/listinfo/discuss> <https://lists.mpich.org/mailman/listinfo/discuss>> _______________________________________________ discuss mailing list discuss(a)mpich.org <mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss>

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Thakur, Rajeev 15 Sep '23

15 Sep '23

Then it is less likely to be a bug in the implementation. Rajeev -----Original Message----- From: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov <mailto:[email protected]>> Date: Friday, September 15, 2023 at 10:01 AM To: "Thakur, Rajeev" <thakur(a)anl.gov <mailto:[email protected]>>, "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Subject: Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Yes, I tried it with OpenMPI and the same problem occurred. Kurt -----Original Message----- From: Thakur, Rajeev <thakur(a)anl.gov <mailto:[email protected]>> Sent: Friday, September 15, 2023 9:59 AM To: discuss(a)mpich.org <mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov <mailto:[email protected]>> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Does it happen with other MPI implementations? Rajeev -----Original Message----- From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Reply-To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Date: Friday, September 15, 2023 at 9:43 AM To: "discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>" <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Subject: Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Joachim, Unfortunately, using MPI_Improbe/MPI_Mrecv didn't solve the problem -- I still am receiving buffers with invalid data objects near the ends of the buffers. The problem goes away when I reduce the size of the job (number of nodes), making me think it is the large number of messages that is causing the problem. 1. Is there a way to detect this kind of overload with an MPI call? 2. Is there an upper bound on the number of messages that can be "in flight"? 3. Is there a upper bound on message length? Or is there some other possible cause that I haven't thought of? Thanks, Kurt -----Original Message----- From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Sent: Thursday, September 14, 2023 3:10 PM To: discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Hi Kurt, just a thought: do you execute single-threaded or multi-threaded? In case of multi-threaded execution, you should look into MPI_Improbe/MPI_Mrecv just to make sure that you really receive the message you probed for. Even in single-threaded execution you might try whether using these functions instead fixes your issue. Best Joachim Am 14.09.23 um 22:02 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > It seems that when I send a process too many non-blocking messages > (with > MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer > > with corrupted data for some of the messages. Usually the corrupted > data objects are at the end of the array that was sent. I checked the > > buffers passed to MPI_Isend, and they are uncorrupted. > > 1. Is there a way to detect this kind of overload with an MPI call? > 2. Is there an upper bound on the number of messages that can be "in > flight"? > 3. Is there a upper bound on message length? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> To > manage subscription options or unsubscribe: > https://list/ > s.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=05%7C01%7Ckurt.e.mccal > l%40nasa.gov%7C9e8bfd71610243562ce808dbb55eaace%7C7005d45845be48ae8140 > d43da96dd17b%7C0%7C0%7C638303190427564481%7CUnknown%7CTWFpbGZsb3d8eyJW > IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% > 7C%7C%7C&sdata=yplpEqPO1a8OpC2Nt3z55G2UtfA0lBqV4Zn5OrMRRx0%3D&reserved > =0 -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> http://www.itc.rwth-aachen.de/ <http://www.itc.rwth-aachen.de/> <http://www.itc.rwth-aachen.de/> <http://www.itc.rwth-aachen.de/>> _______________________________________________ discuss mailing list discuss(a)mpich.org <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> <https://lists.mpich.org/mailman/listinfo/discuss> <https://lists.mpich.org/mailman/listinfo/discuss>>

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Mccall, Kurt E. (MSFC-EV41) 15 Sep '23

15 Sep '23

Yes, I tried it with OpenMPI and the same problem occurred. Kurt -----Original Message----- From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Friday, September 15, 2023 9:59 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Does it happen with other MPI implementations? Rajeev -----Original Message----- From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org <mailto:[email protected]>> Reply-To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Date: Friday, September 15, 2023 at 9:43 AM To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov <mailto:[email protected]>> Subject: Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Joachim, Unfortunately, using MPI_Improbe/MPI_Mrecv didn't solve the problem -- I still am receiving buffers with invalid data objects near the ends of the buffers. The problem goes away when I reduce the size of the job (number of nodes), making me think it is the large number of messages that is causing the problem. 1. Is there a way to detect this kind of overload with an MPI call? 2. Is there an upper bound on the number of messages that can be "in flight"? 3. Is there a upper bound on message length? Or is there some other possible cause that I haven't thought of? Thanks, Kurt -----Original Message----- From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]>> Sent: Thursday, September 14, 2023 3:10 PM To: discuss(a)mpich.org <mailto:[email protected]> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Hi Kurt, just a thought: do you execute single-threaded or multi-threaded? In case of multi-threaded execution, you should look into MPI_Improbe/MPI_Mrecv just to make sure that you really receive the message you probed for. Even in single-threaded execution you might try whether using these functions instead fixes your issue. Best Joachim Am 14.09.23 um 22:02 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > It seems that when I send a process too many non-blocking messages > (with > MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer > > with corrupted data for some of the messages. Usually the corrupted > data objects are at the end of the array that was sent. I checked the > > buffers passed to MPI_Isend, and they are uncorrupted. > > 1. Is there a way to detect this kind of overload with an MPI call? > 2. Is there an upper bound on the number of messages that can be "in > flight"? > 3. Is there a upper bound on message length? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> To > manage subscription options or unsubscribe: > https://list/ > s.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=05%7C01%7Ckurt.e.mccal > l%40nasa.gov%7C9e8bfd71610243562ce808dbb55eaace%7C7005d45845be48ae8140 > d43da96dd17b%7C0%7C0%7C638303190427564481%7CUnknown%7CTWFpbGZsb3d8eyJW > IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% > 7C%7C%7C&sdata=yplpEqPO1a8OpC2Nt3z55G2UtfA0lBqV4Zn5OrMRRx0%3D&reserved > =0 -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de <mailto:[email protected]> http://www.itc.rwth-aachen.de/ <http://www.itc.rwth-aachen.de/> _______________________________________________ discuss mailing list discuss(a)mpich.org <mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss>

1 0

Re: [mpich-discuss] Buffer corruption due to an excessive number of messages
by Thakur, Rajeev 15 Sep '23

15 Sep '23

Does it happen with other MPI implementations? Rajeev -----Original Message----- From: "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org <mailto:[email protected]>> Reply-To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Date: Friday, September 15, 2023 at 9:43 AM To: "discuss(a)mpich.org <mailto:[email protected]>" <discuss(a)mpich.org <mailto:[email protected]>> Cc: "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov <mailto:[email protected]>> Subject: Re: [mpich-discuss] Buffer corruption due to an excessive number of messages Joachim, Unfortunately, using MPI_Improbe/MPI_Mrecv didn't solve the problem -- I still am receiving buffers with invalid data objects near the ends of the buffers. The problem goes away when I reduce the size of the job (number of nodes), making me think it is the large number of messages that is causing the problem. 1. Is there a way to detect this kind of overload with an MPI call? 2. Is there an upper bound on the number of messages that can be "in flight"? 3. Is there a upper bound on message length? Or is there some other possible cause that I haven't thought of? Thanks, Kurt -----Original Message----- From: Joachim Jenke via discuss <discuss(a)mpich.org <mailto:[email protected]>> Sent: Thursday, September 14, 2023 3:10 PM To: discuss(a)mpich.org <mailto:[email protected]> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de <mailto:[email protected]>> Subject: [EXTERNAL] [BULK] Re: [mpich-discuss] Buffer corruption due to an excessive number of messages CAUTION: This email originated from outside of NASA. Please take care when clicking links or opening attachments. Use the "Report Message" button to report suspicious messages to the NASA SOC. Hi Kurt, just a thought: do you execute single-threaded or multi-threaded? In case of multi-threaded execution, you should look into MPI_Improbe/MPI_Mrecv just to make sure that you really receive the message you probed for. Even in single-threaded execution you might try whether using these functions instead fixes your issue. Best Joachim Am 14.09.23 um 22:02 schrieb Mccall, Kurt E. (MSFC-EV41) via discuss: > It seems that when I send a process too many non-blocking messages > (with > MPI_Isend) , MPI_Iprobe/MPI_Recv sometimes returns a buffer > > with corrupted data for some of the messages. Usually the corrupted > data objects are at the end of the array that was sent. I checked the > > buffers passed to MPI_Isend, and they are uncorrupted. > > 1. Is there a way to detect this kind of overload with an MPI call? > 2. Is there an upper bound on the number of messages that can be "in > flight"? > 3. Is there a upper bound on message length? > > Thanks, > > Kurt > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://list/ > s.mpich.org%2Fmailman%2Flistinfo%2Fdiscuss&data=05%7C01%7Ckurt.e.mccal > l%40nasa.gov%7C9e8bfd71610243562ce808dbb55eaace%7C7005d45845be48ae8140 > d43da96dd17b%7C0%7C0%7C638303190427564481%7CUnknown%7CTWFpbGZsb3d8eyJW > IjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000% > 7C%7C%7C&sdata=yplpEqPO1a8OpC2Nt3z55G2UtfA0lBqV4Zn5OrMRRx0%3D&reserved > =0 -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de <mailto:[email protected]> http://www.itc.rwth-aachen.de/ <http://www.itc.rwth-aachen.de/> _______________________________________________ discuss mailing list discuss(a)mpich.org <mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss>

1 0

libmpi size unusually large when building with CUDA support
by Lucas Zacchi de Medeiros 15 Sep '23

15 Sep '23

I work on a project that supports different architectures, so we've built 3 separate versions of MPICH 4.1.1 (cuda11.2, rocm5.4.3 and cpu-only with ucx1.14.1) The cpu-only and the rocm5 libraries are around 50MB each. The CUDA version on the other hand, is close to 1.8GB! There doesn't seem to be anything out of the ordinary with the builds and all versions are working as expected. This is the output of du -sh performed on the install directories: First the rocm5 build: $ du -sh LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/* (...) 53M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.a 40M LINUX_gcc9.3_glibc2.28_rocm5.4.3_ucx1.14.1/lib/libmpi.so.12.3.0 And then the CUDA build: $ du -sh LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/* (...) 1.8G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.a 1.7G LINUX_gcc9.3_glibc2.17_cuda11.2_ucx1.14.1/lib/libmpi.so.12.3.0 This issue really complicates packaging and distribution since we provide both the archive and the shared libraries, I am looking at more than 3GB for MPICH only. After some investigation, it doesn't seem the problem is coming from our end. Is it possible that something in mpich's build process is causing this excessive file size? Thanks for the help. Kind regards, Lucas

1 0