Thank you Ken. Really appreciate this. From: Raffenetti, Ken <[email protected]> Date: Thursday, October 10, 2024 at 9:55 AM To: Niyaz Murshed <[email protected]>, [email protected] <[email protected]> Cc: nd <[email protected]> Subject: Re: [mpich-discuss] MPI_Send over ibverbs This is the post of the recv for the acknowledgement message: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8d... This is where the ack message is sent: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8d... Ken From: Niyaz Murshed <[email protected]> Date: Tuesday, October 8, 2024 at 11:32 AM To: Raffenetti, Ken <[email protected]>, [email protected] <[email protected]> Cc: nd <[email protected]> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 10 AM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <[email protected]> Date: Tuesday, October 8, 2024 at 11:10 AM To: Niyaz Murshed <[email protected]>, [email protected] <[email protected]> Cc: nd <[email protected]> Subject: Re: [mpich-discuss] MPI_Send over ibverbs It’s in MPI scope. To my knowledge, that level of acknowledgement is not exposed from the verbs API. Ken From: Niyaz Murshed <[email protected]> Date: Tuesday, October 8, 2024 at 11:08 AM To: Raffenetti, Ken <[email protected]>, [email protected] <[email protected]> Cc: nd <[email protected]> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 05 AM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <[email protected]> Date: Tuesday, October 8, 2024 at 11:05 AM To: [email protected] <[email protected]> Cc: Niyaz Murshed <[email protected]>, nd <[email protected]> Subject: Re: [mpich-discuss] MPI_Send over ibverbs An internal acknowledgement message is sent from the receiver to the sender indicating that a particular synchronous send operation was matched. Ken From: Niyaz Murshed via discuss <[email protected]> Date: Tuesday, October 8, 2024 at 10:49 AM To: [email protected] <[email protected]> Cc: Niyaz Murshed <[email protected]>, nd <[email protected]> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ –

From the above statement, how do the sender know that the receiver called a matching receive?

I am asking from the point of view of MPI_Send over RDMA. In case of RDMA, internally it will call rdma_send which does not know when a matching rdma_recv was called on destination. How do the sender knows when rdma_recv was called? From: Joachim Jenke via discuss <[email protected]> Date: Tuesday, October 8, 2024 at 9:42 AM To: [email protected] <[email protected]> Cc: Joachim Jenke <[email protected]> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Hi, none of the MPI send calls has a requirement to return after the message was received. MPI_Send can return as soon as the caller can write to buffer without impact to the data being transferred. This might be after the data is put to the network, or the MPI library copied the data to an internal buffer. MPI_Ssend has the requirement to return after the message was matched at the receiver, which implies that the matching Receive was called. But it still does not require that the message was received at this point. Best Joachim Am 08.10.24 um 04:55 schrieb Niyaz Murshed via discuss:

Hi,

Is there a way to check the implementation of MPI_Send?

As I understand there is an implicit barrier which allows MPI_Send to return only after the buffer is received by the destination.

How is this communication done? How does the destination notify the sender that buffer is received?

How is this implemented when using rocev2 ?

Regards,

Niyaz

_______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss_... <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/listinfo/discuss__;!!G_uCfscf7eWS!aTxlD8Xx22PDuMXv--UZcA68fdpz5fzzNTbFlnm4W8ILFpny-KqRwNDxs2-iX7Ob14g7ReYsSsFBpE5IiC8$>

-- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 [email protected] https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!fu... <https://urldefense.us/v3/__http:/www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!aTxlD8Xx22PDuMXv--UZcA68fdpz5fzzNTbFlnm4W8ILFpny-KqRwNDxs2-iX7Ob14g7ReYsSsFBFKt98Uc$>