March 2020 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Binary compatibility
by Sajid Ali 17 Mar '20

17 Mar '20

Hi Saurabh, You can also view the status of the same at : https://abi-laboratory.pro/index.php?view=timeline&l=mpich -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] Binary compatibility
by Wesley Bland 17 Mar '20

17 Mar '20

The ABI compatibility initiative applies within the same implementation as well. So all MPICH versions back to MPICH 3.1 are binary compatible with each other. https://www.mpich.org/abi/ <https://www.mpich.org/abi/> It’s possible that MPICH 3.0 is also compatible, but I don’t think there were any guarantees or tests. You might just get lucky if that’s the version you want. Thanks, Wes > On Mar 17, 2020, at 7:40 AM, Saurabh T via discuss <discuss(a)mpich.org> wrote: > > What is the guarantee for binary (ABI) compatibility between MPICH versions? I am not referring to the ABI compatibility initiative with other MPI implementations, but between different versions of MPICH. For example, is MPICH 3.3 ABI compatible with MPICH 3.0? Thank you. > > saurabh > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Binary compatibility
by Saurabh T 17 Mar '20

17 Mar '20

What is the guarantee for binary (ABI) compatibility between MPICH versions? I am not referring to the ABI compatibility initiative with other MPI implementations, but between different versions of MPICH. For example, is MPICH 3.3 ABI compatible with MPICH 3.0? Thank you. saurabh

1 0

MPICH build errors, cannot find -levent(_pthreads)
by Sajid Ali 16 Mar '20

16 Mar '20

Hi MPICH-developers, I’m seeing a build error for mpich-3.3.2 on a cluster using gcc-9.3.0. I tried installing mpich using spack while linking against ucx and pmix installed by sysadmins at /usr/local. The error is as follows : >> 14144 /usr/bin/ld: cannot find -levent >> 14145 /usr/bin/ld: cannot find -levent_pthreads >> 14146 collect2: error: ld returned 1 exit status >> 14147 make[2]: *** [lib/libmpi.la] Error 1 14148 make[2]: Leaving directory `/tmp/sas4990/spack-stage/spack-stage- mpich-3.3.2-no52nugggqn3fuligbvtucr2uaxdf47d/spack-src' >> 14149 make[1]: *** [all-recursive] Error 1 14150 make[1]: Leaving directory `/tmp/sas4990/spack-stage/spack-stage- mpich-3.3.2-no52nugggqn3fuligbvtucr2uaxdf47d/spack-src' >> 14151 make: *** [all] Error 2 I’m attaching config.log and all the spack logs (build-env, build-logs and config-args). Thanks in advance for the help and hope everyone is staying safe from the virus! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] In-order messages
by Thakur, Rajeev 16 Mar '20

16 Mar '20

Yes, the probes should return in order. The actual data transfer may complete out of order. Rajeev From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, March 16, 2020 at 6:03 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Sorry, a typo. I meant “no more tag=0 messages is coming”. -- Hui Zhou From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, March 16, 2020 at 6:01 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Experts out there, feel free to point out to me if my answer is wrong -- Given the scenario described, my understanding leads to the answer to be affirmative. That is, if you got `tag=1` from the anytag probe, then you can be assured that no more `tag=1` messages that haven’t been recv’ed is coming. -- Hui Zhou From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff

1 0

Re: [mpich-discuss] In-order messages
by Zhou, Hui 16 Mar '20

16 Mar '20

Sorry, a typo. I meant “no more tag=0 messages is coming”. -- Hui Zhou From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, March 16, 2020 at 6:01 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Experts out there, feel free to point out to me if my answer is wrong -- Given the scenario described, my understanding leads to the answer to be affirmative. That is, if you got `tag=1` from the anytag probe, then you can be assured that no more `tag=1` messages that haven’t been recv’ed is coming. -- Hui Zhou From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff

1 0

Re: [mpich-discuss] In-order messages
by Zhou, Hui 16 Mar '20

16 Mar '20

Experts out there, feel free to point out to me if my answer is wrong -- Given the scenario described, my understanding leads to the answer to be affirmative. That is, if you got `tag=1` from the anytag probe, then you can be assured that no more `tag=1` messages that haven’t been recv’ed is coming. -- Hui Zhou From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff

1 0

Re: [mpich-discuss] In-order messages
by Thakur, Rajeev 12 Mar '20

12 Mar '20

The paragraph in Sec 3.5 (reproduced below) talks about the case when two sends match one receive and another case where two receives match one send. In the first case, the receive can get data only from the first send. In the second case, only the first receive can get the data from the send. That is what is meant by non-overtaking. “Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending. If a receiver posts two receives in succession, and both match the same message, then the second receive operation cannot be satisfied by this message, if the first one is still pending. This requirement facilitates matching of sends to receives.” From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Thursday, March 12, 2020 at 4:09 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Paraphrasing the discussion so far, my understanding is: Note: I am only thinking of the case where both messages have the same tag and go to the same destination processor. Section 3.5 of the MPI standard mentions: "Messages are non-overtaking: " I think this must mean that once the messages are in the receive buffer one cannot overtake the one infront. It is trying to make the point that your MPI_Recv cannot reach over message 1 and say I want message 2 (the buffer is FIFO). But this is only in the receive buffer. However, the order the messages are received into the buffer (from the network) cannot be guaranteed to be the same as the order they were sent. Is this correct? Many Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Thursday, March 12, 2020 1:18 PM To: Larson, Jeffrey M. <jmlarson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Navarro, John-Luke Nicolas <jnavarro(a)anl.gov>; Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Matching specifies which buffer the incoming message goes into. Two sends issued in order with the same tag and destination will match two receives issued in order with the same tag or MPI_ANY_TAG at the destination. The two sends could take different paths through the network, one more congested than the other, and hence reach the destination out of order. But they will get placed in the intended buffers. Rajeev From: "Larson, Jeffrey M." <jmlarson(a)anl.gov> Date: Thursday, March 12, 2020 at 1:12 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages On 3/12/20 1:04 PM, Thakur, Rajeev wrote: The standard specifies the order in which messages match, not complete. They can complete in any order. Rajeev From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org><mailto:[email protected]> Reply-To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov><mailto:[email protected]>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov><mailto:[email protected]>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov><mailto:[email protected]> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff Hi Rajeev, What is the difference between matching and completing? Does this mean that the quote from the standard that I gave means the messages between two fixed ranks could overtake each other? That is, they "match" but don't "complete" in the same order?

1 0

Re: [mpich-discuss] In-order messages
by Thakur, Rajeev 12 Mar '20

12 Mar '20

Matching specifies which buffer the incoming message goes into. Two sends issued in order with the same tag and destination will match two receives issued in order with the same tag or MPI_ANY_TAG at the destination. The two sends could take different paths through the network, one more congested than the other, and hence reach the destination out of order. But they will get placed in the intended buffers. Rajeev From: "Larson, Jeffrey M." <jmlarson(a)anl.gov> Date: Thursday, March 12, 2020 at 1:12 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages On 3/12/20 1:04 PM, Thakur, Rajeev wrote: The standard specifies the order in which messages match, not complete. They can complete in any order. Rajeev From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org><mailto:[email protected]> Reply-To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov><mailto:[email protected]>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov><mailto:[email protected]>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov><mailto:[email protected]> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff Hi Rajeev, What is the difference between matching and completing? Does this mean that the quote from the standard that I gave means the messages between two fixed ranks could overtake each other? That is, they "match" but don't "complete" in the same order?

1 0

Re: [mpich-discuss] In-order messages
by Joachim Protze 12 Mar '20

12 Mar '20

Am 12.03.20 um 19:12 schrieb Larson, Jeffrey M. via discuss: > On 3/12/20 1:04 PM, Thakur, Rajeev wrote: > The standard specifies the order in which messages match, not complete. They can complete in any order. > > Rajeev > > > From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org><mailto:[email protected]> > Reply-To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> > Date: Thursday, March 12, 2020 at 12:46 PM > To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> > Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov><mailto:[email protected]>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov><mailto:[email protected]>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov><mailto:[email protected]> > Subject: [mpich-discuss] In-order messages > > Hello MPICH friends, > > Consider the simple two-rank MPI scenario: > > * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. > * When rank 1 is finished, it will send it's last data (or no data) with tag=1. > * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. > Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? > > Looking at section 3.5 of the MPI standard lets me know that > > "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." > > But I'm not sure if this applies to the above case. Is anytag "the same receive"? > If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? > While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? > > Thank you for your help, > Jeff > > Hi Rajeev, > > What is the difference between matching and completing? > > Does this mean that the quote from the standard that I gave means the messages between two fixed ranks could overtake each other? That is, they "match" but don't "complete" in the same order? > Hi Jeff, can you probably fix your mail client to distinguish quotes from new text? Matching is the process of finding send-receive pairs. Once a message is matched, it cannot be received by a different function. This is like answering a call, without having said anything yet. Completion is when the transfer is over and you end the call. Example 3.5 in the standard should show your use case. - Joachim > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0