- discuss - mailman3.cels.anl.gov

MPICH INSTALLATION ERRORS
by Mbaabu Brian 04 Apr '20

04 Apr '20

Hello, My installation produced such errors as are documented in the attached log files. Please help me resolve them.

1 0

Re: [mpich-discuss] In-order messages
by Thakur, Rajeev 23 Mar '20

23 Mar '20

The MPI implementation has to ensure that the probe will not return with message B first. It could use sequence numbers internally, for example. From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Monday, March 23, 2020 at 3:04 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Is it undefined where the matching happens? So the question is: Process 1 sends message A and then message B, both to process 0. message B travels faster through the network, and is received first by process 0. At this point an iprobe on process 0 occurs with MPI_TAG_ANY. My thought is that it finds messageB (which happens to say - ALL FINISHED). A follow up MPI_recv will say all is finished. Is this possible? I've assumed here that the probe is local and so the matching with message A has not yet happened. The given snippet tells me that a probe will return if there is a matching send, but in this case there is two matching sends. ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, March 23, 2020 2:41 PM To: Hudson, Stephen Tobias P <shudson(a)anl.gov>; Larson, Jeffrey M. <jmlarson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Navarro, John-Luke Nicolas <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Pending means not already matched with another message. This is what the standard says about probe: “The MPI implementation of MPI_PROBE and MPI_IPROBE needs to guarantee progress: if a call to MPI_PROBE has been issued by a process, and a send that matches the probe has been initiated by some process, then the call to MPI_PROBE will return, unless the message is received by another concurrent receive operation (that is executed by another thread at the probing process). Similarly, if a process busy waits with MPI_IPROBE and a matching message has been issued, then the call to MPI_IPROBE will eventually return flag = true unless the message is received by another concurrent receive operation or matched by a concurrent matched probe.” Eager protocol or any other protocol have to support the semantics specified by the standard. From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Monday, March 23, 2020 at 2:27 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages The point where I am still a bit unclear is whether "pending" refers to something at the sender's end or in some buffer at the receiver's end. Perhaps an answer to this will help me. Is mpi_probe/iprobe a local operation or does it actually communicate with the sender? Also, what effect does the 'eager' protocal have on this? ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Thursday, March 12, 2020 6:30 PM To: Hudson, Stephen Tobias P <shudson(a)anl.gov>; Larson, Jeffrey M. <jmlarson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Navarro, John-Luke Nicolas <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages The paragraph in Sec 3.5 (reproduced below) talks about the case when two sends match one receive and another case where two receives match one send. In the first case, the receive can get data only from the first send. In the second case, only the first receive can get the data from the send. That is what is meant by non-overtaking. “Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending. If a receiver posts two receives in succession, and both match the same message, then the second receive operation cannot be satisfied by this message, if the first one is still pending. This requirement facilitates matching of sends to receives.” From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Thursday, March 12, 2020 at 4:09 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Paraphrasing the discussion so far, my understanding is: Note: I am only thinking of the case where both messages have the same tag and go to the same destination processor. Section 3.5 of the MPI standard mentions: "Messages are non-overtaking: " I think this must mean that once the messages are in the receive buffer one cannot overtake the one infront. It is trying to make the point that your MPI_Recv cannot reach over message 1 and say I want message 2 (the buffer is FIFO). But this is only in the receive buffer. However, the order the messages are received into the buffer (from the network) cannot be guaranteed to be the same as the order they were sent. Is this correct? Many Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Thursday, March 12, 2020 1:18 PM To: Larson, Jeffrey M. <jmlarson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Navarro, John-Luke Nicolas <jnavarro(a)anl.gov>; Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Matching specifies which buffer the incoming message goes into. Two sends issued in order with the same tag and destination will match two receives issued in order with the same tag or MPI_ANY_TAG at the destination. The two sends could take different paths through the network, one more congested than the other, and hence reach the destination out of order. But they will get placed in the intended buffers. Rajeev From: "Larson, Jeffrey M." <jmlarson(a)anl.gov> Date: Thursday, March 12, 2020 at 1:12 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages On 3/12/20 1:04 PM, Thakur, Rajeev wrote: The standard specifies the order in which messages match, not complete. They can complete in any order. Rajeev From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org><mailto:[email protected]> Reply-To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov><mailto:[email protected]>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov><mailto:[email protected]>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov><mailto:[email protected]> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff Hi Rajeev, What is the difference between matching and completing? Does this mean that the quote from the standard that I gave means the messages between two fixed ranks could overtake each other? That is, they "match" but don't "complete" in the same order?

1 0

Re: [mpich-discuss] In-order messages
by Thakur, Rajeev 23 Mar '20

23 Mar '20

Pending means not already matched with another message. This is what the standard says about probe: “The MPI implementation of MPI_PROBE and MPI_IPROBE needs to guarantee progress: if a call to MPI_PROBE has been issued by a process, and a send that matches the probe has been initiated by some process, then the call to MPI_PROBE will return, unless the message is received by another concurrent receive operation (that is executed by another thread at the probing process). Similarly, if a process busy waits with MPI_IPROBE and a matching message has been issued, then the call to MPI_IPROBE will eventually return flag = true unless the message is received by another concurrent receive operation or matched by a concurrent matched probe.” Eager protocol or any other protocol have to support the semantics specified by the standard. From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Monday, March 23, 2020 at 2:27 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages The point where I am still a bit unclear is whether "pending" refers to something at the sender's end or in some buffer at the receiver's end. Perhaps an answer to this will help me. Is mpi_probe/iprobe a local operation or does it actually communicate with the sender? Also, what effect does the 'eager' protocal have on this? ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Thursday, March 12, 2020 6:30 PM To: Hudson, Stephen Tobias P <shudson(a)anl.gov>; Larson, Jeffrey M. <jmlarson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Navarro, John-Luke Nicolas <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages The paragraph in Sec 3.5 (reproduced below) talks about the case when two sends match one receive and another case where two receives match one send. In the first case, the receive can get data only from the first send. In the second case, only the first receive can get the data from the send. That is what is meant by non-overtaking. “Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending. If a receiver posts two receives in succession, and both match the same message, then the second receive operation cannot be satisfied by this message, if the first one is still pending. This requirement facilitates matching of sends to receives.” From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Thursday, March 12, 2020 at 4:09 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Paraphrasing the discussion so far, my understanding is: Note: I am only thinking of the case where both messages have the same tag and go to the same destination processor. Section 3.5 of the MPI standard mentions: "Messages are non-overtaking: " I think this must mean that once the messages are in the receive buffer one cannot overtake the one infront. It is trying to make the point that your MPI_Recv cannot reach over message 1 and say I want message 2 (the buffer is FIFO). But this is only in the receive buffer. However, the order the messages are received into the buffer (from the network) cannot be guaranteed to be the same as the order they were sent. Is this correct? Many Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Thursday, March 12, 2020 1:18 PM To: Larson, Jeffrey M. <jmlarson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Navarro, John-Luke Nicolas <jnavarro(a)anl.gov>; Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Matching specifies which buffer the incoming message goes into. Two sends issued in order with the same tag and destination will match two receives issued in order with the same tag or MPI_ANY_TAG at the destination. The two sends could take different paths through the network, one more congested than the other, and hence reach the destination out of order. But they will get placed in the intended buffers. Rajeev From: "Larson, Jeffrey M." <jmlarson(a)anl.gov> Date: Thursday, March 12, 2020 at 1:12 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages On 3/12/20 1:04 PM, Thakur, Rajeev wrote: The standard specifies the order in which messages match, not complete. They can complete in any order. Rajeev From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org><mailto:[email protected]> Reply-To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org"<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov><mailto:[email protected]>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov><mailto:[email protected]>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov><mailto:[email protected]> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff Hi Rajeev, What is the difference between matching and completing? Does this mean that the quote from the standard that I gave means the messages between two fixed ranks could overtake each other? That is, they "match" but don't "complete" in the same order?

1 0

Re: [mpich-discuss] In-order messages
by Thakur, Rajeev 23 Mar '20

23 Mar '20

I was referring to the case of nonblocking receives. Blocking receives will sequentialize the receives. From: "Larson, Jeffrey M." <jmlarson(a)anl.gov> Date: Monday, March 23, 2020 at 9:07 AM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages If rank 0 is doing the probing, and rank 0 is doing a blocking receive immediately after a probe says rank 1 is waiting to send, then the fact that the probes return in order, means the data will complete in order, right? ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, March 16, 2020 6:38 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov>; Larson, Jeffrey M. <jmlarson(a)anl.gov>; Hudson, Stephen Tobias P <shudson(a)anl.gov>; Navarro, John-Luke Nicolas <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Yes, the probes should return in order. The actual data transfer may complete out of order. Rajeev From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, March 16, 2020 at 6:03 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Sorry, a typo. I meant “no more tag=0 messages is coming”. -- Hui Zhou From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, March 16, 2020 at 6:01 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Experts out there, feel free to point out to me if my answer is wrong -- Given the scenario described, my understanding leads to the answer to be affirmative. That is, if you got `tag=1` from the anytag probe, then you can be assured that no more `tag=1` messages that haven’t been recv’ed is coming. -- Hui Zhou From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff

1 0

Re: [mpich-discuss] In-order messages
by Larson, Jeffrey M. 23 Mar '20

23 Mar '20

If rank 0 is doing the probing, and rank 0 is doing a blocking receive immediately after a probe says rank 1 is waiting to send, then the fact that the probes return in order, means the data will complete in order, right? ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, March 16, 2020 6:38 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov>; Larson, Jeffrey M. <jmlarson(a)anl.gov>; Hudson, Stephen Tobias P <shudson(a)anl.gov>; Navarro, John-Luke Nicolas <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Yes, the probes should return in order. The actual data transfer may complete out of order. Rajeev From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, March 16, 2020 at 6:03 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Sorry, a typo. I meant “no more tag=0 messages is coming”. -- Hui Zhou From: "Zhou, Hui via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, March 16, 2020 at 6:01 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov> Subject: Re: [mpich-discuss] In-order messages Experts out there, feel free to point out to me if my answer is wrong -- Given the scenario described, my understanding leads to the answer to be affirmative. That is, if you got `tag=1` from the anytag probe, then you can be assured that no more `tag=1` messages that haven’t been recv’ed is coming. -- Hui Zhou From: "Larson, Jeffrey M. via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, March 12, 2020 at 12:46 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Larson, Jeffrey M." <jmlarson(a)anl.gov>, "Navarro, John-Luke Nicolas" <jnavarro(a)anl.gov>, "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] In-order messages Hello MPICH friends, Consider the simple two-rank MPI scenario: * Rank 1 is doing calculations and giving chunks of data to rank 0 using nonblocking sends and tag=0. * When rank 1 is finished, it will send it's last data (or no data) with tag=1. * Rank 0 is using probes to see when data is ready to be received. It receives with any tag, knowing when to stop receiving, or give new data when a tag=0 is received. Is it possible that rank 0 receives a tag=1 message when there are outstanding tag=0 messages? Looking at section 3.5 of the MPI standard lets me know that "Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending." But I'm not sure if this applies to the above case. Is anytag "the same receive"? If rank 1 puts data in its buffer, doesn't the network have to be used to communicate that to the buffer of rank 0? While rank 1 is putting data into its buffer in order, is it possible that a tiny tag=1 message is registered in the rank 0 buffer before a massive tag=0 message? Thank you for your help, Jeff

1 0

Re: [mpich-discuss] MPICH build errors, cannot find -levent(_pthreads)
by Sajid Ali 18 Mar '20

18 Mar '20

Hi Hui, I tried setting LIBRARY_PATH to /lib64 but that didn’t help (neither did setting it /usr). I’m attaching the build logs again (the build-env logs do indeed confirm that LIBRARY_PATH was set). I tried building MPICH against my own pmix library and everything works. (But this pmix is local to my home directory and not linked to slurm). Inspecting the build-env log from the successful build, I see that the libevent’s location is absent from LIBRARY_PATH but is present in the SPACK_INCLUDE/LINK/RPATH_DIRS (and also in pkg-config/cmake paths which I’m guessing isn’t relevant here) which are used by spack’s compiler wrapper to ensure correct include/link/rpath settings. My task right now is to test the installation of mpi libraries with pmix/ucx on a small test cluster before we deploy it on the production cluster at my campus so it’s not of the highest concern to me that MPICH won’t link to a pmix library that doesn’t have rpaths enabled. I’ll ask the sysadmins to rebuild pmix with rpath support and try if that works. If it does, I’ll reply to this thread either asking for help again or indicating that it succeeds (whereby it can (hopefully) serve as a reference for others facing the same issue in the future). Thank You, Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] MPICH build errors, cannot find -levent(_pthreads)
by Zhou, Hui 17 Mar '20

17 Mar '20

Hi Sajid, So does adding /lib64 to your `LIBRARY_PATH` help? -- Hui Zhou From: Sajid Ali <sajidsyed2021(a)u.northwestern.edu> Date: Tuesday, March 17, 2020 at 3:09 PM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH build errors, cannot find -levent(_pthreads) Hi Hui, libevent(-2.0.so.5) is at available at /lib64. pmix depends on libevent as per the spack recipe<https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/package…>. Checking for libpmix linkage I do see that it is linked to libevent : [sas4990@quser10 lib64]$ pwd /usr/local/pmix-3.1.5/lib64 [sas4990@quser10 lib64]$ ldd libpmix.so.2.2.25 linux-vdso.so.1 => (0x00007fffbfbe7000) libm.so.6 => /lib64/libm.so.6 (0x00002b413ffb3000) libutil.so.1 => /lib64/libutil.so.1 (0x00002b41402b5000) libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00002b41404b8000) libevent_pthreads-2.0.so.5 => /lib64/libevent_pthreads-2.0.so.5 (0x00002b4140700000) libz.so.1 => /home/sas4990/packages/spack/opt/spack/linux-rhel7-ivybridge/gcc-4.8.5/zlib-1.2.11-dy4p76tgbhwcznwvv7rcagzfcbxyhsbo/lib/libz.so.1 (0x00002b4140903000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b4140b1a000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b4140d1e000) libc.so.6 => /lib64/libc.so.6 (0x00002b4140f3a000) /lib64/ld-linux-x86-64.so.2 (0x00002b413fa75000) [sas4990@quser10 lib64]$ I’m confused as to why libtool is unable to read use the dependency listing of pmix to find libevent. readelf tells me that libpmix doesn’t have rpaths set for it’s dependencies, could this be the cause of the linker being unable to find libevent ? Thank You Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io<http://s-sajid-ali.github.io>

1 0

Re: [mpich-discuss] MPICH build errors, cannot find -levent(_pthreads)
by Sajid Ali 17 Mar '20

17 Mar '20

Hi Hui, libevent(-2.0.so.5) is at available at /lib64. pmix depends on libevent as per the spack recipe <https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/package…>. Checking for libpmix linkage I do see that it is linked to libevent : [sas4990@quser10 lib64]$ pwd /usr/local/pmix-3.1.5/lib64 [sas4990@quser10 lib64]$ ldd libpmix.so.2.2.25 linux-vdso.so.1 => (0x00007fffbfbe7000) libm.so.6 => /lib64/libm.so.6 (0x00002b413ffb3000) libutil.so.1 => /lib64/libutil.so.1 (0x00002b41402b5000) libevent-2.0.so.5 => /lib64/libevent-2.0.so.5 (0x00002b41404b8000) libevent_pthreads-2.0.so.5 => /lib64/libevent_pthreads-2.0.so.5 (0x00002b4140700000) libz.so.1 => /home/sas4990/packages/spack/opt/spack/linux-rhel7-ivybridge/gcc-4.8.5/zlib-1.2.11-dy4p76tgbhwcznwvv7rcagzfcbxyhsbo/lib/libz.so.1 (0x00002b4140903000) libdl.so.2 => /lib64/libdl.so.2 (0x00002b4140b1a000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b4140d1e000) libc.so.6 => /lib64/libc.so.6 (0x00002b4140f3a000) /lib64/ld-linux-x86-64.so.2 (0x00002b413fa75000) [sas4990@quser10 lib64]$ I’m confused as to why libtool is unable to read use the dependency listing of pmix to find libevent. readelf tells me that libpmix doesn’t have rpaths set for it’s dependencies, could this be the cause of the linker being unable to find libevent ? Thank You Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] MPICH build errors, cannot find -levent(_pthreads)
by Zhou, Hui 17 Mar '20

17 Mar '20

* >> 14144 /usr/bin/ld: cannot find -levent * >> 14145 /usr/bin/ld: cannot find -levent_pthreads * Try find where is `libevent` installed and add it to `LIBRARY_PATH`. I think this dependency is pulled in from libtool library, probably one of the ucx dependency. Usually the libtool package should contain the `-L…` for the library. But the breakage will happen if one installs the library to a different paths than the one specified during configure. These (libtool) magic are wonderful when it works, but so annoyingly difficult to address when it doesn’t 😊 -- Hui Zhou From: Sajid Ali via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, March 17, 2020 at 10:50 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Syed, Sajid Ali" <sajidsyed2021(a)u.northwestern.edu> Subject: [mpich-discuss] MPICH build errors, cannot find -levent(_pthreads) Hi MPICH-developers, I’m seeing a build error for mpich-3.3.2 on a cluster using gcc-9.3.0. I tried installing mpich using spack while linking against ucx and pmix installed by sysadmins at /usr/local. The error is as follows : >> 14144 /usr/bin/ld: cannot find -levent >> 14145 /usr/bin/ld: cannot find -levent_pthreads >> 14146 collect2: error: ld returned 1 exit status >> 14147 make[2]: *** [lib/libmpi.la<http://libmpi.la>] Error 1 14148 make[2]: Leaving directory `/tmp/sas4990/spack-stage/spack-stage- mpich-3.3.2-no52nugggqn3fuligbvtucr2uaxdf47d/spack-src' >> 14149 make[1]: *** [all-recursive] Error 1 14150 make[1]: Leaving directory `/tmp/sas4990/spack-stage/spack-stage- mpich-3.3.2-no52nugggqn3fuligbvtucr2uaxdf47d/spack-src' >> 14151 make: *** [all] Error 2 I’m attaching all the spack logs (build-env, build-out and config-args). Let me know if there is any log file that I can send to help debug the issue. Thanks in advance for the help and hope everyone is staying safe from the virus! PS: I sent this email with config.log before but it didn't go through, hence I'm sending it again without it. Please feel free to delete/reject the first one. -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io<http://s-sajid-ali.github.io>

1 0

MPICH build errors, cannot find -levent(_pthreads)
by Sajid Ali 17 Mar '20

17 Mar '20

Hi MPICH-developers, I’m seeing a build error for mpich-3.3.2 on a cluster using gcc-9.3.0. I tried installing mpich using spack while linking against ucx and pmix installed by sysadmins at /usr/local. The error is as follows : >> 14144 /usr/bin/ld: cannot find -levent >> 14145 /usr/bin/ld: cannot find -levent_pthreads >> 14146 collect2: error: ld returned 1 exit status >> 14147 make[2]: *** [lib/libmpi.la] Error 1 14148 make[2]: Leaving directory `/tmp/sas4990/spack-stage/spack-stage- mpich-3.3.2-no52nugggqn3fuligbvtucr2uaxdf47d/spack-src' >> 14149 make[1]: *** [all-recursive] Error 1 14150 make[1]: Leaving directory `/tmp/sas4990/spack-stage/spack-stage- mpich-3.3.2-no52nugggqn3fuligbvtucr2uaxdf47d/spack-src' >> 14151 make: *** [all] Error 2 I’m attaching all the spack logs (build-env, build-out and config-args). Let me know if there is any log file that I can send to help debug the issue. Thanks in advance for the help and hope everyone is staying safe from the virus! PS: I sent this email with config.log before but it didn't go through, hence I'm sending it again without it. Please feel free to delete/reject the first one. -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0