- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] How to configure mpich without cuda
by David Alexander 13 Jul '21

13 Jul '21

The “—without-cuda" option causes the compile to fail (where without it the compile succeeds)... [qar@qacentos7 shared]$ make 2>&1 | tee build.txt make all-recursive make[1]: Entering directory `/scr_qacentos7/qar/jenkins/workspace/txliball/n/qacentos7/builds-v/mpich-3.4.1/shared' Making all in src/mpl make[2]: Entering directory `/scr_qacentos7/qar/jenkins/workspace/txliball/n/qacentos7/builds-v/mpich-3.4.1/shared/src/mpl' CCLD libmpl.la ./libtool: line 7472: cd: no/lib: No such file or directory libtool: error: cannot determine absolute directory name of 'no/lib' make[2]: *** [libmpl.la] Error 1 make[2]: Leaving directory `/scr_qacentos7/qar/jenkins/workspace/txliball/n/qacentos7/builds-v/mpich-3.4.1/shared/src/mpl' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/scr_qacentos7/qar/jenkins/workspace/txliball/n/qacentos7/builds-v/mpich-3.4.1/shared' make: *** [all] Error 2 > On Jul 12, 2021, at 8:24 PM, Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi John, > > Could you try configure mpich with `--without-cuda`? > From: John Cary via discuss <discuss(a)mpich.org> > Sent: Monday, July 12, 2021 6:27 PM > To: discuss(a)mpich.org <discuss(a)mpich.org>; David Alexander <alexanda(a)txcorp.com> > Cc: John Cary <cary(a)colorado.edu> > Subject: [mpich-discuss] How to configure mpich without cuda > > > Right now it picks up cuda, which links libcuda.so into mpiexec. Is there > any way to prevent this? > > Thx....John Cary > _______________________________________________ > discuss mailing list discuss(a)mpich.org <mailto:[email protected]> > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss>

1 0

Re: [mpich-discuss] How to configure mpich without cuda
by Zhou, Hui 12 Jul '21

12 Jul '21

Hi John, Could you try configure mpich with `--without-cuda`? ________________________________ From: John Cary via discuss <discuss(a)mpich.org> Sent: Monday, July 12, 2021 6:27 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; David Alexander <alexanda(a)txcorp.com> Cc: John Cary <cary(a)colorado.edu> Subject: [mpich-discuss] How to configure mpich without cuda Right now it picks up cuda, which links libcuda.so into mpiexec. Is there any way to prevent this? Thx....John Cary _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

How to configure mpich without cuda
by John Cary 12 Jul '21

12 Jul '21

Right now it picks up cuda, which links libcuda.so into mpiexec.ï¿½ Is there any way to prevent this? Thx....John Cary

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Raffenetti, Kenneth J. 30 Jun '21

30 Jun '21

MPICH might make that eager/rndv decision, or it might delegate to one or more underlying communication libraries depending on configuration. It's probably best to avoid trying to manipulate it. Rajeev is right that what you observe is not necessarily because of rendezvous. The message envelope is transferred the same either way. It has to for match ordering purposes. In Iprobe, we first check if there is any message available matching the message envelope, and return flag=true if it does. If not, we try to make progress. We skip progress in the flag=true case so that Iprobe can be lightweight when allowed. I suppose MPICH could check for the message again *after* making progress (but before returning to the user), but we've so far assumed let the application deal with it. Ken On 6/29/21, 11:42 AM, "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> wrote: Okay, just to check, is there a way to tell it to use eager (or eager threshold), or does MPICH just make that decision? Thanks, Steve ________________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Tuesday, June 29, 2021 11:31 AM To: Hudson, Stephen Tobias P <shudson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. It shouldn’t matter whether it is eager or rendezvous. The envelope in the rendezvous contains the message matching information. Iprobe will initiate progress, so the next call to Iprobe should find a message if there is one. Rajeev From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Tuesday, June 29, 2021 at 11:27 AM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Hudson, Stephen Tobias P 29 Jun '21

29 Jun '21

Okay, just to check, is there a way to tell it to use eager (or eager threshold), or does MPICH just make that decision? Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Tuesday, June 29, 2021 11:31 AM To: Hudson, Stephen Tobias P <shudson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. It shouldn’t matter whether it is eager or rendezvous. The envelope in the rendezvous contains the message matching information. Iprobe will initiate progress, so the next call to Iprobe should find a message if there is one. Rajeev From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Tuesday, June 29, 2021 at 11:27 AM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Thakur, Rajeev 29 Jun '21

29 Jun '21

It shouldn’t matter whether it is eager or rendezvous. The envelope in the rendezvous contains the message matching information. Iprobe will initiate progress, so the next call to Iprobe should find a message if there is one. Rajeev From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Tuesday, June 29, 2021 at 11:27 AM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Hudson, Stephen Tobias P 29 Jun '21

29 Jun '21

Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Thakur, Rajeev 28 Jun '21

28 Jun '21

I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Iprobe does not find all messages.
by Hudson, Stephen Tobias P 28 Jun '21

28 Jun '21

Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] libfabric+psm2 performance
by Antonio J. Peña 23 Jun '21

23 Jun '21

Thank you folks. I'd have loved to get an older libfabric, but an oldish kernel version at MN4 prevented me from compiling a more recent one. I'll give your suggestions a try once I get the chance to spend some more time on this. Toni El 16/6/21 a las 21:47, Zhou, Hui escribió: > > Yeah, as Wesley suggested, it will shed more light if you try set the > environment variable MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG and get > some diagnostics. We haven’t tested against such old libfabric for a > while so some discrepancy in the capability set may have crept in. > > -- > Hui Zhou > > *From: *Bland, Wesley via discuss <discuss(a)mpich.org> > *Date: *Wednesday, June 16, 2021 at 8:47 AM > *To: *Antonio J. Peña <antonio.pena(a)bsc.es> > *Cc: *Bland, Wesley <wesley.bland(a)intel.com>, MPICH Discuss > <discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] libfabric+psm2 performance > > If you set the environment variable > MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG, it will print out the > capability set (and provider) that MPICH is using so you can confirm. > -- Antonio J. Peña (PhD) Team Lead, Accelerators and Communications for HPC | Teaching and Research Staff Sr. Researcher, Computer Sciences Department | Computer Architecture Department Barcelona Supercomputing Center (BSC) | Universitat Politècnica de Catalunya (UPC) http://www.bsc.es/pena-antonio =============================== Looking for job opportunities? Open positions in my team. Please contact me.

1 0