June 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Iprobe does not find all messages.
by Raffenetti, Kenneth J. 30 Jun '21

30 Jun '21

MPICH might make that eager/rndv decision, or it might delegate to one or more underlying communication libraries depending on configuration. It's probably best to avoid trying to manipulate it. Rajeev is right that what you observe is not necessarily because of rendezvous. The message envelope is transferred the same either way. It has to for match ordering purposes. In Iprobe, we first check if there is any message available matching the message envelope, and return flag=true if it does. If not, we try to make progress. We skip progress in the flag=true case so that Iprobe can be lightweight when allowed. I suppose MPICH could check for the message again *after* making progress (but before returning to the user), but we've so far assumed let the application deal with it. Ken On 6/29/21, 11:42 AM, "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> wrote: Okay, just to check, is there a way to tell it to use eager (or eager threshold), or does MPICH just make that decision? Thanks, Steve ________________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Tuesday, June 29, 2021 11:31 AM To: Hudson, Stephen Tobias P <shudson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. It shouldn’t matter whether it is eager or rendezvous. The envelope in the rendezvous contains the message matching information. Iprobe will initiate progress, so the next call to Iprobe should find a message if there is one. Rajeev From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Tuesday, June 29, 2021 at 11:27 AM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Hudson, Stephen Tobias P 29 Jun '21

29 Jun '21

Okay, just to check, is there a way to tell it to use eager (or eager threshold), or does MPICH just make that decision? Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Tuesday, June 29, 2021 11:31 AM To: Hudson, Stephen Tobias P <shudson(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. It shouldn’t matter whether it is eager or rendezvous. The envelope in the rendezvous contains the message matching information. Iprobe will initiate progress, so the next call to Iprobe should find a message if there is one. Rajeev From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Tuesday, June 29, 2021 at 11:27 AM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Thakur, Rajeev 29 Jun '21

29 Jun '21

It shouldn’t matter whether it is eager or rendezvous. The envelope in the rendezvous contains the message matching information. Iprobe will initiate progress, so the next call to Iprobe should find a message if there is one. Rajeev From: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Date: Tuesday, June 29, 2021 at 11:27 AM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Iprobe does not find all messages. Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Hudson, Stephen Tobias P 29 Jun '21

29 Jun '21

Thanks Rajeev, is this going to be using rendezvous protocol, in which case the receiver has to receive an envelope and ping back. In that case, it may make sense to me? Does mpich have a way of controlling message sizes for eager/rendezvous as intel MPI does? I havn't found any environment variables. Thanks, Steve ________________________________ From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Monday, June 28, 2021 9:13 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Hudson, Stephen Tobias P <shudson(a)anl.gov> Subject: Re: [mpich-discuss] Iprobe does not find all messages. I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] Iprobe does not find all messages.
by Thakur, Rajeev 28 Jun '21

28 Jun '21

I guess the reason is that iprobe does not find a message until some MPI function causes progress on the communication. Just a delay will not cause progress. Rajeev From: "Hudson, Stephen Tobias P via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, June 28, 2021 at 4:30 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Hudson, Stephen Tobias P" <shudson(a)anl.gov> Subject: [mpich-discuss] Iprobe does not find all messages. Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Iprobe does not find all messages.
by Hudson, Stephen Tobias P 28 Jun '21

28 Jun '21

Hi, TLDR: Why does iprobe not pick up messages that should be there? In this use-case, a manager is calling iprobe (via mpi4py) to look for messages received from various workers. In the receive function (1.), there is a loop over workers doing an iprobe for each worker, and only if it loops round them all receiving nothing does it come out. Each worker may have sent several messages. loop: 1. receive_func (loop over all workers doing iprobe - until none have anything) 2. do some other stuff. (actual code: https://github.com/Libensemble/libensemble/blob/main/libensemble/manager.py…) The workers are sending with isend, and I know they have passed the isend call for all the required messages. Manager side, the iprobe loop does not necessarily pick up all messages in one call to receive_func (but if I replace each iprobe call with its own loop of several iprobes it does). In my code, the iprobe is part of a loop which may occasionally spend some time in another function (2.), so I want to better understand why it may not pick up all the messages. Even if I put a delay of a few seconds in (2), it still has cycles that do not pick up the messages - as though a certain number of iprobes are required. In particular, I want to understand whether the messages are present on the receiving end, maybe in the unexpected message buffer, but may not get picked up by the iprobe? Or is there some delay on the sending end (I know from timing the code has passed the isend for all messages). Also, as adding several iprobes (in their own sub-loop) always finds the messages, if the delay is not receiver side, then it must be doing some handshake? I have seen answers online, which suggest that this is expected behavior, but I wonder if there is a fairly simple explanation as to why iprobe does not pick everything up? If its too complicated, then okay. I am running on a Ubuntu laptop with $ mpirun --version HYDRA build details: Version: 3.2.1 Thanks, Steve

1 0

Re: [mpich-discuss] libfabric+psm2 performance
by Antonio J. Peña 23 Jun '21

23 Jun '21

Thank you folks. I'd have loved to get an older libfabric, but an oldish kernel version at MN4 prevented me from compiling a more recent one. I'll give your suggestions a try once I get the chance to spend some more time on this. Toni El 16/6/21 a las 21:47, Zhou, Hui escribió: > > Yeah, as Wesley suggested, it will shed more light if you try set the > environment variable MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG and get > some diagnostics. We haven’t tested against such old libfabric for a > while so some discrepancy in the capability set may have crept in. > > -- > Hui Zhou > > *From: *Bland, Wesley via discuss <discuss(a)mpich.org> > *Date: *Wednesday, June 16, 2021 at 8:47 AM > *To: *Antonio J. Peña <antonio.pena(a)bsc.es> > *Cc: *Bland, Wesley <wesley.bland(a)intel.com>, MPICH Discuss > <discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] libfabric+psm2 performance > > If you set the environment variable > MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG, it will print out the > capability set (and provider) that MPICH is using so you can confirm. > -- Antonio J. Peña (PhD) Team Lead, Accelerators and Communications for HPC | Teaching and Research Staff Sr. Researcher, Computer Sciences Department | Computer Architecture Department Barcelona Supercomputing Center (BSC) | Universitat Politècnica de Catalunya (UPC) http://www.bsc.es/pena-antonio =============================== Looking for job opportunities? Open positions in my team. Please contact me.

1 0

Re: [mpich-discuss] libfabric+psm2 performance
by Zhou, Hui 16 Jun '21

16 Jun '21

Yeah, as Wesley suggested, it will shed more light if you try set the environment variable MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG and get some diagnostics. We haven’t tested against such old libfabric for a while so some discrepancy in the capability set may have crept in. -- Hui Zhou From: Bland, Wesley via discuss <discuss(a)mpich.org> Date: Wednesday, June 16, 2021 at 8:47 AM To: Antonio J. Peña <antonio.pena(a)bsc.es> Cc: Bland, Wesley <wesley.bland(a)intel.com>, MPICH Discuss <discuss(a)mpich.org> Subject: Re: [mpich-discuss] libfabric+psm2 performance If you set the environment variable MPIR_CVAR_CH4_OFI_CAPABILITY_SETS_DEBUG, it will print out the capability set (and provider) that MPICH is using so you can confirm.

1 0

Re: [mpich-discuss] the MPI daemon triggers an assertion on an ARM-based Linux system
by Zhou, Hui 16 Jun '21

16 Jun '21

I believe this is the same issue as https://github.com/pmodels/mpich/issues/5309. While we are resolving it, you could try the patch mentioned in the issue, or configure with ch4 using the latest release. -- Hui Zhou From: Fabrice Ducos via discuss <discuss(a)mpich.org> Date: Wednesday, June 16, 2021 at 3:54 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Fabrice Ducos <fabrice.ducos(a)univ-lille.fr> Subject: [mpich-discuss] the MPI daemon triggers an assertion on an ARM-based Linux system Greetings, when running an application (atmospheric science) with MPICH on an AWS ARM (not x86) instance (with Linux Ubuntu Server 20.04), our process crashes at the end of the processing. MPICH was installed precompiled with the Ubuntu apt provisioning tool: $ sudo apt install -y mpich $ apt list [only relevant lines displayed for brevity] libmpich-dev/focal,now 3.3.2-2build1 arm64 [residual-config] libmpich12/focal,now 3.3.2-2build1 arm64 [installed,auto-removable] mpich-doc/focal 3.3.2-2build1 all mpich/focal 3.3.2-2build1 arm64 Luckily, we got some debug information in mpid that can be valuable: Assertion failed in file src/mpid/ch3/channels/nemesis/src/ch3_progress.c at line 530: payload_len >= sizeof (MPIDI_CH3_Pkt_t) 0xffff86833f5f ??? ???:0 0xffff86881eef ??? ???:0 0xffff8683793f ??? ???:0 0xffff8686c543 ??? ???:0 0xffff8676d4b3 ??? ???:0 0xffff8637e6eb ??? ???:0 0xffff8637e85b ??? ???:0 0xffff86369093 ??? ???:0 0xaaaac5491e6f ??? ???:0 internal ABORT - process 22 The same application has been used for years with several MPI implementations (MPICH, OpenMPI, Intel MPI) on x86 systems without problem. It was successfully tested with MPICH on Linux Ubuntu Server 20.04 x86 shortly alongside the ARM test. We also tested the application with another MPI implementation (namely, OpenMPI) on the same ARM instance and it did work. We are perfectly fine using another MPI implementation in this specific case, but we thought that this issue would be of some interest to the MPICH maintenance team. Best regards Fabrice Ducos Ingénieur d’études CNRS Laboratoire d’Optique Atmosphérique - UMR CNRS 8518 Faculté des Sciences et Technologies Bâtiment P5 - Bureau 325 Université de Lille - Cité Scientifique 59655 Villeneuve d’Ascq

1 0

Re: [mpich-discuss] mpiexec (MPICH2) launch too slow on Windows10.
by Raffenetti, Kenneth J. 16 Jun '21

16 Jun '21

Hi, We no longer support use of MPICH on Windows. We encourage users to try MS-MPI. https://docs.microsoft.com/en-us/message-passing-interface/microsoft-mpi Ken On 6/15/21, 8:07 PM, "ZJ CHEN via discuss" <discuss(a)mpich.org> wrote: My OS: windows10mpich2: 1.3-32bit and 1.5-64bit I install both versions successfully, and I test the MPIEXEC -N 10 CPI.EXE example. It works, but with a very slow launching duration of about 4 minutes. How can I fix this problem? Thanks for any help! ZHIJUN

1 0