- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Configuration failure for 4.3.1.
by Zhou, Hui 22 Jul '25

22 Jul '25

Hi Kurt, The error is complaining the file `src/tools/perf/rocm/Makefile` is in use while config.status is trying to recreate the file. I am not sure what is causing it. Did you try again and does the same error occur? Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, July 21, 2025 4:19 PM To: Zhou, Hui via discuss <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Configuration failure for 4.3.1. I’m having some trouble configuring 4. 3. 1. The following command finishes successfully: $ .. /mpich-4. 3. 1/configure --prefix=/opt/mpich --with-device=ch4: ofi --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c. txt But this ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I’m having some trouble configuring 4.3.1. The following command finishes successfully: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt But this one fails with the error below: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt config.status: creating src/tools/perf/rocm/Makefile mv: cannot move './confYHXukm/out' to 'src/tools/perf/rocm/Makefile': Device or resource busy config.status: error: could not create src/tools/perf/rocm/Makefile Are my arguments incorrect? I can send c.txt in a separate message. Redhat 4.18.0, gcc 8.5.0. Thanks, Kurt

1 0

Configuration failure for 4.3.1.
by Mccall, Kurt E. (MSFC-EV41) 21 Jul '25

21 Jul '25

I'm having some trouble configuring 4.3.1. The following command finishes successfully: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ofi --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt But this one fails with the error below: $ ../mpich-4.3.1/configure --prefix=/opt/mpich --with-device=ch4:ucx --with-ucx=embedded --with-slurm -enable-debuginfo --enable-g=debug 2>&1 | tee c.txt config.status: creating src/tools/perf/rocm/Makefile mv: cannot move './confYHXukm/out' to 'src/tools/perf/rocm/Makefile': Device or resource busy config.status: error: could not create src/tools/perf/rocm/Makefile Are my arguments incorrect? I can send c.txt in a separate message. Redhat 4.18.0, gcc 8.5.0. Thanks, Kurt

1 0

MPIX Streams Usage
by Ryan Deng 05 Jul '25

05 Jul '25

Hi, I had some questions about how MPIX_Streams work under the hood. I had posted a couple of times on the github issues page before about it but thought this might be a more appropriate place for it. Right now, my code involves many MPIX_Stream_isends/MPIX_Stream_irecvs as processes asynchronously communicate with one another, and I have a separate thread in the background constantly advancing progress on streams. It's my understanding that all operations on a stream must occur serially. I assume this means that things operations like sending and receiving, calling MPIX_Stream_progress/MPI_Wait cannot occur at the same time without locks? Are there any other operations on streams that cannot occur concurrently? In addition, I had also triggered the assertion here<https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/837914f6e8… >. [https://urldefense.us/v3/__https://opengraph.githubassets.com/a68c91e94d3a9… ]<https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/837914f6e8… > mpich/src/mpid/ch4/shm/posix/posix_progress.h at 837914f6e810bbbd5e7ad391bbad9f546ffe3dab · pmodels/mpich<https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/837914f6e8… > Official MPICH Repository. Contribute to pmodels/mpich development by creating an account on GitHub. github.com Does this mean that I cannot send multiple messages at the same time if it's between two different streams? On the github issue here<https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7492__;!… >, Hui Zhou mentioned that it is due to messaging protocol consistency. I'm not quite sure what he means by it and what that means in terms of what I can and cannot do in my application involving MPIX_Streams. Thanks, Ryan

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Zhou, Hui 25 Jun '25

25 Jun '25

Sorry for the misspelling - Martin 🙂 ________________________________ From: Zhou, Hui via discuss <discuss(a)mpich.org> Sent: Wednesday, June 25, 2025 11:46 AM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hi Margin, Just to say that your support and patience are very much appreciated. As for the hcoll issue, while the fix is simple - in fact we had the PR open since a year ago - it was blocked by the testing. We were not aware of of any hcoll users at the time, thus it received a low priority investigating the testing failures. In general, if you or your users are depending on some features and experience reakages, I would encourage you be harsh - more importantly, be persistent - and it's okay to shout from time to time :) . Stale issue needs a shout to raise attention and we need usage feedback to motivate and properly set priorities. Again, thank you for your support and understanding of our resource constraint. -- Hui ________________________________ From: Audet, Martin via discuss <discuss(a)mpich.org> Sent: Wednesday, June 25, 2025 9:03 AM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello Ken, Thanks for the information. I have recompiled mpich 4. 3. 0 and 4. 3. 1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Ken, Thanks for the information. I have recompiled mpich 4.3.0 and 4.3.1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous message. We should thank you for that great piece of software that is mpich. We use it on our clusters since... 1996. Martin Audet ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 24, 2025 2:03 PM To: Audet, Martin; discuss(a)mpich.org Subject: EXT: Re: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!…> Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Zhou, Hui 25 Jun '25

25 Jun '25

Hi Margin, Just to say that your support and patience are very much appreciated. As for the hcoll issue, while the fix is simple - in fact we had the PR open since a year ago - it was blocked by the testing. We were not aware of of any hcoll users at the time, thus it received a low priority investigating the testing failures. In general, if you or your users are depending on some features and experience reakages, I would encourage you be harsh - more importantly, be persistent - and it's okay to shout from time to time :) . Stale issue needs a shout to raise attention and we need usage feedback to motivate and properly set priorities. Again, thank you for your support and understanding of our resource constraint. -- Hui ________________________________ From: Audet, Martin via discuss <discuss(a)mpich.org> Sent: Wednesday, June 25, 2025 9:03 AM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello Ken, Thanks for the information. I have recompiled mpich 4. 3. 0 and 4. 3. 1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Ken, Thanks for the information. I have recompiled mpich 4.3.0 and 4.3.1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous message. We should thank you for that great piece of software that is mpich. We use it on our clusters since... 1996. Martin Audet ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 24, 2025 2:03 PM To: Audet, Martin; discuss(a)mpich.org Subject: EXT: Re: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!…> Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Audet, Martin 25 Jun '25

25 Jun '25

Hello Ken, Thanks for the information. I have recompiled mpich 4.3.0 and 4.3.1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous message. We should thank you for that great piece of software that is mpich. We use it on our clusters since... 1996. Martin Audet ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 24, 2025 2:03 PM To: Audet, Martin; discuss(a)mpich.org Subject: EXT: Re: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Raffenetti, Ken 24 Jun '25

24 Jun '25

Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Audet, Martin 23 Jun '25

23 Jun '25

Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Raffenetti, Ken 23 Jun '25

23 Jun '25

Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Audet, Martin 23 Jun '25

23 Jun '25

Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0