June 2025 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Zhou, Hui 25 Jun '25

25 Jun '25

Sorry for the misspelling - Martin 🙂 ________________________________ From: Zhou, Hui via discuss <discuss(a)mpich.org> Sent: Wednesday, June 25, 2025 11:46 AM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hi Margin, Just to say that your support and patience are very much appreciated. As for the hcoll issue, while the fix is simple - in fact we had the PR open since a year ago - it was blocked by the testing. We were not aware of of any hcoll users at the time, thus it received a low priority investigating the testing failures. In general, if you or your users are depending on some features and experience reakages, I would encourage you be harsh - more importantly, be persistent - and it's okay to shout from time to time :) . Stale issue needs a shout to raise attention and we need usage feedback to motivate and properly set priorities. Again, thank you for your support and understanding of our resource constraint. -- Hui ________________________________ From: Audet, Martin via discuss <discuss(a)mpich.org> Sent: Wednesday, June 25, 2025 9:03 AM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello Ken, Thanks for the information. I have recompiled mpich 4. 3. 0 and 4. 3. 1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Ken, Thanks for the information. I have recompiled mpich 4.3.0 and 4.3.1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous message. We should thank you for that great piece of software that is mpich. We use it on our clusters since... 1996. Martin Audet ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 24, 2025 2:03 PM To: Audet, Martin; discuss(a)mpich.org Subject: EXT: Re: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!…> Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Zhou, Hui 25 Jun '25

25 Jun '25

Hi Margin, Just to say that your support and patience are very much appreciated. As for the hcoll issue, while the fix is simple - in fact we had the PR open since a year ago - it was blocked by the testing. We were not aware of of any hcoll users at the time, thus it received a low priority investigating the testing failures. In general, if you or your users are depending on some features and experience reakages, I would encourage you be harsh - more importantly, be persistent - and it's okay to shout from time to time :) . Stale issue needs a shout to raise attention and we need usage feedback to motivate and properly set priorities. Again, thank you for your support and understanding of our resource constraint. -- Hui ________________________________ From: Audet, Martin via discuss <discuss(a)mpich.org> Sent: Wednesday, June 25, 2025 9:03 AM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello Ken, Thanks for the information. I have recompiled mpich 4. 3. 0 and 4. 3. 1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Ken, Thanks for the information. I have recompiled mpich 4.3.0 and 4.3.1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous message. We should thank you for that great piece of software that is mpich. We use it on our clusters since... 1996. Martin Audet ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 24, 2025 2:03 PM To: Audet, Martin; discuss(a)mpich.org Subject: EXT: Re: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!…> Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Audet, Martin 25 Jun '25

25 Jun '25

Hello Ken, Thanks for the information. I have recompiled mpich 4.3.0 and 4.3.1 used on our small cluster without the --with-hcoll= option to make sure that hcoll is not used (no problems reported so far). BTW I was a little hash in my previous message. We should thank you for that great piece of software that is mpich. We use it on our clusters since... 1996. Martin Audet ________________________________ From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 24, 2025 2:03 PM To: Audet, Martin; discuss(a)mpich.org Subject: EXT: Re: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Raffenetti, Ken 24 Jun '25

24 Jun '25

Hi Martin, You are correct, we failed to warn users about the possible runtime issues using hcoll. I will have to go back and check when the last successful tests were run. As you guessed, this integration has been neglected somewhat in recent times. I have created a github issue to update our documentation and hopefully backport a fix to the stable branch once confirmed. https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7475__;!… Ken From: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Date: Monday, June 23, 2025 at 2:48 PM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: RE: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Audet, Martin 23 Jun '25

23 Jun '25

Thanks for this quick reply. You say that hcoll don’t work correctly (runtime) so in this case, there should be a warning or something to warn users if they try to use it (like us). Slower but correct results are far better than faster but incorrect ones. I will recompile the library without this option so that it doesn’t create problems for the users of our cluster. Since which version the hcoll don’t work correctly ? I may also disable it in the older mpich versions we keep available for our users. Thanks, Martin From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: June 23, 2025 15:40 To: discuss(a)mpich.org Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: EXT: Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll ***Attention*** This email originated from outside of the NRC. ***Attention*** Ce courriel provient de l'extérieur du CNRC. Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca<mailto:[email protected]>> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Raffenetti, Ken 23 Jun '25

23 Jun '25

Hi Martin, My apologies for the lack of update on this topic. We did not include this patch because even with successful compilation, MPICH hcoll integration does not function correctly at runtime in our tests. Due to other priorities, we have not yet spent the time to fix the issue. Ken From: Audet, Martin via discuss <discuss(a)mpich.org> Date: Monday, June 23, 2025 at 10:04 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Audet, Martin <Martin.Audet(a)cnrc-nrc.gc.ca> Subject: [mpich-discuss] mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll Hello, It seems that the silly compilation problem with hcoll_rte. c I had back in April with mpich 4. 3. 0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4. 3. 1, see: https: //lists. mpich. org/mailman/htdig/discuss/2025-April/006725. html ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/htdig/discuss/202…> It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

mpich 4.3.1 still have compilation problem when using --with-hcoll=/opt/mellanox/hcoll
by Audet, Martin 23 Jun '25

23 Jun '25

Hello, It seems that the silly compilation problem with hcoll_rte.c I had back in April with mpich 4.3.0 when using --with-hcoll=/opt/mellanox/hcoll configuration option is still present in 4.3.1, see: https://urldefense.us/v3/__https://lists.mpich.org/mailman/htdig/discuss/20… It seems that the following very simple patch I was told to try with 4.3.0 haven't been included in 4.3.1: --- src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:54:24.847337975 -0400 +++ src/mpid/common/hcoll/hcoll_rte.c 2025-04-16 12:55:05.428164974 -0400 @@ -55,7 +55,7 @@ /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(&made_progress); + ret = hcoll_do_progress(-1, &made_progress); MPIR_Assert(ret == MPI_SUCCESS); } } So it look like this code path is not compiled very often by mpich developers or it's QA process. BTW applying the same patch fix the compilation problem, but: What does it mean for us users ? Should we still use this option ? BTW hcoll is a very cool mechanism for improving collective operations efficiency. Is this option obsolete ? Was it replaced by something else ? Thanks, Martin Audet

1 0

Re: [mpich-discuss] Assertion Failure
by Zhou, Hui 19 Jun '25

19 Jun '25

Hi Chris, Can you try to dump the value of *comm at the assertion? I suspect the value is corrupted and it results in a wrong comm handle passed to MPI_Comm_free. Hui ________________________________ From: Chris Hewson via discuss <discuss(a)mpich.org> Sent: Wednesday, June 18, 2025 1:41 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Chris Hewson <chris(a)resfrac.com> Subject: [mpich-discuss] Assertion Failure Hi All, I am having a consistent issue when using mpich with petsc. When calling another external library (mkl's pardiso), I will get the following assertion failure: Assertion failed in file src/binding/c/c_binding. c at line 29448: ((*comm)&(0x03ffffff)) ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi All, I am having a consistent issue when using mpich with petsc. When calling another external library (mkl's pardiso), I will get the following assertion failure: Assertion failed in file src/binding/c/c_binding.c at line 29448: ((*comm)&(0x03ffffff)) < MPIR_COMM_N_BUILTIN I'm not really too sure why, if I turn assertions off in mpich, the program will seg fault later on. Stack trace for this looks like this: Assertion failed in file src/binding/c/c_binding.c at line 29448: ((*comm)&(0x03ffffff)) < MPIR_COMM_N_BUILTIN resfrac73bfaea(+0x1e992e0) [0x55b9357d52e0] resfrac73bfaea(+0x1e07ca8) [0x55b935743ca8] resfrac73bfaea(PMPI_Comm_free+0x580) [0x55b935642570] resfrac73bfaea(MKLMPI_Comm_free+0x38) [0x55b9385830f8] resfrac73bfaea(mkl_pds_lp64_reduce_rhs_real+0x4aa) [0x55b938edc2aa] resfrac73bfaea(mkl_pds_lp64_slv_omp_real+0x1c46) [0x55b93869a4c6] resfrac73bfaea(mkl_pds_lp64_solve_slave+0xd84) [0x55b935ba8514] resfrac73bfaea(mkl_pds_lp64_cluster_sparse_solver+0xec4) [0x55b935ba4b74] resfrac73bfaea(+0xdef3de) [0x55b93472b3de] resfrac73bfaea(MatSolve+0x521) [0x55b9349f17ef] resfrac73bfaea(+0x1a29db7) [0x55b935365db7] resfrac73bfaea(PCApply+0x73c) [0x55b934c53065] resfrac73bfaea(+0x19ee12d) [0x55b93532a12d] resfrac73bfaea(+0x115a74c) [0x55b934a9674c] resfrac73bfaea(KSPSolve+0x1b) [0x55b934a98d04] resfrac73bfaea(_ZN11nextstim_ns18PetscSolveParallelERNS_17SystemOfEquationsE+0x28b7) [0x55b934046767] resfrac73bfaea(_ZN11nextstim_ns21SlavePoolWaitAndSolveEv+0x5c7) [0x55b9341d5037] resfrac73bfaea(main+0x5db) [0x55b933eb292b] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f14c99d5d90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f14c99d5e40] resfrac73bfaea(_start+0x25) [0x55b933ecb0d5] The mpich configuration and version that we're using: MPICH Version: 4.3.0 MPICH Release date: Mon Feb 3 09:09:47 AM CST 2025 MPICH ABI: 17:0:5 MPICH Device: ch3:sock MPICH configure: --prefix=/opt/anl/petsc-3.23.2 MAKE=/usr/bin/make --libdir=/opt/anl/petsc-3.23.2/lib CC=gcc CFLAGS=-fPIC -Wno-lto-type-mismatch -Wno-stringop-overflow -O0 AR=/usr/bin/ar ARFLAGS=cr CXX=g++ CXXFLAGS=-Wno-lto-type-mismatch -Wno-psabi -O0 -std=gnu++20 FFLAGS=-ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch FC=gfortran F77=gfortran FCFLAGS=-ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch --disable-shared --with-pm=hydra --disable-java --with-hwloc=embedded --enable-fast=no --enable-error-messages=all --with-device=ch3:sock --enable-g=meminit,dbg PYTHON=/opt/intel/oneapi/intelpython/python3.12/bin/python3 --disable-maintainer-mode --disable-dependency-tracking MPICH CC: gcc -fPIC -Wno-lto-type-mismatch -Wno-stringop-overflow -O0 -g -O0 MPICH CXX: g++ -Wno-lto-type-mismatch -Wno-psabi -O0 -std=gnu++20 -g -O0 MPICH F77: gfortran -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch -g -O0 MPICH FC: gfortran -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch -g -O0 Any information or help on this would be greatly appreciated. Chris Hewson Senior Reservoir Simulation Engineer ResFrac +1.587.575.9792

1 0

Assertion Failure
by Chris Hewson 18 Jun '25

18 Jun '25

Hi All, I am having a consistent issue when using mpich with petsc. When calling another external library (mkl's pardiso), I will get the following assertion failure: Assertion failed in file src/binding/c/c_binding.c at line 29448: ((*comm)&(0x03ffffff)) < MPIR_COMM_N_BUILTIN I'm not really too sure why, if I turn assertions off in mpich, the program will seg fault later on. Stack trace for this looks like this: Assertion failed in file src/binding/c/c_binding.c at line 29448: ((*comm)&(0x03ffffff)) < MPIR_COMM_N_BUILTIN resfrac73bfaea(+0x1e992e0) [0x55b9357d52e0] resfrac73bfaea(+0x1e07ca8) [0x55b935743ca8] resfrac73bfaea(PMPI_Comm_free+0x580) [0x55b935642570] resfrac73bfaea(MKLMPI_Comm_free+0x38) [0x55b9385830f8] resfrac73bfaea(mkl_pds_lp64_reduce_rhs_real+0x4aa) [0x55b938edc2aa] resfrac73bfaea(mkl_pds_lp64_slv_omp_real+0x1c46) [0x55b93869a4c6] resfrac73bfaea(mkl_pds_lp64_solve_slave+0xd84) [0x55b935ba8514] resfrac73bfaea(mkl_pds_lp64_cluster_sparse_solver+0xec4) [0x55b935ba4b74] resfrac73bfaea(+0xdef3de) [0x55b93472b3de] resfrac73bfaea(MatSolve+0x521) [0x55b9349f17ef] resfrac73bfaea(+0x1a29db7) [0x55b935365db7] resfrac73bfaea(PCApply+0x73c) [0x55b934c53065] resfrac73bfaea(+0x19ee12d) [0x55b93532a12d] resfrac73bfaea(+0x115a74c) [0x55b934a9674c] resfrac73bfaea(KSPSolve+0x1b) [0x55b934a98d04] resfrac73bfaea(_ZN11nextstim_ns18PetscSolveParallelERNS_17SystemOfEquationsE+0x28b7) [0x55b934046767] resfrac73bfaea(_ZN11nextstim_ns21SlavePoolWaitAndSolveEv+0x5c7) [0x55b9341d5037] resfrac73bfaea(main+0x5db) [0x55b933eb292b] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f14c99d5d90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7f14c99d5e40] resfrac73bfaea(_start+0x25) [0x55b933ecb0d5] The mpich configuration and version that we're using: MPICH Version: 4.3.0 MPICH Release date: Mon Feb 3 09:09:47 AM CST 2025 MPICH ABI: 17:0:5 MPICH Device: ch3:sock MPICH configure: --prefix=/opt/anl/petsc-3.23.2 MAKE=/usr/bin/make --libdir=/opt/anl/petsc-3.23.2/lib CC=gcc CFLAGS=-fPIC -Wno-lto-type-mismatch -Wno-stringop-overflow -O0 AR=/usr/bin/ar ARFLAGS=cr CXX=g++ CXXFLAGS=-Wno-lto-type-mismatch -Wno-psabi -O0 -std=gnu++20 FFLAGS=-ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch FC=gfortran F77=gfortran FCFLAGS=-ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch --disable-shared --with-pm=hydra --disable-java --with-hwloc=embedded --enable-fast=no --enable-error-messages=all --with-device=ch3:sock --enable-g=meminit,dbg PYTHON=/opt/intel/oneapi/intelpython/python3.12/bin/python3 --disable-maintainer-mode --disable-dependency-tracking MPICH CC: gcc -fPIC -Wno-lto-type-mismatch -Wno-stringop-overflow -O0 -g -O0 MPICH CXX: g++ -Wno-lto-type-mismatch -Wno-psabi -O0 -std=gnu++20 -g -O0 MPICH F77: gfortran -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch -g -O0 MPICH FC: gfortran -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -O0 -fallow-argument-mismatch -g -O0 Any information or help on this would be greatly appreciated. *Chris Hewson* Senior Reservoir Simulation Engineer ResFrac +1.587.575.9792

1 0

Re: [mpich-discuss] [EXTERNAL] [BULK] How to make/run romio tests?
by Latham, Robert J. 13 Jun '25

13 Jun '25

Ok, this fix turned out to be easier once I tried things for myself: the prefix should be "quobyte:" (not "quobytefs:"). ==rob ________________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Wednesday, June 11, 2025 10:25 To: discuss(a)mpich.org Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] Subject: Re: [mpich-discuss] [EXTERNAL] [BULK] How to make/run romio tests? An addendum, it's possible my "bugfix" as detailed in https: //github. com/pmodels/mpich/issues/7452 might not be enough. I have a code that uses `MPI_Info_set` to, well, set infos and I can't seem to affect the fs type. First I set: ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd An addendum, it's possible my "bugfix" as detailed in https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7452__;!… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7452__;!…> might not be enough. I have a code that uses `MPI_Info_set` to, well, set infos and I can't seem to affect the fs type. First I set: export ROMIO_PRINT_HINTS=1 and then in my code I added: call MPI_INFO_SET(info,"romio_filesystem_type","QUOBYTEFS:", status) but, well: key = romio_filesystem_type value = UFS: Generic ROMIO driver for all UNIX-like file systems I also tried it in lowercase as well. No joy. I've also tried: export ROMIO_FSTYPE_FORCE="QUOBYTEFS:" and: export ROMIO_FSTYPE_FORCE="quobytefs:" but in each case, it's still UFS. I do know my Infos are being seen by MPICH because if I toggled: call MPI_INFO_SET(info,"romio_cb_write","enable", status) from enable to disable, the ROMIO_PRINT_HINTS show the change. Matt [signature_3627681122]<https://urldefense.us/v3/__http://www.ssaihq.com/__;!!G_uCfscf7eWS!aqjYcygb…> Matt Thompson Lead Scientific Software Engineer/Supervisor Global Modeling and Assimilation Office Science Systems and Applications, Inc. Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 o: 301-614-6712 matthew.thompson(a)nasa.gov<mailto:[email protected]> From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Date: Wednesday, June 11, 2025 at 9:37 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] [BULK] [mpich-discuss] How to make/run romio tests? All, I'm trying to do some tests on a Quobyte system and we've seen confusing file write timings with Intel MPI and Open MPI. So, a bit of googling seeing if some MPI stack has a mention of Quobyte turns up: https: //www. quobyte. com/blog/quobyte-mpi-io-support-with-kernel-bypass-now-available/ ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd All, I'm trying to do some tests on a Quobyte system and we've seen confusing file write timings with Intel MPI and Open MPI. So, a bit of googling seeing if some MPI stack has a mention of Quobyte turns up: https://urldefense.us/v3/__https://www.quobyte.com/blog/quobyte-mpi-io-supp… <https://urldefense.us/v3/__https://www.quobyte.com/blog/quobyte-mpi-io-supp…> and, yep, quobyte seems to be in MPICH! I grabbed MPICH 4.3.0 and built it (with one bug fix[1]) and well, it seems to run Hello World. So, following that Quobyte page, I get to Step 10 and tried the romio tests and...I am apparently dumb. I went to my build/src/mpi/romio/test directory and I see 'runtests' there, but: > ./runtests make: *** No rule to make target 'default'. Stop. I also tried with -fname and: > ./runtests -fname=quobyte:test_mpi_file ./runtests: line 90: [quobyte:test_mpi_file: command not found make: *** No rule to make target 'default'. Stop. > ./runtests -fname=quobyte:/home/mathomp4/MPI/src/mpich-4.3.0/build-ifort-2021.13/src/mpi/romio/test/test_mpi_file ./runtests: line 90: [quobyte:/home/mathomp4/MPI/src/mpich-4.3.0/build-ifort-2021.13/src/mpi/romio/test/test_mpi_file: No such file or directory make: *** No rule to make target 'default'. Stop. Yeah. I'm stumped. I'm guessing I've missed some simple step along the way. NOTE: I have no idea even how to use romio really, but I was hoping the tests could, well, help me figure that out. Thanks for any help, Matt [1] See https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7452__;!… <https://urldefense.us/v3/__https://github.com/pmodels/mpich/issues/7452__;!…> [signature_3314915376]<https://urldefense.us/v3/__http://www.ssaihq.com/__;!!G_uCfscf7eWS!foSdMay5…> Matt Thompson Lead Scientific Software Engineer/Supervisor Global Modeling and Assimilation Office Science Systems and Applications, Inc. Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 o: 301-614-6712 matthew.thompson(a)nasa.gov<mailto:[email protected]>

1 0