- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Error in MPI_IRECV
by fistrosan 19 Oct '25

19 Oct '25

If Gregor's answer does not solve it try adding -std=legacy to gfortrans compiler flags. Sent from [Proton Mail](https://urldefense.us/v3/__https://proton.me/mail/home__;!!G_uCfscf7eWS!dJKfTCTptV3q_0XDUIiQwsCbaMLbK_wpBUXQlGXJt8y6y91CgHIQadcRpG5GNGcfc6H4q7ws1mpT-QJAKUY$ ) for Android. -------- Original Message -------- On Friday, 10/17/25 at 14:26 Gregor Corbin via discuss <discuss(a)mpich.org> wrote: > Hi, > > let me guess: this code uses `include mpif.h` and is compiled with gfortran. If so, try to add `-fallow-argument-mismatch' to your compiler flags. > > Explanation: With the include file, there are no explicit interfaces and the compiler generates an implicit interface from the call to `MPI_IRECV`. Having two calls with different argument types is an error since some gcc version (10 IIRC). > > Cheers, > > Gregor Corbin > > On 17.10.25 13:30, phani sri via discuss wrote: > >> Hi all, I am trying to run FLASH4. 8, where I am getting the following error. Can someone please, suggest to me why this error is coming. Thank you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & >> ZjQcmQRYFpfptBannerStart >> This Message Is From an External Sender >> This message came from outside your organization. >> >> ZjQcmQRYFpfptBannerEnd >> Hi all, I am trying to run FLASH4.8, where I am getting the following error. Can someone please, suggest to me why this error is coming. >> Thank you in advance. >> 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & >> | 2 >> ...... >> 633 | Call MPI_IRECV(buffert(1,i),buf_size, & >> | 1 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). >> amr_migrate_tree_data.F90:642:23: >> >> 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & >> | 2 >> ...... >> 642 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & >> | 1 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). >> amr_migrate_tree_data.F90:717:23: >> >> 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & >> | 2 >> ...... >> 717 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & >> | 1 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). >> amr_migrate_tree_data.F90:725:23: >> >> 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & >> | 2 >> ...... >> 725 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & >> | 1 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). >> amr_migrate_tree_data.F90:260:25: >> >> 260 | Call MPI_IRECV(buffert(1,i),buf_size, & >> | 1 >> ...... >> 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & >> | 2 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). >> amr_migrate_tree_data.F90:269:25: >> >> 269 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & >> | 1 >> ...... >> 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & >> | 2 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). >> amr_migrate_tree_data.F90:330:25: >> >> 330 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & >> | 1 >> ...... >> 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & >> | 2 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). >> amr_migrate_tree_data.F90:338:25: >> >> 338 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & >> | 1 >> ...... >> 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & >> | 2 >> Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). >> amr_restrict_ec_genorder.F90:310:37: >> >> -- >> >> Dr. D. P. S. L. Kameswari >> >> _______________________________________________ >> discuss mailing list >> discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…

1 0

Re: [mpich-discuss] Error in MPI_IRECV
by Jenke, Joachim 18 Oct '25

18 Oct '25

Hi phani sri, I think your assertion is wrong. It looks like the flags fixed your initial issue. The initial error was raised by the compilation step. Now you run into the next issue raised by the linking step (which was never reached in your previous attempts). Please provide more details about the new error messages so that we have a chance to understand your new issue. Best Joachim Am 18.10.2025 06:05 schrieb phani sri via discuss <discuss(a)mpich.org>: Hi Hui, this also did not work. It is ending up with the error: collect2: error: ld returned 1 exit status. On Fri, 17 Oct 2025 at 21: 11, Zhou, Hui <zhouh@ anl. gov> wrote: Hi phani sri, Try set `FFLAGS=-fallow-argument-mismatch` and `FCFLAGS=-fallow-argument-mismatch` ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui, this also did not work.It is ending up with the error: collect2: error: ld returned 1 exit status. On Fri, 17 Oct 2025 at 21:11, Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi phani sri, Try set `FFLAGS=-fallow-argument-mismatch` and `FCFLAGS=-fallow-argument-mismatch` and let us know if that helps. Hui ________________________________ From: phani sri via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Friday, October 17, 2025 8:07 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: phani sri <phanisri123(a)gmail.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Error in MPI_IRECV Hi Gregor, I have checked this, the code is using mpi. h not mpif. h. Dr. D. P. S. L. Kameswari On Fri, 17 Oct 2025, 5: 56 pm Gregor Corbin via discuss, <discuss@ mpich. org> wrote: Hi, let me guess: this code uses `include mpif. h` and is compiled ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Gregor, I have checked this, the code is using mpi.h not mpif.h. Dr. D. P. S. L. Kameswari On Fri, 17 Oct 2025, 5:56 pm Gregor Corbin via discuss, <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, let me guess: this code uses `include mpif.h` and is compiled with gfortran. If so, try to add `-fallow-argument-mismatch' to your compiler flags. Explanation: With the include file, there are no explicit interfaces and the compiler generates an implicit interface from the call to `MPI_IRECV`. Having two calls with different argument types is an error since some gcc version (10 IIRC). Cheers, Gregor Corbin On 17.10.25 13:30, phani sri via discuss wrote: Hi all, I am trying to run FLASH4. 8, where I am getting the following error. Can someone please, suggest to me why this error is coming. Thank you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I am trying to run FLASH4.8, where I am getting the following error. Can someone please, suggest to me why this error is coming. Thank you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 ...... 633 | Call MPI_IRECV(buffert(1,i),buf_size, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:642:23: 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 ...... 642 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:717:23: 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 ...... 717 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:725:23: 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 ...... 725 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:260:25: 260 | Call MPI_IRECV(buffert(1,i),buf_size, & | 1 ...... 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:269:25: 269 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & | 1 ...... 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:330:25: 330 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & | 1 ...... 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:338:25: 338 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & | 1 ...... 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_restrict_ec_genorder.F90:310:37: -- Dr. D. P. S. L. Kameswari _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…> -- Dr. D. P. S. L. Kameswari

1 0

Re: [mpich-discuss] Error in MPI_IRECV
by phani sri 17 Oct '25

17 Oct '25

Hi Hui, this also did not work.It is ending up with the error: collect2: error: ld returned 1 exit status. On Fri, 17 Oct 2025 at 21:11, Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi phani sri, > > Try set `FFLAGS=-fallow-argument-mismatch` and > `FCFLAGS=-fallow-argument-mismatch` and let us know if that helps. > > Hui > ------------------------------ > *From:* phani sri via discuss <discuss(a)mpich.org> > *Sent:* Friday, October 17, 2025 8:07 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* phani sri <phanisri123(a)gmail.com> > *Subject:* Re: [mpich-discuss] Error in MPI_IRECV > > Hi Gregor, I have checked this, the code is using mpi. h not mpif. h. Dr. > D. P. S. L. Kameswari On Fri, 17 Oct 2025, 5: 56 pm Gregor Corbin via > discuss, <discuss@ mpich. org> wrote: Hi, let me guess: this code uses > `include mpif. h` and is compiled > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi Gregor, I have checked this, the code is using mpi.h not mpif.h. > > *Dr. D. P. S. L. Kameswari* > > > On Fri, 17 Oct 2025, 5:56 pm Gregor Corbin via discuss, <discuss(a)mpich.org> > wrote: > > Hi, > > let me guess: this code uses `include mpif.h` and is compiled with > gfortran. If so, try to add `-fallow-argument-mismatch' to your compiler > flags. > > Explanation: With the include file, there are no explicit interfaces and > the compiler generates an implicit interface from the call to `MPI_IRECV`. > Having two calls with different argument types is an error since some gcc > version (10 IIRC). > > Cheers, > > Gregor Corbin > > > On 17.10.25 13:30, phani sri via discuss wrote: > > Hi all, I am trying to run FLASH4. 8, where I am getting the following > error. Can someone please, suggest to me why this error is coming. Thank > you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi all, I am trying to run FLASH4.8, where I am getting the following > error. Can someone please, suggest to me why this error is coming. > Thank you in advance. > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 633 | Call MPI_IRECV(buffert(1,i),buf_size, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:642:23: > > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 642 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:717:23: > > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 717 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:725:23: > > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 725 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:260:25: > > 260 | Call MPI_IRECV(buffert(1,i),buf_size, > & > | 1 > ...... > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:269:25: > > 269 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, > & > | 1 > ...... > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:330:25: > > 330 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, > & > | 1 > ...... > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:338:25: > > 338 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, > & > | 1 > ...... > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_restrict_ec_genorder.F90:310:37: > > > > -- > *Dr. D. P. S. L. Kameswari* > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://urldefense.us/v3/__https://lists.mpich.org/mailman/list… <https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… > <https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…> > > -- *Dr. D. P. S. L. Kameswari*

1 0

Re: [mpich-discuss] Error in MPI_IRECV
by Zhou, Hui 17 Oct '25

17 Oct '25

Hi phani sri, Try set `FFLAGS=-fallow-argument-mismatch` and `FCFLAGS=-fallow-argument-mismatch` and let us know if that helps. Hui ________________________________ From: phani sri via discuss <discuss(a)mpich.org> Sent: Friday, October 17, 2025 8:07 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: phani sri <phanisri123(a)gmail.com> Subject: Re: [mpich-discuss] Error in MPI_IRECV Hi Gregor, I have checked this, the code is using mpi. h not mpif. h. Dr. D. P. S. L. Kameswari On Fri, 17 Oct 2025, 5: 56 pm Gregor Corbin via discuss, <discuss@ mpich. org> wrote: Hi, let me guess: this code uses `include mpif. h` and is compiled ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Gregor, I have checked this, the code is using mpi.h not mpif.h. Dr. D. P. S. L. Kameswari On Fri, 17 Oct 2025, 5:56 pm Gregor Corbin via discuss, <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, let me guess: this code uses `include mpif.h` and is compiled with gfortran. If so, try to add `-fallow-argument-mismatch' to your compiler flags. Explanation: With the include file, there are no explicit interfaces and the compiler generates an implicit interface from the call to `MPI_IRECV`. Having two calls with different argument types is an error since some gcc version (10 IIRC). Cheers, Gregor Corbin On 17.10.25 13:30, phani sri via discuss wrote: Hi all, I am trying to run FLASH4. 8, where I am getting the following error. Can someone please, suggest to me why this error is coming. Thank you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi all, I am trying to run FLASH4.8, where I am getting the following error. Can someone please, suggest to me why this error is coming. Thank you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 ...... 633 | Call MPI_IRECV(buffert(1,i),buf_size, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:642:23: 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 ...... 642 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:717:23: 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 ...... 717 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:725:23: 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 ...... 725 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:260:25: 260 | Call MPI_IRECV(buffert(1,i),buf_size, & | 1 ...... 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:269:25: 269 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & | 1 ...... 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:330:25: 330 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & | 1 ...... 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:338:25: 338 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & | 1 ...... 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_restrict_ec_genorder.F90:310:37: -- Dr. D. P. S. L. Kameswari _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss…>

1 0

Re: [mpich-discuss] Error in MPI_IRECV
by phani sri 17 Oct '25

17 Oct '25

Hi Gregor, I have checked this, the code is using mpi.h not mpif.h. *Dr. D. P. S. L. Kameswari* On Fri, 17 Oct 2025, 5:56 pm Gregor Corbin via discuss, <discuss(a)mpich.org> wrote: > Hi, > > let me guess: this code uses `include mpif.h` and is compiled with > gfortran. If so, try to add `-fallow-argument-mismatch' to your compiler > flags. > > Explanation: With the include file, there are no explicit interfaces and > the compiler generates an implicit interface from the call to `MPI_IRECV`. > Having two calls with different argument types is an error since some gcc > version (10 IIRC). > > Cheers, > > Gregor Corbin > > > On 17.10.25 13:30, phani sri via discuss wrote: > > Hi all, I am trying to run FLASH4. 8, where I am getting the following > error. Can someone please, suggest to me why this error is coming. Thank > you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi all, I am trying to run FLASH4.8, where I am getting the following > error. Can someone please, suggest to me why this error is coming. > Thank you in advance. > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 633 | Call MPI_IRECV(buffert(1,i),buf_size, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:642:23: > > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 642 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:717:23: > > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 717 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:725:23: > > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 725 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:260:25: > > 260 | Call MPI_IRECV(buffert(1,i),buf_size, > & > | 1 > ...... > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:269:25: > > 269 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, > & > | 1 > ...... > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:330:25: > > 330 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, > & > | 1 > ...... > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:338:25: > > 338 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, > & > | 1 > ...... > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual argument at > (2) (LOGICAL(4)/INTEGER(4)). > amr_restrict_ec_genorder.F90:310:37: > > > > -- > *Dr. D. P. S. L. Kameswari* > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe:https://urldefense.us/v3/__https://lists.mpich.org/mailman/list… > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… >

1 0

Re: [mpich-discuss] Error in MPI_IRECV
by Gregor Corbin 17 Oct '25

17 Oct '25

Hi, let me guess: this code uses `include mpif.h` and is compiled with gfortran. If so, try to add `-fallow-argument-mismatch' to your compiler flags. Explanation: With the include file, there are no explicit interfaces and the compiler generates an implicit interface from the call to `MPI_IRECV`. Having two calls with different argument types is an error since some gcc version (10 IIRC). Cheers, Gregor Corbin On 17.10.25 13:30, phani sri via discuss wrote: > Hi all, I am trying to run FLASH4. 8, where I am getting the following > error. Can someone please, suggest to me why this error is coming. > Thank you in advance. 497 | Call > MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > ZjQcmQRYFpfptBannerEnd > Hi all, I am trying to run FLASH4.8, where I am getting the following > error. Can someone please, suggest to me why this error is coming. > Thank you in advance. > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 633 | Call MPI_IRECV(buffert(1,i),buf_size, & > | 1 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:642:23: > > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 642 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:717:23: > > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 717 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & > | 1 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:725:23: > > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > ...... > 725 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, > & > | 1 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:260:25: > > 260 | Call MPI_IRECV(buffert(1,i),buf_size, > & > | 1 > ...... > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:269:25: > > 269 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, > & > | 1 > ...... > 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (LOGICAL(4)/INTEGER(4)). > amr_migrate_tree_data.F90:330:25: > > 330 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & > | 1 > ...... > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (REAL(8)/INTEGER(4)). > amr_migrate_tree_data.F90:338:25: > > 338 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, > & > | 1 > ...... > 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, > & > | 2 > Error: Type mismatch between actual argument at (1) and actual > argument at (2) (LOGICAL(4)/INTEGER(4)). > amr_restrict_ec_genorder.F90:310:37: > > > > -- > *Dr. D. P. S. L. Kameswari** > * > > _______________________________________________ > discuss mailing listdiscuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Error in MPI_IRECV
by phani sri 17 Oct '25

17 Oct '25

Hi all, I am trying to run FLASH4.8, where I am getting the following error. Can someone please, suggest to me why this error is coming. Thank you in advance. 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 ...... 633 | Call MPI_IRECV(buffert(1,i),buf_size, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:642:23: 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 ...... 642 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:717:23: 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 ...... 717 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:725:23: 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 ...... 725 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & | 1 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:260:25: 260 | Call MPI_IRECV(buffert(1,i),buf_size, & | 1 ...... 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:269:25: 269 | Call MPI_IRECV(newchildt(i),1,MPI_LOGICAL, & | 1 ...... 497 | Call MPI_IRECV(parentt(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_migrate_tree_data.F90:330:25: 330 | Call MPI_SSEND(buffer(1),buf_size,amr_mpi_real, & | 1 ...... 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)). amr_migrate_tree_data.F90:338:25: 338 | Call MPI_SSEND(newchild(i),1,MPI_LOGICAL, & | 1 ...... 513 | Call MPI_SSEND (new_loc(1,i),2,MPI_INTEGER, & | 2 Error: Type mismatch between actual argument at (1) and actual argument at (2) (LOGICAL(4)/INTEGER(4)). amr_restrict_ec_genorder.F90:310:37: -- *Dr. D. P. S. L. Kameswari*

1 0

Recommended configure options for MPICH 4.3.x with Valgrind or address-sanitizer
by Eric Chamberland 02 Oct '25

02 Oct '25

Hi, I have been building MPICH with the following configure options for a long time, mainly to keep my code “Valgrind-clean”: === ./configure \ --enable-g=dbg,meminit \ --with-device=ch3:sock \ --enable-romio === This setup worked reasonably well in the past, but recently I’ve been seeing occasional errors with address-sanitizer or valgrind (with 4.3.0 on a single node) such as: === Fatal error in internal_Allreduce_c: Unknown error class, error stack: internal_Allreduce_c(347)...................: MPI_Allreduce_c(sendbuf=0x7ffdeb0b8e90, recvbuf=0x7ffdeb0b8e98, count=1, dtype=0x4c00083a, MPI_SUM, comm=0x84000003) failed MPIR_Allreduce_impl(4826)...................: MPIR_Allreduce_allcomm_auto(4732)...........: MPIR_Allreduce_intra_recursive_doubling(115): MPIC_Sendrecv(266)..........................: MPIC_Wait(90)...............................: MPIR_Wait(751)..............................: MPIR_Wait_state(708)........................: MPIDI_CH3i_Progress_wait(187)...............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(385)..: MPIDI_CH3I_Socki_handle_read(3647)..........: connection failure (set=0,sock=1,errno=104:Connection reset by peer) === Is CH3 considered legacy? I would like to also ask: 1. What are the recommended configure options in 2025 for building MPICH in a way that works well with Valgrind? 2. Is it preferable now to move to CH4 (e.g. ch4:ofi or ch4:shm) when debugging with Valgrind? 3. Are there any other options (besides --enable-g=dbg,meminit) that you would suggest for catching memory errors while keeping Valgrind reports as clean as possible? 4. Is https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/main/doc/w… <https://urldefense.com/v3/__https:/github.com/pmodels/mpich/blob/main/doc/w… > up-to-date? Any guidance on the “best practice” configuration for this use case would be greatly appreciated. PETSc guys have some options about debug (https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/confi… <https://urldefense.com/v3/__https:/gitlab.com/petsc/petsc/-/blob/main/confi… >) but still uses CH3 by default. However Satish uses the configuration described above, at least for valgrind CI. Thanks a lot, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval On 2025-10-02 08:13, Balay, Satish wrote: > We currently use: > > balay@petsc-gpu-02:~$ > /nfs/gce/projects/petsc/soft/u22.04/mpich-4.3.0-p2-ucx/bin/mpichversion > MPICH Version: 4.3.0 > MPICH Release date: Mon Feb 3 09:09:47 AM CST 2025 > MPICH ABI: 17:0:5 > MPICH Device: ch4:ucx > MPICH configure: > --prefix=/nfs/gce/projects/petsc/soft/u22.04/mpich-4.3.0-p2-ucx > --enable-shared --with-device=ch4:ucx --with-pm=hydra --enable-fast=no > --enable-error-messages=all --enable-g=meminit --disable-java > --without-hwloc --disable-opencl --without-cuda --without-hip > MPICH CC: gcc -O0 > MPICH CXX: g++ -O0 > MPICH F77: gfortran -O0 > MPICH FC: gfortran -O0 > MPICH features: threadcomm > > With: > > #MPICH OFI/UCX/valgrind > export FI_PROVIDER=^psm3 > export UCX_SYSV_HUGETLB_MODE=n > export UCX_LOG_LEVEL=error > > Satish > ------------------------------------------------------------------------

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Howard Pritchard 03 Sep '25

03 Sep '25

HI Ken, Interesting. Switching to using PMI supplied by the vendor and using srun --mpi=pmi2. the results look more like what I would expect for this interconnect: # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.94 1 2.00 2 1.99 4 1.99 8 1.99 16 1.99 32 1.99 64 1.99 128 2.47 256 2.61 512 2.66 1024 2.78 2048 2.96 4096 3.01 8192 6.03 16384 6.73 32768 7.49 65536 8.86 131072 11.58 262144 17.21 524288 28.05 1048576 49.52 2097152 93.09 4194304 179.85 Am Do., 28. Aug. 2025 um 15:24 Uhr schrieb Raffenetti, Ken via discuss < discuss(a)mpich.org>: > I also tried on Frontier and get the expected (good) latency results with > the current main branch. The Slurm installation on Frontier does not > support PMIx, so it is not quite apples to apples. Does your system have > the Cray PMI library to try? > > Ken > > *From: *Raffenetti, Ken via discuss <discuss(a)mpich.org> > *Date: *Friday, August 22, 2025 at 11:03 AM > *To: *discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov> > *Cc: *Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org < > discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus > more - a slurm problem > > Hi Howard, > > The PMIx stuff is likely related to the new sessions implementation coming > in 5.0.x. I’ll look for a Slurm cluster to try and figure out what’s going > on with that. > > What commit hash are you working with that shows the poor latency? I just > built from the HEAD of main and don’t see the behavior on Aurora. > > Ken > > *From: *Howard Pritchard via discuss <discuss(a)mpich.org> > *Date: *Thursday, August 21, 2025 at 4:40 PM > *To: *Zhou, Hui <zhouh(a)anl.gov> > *Cc: *Howard Pritchard <hppritcha(a)gmail.com>, discuss(a)mpich.org < > discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus > more - a slurm problem > > This Message Is From an External Sender > This message came from outside your organization. > > Here you go Hui! > > MPICH debug output and slurm steps output to boot. Again no such > slurmy errors with the 4.3.1 release. > Something must have changed in the way MPICH is using the PMIX group > constructor ops or something like that. > > Required minimum FI_VERSION: 0, current version: 10016 > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] > > Required minimum FI_VERSION: 10005, current version: 10016 > > ==== Capability set configuration ==== > > libfabric provider: cxi - cxi > > MPIDI_OFI_ENABLE_DATA: 1 > > MPIDI_OFI_ENABLE_AV_TABLE: 1 > > MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0 > > MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 > > MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0 > > MPIDI_OFI_ENABLE_MR_ALLOCATED: 1 > > MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 0 > > MPIDI_OFI_ENABLE_MR_PROV_KEY: 0 > > MPIDI_OFI_ENABLE_TAGGED: 1 > > MPIDI_OFI_ENABLE_AM: 1 > > MPIDI_OFI_ENABLE_RMA: 1 > > MPIDI_OFI_ENABLE_ATOMICS: 1 > > MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 > > MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 > > MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 > > MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 > > MPIDI_OFI_ENABLE_TRIGGERED: 0 > > MPIDI_OFI_ENABLE_HMEM: 0 > > MPIDI_OFI_NUM_AM_BUFFERS: 8 > > MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0 > > MPIDI_OFI_CONTEXT_BITS: 20 > > MPIDI_OFI_SOURCE_BITS: 0 > > MPIDI_OFI_TAG_BITS: 20 > > MPIDI_OFI_VNI_USE_DOMAIN: 1 > > MAXIMUM SUPPORTED RANKS: 4294967296 > > MAXIMUM TAG: 1048576 > > ==== Provider global thresholds ==== > > max_buffered_send: 192 > > max_buffered_write: 192 > > max_msg_size: 4294967295 > > max_order_raw: -1 > > max_order_war: -1 > > max_order_waw: -1 > > tx_iov_limit: 1 > > rx_iov_limit: 1 > > rma_iov_limit: 1 > > max_mr_key_size: 4 > > ==== Various sizes and limits ==== > > MPIDI_OFI_AM_MSG_HEADER_SIZE: 24 > > MPIDI_OFI_MAX_AM_HDR_SIZE: 255 > > sizeof(MPIDI_OFI_am_request_header_t): 416 > > sizeof(MPIDI_OFI_per_vci_t): 52480 > > MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024 > > MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384 > > ====================================== > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001406 [1]: > pmixp_coll.c:280: No process controlled by this slurmstepd is involved in > this collective. > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001406 [1]: > pmixp_server.c:923: Unable to pmixp_state_coll_get() > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001405 [0]: > pmixp_coll_ring.c:614: 0x14b448006e10: unexpected contrib from nid001406:1, > expected is 0 > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001405 [0]: > pmixp_server.c:937: 0x14b448006e10: unexpected contrib from nid001406:1, > coll->seq=0, seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001405 > [0]: pmixp_coll_ring.c:738: 0x14b454052fc0: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001405 [0]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:756: 0x14b454052fc0: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:758: my peerid: 0:nid001405 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:765: neighbor id: next 1:nid001406, prev 1:nid001406 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:775: Context ptr=0x14b454053038, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:775: Context ptr=0x14b454053070, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:775: Context ptr=0x14b4540530a8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:823: wait contrib: nid001406 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001406 > [1]: pmixp_coll_ring.c:738: 0x14aa28053100: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001406 [1]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:756: 0x14aa28053100: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:758: my peerid: 1:nid001406 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:765: neighbor id: next 0:nid001405, prev 0:nid001405 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:775: Context ptr=0x14aa28053178, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:775: Context ptr=0x14aa280531b0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:775: Context ptr=0x14aa280531e8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:823: wait contrib: nid001405 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > ==== Various sizes and limits ==== > > sizeof(MPIDI_per_vci_t): 128 > > ==== collective selection ==== > > MPIR_CVAR_DEVICE_COLLECTIVES: percoll > > MPIR: MPII_coll_generic_json > > MPID: MPIDI_coll_generic_json > > MPID/shm: MPIDI_POSIX_coll_generic_json > > ==== OFI dynamic settings ==== > > num_vcis: 1 > > num_nics: 1 > > ====================================== > > error checking : disabled > > QMPI : disabled > > debugger support : disabled > > thread level : MPI_THREAD_SINGLE > > thread CS : per-vci > > threadcomm : enabled > > ==== data structure summary ==== > > sizeof(MPIR_Comm): 1832 > > sizeof(MPIR_Request): 520 > > sizeof(MPIR_Datatype): 280 > > ================================ > > # OSU MPI Latency Test v5.8 > > # Size Latency (us) > > 0 2.04 > > 1 10.08 > > 2 10.10 > > 4 10.11 > > 8 10.12 > > 16 10.12 > > 32 10.13 > > 64 10.12 > > 128 10.67 > > 256 8.10 > > 512 8.18 > > 1024 8.11 > > 2048 7.86 > > 4096 7.80 > > 8192 10.25 > > 16384 11.04 > > 32768 12.04 > > 65536 14.05 > > 131072 17.89 > > 262144 24.61 > > 524288 37.51 > > 1048576 61.48 > > 2097152 110.06 > > 4194304 228.67 > > > Am Mi., 13. Aug. 2025 um 13:10 Uhr schrieb Zhou, Hui <zhouh(a)anl.gov>: > > Hi Howard, > > Could you run with `MPIR_CVAR_DEBUG_SUMMARY=1`? It should print some debug > messages. I want to confirm it is running the `cxi` provider. > > > Hui > ------------------------------ > *From:* Howard Pritchard <hppritcha(a)gmail.com> > *Sent:* Wednesday, July 30, 2025 4:37 PM > *To:* Thakur, Rajeev <thakur(a)anl.gov> > *Cc:* discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> > *Subject:* Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus > more - a slurm problem > > You don't often get email from hppritcha(a)gmail.com. Learn why this is > important > <https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;…> > This Message Is From an External Sender > This message came from outside your organization. > > Hi Rajeev, > > Here are the results for 4.3.x branch: > > hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun > --mpi=pmix -n 2 ./osu_latency > > # OSU MPI Latency Test v5.8 > > # Size Latency (us) > > 0 1.92 > > 1 1.98 > > 2 1.98 > > 4 1.98 > > 8 1.98 > > 16 1.98 > > 32 1.99 > > 64 1.99 > > 128 2.47 > > 256 2.59 > > 512 2.65 > > 1024 2.76 > > 2048 2.95 > > 4096 3.00 > > 8192 5.96 > > 16384 6.64 > > 32768 7.44 > > 65536 8.75 > > 131072 11.52 > > 262144 17.08 > > 524288 27.96 > > 1048576 49.38 > > 2097152 92.96 > > 4194304 179.74 > > These are more like i would expect for SS11/OFI CXI provider. > > Howard > > Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <thakur(a)anl.gov > >: > > Hi Howard, > > What was the latency with the 4.3.x branch? > > > > Rajeev > > > > > > *From: *Howard Pritchard via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Wednesday, July 30, 2025 at 1:43 PM > *To: *"Zhou, Hui" <zhouh(a)anl.gov> > *Cc: *Howard Pritchard <hppritcha(a)gmail.com>, "discuss(a)mpich.org" < > discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus > more - a slurm problem > > > > Hi Hui That didn’t help. I am not surprised though as our cluster is an > NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch > and latency results are nominal and the slurm problem went away too. So we > will stick with that branch. > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender* > > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > > Hi Hui > > > > That didn’t help. I am not surprised though as our cluster is an NVIDIA > free zone. What did help is to switch to the mpich 4.3.x branch and > latency results are nominal and the slurm problem went away too. So we > will stick with that branch. > > > > Howard > > > > On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Howard, > > > > I wonder whether it is due to the overhead of querying pointer > attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` > and see if the latency improves? > > > > Hui > ------------------------------ > > *From:* Howard Pritchard via discuss <discuss(a)mpich.org> > *Sent:* Monday, July 28, 2025 9:41 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Howard Pritchard <hppritcha(a)gmail.com> > *Subject:* [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more > - a slurm problem > > > > Hi Folks, We are seeing a strange performance issue on our HPE SS11 system > when testing osu_latency inter-node with MPICH. First the info: system > using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: > MPICH Version: 5. 0. 0a1 > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender* > > This message came from outside your organization. > > > > ZjQcmQRYFpfptBannerEnd > > Hi Folks, > > > > We are seeing a strange performance issue on our HPE SS11 system when > testing osu_latency inter-node with MPICH. > > > > First the info: > > system using libfabric 1.22.0 > > slurm - 24.11.5 > > > > Here's my mpichversion output: > > > > MPICH Version: 5.0.0a1 > > MPICH Release date: unreleased development copy > > MPICH ABI: 0:0:0 > > MPICH Device: ch4:ofi > > MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no > --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple > --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci > --with-libfabric=/opt/cray/libfabric/1.22.0 > --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 > --enable-fast=O3 > > MPICH CC: gcc -O3 > > MPICH CXX: g++ -O3 > > MPICH F77: gfortran -O3 > > MPICH FC: gfortran -O3 > > MPICH features: threadcomm > > > > And here's the OSU latency results: > > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: > pmixp_coll.c:280: No process controlled by this slurmstepd is involved in > this collective. > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: > pmixp_server.c:923: Unable to pmixp_state_coll_get() > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: > pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, > expected is 0 > > slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: > pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, > coll->seq=0, seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 > [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:758: my peerid: 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:823: wait contrib: nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 > [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: > pmixp_coll.c:286: Dumping collective state > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:758: my peerid: 1:nid001439 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:788: neighbor contribs [2]: > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:821: done contrib: - > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:823: wait contrib: nid001438 > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS > > slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: > pmixp_coll_ring.c:829: buf (offset/size): 36/16384 > > # OSU MPI Latency Test v5.8 > > # Size Latency (us) > > 0 1.66 > > 1 9.29 > > 2 9.57 > > 4 9.69 > > 8 9.76 > > 16 9.77 > > 32 9.76 > > 64 9.77 > > 128 10.32 > > 256 7.54 > > 512 7.45 > > 1024 7.38 > > 2048 7.37 > > 4096 7.45 > > 8192 9.21 > > 16384 9.70 > > 32768 10.63 > > 65536 13.15 > > 131072 16.96 > > 262144 23.84 > > 524288 36.16 > > 1048576 60.36 > > 2097152 108.43 > > 4194304 228.31 > > > > Note the slurm behavior is - I launch the job. Go get coffee, do some > duo-lingo, read some emails, then after about 10 minutes the osu latency > runs. > > > > I did not get the slurm problems using an older mpich 4.3.1 but did get > the same performance issue. 9 usecs doesn't seem right for an 8-byte > pingpong over libfabric S11. I was expecting more like 1.6 or so. > > > > I am confident the slurm issue is unrelated to the latency issue. > > Thanks for any suggestions on how to address either issue however. > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… >

1 0

Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem
by Raffenetti, Ken 28 Aug '25

28 Aug '25

I also tried on Frontier and get the expected (good) latency results with the current main branch. The Slurm installation on Frontier does not support PMIx, so it is not quite apples to apples. Does your system have the Cray PMI library to try? Ken From: Raffenetti, Ken via discuss <discuss(a)mpich.org> Date: Friday, August 22, 2025 at 11:03 AM To: discuss(a)mpich.org <discuss(a)mpich.org>, Zhou, Hui <zhouh(a)anl.gov> Cc: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Howard, The PMIx stuff is likely related to the new sessions implementation coming in 5.0.x. I’ll look for a Slurm cluster to try and figure out what’s going on with that. What commit hash are you working with that shows the poor latency? I just built from the HEAD of main and don’t see the behavior on Aurora. Ken From: Howard Pritchard via discuss <discuss(a)mpich.org> Date: Thursday, August 21, 2025 at 4:40 PM To: Zhou, Hui <zhouh(a)anl.gov> Cc: Howard Pritchard <hppritcha(a)gmail.com>, discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem This Message Is From an External Sender This message came from outside your organization. Here you go Hui! MPICH debug output and slurm steps output to boot. Again no such slurmy errors with the 4.3.1 release. Something must have changed in the way MPICH is using the PMIX group constructor ops or something like that. Required minimum FI_VERSION: 0, current version: 10016 provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] provider: cxi, score = 5, pref = 0, FI_FORMAT_UNSPEC [8] Required minimum FI_VERSION: 10005, current version: 10016 ==== Capability set configuration ==== libfabric provider: cxi - cxi MPIDI_OFI_ENABLE_DATA: 1 MPIDI_OFI_ENABLE_AV_TABLE: 1 MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS: 0 MPIDI_OFI_ENABLE_SHARED_CONTEXTS: 0 MPIDI_OFI_ENABLE_MR_VIRT_ADDRESS: 0 MPIDI_OFI_ENABLE_MR_ALLOCATED: 1 MPIDI_OFI_ENABLE_MR_REGISTER_NULL: 0 MPIDI_OFI_ENABLE_MR_PROV_KEY: 0 MPIDI_OFI_ENABLE_TAGGED: 1 MPIDI_OFI_ENABLE_AM: 1 MPIDI_OFI_ENABLE_RMA: 1 MPIDI_OFI_ENABLE_ATOMICS: 1 MPIDI_OFI_FETCH_ATOMIC_IOVECS: 1 MPIDI_OFI_ENABLE_DATA_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_CONTROL_AUTO_PROGRESS: 0 MPIDI_OFI_ENABLE_PT2PT_NOPACK: 1 MPIDI_OFI_ENABLE_TRIGGERED: 0 MPIDI_OFI_ENABLE_HMEM: 0 MPIDI_OFI_NUM_AM_BUFFERS: 8 MPIDI_OFI_NUM_OPTIMIZED_MEMORY_REGIONS: 0 MPIDI_OFI_CONTEXT_BITS: 20 MPIDI_OFI_SOURCE_BITS: 0 MPIDI_OFI_TAG_BITS: 20 MPIDI_OFI_VNI_USE_DOMAIN: 1 MAXIMUM SUPPORTED RANKS: 4294967296 MAXIMUM TAG: 1048576 ==== Provider global thresholds ==== max_buffered_send: 192 max_buffered_write: 192 max_msg_size: 4294967295 max_order_raw: -1 max_order_war: -1 max_order_waw: -1 tx_iov_limit: 1 rx_iov_limit: 1 rma_iov_limit: 1 max_mr_key_size: 4 ==== Various sizes and limits ==== MPIDI_OFI_AM_MSG_HEADER_SIZE: 24 MPIDI_OFI_MAX_AM_HDR_SIZE: 255 sizeof(MPIDI_OFI_am_request_header_t): 416 sizeof(MPIDI_OFI_per_vci_t): 52480 MPIDI_OFI_AM_HDR_POOL_CELL_SIZE: 1024 MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE: 16384 ====================================== slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001406 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001406 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001405 [0]: pmixp_coll_ring.c:614: 0x14b448006e10: unexpected contrib from nid001406:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001405 [0]: pmixp_server.c:937: 0x14b448006e10: unexpected contrib from nid001406:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001405 [0]: pmixp_coll_ring.c:738: 0x14b454052fc0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001405 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:756: 0x14b454052fc0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001405 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001406, prev 1:nid001406 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:775: Context ptr=0x14b454053038, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:775: Context ptr=0x14b454053070, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:775: Context ptr=0x14b4540530a8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:823: wait contrib: nid001406 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001405 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001406 [1]: pmixp_coll_ring.c:738: 0x14aa28053100: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001406 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:756: 0x14aa28053100: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001406 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001405, prev 0:nid001405 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:775: Context ptr=0x14aa28053178, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:775: Context ptr=0x14aa280531b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:775: Context ptr=0x14aa280531e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:823: wait contrib: nid001405 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001406 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 ==== Various sizes and limits ==== sizeof(MPIDI_per_vci_t): 128 ==== collective selection ==== MPIR_CVAR_DEVICE_COLLECTIVES: percoll MPIR: MPII_coll_generic_json MPID: MPIDI_coll_generic_json MPID/shm: MPIDI_POSIX_coll_generic_json ==== OFI dynamic settings ==== num_vcis: 1 num_nics: 1 ====================================== error checking : disabled QMPI : disabled debugger support : disabled thread level : MPI_THREAD_SINGLE thread CS : per-vci threadcomm : enabled ==== data structure summary ==== sizeof(MPIR_Comm): 1832 sizeof(MPIR_Request): 520 sizeof(MPIR_Datatype): 280 ================================ # OSU MPI Latency Test v5.8 # Size Latency (us) 0 2.04 1 10.08 2 10.10 4 10.11 8 10.12 16 10.12 32 10.13 64 10.12 128 10.67 256 8.10 512 8.18 1024 8.11 2048 7.86 4096 7.80 8192 10.25 16384 11.04 32768 12.04 65536 14.05 131072 17.89 262144 24.61 524288 37.51 1048576 61.48 2097152 110.06 4194304 228.67 Am Mi., 13. Aug. 2025 um 13:10 Uhr schrieb Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>: Hi Howard, Could you run with `MPIR_CVAR_DEBUG_SUMMARY=1`? It should print some debug messages. I want to confirm it is running the `cxi` provider. Hui ________________________________ From: Howard Pritchard <hppritcha(a)gmail.com<mailto:[email protected]>> Sent: Wednesday, July 30, 2025 4:37 PM To: Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> Cc: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>; Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem You don't often get email from hppritcha(a)gmail.com<mailto:[email protected]>. Learn why this is important<https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;…> This Message Is From an External Sender This message came from outside your organization. Hi Rajeev, Here are the results for 4.3.x branch: hpp@nid001293:/usr/projects/artab/users/hpp/osu-micro-benchmarks-5.8-mpich/mpi/pt2pt>srun --mpi=pmix -n 2 ./osu_latency # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.92 1 1.98 2 1.98 4 1.98 8 1.98 16 1.98 32 1.99 64 1.99 128 2.47 256 2.59 512 2.65 1024 2.76 2048 2.95 4096 3.00 8192 5.96 16384 6.64 32768 7.44 65536 8.75 131072 11.52 262144 17.08 524288 27.96 1048576 49.38 2097152 92.96 4194304 179.74 These are more like i would expect for SS11/OFI CXI provider. Howard Am Mi., 30. Juli 2025 um 12:48 Uhr schrieb Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>>: Hi Howard, What was the latency with the 4.3.x branch? Rajeev From: Howard Pritchard via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, July 30, 2025 at 1:43 PM To: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>> Cc: Howard Pritchard <hppritcha(a)gmail.com<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4. 3. x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Hui That didn’t help. I am not surprised though as our cluster is an NVIDIA free zone. What did help is to switch to the mpich 4.3.x branch and latency results are nominal and the slurm problem went away too. So we will stick with that branch. Howard On Mon, Jul 28, 2025 at 4:15 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Howard, I wonder whether it is due to the overhead of querying pointer attributes. Could you try disable GPU support with `MPIR_CVAR_ENABLE_GPU=0` and see if the latency improves? Hui ________________________________ From: Howard Pritchard via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, July 28, 2025 9:41 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Howard Pritchard <hppritcha(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] MPICH 5.0.1 performance on HPE SS11 plus more - a slurm problem Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1. 22. 0 slurm - 24. 11. 5 Here's my mpichversion output: MPICH Version: 5. 0. 0a1 ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi Folks, We are seeing a strange performance issue on our HPE SS11 system when testing osu_latency inter-node with MPICH. First the info: system using libfabric 1.22.0 slurm - 24.11.5 Here's my mpichversion output: MPICH Version: 5.0.0a1 MPICH Release date: unreleased development copy MPICH ABI: 0:0:0 MPICH Device: ch4:ofi MPICH configure: --prefix=/XXXX/mpich_again/install --enable-g=no --enable-error-checking=no --with-device=ch4:ofi --enable-threads=multiple --with-ch4-shmmods=posix,xpmem --enable-thread-cs=per-vci --with-libfabric=/opt/cray/libfabric/1.22.0 --with-xpmem=/opt/cray/xpmem/default --with-pmix=/opt/pmix/gcc4x/5.0.8 --enable-fast=O3 MPICH CC: gcc -O3 MPICH CXX: g++ -O3 MPICH F77: gfortran -O3 MPICH FC: gfortran -O3 MPICH features: threadcomm And here's the OSU latency results: slurmstepd: error: mpi/pmix_v4: pmixp_coll_belong_chk: nid001439 [1]: pmixp_coll.c:280: No process controlled by this slurmstepd is involved in this collective. slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001439 [1]: pmixp_server.c:923: Unable to pmixp_state_coll_get() slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_check: nid001438 [0]: pmixp_coll_ring.c:614: 0x15005c005dc0: unexpected contrib from nid001439:1, expected is 0 slurmstepd: error: mpi/pmix_v4: _process_server_request: nid001438 [0]: pmixp_server.c:937: 0x15005c005dc0: unexpected contrib from nid001439:1, coll->seq=0, seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001438 [0]: pmixp_coll_ring.c:738: 0x1500580532f0: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001438 [0]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:756: 0x1500580532f0: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:758: my peerid: 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:765: neighbor id: next 1:nid001439, prev 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x150058053368, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533a0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:775: Context ptr=0x1500580533d8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:823: wait contrib: nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001438 [0]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_reset_if_to: nid001439 [1]: pmixp_coll_ring.c:738: 0x151d0c053400: collective timeout seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_log: nid001439 [1]: pmixp_coll.c:286: Dumping collective state slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:756: 0x151d0c053400: COLL_FENCE_RING state seq=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:758: my peerid: 1:nid001439 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:765: neighbor id: next 0:nid001438, prev 0:nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c053478, #0, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534b0, #1, in-use=0 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:775: Context ptr=0x151d0c0534e8, #2, in-use=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:786: seq=0 contribs: loc=1/prev=0/fwd=1 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:788: neighbor contribs [2]: slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:821: done contrib: - slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:823: wait contrib: nid001438 slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:825: status=PMIXP_COLL_RING_PROGRESS slurmstepd: error: mpi/pmix_v4: pmixp_coll_ring_log: nid001439 [1]: pmixp_coll_ring.c:829: buf (offset/size): 36/16384 # OSU MPI Latency Test v5.8 # Size Latency (us) 0 1.66 1 9.29 2 9.57 4 9.69 8 9.76 16 9.77 32 9.76 64 9.77 128 10.32 256 7.54 512 7.45 1024 7.38 2048 7.37 4096 7.45 8192 9.21 16384 9.70 32768 10.63 65536 13.15 131072 16.96 262144 23.84 524288 36.16 1048576 60.36 2097152 108.43 4194304 228.31 Note the slurm behavior is - I launch the job. Go get coffee, do some duo-lingo, read some emails, then after about 10 minutes the osu latency runs. I did not get the slurm problems using an older mpich 4.3.1 but did get the same performance issue. 9 usecs doesn't seem right for an 8-byte pingpong over libfabric S11. I was expecting more like 1.6 or so. I am confident the slurm issue is unrelated to the latency issue. Thanks for any suggestions on how to address either issue however.

1 0