- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Occasional hang with MPI_Intercomm_merge and OFI+provider verbs
by Iker Martín Álvarez 29 Oct '24

29 Oct '24

Hi Hui, Yes, it works for that provider, or at least I haven't had any issues with it. Kind regards, Iker Missatge de Zhou, Hui <zhouh(a)anl.gov> del dia dl., 28 d’oct. 2024 a les 20:01: > Hi Iker, > > Does it work with `FI_PROVIDER="verbs;ofi_rxm"? > > Hui > ------------------------------ > *From:* Iker Martín Álvarez via discuss <discuss(a)mpich.org> > *Sent:* Monday, October 28, 2024 12:34 PM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Iker Martín Álvarez <martini(a)uji.es> > *Subject:* [mpich-discuss] Occasional hang with MPI_Intercomm_merge and > OFI+provider verbs > > Hi, Lately I have been dealing with an unexpected problem when using > MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my > application hangs when two conditions are met. Specifically, the hang > occurs when using the resulting Intracommunicator > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > > ZjQcmQRYFpfptBannerEnd > Hi, > > Lately I have been dealing with an unexpected problem when using > MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my > application hangs when two conditions are met. > > Specifically, the hang occurs when using the resulting Intracommunicator > of MPI_Intercomm_merge in collective operations as MPI_Bcast. The > conditions are > - There is an oversubscription state. The number of processes is greater > than the available number of physical cores. > - Using CH4:ofi with FI_PROVIDER="verbs:ofi_rxd". > > I tested a minimal code with MPICH 4.2.0 and MPICH 4.2.3 configured as: > ./configure --prefix=... --with-device=ch4:ofi --disable-psm3 > > The minimal code > <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of…> > to reproduce the problem is the following: > ========================== > > #include <stdio.h>#include <stdlib.h>#include <mpi.h>int main(int argc, char* argv[]) { int rank, numP, numO; int rootBcast, order; double test = 0; int solution = 0; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm intercomm, intracomm; MPI_Comm_get_parent(&intercomm); if(intercomm == MPI_COMM_NULL) { numO = atoi(argv[1]); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numO, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE); order = 0; } else { order = 1; } MPI_Intercomm_merge(intercomm, order, &intracomm); printf("TEST 1 P%02d/%d\n", rank, numP); MPI_Bcast(&test, 1, MPI_DOUBLE, 0, intracomm); // Hangs here if(solution) { MPI_Barrier(intercomm); } printf("TEST 2 P%02d/%d\n", rank, numP); MPI_Finalize(); return 0;} > > ========================== > The code only hangs at the MPI_Bcast operation for some of the spawned > processes. All my executions have been with a single node of 20 cores, with > 10 initial processes and spawning 20 at the MPI_Comm_spawn function. If I > change the variable "solution" to 1, I rarely get a hang of the > application, but it still happens on some occasions. > > From my perspective, the code seems to follow the standard. Is this the > case? I have been able to run the code with other providers for OFI, but I > am confused as to why it does not work in this case. > > Thank you for your time. > Best regards, > Iker >

1 0

Re: [mpich-discuss] Occasional hang with MPI_Intercomm_merge and OFI+provider verbs
by Zhou, Hui 28 Oct '24

28 Oct '24

Hi Iker, Does it work with `FI_PROVIDER="verbs;ofi_rxm"? Hui ________________________________ From: Iker Martín Álvarez via discuss <discuss(a)mpich.org> Sent: Monday, October 28, 2024 12:34 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Iker Martín Álvarez <martini(a)uji.es> Subject: [mpich-discuss] Occasional hang with MPI_Intercomm_merge and OFI+provider verbs Hi, Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met. Specifically, the hang occurs when using the resulting Intracommunicator ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met. Specifically, the hang occurs when using the resulting Intracommunicator of MPI_Intercomm_merge in collective operations as MPI_Bcast. The conditions are - There is an oversubscription state. The number of processes is greater than the available number of physical cores. - Using CH4:ofi with FI_PROVIDER="verbs:ofi_rxd". I tested a minimal code with MPICH 4.2.0 and MPICH 4.2.3 configured as: ./configure --prefix=... --with-device=ch4:ofi --disable-psm3 The minimal code<https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of…> to reproduce the problem is the following: ========================== #include <stdio.h> #include <stdlib.h> #include <mpi.h> int main(int argc, char* argv[]) { int rank, numP, numO; int rootBcast, order; double test = 0; int solution = 0; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm intercomm, intracomm; MPI_Comm_get_parent(&intercomm); if(intercomm == MPI_COMM_NULL) { numO = atoi(argv[1]); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numO, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE); order = 0; } else { order = 1; } MPI_Intercomm_merge(intercomm, order, &intracomm); printf("TEST 1 P%02d/%d\n", rank, numP); MPI_Bcast(&test, 1, MPI_DOUBLE, 0, intracomm); // Hangs here if(solution) { MPI_Barrier(intercomm); } printf("TEST 2 P%02d/%d\n", rank, numP); MPI_Finalize(); return 0; } ========================== The code only hangs at the MPI_Bcast operation for some of the spawned processes. All my executions have been with a single node of 20 cores, with 10 initial processes and spawning 20 at the MPI_Comm_spawn function. If I change the variable "solution" to 1, I rarely get a hang of the application, but it still happens on some occasions. >From my perspective, the code seems to follow the standard. Is this the case? I have been able to run the code with other providers for OFI, but I am confused as to why it does not work in this case. Thank you for your time. Best regards, Iker

1 0

Occasional hang with MPI_Intercomm_merge and OFI+provider verbs
by Iker Martín Álvarez 28 Oct '24

28 Oct '24

Hi, Lately I have been dealing with an unexpected problem when using MPI_Comm_spawn + MPI_Intercomm_merge, where on some occasions my application hangs when two conditions are met. Specifically, the hang occurs when using the resulting Intracommunicator of MPI_Intercomm_merge in collective operations as MPI_Bcast. The conditions are - There is an oversubscription state. The number of processes is greater than the available number of physical cores. - Using CH4:ofi with FI_PROVIDER="verbs:ofi_rxd". I tested a minimal code with MPICH 4.2.0 and MPICH 4.2.3 configured as: ./configure --prefix=... --with-device=ch4:ofi --disable-psm3 The minimal code <https://urldefense.us/v3/__https://lorca.act.uji.es/gitlab/martini/mpich_of… > to reproduce the problem is the following: ========================== #include <stdio.h>#include <stdlib.h>#include <mpi.h>int main(int argc, char* argv[]) { int rank, numP, numO; int rootBcast, order; double test = 0; int solution = 0; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm intercomm, intracomm; MPI_Comm_get_parent(&intercomm); if(intercomm == MPI_COMM_NULL) { numO = atoi(argv[1]); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numO, MPI_INFO_NULL, 0, MPI_COMM_WORLD, &intercomm, MPI_ERRCODES_IGNORE); order = 0; } else { order = 1; } MPI_Intercomm_merge(intercomm, order, &intracomm); printf("TEST 1 P%02d/%d\n", rank, numP); MPI_Bcast(&test, 1, MPI_DOUBLE, 0, intracomm); // Hangs here if(solution) { MPI_Barrier(intercomm); } printf("TEST 2 P%02d/%d\n", rank, numP); MPI_Finalize(); return 0;} ========================== The code only hangs at the MPI_Bcast operation for some of the spawned processes. All my executions have been with a single node of 20 cores, with 10 initial processes and spawning 20 at the MPI_Comm_spawn function. If I change the variable "solution" to 1, I rarely get a hang of the application, but it still happens on some occasions. >From my perspective, the code seems to follow the standard. Is this the case? I have been able to run the code with other providers for OFI, but I am confused as to why it does not work in this case. Thank you for your time. Best regards, Iker

1 0

How to have valgrind suppression files installed?
by Eric Chamberland 18 Oct '24

18 Oct '24

Hi, we are compiling and running with an mpich configured with "--enable-g=dbg,meminit" option, we have installed valgrind-devel package. Configuring MPICH version 4.2.2 with '--prefix=/opt/mpi/mpich-4.2.2-debug' '--enable-debuginfo' '--enable-g=dbg,meminit' 'CC=/usr/bin/gcc' 'CXX=/usr/bin/g++' 'FC=/usr/bin/gfortran' 'F77=/usr/bin/gfortran' '--with-device=ch3:sock' '--enable-romio' We saw there are some valgrind suppression files into de repo but they are not "installed" when we launch a "make install". find . -name "*valg*" ./modules/hwloc/contrib/hwloc-valgrind.supp ./modules/ucx/contrib/valgrind.supp ./src/pm/hydra/modules/hwloc/contrib/hwloc-valgrind.supp Is this normal? Mpich finds valgrind headers: checking valgrind/valgrind.h usability... yes checking valgrind/valgrind.h presence... yes checking for valgrind/valgrind.h... yes checking valgrind/memcheck.h usability... yes checking valgrind/memcheck.h presence... yes checking for valgrind/memcheck.h... yes checking helgrind.h usability... no checking helgrind.h presence... no checking for helgrind.h... no checking valgrind/helgrind.h usability... yes checking valgrind/helgrind.h presence... yes checking for valgrind/helgrind.h... yes checking drd.h usability... no checking drd.h presence... no checking for drd.h... no checking valgrind/drd.h usability... yes checking valgrind/drd.h presence... yes checking for valgrind/drd.h... yes checking whether the valgrind headers are broken or too old... no I also found this in src/pm/hydra/modules/hwloc/Makefile.in # Only install the valgrind suppressions file if we're building in # standalone mode @HWLOC_BUILD_STANDALONE_TRUE@dist_pkgdata_DATA = contrib/hwloc-valgrind.supp all:all-recursive What is the "standalone mode" ? Thanks, Eric -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0

Re: [mpich-discuss] MPI_Send over ibverbs
by Niyaz Murshed 10 Oct '24

10 Oct '24

Thank you Ken. Really appreciate this. From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Thursday, October 10, 2024 at 9:55 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs This is the post of the recv for the acknowledgement message: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8… This is where the ack message is sent: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8… Ken From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Tuesday, October 8, 2024 at 11:32 AM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 10 AM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, October 8, 2024 at 11:10 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs It’s in MPI scope. To my knowledge, that level of acknowledgement is not exposed from the verbs API. Ken From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Tuesday, October 8, 2024 at 11:08 AM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 05 AM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, October 8, 2024 at 11:05 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs An internal acknowledgement message is sent from the receiver to the sender indicating that a particular synchronous send operation was matched. Ken From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Tuesday, October 8, 2024 at 10:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – >From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point of view of MPI_Send over RDMA. In case of RDMA, internally it will call rdma_send which does not know when a matching rdma_recv was called on destination. How do the sender knows when rdma_recv was called? From: Joachim Jenke via discuss <discuss(a)mpich.org> Date: Tuesday, October 8, 2024 at 9:42 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Hi, none of the MPI send calls has a requirement to return after the message was received. MPI_Send can return as soon as the caller can write to buffer without impact to the data being transferred. This might be after the data is put to the network, or the MPI library copied the data to an internal buffer. MPI_Ssend has the requirement to return after the message was matched at the receiver, which implies that the matching Receive was called. But it still does not require that the message was received at this point. Best Joachim Am 08.10.24 um 04:55 schrieb Niyaz Murshed via discuss: > Hi, > > Is there a way to check the implementation of MPI_Send? > > As I understand there is an implicit barrier which allows MPI_Send to > return only after the buffer is received by the destination. > > How is this communication done? How does the destination notify the > sender that buffer is received? > > How is this implemented when using rocev2 ? > > Regards, > > Niyaz > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/listinfo/discuss_…> -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!f… <https://urldefense.us/v3/__http:/www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!aT…>

1 0

Re: [mpich-discuss] MPI_Send over ibverbs
by Raffenetti, Ken 10 Oct '24

10 Oct '24

This is the post of the recv for the acknowledgement message: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8… This is where the ack message is sent: https://urldefense.us/v3/__https://github.com/pmodels/mpich/blob/824b5b37c8… Ken From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Tuesday, October 8, 2024 at 11:32 AM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 10 AM To: Niyaz Murshed <Niyaz. Murshed@ arm. com>, discuss@ mpich. org ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, October 8, 2024 at 11:10 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs It’s in MPI scope. To my knowledge, that level of acknowledgement is not exposed from the verbs API. Ken From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Tuesday, October 8, 2024 at 11:08 AM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 05 AM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, October 8, 2024 at 11:05 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs An internal acknowledgement message is sent from the receiver to the sender indicating that a particular synchronous send operation was matched. Ken From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Tuesday, October 8, 2024 at 10:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – >From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point of view of MPI_Send over RDMA. In case of RDMA, internally it will call rdma_send which does not know when a matching rdma_recv was called on destination. How do the sender knows when rdma_recv was called? From: Joachim Jenke via discuss <discuss(a)mpich.org> Date: Tuesday, October 8, 2024 at 9:42 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Hi, none of the MPI send calls has a requirement to return after the message was received. MPI_Send can return as soon as the caller can write to buffer without impact to the data being transferred. This might be after the data is put to the network, or the MPI library copied the data to an internal buffer. MPI_Ssend has the requirement to return after the message was matched at the receiver, which implies that the matching Receive was called. But it still does not require that the message was received at this point. Best Joachim Am 08.10.24 um 04:55 schrieb Niyaz Murshed via discuss: > Hi, > > Is there a way to check the implementation of MPI_Send? > > As I understand there is an implicit barrier which allows MPI_Send to > return only after the buffer is received by the destination. > > How is this communication done? How does the destination notify the > sender that buffer is received? > > How is this implemented when using rocev2 ? > > Regards, > > Niyaz > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/listinfo/discuss_…> -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!Y… <https://urldefense.us/v3/__http:/www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!aT…>

1 0

Re: [mpich-discuss] Maximum number of communicators
by Zhou, Hui 09 Oct '24

09 Oct '24

Hi Bruce, Could you create a github issue for this? Hui ________________________________ From: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Sent: Wednesday, October 9, 2024 1:37 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> Subject: Re: Maximum number of communicators I may have spoken too soon. This fixed the issue on one SMP node using 24 processes. When I increase the number of nodes to 4 with 96 processes, I can’t even get past MPI_Init. I’m seeing the following in standard out =================================================================================== ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd I may have spoken too soon. This fixed the issue on one SMP node using 24 processes. When I increase the number of nodes to 4 with 96 processes, I can’t even get past MPI_Init. I’m seeing the following in standard out =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 483326 RUNNING AT j006 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions I get the following in standard error: Abort(613541135) on node 65: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(49162).............: MPI_Init(argc=0x7ffff3c5646c, argv=0x7ffff3c56460) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(302)...............: MPID_Isend(63)...................: MPIDI_isend(35)..................: MPIDI_OFI_send_fallback(549).....: OFI call tsendv failed (ofi_send.h:549:MPIDI_OFI_send_fallback:No such file or dir ectory) Bruce From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Date: Monday, October 7, 2024 at 1:14 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Maximum number of communicators Check twice before you click! This email originated from outside PNNL. That seemed to fix the issue. Thanks! Bruce From: Zhou, Hui <zhouh@ anl. gov> Date: Thursday, October 3, 2024 at 11: 01 AM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> Subject: Re: Maximum ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd That seemed to fix the issue. Thanks! Bruce From: Zhou, Hui <zhouh(a)anl.gov> Date: Thursday, October 3, 2024 at 11:01 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Maximum number of communicators Hi Bruce, Try configure mpich using --with-device=ch4:ofi --enable-extended-context-bits. Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Thursday, October 3, 2024 11:44 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Maximum number of communicators Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is built on top of 4 conventional global arrays and each global array uses one MPI Window. Each Window appears to be creating its own communicator and I’m hitting an internal limit at 2048 communicators. Is there a way to increase the number of communicators? Bruce Palmer

1 0

Re: [mpich-discuss] Maximum number of communicators
by Palmer, Bruce J 09 Oct '24

09 Oct '24

I may have spoken too soon. This fixed the issue on one SMP node using 24 processes. When I increase the number of nodes to 4 with 96 processes, I can’t even get past MPI_Init. I’m seeing the following in standard out =================================================================================== = BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES = PID 483326 RUNNING AT j006 = EXIT CODE: 9 = CLEANING UP REMAINING PROCESSES = YOU CAN IGNORE THE BELOW CLEANUP MESSAGES =================================================================================== YOUR APPLICATION TERMINATED WITH THE EXIT STRING: Killed (signal 9) This typically refers to a problem with your application. Please see the FAQ page for debugging suggestions I get the following in standard error: Abort(613541135) on node 65: Fatal error in internal_Init: Other MPI error, error stack: internal_Init(49162).............: MPI_Init(argc=0x7ffff3c5646c, argv=0x7ffff3c56460) failed MPII_Init_thread(265)............: MPIR_init_comm_world(34).........: MPIR_Comm_commit(823)............: MPID_Comm_commit_post_hook(222)..: MPIDI_world_post_init(660).......: MPIDI_OFI_init_vcis(842).........: check_num_nics(891)..............: MPIR_Allreduce_allcomm_auto(4726): MPIC_Sendrecv(302)...............: MPID_Isend(63)...................: MPIDI_isend(35)..................: MPIDI_OFI_send_fallback(549).....: OFI call tsendv failed (ofi_send.h:549:MPIDI_OFI_send_fallback:No such file or dir ectory) Bruce From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Date: Monday, October 7, 2024 at 1:14 PM To: Zhou, Hui <zhouh(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] Maximum number of communicators Check twice before you click! This email originated from outside PNNL. That seemed to fix the issue. Thanks! Bruce From: Zhou, Hui <zhouh@ anl. gov> Date: Thursday, October 3, 2024 at 11: 01 AM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Palmer, Bruce J <Bruce. Palmer@ pnnl. gov> Subject: Re: Maximum ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd That seemed to fix the issue. Thanks! Bruce From: Zhou, Hui <zhouh(a)anl.gov> Date: Thursday, October 3, 2024 at 11:01 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: Re: Maximum number of communicators Hi Bruce, Try configure mpich using --with-device=ch4:ofi --enable-extended-context-bits. Hui ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Thursday, October 3, 2024 11:44 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce J <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] Maximum number of communicators Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hi, I’m looking at using MPI RMA to support sparse data structures in Global Arrays. I’ve got an application that uses a large number of sparse arrays and it is failing when the number of sparse arrays reaches about 500. Each sparse array is built on top of 4 conventional global arrays and each global array uses one MPI Window. Each Window appears to be creating its own communicator and I’m hitting an internal limit at 2048 communicators. Is there a way to increase the number of communicators? Bruce Palmer

1 0

Re: [mpich-discuss] MPI_Send over ibverbs
by Niyaz Murshed 08 Oct '24

08 Oct '24

Thank you Ken. Do you know where I can see the code of these acks ; message sequences. From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, October 8, 2024 at 11:10 AM To: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs It’s in MPI scope. To my knowledge, that level of acknowledgement is not exposed from the verbs API. Ken From: Niyaz Murshed <Niyaz.Murshed(a)arm.com> Date: Tuesday, October 8, 2024 at 11:08 AM To: Raffenetti, Ken <raffenet(a)anl.gov>, discuss(a)mpich.org <discuss(a)mpich.org> Cc: nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet@ anl. gov> Date: Tuesday, October 8, 2024 at 11: 05 AM To: discuss@ mpich. org <discuss@ mpich. org> Cc: Niyaz Murshed <Niyaz. Murshed@ arm. com>, ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Are you talking about the ack of RC connection? Or is it MPI’s scope? From: Raffenetti, Ken <raffenet(a)anl.gov> Date: Tuesday, October 8, 2024 at 11:05 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs An internal acknowledgement message is sent from the receiver to the sender indicating that a particular synchronous send operation was matched. Ken From: Niyaz Murshed via discuss <discuss(a)mpich.org> Date: Tuesday, October 8, 2024 at 10:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Niyaz Murshed <Niyaz.Murshed(a)arm.com>, nd <nd(a)arm.com> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thank you for the reply Joachim. ‘MPI_Ssend has the requirement to return after the message was matched at the receiver’ – >From the above statement, how do the sender know that the receiver called a matching receive? I am asking from the point of view of MPI_Send over RDMA. In case of RDMA, internally it will call rdma_send which does not know when a matching rdma_recv was called on destination. How do the sender knows when rdma_recv was called? From: Joachim Jenke via discuss <discuss(a)mpich.org> Date: Tuesday, October 8, 2024 at 9:42 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Joachim Jenke <jenke(a)itc.rwth-aachen.de> Subject: Re: [mpich-discuss] MPI_Send over ibverbs Hi, none of the MPI send calls has a requirement to return after the message was received. MPI_Send can return as soon as the caller can write to buffer without impact to the data being transferred. This might be after the data is put to the network, or the MPI library copied the data to an internal buffer. MPI_Ssend has the requirement to return after the message was matched at the receiver, which implies that the matching Receive was called. But it still does not require that the message was received at this point. Best Joachim Am 08.10.24 um 04:55 schrieb Niyaz Murshed via discuss: > Hi, > > Is there a way to check the implementation of MPI_Send? > > As I understand there is an implicit barrier which allows MPI_Send to > return only after the buffer is received by the destination. > > How is this communication done? How does the destination notify the > sender that buffer is received? > > How is this implemented when using rocev2 ? > > Regards, > > Niyaz > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://urldefense.us/v3/__https://lists.mpich.org/mailman/listinfo/discuss… <https://urldefense.us/v3/__https:/lists.mpich.org/mailman/listinfo/discuss_…> -- Dr. rer. nat. Joachim Jenke IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 jenke(a)itc.rwth-aachen.de https://urldefense.us/v3/__http://www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!f… <https://urldefense.us/v3/__http:/www.itc.rwth-aachen.de__;!!G_uCfscf7eWS!aT…>

1 0

Re: [mpich-discuss] How to use ppr with mpiexec?
by Jeffrey Layton 08 Oct '24

08 Oct '24

Thanks! I used the second option and it worked just fine. Thanks for the help! Jeff On Tue, Oct 8, 2024 at 11:32 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > If you are using hydra – the default launcher installed with MPICH, the > “-hosts” option accepts a comma separated list, so something like “-hosts > node1:4,node2:4,node3:4”. The “:4” tells the launcher to assign 4 processes > per host, so you don’t need another “-map-by” option. Alternatively, you > could use “-hosts node1,node2,node3 -ppn 4” for the same effect. > > > > -- > Hui Zhou > > > > > > *From: *Jeffrey Layton via discuss <discuss(a)mpich.org> > *Date: *Tuesday, October 8, 2024 at 6:50 AM > *To: *discuss(a)mpich.org <discuss(a)mpich.org> > *Cc: *Jeffrey Layton <laytonjb(a)gmail.com> > *Subject: *[mpich-discuss] How to use ppr with mpiexec? > > Good morning, I'm trying to "port" an Open MPI script with MPICH and I'm > having some trouble. I've got a few questions. What is the form of the > "-hosts" option? Is it something like '-hosts node1: 4 node2: 4 > > ZjQcmQRYFpfptBannerStart > > *This Message Is From an External Sender * > > This message came from outside your organization. > > > > ZjQcmQRYFpfptBannerEnd > > Good morning, > > > > I'm trying to "port" an Open MPI script with MPICH and I'm having some > trouble. I've got a few questions. > > 1. What is the form of the "-hosts" option? Is it something like '-hosts > node1:4 node2:4 node3:4 ...'? > > 2. The Open MPI script has an option "-map-by ppr:4:node" that tells it > to run 4 processes per node and then it runs as many nodes as needed to > meet the number of processes. How can I do this in MPICH? > > Thanks for the help! > > > > Jeff > > >

1 0