November 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Waitany got abort signal.
by Zhou, Hui 24 Nov '21

24 Nov '21

Hi Anatoly, Without a reproducible simple case, it will be difficult to pin down the issue. To debug, I would start injecting prints right above the assertion to see what is actually in the pkt_type. But since mpich-3.1 is very old, can you try a newer release? The latest is mpich 4.0b1 -- https://www.mpich.org/downloads/ -- Hui Zhou [https://www.mpich.org/files/2012/10/rnd100_home.jpg1zB]<https://www.mpich.org/downloads/> Downloads | MPICH<https://www.mpich.org/downloads/> Downloads MPICH is distributed under a BSD-like license. NOTE: MPICH binary packages are available in many UNIX distributions and for Windows. For example, you can search for it using “yum” (on Fedora), “apt” (Debian/Ubuntu), “pkg_add” (FreeBSD) or “port”/”brew” (Mac OS). www.mpich.org ________________________________ From: Anatoly G via discuss <discuss(a)mpich.org> Sent: Wednesday, November 24, 2021 3:49 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Anatoly G <anatolyrishon(a)gmail.com> Subject: [mpich-discuss] MPI_Waitany got abort signal. Hello Mpich, I use mpich-3.1 on Ubuntu 14. Each process has complicated logic except process 0. Process 0 is used as a router to communicate with an application and broadcast/collect results from other processes. During night runs, sometimes I see a single failure of process with rank 0. From process 0 I get wallowing print: Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 0 *** Error in `/export/home/fpd/versions/current_ver/third_party/MPI_Scheduler': double free or corruption (fasttop): 0x00007f27003f59a0 *** Stack trace: 2d40 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f2712ff2d40] gsignal + 57 abort + 328 f394 /lib/x86_64-linux-gnu/libc.so.6(+0x73394) [0x7f271302f394] b66e /lib/x86_64-linux-gnu/libc.so.6(+0x7f66e) [0x7f271303b66e] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() + 31 8259 /lib/x86_64-linux-gnu/libc.so.6(+0x3c259) [0x7f2712ff8259] 82a5 /lib/x86_64-linux-gnu/libc.so.6(+0x3c2a5) [0x7f2712ff82a5] d049 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x224049) [0x7f2714bdd049] MPID_Abort + 103 MPIR_Assert_fail + 37 d598 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x244598) [0x7f2714bfd598] MPID_nem_tcp_connpoll + 366 MPIDI_CH3I_Progress + 1408 MPI_Waitany + 1072 From the rest of processes I get: terminate called after throwing an instance of 'int' s/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1 *** Error in `Scheduler': double free or corruption (fasttop): 0x0000000003e16d60 *** Unfortunately, I can't reproduce this failure on a simplified system. Even on a real system failure can happen once in at night. We have a memory monitor which shows that we have free memory on the computer. Can you please advise me, what can be the reason for failure? Regards, Anatoly.

1 0

MPI_Waitany got abort signal.
by Anatoly G 24 Nov '21

24 Nov '21

Hello Mpich, I use mpich-3.1 on Ubuntu 14. Each process has complicated logic except process 0. Process 0 is used as a router to communicate with an application and broadcast/collect results from other processes. During night runs, sometimes I see a single failure of process with rank 0. >From process 0 I get wallowing print: Assertion failed in file src/mpid/ch3/channels/nemesis/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 0 *** Error in `/export/home/fpd/versions/current_ver/third_party/MPI_Scheduler': double free or corruption (fasttop): 0x00007f27003f59a0 *** *Stack trace:* 2d40 /lib/x86_64-linux-gnu/libc.so.6(+0x36d40) [0x7f2712ff2d40] gsignal + 57 abort + 328 f394 /lib/x86_64-linux-gnu/libc.so.6(+0x73394) [0x7f271302f394] b66e /lib/x86_64-linux-gnu/libc.so.6(+0x7f66e) [0x7f271303b66e] std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() + 31 8259 /lib/x86_64-linux-gnu/libc.so.6(+0x3c259) [0x7f2712ff8259] 82a5 /lib/x86_64-linux-gnu/libc.so.6(+0x3c2a5) [0x7f2712ff82a5] d049 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x224049) [0x7f2714bdd049] MPID_Abort + 103 MPIR_Assert_fail + 37 d598 /export/home/fpd/versions/current_ver/third_party/libMPIServices.so(+0x244598) [0x7f2714bfd598] MPID_nem_tcp_connpoll + 366 MPIDI_CH3I_Progress + 1408 MPI_Waitany + 1072 >From the rest of processes I get: terminate called after throwing an instance of 'int' s/netmod/tcp/socksm.c at line 596: hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_ID_INFO || hdr.pkt_type == MPIDI_NEM_TCP_SOCKSM_PKT_TMPVC_INFO internal ABORT - process 1 *** Error in `Scheduler': double free or corruption (fasttop): 0x0000000003e16d60 *** Unfortunately, I can't reproduce this failure on a simplified system. Even on a real system failure can happen once in at night. We have a memory monitor which shows that we have free memory on the computer. Can you please advise me, what can be the reason for failure? Regards, Anatoly.

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Zhou, Hui 22 Nov '21

22 Nov '21

We just released 4.0b1 last week! We will release 4.0rc1 and 4.0 GA in roughly two weeks apart if no major issues are discovered during this period. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Monday, November 22, 2021 at 9:13 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, A reboot of that failing node fixed the problem. My original question was if there was a limit on the number of inter-communicators. With 4.0a2, my job runs fine with the increased number of inter-communicators, filling up each node nicely. Should I just step up to 3.4.2, or wait until 4.0 is released? Do you know when 4.0 will be released? Thanks for your help, Kurt Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, November 22, 2021 8:10 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] [EXTERNAL] Re: Maximum number of inter-communicators? Hui, just one process on a particular node failed (without any error message). We just had an O/S upgrade on the whole cluster, and may an error was made on that node in the upgrade. I’m trying to figure out what is different about that node. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, November 19, 2021 4:29 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Kurt, That usually means the process went away without telling the process manager (hydra) first. Those messages are from hydra when the connection to the process disconnected unexpectedly. Did all processes call `MPI_Finalize` before exit? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, November 19, 2021 at 2:26 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Thanks Hui, replacing MPI_LOGICAL with MPI_CXX_BOOL enabled the code to run without the segfault (using 4.0a2). Do you have any clue what this error message means? [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:206): assert (!closed) failed [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0a2/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip.c:169): de[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_ Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, November 15, 2021 9:44 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, Did you build mpich with Fortran disabled? MPI_LOGICAL is a Fortran datatype and is unavailable when Fortran binding is disabled. Try use MPI_C_BOOL instead. If you didn't disable cxx, you may also try use MPI_CXX_BOOL since you are programming in C++. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Saturday, November 13, 2021 2:56 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below. Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace. Program received signal SIGSEGV, Segmentation fault. MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); (gdb) where #0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410, displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410, array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 #6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0, array_of_types=0x7fffcaa24410, newtype=0x12853fc) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97 #8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, world_size=21) at src/NeedlesMpiManager.cpp:204 #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142 (gdb) Here is my code that creates the custom type and then calls MPI_Type_commit: MsgBasic obj; int struct_len = 8, i; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; i = 0; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; // the int array "start_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the int array "end_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the boolean "success_" member ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version displacements[i] = (size_t) &obj.success_ - (size_t) &obj; // the double "run_time_sec_" member ++i; block_len[i] = 1; types[i] = MPI_DOUBLE; displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &msg_data_type_); MPI_Type_commit(&msg_data_type_); Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 22 Nov '21

22 Nov '21

Hui, A reboot of that failing node fixed the problem. My original question was if there was a limit on the number of inter-communicators. With 4.0a2, my job runs fine with the increased number of inter-communicators, filling up each node nicely. Should I just step up to 3.4.2, or wait until 4.0 is released? Do you know when 4.0 will be released? Thanks for your help, Kurt Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, November 22, 2021 8:10 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] [EXTERNAL] Re: Maximum number of inter-communicators? Hui, just one process on a particular node failed (without any error message). We just had an O/S upgrade on the whole cluster, and may an error was made on that node in the upgrade. I’m trying to figure out what is different about that node. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Friday, November 19, 2021 4:29 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Kurt, That usually means the process went away without telling the process manager (hydra) first. Those messages are from hydra when the connection to the process disconnected unexpectedly. Did all processes call `MPI_Finalize` before exit? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, November 19, 2021 at 2:26 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Thanks Hui, replacing MPI_LOGICAL with MPI_CXX_BOOL enabled the code to run without the segfault (using 4.0a2). Do you have any clue what this error message means? [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:206): assert (!closed) failed [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0a2/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip.c:169): de[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_ Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, November 15, 2021 9:44 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, Did you build mpich with Fortran disabled? MPI_LOGICAL is a Fortran datatype and is unavailable when Fortran binding is disabled. Try use MPI_C_BOOL instead. If you didn't disable cxx, you may also try use MPI_CXX_BOOL since you are programming in C++. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Saturday, November 13, 2021 2:56 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below. Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace. Program received signal SIGSEGV, Segmentation fault. MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); (gdb) where #0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410, displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410, array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 #6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0, array_of_types=0x7fffcaa24410, newtype=0x12853fc) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97 #8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, world_size=21) at src/NeedlesMpiManager.cpp:204 #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142 (gdb) Here is my code that creates the custom type and then calls MPI_Type_commit: MsgBasic obj; int struct_len = 8, i; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; i = 0; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; // the int array "start_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the int array "end_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the boolean "success_" member ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version displacements[i] = (size_t) &obj.success_ - (size_t) &obj; // the double "run_time_sec_" member ++i; block_len[i] = 1; types[i] = MPI_DOUBLE; displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &msg_data_type_); MPI_Type_commit(&msg_data_type_); Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 22 Nov '21

22 Nov '21

Hui, just one process on a particular node failed (without any error message). We just had an O/S upgrade on the whole cluster, and may an error was made on that node in the upgrade. I’m trying to figure out what is different about that node. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, November 19, 2021 4:29 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Kurt, That usually means the process went away without telling the process manager (hydra) first. Those messages are from hydra when the connection to the process disconnected unexpectedly. Did all processes call `MPI_Finalize` before exit? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, November 19, 2021 at 2:26 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Thanks Hui, replacing MPI_LOGICAL with MPI_CXX_BOOL enabled the code to run without the segfault (using 4.0a2). Do you have any clue what this error message means? [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:206): assert (!closed) failed [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0a2/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip.c:169): de[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_ Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, November 15, 2021 9:44 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, Did you build mpich with Fortran disabled? MPI_LOGICAL is a Fortran datatype and is unavailable when Fortran binding is disabled. Try use MPI_C_BOOL instead. If you didn't disable cxx, you may also try use MPI_CXX_BOOL since you are programming in C++. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Saturday, November 13, 2021 2:56 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below. Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace. Program received signal SIGSEGV, Segmentation fault. MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); (gdb) where #0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410, displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410, array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 #6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0, array_of_types=0x7fffcaa24410, newtype=0x12853fc) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97 #8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, world_size=21) at src/NeedlesMpiManager.cpp:204 #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142 (gdb) Here is my code that creates the custom type and then calls MPI_Type_commit: MsgBasic obj; int struct_len = 8, i; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; i = 0; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; // the int array "start_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the int array "end_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the boolean "success_" member ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version displacements[i] = (size_t) &obj.success_ - (size_t) &obj; // the double "run_time_sec_" member ++i; block_len[i] = 1; types[i] = MPI_DOUBLE; displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &msg_data_type_); MPI_Type_commit(&msg_data_type_); Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Zhou, Hui 19 Nov '21

19 Nov '21

Kurt, That usually means the process went away without telling the process manager (hydra) first. Those messages are from hydra when the connection to the process disconnected unexpectedly. Did all processes call `MPI_Finalize` before exit? -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Friday, November 19, 2021 at 2:26 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Thanks Hui, replacing MPI_LOGICAL with MPI_CXX_BOOL enabled the code to run without the segfault (using 4.0a2). Do you have any clue what this error message means? [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:206): assert (!closed) failed [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0a2/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip.c:169): de[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_ Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, November 15, 2021 9:44 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, Did you build mpich with Fortran disabled? MPI_LOGICAL is a Fortran datatype and is unavailable when Fortran binding is disabled. Try use MPI_C_BOOL instead. If you didn't disable cxx, you may also try use MPI_CXX_BOOL since you are programming in C++. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Saturday, November 13, 2021 2:56 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below. Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace. Program received signal SIGSEGV, Segmentation fault. MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); (gdb) where #0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410, displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410, array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 #6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0, array_of_types=0x7fffcaa24410, newtype=0x12853fc) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97 #8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, world_size=21) at src/NeedlesMpiManager.cpp:204 #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142 (gdb) Here is my code that creates the custom type and then calls MPI_Type_commit: MsgBasic obj; int struct_len = 8, i; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; i = 0; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; // the int array "start_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the int array "end_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the boolean "success_" member ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version displacements[i] = (size_t) &obj.success_ - (size_t) &obj; // the double "run_time_sec_" member ++i; block_len[i] = 1; types[i] = MPI_DOUBLE; displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &msg_data_type_); MPI_Type_commit(&msg_data_type_); Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 19 Nov '21

19 Nov '21

Thanks Hui, replacing MPI_LOGICAL with MPI_CXX_BOOL enabled the code to run without the segfault (using 4.0a2). Do you have any clue what this error message means? [mpiexec(a)n022.cluster.com] control_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_cb.c:206): assert (!closed) failed [mpiexec(a)n022.cluster.com] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [mpiexec(a)n022.cluster.com] HYD_pmci_wait_for_completion (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:160): error waiting for event [mpiexec(a)n022.cluster.com] main (../../../../mpich-4.0a2/src/pm/hydra/ui/mpich/mpiexec.c:325): process manager error waiting for completion [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demux/demux_poll.c:76): callback returned error status [proxy:0:[email protected]] main (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip.c:169): de[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (../../../../mpich-4.0a2/src/pm/hydra/tools/demu[proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (../../../../mpich-4.0a2/src/pm/hydra/pm/pmiserv/pmip_cb.c:896): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_ Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, November 15, 2021 9:44 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, Did you build mpich with Fortran disabled? MPI_LOGICAL is a Fortran datatype and is unavailable when Fortran binding is disabled. Try use MPI_C_BOOL instead. If you didn't disable cxx, you may also try use MPI_CXX_BOOL since you are programming in C++. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Saturday, November 13, 2021 2:56 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below. Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace. Program received signal SIGSEGV, Segmentation fault. MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); (gdb) where #0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410, displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410, array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 #6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0, array_of_types=0x7fffcaa24410, newtype=0x12853fc) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97 #8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, world_size=21) at src/NeedlesMpiManager.cpp:204 #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142 (gdb) Here is my code that creates the custom type and then calls MPI_Type_commit: MsgBasic obj; int struct_len = 8, i; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; i = 0; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; // the int array "start_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the int array "end_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the boolean "success_" member ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version displacements[i] = (size_t) &obj.success_ - (size_t) &obj; // the double "run_time_sec_" member ++i; block_len[i] = 1; types[i] = MPI_DOUBLE; displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &msg_data_type_); MPI_Type_commit(&msg_data_type_); Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Join the MPICH BoF this evening at SC21
by Raffenetti, Ken 17 Nov '21

17 Nov '21

All, A reminder that the MPICH team will host a virtual Birds-of-a-Feather session tonight at 5:15pm CST as part of the SC21 proceedings. We hope to see you there to discuss the latest developments in MPICH, and to hear from partners Intel, HPE, ParTec, MVAPICH, and Cornelis Networks. See the link below for join info! https://sc21.hubb.me/fe/schedule-builder/sessions/879718 Cheers, The MPICH team

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Jeff Hammond 16 Nov '21

16 Nov '21

Unless MPICH configure has changed recently, —enable-g=debug enables debug symbols, and is all you need. Your choice is sufficient but overkill. It may introduce nontrivial performance overheads. Jeff On Sat, Nov 13, 2021 at 10:57 PM Mccall, Kurt E. (MSFC-EV41) via discuss < discuss(a)mpich.org> wrote: > Hui, > > > > I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to > “configure” so that debugging symbols would be present. The code is > crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this > stack trace below. Since MPICH 3.3.2, has there been changes in how custom > types are created (the code worked in 3.3.2)? I included my type-creating > code after the stack trace. > > > > Program received signal SIGSEGV, Segmentation fault. > > MPIR_Typerep_create_struct (count=count@entry=8, > array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, > > array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, > > array_of_types=array_of_types@entry=0x7fffcaa24410, > > newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) > > at > ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 > > 659 > MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); > > > > (gdb) where > > > > #0 MPIR_Typerep_create_struct (count=count@entry=8, > array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, > > array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, > > array_of_types=array_of_types@entry=0x7fffcaa24410, > > newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) > > at > ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 > > #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, > blocklength_array=blocklength_array@entry=0x128b6b0, > > displacement_array=displacement_array@entry=0x7fffcaa243c0, > oldtype_array=oldtype_array@entry=0x7fffcaa24410, > > newtype=newtype@entry=0x7fffcaa242dc) at > ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 > > #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, > oldtype_array=0x7fffcaa24410, > > displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, > count=8) > > at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 > > #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, > > displacement_array=displacement_array@entry=0x7fffcaa243c0, > oldtype_array=oldtype_array@entry=0x7fffcaa24410, > > newtype=newtype@entry=0x7fffcaa242dc) at > ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 > > #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, > > > array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, > > array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, > > array_of_types=array_of_types@entry=0x7fffcaa24410, > newtype=newtype@entry=0x12853fc) > > at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 > > #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, > array_of_types=0x7fffcaa24410, > > array_of_displacements=<optimized out>, > array_of_blocklengths=0x7fffcaa24440, count=8) > > at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 > > #6 PMPI_Type_create_struct (count=8, > array_of_blocklengths=0x7fffcaa24440, > array_of_displacements=0x7fffcaa243c0, > > array_of_types=0x7fffcaa24410, newtype=0x12853fc) > > at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 > > #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType > (this=0x12853fc) at src/MsgBasic.cpp:97 > > #8 0x0000000000412b77 in needles::NeedlesMpiManager::init > (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, > > world_size=21) at src/NeedlesMpiManager.cpp:204 > > #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at > src/NeedlesMpiManagerMain.cpp:142 > > (gdb) > > > > > > > > Here is my code that creates the custom type and then calls > MPI_Type_commit: > > > > MsgBasic obj; > > int struct_len = 8, i; > > > > int block_len[struct_len]; > > MPI_Datatype types[struct_len]; > > MPI_Aint displacements[struct_len]; > > > > i = 0; > > block_len[i] = 1; > > types[i] = MPI_LOGICAL; > > displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; > > > > ++i; > > block_len[i] = 1; > > types[i] = MPI_LOGICAL; > > displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; > > > > // the int array "start_" member > > ++i; > > block_len[i] = Tuple::N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.start_ - (size_t) &obj; > > > > // the int array "end_" member > > ++i; > > block_len[i] = Tuple::N_INDICES_MAX_; > > types[i] = MPI_SHORT; > > displacements[i] = (size_t) &obj.end_ - (size_t) &obj; > > > > // the integer "opcode_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_INT; > > displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; > > > > // the boolean "success_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version > > displacements[i] = (size_t) &obj.success_ - (size_t) &obj; > > > > // the double "run_time_sec_" member > > ++i; > > block_len[i] = 1; > > types[i] = MPI_DOUBLE; > > displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; > > > > // the char array "error_msg_" member > > ++i; > > block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; > > types[i] = MPI_CHAR; > > displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; > > > > MPI_Type_create_struct(struct_len, block_len, displacements, > > types, &msg_data_type_); > > MPI_Type_commit(&msg_data_type_); > > > > Thanks, > > Kurt > > > > *From:* Zhou, Hui <zhouh(a)anl.gov> > *Sent:* Sunday, October 24, 2021 6:46 PM > *To:* discuss(a)mpich.org > *Cc:* Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > *Subject:* [EXTERNAL] Re: Maximum number of inter-communicators? > > > > Hi Kurt, > > > > There is indeed a limit on maximum number of communicators that you can > have, including both intra communicators and inter-communicators. Try free > the communicators that you no longer need. In older version of MPICH, there > may be additional limit on how many dynamic processes one can connect. If > you still hit crash after making sure there isn't too many simultaneous > active communicators, could you try the latest release -- > http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz > <https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, > and see if the issue persist? > > > > -- > > Hui > ------------------------------ > > *From:* Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> > *Sent:* Sunday, October 24, 2021 2:37 PM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> > *Subject:* [mpich-discuss] Maximum number of inter-communicators? > > > > Hi, > > > > Based on a paper I read about giving an MPI job some fault tolerance, I’m > exclusively connecting my processes with inter-communicators. > > I’ve found that if I increase the number of processes beyond a certain > point, many processes don’t get created at all and the whole job > > crashes. Am I running up against an operating system limit (like the > number of open file descriptors – it is set at 1024), or some sort of > > MPICH limit? > > > > If it matters, my process architecture (a tree) is as follows: one > master process connected to 21 manager processes on 21 other nodes, > > and each manager connected to 8 worker processes on the manager’s own > node. This is the largest job I’ve been able to create > > without it crashing. Attempting to increase the number of workers > beyond 8 results in a crash. > > > > I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the > Portland Group compiler pgc++ 19.5-0. > > > > Thanks, > > Kurt > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Zhou, Hui 15 Nov '21

15 Nov '21

Hi Kurt, Did you build mpich with Fortran disabled? MPI_LOGICAL is a Fortran datatype and is unavailable when Fortran binding is disabled. Try use MPI_C_BOOL instead. If you didn't disable cxx, you may also try use MPI_CXX_BOOL since you are programming in C++. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Saturday, November 13, 2021 2:56 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] Maximum number of inter-communicators? Hui, I built MPICH 4.0a2 with gcc 4.8.5, and included the -enable-g=all flag to “configure” so that debugging symbols would be present. The code is crashing my call to MPI_Type_commit, in libpthreads.so. gdb give this stack trace below. Since MPICH 3.3.2, has there been changes in how custom types are created (the code worked in 3.3.2)? I included my type-creating code after the stack trace. Program received signal SIGSEGV, Segmentation fault. MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 659 MPIR_Ensure_Aint_fits_in_int(old_dtp->builtin_element_size); (gdb) where #0 MPIR_Typerep_create_struct (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x128b6b0, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fec8bedd258 <MPIR_Datatype_direct+1400>) at ../mpich-4.0a2/src/mpi/datatype/typerep/src/typerep_dataloop_create.c:659 #1 0x00007fec8b9b2608 in type_struct (count=count@entry=8, blocklength_array=blocklength_array@entry=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:206 #2 0x00007fec8b9b4b9e in type_struct (newtype=0x7fffcaa242dc, oldtype_array=0x7fffcaa24410, displacement_array=0x7fffcaa243c0, blocklength_array=0x128b6b0, count=8) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:227 #3 MPIR_Type_struct (count=count@entry=8, blocklength_array=0x128b6b0, displacement_array=displacement_array@entry=0x7fffcaa243c0, oldtype_array=oldtype_array@entry=0x7fffcaa24410, newtype=newtype@entry=0x7fffcaa242dc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:235 #4 0x00007fec8b9b7b08 in MPIR_Type_create_struct_impl (count=count@entry=8, array_of_blocklengths=array_of_blocklengths@entry=0x7fffcaa24440, array_of_displacements=array_of_displacements@entry=0x7fffcaa243c0, array_of_types=array_of_types@entry=0x7fffcaa24410, newtype=newtype@entry=0x12853fc) at ../mpich-4.0a2/src/mpi/datatype/type_create.c:908 #5 0x00007fec8b85ad26 in internal_Type_create_struct (newtype=0x12853fc, array_of_types=0x7fffcaa24410, array_of_displacements=<optimized out>, array_of_blocklengths=0x7fffcaa24440, count=8) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:79 #6 PMPI_Type_create_struct (count=8, array_of_blocklengths=0x7fffcaa24440, array_of_displacements=0x7fffcaa243c0, array_of_types=0x7fffcaa24410, newtype=0x12853fc) at ../mpich-4.0a2/src/binding/c/datatype/type_create_struct.c:164 #7 0x0000000000438dfb in needles::MpiMsgBasic::createMsgDataType (this=0x12853fc) at src/MsgBasic.cpp:97 #8 0x0000000000412b77 in needles::NeedlesMpiManager::init (this=0x12853a0, argc=23, argv=0x7fffcaa24e08, rank=20, world_size=21) at src/NeedlesMpiManager.cpp:204 #9 0x000000000040605f in main (argc=23, argv=0x7fffcaa24e08) at src/NeedlesMpiManagerMain.cpp:142 (gdb) Here is my code that creates the custom type and then calls MPI_Type_commit: MsgBasic obj; int struct_len = 8, i; int block_len[struct_len]; MPI_Datatype types[struct_len]; MPI_Aint displacements[struct_len]; i = 0; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_valid_ - (size_t) &obj; ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; displacements[i] = (size_t) &obj.tuple_seq_valid_ - (size_t) &obj; // the int array "start_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.start_ - (size_t) &obj; // the int array "end_" member ++i; block_len[i] = Tuple::N_INDICES_MAX_; types[i] = MPI_SHORT; displacements[i] = (size_t) &obj.end_ - (size_t) &obj; // the integer "opcode_" member ++i; block_len[i] = 1; types[i] = MPI_INT; displacements[i] = (size_t) &obj.opcode_ - (size_t) &obj; // the boolean "success_" member ++i; block_len[i] = 1; types[i] = MPI_LOGICAL; // NOTE: might be MPI_BOOLEAN in later version displacements[i] = (size_t) &obj.success_ - (size_t) &obj; // the double "run_time_sec_" member ++i; block_len[i] = 1; types[i] = MPI_DOUBLE; displacements[i] = (size_t) &obj.run_time_sec_ - (size_t) &obj; // the char array "error_msg_" member ++i; block_len[i] = NeedlesMpi::ERROR_MSG_LEN_ + 1; types[i] = MPI_CHAR; displacements[i] = (size_t) &obj.error_msg_[0] - (size_t) &obj; MPI_Type_create_struct(struct_len, block_len, displacements, types, &msg_data_type_); MPI_Type_commit(&msg_data_type_); Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0