September 2023 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Core dump in parse_args
by kumar.tarun＠siemens.com 22 Sep '23

22 Sep '23

I tried -configfile option and it further crashed with a memory failure. The configfile itself has 1199 words/23777 characters. Overall it has 4 lines for 4 partitions. *** Error in `vmpiexec': double free or corruption (out): 0x00000000007001e0 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x81329)[0x7ffff7206329] vmpiexec[0x4262a9] vmpiexec[0x4220ed] vmpiexec[0x409d23] vmpiexec[0x4035da] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7ffff71a7555] vmpiexec[0x404c05] ======= Memory map: ======== 00400000-0047c000 r-xp 00000000 00:77 2085572302 ./mpich2/linux_x86_64/bin/mpiexec.hydra 0067b000-0067e000 rw-p 0007b000 00:77 2085572302 ./mpich2/linux_x86_64/bin/mpiexec.hydra 0067e000-00bd9000 rw-p 00000000 00:00 0 [heap] 7ffff0000000-7ffff0021000 rw-p 00000000 00:00 0 7ffff0021000-7ffff4000000 ---p 00000000 00:00 0 7ffff6b45000-7ffff6b6a000 r-xp 00000000 08:02 45522 /usr/lib64/liblzma.so.5.2.2 7ffff6b6a000-7ffff6d69000 ---p 00025000 08:02 45522 /usr/lib64/liblzma.so.5.2.2 7ffff6d69000-7ffff6d6a000 r--p 00024000 08:02 45522 /usr/lib64/liblzma.so.5.2.2 7ffff6d6a000-7ffff6d6b000 rw-p 00025000 08:02 45522 /usr/lib64/liblzma.so.5.2.2 7ffff6d6b000-7ffff6d80000 r-xp 00000000 08:02 447697 /usr/lib64/libz.so.1.2.7 7ffff6d80000-7ffff6f7f000 ---p 00015000 08:02 447697 /usr/lib64/libz.so.1.2.7 7ffff6f7f000-7ffff6f80000 r--p 00014000 08:02 447697 /usr/lib64/libz.so.1.2.7 7ffff6f80000-7ffff6f81000 rw-p 00015000 08:02 447697 /usr/lib64/libz.so.1.2.7 7ffff6f81000-7ffff6f83000 r-xp 00000000 08:02 45003 /usr/lib64/libdl-2.17.so 7ffff6f83000-7ffff7183000 ---p 00002000 08:02 45003 /usr/lib64/libdl-2.17.so 7ffff7183000-7ffff7184000 r--p 00002000 08:02 45003 /usr/lib64/libdl-2.17.so Regards Tarun From: Kumar, Tarun (DI SW ICS DVT RD QSCE) Sent: Monday, September 18, 2023 3:16 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org; Thakur, Rajeev <thakur(a)anl.gov> Cc: Raffenetti, Ken <raffenet(a)anl.gov> Subject: RE: [mpich-discuss] Core dump in parse_args Thanks. Good to know. I will explore further. Regards Tarun From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, September 18, 2023 2:34 PM To: discuss(a)mpich.org<mailto:[email protected]>; Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> Cc: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>>; Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args Hydra supports -configfile {name} option, so one could try putting all his commandline arguments in a file and pass that to hydra. Hui ________________________________ From: Raffenetti, Ken via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, September 18, 2023 4:22 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>; Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> Cc: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>>; kumar.tarun(a)siemens.com<mailto:[email protected]> <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args I’m not aware of a hard-coded limit, but I am able to reproduce a segfault reliably by passing 1000 arguments a toy executable. Will update with more info when we have it. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:57 PM To: "Thakur, Rajeev" <thakur(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args Thanks Rajeev for your reply. I don’t have the exact number as of now. It’s definitely more than 1000. Is there a hard coded limit? Regards Tarun From: Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> Sent: Friday, September 15, 2023 2:40 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don’t see the crash. Is it a known issue? Regards Tarun

1 0

Re: [mpich-discuss] [EXTERNAL] Re: MPI_alloc_mem call code compiles with Open MPI, not MPICH
by Zhou, Hui 22 Sep '23

22 Sep '23

I created issue https://github.com/pmodels/mpich/issues/6691 to track this issue. Hui ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Sent: Friday, September 22, 2023 11:02 AM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [EXTERNAL] Re: MPI_alloc_mem call code compiles with Open MPI, not MPICH Hui, The option flag doesn't seem to help: $ mpifort -fallow-argument-mismatch support_for_mpi_alloc_mem_cptr.F90 support_for_mpi_alloc_mem_cptr.F90:9:53: 9 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) I did test and, yes, mpi_f08 does let it work. I think the issue is I'm not sure the library this is in can assume mpi_f08 support for all supported compilers. For example, I know NAG + Open MPI does not build mpi_f08. I haven't tried NAG + MPICH...if that does support it, maybe we could move there? I did look around online, and the MPI Forum does sort of say my code should be supported: https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node216.htm "If the Fortran compiler provides TYPE(C_PTR), then the following generic interface must be provided in the mpi module and should be provided in mpif.h through overloading, i.e., with the same routine name as the routine with INTEGER(KIND=MPI_ADDRESS_KIND) BASEPTR, but with a different specific procedure name ... The base procedure name of this overloaded function is MPI_ALLOC_MEM_CPTR" If I look at the Open MPI source code, I do see "MPI_alloc_mem_cptr" there: https://github.com/open-mpi/ompi/blob/main/ompi/mpi/fortran/use-mpi-ignore-… but not in MPICH's. Now, I tried changing it to use MPI_ADDRESS_KIND like on the page above (thinking maybe that was it) but: $ cat support_for_mpi_alloc_mem_cptr.address_kind.F90 program main use mpi use iso_c_binding, only: C_PTR integer(kind=MPI_ADDRESS_KIND) :: sz type (c_ptr) :: ptr call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) end program main $ mpifort support_for_mpi_alloc_mem_cptr.address_kind.F90 support_for_mpi_alloc_mem_cptr.address_kind.F90:8:53: 8 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Friday, September 22, 2023 at 11:29 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: MPI_alloc_mem call code compiles with Open MPI, not MPICH Hi Matt, Are you able to test `use mpi_f08` instead? We'll open an issue to track this for "use mpi". In the meantime, you may be able to bypass this issue by adding the `-fallow-argument-mismatch flag to mpifort. Hui ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Friday, September 22, 2023 9:11 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] MPI_alloc_mem call code compiles with Open MPI, not MPICH All, I'm sort of new to MPICH and I'm encountering one call that works with Open MPI but not MPICH. As such, I'm wondering if I built MPICH incorrectly or incompletely (always easy to miss a configure flag!). To wit, I have a code like: program main use mpi use iso_fortran_env, only: INT64 use iso_c_binding, only: C_PTR integer(kind=INT64) :: sz type (c_ptr) :: ptr call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) end program main If I compile with GCC 12.1 + Open MPI 4.1.3, all is well: $ mpifort --showme:version mpifort: Open MPI 4.1.3 (Language: Fortran) mathomp4@discover23 ~/MPITests/CPtr master ?25 $ mpifort support_for_mpi_alloc_mem_cptr.F90 mathomp4@discover23 ~/MPITests/CPtr master ?25 $ echo $? 0 I'll also say Intel MPI builds this code as well. But with MPICH 4.1.2 I just built: $ mpifort support_for_mpi_alloc_mem_cptr.F90 support_for_mpi_alloc_mem_cptr.F90:9:53: 9 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) Now, my configure lines to Open MPI and MPICH are pretty boring, so the resulting mpiforts are as well: $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/include -pthread -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -L/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -L/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -lmpifort -Wl,-rpath -Wl,/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -Wl,--enable-new-dtags -lmpi But I see Open MPI has that "mpi_usempi_ignore_tkr", is it possible that's what is doing it? Perhaps we just need to recode the code (using transfer, etc?) Thanks for any help, Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] [EXTERNAL] Re: MPI_alloc_mem call code compiles with Open MPI, not MPICH
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 22 Sep '23

22 Sep '23

Hui, The option flag doesn't seem to help: $ mpifort -fallow-argument-mismatch support_for_mpi_alloc_mem_cptr.F90 support_for_mpi_alloc_mem_cptr.F90:9:53: 9 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) I did test and, yes, mpi_f08 does let it work. I think the issue is I'm not sure the library this is in can assume mpi_f08 support for all supported compilers. For example, I know NAG + Open MPI does not build mpi_f08. I haven't tried NAG + MPICH...if that does support it, maybe we could move there? I did look around online, and the MPI Forum does sort of say my code should be supported: https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node216.htm "If the Fortran compiler provides TYPE(C_PTR), then the following generic interface must be provided in the mpi module and should be provided in mpif.h through overloading, i.e., with the same routine name as the routine with INTEGER(KIND=MPI_ADDRESS_KIND) BASEPTR, but with a different specific procedure name ... The base procedure name of this overloaded function is MPI_ALLOC_MEM_CPTR" If I look at the Open MPI source code, I do see "MPI_alloc_mem_cptr" there: https://github.com/open-mpi/ompi/blob/main/ompi/mpi/fortran/use-mpi-ignore-… but not in MPICH's. Now, I tried changing it to use MPI_ADDRESS_KIND like on the page above (thinking maybe that was it) but: $ cat support_for_mpi_alloc_mem_cptr.address_kind.F90 program main use mpi use iso_c_binding, only: C_PTR integer(kind=MPI_ADDRESS_KIND) :: sz type (c_ptr) :: ptr call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) end program main $ mpifort support_for_mpi_alloc_mem_cptr.address_kind.F90 support_for_mpi_alloc_mem_cptr.address_kind.F90:8:53: 8 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Friday, September 22, 2023 at 11:29 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC]" <matthew.thompson(a)nasa.gov> Subject: [EXTERNAL] Re: MPI_alloc_mem call code compiles with Open MPI, not MPICH Hi Matt, Are you able to test `use mpi_f08` instead? We'll open an issue to track this for "use mpi". In the meantime, you may be able to bypass this issue by adding the `-fallow-argument-mismatch flag to mpifort. Hui ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Friday, September 22, 2023 9:11 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] MPI_alloc_mem call code compiles with Open MPI, not MPICH All, I'm sort of new to MPICH and I'm encountering one call that works with Open MPI but not MPICH. As such, I'm wondering if I built MPICH incorrectly or incompletely (always easy to miss a configure flag!). To wit, I have a code like: program main use mpi use iso_fortran_env, only: INT64 use iso_c_binding, only: C_PTR integer(kind=INT64) :: sz type (c_ptr) :: ptr call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) end program main If I compile with GCC 12.1 + Open MPI 4.1.3, all is well: $ mpifort --showme:version mpifort: Open MPI 4.1.3 (Language: Fortran) mathomp4@discover23 ~/MPITests/CPtr master ?25 $ mpifort support_for_mpi_alloc_mem_cptr.F90 mathomp4@discover23 ~/MPITests/CPtr master ?25 $ echo $? 0 I'll also say Intel MPI builds this code as well. But with MPICH 4.1.2 I just built: $ mpifort support_for_mpi_alloc_mem_cptr.F90 support_for_mpi_alloc_mem_cptr.F90:9:53: 9 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) Now, my configure lines to Open MPI and MPICH are pretty boring, so the resulting mpiforts are as well: $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/include -pthread -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -L/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -L/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -lmpifort -Wl,-rpath -Wl,/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -Wl,--enable-new-dtags -lmpi But I see Open MPI has that "mpi_usempi_ignore_tkr", is it possible that's what is doing it? Perhaps we just need to recode the code (using transfer, etc?) Thanks for any help, Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] MPI_alloc_mem call code compiles with Open MPI, not MPICH
by Zhou, Hui 22 Sep '23

22 Sep '23

Hi Matt, Are you able to test `use mpi_f08` instead? We'll open an issue to track this for "use mpi". In the meantime, you may be able to bypass this issue by adding the `-fallow-argument-mismatch flag to mpifort. Hui ________________________________ From: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] via discuss <discuss(a)mpich.org> Sent: Friday, September 22, 2023 9:11 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] <matthew.thompson(a)nasa.gov> Subject: [mpich-discuss] MPI_alloc_mem call code compiles with Open MPI, not MPICH All, I'm sort of new to MPICH and I'm encountering one call that works with Open MPI but not MPICH. As such, I'm wondering if I built MPICH incorrectly or incompletely (always easy to miss a configure flag!). To wit, I have a code like: program main use mpi use iso_fortran_env, only: INT64 use iso_c_binding, only: C_PTR integer(kind=INT64) :: sz type (c_ptr) :: ptr call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) end program main If I compile with GCC 12.1 + Open MPI 4.1.3, all is well: $ mpifort --showme:version mpifort: Open MPI 4.1.3 (Language: Fortran) mathomp4@discover23 ~/MPITests/CPtr master ?25 $ mpifort support_for_mpi_alloc_mem_cptr.F90 mathomp4@discover23 ~/MPITests/CPtr master ?25 $ echo $? 0 I'll also say Intel MPI builds this code as well. But with MPICH 4.1.2 I just built: $ mpifort support_for_mpi_alloc_mem_cptr.F90 support_for_mpi_alloc_mem_cptr.F90:9:53: 9 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) Now, my configure lines to Open MPI and MPICH are pretty boring, so the resulting mpiforts are as well: $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/include -pthread -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -L/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -L/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -lmpifort -Wl,-rpath -Wl,/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -Wl,--enable-new-dtags -lmpi But I see Open MPI has that "mpi_usempi_ignore_tkr", is it possible that's what is doing it? Perhaps we just need to recode the code (using transfer, etc?) Thanks for any help, Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

MPI_alloc_mem call code compiles with Open MPI, not MPICH
by Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] 22 Sep '23

22 Sep '23

All, I'm sort of new to MPICH and I'm encountering one call that works with Open MPI but not MPICH. As such, I'm wondering if I built MPICH incorrectly or incompletely (always easy to miss a configure flag!). To wit, I have a code like: program main use mpi use iso_fortran_env, only: INT64 use iso_c_binding, only: C_PTR integer(kind=INT64) :: sz type (c_ptr) :: ptr call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) end program main If I compile with GCC 12.1 + Open MPI 4.1.3, all is well: $ mpifort --showme:version mpifort: Open MPI 4.1.3 (Language: Fortran) mathomp4@discover23 ~/MPITests/CPtr master ?25 $ mpifort support_for_mpi_alloc_mem_cptr.F90 mathomp4@discover23 ~/MPITests/CPtr master ?25 $ echo $? 0 I'll also say Intel MPI builds this code as well. But with MPICH 4.1.2 I just built: $ mpifort support_for_mpi_alloc_mem_cptr.F90 support_for_mpi_alloc_mem_cptr.F90:9:53: 9 | call MPI_Alloc_mem(sz, MPI_INFO_NULL, ptr, ierror) | 1 Error: Type mismatch in argument 'baseptr' at (1); passed TYPE(c_ptr) to INTEGER(8) Now, my configure lines to Open MPI and MPICH are pretty boring, so the resulting mpiforts are as well: $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/include -pthread -I/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -L/discover/swdev/gmao_SIteam/MPI/openmpi/4.1.3/gcc-12.1.0/lib -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi $ mpifort -show gfortran -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -I/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/include -L/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -lmpifort -Wl,-rpath -Wl,/discover/swdev/gmao_SIteam/MPI/mpich/4.1.2/gcc-12.1.0/lib -Wl,--enable-new-dtags -lmpi But I see Open MPI has that "mpi_usempi_ignore_tkr", is it possible that's what is doing it? Perhaps we just need to recode the code (using transfer, etc?) Thanks for any help, Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson

1 0

Re: [mpich-discuss] Core dump in parse_args
by kumar.tarun＠siemens.com 18 Sep '23

18 Sep '23

Thanks for that. Regards Tarun From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Monday, September 18, 2023 2:44 PM To: discuss(a)mpich.org; Thakur, Rajeev <thakur(a)anl.gov> Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args Hi Tarun, We did find that there is a hardcoded limit of ~1000 command line arguments. If you want a temporary workaround, you can increase the value of this define in the MPICH source to something larger: #define HYD_NUM_TMP_STRINGS 1000 I created https://github.com/pmodels/mpich/issues/6681 to track progress on a more robust fix. Ken From: "Raffenetti, Ken via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Monday, September 18, 2023 at 4:23 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>>, "Thakur, Rajeev" <thakur(a)anl.gov<mailto:[email protected]>> Cc: "Raffenetti, Ken" <raffenet(a)anl.gov<mailto:[email protected]>>, "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args I'm not aware of a hard-coded limit, but I am able to reproduce a segfault reliably by passing 1000 arguments a toy executable. Will update with more info when we have it. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:57 PM To: "Thakur, Rajeev" <thakur(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args Thanks Rajeev for your reply. I don't have the exact number as of now. It's definitely more than 1000. Is there a hard coded limit? Regards Tarun From: Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> Sent: Friday, September 15, 2023 2:40 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don't see the crash. Is it a known issue? Regards Tarun

1 0

Re: [mpich-discuss] Core dump in parse_args
by kumar.tarun＠siemens.com 18 Sep '23

18 Sep '23

Thanks. Good to know. I will explore further. Regards Tarun From: Zhou, Hui <zhouh(a)anl.gov> Sent: Monday, September 18, 2023 2:34 PM To: discuss(a)mpich.org; Thakur, Rajeev <thakur(a)anl.gov> Cc: Raffenetti, Ken <raffenet(a)anl.gov>; Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args Hydra supports -configfile {name} option, so one could try putting all his commandline arguments in a file and pass that to hydra. Hui ________________________________ From: Raffenetti, Ken via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, September 18, 2023 4:22 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>>; Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> Cc: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>>; kumar.tarun(a)siemens.com<mailto:[email protected]> <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args I’m not aware of a hard-coded limit, but I am able to reproduce a segfault reliably by passing 1000 arguments a toy executable. Will update with more info when we have it. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:57 PM To: "Thakur, Rajeev" <thakur(a)anl.gov<mailto:[email protected]>>, "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args Thanks Rajeev for your reply. I don’t have the exact number as of now. It’s definitely more than 1000. Is there a hard coded limit? Regards Tarun From: Thakur, Rajeev <thakur(a)anl.gov<mailto:[email protected]>> Sent: Friday, September 15, 2023 2:40 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Core dump in parse_args How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don’t see the crash. Is it a known issue? Regards Tarun

1 0

Re: [mpich-discuss] Core dump in parse_args
by Raffenetti, Ken 18 Sep '23

18 Sep '23

Hi Tarun, We did find that there is a hardcoded limit of ~1000 command line arguments. If you want a temporary workaround, you can increase the value of this define in the MPICH source to something larger: #define HYD_NUM_TMP_STRINGS 1000 I created https://github.com/pmodels/mpich/issues/6681 to track progress on a more robust fix. Ken From: "Raffenetti, Ken via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Monday, September 18, 2023 at 4:23 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org>, "Thakur, Rajeev" <thakur(a)anl.gov> Cc: "Raffenetti, Ken" <raffenet(a)anl.gov>, "kumar.tarun(a)siemens.com" <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args I’m not aware of a hard-coded limit, but I am able to reproduce a segfault reliably by passing 1000 arguments a toy executable. Will update with more info when we have it. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, September 15, 2023 at 4:57 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "kumar.tarun(a)siemens.com" <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args Thanks Rajeev for your reply. I don’t have the exact number as of now. It’s definitely more than 1000. Is there a hard coded limit? Regards Tarun From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Friday, September 15, 2023 2:40 PM To: discuss(a)mpich.org Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don’t see the crash. Is it a known issue? Regards Tarun

1 0

Re: [mpich-discuss] Core dump in parse_args
by Zhou, Hui 18 Sep '23

18 Sep '23

Hydra supports -configfile {name} option, so one could try putting all his commandline arguments in a file and pass that to hydra. Hui ________________________________ From: Raffenetti, Ken via discuss <discuss(a)mpich.org> Sent: Monday, September 18, 2023 4:22 PM To: discuss(a)mpich.org <discuss(a)mpich.org>; Thakur, Rajeev <thakur(a)anl.gov> Cc: Raffenetti, Ken <raffenet(a)anl.gov>; kumar.tarun(a)siemens.com <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args I’m not aware of a hard-coded limit, but I am able to reproduce a segfault reliably by passing 1000 arguments a toy executable. Will update with more info when we have it. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, September 15, 2023 at 4:57 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "kumar.tarun(a)siemens.com" <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args Thanks Rajeev for your reply. I don’t have the exact number as of now. It’s definitely more than 1000. Is there a hard coded limit? Regards Tarun From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Friday, September 15, 2023 2:40 PM To: discuss(a)mpich.org Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don’t see the crash. Is it a known issue? Regards Tarun

1 0

Re: [mpich-discuss] Core dump in parse_args
by Raffenetti, Ken 18 Sep '23

18 Sep '23

I’m not aware of a hard-coded limit, but I am able to reproduce a segfault reliably by passing 1000 arguments a toy executable. Will update with more info when we have it. Ken From: "kumar.tarun--- via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, September 15, 2023 at 4:57 PM To: "Thakur, Rajeev" <thakur(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "kumar.tarun(a)siemens.com" <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args Thanks Rajeev for your reply. I don’t have the exact number as of now. It’s definitely more than 1000. Is there a hard coded limit? Regards Tarun From: Thakur, Rajeev <thakur(a)anl.gov> Sent: Friday, September 15, 2023 2:40 PM To: discuss(a)mpich.org Cc: Kumar, Tarun (DI SW ICS DVT RD QSCE) <kumar.tarun(a)siemens.com> Subject: Re: [mpich-discuss] Core dump in parse_args How many arguments? Rajeev From: "kumar.tarun--- via discuss" <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Friday, September 15, 2023 at 4:22 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "kumar.tarun(a)siemens.com<mailto:[email protected]>" <kumar.tarun(a)siemens.com<mailto:[email protected]>> Subject: [mpich-discuss] Core dump in parse_args Hi, I recently encountered this crash where mpiexec when executed from a bash script and with a large number of arguments crashes. The core dump is as follows: #0 0x0000000000408ec3 in parse_args () #1 0x0000000000409f26 in HYD_uii_mpx_get_parameters () #2 0x000000000040397a in main () If I reduce the number of arguments then I don’t see the crash. Is it a known issue? Regards Tarun

1 0