- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Receiving fortran-related errors when compiling MPICH 3.3.2
by Zhou, Hui 08 Jul '20

08 Jul '20

Hi Arham, I think you have multiple MPI installations mixing up. Please double check that you are using the correct `mpicc` and check the results with `ldd`. -- Hui Zhou From: Arham Amouei via discuss <discuss(a)mpich.org> Reply-To: Arham Amouei <arham.amouei(a)protonmail.com>, "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, July 8, 2020 at 10:48 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Arham Amouei <arham.amouei(a)protonmail.com> Subject: [mpich-discuss] Receiving fortran-related errors when compiling MPICH 3.3.2 Hello. Despite I have disabled fortran by --disable-fortran, I receive errors related to fortran when compiling MPICH 3.3.2. Example: .../linux/mpi/intel64/lib/libmpifort.so.12: undefined reference to `PMPI_Type_create_f90_complex' How can I get rid of those errors without changing other installed softwares? Please find attached m.txt and c.txt files. Kind regards, Arham

1 0

Receiving fortran-related errors when compiling MPICH 3.3.2
by Arham Amouei 08 Jul '20

08 Jul '20

Hello. Despite I have disabled fortran by --disable-fortran, I receive errors related to fortran when compiling MPICH 3.3.2. Example: .../linux/mpi/intel64/lib/libmpifort.so.12: undefined reference to `PMPI_Type_create_f90_complex' How can I get rid of those errors without changing other installed softwares? Please find attached m.txt and c.txt files. Kind regards, Arham

1 0

Re: [mpich-discuss] Multi NIC configuration question
by Balaji, Pavan 06 Jul '20

06 Jul '20

Sudhakar, That depends on the network and on whether the underlying communication library we are using supports it. We'd need more details on what network you are using. -- Pavan On 6/24/20, 5:50 PM, "Sudhakar D via discuss" <discuss(a)mpich.org> wrote: Hello, Does MPICH support multiple interfaces for each node ? For example, if I have two NIC cards on the local and remote system, can I load balance across all the interfaces ? If yes, is there any documentation to configure the MPICH to use all the interfaces ? regards, Sudhakar

1 0

Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE
by Zhou, Hui 02 Jul '20

02 Jul '20

Hi Patrick, Thanks for referring the `mpi4pi` issue. I am cross linking it to https://github.com/pmodels/mpich/issues/4130. The problem is how FORTRAN common blocks works, or how the way it works in a non-standard way. When libmpifort.so is linked in later, it creates separate common blocks that are disconnected from the previous one. This is probably due to the fact that MPICH still embraces the F77 compatibility. So rather than saying this is not an MPICH issue, it is an issue that there isn’t simple fix currently. -- Hui Zhou From: Patrick McNally via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, July 2, 2020 at 5:34 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Patrick McNally <rpmcnally(a)gmail.com> Subject: Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE Thank you both for the follow up. I did engage with Lisandro first, which you can find here: https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks…. The short version is that this isn't an mpi4py issue either. For my particular case, mpi4py could possibly fix it by linking to the Fortran bindings even though it doesn't need or use them, but I think that focusing on mpi4py misses the larger point. As the standalone test files he provided show, this issue can definitely happen without mpi4py in the loop. It would appear that using any 3rd party Python/C library that uses MPI (or custom Python bindings to the MPI C library) would cause later invocation of Fortran code to behave incorrectly. Having the Fortran accessible C wrappers testing the buffer against the address of a variable defined in another library seems odd to me, but I don't understand your codebase well enough to propose an alternative. I would say that even if this is a problem that the end user needs to fix, it would be nice if there were some user-facing documentation to explain and provide a remedy for the problem. -Patrick On Wed, Jul 1, 2020 at 8:40 PM Jeff Hammond via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Has Lisandro provided any feedback on this anywhere? I agree with Hui. This appears to be an artifact of the way MPI uses magic values for MPI_IN_PLACE and how those get encoded in binaries. I naively speculate that MPICH could encode the magic values differently to work around this, but that would likely break ABI compatibility, at least on the Fortran side. Jeff On Wed, Jun 10, 2020 at 3:31 PM Zhou, Hui via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: I am not sure this is an issue for mpich. Seems more to be an issue of `mpi4py`. The issue is exactly as you suspected -- `libmpifort.so` need be loaded before `libmpi.so`, or the external symbol in `libmpifort.so` won’t get resolved. > It works if you load something linked to libmpifort first or load everything with RTLD_GLOBAL. Seems plausible. I don’t have any better idea other than fixing `mpi4py` so it always load `libmpifort.so` first. -- Hui Zhou From: Patrick McNally via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Wednesday, June 10, 2020 at 1:09 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: Patrick McNally <rpmcnally(a)gmail.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE I hate to bug, but this is a pretty serious issue. I suspect it is why we get segfaults trying to use similar variables like MPI_STATUSES_IGNORE. Any insight would be appreciated. Thanks, Patrick On Wed, May 27, 2020 at 10:25 AM Patrick McNally <rpmcnally(a)gmail.com<mailto:[email protected]>> wrote: Our application consists primarily of a Python head calling into Fortran routines to do the heavy lifting. We have never been able to successfully use MPI_IN_PLACE in Fortran but weren't sure why. Recently, we discovered that it works fine in standalone Fortran code and is only broken when the Fortran code is run through our Python modules. The issue appears to be related to having code that only links to the C libmpi library loaded first and with RTLD_LOCAL, as happens when we load mpi4py. It works if you load something linked to libmpifort first or load everything with RTLD_GLOBAL. I'm assuming this has something to do with how MPICH tests the address of MPIR_F08_MPI_IN_PLACE but I don't understand SO loading well enough to fully grasp the issue. Below is some standalone code to show the issue. I'd appreciate any insight you can provide into why this is happening. Relevant system details: RHEL 7.8 Python 2.7 GCC 7.3.0 MPICH 3.3.2 (and 3.2) The below files are also available towards the end of the bug report at the following link: https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks… Thanks, Patrick makefile ----------- libs = testc.so testf.so all: $(libs) testc.so: testc.c mpicc -shared -fPIC $< -o $@ testf.so: testf.f90 mpifort -shared -fPIC $< -o $@ clean: $(RM) $(libs) testc.c --------- #include <stddef.h> #include <stdio.h> #include <mpi.h> extern void initc(void); extern void testc(void); void initc(void) { MPI_Init(NULL,NULL); } void testc(void) { int val = 1; MPI_Allreduce(MPI_IN_PLACE, &val, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); printf("C val: %2d\n",val); } testf.f90 ----------- subroutine initf() bind(C) use mpi integer ierr call MPI_Init(ierr) end subroutine initf subroutine testf() bind(C) use mpi integer ierr integer val val = 1 call MPI_Allreduce(MPI_IN_PLACE, val, 1, MPI_INTEGER, MPI_SUM, MPI_COMM_WORLD, ierr) print '(A,I2)', 'F val: ', val end subroutine testf test.py --------- from ctypes import CDLL, RTLD_LOCAL, RTLD_GLOBAL mode = RTLD_LOCAL cfirst = True if cfirst: # it does not work! libc = CDLL("./testc.so", mode) libf = CDLL("./testf.so", mode) else: # it works! libf = CDLL("./testf.so", mode) libc = CDLL("./testc.so", mode) libc.initc.restype = None libc.testc.argtypes = [] libf.initf.restype = None libf.testf.argtypes = [] libc.initc() libc.testc() libf.testf() _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE
by Patrick McNally 02 Jul '20

02 Jul '20

Thank you both for the follow up. I did engage with Lisandro first, which you can find here: https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks…. The short version is that this isn't an mpi4py issue either. For my particular case, mpi4py could possibly fix it by linking to the Fortran bindings even though it doesn't need or use them, but I think that focusing on mpi4py misses the larger point. As the standalone test files he provided show, this issue can definitely happen without mpi4py in the loop. It would appear that using any 3rd party Python/C library that uses MPI (or custom Python bindings to the MPI C library) would cause later invocation of Fortran code to behave incorrectly. Having the Fortran accessible C wrappers testing the buffer against the address of a variable defined in another library seems odd to me, but I don't understand your codebase well enough to propose an alternative. I would say that even if this is a problem that the end user needs to fix, it would be nice if there were some user-facing documentation to explain and provide a remedy for the problem. -Patrick On Wed, Jul 1, 2020 at 8:40 PM Jeff Hammond via discuss <discuss(a)mpich.org> wrote: > Has Lisandro provided any feedback on this anywhere? I agree with Hui. > This appears to be an artifact of the way MPI uses magic values for > MPI_IN_PLACE and how those get encoded in binaries. I naively speculate > that MPICH could encode the magic values differently to work around this, > but that would likely break ABI compatibility, at least on the Fortran side. > > Jeff > > On Wed, Jun 10, 2020 at 3:31 PM Zhou, Hui via discuss <discuss(a)mpich.org> > wrote: > >> I am not sure this is an issue for mpich. Seems more to be an issue of >> `mpi4py`. The issue is exactly as you suspected -- `libmpifort.so` need be >> loaded before `libmpi.so`, or the external symbol in `libmpifort.so` won’t >> get resolved. >> >> >> >> > It works if you load something linked to libmpifort first or load >> everything with RTLD_GLOBAL. >> >> Seems plausible. I don’t have any better idea other than fixing `mpi4py` >> so it always load `libmpifort.so` first. >> >> >> >> -- >> Hui Zhou >> >> >> >> >> >> *From: *Patrick McNally via discuss <discuss(a)mpich.org> >> *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> >> *Date: *Wednesday, June 10, 2020 at 1:09 PM >> *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> >> *Cc: *Patrick McNally <rpmcnally(a)gmail.com> >> *Subject: *Re: [mpich-discuss] Using MPICH in Python breaks Fortran >> MPI_IN_PLACE >> >> >> >> I hate to bug, but this is a pretty serious issue. I suspect it is why >> we get segfaults trying to use similar variables like MPI_STATUSES_IGNORE. >> Any insight would be appreciated. >> >> >> >> Thanks, >> >> Patrick >> >> >> >> On Wed, May 27, 2020 at 10:25 AM Patrick McNally <rpmcnally(a)gmail.com> >> wrote: >> >> Our application consists primarily of a Python head calling into Fortran >> routines to do the heavy lifting. We have never been able to successfully >> use MPI_IN_PLACE in Fortran but weren't sure why. Recently, we discovered >> that it works fine in standalone Fortran code and is only broken when the >> Fortran code is run through our Python modules. >> >> The issue appears to be related to having code that only links to the C >> libmpi library loaded first and with RTLD_LOCAL, as happens when we load >> mpi4py. It works if you load something linked to libmpifort first or load >> everything with RTLD_GLOBAL. I'm assuming this has something to do with >> how MPICH tests the address of MPIR_F08_MPI_IN_PLACE but I don't understand >> SO loading well enough to fully grasp the issue. Below is some standalone >> code to show the issue. I'd appreciate any insight you can provide into >> why this is happening. >> >> Relevant system details: >> RHEL 7.8 >> Python 2.7 >> GCC 7.3.0 >> MPICH 3.3.2 (and 3.2) >> >> The below files are also available towards the end of the bug report at >> the following link: >> >> https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks… >> >> >> >> Thanks, >> >> Patrick >> >> >> makefile >> ----------- >> libs = testc.so testf.so >> all: $(libs) >> >> testc.so: testc.c >> mpicc -shared -fPIC $< -o $@ >> >> testf.so: testf.f90 >> mpifort -shared -fPIC $< -o $@ >> >> clean: >> $(RM) $(libs) >> >> testc.c >> --------- >> #include <stddef.h> >> #include <stdio.h> >> #include <mpi.h> >> >> extern void initc(void); >> extern void testc(void); >> >> void initc(void) >> { >> MPI_Init(NULL,NULL); >> } >> >> void testc(void) >> { >> int val = 1; >> MPI_Allreduce(MPI_IN_PLACE, &val, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); >> printf("C val: %2d\n",val); >> } >> >> testf.f90 >> ----------- >> subroutine initf() bind(C) >> use mpi >> integer ierr >> call MPI_Init(ierr) >> end subroutine initf >> >> subroutine testf() bind(C) >> use mpi >> integer ierr >> integer val >> val = 1 >> call MPI_Allreduce(MPI_IN_PLACE, val, 1, MPI_INTEGER, MPI_SUM, >> MPI_COMM_WORLD, ierr) >> print '(A,I2)', 'F val: ', val >> end subroutine testf >> >> test.py >> --------- >> from ctypes import CDLL, RTLD_LOCAL, RTLD_GLOBAL >> >> mode = RTLD_LOCAL >> cfirst = True >> >> if cfirst: # it does not work! >> libc = CDLL("./testc.so", mode) >> libf = CDLL("./testf.so", mode) >> else: # it works! >> libf = CDLL("./testf.so", mode) >> libc = CDLL("./testc.so", mode) >> >> libc.initc.restype = None >> libc.testc.argtypes = [] >> libf.initf.restype = None >> libf.testf.argtypes = [] >> >> libc.initc() >> libc.testc() >> libf.testf() >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE
by Jeff Hammond 01 Jul '20

01 Jul '20

Has Lisandro provided any feedback on this anywhere? I agree with Hui. This appears to be an artifact of the way MPI uses magic values for MPI_IN_PLACE and how those get encoded in binaries. I naively speculate that MPICH could encode the magic values differently to work around this, but that would likely break ABI compatibility, at least on the Fortran side. Jeff On Wed, Jun 10, 2020 at 3:31 PM Zhou, Hui via discuss <discuss(a)mpich.org> wrote: > I am not sure this is an issue for mpich. Seems more to be an issue of > `mpi4py`. The issue is exactly as you suspected -- `libmpifort.so` need be > loaded before `libmpi.so`, or the external symbol in `libmpifort.so` won’t > get resolved. > > > > > It works if you load something linked to libmpifort first or load > everything with RTLD_GLOBAL. > > Seems plausible. I don’t have any better idea other than fixing `mpi4py` > so it always load `libmpifort.so` first. > > > > -- > Hui Zhou > > > > > > *From: *Patrick McNally via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Wednesday, June 10, 2020 at 1:09 PM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *Patrick McNally <rpmcnally(a)gmail.com> > *Subject: *Re: [mpich-discuss] Using MPICH in Python breaks Fortran > MPI_IN_PLACE > > > > I hate to bug, but this is a pretty serious issue. I suspect it is why we > get segfaults trying to use similar variables like MPI_STATUSES_IGNORE. > Any insight would be appreciated. > > > > Thanks, > > Patrick > > > > On Wed, May 27, 2020 at 10:25 AM Patrick McNally <rpmcnally(a)gmail.com> > wrote: > > Our application consists primarily of a Python head calling into Fortran > routines to do the heavy lifting. We have never been able to successfully > use MPI_IN_PLACE in Fortran but weren't sure why. Recently, we discovered > that it works fine in standalone Fortran code and is only broken when the > Fortran code is run through our Python modules. > > The issue appears to be related to having code that only links to the C > libmpi library loaded first and with RTLD_LOCAL, as happens when we load > mpi4py. It works if you load something linked to libmpifort first or load > everything with RTLD_GLOBAL. I'm assuming this has something to do with > how MPICH tests the address of MPIR_F08_MPI_IN_PLACE but I don't understand > SO loading well enough to fully grasp the issue. Below is some standalone > code to show the issue. I'd appreciate any insight you can provide into > why this is happening. > > Relevant system details: > RHEL 7.8 > Python 2.7 > GCC 7.3.0 > MPICH 3.3.2 (and 3.2) > > The below files are also available towards the end of the bug report at > the following link: > > https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks… > > > > Thanks, > > Patrick > > > makefile > ----------- > libs = testc.so testf.so > all: $(libs) > > testc.so: testc.c > mpicc -shared -fPIC $< -o $@ > > testf.so: testf.f90 > mpifort -shared -fPIC $< -o $@ > > clean: > $(RM) $(libs) > > testc.c > --------- > #include <stddef.h> > #include <stdio.h> > #include <mpi.h> > > extern void initc(void); > extern void testc(void); > > void initc(void) > { > MPI_Init(NULL,NULL); > } > > void testc(void) > { > int val = 1; > MPI_Allreduce(MPI_IN_PLACE, &val, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); > printf("C val: %2d\n",val); > } > > testf.f90 > ----------- > subroutine initf() bind(C) > use mpi > integer ierr > call MPI_Init(ierr) > end subroutine initf > > subroutine testf() bind(C) > use mpi > integer ierr > integer val > val = 1 > call MPI_Allreduce(MPI_IN_PLACE, val, 1, MPI_INTEGER, MPI_SUM, > MPI_COMM_WORLD, ierr) > print '(A,I2)', 'F val: ', val > end subroutine testf > > test.py > --------- > from ctypes import CDLL, RTLD_LOCAL, RTLD_GLOBAL > > mode = RTLD_LOCAL > cfirst = True > > if cfirst: # it does not work! > libc = CDLL("./testc.so", mode) > libf = CDLL("./testf.so", mode) > else: # it works! > libf = CDLL("./testf.so", mode) > libc = CDLL("./testc.so", mode) > > libc.initc.restype = None > libc.testc.argtypes = [] > libf.initf.restype = None > libf.testf.argtypes = [] > > libc.initc() > libc.testc() > libf.testf() > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] MPI over infiniband
by Zhou, Hui 01 Jul '20

01 Jul '20

Hi Sabu, Could you try configure MPICH with `--with-device=ch4:ucx`? -- Hui Zhou From: Sabu Best via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, July 1, 2020 at 6:12 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Sabu Best <Sabu.Best(a)barbados.gov.bb> Subject: [mpich-discuss] MPI over infiniband Good day to all, Question, we have been trying to use MPICH over infiniband fabric, but we can't seem to get it to work unless it is IPoIB. We have seen options in OpenMPI that automatically selects Infiniband as the first choice path, but I don't see anything similar for MPICH. Looking forward to your input on this. Thanks!

1 0

MPI over infiniband
by Sabu Best 01 Jul '20

01 Jul '20

Good day to all, Question, we have been trying to use MPICH over infiniband fabric, but we can't seem to get it to work unless it is IPoIB. We have seen options in OpenMPI that automatically selects Infiniband as the first choice path, but I don't see anything similar for MPICH. Looking forward to your input on this. Thanks!

1 0

Multi NIC configuration question
by Sudhakar D 24 Jun '20

24 Jun '20

Hello, Does MPICH support multiple interfaces for each node ? For example, if I have two NIC cards on the local and remote system, can I load balance across all the interfaces ? If yes, is there any documentation to configure the MPICH to use all the interfaces ? regards, Sudhakar

1 0

Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1
by Raffenetti, Kenneth J. 24 Jun '20

24 Jun '20

Hi Kent, Thanks for your report. We have not seen this issue with any compiler/OS combination in our nightly tests. We are using PGI 19.4 at this time. I will request 20.1 be installed so we can investigate further. Ken On 6/23/20, 8:26 AM, "Kent Cheung via discuss" <discuss(a)mpich.org> wrote: I'm running into an issue where processes sometimes hang when calling MPI_Finalize. This happens with both versions 3.3.2 and 3.4a2 on a single node RedHat 7.5 x86-64 machine, when MPICH is compiled with PGI 20.1 with these configuration flags --enable-debug --enable-shared --enable-debuginfo --enable-sharedlib=gcc If I change the default optimization level (-O2) by configuring with --enable-fast=O1 as well, the hang doesn't occur. Another data point is that the hang does not occur with PGI 19.5 with either optimization levels. I have been testing with the cpi.c code in the examples folder built with just mpicc cpi.c mpiexec -n 3 ./a.out Here is a the backtrace from one of the processes that is hanging (gdb) bt #0 MPID_nem_mpich_blocking_recv () at /tmp/mpich-3.3.2/build/../src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:1038 #1 MPIDI_CH3I_Progress () at ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:506 #2 0x00000000004fc88d in MPIDI_CH3U_VC_WaitForClose () at ../src/mpid/ch3/src/ch3u_handle_connection.c:383 #3 0x0000000000442364 in MPID_Finalize () at ../src/mpid/ch3/src/mpid_finalize.c:110 #4 0x0000000000408621 in PMPI_Finalize () at ../src/mpi/init/finalize.c:260 #5 0x00000000004023e5 in main () at cpi.c:59 Is there a potential fix to be made to MPICH to prevent processes hanging when MPICH is compiled with the default optimization level? Thanks, Kent IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0