June 2020 - discuss - mailman3.cels.anl.gov

Multi NIC configuration question
by Sudhakar D 24 Jun '20

24 Jun '20

Hello, Does MPICH support multiple interfaces for each node ? For example, if I have two NIC cards on the local and remote system, can I load balance across all the interfaces ? If yes, is there any documentation to configure the MPICH to use all the interfaces ? regards, Sudhakar

1 0

Re: [mpich-discuss] Intermittent hang in MPI_Finalize with PGI 20.1
by Raffenetti, Kenneth J. 24 Jun '20

24 Jun '20

Hi Kent, Thanks for your report. We have not seen this issue with any compiler/OS combination in our nightly tests. We are using PGI 19.4 at this time. I will request 20.1 be installed so we can investigate further. Ken On 6/23/20, 8:26 AM, "Kent Cheung via discuss" <discuss(a)mpich.org> wrote: I'm running into an issue where processes sometimes hang when calling MPI_Finalize. This happens with both versions 3.3.2 and 3.4a2 on a single node RedHat 7.5 x86-64 machine, when MPICH is compiled with PGI 20.1 with these configuration flags --enable-debug --enable-shared --enable-debuginfo --enable-sharedlib=gcc If I change the default optimization level (-O2) by configuring with --enable-fast=O1 as well, the hang doesn't occur. Another data point is that the hang does not occur with PGI 19.5 with either optimization levels. I have been testing with the cpi.c code in the examples folder built with just mpicc cpi.c mpiexec -n 3 ./a.out Here is a the backtrace from one of the processes that is hanging (gdb) bt #0 MPID_nem_mpich_blocking_recv () at /tmp/mpich-3.3.2/build/../src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:1038 #1 MPIDI_CH3I_Progress () at ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:506 #2 0x00000000004fc88d in MPIDI_CH3U_VC_WaitForClose () at ../src/mpid/ch3/src/ch3u_handle_connection.c:383 #3 0x0000000000442364 in MPID_Finalize () at ../src/mpid/ch3/src/mpid_finalize.c:110 #4 0x0000000000408621 in PMPI_Finalize () at ../src/mpi/init/finalize.c:260 #5 0x00000000004023e5 in main () at cpi.c:59 Is there a potential fix to be made to MPICH to prevent processes hanging when MPICH is compiled with the default optimization level? Thanks, Kent IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0

Intermittent hang in MPI_Finalize with PGI 20.1
by Kent Cheung 23 Jun '20

23 Jun '20

I'm running into an issue where processes sometimes hang when calling MPI_Finalize. This happens with both versions 3.3.2 and 3.4a2 on a single node RedHat 7.5 x86-64 machine, when MPICH is compiled with PGI 20.1 with these configuration flags --enable-debug --enable-shared --enable-debuginfo --enable-sharedlib=gcc If I change the default optimization level (-O2) by configuring with --enable-fast=O1 as well, the hang doesn't occur. Another data point is that the hang does not occur with PGI 19.5 with either optimization levels. I have been testing with the cpi.c code in the examples folder built with just mpicc cpi.c mpiexec -n 3 ./a.out Here is a the backtrace from one of the processes that is hanging (gdb) bt #0 MPID_nem_mpich_blocking_recv () at /tmp/mpich-3.3.2/build/../src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h:1038 #1 MPIDI_CH3I_Progress () at ../src/mpid/ch3/channels/nemesis/src/ch3_progress.c:506 #2 0x00000000004fc88d in MPIDI_CH3U_VC_WaitForClose () at ../src/mpid/ch3/src/ch3u_handle_connection.c:383 #3 0x0000000000442364 in MPID_Finalize () at ../src/mpid/ch3/src/mpid_finalize.c:110 #4 0x0000000000408621 in PMPI_Finalize () at ../src/mpi/init/finalize.c:260 #5 0x00000000004023e5 in main () at cpi.c:59 Is there a potential fix to be made to MPICH to prevent processes hanging when MPICH is compiled with the default optimization level? Thanks, Kent IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

1 0

launcher manual and invalid pointer
by bruno.guerraz＠orange.com 15 Jun '20

15 Jun '20

Hi, I am trying to launch an application using mpi on top of apache Yarn. I am using the trick described in this old post https://lists.mpich.org/mailman/htdig/devel/2016-July/000717.html. When I try it outside Yarn, all works fine, but inside Yarn I have an issue in mpiexec : "munmap_chunk(): invalid pointer" I cannot reproduce the bug outside of yarn. I have try with the release 3.4a2 of hydra, but I still have the bug. Here is the backtrace I got from the core dump (on the 3.4a2 release). #1 0x00007f0811d88801 in __GI_abort () at abort.c:79 #2 0x00007f0811dd1897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7f0811efeb9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181 #3 0x00007f0811dd890a in malloc_printerr (str=str@entry=0x7f0811f007a8 "munmap_chunk(): invalid pointer") at malloc.c:5350 #4 0x00007f0811ddfecc in munmap_chunk (p=0x7fffbcc9a8cc) at malloc.c:2846 #5 __GI___libc_free (mem=mem@entry=0x7fffbcc9a8dc) at malloc.c:3117 #6 0x000055ad7edca224 in HYDU_find_in_path (execname=execname@entry=0x7fffbcc9a8ab "mpiexec", path=0x55ad7f0b3990 <HYD_server_info+144>) at utils/args/args.c:108 #7 0x000055ad7edb1c48 in HYD_uii_mpx_get_parameters (t_argv=<optimized out>) at ui/mpich/utils.c:1751 #8 0x000055ad7edaa9d6 in main (argc=<optimized out>, argv=0x7fffbcc99d68) at ui/mpich/mpiexec.c:147 Any help would be greatly appreciated Bruno _________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.

1 0

Re: [mpich-discuss] Excessive disk activity in MPI processes
by Mccall, Kurt E. (MSFC-EV41) 11 Jun '20

11 Jun '20

Hui, No, I was asking if MPI itself did any optional disk I/O that could be turned off. Ken Raffenetti said that there wasn't any. Your idea of launching with a wrapper might help, though, since my application does some writing to stdout that I cannot silence. Thanks, Kurt -----Original Message----- From: Zhou, Hui via discuss <discuss(a)mpich.org> Sent: Wednesday, June 10, 2020 5:16 PM To: discuss(a)mpich.org Cc: Zhou, Hui <zhouh(a)anl.gov> Subject: [EXTERNAL] Re: [mpich-discuss] Excessive disk activity in MPI processes Correct me if I was off, but I believe in the OP's case, it is the application itself heavily outputing to stdout and stderr -- from his anecdote of telling qsub not to save stdout and stderr. Hydra collects stdout and stderr by default, so I think that is causing the I/O simply due to the amount of output. I think what OP was asking is whether there is an option to simply drop stdout/stderr. Kurt, not that I know of. If the application doesn't have an option to silence itself, could you try launch with a script wrapper, for example: ``` app.sh [your app] 2>&1 > /dev/null # or redirect to a log file ``` Then launch with `mpirun app.sh`? -- Hui Zhou On 6/10/20, 4:57 PM, "Raffenetti, Kenneth J. via discuss" <discuss(a)mpich.org> wrote: MPICH queries a bunch of information from the /proc filesystem using hwloc, but that should all be during MPI_Init. Nevertheless, you can disable it in 3.3.2 using --with-hwloc-prefix=no. I'm not able to think of other optional I/O. Ken On 6/2/20, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Sorry, I should have mentioned that our file system is NFS hosted on the cluster head node, with infiniband from the compute nodes to the head. ****** I am attempting to run an MPI Monte Carlo: multiple instances of a simulation “Mav” under the control of MPI. Mav does a huge amount of output to disk, but the MPI processes that oversee the Mav instances are lightweight and only wake up every 5 seconds. The MPI Monte Carlo is much slower than running each Mav as an independent PBS Torque job, and according to “iotop”, the MPI-supervised Mav instances regularly become blocked waiting for I/O. iotop often shows the Mavs’s spending 99% of their time waiting for I/O. The independent Torque jobs don’t do that. I got some significant improvement in the MPI performance by telling qsub not to save the stdout and stderr output (qusb –k n) to a file. My question is: are there any other optional types of disk activity performed by MPI that I can disable? What else could be happening here? I’m using MPICH 3.3.2 compiled with pgc++, PBS Torque 5.1.1, CentOS 3.10.0. Here are my MPICH configure options: '--disable-option-checking' '--prefix=/opt/mpich_pgc' '--with-pbs=/opt/torque' 'CC=pgcc' 'CXX=pgc++' '-enable-debuginfo' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/openpa/src -I/home/kmccall/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/home/kmccall/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl' Thanks, Kurt _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma… _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.mpich.org_mailma…

1 0

Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE
by Zhou, Hui 10 Jun '20

10 Jun '20

I am not sure this is an issue for mpich. Seems more to be an issue of `mpi4py`. The issue is exactly as you suspected -- `libmpifort.so` need be loaded before `libmpi.so`, or the external symbol in `libmpifort.so` won’t get resolved. > It works if you load something linked to libmpifort first or load everything with RTLD_GLOBAL. Seems plausible. I don’t have any better idea other than fixing `mpi4py` so it always load `libmpifort.so` first. -- Hui Zhou From: Patrick McNally via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Wednesday, June 10, 2020 at 1:09 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Patrick McNally <rpmcnally(a)gmail.com> Subject: Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE I hate to bug, but this is a pretty serious issue. I suspect it is why we get segfaults trying to use similar variables like MPI_STATUSES_IGNORE. Any insight would be appreciated. Thanks, Patrick On Wed, May 27, 2020 at 10:25 AM Patrick McNally <rpmcnally(a)gmail.com<mailto:[email protected]>> wrote: Our application consists primarily of a Python head calling into Fortran routines to do the heavy lifting. We have never been able to successfully use MPI_IN_PLACE in Fortran but weren't sure why. Recently, we discovered that it works fine in standalone Fortran code and is only broken when the Fortran code is run through our Python modules. The issue appears to be related to having code that only links to the C libmpi library loaded first and with RTLD_LOCAL, as happens when we load mpi4py. It works if you load something linked to libmpifort first or load everything with RTLD_GLOBAL. I'm assuming this has something to do with how MPICH tests the address of MPIR_F08_MPI_IN_PLACE but I don't understand SO loading well enough to fully grasp the issue. Below is some standalone code to show the issue. I'd appreciate any insight you can provide into why this is happening. Relevant system details: RHEL 7.8 Python 2.7 GCC 7.3.0 MPICH 3.3.2 (and 3.2) The below files are also available towards the end of the bug report at the following link: https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks… Thanks, Patrick makefile ----------- libs = testc.so testf.so all: $(libs) testc.so: testc.c mpicc -shared -fPIC $< -o $@ testf.so: testf.f90 mpifort -shared -fPIC $< -o $@ clean: $(RM) $(libs) testc.c --------- #include <stddef.h> #include <stdio.h> #include <mpi.h> extern void initc(void); extern void testc(void); void initc(void) { MPI_Init(NULL,NULL); } void testc(void) { int val = 1; MPI_Allreduce(MPI_IN_PLACE, &val, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); printf("C val: %2d\n",val); } testf.f90 ----------- subroutine initf() bind(C) use mpi integer ierr call MPI_Init(ierr) end subroutine initf subroutine testf() bind(C) use mpi integer ierr integer val val = 1 call MPI_Allreduce(MPI_IN_PLACE, val, 1, MPI_INTEGER, MPI_SUM, MPI_COMM_WORLD, ierr) print '(A,I2)', 'F val: ', val end subroutine testf test.py --------- from ctypes import CDLL, RTLD_LOCAL, RTLD_GLOBAL mode = RTLD_LOCAL cfirst = True if cfirst: # it does not work! libc = CDLL("./testc.so", mode) libf = CDLL("./testf.so", mode) else: # it works! libf = CDLL("./testf.so", mode) libc = CDLL("./testc.so", mode) libc.initc.restype = None libc.testc.argtypes = [] libf.initf.restype = None libf.testf.argtypes = [] libc.initc() libc.testc() libf.testf()

1 0

Re: [mpich-discuss] Excessive disk activity in MPI processes
by Zhou, Hui 10 Jun '20

10 Jun '20

Correct me if I was off, but I believe in the OP's case, it is the application itself heavily outputing to stdout and stderr -- from his anecdote of telling qsub not to save stdout and stderr. Hydra collects stdout and stderr by default, so I think that is causing the I/O simply due to the amount of output. I think what OP was asking is whether there is an option to simply drop stdout/stderr. Kurt, not that I know of. If the application doesn't have an option to silence itself, could you try launch with a script wrapper, for example: ``` app.sh [your app] 2>&1 > /dev/null # or redirect to a log file ``` Then launch with `mpirun app.sh`? -- Hui Zhou On 6/10/20, 4:57 PM, "Raffenetti, Kenneth J. via discuss" <discuss(a)mpich.org> wrote: MPICH queries a bunch of information from the /proc filesystem using hwloc, but that should all be during MPI_Init. Nevertheless, you can disable it in 3.3.2 using --with-hwloc-prefix=no. I'm not able to think of other optional I/O. Ken On 6/2/20, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Sorry, I should have mentioned that our file system is NFS hosted on the cluster head node, with infiniband from the compute nodes to the head. ****** I am attempting to run an MPI Monte Carlo: multiple instances of a simulation “Mav” under the control of MPI. Mav does a huge amount of output to disk, but the MPI processes that oversee the Mav instances are lightweight and only wake up every 5 seconds. The MPI Monte Carlo is much slower than running each Mav as an independent PBS Torque job, and according to “iotop”, the MPI-supervised Mav instances regularly become blocked waiting for I/O. iotop often shows the Mavs’s spending 99% of their time waiting for I/O. The independent Torque jobs don’t do that. I got some significant improvement in the MPI performance by telling qsub not to save the stdout and stderr output (qusb –k n) to a file. My question is: are there any other optional types of disk activity performed by MPI that I can disable? What else could be happening here? I’m using MPICH 3.3.2 compiled with pgc++, PBS Torque 5.1.1, CentOS 3.10.0. Here are my MPICH configure options: '--disable-option-checking' '--prefix=/opt/mpich_pgc' '--with-pbs=/opt/torque' 'CC=pgcc' 'CXX=pgc++' '-enable-debuginfo' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/openpa/src -I/home/kmccall/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/home/kmccall/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl' Thanks, Kurt _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Excessive disk activity in MPI processes
by Raffenetti, Kenneth J. 10 Jun '20

10 Jun '20

MPICH queries a bunch of information from the /proc filesystem using hwloc, but that should all be during MPI_Init. Nevertheless, you can disable it in 3.3.2 using --with-hwloc-prefix=no. I'm not able to think of other optional I/O. Ken On 6/2/20, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Sorry, I should have mentioned that our file system is NFS hosted on the cluster head node, with infiniband from the compute nodes to the head. ****** I am attempting to run an MPI Monte Carlo: multiple instances of a simulation “Mav” under the control of MPI. Mav does a huge amount of output to disk, but the MPI processes that oversee the Mav instances are lightweight and only wake up every 5 seconds. The MPI Monte Carlo is much slower than running each Mav as an independent PBS Torque job, and according to “iotop”, the MPI-supervised Mav instances regularly become blocked waiting for I/O. iotop often shows the Mavs’s spending 99% of their time waiting for I/O. The independent Torque jobs don’t do that. I got some significant improvement in the MPI performance by telling qsub not to save the stdout and stderr output (qusb –k n) to a file. My question is: are there any other optional types of disk activity performed by MPI that I can disable? What else could be happening here? I’m using MPICH 3.3.2 compiled with pgc++, PBS Torque 5.1.1, CentOS 3.10.0. Here are my MPICH configure options: '--disable-option-checking' '--prefix=/opt/mpich_pgc' '--with-pbs=/opt/torque' 'CC=pgcc' 'CXX=pgc++' '-enable-debuginfo' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LDFLAGS=' 'LIBS=' 'CPPFLAGS= -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/mpl/include -I/home/kmccall/mpich-3.3.2/src/openpa/src -I/home/kmccall/mpich-3.3.2/src/openpa/src -D_REENTRANT -I/home/kmccall/mpich-3.3.2/src/mpi/romio/include' 'MPLLIBNAME=mpl' Thanks, Kurt

1 0

Re: [mpich-discuss] Using MPICH in Python breaks Fortran MPI_IN_PLACE
by Patrick McNally 10 Jun '20

10 Jun '20

I hate to bug, but this is a pretty serious issue. I suspect it is why we get segfaults trying to use similar variables like MPI_STATUSES_IGNORE. Any insight would be appreciated. Thanks, Patrick On Wed, May 27, 2020 at 10:25 AM Patrick McNally <rpmcnally(a)gmail.com> wrote: > Our application consists primarily of a Python head calling into Fortran > routines to do the heavy lifting. We have never been able to successfully > use MPI_IN_PLACE in Fortran but weren't sure why. Recently, we discovered > that it works fine in standalone Fortran code and is only broken when the > Fortran code is run through our Python modules. > > The issue appears to be related to having code that only links to the C > libmpi library loaded first and with RTLD_LOCAL, as happens when we load > mpi4py. It works if you load something linked to libmpifort first or load > everything with RTLD_GLOBAL. I'm assuming this has something to do with > how MPICH tests the address of MPIR_F08_MPI_IN_PLACE but I don't understand > SO loading well enough to fully grasp the issue. Below is some standalone > code to show the issue. I'd appreciate any insight you can provide into > why this is happening. > > Relevant system details: > RHEL 7.8 > Python 2.7 > GCC 7.3.0 > MPICH 3.3.2 (and 3.2) > > The below files are also available towards the end of the bug report at > the following link: > > https://bitbucket.org/mpi4py/mpi4py/issues/162/mpi4py-initialization-breaks… > > Thanks, > Patrick > > makefile > ----------- > libs = testc.so testf.so > all: $(libs) > > testc.so: testc.c > mpicc -shared -fPIC $< -o $@ > > testf.so: testf.f90 > mpifort -shared -fPIC $< -o $@ > > clean: > $(RM) $(libs) > > testc.c > --------- > #include <stddef.h> > #include <stdio.h> > #include <mpi.h> > > extern void initc(void); > extern void testc(void); > > void initc(void) > { > MPI_Init(NULL,NULL); > } > > void testc(void) > { > int val = 1; > MPI_Allreduce(MPI_IN_PLACE, &val, 1, MPI_INT, MPI_SUM, MPI_COMM_WORLD); > printf("C val: %2d\n",val); > } > > testf.f90 > ----------- > subroutine initf() bind(C) > use mpi > integer ierr > call MPI_Init(ierr) > end subroutine initf > > subroutine testf() bind(C) > use mpi > integer ierr > integer val > val = 1 > call MPI_Allreduce(MPI_IN_PLACE, val, 1, MPI_INTEGER, MPI_SUM, > MPI_COMM_WORLD, ierr) > print '(A,I2)', 'F val: ', val > end subroutine testf > > test.py > --------- > from ctypes import CDLL, RTLD_LOCAL, RTLD_GLOBAL > > mode = RTLD_LOCAL > cfirst = True > > if cfirst: # it does not work! > libc = CDLL("./testc.so", mode) > libf = CDLL("./testf.so", mode) > else: # it works! > libf = CDLL("./testf.so", mode) > libc = CDLL("./testc.so", mode) > > libc.initc.restype = None > libc.testc.argtypes = [] > libf.initf.restype = None > libf.testf.argtypes = [] > > libc.initc() > libc.testc() > libf.testf() >

1 0

MPICH installation
by Anthony Sirico 10 Jun '20

10 Jun '20

Good morning, I have been struggling with getting my machines to work as one. I installed Linux mint on all machines, creating the same username and passwords. I also setup a share file through NFS. My first question is, I cannot ssh from the master node, wolf00, to a slave node, wolf01 etc., without entering a password, how can i fix that? i've been through guides, setting up keychains and it is still not working. I would like to start over setting that up. My second question is, during my Beowulf setup, i created a user in the NFS shared folder (/mirror), does it make sense to install MPICH there using the installation guide, or does if have to be installed on just the master node's machine, or all machines? Looking forward to working with you. Regards, Anthony

1 0