- discuss - mailman3.cels.anl.gov

Crash on MPI_Rput
by Palmer, Bruce J 27 Sep '22

27 Sep '22

Hi, I'm testing the MPI-RMA runtime in Global Arrays and I'm getting a lot more crashes than I've seen in the past. The MPI-RMA runtime code is fairly stable and hasn't been modified much recently and all the tests used to pass using one of the more recent MPICH releases. However, I'm getting significant crashes at this point. One of them occurs in a program designed to test non-blocking communication. It creates an MPI window, using MPI_Alloc_mem followed by MPI_Win_create and then calls MPI_Win_lock_all on the window. The code currently crashes when it gets to an MPI_Rput call. I'm trying to see if there is something different in the environment that might be causing this. I'm currently up to MPICH-4.0b1 configured with ./configure --prefix=/people/d3g293/mpich/mpich-4.0b1/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ #./configure --prefix=/people/d3g293/mpich/mpich-3.4.1/install-newell-nocuda --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ I've tried other recent vintages of MPICH, but I get similar results. The error I'm seeing when the program crashes is [proxy:0:[email protected]] HYD_pmcd_pmip_control_cmd_cb (pm/pmiserv/pmip_cb.c:899): assert (!closed) failed [proxy:0:[email protected]] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:76): callback returned error status srun: error: node003: task 1: Exited with exit code 7 [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:169): demux engine error waiting for event [mpiexec(a)node002.local] HYDT_bscu_wait_for_completion (tools/bootstrap/utils/bscu_wait.c:74): one of the processes terminated badly; aborting [mpiexec(a)node002.local] HYDT_bsci_wait_for_completion (tools/bootstrap/src/bsci_wait.c:21): launcher returned error waiting for completion [mpiexec(a)node002.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:179): launcher returned error waiting for completion [mpiexec(a)node002.local] main (ui/mpich/mpiexec.c:325): process manager error waiting for completion Any suggestions about what might be going wrong here? It could be a problem with the machine configuration, since this code seemed to be running fine a while ago and has not been modified since then. I'll try building the latest stable release and see if that fixes anything, but as I mentioned none of the recent releases seems to work. Bruce Palmer Computer Scientist Pacific Northwest National Laboratory (509) 375-3899

1 0

Re: [mpich-discuss] GPU-aware MPICH
by Zhou, Hui 13 Jul '22

13 Jul '22

Yes, all datatypes are supported. -- Hui Zhou ________________________________ From: Palmer, Bruce J via discuss <discuss(a)mpich.org> Sent: Wednesday, July 13, 2022 5:29 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Palmer, Bruce <bruce.palmer(a)pnnl.gov> Subject: [mpich-discuss] GPU-aware MPICH Hi, Does anyone know if GPU-aware MPICH supports MPI datatypes? In particular, can you use the subarray datatype (created with MPI_Type_create_subarray)? Bruce Palmer

1 0

GPU-aware MPICH
by Palmer, Bruce J 13 Jul '22

13 Jul '22

Hi, Does anyone know if GPU-aware MPICH supports MPI datatypes? In particular, can you use the subarray datatype (created with MPI_Type_create_subarray)? Bruce Palmer

1 0

Re: [mpich-discuss] How to specify which Ethernet / IP interface to use?
by Jonathon Anderson 01 Jul '22

01 Jul '22

Thanks again, Hui. It looks like I hadn't understood the structure or delegation of responsibility between mpich and libfabric, nor the extent to which libfabric itself comprised part of the runtime interface. I'll give this a try, and I expect this knowledge will be useful to me for other MPI that use libfabric as well! ~jonathon On Fri, Jul 1, 2022 at 7:46 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > For libfabric, which used by ch4:ofi, you can these settings from > libfabric manpages. It depends on your provider. For example, > https://ofiwg.github.io/libfabric/v1.11.1/man/fi_tcp.7.html > fi_tcp(7) - GitHub Pages > <https://ofiwg.github.io/libfabric/v1.11.1/man/fi_tcp.7.html> > NAME. fi_tcp - The msg sockets Fabric Provider. OVERVIEW. The tcp provider > can be used on any system that supports TCP sockets. The provider is not > intended to provide performance improvements over regular TCP sockets, but > rather to allow developers to write, test,and debug application code even > on platforms that do not have high-performance fabric hardware. > ofiwg.github.io > > ------------------------------ > *From:* Jonathon Anderson <janderson(a)ciq.co> > *Sent:* Friday, July 1, 2022 8:40 AM > *To:* Zhou, Hui <zhouh(a)anl.gov> > *Cc:* discuss(a)mpich.org <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] How to specify which Ethernet / IP > interface to use? > > Thank you very much for this information, Hi! Can you point me to where > these environment variables are documented for future reference? I tried to > find something like this and was unable. > > ~jonathon > > > On Fri, Jul 1, 2022, 7:35 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Jonathon, > > Since the binary name is "mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64", > I assume it is mpich -3.4.2 configured with ch4:ofi. We no longer use > mpich2 to refer to the project, just mpich. > > With ch4:ofi, try setting the following environment variables: > FI_PROVIDER=tcp > FI_TCP_IFACE=eth0 > > That assumes you want to use the tcp provider and the private interface is > eth0. There are other providers that you may choose and the iface selection > variables will be different. You can set MPIR_CVAR_DEBUG_SUMMARY=1 to show > the list of providers mpich probes during init. > > Best wishes, > -- > Hui > ------------------------------ > *From:* Jonathon Anderson via discuss <discuss(a)mpich.org> > *Sent:* Thursday, June 30, 2022 10:49 PM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Jonathon Anderson <janderson(a)ciq.co> > *Subject:* Re: [mpich-discuss] How to specify which Ethernet / IP > interface to use? > > I'm trying to use mpich2 [1] on a cluster of compute nodes each of > which has two Ethernet interfaces: one, an encumbered public / > Internet interface, and another a private / intranet interface. mpich > is defaulting to trying to use the public interface, which then hangs > indefinitely with both nodes in my job stuck listening via UDP. > > I experience the same behavior whether I use mpiexec or srun --mpi=pmi2. > > How do I tell mpich which interface to use? > > Thanks! > > ~jonathon > > > [1]: mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] How to specify which Ethernet / IP interface to use?
by Zhou, Hui 01 Jul '22

01 Jul '22

For libfabric, which used by ch4:ofi, you can these settings from libfabric manpages. It depends on your provider. For example, https://ofiwg.github.io/libfabric/v1.11.1/man/fi_tcp.7.html fi_tcp(7) - GitHub Pages<https://ofiwg.github.io/libfabric/v1.11.1/man/fi_tcp.7.html> NAME. fi_tcp - The msg sockets Fabric Provider. OVERVIEW. The tcp provider can be used on any system that supports TCP sockets. The provider is not intended to provide performance improvements over regular TCP sockets, but rather to allow developers to write, test,and debug application code even on platforms that do not have high-performance fabric hardware. ofiwg.github.io ________________________________ From: Jonathon Anderson <janderson(a)ciq.co> Sent: Friday, July 1, 2022 8:40 AM To: Zhou, Hui <zhouh(a)anl.gov> Cc: discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] How to specify which Ethernet / IP interface to use? Thank you very much for this information, Hi! Can you point me to where these environment variables are documented for future reference? I tried to find something like this and was unable. ~jonathon On Fri, Jul 1, 2022, 7:35 AM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Jonathon, Since the binary name is "mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64", I assume it is mpich -3.4.2 configured with ch4:ofi. We no longer use mpich2 to refer to the project, just mpich. With ch4:ofi, try setting the following environment variables: FI_PROVIDER=tcp FI_TCP_IFACE=eth0 That assumes you want to use the tcp provider and the private interface is eth0. There are other providers that you may choose and the iface selection variables will be different. You can set MPIR_CVAR_DEBUG_SUMMARY=1 to show the list of providers mpich probes during init. Best wishes, -- Hui ________________________________ From: Jonathon Anderson via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Thursday, June 30, 2022 10:49 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Jonathon Anderson <janderson(a)ciq.co<mailto:[email protected]>> Subject: Re: [mpich-discuss] How to specify which Ethernet / IP interface to use? I'm trying to use mpich2 [1] on a cluster of compute nodes each of which has two Ethernet interfaces: one, an encumbered public / Internet interface, and another a private / intranet interface. mpich is defaulting to trying to use the public interface, which then hangs indefinitely with both nodes in my job stuck listening via UDP. I experience the same behavior whether I use mpiexec or srun --mpi=pmi2. How do I tell mpich which interface to use? Thanks! ~jonathon [1]: mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64 _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to specify which Ethernet / IP interface to use?
by Jonathon Anderson 01 Jul '22

01 Jul '22

Thank you very much for this information, Hi! Can you point me to where these environment variables are documented for future reference? I tried to find something like this and was unable. ~jonathon On Fri, Jul 1, 2022, 7:35 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > Hi Jonathon, > > Since the binary name is "mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64", > I assume it is mpich -3.4.2 configured with ch4:ofi. We no longer use > mpich2 to refer to the project, just mpich. > > With ch4:ofi, try setting the following environment variables: > FI_PROVIDER=tcp > FI_TCP_IFACE=eth0 > > That assumes you want to use the tcp provider and the private interface is > eth0. There are other providers that you may choose and the iface selection > variables will be different. You can set MPIR_CVAR_DEBUG_SUMMARY=1 to show > the list of providers mpich probes during init. > > Best wishes, > -- > Hui > ------------------------------ > *From:* Jonathon Anderson via discuss <discuss(a)mpich.org> > *Sent:* Thursday, June 30, 2022 10:49 PM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Jonathon Anderson <janderson(a)ciq.co> > *Subject:* Re: [mpich-discuss] How to specify which Ethernet / IP > interface to use? > > I'm trying to use mpich2 [1] on a cluster of compute nodes each of > which has two Ethernet interfaces: one, an encumbered public / > Internet interface, and another a private / intranet interface. mpich > is defaulting to trying to use the public interface, which then hangs > indefinitely with both nodes in my job stuck listening via UDP. > > I experience the same behavior whether I use mpiexec or srun --mpi=pmi2. > > How do I tell mpich which interface to use? > > Thanks! > > ~jonathon > > > [1]: mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64 > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] How to specify which Ethernet / IP interface to use?
by Zhou, Hui 01 Jul '22

01 Jul '22

Hi Jonathon, Since the binary name is "mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64", I assume it is mpich -3.4.2 configured with ch4:ofi. We no longer use mpich2 to refer to the project, just mpich. With ch4:ofi, try setting the following environment variables: FI_PROVIDER=tcp FI_TCP_IFACE=eth0 That assumes you want to use the tcp provider and the private interface is eth0. There are other providers that you may choose and the iface selection variables will be different. You can set MPIR_CVAR_DEBUG_SUMMARY=1 to show the list of providers mpich probes during init. Best wishes, -- Hui ________________________________ From: Jonathon Anderson via discuss <discuss(a)mpich.org> Sent: Thursday, June 30, 2022 10:49 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Jonathon Anderson <janderson(a)ciq.co> Subject: Re: [mpich-discuss] How to specify which Ethernet / IP interface to use? I'm trying to use mpich2 [1] on a cluster of compute nodes each of which has two Ethernet interfaces: one, an encumbered public / Internet interface, and another a private / intranet interface. mpich is defaulting to trying to use the public interface, which then hangs indefinitely with both nodes in my job stuck listening via UDP. I experience the same behavior whether I use mpiexec or srun --mpi=pmi2. How do I tell mpich which interface to use? Thanks! ~jonathon [1]: mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64 _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] How to specify which Ethernet / IP interface to use?
by Jonathon Anderson 30 Jun '22

30 Jun '22

I'm trying to use mpich2 [1] on a cluster of compute nodes each of which has two Ethernet interfaces: one, an encumbered public / Internet interface, and another a private / intranet interface. mpich is defaulting to trying to use the public interface, which then hangs indefinitely with both nodes in my job stuck listening via UDP. I experience the same behavior whether I use mpiexec or srun --mpi=pmi2. How do I tell mpich which interface to use? Thanks! ~jonathon [1]: mpich-ofi-gnu9-ohpc-3.4.2-3.1.ohpc.2.4.x86_64

1 0

Re: [mpich-discuss] mpiexec fails to launch any processes
by Zhou, Hui 14 Jun '22

14 Jun '22

Kurt, The stack trace is showing mpiexec in the polling loop. Since there are no other PMI messages being logged, that means it is still waiting for the processes to call PMI_Init. Can you make your program print something before and after MPI_Init? That will tell us whether the program is stuck in MPI_Init or before MPI_Init. Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Tuesday, June 14, 2022 2:21 AM To: discuss(a)mpich.org <discuss(a)mpich.org>; Zhou, Hui <zhouh(a)anl.gov> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: mpiexec fails to launch any processes Hui, Slurm doesn’t seem to be killing the job, as it still shows up when I run squeue. A gdb stack trace shows where mpiexec is stuck – does this tell you anything? #0 0x00007f9c895ddaa8 in poll () from /lib64/libc.so.6 #1 0x000000000045352c in HYDT_dmxu_poll_wait_for_event (wtime=-1) at ../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:39 #2 0x0000000000452e9a in HYDT_dmx_wait_for_event (wtime=-1) at ../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux.c:168 #3 0x000000000040cda4 in HYD_pmci_wait_for_completion (timeout=-1) at ../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:157 #4 0x0000000000404177 in main (argc=33, argv=0x7fff054e6888) at ../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:324 Thanks, Kurt From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, June 13, 2022 4:16 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] [EXTERNAL] Re: mpiexec fails to launch any processes Hui, That worked too. I guess I’ll have to find a way to pass a “verbose” argument to sbatch and see why Slurm is killing my application. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 4:11 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [EXTERNAL] Re: mpiexec fails to launch any processes Kurt, Could you try launch hostname with the same command? mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 hostname If that went okay, it then seems to point to your application. Something in your code made Slurm kill the job. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 4:02 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: RE: [EXTERNAL] Re: mpiexec fails to launch any processes Hui, $ mpiexec -N 10 -hostfile MySlurmNodeFile2 hostname works properly, reporting from each of 10 nodes. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 2:44 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: mpiexec fails to launch any processes Hi Kurt, I don't have much clue. Are you able to launch some trivial applications, for example, "hostname"? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, June 13, 2022 12:29 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] mpiexec fails to launch any processes Outlook blocked the output file slurm.out that I had attached. Trying to send it again as slurm.txt. Kurt Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn’t see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args…> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0

Re: [mpich-discuss] mpiexec fails to launch any processes
by Mccall, Kurt E. (MSFC-EV41) 14 Jun '22

14 Jun '22

Hui, Slurm doesn’t seem to be killing the job, as it still shows up when I run squeue. A gdb stack trace shows where mpiexec is stuck – does this tell you anything? #0 0x00007f9c895ddaa8 in poll () from /lib64/libc.so.6 #1 0x000000000045352c in HYDT_dmxu_poll_wait_for_event (wtime=-1) at ../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux_poll.c:39 #2 0x0000000000452e9a in HYDT_dmx_wait_for_event (wtime=-1) at ../../../../mpich-4.0.1/src/pm/hydra/tools/demux/demux.c:168 #3 0x000000000040cda4 in HYD_pmci_wait_for_completion (timeout=-1) at ../../../../mpich-4.0.1/src/pm/hydra/pm/pmiserv/pmiserv_pmci.c:157 #4 0x0000000000404177 in main (argc=33, argv=0x7fff054e6888) at ../../../../mpich-4.0.1/src/pm/hydra/ui/mpich/mpiexec.c:324 Thanks, Kurt From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Monday, June 13, 2022 4:16 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] [EXTERNAL] Re: mpiexec fails to launch any processes Hui, That worked too. I guess I’ll have to find a way to pass a “verbose” argument to sbatch and see why Slurm is killing my application. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 4:11 PM To: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> Subject: Re: [EXTERNAL] Re: mpiexec fails to launch any processes Kurt, Could you try launch hostname with the same command? mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 hostname If that went okay, it then seems to point to your application. Something in your code made Slurm kill the job. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 4:02 PM To: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Subject: RE: [EXTERNAL] Re: mpiexec fails to launch any processes Hui, $ mpiexec -N 10 -hostfile MySlurmNodeFile2 hostname works properly, reporting from each of 10 nodes. Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Monday, June 13, 2022 2:44 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: mpiexec fails to launch any processes Hi Kurt, I don't have much clue. Are you able to launch some trivial applications, for example, "hostname"? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Monday, June 13, 2022 12:29 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] mpiexec fails to launch any processes Outlook blocked the output file slurm.out that I had attached. Trying to send it again as slurm.txt. Kurt Hi, My mpiexec command fails to launch any processes. I ran it with the -verbose option but didn’t see any obvious errors in the output (attached). The command is: mpiexec -launcher ssh -verbose -print-all-exitcodes -wdir <directory> -np 20 -ppn 1 <more args…> I am running MPICH 4.0.1 under Slurm 20.11.8. Thanks for any help. Kurt

1 0