February 2022 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Init hangs under Slurm
by Raffenetti, Ken 18 Feb '22

18 Feb '22

So it looks like only nodes 1 and 2 successfully launched. n001.cluster.pssclabs.com n002.cluster.pssclabs.com while the others are stuck trying to get there over ssh. Basically, it's a system configuration issue. That said, it is preferable to use the Slurm launcher instead of ssh, but in the past I recall there was an issue spawning new processes getting an error like: srun: Job 84993 step creation temporarily disabled, retrying (Requested nodes are busy) Can you try removing the --exclusive from your sbatch command and also remove -launcher ssh from your mpiexec command? I think what might be happening is the Slurm launcher won't spawn any processes because it thinks the currently running processes need access to the whole node. Ken On 2/18/22, 2:08 PM, "Mccall, Kurt E. (MSFC-EV41)" <kurt.e.mccall(a)nasa.gov> wrote: I’m clueless. Here is the correct output, running mpiexec inside of sbatch. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, February 18, 2022 2:00 PM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Init hangs under Slurm Hi Kurt, Did you run mpiexec inside sbatch? It will need sbatch to allocate the nodes. -- Hui ________________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Friday, February 18, 2022 1:39 PM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] MPI_Init hangs under Slurm Here is the --verbose output. Is it trying to launch the processes all on the head node rocci.ndc.nasa.gov? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 18, 2022 1:31 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Init hangs under Slurm From the looks of it, using the ssh launcher might not be able to access all the nodes. To confirm, can you try launching a non-MPI program? Something like mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -ppn1 hostname Ken On 2/17/22, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Sorry, my attachment with an .out extension was blocked. Here is the file with a .txt extension. From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Thursday, February 17, 2022 2:36 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: MPI_Init hangs under Slurm Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of “mpiexec -verbose” is attached, and the stack trace at the point where it hangs is below. In the “mpiexec -verbose” output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the “host” info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive –verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args…> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

Re: [mpich-discuss] MPI_Init hangs under Slurm
by Mccall, Kurt E. (MSFC-EV41) 18 Feb '22

18 Feb '22

I’m clueless. Here is the correct output, running mpiexec inside of sbatch. Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Friday, February 18, 2022 2:00 PM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Init hangs under Slurm Hi Kurt, Did you run mpiexec inside sbatch? It will need sbatch to allocate the nodes. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Friday, February 18, 2022 1:39 PM To: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>>; discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: Re: [mpich-discuss] MPI_Init hangs under Slurm Here is the --verbose output. Is it trying to launch the processes all on the head node rocci.ndc.nasa.gov? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov<mailto:[email protected]>> Sent: Friday, February 18, 2022 1:31 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Init hangs under Slurm From the looks of it, using the ssh launcher might not be able to access all the nodes. To confirm, can you try launching a non-MPI program? Something like mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -ppn1 hostname Ken On 2/17/22, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org<mailto:[email protected]>> wrote: Sorry, my attachment with an .out extension was blocked. Here is the file with a .txt extension. From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Sent: Thursday, February 17, 2022 2:36 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: MPI_Init hangs under Slurm Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of “mpiexec -verbose” is attached, and the stack trace at the point where it hangs is below. In the “mpiexec -verbose” output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the “host” info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive –verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args…> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

Re: [mpich-discuss] MPI_Init hangs under Slurm
by Zhou, Hui 18 Feb '22

18 Feb '22

Hi Kurt, Did you run mpiexec inside sbatch? It will need sbatch to allocate the nodes. -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Friday, February 18, 2022 1:39 PM To: Raffenetti, Ken <raffenet(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] MPI_Init hangs under Slurm Here is the --verbose output. Is it trying to launch the processes all on the head node rocci.ndc.nasa.gov? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 18, 2022 1:31 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Init hangs under Slurm From the looks of it, using the ssh launcher might not be able to access all the nodes. To confirm, can you try launching a non-MPI program? Something like mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -ppn1 hostname Ken On 2/17/22, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Sorry, my attachment with an .out extension was blocked. Here is the file with a .txt extension. From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Thursday, February 17, 2022 2:36 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: MPI_Init hangs under Slurm Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of “mpiexec -verbose” is attached, and the stack trace at the point where it hangs is below. In the “mpiexec -verbose” output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the “host” info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive –verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args…> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

Re: [mpich-discuss] MPI_Init hangs under Slurm
by Mccall, Kurt E. (MSFC-EV41) 18 Feb '22

18 Feb '22

Here is the --verbose output. Is it trying to launch the processes all on the head node rocci.ndc.nasa.gov? Kurt -----Original Message----- From: Raffenetti, Ken <raffenet(a)anl.gov> Sent: Friday, February 18, 2022 1:31 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: [mpich-discuss] MPI_Init hangs under Slurm From the looks of it, using the ssh launcher might not be able to access all the nodes. To confirm, can you try launching a non-MPI program? Something like mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -ppn1 hostname Ken On 2/17/22, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Sorry, my attachment with an .out extension was blocked. Here is the file with a .txt extension. From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Thursday, February 17, 2022 2:36 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: MPI_Init hangs under Slurm Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of “mpiexec -verbose” is attached, and the stack trace at the point where it hangs is below. In the “mpiexec -verbose” output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the “host” info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive –verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args…> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

Re: [mpich-discuss] MPI_Init hangs under Slurm
by Raffenetti, Ken 18 Feb '22

18 Feb '22

From the looks of it, using the ssh launcher might not be able to access all the nodes. To confirm, can you try launching a non-MPI program? Something like mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -ppn1 hostname Ken On 2/17/22, 2:39 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Sorry, my attachment with an .out extension was blocked. Here is the file with a .txt extension. From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Thursday, February 17, 2022 2:36 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: MPI_Init hangs under Slurm Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of “mpiexec -verbose” is attached, and the stack trace at the point where it hangs is below. In the “mpiexec -verbose” output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the “host” info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive –verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args…> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

Re: [mpich-discuss] MPI_Init hangs under Slurm
by Zhou, Hui 17 Feb '22

17 Feb '22

Hi Kurt, We have fixed the bug when launching using PMI1 and mpiexec -- https://github.com/pmodels/mpich/issues/5835. Could you checkout the latest main branch on github and test? Let us know if you need instructions on building from git checkouts. We are working on the remaining scenarios. -- Hui Zhou [https://opengraph.githubassets.com/625f481fa04d88c26174a96205e1a5fc12fb0eb7…]<https://github.com/pmodels/mpich/issues/5835> MPI_Comm_spawn in Slurm environment · Issue #5835 · pmodels/mpich<https://github.com/pmodels/mpich/issues/5835> Originated from user email https://lists.mpich.org/pipermail/discuss/2022-January/006360.html. 1. MPICH + Hydra + PMI1 (crashes) Fixed in #5838 2. MPICH + Hydra + PMI2 (works but ignores "host... github.com ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Thursday, February 17, 2022 2:38 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: [mpich-discuss] MPI_Init hangs under Slurm Sorry, my attachment with an .out extension was blocked. Here is the file with a .txt extension. From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Thursday, February 17, 2022 2:36 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: MPI_Init hangs under Slurm Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of “mpiexec -verbose” is attached, and the stack trace at the point where it hangs is below. In the “mpiexec -verbose” output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the “host” info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive –verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args…> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

Re: [mpich-discuss] MPI_Init hangs under Slurm
by Mccall, Kurt E. (MSFC-EV41) 17 Feb '22

17 Feb '22

Sorry, my attachment with an .out extension was blocked. Here is the file with a .txt extension. From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Thursday, February 17, 2022 2:36 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: MPI_Init hangs under Slurm Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of "mpiexec -verbose" is attached, and the stack trace at the point where it hangs is below. In the "mpiexec -verbose" output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the "host" info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive -verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args...> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

MPI_Init hangs under Slurm
by Mccall, Kurt E. (MSFC-EV41) 17 Feb '22

17 Feb '22

Things were working fine when I was launching 1 node jobs under Slurm 20.11.8, but when I launched a 20 node job, MPICH hangs in MPI_Init. The output of "mpiexec -verbose" is attached, and the stack trace at the point where it hangs is below. In the "mpiexec -verbose" output, I wonder why variables such as PATH_modshare point to our Intel MPI implementation, which I am no using. I am using MPICH 4.0 with a patch that Ken Raffenetti provided (which makes MPICH recognize the "host" info key). My $PATH and $LD_LIBRARY_PATH variables definitely point to the correct MPICH installation. I appreciate any help you can give. Here is the Slurm sbatch command: sbatch --nodes=20 --ntasks=20 --job-name $job_name --exclusive -verbose Here is the mpiexec command: mpiexec -verbose -launcher ssh -print-all-exitcodes -np 20 -wdir ${work_dir} -env DISPLAY localhost:10.0 --ppn 1 <many more args...> Stack trace at MPI_Init: #0 0x00007f6d85f499b2 in read () from /lib64/libpthread.so.0 #1 0x00007f6d87a5753a in PMIU_readline (fd=5, buf=buf@entry=0x7ffd6fb596e0 "", maxlen=maxlen@entry=1024) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmiutil.c:134 #2 0x00007f6d87a57a56 in GetResponse (request=0x7f6d87b48351 "cmd=barrier_in\n", expectedCmd=0x7f6d87b48345 "barrier_out", checkRc=0) at ../mpich-slurm-patch-4.0/src/pmi/simple/simple_pmi.c:818 #3 0x00007f6d87a29915 in MPIDI_PG_SetConnInfo (rank=rank@entry=0, connString=connString@entry=0x1bbf5a0 "description#n001$port#33403$ifname#172.16.56.1$") at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpidi_pg.c:559 #4 0x00007f6d87a38611 in MPID_nem_init (pg_rank=pg_rank@entry=0, pg_p=pg_p@entry=0x1bbf850, has_parent=<optimized out>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c:393 #5 0x00007f6d87a2ad93 in MPIDI_CH3_Init (has_parent=<optimized out>, pg_p=0x1bbf850, pg_rank=0) at ../mpich-slurm-patch-4.0/src/mpid/ch3/channels/nemesis/src/ch3_init.c:83 #6 0x00007f6d87a1b3b7 in init_world () at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:190 #7 MPID_Init (requested=<optimized out>, provided=provided@entry=0x7f6d87e03540 <MPIR_ThreadInfo>) at ../mpich-slurm-patch-4.0/src/mpid/ch3/src/mpid_init.c:76 #8 0x00007f6d879828eb in MPII_Init_thread (argc=argc@entry=0x7ffd6fb5a5cc, argv=argv@entry=0x7ffd6fb5a5c0, user_required=0, provided=provided@entry=0x7ffd6fb5a574, p_session_ptr=p_session_ptr@entry=0x0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:208 #9 0x00007f6d879832a5 in MPIR_Init_impl (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/mpi/init/mpir_init.c:93 #10 0x00007f6d8786388e in PMPI_Init (argc=0x7ffd6fb5a5cc, argv=0x7ffd6fb5a5c0) at ../mpich-slurm-patch-4.0/src/binding/c/init/init.c:46 #11 0x000000000040640d in main (argc=23, argv=0x7ffd6fb5ad68) at src/NeedlesMpiManagerMain.cpp:53

1 0

Re: [mpich-discuss] new to mpi: performance issues when using mpich
by Raffenetti, Ken 16 Feb '22

16 Feb '22

--cpus-per-task is not equivalent to -ppn when using mpiexec/mpirun. -ppn says launch this many processes per node. --cpus-per-tasks says launch as many processes as you can so that each has access to at least 8 CPU cores. Trying tuning the launch options so you are sure you are launching the same number of procs in either configuration. Ken On 2/16/22, 12:36 PM, "Sopena Ballesteros Manuel via discuss" <discuss(a)mpich.org> wrote: I am learning how to install and run mpi aplications. My goal is to install mpich and run osu micro benchmark in order to understand how it works. so far I got it working but I must be doing wrong because the latency tests does not looks right to me. /scratch/e1000/msopena/mpich-installed/bin/mpirun -f /scratch/e1000/msopena/hosts -ppn 8 /scratch/e1000/msopena/osu-micro-benchmarks-5.8/mpi/collective/osu_alltoall # OSU MPI All-to-All Personalized Exchange Latency Test v5.8 # Size Avg Latency(us) 1 104.87 2 101.44 4 105.47 8 94.35 16 95.14 32 239.45 64 293.59 128 437.93 256 414.21 512 444.74 1024 655.93 2048 529.20 4096 551.84 8192 574.99 16384 644.90 32768 790.70 65536 4169.19 131072 4756.38 262144 6782.57 524288 12299.24 1048576 22880.86 However, I get notmal results when the osu benchmarks through srun eg $ srun -N3 --cpus-per-task=8 /scratch/e1000/msopena/osu-micro-benchmarks-5.8/mpi/collective/osu_alltoall # OSU MPI All-to-All Personalized Exchange Latency Test v5.8 # Size Avg Latency(us) 1 27.43 2 27.97 4 29.26 8 30.69 16 31.77 32 34.89 64 42.06 128 60.83 256 115.84 512 205.33 1024 282.61 2048 446.41 4096 733.10 8192 1421.99 16384 2832.45 32768 5631.33 65536 11242.08 131072 22645.46 262144 45544.75 524288 90440.93 1048576 186709.32 Could someone please help me to troubleshoot could be wrong when I use mpich stand alone? thank you very much

1 0

new to mpi: performance issues when using mpich
by Sopena Ballesteros Manuel 16 Feb '22

16 Feb '22

I am learning how to install and run mpi aplications. My goal is to install mpich and run osu micro benchmark in order to understand how it works. so far I got it working but I must be doing wrong because the latency tests does not looks right to me. /scratch/e1000/msopena/mpich-installed/bin/mpirun -f /scratch/e1000/msopena/hosts -ppn 8 /scratch/e1000/msopena/osu-micro-benchmarks-5.8/mpi/collective/osu_alltoall # OSU MPI All-to-All Personalized Exchange Latency Test v5.8 # Size Avg Latency(us) 1 104.87 2 101.44 4 105.47 8 94.35 16 95.14 32 239.45 64 293.59 128 437.93 256 414.21 512 444.74 1024 655.93 2048 529.20 4096 551.84 8192 574.99 16384 644.90 32768 790.70 65536 4169.19 131072 4756.38 262144 6782.57 524288 12299.24 1048576 22880.86 However, I get notmal results when the osu benchmarks through srun eg $ srun -N3 --cpus-per-task=8 /scratch/e1000/msopena/osu-micro-benchmarks-5.8/mpi/collective/osu_alltoall # OSU MPI All-to-All Personalized Exchange Latency Test v5.8 # Size Avg Latency(us) 1 27.43 2 27.97 4 29.26 8 30.69 16 31.77 32 34.89 64 42.06 128 60.83 256 115.84 512 205.33 1024 282.61 2048 446.41 4096 733.10 8192 1421.99 16384 2832.45 32768 5631.33 65536 11242.08 131072 22645.46 262144 45544.75 524288 90440.93 1048576 186709.32 Could someone please help me to troubleshoot could be wrong when I use mpich stand alone? thank you very much

1 0