- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Error in MPI call.
by Zhou, Hui 29 Apr '20

29 Apr '20

HI hritikesh semwal, The error says a connection error: Connection reset by peer. Are you running 8 processes on a single node or 8 nodes? -- Hui Zhou From: hritikesh semwal via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Tuesday, April 28, 2020 at 8:45 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: hritikesh semwal <hritikesh.semwal(a)gmail.com> Subject: [mpich-discuss] Error in MPI call. Hello all, Greetings of the day. I am working on the development of a parallel CFD solver in which I am using distributed graph communicator with MPI neighbourhood collective operation MPI_Neighbor_alltoallw. While running on 2 and 4 processors, the code is working fine but when I tried it on 8 processors it is showing following error, Fatal error in PMPI_Neighbor_alltoallw: Unknown error class, error stack: PMPI_Neighbor_alltoallw(299)..............: MPI_Neighbor_alltoallw(sendbuf=0x2fa8c20, sendcounts=0x1cd2300, sdispls=0x1cd2270, sendtypes=0x1cd2180, recvbuf=0x2fac1d0, recvcounts=0x7f167615d010, rdispls=0x7f1667ace010, recvtypes=0x7f1660786010, comm=comm=0x84000002) PMPI_Neighbor_alltoallw(285)..............: MPIR_Neighbor_alltoallw_impl(174).........: MPIR_Neighbor_alltoallw_intra_auto(93)....: MPIR_Neighbor_alltoallw_nb(33)............: MPIR_Wait_impl(77)........................: MPIDI_CH3i_Progress_wait(242).............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(479): MPIDI_CH3I_Socki_handle_read(4013)........: connection failure (set=0,sock=1,errno=104:Connection reset by peer) Can anyone tell me what is the solution to this problem? This would be a great help. Thank you.

1 0

Error in MPI call.
by hritikesh semwal 28 Apr '20

28 Apr '20

Hello all, Greetings of the day. I am working on the development of a parallel CFD solver in which I am using distributed graph communicator with MPI neighbourhood collective operation MPI_Neighbor_alltoallw. While running on 2 and 4 processors, the code is working fine but when I tried it on 8 processors it is showing following error, Fatal error in PMPI_Neighbor_alltoallw: Unknown error class, error stack: PMPI_Neighbor_alltoallw(299)..............: MPI_Neighbor_alltoallw(sendbuf=0x2fa8c20, sendcounts=0x1cd2300, sdispls=0x1cd2270, sendtypes=0x1cd2180, recvbuf=0x2fac1d0, recvcounts=0x7f167615d010, rdispls=0x7f1667ace010, recvtypes=0x7f1660786010, comm=comm=0x84000002) PMPI_Neighbor_alltoallw(285)..............: MPIR_Neighbor_alltoallw_impl(174).........: MPIR_Neighbor_alltoallw_intra_auto(93)....: MPIR_Neighbor_alltoallw_nb(33)............: MPIR_Wait_impl(77)........................: MPIDI_CH3i_Progress_wait(242).............: an error occurred while handling an event returned by MPIDI_CH3I_Sock_Wait() MPIDI_CH3I_Progress_handle_sock_event(479): MPIDI_CH3I_Socki_handle_read(4013)........: connection failure (set=0,sock=1,errno=104:Connection reset by peer) Can anyone tell me what is the solution to this problem? This would be a great help. Thank you.

1 0

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by Junchao Zhang 28 Apr '20

28 Apr '20

Got it. Thanks a lot. --Junchao Zhang On Tue, Apr 28, 2020 at 2:35 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > HI Junchao, > > > > As far as I know MPICH does not trap any signals. A segfault will simply > kill the mpi process. Hydra will see one of its child being killed by > signal and report so. So no, a segfault will not cause mpich to run > `MPIR_Handle_fatal_error()`. > > > > -- > Hui Zhou > > > > > > *From: *Junchao Zhang <junchao.zhang(a)gmail.com> > *Date: *Tuesday, April 28, 2020 at 2:17 PM > *To: *"Zhou, Hui" <zhouh(a)anl.gov> > *Cc: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Subject: *Re: [mpich-discuss] Should MPI_Abort() call exit or _exit? > > > > Then, will a segfault in MPICH cause it to eventually execute > MPIR_Handle_fatal_error()? > > --Junchao Zhang > > > > > > On Tue, Apr 28, 2020 at 1:11 PM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Hi Junchao, > > > > What discussed in this thread is to call `MPI_Abort` in a signal handler. > Since `MPI_Abort` is not interrupt safe – nearly all MPI functions for that > matter – we really shouldn’t call `MPI_Abort` in signal handlers. > > The normal error handling in MPICH are run normally, i.e. not in a signal > handler. MPICH functions are thread-safe, so it generally does not cause > problems. > > > > -- > Hui Zhou > > > > > > *From: *Junchao Zhang <junchao.zhang(a)gmail.com> > *Date: *Tuesday, April 28, 2020 at 1:05 PM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *"Zhou, Hui" <zhouh(a)anl.gov>, John Peterson <jwpeterson(a)gmail.com> > *Subject: *Re: [mpich-discuss] Should MPI_Abort() call exit or _exit? > > > > Hui, > > I am a bit confused. I browsed MPICH source code. The default MPI error > handler in MPICH is implemented by MPIR_Handle_fatal_error(), which calls > MPID_Abort(), which in turn calls MPL_exit(). So even PETSc does not take > over the error handling, users can still run into problems reported in this > email thread. Am I missing something? > > > > Thanks. > > --Junchao Zhang > > > > > > On Tue, Apr 21, 2020 at 1:18 PM Zhou, Hui via discuss <discuss(a)mpich.org> > wrote: > > Let’s make it clear – inside a signal handler, one is not allowed to call > any interrupt unsafe functions, that includes all MPI functions including > `MPI_Abort`, which means `MPL_exit` should never be called inside a signal > handler. Back to the question, since `MPL_exit` is not inside a signal > handler, it is allowed to call `exit`. > > > > I don’t think it is possible to do any real clean up inside a signal > handler. The best a signal handler can do is to flip some atomic flags, the > your applications should have checkpoints check those flags and do your > graceful exit if necessary. > > > > -- > Hui Zhou > > > > > > *From: *John Peterson via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Tuesday, April 21, 2020 at 1:06 PM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *John Peterson <jwpeterson(a)gmail.com> > *Subject: *[mpich-discuss] Should MPI_Abort() call exit or _exit? > > > > Hi, > > I have what is a fairly convoluted question involving several different > libraries, so I'll try to make it as succinct as possible. The issue is > that we have an MPI job that is canceled by slurm's "scancel", but instead > of exiting cleanly, the job (sometimes) hangs. I think we've tracked it > down to the job being canceled while in the middle of a call to "free" and > then "free" subsequently being called again from a function called by the > signal handler, which leads to the deadlock. The general rule seems to be > that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are > allowed to be called in signal handlers [0]. > > > > In our specific case, the stack trace of one of the hung jobs is: > > > > #0 __lll_lock_wait_private () at > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 > #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, > av=0x7ff175576c40 <main_arena>) at malloc.c:4266 > #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 > #3 0x00007ff1714938aa in H5MM_xfree () > #4 0x00007ff17147e5da in H5L_term_package () > #5 0x00007ff17133a766 in H5_term_library () > #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, > listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, > run_dtors=run_dtors@entry=true) at exit.c:108 > #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 > #8 0x00007ff176f3c809 in MPL_exit () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #9 0x00007ff176eefd4c in ?? () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #10 0x00007ff176e3ea59 in PMPI_Abort () from > /usr/lib/x86_64-linux-gnu/libmpich.so.0 > #11 0x00007ff177635531 in PetscSignalHandlerDefault () > > #12 0x00007ff177635270 in PetscSignalHandler_Private () > #13 <signal handler called> > #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, > av=0x7ff175576c40 <main_arena>) at malloc.c:4280 > #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 > > > > To summarize this: > > 1.) We are in a call to "free" > > 2.) The process receives a sigterm/sigkill signal, which petsc handles > 3.) petsc calls mpich's abort function > 4.) mpich calls "exit", which causes the "atexit" functions to run > > 5.) HDF5 registers an "atexit" function which also calls "free" > > 6.) Deadlock > > > > This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom > of registering "atexit" functions which free memory -- the program is > exiting after all. But, I also wanted to confirm whether calling exit() > from MPL_exit() is a deliberate choice, or if it could perhaps be changed > to _exit, which I think would avoid this particular problem. > > > > Thanks, > > John > > > > [0]: > https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono… > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > >

1 0

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by Zhou, Hui 28 Apr '20

28 Apr '20

HI Junchao, As far as I know MPICH does not trap any signals. A segfault will simply kill the mpi process. Hydra will see one of its child being killed by signal and report so. So no, a segfault will not cause mpich to run `MPIR_Handle_fatal_error()`. -- Hui Zhou From: Junchao Zhang <junchao.zhang(a)gmail.com> Date: Tuesday, April 28, 2020 at 2:17 PM To: "Zhou, Hui" <zhouh(a)anl.gov> Cc: "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Should MPI_Abort() call exit or _exit? Then, will a segfault in MPICH cause it to eventually execute MPIR_Handle_fatal_error()? --Junchao Zhang On Tue, Apr 28, 2020 at 1:11 PM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Junchao, What discussed in this thread is to call `MPI_Abort` in a signal handler. Since `MPI_Abort` is not interrupt safe – nearly all MPI functions for that matter – we really shouldn’t call `MPI_Abort` in signal handlers. The normal error handling in MPICH are run normally, i.e. not in a signal handler. MPICH functions are thread-safe, so it generally does not cause problems. -- Hui Zhou From: Junchao Zhang <junchao.zhang(a)gmail.com<mailto:[email protected]>> Date: Tuesday, April 28, 2020 at 1:05 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: "Zhou, Hui" <zhouh(a)anl.gov<mailto:[email protected]>>, John Peterson <jwpeterson(a)gmail.com<mailto:[email protected]>> Subject: Re: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hui, I am a bit confused. I browsed MPICH source code. The default MPI error handler in MPICH is implemented by MPIR_Handle_fatal_error(), which calls MPID_Abort(), which in turn calls MPL_exit(). So even PETSc does not take over the error handling, users can still run into problems reported in this email thread. Am I missing something? Thanks. --Junchao Zhang On Tue, Apr 21, 2020 at 1:18 PM Zhou, Hui via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Let’s make it clear – inside a signal handler, one is not allowed to call any interrupt unsafe functions, that includes all MPI functions including `MPI_Abort`, which means `MPL_exit` should never be called inside a signal handler. Back to the question, since `MPL_exit` is not inside a signal handler, it is allowed to call `exit`. I don’t think it is possible to do any real clean up inside a signal handler. The best a signal handler can do is to flip some atomic flags, the your applications should have checkpoints check those flags and do your graceful exit if necessary. -- Hui Zhou From: John Peterson via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, April 21, 2020 at 1:06 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: John Peterson <jwpeterson(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hi, I have what is a fairly convoluted question involving several different libraries, so I'll try to make it as succinct as possible. The issue is that we have an MPI job that is canceled by slurm's "scancel", but instead of exiting cleanly, the job (sometimes) hangs. I think we've tracked it down to the job being canceled while in the middle of a call to "free" and then "free" subsequently being called again from a function called by the signal handler, which leads to the deadlock. The general rule seems to be that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are allowed to be called in signal handlers [0]. In our specific case, the stack trace of one of the hung jobs is: #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, av=0x7ff175576c40 <main_arena>) at malloc.c:4266 #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 #3 0x00007ff1714938aa in H5MM_xfree () #4 0x00007ff17147e5da in H5L_term_package () #5 0x00007ff17133a766 in H5_term_library () #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 #8 0x00007ff176f3c809 in MPL_exit () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #9 0x00007ff176eefd4c in ?? () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #10 0x00007ff176e3ea59 in PMPI_Abort () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #11 0x00007ff177635531 in PetscSignalHandlerDefault () #12 0x00007ff177635270 in PetscSignalHandler_Private () #13 <signal handler called> #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, av=0x7ff175576c40 <main_arena>) at malloc.c:4280 #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 To summarize this: 1.) We are in a call to "free" 2.) The process receives a sigterm/sigkill signal, which petsc handles 3.) petsc calls mpich's abort function 4.) mpich calls "exit", which causes the "atexit" functions to run 5.) HDF5 registers an "atexit" function which also calls "free" 6.) Deadlock This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom of registering "atexit" functions which free memory -- the program is exiting after all. But, I also wanted to confirm whether calling exit() from MPL_exit() is a deliberate choice, or if it could perhaps be changed to _exit, which I think would avoid this particular problem. Thanks, John [0]: https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono… _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Should MPI_Abort() call exit or _exit?
by Zhou, Hui 28 Apr '20

28 Apr '20

Hi Junchao, What discussed in this thread is to call `MPI_Abort` in a signal handler. Since `MPI_Abort` is not interrupt safe – nearly all MPI functions for that matter – we really shouldn’t call `MPI_Abort` in signal handlers. The normal error handling in MPICH are run normally, i.e. not in a signal handler. MPICH functions are thread-safe, so it generally does not cause problems. -- Hui Zhou From: Junchao Zhang <junchao.zhang(a)gmail.com> Date: Tuesday, April 28, 2020 at 1:05 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Zhou, Hui" <zhouh(a)anl.gov>, John Peterson <jwpeterson(a)gmail.com> Subject: Re: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hui, I am a bit confused. I browsed MPICH source code. The default MPI error handler in MPICH is implemented by MPIR_Handle_fatal_error(), which calls MPID_Abort(), which in turn calls MPL_exit(). So even PETSc does not take over the error handling, users can still run into problems reported in this email thread. Am I missing something? Thanks. --Junchao Zhang On Tue, Apr 21, 2020 at 1:18 PM Zhou, Hui via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Let’s make it clear – inside a signal handler, one is not allowed to call any interrupt unsafe functions, that includes all MPI functions including `MPI_Abort`, which means `MPL_exit` should never be called inside a signal handler. Back to the question, since `MPL_exit` is not inside a signal handler, it is allowed to call `exit`. I don’t think it is possible to do any real clean up inside a signal handler. The best a signal handler can do is to flip some atomic flags, the your applications should have checkpoints check those flags and do your graceful exit if necessary. -- Hui Zhou From: John Peterson via discuss <discuss(a)mpich.org<mailto:[email protected]>> Reply-To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Date: Tuesday, April 21, 2020 at 1:06 PM To: "discuss(a)mpich.org<mailto:[email protected]>" <discuss(a)mpich.org<mailto:[email protected]>> Cc: John Peterson <jwpeterson(a)gmail.com<mailto:[email protected]>> Subject: [mpich-discuss] Should MPI_Abort() call exit or _exit? Hi, I have what is a fairly convoluted question involving several different libraries, so I'll try to make it as succinct as possible. The issue is that we have an MPI job that is canceled by slurm's "scancel", but instead of exiting cleanly, the job (sometimes) hangs. I think we've tracked it down to the job being canceled while in the middle of a call to "free" and then "free" subsequently being called again from a function called by the signal handler, which leads to the deadlock. The general rule seems to be that only "asynchronous-safe" functions (abort(), _Exit(), etc.) are allowed to be called in signal handlers [0]. In our specific case, the stack trace of one of the hung jobs is: #0 __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95 #1 0x00007ff175223114 in _int_free (have_lock=0, p=0x55cd9120f2a0, av=0x7ff175576c40 <main_arena>) at malloc.c:4266 #2 __GI___libc_free (mem=0x55cd9120f2b0) at malloc.c:3124 #3 0x00007ff1714938aa in H5MM_xfree () #4 0x00007ff17147e5da in H5L_term_package () #5 0x00007ff17133a766 in H5_term_library () #6 0x00007ff1751ce041 in __run_exit_handlers (status=59, listp=0x7ff175576718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #7 0x00007ff1751ce13a in __GI_exit (status=<optimized out>) at exit.c:139 #8 0x00007ff176f3c809 in MPL_exit () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #9 0x00007ff176eefd4c in ?? () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #10 0x00007ff176e3ea59 in PMPI_Abort () from /usr/lib/x86_64-linux-gnu/libmpich.so.0 #11 0x00007ff177635531 in PetscSignalHandlerDefault () #12 0x00007ff177635270 in PetscSignalHandler_Private () #13 <signal handler called> #14 0x00007ff175222c6f in _int_free (have_lock=0, p=0x55cd9345e360, av=0x7ff175576c40 <main_arena>) at malloc.c:4280 #15 __GI___libc_free (mem=0x55cd9345e370) at malloc.c:3124 To summarize this: 1.) We are in a call to "free" 2.) The process receives a sigterm/sigkill signal, which petsc handles 3.) petsc calls mpich's abort function 4.) mpich calls "exit", which causes the "atexit" functions to run 5.) HDF5 registers an "atexit" function which also calls "free" 6.) Deadlock This could definitely be seen as an HDF5 issue: I'm not sure of the wisdom of registering "atexit" functions which free memory -- the program is exiting after all. But, I also wanted to confirm whether calling exit() from MPL_exit() is a deliberate choice, or if it could perhaps be changed to _exit, which I think would avoid this particular problem. Thanks, John [0]: https://wiki.sei.cmu.edu/confluence/display/c/SIG30-C.+Call+only+asynchrono… _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH configure
by Zhou, Hui 24 Apr '20

24 Apr '20

Hi Bruce, Before give you solution (which I have no clue yet), let’s understand what is needed. Do you have a firewall rules between nodes? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Friday, April 24, 2020 at 2:54 PM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hui, mpirun is pointing to mpiexec.hydra. I’m using “mpirun -n 6 executable.x” to launch jobs. The system guys have given me some information about linking to the pmi libraries that I’m going to try to see if I can get srun to work properly. I will give that a try and see if it enables me to use srun. They also suggested trying to configure MPICH with --add-pmi, although that doesn’t look like an MPICH configuration option. I did see a –enable-pmiport option. Is that something I should try? Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Friday, April 24, 2020 at 10:59 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Mpirun, mpiexec, and mpiexec.hydra, are all the same binary, with previous two symbolic links to the last. Please verify. If not, then you have installation issue. Hydra will detect environment and utilize information gathered from, e.g. slurm. For example, it will gather that you are launching with a given number of processes. When in doubt, use explicit command line option, such as `-n <numprocs>`. By the way, what is your complete command line that you used to launch jobs in all the cases? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Friday, April 24, 2020 at 11:49 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I’m not sure what is going on with srun, but it doesn’t seem to work the way you are describing, at least on our system. I’ve got an inquiry into our system administrators asking about it but generally, when I launch with srun it looks like MPI_Comm_size is returns 1 for the size of MPI_COMM_WORLD no matter how many processors I’m actually running on. I’ve tried the following combinations: mpich-3.3.2 built without slurm Launch with mpiexec.hydra: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init Launch with mpiexec: hangs in MPI_Init mpich-3.3.2 built with slurm Launch with mpiexec: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init mpich-3.3.1 built with slurm Launch with mpiexec: runs okay Launch with srun: only get 1 processor Launch with mpirun: runs okay I was under the impression that MPICH created its own version of mpirun/mpiexec depending on what it found out about the scheduling system during configuration and then built mpirun or mpiexec accordingly. Is this not correct? Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Thursday, April 23, 2020 at 1:31 PM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, I noticed you are mixing up things a bit, so let’s clear it up first: * If you compile with slurm then you should only launch with srun, not mpirun. Otherwise it won’t work. * If you compile without slurm, then you should launch with mpirun, or more precisely, mpiexec.hydra. Depend on which you are doing, each may have issues in your envioronment, but whatever issues are probably unrelated, and should not be discussed in a same context. What is working with your with 3.3.1? With slurm? If that’s what’s working for you, then let’s focus on compile and run 3.3.2 with slurm as well. -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, April 23, 2020 at 11:08 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I was launching the jobs with mpirun. I tried launching the jobs with srun and they no longer hang, but it looks like they are returning an MPI_COMM_WORLD with only one process, although I didn’t investigate this extensively. I also tried Pavan’s suggestion and rebuilt 3.3.2 without slurm and ran it with mpiexec. This also hangs and the error I was seeing previously reappears [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!) srun: error: node168: task 1: Exited with exit code 5 srun: Job step aborted: Waiting up to 2 seconds for job step to finish. slurmstepd: *** STEP 13201325.0 CANCELLED AT 2020-04-23T08:58:02 *** on node100 slurmstepd: *** JOB 13201325 CANCELLED AT 2020-04-23T08:58:02 *** on node100 [mpiexec(a)node100.local] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)node100.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)node100.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)node100.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)node100.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)node100.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion I suppose this could be a firewall issue, but there must also be some changes between 3.3.1 and 3.3.2, otherwise it should be a problem for all versions of mpich. Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 10:22 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, How did you launch your jobs? Since you configured with slurm, you should launch your job with `srun`, right? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:55 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, Its currently 14.03.8 Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 9:36 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Thanks for the effort checking these versions. What is the slurm versions that you have on cluster? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:02 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran 3.3rc1: Works 3.3: Works 3.3.1: Works 3.3.2: Hangs 3.4a2: Hangs I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb (gdb) where #0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0 #1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #6 0x0000000000408d0e in main () Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Monday, April 20, 2020 at 10:25 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Re: [mpich-discuss] MPICH configure
by Palmer, Bruce J 24 Apr '20

24 Apr '20

Hui, mpirun is pointing to mpiexec.hydra. I’m using “mpirun -n 6 executable.x” to launch jobs. The system guys have given me some information about linking to the pmi libraries that I’m going to try to see if I can get srun to work properly. I will give that a try and see if it enables me to use srun. They also suggested trying to configure MPICH with --add-pmi, although that doesn’t look like an MPICH configuration option. I did see a –enable-pmiport option. Is that something I should try? Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Friday, April 24, 2020 at 10:59 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Mpirun, mpiexec, and mpiexec.hydra, are all the same binary, with previous two symbolic links to the last. Please verify. If not, then you have installation issue. Hydra will detect environment and utilize information gathered from, e.g. slurm. For example, it will gather that you are launching with a given number of processes. When in doubt, use explicit command line option, such as `-n <numprocs>`. By the way, what is your complete command line that you used to launch jobs in all the cases? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Friday, April 24, 2020 at 11:49 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I’m not sure what is going on with srun, but it doesn’t seem to work the way you are describing, at least on our system. I’ve got an inquiry into our system administrators asking about it but generally, when I launch with srun it looks like MPI_Comm_size is returns 1 for the size of MPI_COMM_WORLD no matter how many processors I’m actually running on. I’ve tried the following combinations: mpich-3.3.2 built without slurm Launch with mpiexec.hydra: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init Launch with mpiexec: hangs in MPI_Init mpich-3.3.2 built with slurm Launch with mpiexec: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init mpich-3.3.1 built with slurm Launch with mpiexec: runs okay Launch with srun: only get 1 processor Launch with mpirun: runs okay I was under the impression that MPICH created its own version of mpirun/mpiexec depending on what it found out about the scheduling system during configuration and then built mpirun or mpiexec accordingly. Is this not correct? Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Thursday, April 23, 2020 at 1:31 PM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, I noticed you are mixing up things a bit, so let’s clear it up first: * If you compile with slurm then you should only launch with srun, not mpirun. Otherwise it won’t work. * If you compile without slurm, then you should launch with mpirun, or more precisely, mpiexec.hydra. Depend on which you are doing, each may have issues in your envioronment, but whatever issues are probably unrelated, and should not be discussed in a same context. What is working with your with 3.3.1? With slurm? If that’s what’s working for you, then let’s focus on compile and run 3.3.2 with slurm as well. -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, April 23, 2020 at 11:08 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I was launching the jobs with mpirun. I tried launching the jobs with srun and they no longer hang, but it looks like they are returning an MPI_COMM_WORLD with only one process, although I didn’t investigate this extensively. I also tried Pavan’s suggestion and rebuilt 3.3.2 without slurm and ran it with mpiexec. This also hangs and the error I was seeing previously reappears [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!) srun: error: node168: task 1: Exited with exit code 5 srun: Job step aborted: Waiting up to 2 seconds for job step to finish. slurmstepd: *** STEP 13201325.0 CANCELLED AT 2020-04-23T08:58:02 *** on node100 slurmstepd: *** JOB 13201325 CANCELLED AT 2020-04-23T08:58:02 *** on node100 [mpiexec(a)node100.local] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)node100.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)node100.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)node100.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)node100.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)node100.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion I suppose this could be a firewall issue, but there must also be some changes between 3.3.1 and 3.3.2, otherwise it should be a problem for all versions of mpich. Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 10:22 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, How did you launch your jobs? Since you configured with slurm, you should launch your job with `srun`, right? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:55 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, Its currently 14.03.8 Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 9:36 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Thanks for the effort checking these versions. What is the slurm versions that you have on cluster? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:02 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran 3.3rc1: Works 3.3: Works 3.3.1: Works 3.3.2: Hangs 3.4a2: Hangs I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb (gdb) where #0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0 #1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #6 0x0000000000408d0e in main () Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Monday, April 20, 2020 at 10:25 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Re: [mpich-discuss] MPICH configure
by Zhou, Hui 24 Apr '20

24 Apr '20

Hi Bruce, Mpirun, mpiexec, and mpiexec.hydra, are all the same binary, with previous two symbolic links to the last. Please verify. If not, then you have installation issue. Hydra will detect environment and utilize information gathered from, e.g. slurm. For example, it will gather that you are launching with a given number of processes. When in doubt, use explicit command line option, such as `-n <numprocs>`. By the way, what is your complete command line that you used to launch jobs in all the cases? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Friday, April 24, 2020 at 11:49 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I’m not sure what is going on with srun, but it doesn’t seem to work the way you are describing, at least on our system. I’ve got an inquiry into our system administrators asking about it but generally, when I launch with srun it looks like MPI_Comm_size is returns 1 for the size of MPI_COMM_WORLD no matter how many processors I’m actually running on. I’ve tried the following combinations: mpich-3.3.2 built without slurm Launch with mpiexec.hydra: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init Launch with mpiexec: hangs in MPI_Init mpich-3.3.2 built with slurm Launch with mpiexec: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init mpich-3.3.1 built with slurm Launch with mpiexec: runs okay Launch with srun: only get 1 processor Launch with mpirun: runs okay I was under the impression that MPICH created its own version of mpirun/mpiexec depending on what it found out about the scheduling system during configuration and then built mpirun or mpiexec accordingly. Is this not correct? Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Thursday, April 23, 2020 at 1:31 PM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, I noticed you are mixing up things a bit, so let’s clear it up first: * If you compile with slurm then you should only launch with srun, not mpirun. Otherwise it won’t work. * If you compile without slurm, then you should launch with mpirun, or more precisely, mpiexec.hydra. Depend on which you are doing, each may have issues in your envioronment, but whatever issues are probably unrelated, and should not be discussed in a same context. What is working with your with 3.3.1? With slurm? If that’s what’s working for you, then let’s focus on compile and run 3.3.2 with slurm as well. -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, April 23, 2020 at 11:08 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I was launching the jobs with mpirun. I tried launching the jobs with srun and they no longer hang, but it looks like they are returning an MPI_COMM_WORLD with only one process, although I didn’t investigate this extensively. I also tried Pavan’s suggestion and rebuilt 3.3.2 without slurm and ran it with mpiexec. This also hangs and the error I was seeing previously reappears [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!) srun: error: node168: task 1: Exited with exit code 5 srun: Job step aborted: Waiting up to 2 seconds for job step to finish. slurmstepd: *** STEP 13201325.0 CANCELLED AT 2020-04-23T08:58:02 *** on node100 slurmstepd: *** JOB 13201325 CANCELLED AT 2020-04-23T08:58:02 *** on node100 [mpiexec(a)node100.local] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)node100.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)node100.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)node100.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)node100.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)node100.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion I suppose this could be a firewall issue, but there must also be some changes between 3.3.1 and 3.3.2, otherwise it should be a problem for all versions of mpich. Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 10:22 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, How did you launch your jobs? Since you configured with slurm, you should launch your job with `srun`, right? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:55 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, Its currently 14.03.8 Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 9:36 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Thanks for the effort checking these versions. What is the slurm versions that you have on cluster? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:02 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran 3.3rc1: Works 3.3: Works 3.3.1: Works 3.3.2: Hangs 3.4a2: Hangs I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb (gdb) where #0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0 #1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #6 0x0000000000408d0e in main () Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Monday, April 20, 2020 at 10:25 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Re: [mpich-discuss] MPICH configure
by Congiu, Giuseppe 24 Apr '20

24 Apr '20

Hi Bruce, I am not sure why you are getting only one process with srun. There is a bug in all those versions of mpich you have listed when running with slurm, but that concerns the misdetection of nodes when they use some naming scheme that is not recognized. If this can be of any help, that bug was fixed in the following PR and was included in version 3.4a2 https://github.com/pmodels/mpich/pull/3947 Best, Giuseppe From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Friday, April 24, 2020 at 11:49 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I’m not sure what is going on with srun, but it doesn’t seem to work the way you are describing, at least on our system. I’ve got an inquiry into our system administrators asking about it but generally, when I launch with srun it looks like MPI_Comm_size is returns 1 for the size of MPI_COMM_WORLD no matter how many processors I’m actually running on. I’ve tried the following combinations: mpich-3.3.2 built without slurm Launch with mpiexec.hydra: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init Launch with mpiexec: hangs in MPI_Init mpich-3.3.2 built with slurm Launch with mpiexec: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init mpich-3.3.1 built with slurm Launch with mpiexec: runs okay Launch with srun: only get 1 processor Launch with mpirun: runs okay I was under the impression that MPICH created its own version of mpirun/mpiexec depending on what it found out about the scheduling system during configuration and then built mpirun or mpiexec accordingly. Is this not correct? Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Thursday, April 23, 2020 at 1:31 PM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, I noticed you are mixing up things a bit, so let’s clear it up first: * If you compile with slurm then you should only launch with srun, not mpirun. Otherwise it won’t work. * If you compile without slurm, then you should launch with mpirun, or more precisely, mpiexec.hydra. Depend on which you are doing, each may have issues in your envioronment, but whatever issues are probably unrelated, and should not be discussed in a same context. What is working with your with 3.3.1? With slurm? If that’s what’s working for you, then let’s focus on compile and run 3.3.2 with slurm as well. -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, April 23, 2020 at 11:08 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I was launching the jobs with mpirun. I tried launching the jobs with srun and they no longer hang, but it looks like they are returning an MPI_COMM_WORLD with only one process, although I didn’t investigate this extensively. I also tried Pavan’s suggestion and rebuilt 3.3.2 without slurm and ran it with mpiexec. This also hangs and the error I was seeing previously reappears [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!) srun: error: node168: task 1: Exited with exit code 5 srun: Job step aborted: Waiting up to 2 seconds for job step to finish. slurmstepd: *** STEP 13201325.0 CANCELLED AT 2020-04-23T08:58:02 *** on node100 slurmstepd: *** JOB 13201325 CANCELLED AT 2020-04-23T08:58:02 *** on node100 [mpiexec(a)node100.local] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)node100.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)node100.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)node100.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)node100.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)node100.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion I suppose this could be a firewall issue, but there must also be some changes between 3.3.1 and 3.3.2, otherwise it should be a problem for all versions of mpich. Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 10:22 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, How did you launch your jobs? Since you configured with slurm, you should launch your job with `srun`, right? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:55 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, Its currently 14.03.8 Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 9:36 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Thanks for the effort checking these versions. What is the slurm versions that you have on cluster? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:02 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran 3.3rc1: Works 3.3: Works 3.3.1: Works 3.3.2: Hangs 3.4a2: Hangs I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb (gdb) where #0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0 #1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #6 0x0000000000408d0e in main () Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Monday, April 20, 2020 at 10:25 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0

Re: [mpich-discuss] MPICH configure
by Palmer, Bruce J 24 Apr '20

24 Apr '20

Hi Hui, I’m not sure what is going on with srun, but it doesn’t seem to work the way you are describing, at least on our system. I’ve got an inquiry into our system administrators asking about it but generally, when I launch with srun it looks like MPI_Comm_size is returns 1 for the size of MPI_COMM_WORLD no matter how many processors I’m actually running on. I’ve tried the following combinations: mpich-3.3.2 built without slurm Launch with mpiexec.hydra: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init Launch with mpiexec: hangs in MPI_Init mpich-3.3.2 built with slurm Launch with mpiexec: hangs in MPI_Init Launch with srun: only get 1 processor Launch with mpirun: hangs in MPI_Init mpich-3.3.1 built with slurm Launch with mpiexec: runs okay Launch with srun: only get 1 processor Launch with mpirun: runs okay I was under the impression that MPICH created its own version of mpirun/mpiexec depending on what it found out about the scheduling system during configuration and then built mpirun or mpiexec accordingly. Is this not correct? Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Thursday, April 23, 2020 at 1:31 PM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, I noticed you are mixing up things a bit, so let’s clear it up first: * If you compile with slurm then you should only launch with srun, not mpirun. Otherwise it won’t work. * If you compile without slurm, then you should launch with mpirun, or more precisely, mpiexec.hydra. Depend on which you are doing, each may have issues in your envioronment, but whatever issues are probably unrelated, and should not be discussed in a same context. What is working with your with 3.3.1? With slurm? If that’s what’s working for you, then let’s focus on compile and run 3.3.2 with slurm as well. -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, April 23, 2020 at 11:08 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I was launching the jobs with mpirun. I tried launching the jobs with srun and they no longer hang, but it looks like they are returning an MPI_COMM_WORLD with only one process, although I didn’t investigate this extensively. I also tried Pavan’s suggestion and rebuilt 3.3.2 without slurm and ran it with mpiexec. This also hangs and the error I was seeing previously reappears [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!) srun: error: node168: task 1: Exited with exit code 5 srun: Job step aborted: Waiting up to 2 seconds for job step to finish. slurmstepd: *** STEP 13201325.0 CANCELLED AT 2020-04-23T08:58:02 *** on node100 slurmstepd: *** JOB 13201325 CANCELLED AT 2020-04-23T08:58:02 *** on node100 [mpiexec(a)node100.local] HYDU_sock_write (utils/sock/sock.c:256): write error (Bad file descriptor) [mpiexec(a)node100.local] HYD_pmcd_pmiserv_send_signal (pm/pmiserv/pmiserv_cb.c:178): unable to write data to proxy [mpiexec(a)node100.local] ui_cmd_cb (pm/pmiserv/pmiserv_pmci.c:77): unable to send signal downstream [mpiexec(a)node100.local] HYDT_dmxu_poll_wait_for_event (tools/demux/demux_poll.c:77): callback returned error status [mpiexec(a)node100.local] HYD_pmci_wait_for_completion (pm/pmiserv/pmiserv_pmci.c:196): error waiting for event [mpiexec(a)node100.local] main (ui/mpich/mpiexec.c:336): process manager error waiting for completion I suppose this could be a firewall issue, but there must also be some changes between 3.3.1 and 3.3.2, otherwise it should be a problem for all versions of mpich. Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 10:22 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, How did you launch your jobs? Since you configured with slurm, you should launch your job with `srun`, right? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:55 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, Its currently 14.03.8 Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Wednesday, April 22, 2020 at 9:36 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Thanks for the effort checking these versions. What is the slurm versions that you have on cluster? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Wednesday, April 22, 2020 at 11:02 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I rebuilt everything from scratch and tried running several versions of mpich. Release 3.3.1 seems to work okay but 3.3.2 hangs. Here is a complete summary of the versions I ran 3.3rc1: Works 3.3: Works 3.3.1: Works 3.3.2: Hangs 3.4a2: Hangs I’m not seeing the error message from hydra anymore (I have no idea why not), but I logged into one of the hung processes when running with 3.3.2 and got the following listing from gdb (gdb) where #0 0x0000003d7ce0e810 in __read_nocancel () from /lib64/libpthread.so.0 #1 0x00002aaaac6e936e in PMIU_readline () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #2 0x00002aaaac6e985b in GetResponse.part.0 () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #3 0x00002aaaac6e4e36 in MPIDU_shm_seg_commit () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #4 0x00002aaaabc541dc in MPIR_Init_thread () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #5 0x00002aaaabc3db7e in PMPI_Init () from /people/d3g293/mpich/mpich-3.3.2/install/lib/libmpi.so.12 #6 0x0000000000408d0e in main () Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Monday, April 20, 2020 at 10:25 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: Re: [mpich-discuss] MPICH configure The error is from `hydra`, which should not have changed much between the versions. Could you verify that 3.3.1 still works for you? -- Hui Zhou From: "Palmer, Bruce J via discuss" <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, April 16, 2020 at 5:48 PM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Subject: [mpich-discuss] MPICH configure Hi, I’ve been building MPICH on are aging Infiniband cluster using the following formula ./configure --prefix=/people/d3g293/mpich/mpich-3.3.2/install --with-device=ch4:ofi:sockets --with-libfabric=embedded --enable-threads=multiple --with-slurm CC=gcc CXX=g++ It’s been working pretty well but I recently tried to build mpich-3.3.2 and mpich-3.4a2 and although the build seems to work okay, I’m having problems actually running anything. If I run on 2 nodes the code seems to hang on MPI_Init and it looks like it is producing the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node013.l ocal" to "node012.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node012.local at p ort 37769 (check for firewalls!) srun: error: node013: task 1: Exited with exit code 5 If I run on a single node, things seem to work. Any idea what is going on here? I’ve got a working build of mpich-3.3, so things were okay up until recently. Has something in MPICH changed and my configuration formula is no good, or is this more likely to be due to some system modification? Bruce Palmer Senior Research Scientist Pacific Northwest National Laboratory Richland, WA 99352 (509) 375-3899

1 0