February 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] MPI_Info key host error in SLURM
by Raffenetti, Kenneth J. 26 Feb '21

26 Feb '21

Hi, Could you add "-v" to your mpiexec command and provide the output? The "host" info key is handled by the process manager when executing the spawn command. Ken On 2/24/21, 11:55 AM, "Iker Martín Álvarez via discuss" <discuss(a)mpich.org> wrote: Hi, I have been working around the MPI_Comm_spawn function with the key "host" along a value in a system with the SLURM resource manager (slurm-wlm 17.11.2). The function works as expected, but when I send to execute the code with the sbatch command, an error arises. This does not happen when I execute directly the code in the same machine SLURM decided to execute it when it was sended with sbatch. In both cases with the key "host", as when I do not use the key, it works just fine. The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives different errors. Also, I tried it with other implementations (OpenMPI and Intel MPI), which works as expected creating the processes in the indicated host. I would like to create processes by MPI_Comm_spawn in an assigned host, so if there are other key values for the Info argument, I could try them, but I have not found any looking through MPICH documentation. Here is the code I have been using: int main(int argc, char ** argv) { int myId, numP; MPI_Info info; MPI_Comm comm; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myId); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm comm_par; MPI_Comm_get_parent(&comm_par); if(comm_par != MPI_COMM_NULL ) { if(myId == ROOT) { printf("SONS\n"); fflush(stdout); } } else { if(myId == ROOT) { printf("FATHERS\n"); fflush(stdout); } MPI_Info_create(&info); MPI_Info_set(info, "host", "n00"); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE); } MPI_Finalize(); return 0; } For MPICH 3.4.1 there is sometimes no error and the code stops working at MPI_Comm_spawn function, other times this error is shown: Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Internal MPI error!, error stack: MPIR_Init_thread(152)...: MPID_Init(562)..........: MPIDU_Init_shm_init(195): Init_shm_barrier(94)....: Internal MPI error! barrier not initialized Also, the error code for MPICH 3.3.2: Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable internal ABORT - process 0 Thanks, Iker

1 0

MPI_Info key host error in SLURM
by Iker Martín Álvarez 24 Feb '21

24 Feb '21

Hi, I have been working around the MPI_Comm_spawn function with the key "host" along a value in a system with the SLURM resource manager (slurm-wlm 17.11.2). The function works as expected, but when I send to execute the code with the sbatch command, an error arises. This does not happen when I execute directly the code in the same machine SLURM decided to execute it when it was sended with *sbatch*. In both cases with the key "host", as when I do not use the key, it works just fine. The same code has been tested with MPICH 3.3.2 and 3.4.1, which gives different errors. Also, I tried it with other implementations (OpenMPI and Intel MPI), which works as expected creating the processes in the indicated host. I would like to create processes by MPI_Comm_spawn in an assigned host, so if there are other key values for the Info argument, I could try them, but I have not found any looking through MPICH documentation. Here is the code I have been using: int main(int argc, char ** argv) { int myId, numP; MPI_Info info; MPI_Comm comm; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &myId); MPI_Comm_size(MPI_COMM_WORLD, &numP); MPI_Comm comm_par; MPI_Comm_get_parent(&comm_par); if(comm_par != MPI_COMM_NULL ) { if(myId == ROOT) { printf("SONS\n"); fflush(stdout); } } else { if(myId == ROOT) { printf("FATHERS\n"); fflush(stdout); } MPI_Info_create(&info); MPI_Info_set(info, "host", "n00"); MPI_Comm_spawn(argv[0], MPI_ARGV_NULL, numP, info, ROOT, MPI_COMM_WORLD, &comm, MPI_ERRCODES_IGNORE); } MPI_Finalize(); return 0; } For MPICH 3.4.1 there is sometimes no error and the code stops working at MPI_Comm_spawn function, other times this error is shown: Abort(1615120) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init: Internal MPI error!, error stack: MPIR_Init_thread(152)...: MPID_Init(562)..........: MPIDU_Init_shm_init(195): Init_shm_barrier(94)....: Internal MPI error! barrier not initialized Also, the error code for MPICH 3.3.2: Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size Assertion failed in file src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c at line 683: our_pg_rank < pg->size libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable libbacktrace: no debug info in ELF executable internal ABORT - process 0 Thanks, Iker

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Calling MPI_Finalize or not
by Raffenetti, Kenneth J. 21 Feb '21

21 Feb '21

Kurt, Yes, that is correct. Spawned processes should be sure to call MPI_COMM_DISCONNECT on their parent communicator before MPI_FINALIZE. Ken On 2/21/21, 4:13 PM, "Mccall, Kurt E. (MSFC-EV41) via discuss" <discuss(a)mpich.org> wrote: Hui, So are you saying that the worker can call MPI_Finalize and it won’t wait for the manager that spawned it to do the same? That would be ideal. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Sunday, February 21, 2021 4:07 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Calling MPI_Finalize or not The worker should call `MPI_Finalize` before exit. If somehow it hangs, assuming all the resources have been freed/disconnected, it must be a bug, please file an issue and we’ll track it down. Each spawned process has a different `MPI_COMM_WORLD` from the process who spawned it. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Sunday, February 21, 2021 at 3:06 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Calling MPI_Finalize or not Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn’t sit there taking up a core’s resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0

Re: [mpich-discuss] Does mpich have or have planned fine grained affinity control?
by Zhou, Hui 21 Feb '21

21 Feb '21

You can manually set the exact binding you desire. For example, assuming you have 8 cores, you may try – mpiexec -bind-to user:0,1+2+3+4+5+6+7 -ppn 2 -n …. to bind the first process to core 0, and the 2nd process to 1-7. Additional help is available with mpiexec -bind-to -help -- Hui Zhou From: John Cary via discuss <discuss(a)mpich.org> Date: Sunday, February 21, 2021 at 3:18 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: John Cary <cary(a)colorado.edu> Subject: [mpich-discuss] Does mpich have or have planned fine grained affinity control? I would like to, eg, launch 2 processes per node, with one bound to one core (used to drive a GPU), and the other to use the remaining cores for doing thread-parallel computing. Is this possible with MPICH? Thx.....John Cary _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Calling MPI_Finalize or not
by Mccall, Kurt E. (MSFC-EV41) 21 Feb '21

21 Feb '21

Hui, So are you saying that the worker can call MPI_Finalize and it won't wait for the manager that spawned it to do the same? That would be ideal. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Sunday, February 21, 2021 4:07 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Calling MPI_Finalize or not The worker should call `MPI_Finalize` before exit. If somehow it hangs, assuming all the resources have been freed/disconnected, it must be a bug, please file an issue and we'll track it down. Each spawned process has a different `MPI_COMM_WORLD` from the process who spawned it. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Sunday, February 21, 2021 at 3:06 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Calling MPI_Finalize or not Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn't sit there taking up a core's resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0

Re: [mpich-discuss] Calling MPI_Finalize or not
by Zhou, Hui 21 Feb '21

21 Feb '21

The worker should call `MPI_Finalize` before exit. If somehow it hangs, assuming all the resources have been freed/disconnected, it must be a bug, please file an issue and we’ll track it down. Each spawned process has a different `MPI_COMM_WORLD` from the process who spawned it. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Sunday, February 21, 2021 at 3:06 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Calling MPI_Finalize or not Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn’t sit there taking up a core’s resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0

Does mpich have or have planned fine grained affinity control?
by John Cary 21 Feb '21

21 Feb '21

I would like to, eg, launch 2 processes per node, with one bound to one core (used to drive a GPU), and the other to use the remaining cores for doing thread-parallel computing. Is this possible with MPICH? Thx.....John Cary

1 0

Calling MPI_Finalize or not
by Mccall, Kurt E. (MSFC-EV41) 21 Feb '21

21 Feb '21

Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn't sit there taking up a core's resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0

Re: [mpich-discuss] Fatal error in PMPI_Isend: Unknown error class
by Raffenetti, Kenneth J. 19 Feb '21

19 Feb '21

On 2/18/21, 1:03 AM, "Brent Morgan via discuss" <discuss(a)mpich.org> wrote: all the questions we've submitted, none of them were answered or considered at all in neither discuss(a)mpich.org or devel(a)mpich.org <mailto:[email protected]> Can you let us know which emails have gone unanswered? I see that aside from this email which went unanswered for 4 days, the 2 previous questions you posted to discuss and devel had some back and forth. Maybe we are missing some other questions? As others have already said, sample code and/or minimal reproducers would be a sure way to accelerate productive responses. Ken

1 0

Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur)
by Eric Borisch 19 Feb '21

19 Feb '21

Yes: (Again, only testing locally) $ MPIR_CVAR_OFI_SKIP_IPV6=1 mpiexec -np 2 ./cpi Process 0 of 2 is on <HOSTNAME> Process 1 of 2 is on <HOSTNAME> pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.000268 while without: $ mpiexec -np 2 ./cpi Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_init.c at line 1988: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_init.c at line 1988: mapped_table[i] != FI_ADDR_NOTAVAIL 0 libpmpi.12.dylib 0x000000010d3e8b04 MPL_backtrace_show + 52 Thanks! I'll patch our installation to crowbar that to 1 for now, if that seems reasonable. Thanks, - Eric On Fri, Feb 19, 2021 at 7:56 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Does setting environment variable `MPIR_CVAR_OFI_SKIP_IPV6=1` work? > > > > -- > Hui Zhou > > > > > > From: Eric Borisch via discuss <discuss(a)mpich.org> > Date: Thursday, February 18, 2021 at 7:36 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Eric Borisch <eborisch(a)gmail.com> > Subject: Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur) > > Blaise, > > > > This is 3.4.1 and CH4? > > > > With —enable-fast=all, asserts are disabled; can you see if it will compile and execute mpi binaries when built without that option? I suspect it will catch the invalid condition (with the non-elided assert), and would likewise crash if it was used in actual distributed (non-localhost) execution. > > > > Let me know; if it doesn’t crash for you I would love to figure out why. > > > > - Eric > > > > On Thu, Feb 18, 2021 at 3:59 PM Blaise A Bourdin via discuss <discuss(a)mpich.org> wrote: > > Hi, > > Absolutely no issues here when using gcc10 from homebrew both on intel and ARM. > > Here is my configuration options: > > ./configure --prefix=$MPICH_HOME \ > > --enable-fast=all,O3 \ > > --enable-g=dbg \ > > --enable-romio \ > > --enable-shared \ > > --enable-sharedlibs=gcc-osx \ > > --with-pm=hydra \ > > CC=gcc-10 CXX=g++-10 FC=gfortran-10 F77=gfortran-10 \ > > CFLAGS="-fgnu89-inline -Wno-deprecated" \ > > CXXFLAGS="-Wno-deprecated" \ > > FFLAGS="-fallow-argument-mismatch" > > > > Blaise > > > > On Feb 18, 2021, at 3:53 PM, Allen Sanderson via discuss <discuss(a)mpich.org> wrote: > > > > Hello, > > > > First post to the discussion board. I made the mistake of upgrading to macOS 11.2 (aka Big Sur). I was able to build MPICH 3.4.1 successfully and integrate it with an application. However, when I ran, MPICH crashed. Here is the initial output: > > > > Abort(1) on node 0: Internal error > > 0 libpmpi.12.dylib 0x00000001068c2159 MPL_backtrace_show + 57 > > 1 libpmpi.12.dylib 0x000000010684e874 MPIR_Assert_fail + 36 > > 2 libpmpi.12.dylib 0x0000000106897bc8 MPIDI_OFI_mpi_init_hook + 4472 > > 3 libpmpi.12.dylib 0x00000001068790d3 MPID_Init + 3091 > > 4 libpmpi.12.dylib 0x00000001066ae70b MPIR_Init_thread + 219 > > 5 libmpi.12.dylib 0x00000001064bb897 MPI_Init + 279 > > > > I have not yet dug into the issue any further. Before doing so I wanted to first check to see if others have successfully built and run via macOS 11.2. If so I would appreciate any pointers. If it is known that it will not yet work, that would be good information as well so not to spend anytime digging. > > > > I should note that on another machine, I was wiser and upgraded to macOS 10.5.7 (aka Catalina) and built and ran with 3.3.1 that worked fine running in parallel. > > > > Cheers, > > > > Allen > > > > > > Allen Sanderson > > SCI Institute > > University of Utah > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > > A.K. & Shirley Barton Professor of Mathematics > > Adjunct Professor of Mechanical Engineering > > Adjunct of the Center for Computation & Technology > > Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA > > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0