- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Does mpich have or have planned fine grained affinity control?
by Zhou, Hui 21 Feb '21

21 Feb '21

You can manually set the exact binding you desire. For example, assuming you have 8 cores, you may try – mpiexec -bind-to user:0,1+2+3+4+5+6+7 -ppn 2 -n …. to bind the first process to core 0, and the 2nd process to 1-7. Additional help is available with mpiexec -bind-to -help -- Hui Zhou From: John Cary via discuss <discuss(a)mpich.org> Date: Sunday, February 21, 2021 at 3:18 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: John Cary <cary(a)colorado.edu> Subject: [mpich-discuss] Does mpich have or have planned fine grained affinity control? I would like to, eg, launch 2 processes per node, with one bound to one core (used to drive a GPU), and the other to use the remaining cores for doing thread-parallel computing. Is this possible with MPICH? Thx.....John Cary _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] [EXTERNAL] Re: Calling MPI_Finalize or not
by Mccall, Kurt E. (MSFC-EV41) 21 Feb '21

21 Feb '21

Hui, So are you saying that the worker can call MPI_Finalize and it won't wait for the manager that spawned it to do the same? That would be ideal. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Sunday, February 21, 2021 4:07 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Calling MPI_Finalize or not The worker should call `MPI_Finalize` before exit. If somehow it hangs, assuming all the resources have been freed/disconnected, it must be a bug, please file an issue and we'll track it down. Each spawned process has a different `MPI_COMM_WORLD` from the process who spawned it. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Date: Sunday, February 21, 2021 at 3:06 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Calling MPI_Finalize or not Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn't sit there taking up a core's resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0

Re: [mpich-discuss] Calling MPI_Finalize or not
by Zhou, Hui 21 Feb '21

21 Feb '21

The worker should call `MPI_Finalize` before exit. If somehow it hangs, assuming all the resources have been freed/disconnected, it must be a bug, please file an issue and we’ll track it down. Each spawned process has a different `MPI_COMM_WORLD` from the process who spawned it. -- Hui Zhou From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Date: Sunday, February 21, 2021 at 3:06 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Calling MPI_Finalize or not Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn’t sit there taking up a core’s resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0

Does mpich have or have planned fine grained affinity control?
by John Cary 21 Feb '21

21 Feb '21

I would like to, eg, launch 2 processes per node, with one bound to one core (used to drive a GPU), and the other to use the remaining cores for doing thread-parallel computing. Is this possible with MPICH? Thx.....John Cary

1 0

Calling MPI_Finalize or not
by Mccall, Kurt E. (MSFC-EV41) 21 Feb '21

21 Feb '21

Two questions: I have a group of manager processes, each of which uses MPI_Comm_spawn() to create individual worker processes one at a time. Occasionally, a worker has to exit and is replaced by another worker via MPI_Comm_spawn(). If the exiting worker calls MPI_Finalize(), will it sit there waiting for all other managers and workers in MPI_COMM_WORLD to do the same, or will it wait only for the manager that created its inter-communicator to call MPI_Finalize? In either case, maybe it is better for the worker to NOT call MPI_Finalize at all so that it doesn't sit there taking up a core's resources while it waits for the others. What are the implications of not calling MPI_Finalize in the workers? Thanks, Kurt

1 0

Re: [mpich-discuss] Fatal error in PMPI_Isend: Unknown error class
by Raffenetti, Kenneth J. 19 Feb '21

19 Feb '21

On 2/18/21, 1:03 AM, "Brent Morgan via discuss" <discuss(a)mpich.org> wrote: all the questions we've submitted, none of them were answered or considered at all in neither discuss(a)mpich.org or devel(a)mpich.org <mailto:[email protected]> Can you let us know which emails have gone unanswered? I see that aside from this email which went unanswered for 4 days, the 2 previous questions you posted to discuss and devel had some back and forth. Maybe we are missing some other questions? As others have already said, sample code and/or minimal reproducers would be a sure way to accelerate productive responses. Ken

1 0

Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur)
by Eric Borisch 19 Feb '21

19 Feb '21

Yes: (Again, only testing locally) $ MPIR_CVAR_OFI_SKIP_IPV6=1 mpiexec -np 2 ./cpi Process 0 of 2 is on <HOSTNAME> Process 1 of 2 is on <HOSTNAME> pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.000268 while without: $ mpiexec -np 2 ./cpi Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_init.c at line 1988: mapped_table[i] != FI_ADDR_NOTAVAIL Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_init.c at line 1988: mapped_table[i] != FI_ADDR_NOTAVAIL 0 libpmpi.12.dylib 0x000000010d3e8b04 MPL_backtrace_show + 52 Thanks! I'll patch our installation to crowbar that to 1 for now, if that seems reasonable. Thanks, - Eric On Fri, Feb 19, 2021 at 7:56 AM Zhou, Hui <zhouh(a)anl.gov> wrote: > > Does setting environment variable `MPIR_CVAR_OFI_SKIP_IPV6=1` work? > > > > -- > Hui Zhou > > > > > > From: Eric Borisch via discuss <discuss(a)mpich.org> > Date: Thursday, February 18, 2021 at 7:36 PM > To: discuss(a)mpich.org <discuss(a)mpich.org> > Cc: Eric Borisch <eborisch(a)gmail.com> > Subject: Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur) > > Blaise, > > > > This is 3.4.1 and CH4? > > > > With —enable-fast=all, asserts are disabled; can you see if it will compile and execute mpi binaries when built without that option? I suspect it will catch the invalid condition (with the non-elided assert), and would likewise crash if it was used in actual distributed (non-localhost) execution. > > > > Let me know; if it doesn’t crash for you I would love to figure out why. > > > > - Eric > > > > On Thu, Feb 18, 2021 at 3:59 PM Blaise A Bourdin via discuss <discuss(a)mpich.org> wrote: > > Hi, > > Absolutely no issues here when using gcc10 from homebrew both on intel and ARM. > > Here is my configuration options: > > ./configure --prefix=$MPICH_HOME \ > > --enable-fast=all,O3 \ > > --enable-g=dbg \ > > --enable-romio \ > > --enable-shared \ > > --enable-sharedlibs=gcc-osx \ > > --with-pm=hydra \ > > CC=gcc-10 CXX=g++-10 FC=gfortran-10 F77=gfortran-10 \ > > CFLAGS="-fgnu89-inline -Wno-deprecated" \ > > CXXFLAGS="-Wno-deprecated" \ > > FFLAGS="-fallow-argument-mismatch" > > > > Blaise > > > > On Feb 18, 2021, at 3:53 PM, Allen Sanderson via discuss <discuss(a)mpich.org> wrote: > > > > Hello, > > > > First post to the discussion board. I made the mistake of upgrading to macOS 11.2 (aka Big Sur). I was able to build MPICH 3.4.1 successfully and integrate it with an application. However, when I ran, MPICH crashed. Here is the initial output: > > > > Abort(1) on node 0: Internal error > > 0 libpmpi.12.dylib 0x00000001068c2159 MPL_backtrace_show + 57 > > 1 libpmpi.12.dylib 0x000000010684e874 MPIR_Assert_fail + 36 > > 2 libpmpi.12.dylib 0x0000000106897bc8 MPIDI_OFI_mpi_init_hook + 4472 > > 3 libpmpi.12.dylib 0x00000001068790d3 MPID_Init + 3091 > > 4 libpmpi.12.dylib 0x00000001066ae70b MPIR_Init_thread + 219 > > 5 libmpi.12.dylib 0x00000001064bb897 MPI_Init + 279 > > > > I have not yet dug into the issue any further. Before doing so I wanted to first check to see if others have successfully built and run via macOS 11.2. If so I would appreciate any pointers. If it is known that it will not yet work, that would be good information as well so not to spend anytime digging. > > > > I should note that on another machine, I was wiser and upgraded to macOS 10.5.7 (aka Catalina) and built and ran with 3.3.1 that worked fine running in parallel. > > > > Cheers, > > > > Allen > > > > > > Allen Sanderson > > SCI Institute > > University of Utah > > > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > > -- > > A.K. & Shirley Barton Professor of Mathematics > > Adjunct Professor of Mechanical Engineering > > Adjunct of the Center for Computation & Technology > > Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA > > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur)
by Zhou, Hui 19 Feb '21

19 Feb '21

Does setting environment variable `MPIR_CVAR_OFI_SKIP_IPV6=1` work? -- Hui Zhou From: Eric Borisch via discuss <discuss(a)mpich.org> Date: Thursday, February 18, 2021 at 7:36 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Eric Borisch <eborisch(a)gmail.com> Subject: Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur) Blaise, This is 3.4.1 and CH4? With —enable-fast=all, asserts are disabled; can you see if it will compile and execute mpi binaries when built without that option? I suspect it will catch the invalid condition (with the non-elided assert), and would likewise crash if it was used in actual distributed (non-localhost) execution. Let me know; if it doesn’t crash for you I would love to figure out why. - Eric On Thu, Feb 18, 2021 at 3:59 PM Blaise A Bourdin via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi, Absolutely no issues here when using gcc10 from homebrew both on intel and ARM. Here is my configuration options: ./configure --prefix=$MPICH_HOME \ --enable-fast=all,O3 \ --enable-g=dbg \ --enable-romio \ --enable-shared \ --enable-sharedlibs=gcc-osx \ --with-pm=hydra \ CC=gcc-10 CXX=g++-10 FC=gfortran-10 F77=gfortran-10 \ CFLAGS="-fgnu89-inline -Wno-deprecated" \ CXXFLAGS="-Wno-deprecated" \ FFLAGS="-fallow-argument-mismatch" Blaise On Feb 18, 2021, at 3:53 PM, Allen Sanderson via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hello, First post to the discussion board. I made the mistake of upgrading to macOS 11.2 (aka Big Sur). I was able to build MPICH 3.4.1 successfully and integrate it with an application. However, when I ran, MPICH crashed. Here is the initial output: Abort(1) on node 0: Internal error 0 libpmpi.12.dylib 0x00000001068c2159 MPL_backtrace_show + 57 1 libpmpi.12.dylib 0x000000010684e874 MPIR_Assert_fail + 36 2 libpmpi.12.dylib 0x0000000106897bc8 MPIDI_OFI_mpi_init_hook + 4472 3 libpmpi.12.dylib 0x00000001068790d3 MPID_Init + 3091 4 libpmpi.12.dylib 0x00000001066ae70b MPIR_Init_thread + 219 5 libmpi.12.dylib 0x00000001064bb897 MPI_Init + 279 I have not yet dug into the issue any further. Before doing so I wanted to first check to see if others have successfully built and run via macOS 11.2. If so I would appreciate any pointers. If it is known that it will not yet work, that would be good information as well so not to spend anytime digging. I should note that on another machine, I was wiser and upgraded to macOS 10.5.7 (aka Catalina) and built and ran with 3.3.1 that worked fine running in parallel. Cheers, Allen Allen Sanderson SCI Institute University of Utah _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss -- A.K. & Shirley Barton Professor of Mathematics Adjunct Professor of Mechanical Engineering Adjunct of the Center for Computation & Technology Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur)
by Eric Borisch 18 Feb '21

18 Feb '21

Blaise, This is 3.4.1 and CH4? With —enable-fast=all, asserts are disabled; can you see if it will compile and execute mpi binaries when built without that option? I suspect it will catch the invalid condition (with the non-elided assert), and would likewise crash if it was used in actual distributed (non-localhost) execution. Let me know; if it doesn’t crash for you I would love to figure out why. - Eric On Thu, Feb 18, 2021 at 3:59 PM Blaise A Bourdin via discuss < discuss(a)mpich.org> wrote: > Hi, > Absolutely no issues here when using gcc10 from homebrew both on intel and > ARM. > Here is my configuration options: > ./configure --prefix=$MPICH_HOME \ > --enable-fast=all,O3 \ > --enable-g=dbg \ > --enable-romio \ > --enable-shared \ > --enable-sharedlibs=gcc-osx \ > --with-pm=hydra \ > CC=gcc-10 CXX=g++-10 FC=gfortran-10 F77=gfortran-10 \ > CFLAGS="-fgnu89-inline -Wno-deprecated" \ > CXXFLAGS="-Wno-deprecated" \ > FFLAGS="-fallow-argument-mismatch" > > Blaise > > On Feb 18, 2021, at 3:53 PM, Allen Sanderson via discuss < > discuss(a)mpich.org> wrote: > > Hello, > > First post to the discussion board. I made the mistake of upgrading to > macOS 11.2 (aka Big Sur). I was able to build MPICH 3.4.1 successfully and > integrate it with an application. However, when I ran, MPICH crashed. Here > is the initial output: > > Abort(1) on node 0: Internal error > 0 libpmpi.12.dylib 0x00000001068c2159 > MPL_backtrace_show + 57 > 1 libpmpi.12.dylib 0x000000010684e874 > MPIR_Assert_fail + 36 > 2 libpmpi.12.dylib 0x0000000106897bc8 > MPIDI_OFI_mpi_init_hook + 4472 > 3 libpmpi.12.dylib 0x00000001068790d3 MPID_Init + 3091 > 4 libpmpi.12.dylib 0x00000001066ae70b > MPIR_Init_thread + 219 > 5 libmpi.12.dylib 0x00000001064bb897 MPI_Init + 279 > > I have not yet dug into the issue any further. Before doing so I wanted to > first check to see if others have successfully built and run via macOS > 11.2. If so I would appreciate any pointers. If it is known that it will > not yet work, that would be good information as well so not to spend > anytime digging. > > I should note that on another machine, I was wiser and upgraded to macOS > 10.5.7 (aka Catalina) and built and ran with 3.3.1 that worked fine running > in parallel. > > Cheers, > > Allen > > > Allen Sanderson > SCI Institute > University of Utah > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > > > -- > A.K. & Shirley Barton Professor of Mathematics > Adjunct Professor of Mechanical Engineering > Adjunct of the Center for Computation & Technology > Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, > USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web > http://www.math.lsu.edu/~bourdin > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] MPICH 3.4.1 and macOS 11.2 (aka Big Sur)
by Allen Sanderson 18 Feb '21

18 Feb '21

Hello Thanks for that data point. I am using the native clang compilers. I dug into the issue as there was an assertion: Assertion failed in file src/mpid/ch4/netmod/ofi/ofi_init.c at line 1988: mapped_table[i] != FI_ADDR_NOTAVAIL And it is related to this issue: https://github.com/pmodels/mpich/issues/5041 I added the "--with-device=ch3” to my configure and happy to report that I am able to run in parallel. Cheers, Allen Allen Sanderson SCI Institute University of Utah > On Feb 18, 2021, at 2:59 PM, Blaise A Bourdin <bourdin(a)lsu.edu> wrote: > > Hi, > Absolutely no issues here when using gcc10 from homebrew both on intel and ARM. > Here is my configuration options: > ./configure --prefix=$MPICH_HOME \ > --enable-fast=all,O3 \ > --enable-g=dbg \ > --enable-romio \ > --enable-shared \ > --enable-sharedlibs=gcc-osx \ > --with-pm=hydra \ > CC=gcc-10 CXX=g++-10 FC=gfortran-10 F77=gfortran-10 \ > CFLAGS="-fgnu89-inline -Wno-deprecated" \ > CXXFLAGS="-Wno-deprecated" \ > FFLAGS="-fallow-argument-mismatch" > > Blaise > >> On Feb 18, 2021, at 3:53 PM, Allen Sanderson via discuss <discuss(a)mpich.org <mailto:[email protected]>> wrote: >> >> Hello, >> >> First post to the discussion board. I made the mistake of upgrading to macOS 11.2 (aka Big Sur). I was able to build MPICH 3.4.1 successfully and integrate it with an application. However, when I ran, MPICH crashed. Here is the initial output: >> >> Abort(1) on node 0: Internal error >> 0 libpmpi.12.dylib 0x00000001068c2159 MPL_backtrace_show + 57 >> 1 libpmpi.12.dylib 0x000000010684e874 MPIR_Assert_fail + 36 >> 2 libpmpi.12.dylib 0x0000000106897bc8 MPIDI_OFI_mpi_init_hook + 4472 >> 3 libpmpi.12.dylib 0x00000001068790d3 MPID_Init + 3091 >> 4 libpmpi.12.dylib 0x00000001066ae70b MPIR_Init_thread + 219 >> 5 libmpi.12.dylib 0x00000001064bb897 MPI_Init + 279 >> >> I have not yet dug into the issue any further. Before doing so I wanted to first check to see if others have successfully built and run via macOS 11.2. If so I would appreciate any pointers. If it is known that it will not yet work, that would be good information as well so not to spend anytime digging. >> >> I should note that on another machine, I was wiser and upgraded to macOS 10.5.7 (aka Catalina) and built and ran with 3.3.1 that worked fine running in parallel. >> >> Cheers, >> >> Allen >> >> >> Allen Sanderson >> SCI Institute >> University of Utah >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org <mailto:[email protected]> >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss <https://lists.mpich.org/mailman/listinfo/discuss> > > -- > A.K. & Shirley Barton Professor of Mathematics > Adjunct Professor of Mechanical Engineering > Adjunct of the Center for Computation & Technology > Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin <http://www.math.lsu.edu/~bourdin>

1 0