November 2019 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Question about mpich bandwidth vs message-size plot
by Congiu, Giuseppe 26 Nov '19

26 Nov '19

Hi Sajid, You are correct, by default MPICH 3.3 uses ch3:nemesis:tcp. If you have Mellanox IB and you want to use that you have to specify `—with-device=ch4:ucx` at MPICH configuration time. As for why you see that drop in bandwidth for some large message sizes it’s difficult to pin-point the reason by just looking at the config.log. For large messages (you can probably look up the threshold from the CH3 code, I think it’s either 16 or 32 KBs) MPICH will switch from EAGER to RENDEZVOUS protocol, which requires some additional handshaking between sender and receiver before data can be actually transferred; This might impact performance. One thing that surprises me more however is that you have much better latency and bandwidth for small messages in MPICH compared with other MPI libraries. I would make sure you are running on separate nodes when using MPICH and not having your processes on the same node instead. There was a bug in MPICH hydra with slurm process manager for some time that would silently fail to detect separate nodes depending on whether they had some hostname that was not recognized properly. Hope this helps, Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Nov 26, 2019, at 12:55 PM, Sajid Ali via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi MPICH-developers, I’ve been trying to benchmark mpi software currently available on my univ. cluster with OSU micro benchmarks. I used this script to build and run the benchmark : https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/curr_mpi_bench/subm… And here is a plot of the results I obtained : <curr_mpi.png> Could someone give me a clue as to why the behaviour of mpich bandwidth vs latency is so odd ? Why does the bandwidth increase first and then fall off at high message sizes? I’m also attaching the config.log for this particular mpich build which seems to indicate that MPICH was built with ch3 and nemesis:tcp. Does this mean that it didn’t pick up the infiniband libraries or that it is somehow using IPoIB? Is there a way to get this information ? Thanks in advance for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io<http://s-sajid-ali.github.io/> <config.log>_______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

UCX run-time warnings
by Sajid Ali 26 Nov '19

26 Nov '19

Hi MPICH-developers, I’ve built MPICH-3.3.2 from source on my univ. cluster to ensure that jobs can be launched with slurm’s srun (via —mpi=pmi2, slurm’s native pmi version). I built mpich from source using spack using ch4:ucx. I see the following errors in my outfile (from osu-micro-benchmarks runs): .... slurmstepd: error: mpi/pmi2: no value for key in req .... [1574612953.443924] [qnode5038:30333:0] mpool.c:38 UCX WARN object 0x1b88d00 was not returned to mpool ucp_rkeys ... ` as shown in a typical outfile here. I don’t see such errors with using openmpi-4.0.2 which was also built with UCX as the primary transport layer protocol where I compiled a newer version of UCX from source. I’m attaching the build log from mpich install for reference. Windows Terminal[sas4990@quser10 ~]$ spack find -ldv mpich%gcc(a)8.3.0 ==> 1 installed package -- linux-rhel7-ivybridge / gcc(a)8.3.0 ---------------------------- btlyk64 mpich(a)3.3.2 device=ch4 +hydra netmod=ucx +pci pmi=pmi2 +romio+slurm~verbs+wrapperrpath eb74crw libpciaccess(a)0.13.5 t6h6tgg libxml2(a)2.9.9~python 46i2h6v libiconv(a)1.16 dnbfmo2 xz(a)5.2.4 xaq5v23 zlib(a)1.2.11+optimize+pic+shared 77kx7yy slurm@19-05-3-2~gtk~hdf5~hwloc~mariadb~pmix+readline [sas4990@quser10 ~]$ Thanks in advance for your help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Question about mpich bandwidth vs message-size plot
by Sajid Ali 26 Nov '19

26 Nov '19

Hi MPICH-developers, I’ve been trying to benchmark mpi software currently available on my univ. cluster with OSU micro benchmarks. I used this script to build and run the benchmark : https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/curr_mpi_bench/subm… And here is a plot of the results I obtained : [image: curr_mpi.png] Could someone give me a clue as to why the behaviour of mpich bandwidth vs latency is so odd ? Why does the bandwidth increase first and then fall off at high message sizes? I’m also attaching the config.log for this particular mpich build which seems to indicate that MPICH was built with ch3 and nemesis:tcp. Does this mean that it didn’t pick up the infiniband libraries or that it is somehow using IPoIB? Is there a way to get this information ? Thanks in advance for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Zhang, Junchao 22 Nov '19

22 Nov '19

Thomas, If you can build MPICH with gfortran (which is easy in my opinion), then you can easily do the experiment. Attaching a debugger, within a few steps, you run into gfortran code. Watching the address where the array's start index is stored, you can immediately find the bug. Otherwise, Hui has to separate out the few C/Fortran wrappers and make them MPI independent to make a test code. --Junchao Zhang On Thu, Nov 21, 2019 at 11:17 AM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Thomas, Just to follow up. I will try to isolate a reproducing example at some point (unless someone beats me). — Hui Zhou On Nov 15, 2019, at 12:48 PM, Thomas König via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi Junchao, Could you check if it is a gcc bug in handling type(*), dimension(..) Unfortunately, I don't know the code that well to be able to do it. If you could put together a simple, self-contained test case showing i there is a problem, you could submit that to gcc's bugzilla (or I could do that for you). I could also check a simplified test case against current gfortran trunk, which will not compile mpich 3.3.1 because it now flags argument mismatches as errors. (And yes, argument mismatches are real errors, which cause wrong code; see for example https://bugs.debian.org/944127 .) _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Zhou, Hui 21 Nov '19

21 Nov '19

Hi Thomas, Just to follow up. I will try to isolate a reproducing example at some point (unless someone beats me). — Hui Zhou On Nov 15, 2019, at 12:48 PM, Thomas König via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi Junchao, Could you check if it is a gcc bug in handling type(*), dimension(..) Unfortunately, I don't know the code that well to be able to do it. If you could put together a simple, self-contained test case showing i there is a problem, you could submit that to gcc's bugzilla (or I could do that for you). I could also check a simplified test case against current gfortran trunk, which will not compile mpich 3.3.1 because it now flags argument mismatches as errors. (And yes, argument mismatches are real errors, which cause wrong code; see for example https://bugs.debian.org/944127 .) _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Zhou, Hui 21 Nov '19

21 Nov '19

Hi Pedro Ricardo, I have added some follow up for the reported issue at https://github.com/pmodels/mpich/issues/4170. gfortran 9.2 confirms your observation. Intel compiler, ifort, works correctly. I will try further narrow down the issue when I find some time; but it appears this is an issue in gfortran implementation. The `use_f08` binding can be implemented with a thin Fortran layer, essentially wrapping around the `F77` binding, or it can be implemented nearly entirely with Fortran iso-c-binding, wrapping around a native mpi c library. MPICH adopts the latter approach. If you are interested, you may refer to this paper: https://www.mcs.anl.gov/~balaji/pubs/2014/eurompi/eurompi14.f08binding.pdf — Hui Zhou On Nov 13, 2019, at 4:45 PM, Zhou, Hui via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi Pedro Ricardo, Thanks for reporting the issue. I have opened a GitHub issue, https://github.com/pmodels/mpich/issues/4170, for better tracking. `mpi_f08` tries to use the iso-c-binding as much as it can and it has not been receiving as much test as `use mpi`. One of the reason is the compiler support has been sporadic. May I ask which compiler (include version) were you using? Anyway, we’ll look into it and hopefully get it fixed shortly. — Hui Zhou Principal Software Development Specialist MCS | ANL On Nov 13, 2019, at 1:59 PM, Pedro Ricardo Souza via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi there, My name is Pedro Ricardo and I'm currently a researcher at a Fluid Dynamics Laboratory in Brazil. We have a CFD software with 12 continuous years of development mostly in Fortran, and we've been using Mpich and Openmpi since the very beginning. Recently we started to refactor the code and write it using Fortran2008 standard and this includes the usage of the the newest MPI module for Fortran "mpi_f08". So far no errors were present in Openmpi implementations and everything worked like a charm, but when we started compiling with Mpich lots of errors happened during execution. I was able so isolate and reproduce the error in a simpler code. Whenever "use mpi_f08" is present, the arrays passed to a MPI subroutine return with C-like indexes. If I allocate an standard array in Fortran (that starts with index 1), after the call of some MPI routine this array come back starting with index 0. And you can imagine that all access to this array will be wrong after that. It is a funny behavior because if you put "use mpi" instead of "use mpi_f08", the indexes do not change. The test is simple, allocate and array, print it's bound indexes, calls a MPI_Reduce routine and print the array bounds again. I've also tested other routines like MPI_ISend and the error also happens. program check_bounds !Check Works ! use mpi !Check Fails use mpi_f08 implicit none integer:: mpi_ierr, proc_id, n_proc real, allocatable, dimension(:):: t1, t1_0 call MPI_Init(mpi_ierr) call MPI_Comm_rank(MPI_COMM_WORLD, proc_id, mpi_ierr) call MPI_Comm_size(MPI_COMM_WORLD, n_proc, mpi_ierr) allocate(t1(5), t1_0(5)) t1 = 1.0; t1_0 = 1.0 if (proc_id==0) then write(*,*)'Array Configuration Before:' write(*,'(a6,2(1x,i2))')'Low:',lbound(t1),lbound(t1_0) write(*,'(a6,2(1x,i2))')'Upper:',ubound(t1),ubound(t1_0) end if CALL MPI_REDUCE(t1, t1_0, 5, MPI_REAL, MPI_SUM, 0, MPI_COMM_WORLD, mpi_ierr) if (proc_id==0) then write(*,*)'Array Configuration After:' write(*,'(a6,2(1x,i2))')'Low:',lbound(t1),lbound(t1_0) write(*,'(a6,2(1x,i2))')'Upper:',ubound(t1),ubound(t1_0) end if deallocate(t1, t1_0) call MPI_Finalize(mpi_ierr) end program check_bounds Thanks for the attention, _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Overlapping non-blocking collectives leads to deadlock
by Raffenetti, Kenneth J. 18 Nov '19

18 Nov '19

Correct. The ordering requirement is per-communicator, so creating additional communicators is one way to ensure safety. Ken On 11/18/19 11:52 AM, Joachim Protze via discuss wrote: > Hi Mark, > > I agree with Guiseppe. This is an issue in your code. You can use > different communicators (e.g., duplicates of comm-world) if you want to > concurrently execute collective communication by different threads. > > - Joachim > > On 11/18/19 5:25 PM, Mark Davis via discuss wrote: >> Hi Giuseppe, thank you for the fast response -- that clarifies things >> for me. >> >> On Mon, Nov 18, 2019 at 10:39 AM Congiu, Giuseppe via discuss >> <discuss(a)mpich.org> wrote: >>> >>> Hello Mark, >>> >>> I don’t think that is a bug in MPICH, it’s a bug in your code. The >>> MPI standard requires that collectives (non-blocking ones are no >>> exception) must be invoked in the same order in all processes. If T0 >>> in process 0 runs first and T1 in process 1 runs first you have a >>> mismatch and a resulting deadlock. >>> >>> —Giuseppe >>> >>>> On Nov 18, 2019, at 8:27 AM, Mark Davis via discuss >>>> <discuss(a)mpich.org> wrote: >>>> >>>> I realized something else relevant: I mentioned above that this >>>> deadlock occurs sometimes but not all of the time; I think I've >>>> narrowed down when it happens. Here's the above example with thread >>>> IDs annotated in: >>>> >>>> >>>> PROCESS 0 (root for ireduce and ibcast): >>>> // T0 is always the thread that calls MPI functions >>>> T0: MPI_Ireduce(..., &req) >>>> T0: MPI_Wait(&req); <-- blocking here >>>> ... >>>> T0: MPI_Ibcast(..., &req2); >>>> T0: MPI_Wait(&req2); >>>> >>>> PROCESS 1 (non-root for ireduce and ibcast): >>>> // T0 is the root for the reduce >>>> T0: MPI_Ireduce(..., &req) >>>> T0:MPI_Wait(&req); >>>> ... >>>> // T1 is the root for the bcast >>>> T1: MPI_Ibcast(..., &req2); >>>> T1: MPI_Wait(&req2); <-- blocking here >>>> >>>> Note that the non-root process has two different threads, T0 and T1, >>>> and T0 does the Ireduce and T1 does the bcast. I believe the T0 call >>>> to MPI_Ireduce is concurrent with the T1 call to MPI_Ibcast (both as >>>> non-roots). >>>> >>>> So, I believe the question is: is it legal in MPI to have two threads >>>> in a given MPI process call different non-blocking collectives (e.g., >>>> reduce and bcast) concurrently with MPI_THREAD_MULTIPLE enabled? >>>> >>>> Thank you >>>> >>>> On Mon, Nov 18, 2019 at 10:05 AM Mark Davis >>>> <markdavisinboston(a)gmail.com> wrote: >>>>> >>>>> Hello, I'm experimenting with non-blocking collectives using MPICH in >>>>> a multithreaded C++ program (with MPI_THREAD_MULTIPLE initialization). >>>>> >>>>> I'm currently doing a non-blocking reduce followed by a non-blocking >>>>> broadcast (I realize I can just use an allreduce but for my >>>>> experiment, I need to serialize these operations). I was able to >>>>> produce this bug with only two MPI processes. I see in gdb that the >>>>> root process is stuck trying to execute the MPI_Ireduce in cases where >>>>> the non-root process does the MPI_Ireduce and gets to the MPI_Ibcast >>>>> quickly. That is, process 0 (root) isn't able to complete the >>>>> MPI_Ireduce wait while process 1 is stuck in the MPI_Ibcast wait. >>>>> >>>>> PROCESS 0 (root for ireduce and ibcast): >>>>> MPI_Ireduce(..., &req) >>>>> MPI_Wait(&req); <-- blocking here >>>>> ... >>>>> MPI_Ibcast(..., &req2); >>>>> MPI_Wait(&req2); >>>>> >>>>> PROCESS 1 (non-root for ireduce and ibcast): >>>>> MPI_Ireduce(..., &req) >>>>> MPI_Wait(&req); >>>>> ... >>>>> MPI_Ibcast(..., &req2); >>>>> MPI_Wait(&req2); <-- blocking here >>>>> >>>>> Much of the time, the program deadlocks as shown above; sometimes this >>>>> works fine, though, perhaps due to subtle timing differences. I >>>>> mentioned above that this is a multithreaded program. I'm able to >>>>> produce the issue with two threads with two MPI procs. The other >>>>> threads are not calling MPI functions -- they are helping with other >>>>> computation. I've verified that I don't have any TSAN or ASAN errors >>>>> in this program. However, when I only have one thread per process, I >>>>> don't have this issue. I think there's a decent chance, though, that >>>>> this has to do with timing differences as opposed to changing anything >>>>> with the MPI calls. I have verified that only one thread per process >>>>> is calling the MPI routines in the multithreaded case. >>>>> >>>>> When I change the MPI_Ireduce to a blocking MPI_Reduce and I keep the >>>>> MPI_Ibcast non-blocking, the program runs fine. Only when BOTH >>>>> MPI_Ireduce and MPI_Ibcast happen serially do I see this deadlock >>>>> (again, some of the time). >>>>> >>>>> Unfortunately, this program is part of a very large system and it's >>>>> not straightforward to give a fully working example. So, I'm just >>>>> looking for any ideas anyone has for what sort of thing may be >>>>> happening, any information that may be helpful about how two >>>>> coincident non-blocking requests could interact with each other, etc. >>>>> >>>>> Also, if anyone has tips on how to debug this sort of thing in gdb >>>>> that would be helpful. For example, are there ways to introspect the >>>>> MPI_Request object, etc.? >>>>> >>>>> Thanks >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Overlapping non-blocking collectives leads to deadlock
by Joachim Protze 18 Nov '19

18 Nov '19

Hi Mark, I agree with Guiseppe. This is an issue in your code. You can use different communicators (e.g., duplicates of comm-world) if you want to concurrently execute collective communication by different threads. - Joachim On 11/18/19 5:25 PM, Mark Davis via discuss wrote: > Hi Giuseppe, thank you for the fast response -- that clarifies things for me. > > On Mon, Nov 18, 2019 at 10:39 AM Congiu, Giuseppe via discuss > <discuss(a)mpich.org> wrote: >> >> Hello Mark, >> >> I don’t think that is a bug in MPICH, it’s a bug in your code. The MPI standard requires that collectives (non-blocking ones are no exception) must be invoked in the same order in all processes. If T0 in process 0 runs first and T1 in process 1 runs first you have a mismatch and a resulting deadlock. >> >> —Giuseppe >> >>> On Nov 18, 2019, at 8:27 AM, Mark Davis via discuss <discuss(a)mpich.org> wrote: >>> >>> I realized something else relevant: I mentioned above that this >>> deadlock occurs sometimes but not all of the time; I think I've >>> narrowed down when it happens. Here's the above example with thread >>> IDs annotated in: >>> >>> >>> PROCESS 0 (root for ireduce and ibcast): >>> // T0 is always the thread that calls MPI functions >>> T0: MPI_Ireduce(..., &req) >>> T0: MPI_Wait(&req); <-- blocking here >>> ... >>> T0: MPI_Ibcast(..., &req2); >>> T0: MPI_Wait(&req2); >>> >>> PROCESS 1 (non-root for ireduce and ibcast): >>> // T0 is the root for the reduce >>> T0: MPI_Ireduce(..., &req) >>> T0:MPI_Wait(&req); >>> ... >>> // T1 is the root for the bcast >>> T1: MPI_Ibcast(..., &req2); >>> T1: MPI_Wait(&req2); <-- blocking here >>> >>> Note that the non-root process has two different threads, T0 and T1, >>> and T0 does the Ireduce and T1 does the bcast. I believe the T0 call >>> to MPI_Ireduce is concurrent with the T1 call to MPI_Ibcast (both as >>> non-roots). >>> >>> So, I believe the question is: is it legal in MPI to have two threads >>> in a given MPI process call different non-blocking collectives (e.g., >>> reduce and bcast) concurrently with MPI_THREAD_MULTIPLE enabled? >>> >>> Thank you >>> >>> On Mon, Nov 18, 2019 at 10:05 AM Mark Davis <markdavisinboston(a)gmail.com> wrote: >>>> >>>> Hello, I'm experimenting with non-blocking collectives using MPICH in >>>> a multithreaded C++ program (with MPI_THREAD_MULTIPLE initialization). >>>> >>>> I'm currently doing a non-blocking reduce followed by a non-blocking >>>> broadcast (I realize I can just use an allreduce but for my >>>> experiment, I need to serialize these operations). I was able to >>>> produce this bug with only two MPI processes. I see in gdb that the >>>> root process is stuck trying to execute the MPI_Ireduce in cases where >>>> the non-root process does the MPI_Ireduce and gets to the MPI_Ibcast >>>> quickly. That is, process 0 (root) isn't able to complete the >>>> MPI_Ireduce wait while process 1 is stuck in the MPI_Ibcast wait. >>>> >>>> PROCESS 0 (root for ireduce and ibcast): >>>> MPI_Ireduce(..., &req) >>>> MPI_Wait(&req); <-- blocking here >>>> ... >>>> MPI_Ibcast(..., &req2); >>>> MPI_Wait(&req2); >>>> >>>> PROCESS 1 (non-root for ireduce and ibcast): >>>> MPI_Ireduce(..., &req) >>>> MPI_Wait(&req); >>>> ... >>>> MPI_Ibcast(..., &req2); >>>> MPI_Wait(&req2); <-- blocking here >>>> >>>> Much of the time, the program deadlocks as shown above; sometimes this >>>> works fine, though, perhaps due to subtle timing differences. I >>>> mentioned above that this is a multithreaded program. I'm able to >>>> produce the issue with two threads with two MPI procs. The other >>>> threads are not calling MPI functions -- they are helping with other >>>> computation. I've verified that I don't have any TSAN or ASAN errors >>>> in this program. However, when I only have one thread per process, I >>>> don't have this issue. I think there's a decent chance, though, that >>>> this has to do with timing differences as opposed to changing anything >>>> with the MPI calls. I have verified that only one thread per process >>>> is calling the MPI routines in the multithreaded case. >>>> >>>> When I change the MPI_Ireduce to a blocking MPI_Reduce and I keep the >>>> MPI_Ibcast non-blocking, the program runs fine. Only when BOTH >>>> MPI_Ireduce and MPI_Ibcast happen serially do I see this deadlock >>>> (again, some of the time). >>>> >>>> Unfortunately, this program is part of a very large system and it's >>>> not straightforward to give a fully working example. So, I'm just >>>> looking for any ideas anyone has for what sort of thing may be >>>> happening, any information that may be helpful about how two >>>> coincident non-blocking requests could interact with each other, etc. >>>> >>>> Also, if anyone has tips on how to debug this sort of thing in gdb >>>> that would be helpful. For example, are there ways to introspect the >>>> MPI_Request object, etc.? >>>> >>>> Thanks >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Dipl.-Inf. Joachim Protze IT Center Group: High Performance Computing Division: Computational Science and Engineering RWTH Aachen University Seffenter Weg 23 D 52074 Aachen (Germany) Tel: +49 241 80- 24765 Fax: +49 241 80-624765 protze(a)itc.rwth-aachen.de www.itc.rwth-aachen.de

1 0

Re: [mpich-discuss] Overlapping non-blocking collectives leads to deadlock
by Mark Davis 18 Nov '19

18 Nov '19

Hi Giuseppe, thank you for the fast response -- that clarifies things for me. On Mon, Nov 18, 2019 at 10:39 AM Congiu, Giuseppe via discuss <discuss(a)mpich.org> wrote: > > Hello Mark, > > I don’t think that is a bug in MPICH, it’s a bug in your code. The MPI standard requires that collectives (non-blocking ones are no exception) must be invoked in the same order in all processes. If T0 in process 0 runs first and T1 in process 1 runs first you have a mismatch and a resulting deadlock. > > —Giuseppe > > > On Nov 18, 2019, at 8:27 AM, Mark Davis via discuss <discuss(a)mpich.org> wrote: > > > > I realized something else relevant: I mentioned above that this > > deadlock occurs sometimes but not all of the time; I think I've > > narrowed down when it happens. Here's the above example with thread > > IDs annotated in: > > > > > > PROCESS 0 (root for ireduce and ibcast): > > // T0 is always the thread that calls MPI functions > > T0: MPI_Ireduce(..., &req) > > T0: MPI_Wait(&req); <-- blocking here > > ... > > T0: MPI_Ibcast(..., &req2); > > T0: MPI_Wait(&req2); > > > > PROCESS 1 (non-root for ireduce and ibcast): > > // T0 is the root for the reduce > > T0: MPI_Ireduce(..., &req) > > T0:MPI_Wait(&req); > > ... > > // T1 is the root for the bcast > > T1: MPI_Ibcast(..., &req2); > > T1: MPI_Wait(&req2); <-- blocking here > > > > Note that the non-root process has two different threads, T0 and T1, > > and T0 does the Ireduce and T1 does the bcast. I believe the T0 call > > to MPI_Ireduce is concurrent with the T1 call to MPI_Ibcast (both as > > non-roots). > > > > So, I believe the question is: is it legal in MPI to have two threads > > in a given MPI process call different non-blocking collectives (e.g., > > reduce and bcast) concurrently with MPI_THREAD_MULTIPLE enabled? > > > > Thank you > > > > On Mon, Nov 18, 2019 at 10:05 AM Mark Davis <markdavisinboston(a)gmail.com> wrote: > >> > >> Hello, I'm experimenting with non-blocking collectives using MPICH in > >> a multithreaded C++ program (with MPI_THREAD_MULTIPLE initialization). > >> > >> I'm currently doing a non-blocking reduce followed by a non-blocking > >> broadcast (I realize I can just use an allreduce but for my > >> experiment, I need to serialize these operations). I was able to > >> produce this bug with only two MPI processes. I see in gdb that the > >> root process is stuck trying to execute the MPI_Ireduce in cases where > >> the non-root process does the MPI_Ireduce and gets to the MPI_Ibcast > >> quickly. That is, process 0 (root) isn't able to complete the > >> MPI_Ireduce wait while process 1 is stuck in the MPI_Ibcast wait. > >> > >> PROCESS 0 (root for ireduce and ibcast): > >> MPI_Ireduce(..., &req) > >> MPI_Wait(&req); <-- blocking here > >> ... > >> MPI_Ibcast(..., &req2); > >> MPI_Wait(&req2); > >> > >> PROCESS 1 (non-root for ireduce and ibcast): > >> MPI_Ireduce(..., &req) > >> MPI_Wait(&req); > >> ... > >> MPI_Ibcast(..., &req2); > >> MPI_Wait(&req2); <-- blocking here > >> > >> Much of the time, the program deadlocks as shown above; sometimes this > >> works fine, though, perhaps due to subtle timing differences. I > >> mentioned above that this is a multithreaded program. I'm able to > >> produce the issue with two threads with two MPI procs. The other > >> threads are not calling MPI functions -- they are helping with other > >> computation. I've verified that I don't have any TSAN or ASAN errors > >> in this program. However, when I only have one thread per process, I > >> don't have this issue. I think there's a decent chance, though, that > >> this has to do with timing differences as opposed to changing anything > >> with the MPI calls. I have verified that only one thread per process > >> is calling the MPI routines in the multithreaded case. > >> > >> When I change the MPI_Ireduce to a blocking MPI_Reduce and I keep the > >> MPI_Ibcast non-blocking, the program runs fine. Only when BOTH > >> MPI_Ireduce and MPI_Ibcast happen serially do I see this deadlock > >> (again, some of the time). > >> > >> Unfortunately, this program is part of a very large system and it's > >> not straightforward to give a fully working example. So, I'm just > >> looking for any ideas anyone has for what sort of thing may be > >> happening, any information that may be helpful about how two > >> coincident non-blocking requests could interact with each other, etc. > >> > >> Also, if anyone has tips on how to debug this sort of thing in gdb > >> that would be helpful. For example, are there ways to introspect the > >> MPI_Request object, etc.? > >> > >> Thanks > > _______________________________________________ > > discuss mailing list discuss(a)mpich.org > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/discuss > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Overlapping non-blocking collectives leads to deadlock
by Congiu, Giuseppe 18 Nov '19

18 Nov '19

Hello Mark, I don’t think that is a bug in MPICH, it’s a bug in your code. The MPI standard requires that collectives (non-blocking ones are no exception) must be invoked in the same order in all processes. If T0 in process 0 runs first and T1 in process 1 runs first you have a mismatch and a resulting deadlock. —Giuseppe > On Nov 18, 2019, at 8:27 AM, Mark Davis via discuss <discuss(a)mpich.org> wrote: > > I realized something else relevant: I mentioned above that this > deadlock occurs sometimes but not all of the time; I think I've > narrowed down when it happens. Here's the above example with thread > IDs annotated in: > > > PROCESS 0 (root for ireduce and ibcast): > // T0 is always the thread that calls MPI functions > T0: MPI_Ireduce(..., &req) > T0: MPI_Wait(&req); <-- blocking here > ... > T0: MPI_Ibcast(..., &req2); > T0: MPI_Wait(&req2); > > PROCESS 1 (non-root for ireduce and ibcast): > // T0 is the root for the reduce > T0: MPI_Ireduce(..., &req) > T0:MPI_Wait(&req); > ... > // T1 is the root for the bcast > T1: MPI_Ibcast(..., &req2); > T1: MPI_Wait(&req2); <-- blocking here > > Note that the non-root process has two different threads, T0 and T1, > and T0 does the Ireduce and T1 does the bcast. I believe the T0 call > to MPI_Ireduce is concurrent with the T1 call to MPI_Ibcast (both as > non-roots). > > So, I believe the question is: is it legal in MPI to have two threads > in a given MPI process call different non-blocking collectives (e.g., > reduce and bcast) concurrently with MPI_THREAD_MULTIPLE enabled? > > Thank you > > On Mon, Nov 18, 2019 at 10:05 AM Mark Davis <markdavisinboston(a)gmail.com> wrote: >> >> Hello, I'm experimenting with non-blocking collectives using MPICH in >> a multithreaded C++ program (with MPI_THREAD_MULTIPLE initialization). >> >> I'm currently doing a non-blocking reduce followed by a non-blocking >> broadcast (I realize I can just use an allreduce but for my >> experiment, I need to serialize these operations). I was able to >> produce this bug with only two MPI processes. I see in gdb that the >> root process is stuck trying to execute the MPI_Ireduce in cases where >> the non-root process does the MPI_Ireduce and gets to the MPI_Ibcast >> quickly. That is, process 0 (root) isn't able to complete the >> MPI_Ireduce wait while process 1 is stuck in the MPI_Ibcast wait. >> >> PROCESS 0 (root for ireduce and ibcast): >> MPI_Ireduce(..., &req) >> MPI_Wait(&req); <-- blocking here >> ... >> MPI_Ibcast(..., &req2); >> MPI_Wait(&req2); >> >> PROCESS 1 (non-root for ireduce and ibcast): >> MPI_Ireduce(..., &req) >> MPI_Wait(&req); >> ... >> MPI_Ibcast(..., &req2); >> MPI_Wait(&req2); <-- blocking here >> >> Much of the time, the program deadlocks as shown above; sometimes this >> works fine, though, perhaps due to subtle timing differences. I >> mentioned above that this is a multithreaded program. I'm able to >> produce the issue with two threads with two MPI procs. The other >> threads are not calling MPI functions -- they are helping with other >> computation. I've verified that I don't have any TSAN or ASAN errors >> in this program. However, when I only have one thread per process, I >> don't have this issue. I think there's a decent chance, though, that >> this has to do with timing differences as opposed to changing anything >> with the MPI calls. I have verified that only one thread per process >> is calling the MPI routines in the multithreaded case. >> >> When I change the MPI_Ireduce to a blocking MPI_Reduce and I keep the >> MPI_Ibcast non-blocking, the program runs fine. Only when BOTH >> MPI_Ireduce and MPI_Ibcast happen serially do I see this deadlock >> (again, some of the time). >> >> Unfortunately, this program is part of a very large system and it's >> not straightforward to give a fully working example. So, I'm just >> looking for any ideas anyone has for what sort of thing may be >> happening, any information that may be helpful about how two >> coincident non-blocking requests could interact with each other, etc. >> >> Also, if anyone has tips on how to debug this sort of thing in gdb >> that would be helpful. For example, are there ways to introspect the >> MPI_Request object, etc.? >> >> Thanks > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss

1 0