- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] UCX run-time warnings
by Raffenetti, Kenneth J. 27 Nov '19

27 Nov '19

Those warning messages are not coming from MPICH. Can you run ldd one of the benchmark binaries and confirm all the expected libraries are linked to it? Are you perhaps also linked against an hcoll library? Ken On 11/27/19 12:58 PM, Sajid Ali wrote: > Hi Kenneth, > > Running the same test with mpich@develop, I still see the following > errors in a typical outfile : > > |[sas4990@quser10 mdb6i63]$ cat lat_10 > ---------------------------------------- ... > ---------------------------------------- slurmstepd: error: mpi/pmi2: no > value for key in req slurmstepd: error: mpi/pmi2: no value for key in > req slurmstepd: error: mpi/pmi2: no value for key in req > [1574880837.368994] [qnode5038:4743 :0] ucp_context.c:1131 UCX WARN UCP > version is incompatible, required: 1.5, actual: 1.4 (release 0) > slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: > mpi/pmi2: no value for key in req [1574880837.377860] [qnode5039:4546 > :0] ucp_context.c:1131 UCX WARN UCP version is incompatible, required: > 1.5, actual: 1.4 (release 0) slurmstepd: error: mpi/pmi2: no value for > key in req slurmstepd: error: mpi/pmi2: no value for key in req > slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: > mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value > for key in req # OSU MPI_Get latency Test v5.6.1 # Window creation: > MPI_Win_allocate # Synchronization: MPI_Win_flush # Size Latency (us) 1 > 2.13 .... | > > > Since UCX is a vendored dependency of MPICH, why is MPICH complaining > about an incompatible UCX version ? > > > Thank You, > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io <http://s-sajid-ali.github.io> >

1 0

Re: [mpich-discuss] Question about mpich bandwidth vs message-size plot
by Congiu, Giuseppe 27 Nov '19

27 Nov '19

Sajid please use latest mpich master branch from www.github.com/pmodels/mpich<http://www.github.com/pmodels/mpich>. This contains the bug fix that was merged with the following PR: https://github.com/pmodels/mpich/pull/3947 Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Nov 27, 2019, at 11:15 AM, Sajid Ali via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi Jeff/Giuseppe, I spoke to the computational specialist from research IT and he confirmed that this particular build of MPICH is indeed broken. As Giuseppe pointed out the hydra launcher was running multi threaded jobs on each node because it was unable to read the nodelist. When we passed a nodelist to mpirun and repeated the same test, the results looked more reasonable (~80us latency instead of ~0.05us due to ch3:TCP). Thanks for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io<http://s-sajid-ali.github.io/> _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] UCX run-time warnings
by Sajid Ali 27 Nov '19

27 Nov '19

Hi Kenneth, Running the same test with mpich@develop, I still see the following errors in a typical outfile : [sas4990@quser10 mdb6i63]$ cat lat_10 ---------------------------------------- ... ---------------------------------------- slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req [1574880837.368994] [qnode5038:4743 :0] ucp_context.c:1131 UCX WARN UCP version is incompatible, required: 1.5, actual: 1.4 (release 0) slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req [1574880837.377860] [qnode5039:4546 :0] ucp_context.c:1131 UCX WARN UCP version is incompatible, required: 1.5, actual: 1.4 (release 0) slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req slurmstepd: error: mpi/pmi2: no value for key in req # OSU MPI_Get latency Test v5.6.1 # Window creation: MPI_Win_allocate # Synchronization: MPI_Win_flush # Size Latency (us) 1 2.13 .... Since UCX is a vendored dependency of MPICH, why is MPICH complaining about an incompatible UCX version ? Thank You, Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] Question about mpich bandwidth vs message-size plot
by Sajid Ali 27 Nov '19

27 Nov '19

Hi Jeff/Giuseppe, I spoke to the computational specialist from research IT and he confirmed that this particular build of MPICH is indeed broken. As Giuseppe pointed out the hydra launcher was running multi threaded jobs on each node because it was unable to read the nodelist. When we passed a nodelist to mpirun and repeated the same test, the results looked more reasonable (~80us latency instead of ~0.05us due to ch3:TCP). Thanks for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] UCX run-time warnings
by Raffenetti, Kenneth J. 27 Nov '19

27 Nov '19

Could you try an development build (mpich@develop) and see if the warnings persist? 3.3.2 only contains critical bug fixes. Our master git branch has diverged quite a bit, especially the ch4 device. Ken On 11/26/19 1:04 PM, Sajid Ali via discuss wrote: > Hi MPICH-developers, > > I’ve built MPICH-3.3.2 from source on my univ. cluster to ensure that > jobs can be launched with slurm’s srun (via —mpi=pmi2, slurm’s native > pmi version). I built mpich from source using spack using ch4:ucx. > > I see the following errors in my outfile (from osu-micro-benchmarks runs): > > |.... slurmstepd: error: mpi/pmi2: no value for key in req .... > [1574612953.443924] [qnode5038:30333:0] mpool.c:38 UCX WARN object > 0x1b88d00 was not returned to mpool ucp_rkeys ... ` | > > as shown in a typical outfile here. > > I don’t see such errors with using openmpi-4.0.2 which was also built > with UCX as the primary transport layer protocol where I compiled a > newer version of UCX from source. > > I’m attaching the build log from mpich install for reference. > > |Windows Terminal[sas4990@quser10 ~]$ spack find -ldv mpich%gcc(a)8.3.0 > ==> 1 installed package -- linux-rhel7-ivybridge / gcc(a)8.3.0 > ---------------------------- btlyk64 mpich(a)3.3.2 device=ch4 +hydra > netmod=ucx +pci pmi=pmi2 +romio+slurm~verbs+wrapperrpath eb74crw > libpciaccess(a)0.13.5 t6h6tgg libxml2(a)2.9.9~python 46i2h6v libiconv(a)1.16 > dnbfmo2 xz(a)5.2.4 xaq5v23 zlib(a)1.2.11+optimize+pic+shared 77kx7yy > slurm@19-05-3-2~gtk~hdf5~hwloc~mariadb~pmix+readline [sas4990@quser10 ~]$ | > > Thanks in advance for your help! > > -- > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io <http://s-sajid-ali.github.io> > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] Question about mpich bandwidth vs message-size plot
by Jeff Hammond 26 Nov '19

26 Nov '19

There is no way for internode latency to be less than ~1 us. Thus, I conclude you measure intranode results with MPICH and internode with everything else. Check your mpirun commands and try again. Jeff On Tue, Nov 26, 2019 at 10:57 AM Sajid Ali via discuss <discuss(a)mpich.org> wrote: > Hi MPICH-developers, > > I’ve been trying to benchmark mpi software currently available on my univ. > cluster with OSU micro benchmarks. I used this script to build and run the > benchmark : > https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/curr_mpi_bench/subm… > > And here is a plot of the results I obtained : > [image: curr_mpi.png] > > Could someone give me a clue as to why the behaviour of mpich bandwidth vs > latency is so odd ? Why does the bandwidth increase first and then fall off > at high message sizes? > > I’m also attaching the config.log for this particular mpich build which > seems to indicate that MPICH was built with ch3 and nemesis:tcp. Does this > mean that it didn’t pick up the infiniband libraries or that it is somehow > using IPoIB? Is there a way to get this information ? > > Thanks in advance for the help! > > > -- > Sajid Ali | PhD Candidate > Applied Physics > Northwestern University > s-sajid-ali.github.io > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Question about mpich bandwidth vs message-size plot
by Congiu, Giuseppe 26 Nov '19

26 Nov '19

Hi Sajid, You are correct, by default MPICH 3.3 uses ch3:nemesis:tcp. If you have Mellanox IB and you want to use that you have to specify `—with-device=ch4:ucx` at MPICH configuration time. As for why you see that drop in bandwidth for some large message sizes it’s difficult to pin-point the reason by just looking at the config.log. For large messages (you can probably look up the threshold from the CH3 code, I think it’s either 16 or 32 KBs) MPICH will switch from EAGER to RENDEZVOUS protocol, which requires some additional handshaking between sender and receiver before data can be actually transferred; This might impact performance. One thing that surprises me more however is that you have much better latency and bandwidth for small messages in MPICH compared with other MPI libraries. I would make sure you are running on separate nodes when using MPICH and not having your processes on the same node instead. There was a bug in MPICH hydra with slurm process manager for some time that would silently fail to detect separate nodes depending on whether they had some hostname that was not recognized properly. Hope this helps, Giuseppe Congiu Postdoctoral Appointee MCS Division Argonne National Laboratory 9700 South Cass Ave., Lemont, IL 60439 On Nov 26, 2019, at 12:55 PM, Sajid Ali via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi MPICH-developers, I’ve been trying to benchmark mpi software currently available on my univ. cluster with OSU micro benchmarks. I used this script to build and run the benchmark : https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/curr_mpi_bench/subm… And here is a plot of the results I obtained : <curr_mpi.png> Could someone give me a clue as to why the behaviour of mpich bandwidth vs latency is so odd ? Why does the bandwidth increase first and then fall off at high message sizes? I’m also attaching the config.log for this particular mpich build which seems to indicate that MPICH was built with ch3 and nemesis:tcp. Does this mean that it didn’t pick up the infiniband libraries or that it is somehow using IPoIB? Is there a way to get this information ? Thanks in advance for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io<http://s-sajid-ali.github.io/> <config.log>_______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

UCX run-time warnings
by Sajid Ali 26 Nov '19

26 Nov '19

Hi MPICH-developers, I’ve built MPICH-3.3.2 from source on my univ. cluster to ensure that jobs can be launched with slurm’s srun (via —mpi=pmi2, slurm’s native pmi version). I built mpich from source using spack using ch4:ucx. I see the following errors in my outfile (from osu-micro-benchmarks runs): .... slurmstepd: error: mpi/pmi2: no value for key in req .... [1574612953.443924] [qnode5038:30333:0] mpool.c:38 UCX WARN object 0x1b88d00 was not returned to mpool ucp_rkeys ... ` as shown in a typical outfile here. I don’t see such errors with using openmpi-4.0.2 which was also built with UCX as the primary transport layer protocol where I compiled a newer version of UCX from source. I’m attaching the build log from mpich install for reference. Windows Terminal[sas4990@quser10 ~]$ spack find -ldv mpich%gcc(a)8.3.0 ==> 1 installed package -- linux-rhel7-ivybridge / gcc(a)8.3.0 ---------------------------- btlyk64 mpich(a)3.3.2 device=ch4 +hydra netmod=ucx +pci pmi=pmi2 +romio+slurm~verbs+wrapperrpath eb74crw libpciaccess(a)0.13.5 t6h6tgg libxml2(a)2.9.9~python 46i2h6v libiconv(a)1.16 dnbfmo2 xz(a)5.2.4 xaq5v23 zlib(a)1.2.11+optimize+pic+shared 77kx7yy slurm@19-05-3-2~gtk~hdf5~hwloc~mariadb~pmix+readline [sas4990@quser10 ~]$ Thanks in advance for your help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Question about mpich bandwidth vs message-size plot
by Sajid Ali 26 Nov '19

26 Nov '19

Hi MPICH-developers, I’ve been trying to benchmark mpi software currently available on my univ. cluster with OSU micro benchmarks. I used this script to build and run the benchmark : https://github.com/s-sajid-ali/nu-quest-mpi/blob/master/curr_mpi_bench/subm… And here is a plot of the results I obtained : [image: curr_mpi.png] Could someone give me a clue as to why the behaviour of mpich bandwidth vs latency is so odd ? Why does the bandwidth increase first and then fall off at high message sizes? I’m also attaching the config.log for this particular mpich build which seems to indicate that MPICH was built with ch3 and nemesis:tcp. Does this mean that it didn’t pick up the infiniband libraries or that it is somehow using IPoIB? Is there a way to get this information ? Thanks in advance for the help! -- Sajid Ali | PhD Candidate Applied Physics Northwestern University s-sajid-ali.github.io

1 0

Re: [mpich-discuss] Fortran 2008 MPI Interface Bug
by Zhang, Junchao 22 Nov '19

22 Nov '19

Thomas, If you can build MPICH with gfortran (which is easy in my opinion), then you can easily do the experiment. Attaching a debugger, within a few steps, you run into gfortran code. Watching the address where the array's start index is stored, you can immediately find the bug. Otherwise, Hui has to separate out the few C/Fortran wrappers and make them MPI independent to make a test code. --Junchao Zhang On Thu, Nov 21, 2019 at 11:17 AM Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> wrote: Hi Thomas, Just to follow up. I will try to isolate a reproducing example at some point (unless someone beats me). — Hui Zhou On Nov 15, 2019, at 12:48 PM, Thomas König via discuss <discuss(a)mpich.org<mailto:[email protected]>> wrote: Hi Junchao, Could you check if it is a gcc bug in handling type(*), dimension(..) Unfortunately, I don't know the code that well to be able to do it. If you could put together a simple, self-contained test case showing i there is a problem, you could submit that to gcc's bugzilla (or I could do that for you). I could also check a simplified test case against current gfortran trunk, which will not compile mpich 3.3.1 because it now flags argument mismatches as errors. (And yes, argument mismatches are real errors, which cause wrong code; see for example https://bugs.debian.org/944127 .) _______________________________________________ discuss mailing list discuss(a)mpich.org<mailto:[email protected]> To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0