An update! I started reading all the MPICH wiki pages I could find and thought I should try -hosts or -f, and that *does* work:
(1024) $ mpirun -f machinefile -np 96 ./GEOSgcm.x srun.slurm: cluster configuration lacks support for cpu binding
In MAPL_Shmem: NumCores per Node varies from 12 to 28 NumNodes in use = 4 Total PEs = 96
So, I guess the answer is that MPICH 3.2 can't quite decode the SLURM environment to figure out a machinefile, so I need to make one myself. Would this be the best way to do this, or is there a way to build/configure MPICH to better support this? Next up: trying to figure out how to get Inifiniband supported as I think I'm using TCP:
(1108) $ mpichversion | grep Device MPICH Device: ch3:nemesis
On 09/05/2017 10:34 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote:
All,
I've been evaluating different MPI stacks on our cluster and found that MPICH 3.2 does really well on some simple little benchmarks. It also runs Hello World just fine, so I decided to apply it to our climate model (GEOS).
However, the first time I did that, things went a bit nuts. Essentially:
(1065) $ mpirun -np 96 ./GEOSgcm.x | & tee withoutppn.log srun.slurm: cluster configuration lacks support for cpu binding Fatal error in PMPI_Comm_create: Unknown error class, error stack: PMPI_Comm_create(564).................: MPI_Comm_create(MPI_COMM_WORLD, group=0x88000000, new_comm=0x106d1740) failed PMPI_Comm_create(541).................: MPIR_Comm_create_intra(215)...........: MPIR_Get_contextid_sparse_group(500)..: MPIR_Allreduce_impl(764)..............: MPIR_Allreduce_intra(257).............: allreduce_intra_or_coll_fn(163).......: MPIR_Allreduce_intra(417).............: MPIDU_Complete_posted_with_error(1137): Process failed MPIR_Allreduce_intra(417).............: MPIDU_Complete_posted_with_error(1137): Process failed MPIR_Allreduce_intra(268).............: MPIR_Bcast_impl(1452).................: MPIR_Bcast(1476)......................: MPIR_Bcast_intra(1287)................: MPIR_Bcast_binomial(310)..............: Failure during collective
(NOTE: The srun.slurm thing is just an error/warning we always get. Doesn't matter if it's MPT, Open MPI, MVAPICH2, Intel MPI...it happens.)
The thing is, it works just fine at (NX-by-NY) of 1x6 and 2x12, but once I go to 3x18, boom, collapse. As I am on 28-core nodes, my first thought was it was due to crossing nodes. But, those benchmarks I ran did just fine for 192 nodes, so...hmm.
Out of desperation, I finally thought, what if it was the fact that 28 doesn't divide 96 and passed in -ppn and:
(1068) $ mpirun -ppn 12 -np 96 ./GEOSgcm.x |& tee withppn.log srun.slurm: cluster configuration lacks support for cpu binding
In MAPL_Shmem: NumCores per Node = 12 NumNodes in use = 8 Total PEs = 96 ...
Starts up just fine! Note that every other MPI stack (MPT, Intel MPI, MVAPICH2, and Open MPI) handle the non-ppn type job just fine, but it's possible that they are evenly distributing the processes themselves. And the "MAPL_Shmem" lines you see are just reporting what the process structure looks like. I've added some print statements including this:
if (present(CommIn)) then CommCap = CommIn else CommCap = MPI_COMM_WORLD end if
if (.not.present(CommIn)) then call mpi_init(status) VERIFY_(STATUS) end if write (*,*) "MPI Initialized."
So, boring, and CommIn is *not* present, so we are using MPI_COMM_WORLD, and mpi_init is called as one would. Now if I run:
mpirun -np 96 ./GEOSgcm.x | grep 'MPI Init' | wc -l
to count the number initialized, multiple times, I get results like: 40, 56, 56, 45, 68. Never consistent.
So, I'm a bit at a loss. I freely admit I might have built MPICH3 incorrectly. It was essentially my first time. I configured with:
./configure --prefix=$SWDEV/MPI/mpich/3.2/intel_17.0.4.196 \ --disable-wrapper-rpath CC=icc CXX=icpc FC=ifort F77=ifort \ --enable-fortran=all --enable-cxx | & tee configure.intel_17.0.4.196.log
which might be too vanilla for a SLURM/Infiniband cluster, but, yet, it works with -ppn. But maybe I need extra options to work at all times? --with-ibverbs? --with-slurm?
Any ideas on what's happening and what I might have done wrong?
Thanks, Matt
-- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson _______________________________________________ discuss mailing list [email protected] To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss