On 09/06/2017 11:02 AM, Kenneth Raffenetti wrote:

On 09/05/2017 10:28 AM, Thompson, Matt (GSFC-610.1)[SCIENCE SYSTEMS AND APPLICATIONS INC] wrote:

...

An update!

I started reading all the MPICH wiki pages I could find and thought I should try -hosts or -f, and that *does* work:

...

(1024) $ mpirun -f machinefile -np 96 ./GEOSgcm.x srun.slurm: cluster configuration lacks support for cpu binding

 In MAPL_Shmem:      NumCores per Node varies from           12  to           28      NumNodes in use   =            4      Total PEs         =           96

So, I guess the answer is that MPICH 3.2 can't quite decode the SLURM environment to figure out a machinefile, so I need to make one myself.

Can you check the config.log in src/pm/hydra and see if slurm was detected? If not, you can specify --with-slurm=<path/to/install>. Hydra should be able to detect and understand the slurm host list from the job environment.

I tried compiling three different ways: If I just add --with-slurm, nothing happens. Configure seems not to react and I get the same mpirun behavior. If I add --with-slurm and --with-pmi=slurm, configure dies saying: configure: error: The PM chosen (hydra) requires the PMI implementation simple but slurm was selected as the PMI implementation. If I do --with-slurm --with-pmi=slurm --with-pm=none, then I don't get mpiexec.hydra built. So srun is all that can work.

...

Would this be the best way to do this, or is there a way to build/configure MPICH to better support this?

Next up: trying to figure out how to get Inifiniband supported as I think I'm using TCP:

If you are using Mellanox infiniband, try --with-device=ch3:nemesis:mxm --with-mxm=<path/to/install>. The MXM library is part of MOFED, or can be downloaded separately from the Mellanox website.

Now this I was able to do, and I could see from the configure that it found it and did something different. The problem? If I run a simple Hello World, the code locks up after printing "Hello world from..." It's like it never finalizes, or gets stuck somewhere. Matt -- Matt Thompson, SSAI, Sr Scientific Programmer/Analyst NASA GSFC, Global Modeling and Assimilation Office Code 610.1, 8800 Greenbelt Rd, Greenbelt, MD 20771 Phone: 301-614-6712 Fax: 301-614-6246 http://science.gsfc.nasa.gov/sed/bio/matthew.thompson