August 2018 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] using mpi4py in a Singluarity container run at a large computing center with Slurm installed
by Martin Cuma 31 Aug '18

31 Aug '18

Hi Heather, this is a nicely complex problem that I can't say I know a solution of, but, let me say what I know and perhaps it'll shed some light on the problem. To answer your question on how mpirun interacts with srun (or SLURM in general), most MPIs (or better to say, PMIs that MPI uses for process launch) these days have SLURM support so when built they can leverage SLURM. Or the SLURM is set up to facilitate the remote node connection (e.g. by hijacking ssh through its own PMI - I don't know this just guessing). So, for the MPI distros that I tried (MPICH and derivatives - Intel MPI, MVAPICH2; and OpenMPI), mpirun at some point calls srun, no matter if it was built with SLURM support explicitly or not. Which would explain the srun error you are getting. Now, what I think is happening in your case is that you are calling the mpirun (or its equivalent inside mpi4py) from INSIDE of the container, where there's no srun. Notice that most MPI container examples, including the very well written ANL page, instruct you to use mpirun (or aprun in Cray's case) OUTSIDE of the container (the host), and launch N instances of the container through the mpirun. I reproduced your problem on our system in the following way: 1. Build a Singularity container with local MPI installation, e.g. https://github.com/CHPC-UofU/Singularity-ubuntu-mpi 2. shell into the container and build some mpi program (e.g. I have the cpi.c example from mpich - mpicc cpi.c -o cpi). 3. This then runs OK in the container on an interactive node (= outside SLURM job - mpirun does not use SLURM's PMI = does not use srun). 4. Launch the job, then shell into the container, and try to run mpirun -np 2 ./cpi - I get the same error you get, since $ which srun $ Now, I can try to set the path to the SLURM binaries $ export PATH="/uufs/notchpeak.peaks/sys/installdir/slurm/std/bin:$PATH" $ which srun /uufs/notchpeak.peaks/sys/installdir/slurm/std/bin/srun but then get another error: $ mpirun -np 2 ./cpi.c srun: error: Invalid user for SlurmUser slurm, ignored srun: fatal: Unable to process configuration file so the environment needs some more changes to get the srun to work correctly from inside the container. Though I think this would still only be hackable for an intra-node MPI launch, as inter-node you'll rely on SLURM that would have to get accessed from outside of the container. So, bottom line, launching mpirun from the host is preferable. I am not sure how you can launch the mpi4py from the host, since I don't use mpi4py, but, in theory it should not be any different than launching MPI binaries. Though I figure modifying your launch scripts around this may be complicated. BTW, I have had a reasonable success with mixing ABI compatible MPIs (MPICH, MVAPICH2, Intel MPI) in and out of the container. It often works but sometimes it does not. HTH, MC -- Martin Cuma Center for High Performance Computing Department of Geology and Geophysics University of Utah _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] using mpi4py in a Singluarity container run at a large computing center with Slurm installed
by Heather Kelly 31 Aug '18

31 Aug '18

Hi Martin, Thanks for the response. Unfortunately, I don't have direct access to the Dockerfile used to create this image, but I have at times built my own when necessary. Here is an example: https://github.com/LSSTDESC/dockerfiles/blob/master/lsst_sims/Dockerfile-bu… There's a bit going on there with the newinstall.sh script that includes installing miniconda plus a bunch of python packages including mpich 3.2.1, then downloading and building some C/C++ code. In my search for clues, I see you are absolutely right, there is a hope to reuse the same (or similar) version of mpi in the container as is available on the native system: https://www.alcf.anl.gov/user-guides/singularity. That is not what is happening here, and I'm not certain how easily I can change that. I might be able to modify this one package that includes the python code I want to run. I've tried to simplify what I'm up to, so I connect to one of the compute nodes and there I am able to start up shifter or singularity (depending on what center I'm working on - the error is the same in both cases). Here I'm using shifter on a compute node, and I completely skip using srun, but the result is the same: shifter --image=<imageName> ./run_shifter_smp.sh $PWD/drp-test $PWD/filesToIngest.txt |& tee smp.log root INFO: Loading config overrride file '/global/cscratch1/sd/desc/DC2/Run1.2iTest-20180830/obs_lsstCam/config/ingest.py' LsstCamMapper WARN: Unable to find calib root directory CameraMapper INFO: Loading Posix exposure registry from /global/cscratch1/sd/desc/DC2/Run1.2i-20180830/drp-test [mpiexec@nid00011] HYDU_create_process (utils/launch/launch.c:75): execvp error on file srun (No such file or directory) that run_shifter_smp.sh script just sets up my env and calls the routine where I can ask that it run in SMP mode: ingestDriver.py $1 @$2 --cores 20 --mode link --batch-type smp I can double check where mpiexec is pointing and see: (lsst-scipipe-10a4fa6) sh-4.2$ type mpiexec mpiexec is /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpich/3.2.1/bin/mpiexec which is in our software, rather than the system's mpiexec I also took a look at the slew of environment variables containing "MPI", it's at the end of this email in case it's of any use. Here's a look at the modules loaded in this env: module list Currently Loaded Modulefiles: 1) modules/3.2.10.6 13) dvs/2.7_2.2.65-6.0.5.2_16.2__gbec2cb0 2) intel/18.0.1.163 14) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari 3) craype-network-aries 15) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari 4) craype/2.5.14 16) atp/2.1.1 5) cray-libsci/18.03.1 17) PrgEnv-intel/6.0.4 6) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari 18) craype-haswell 7) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari 19) cray-mpich-abi/7.7.0 8) pmi/5.0.13 20) nano/2.2.6 9) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari 21) altd/2.0 10) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari 22) darshan/3.1.4 11) xpmem/2.2.4-6.0.5.1_8.18__g35d5e73.ari 23) Base-opts/2.4.123-6.0.5.0_11.2__g6460790.ari 12) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari Reading the FAQ, I was curious about this statement when MPICH is set up with Slurm to use srun: "Once configured with slurm, no internal process manager is built for MPICH; the user is expected to use SLURM's launch models (such as srun)." https://wiki.mpich.org/mpich/index.php/FAQ#Q:_How_do_I_use_MPICH_with_slurm… Does that mean the host system's mpi likely only knows about SLURM's launch models and wouldn't even be able to use SMP? Take care, Heather (lsst-scipipe-10a4fa6) sh-4.2$ set|grep MPI CRAY_MPICH2_DIR=/opt/cray/pe/mpt/7.7.0/gni/mpich-intel/16.0 CRAY_MPICH2_VER=7.7.0 CRAY_MPICH_BASEDIR=/opt/cray/pe/mpt/7.7.0/gni CRAY_MPICH_DIR=/opt/cray/pe/mpt/7.7.0/gni/mpich-intel/16.0 CRAY_MPICH_ROOTDIR=/opt/cray/pe/mpt/7.7.0 CRAY_PRE_COMPILE_OPTS=-hnetwork=aries MPI4PY_DIR=/opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpi4py/2.0.0+6 MPICH_ABORT_ON_ERROR=1 MPICH_DIR=/opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpich/3.2.1 MPICH_MPIIO_DVS_MAXNODES=32 MPI_DIR=/opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6/Linux64/mpi/0.0.1+3 PE_FFTW2_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH PE_FFTW_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH PE_GA_DEFAULT_GENCOMPILERS_GNU='5.1 4.9' PE_HDF5_DEFAULT_GENCOMPILERS_GNU='5.1 4.9' PE_HDF5_PARALLEL_DEFAULT_GENCOMPILERS_GNU='5.1 4.9' PE_HDF5_PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH PE_LIBSCI_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6 PE_LIBSCI_DEFAULT_GENCOMPILERS_GNU_x86_64='7.1 6.1 5.1 4.9' PE_LIBSCI_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0 PE_LIBSCI_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH PE_LIBSCI_GENCOMPILERS_CRAY_x86_64=8.6 PE_LIBSCI_GENCOMPILERS_GNU_x86_64='7.1 6.1 5.1 4.9' PE_LIBSCI_GENCOMPILERS_INTEL_x86_64=16.0 PE_LIBSCI_REQUIRED_PRODUCTS=PE_MPICH PE_MPICH_ALTERNATE_LIBS_dpm=_dpm PE_MPICH_ALTERNATE_LIBS_multithreaded=_mt PE_MPICH_CXX_PKGCONFIG_LIBS=mpichcxx PE_MPICH_DEFAULT_DIR_CRAY_DEFAULT64=64 PE_MPICH_DEFAULT_FIXED_PRGENV=INTEL PE_MPICH_DEFAULT_GENCOMPILERS_CRAY=8.6 PE_MPICH_DEFAULT_GENCOMPILERS_GNU='5.1 4.9' PE_MPICH_DEFAULT_GENCOMPS_CRAY=86 PE_MPICH_DEFAULT_GENCOMPS_GNU='51 49' PE_MPICH_DEFAULT_VOLATILE_PKGCONFIG_PATH=/opt/cray/pe/mpt/7.7.0/gni/mpich-@PRGENV @@PE_MPICH_DEFAULT_DIR_DEFAULT64@/@PE_MPICH_DEFAULT_GENCOMPS@ /lib/pkgconfig PE_MPICH_DEFAULT_VOLATILE_PRGENV='CRAY GNU' PE_MPICH_DIR_CRAY_DEFAULT64=64 PE_MPICH_FIXED_PRGENV=INTEL PE_MPICH_FORTRAN_PKGCONFIG_LIBS=mpichf90 PE_MPICH_GENCOMPILERS_CRAY=8.6 PE_MPICH_GENCOMPILERS_GNU='5.1 4.9' PE_MPICH_GENCOMPS_CRAY=86 PE_MPICH_GENCOMPS_GNU='51 49' PE_MPICH_MODULE_NAME=cray-mpich PE_MPICH_NV_LIBS= PE_MPICH_NV_LIBS_nvidia20=-lcudart PE_MPICH_NV_LIBS_nvidia35=-lcudart PE_MPICH_NV_LIBS_nvidia60=-lcudart PE_MPICH_PKGCONFIG_LIBS=mpich PE_MPICH_PKGCONFIG_VARIABLES=PE_MPICH_NV_LIBS_@accelerator @:PE_MPICH_ALTERNATE_LIBS_@multithreaded@:PE_MPICH_ALTERNATE_LIBS_@dpm@ PE_MPICH_TARGET_VAR_nvidia20=-lcudart PE_MPICH_TARGET_VAR_nvidia35=-lcudart PE_MPICH_VOLATILE_PKGCONFIG_PATH=/opt/cray/pe/mpt/7.7.0/gni/mpich-@PRGENV @@PE_MPICH_DIR_DEFAULT64@/@PE_MPICH_GENCOMPS@/lib/pkgconfig PE_MPICH_VOLATILE_PRGENV='CRAY GNU' PE_NETCDF_DEFAULT_GENCOMPILERS_GNU='5.1 4.9' PE_NETCDF_HDF5PARALLEL_DEFAULT_GENCOMPILERS_GNU='5.1 4.9' PE_NETCDF_HDF5PARALLEL_DEFAULT_REQUIRED_PRODUCTS=PE_HDF5_PARALLEL:PE_MPICH PE_PARALLEL_NETCDF_DEFAULT_GENCOMPILERS_GNU='5.1 4.9' PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_haswell=8.6 PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_interlagos=8.6 PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_mic_knl=8.6 PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_sandybridge=8.6 PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_skylake=8.6 PE_PETSC_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6 PE_PETSC_DEFAULT_GENCOMPILERS_GNU_haswell='5.3 4.9' PE_PETSC_DEFAULT_GENCOMPILERS_GNU_interlagos='5.3 4.9' PE_PETSC_DEFAULT_GENCOMPILERS_GNU_mic_knl=5.3 PE_PETSC_DEFAULT_GENCOMPILERS_GNU_sandybridge='5.3 4.9' PE_PETSC_DEFAULT_GENCOMPILERS_GNU_skylake=6.1 PE_PETSC_DEFAULT_GENCOMPILERS_GNU_x86_64='5.3 4.9' PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_haswell=16.0 PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_interlagos=16.0 PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_mic_knl=16.0 PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_sandybridge=16.0 PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_skylake=16.0 PE_PETSC_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0 PE_PETSC_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI:PE_HDF5_PARALLEL:PE_TPSL PE_PKGCONFIG_DEFAULT_PRODUCTS=PE_TRILINOS:PE_TPSL_64:PE_TPSL:PE_PETSC:PE_PARALLEL_NETCDF:PE_NETCDF_HDF5PARALLEL:PE_NETCDF:PE_MPICH:PE_LIBSCI:PE_HDF5_PARALLEL:PE_HDF5:PE_GA:PE_FFTW2:PE_FFTW PE_PKGCONFIG_PRODUCTS=PE_MPICH:PE_LIBSCI PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_haswell=8.6 PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_interlagos=8.6 PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_mic_knl=8.6 PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_sandybridge=8.6 PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6 PE_TPSL_64_DEFAULT_GENCOMPILERS_CRAY_x86_skylake=8.6 PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_haswell='5.1 4.9' PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_interlagos='5.1 4.9' PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_mic_knl=5.1 PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_sandybridge='5.1 4.9' PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_x86_64='5.1 4.9' PE_TPSL_64_DEFAULT_GENCOMPILERS_GNU_x86_skylake=6.1 PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_haswell=16.0 PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_interlagos=16.0 PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_mic_knl=16.0 PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_sandybridge=16.0 PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0 PE_TPSL_64_DEFAULT_GENCOMPILERS_INTEL_x86_skylake=16.0 PE_TPSL_64_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_haswell=8.6 PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_interlagos=8.6 PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_mic_knl=8.6 PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_sandybridge=8.6 PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6 PE_TPSL_DEFAULT_GENCOMPILERS_CRAY_x86_skylake=8.6 PE_TPSL_DEFAULT_GENCOMPILERS_GNU_haswell='5.1 4.9' PE_TPSL_DEFAULT_GENCOMPILERS_GNU_interlagos='5.1 4.9' PE_TPSL_DEFAULT_GENCOMPILERS_GNU_mic_knl=5.1 PE_TPSL_DEFAULT_GENCOMPILERS_GNU_sandybridge='5.1 4.9' PE_TPSL_DEFAULT_GENCOMPILERS_GNU_x86_64='5.1 4.9' PE_TPSL_DEFAULT_GENCOMPILERS_GNU_x86_skylake=6.1 PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_haswell=16.0 PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_interlagos=16.0 PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_mic_knl=16.0 PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_sandybridge=16.0 PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0 PE_TPSL_DEFAULT_GENCOMPILERS_INTEL_x86_skylake=16.0 PE_TPSL_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_LIBSCI PE_TRILINOS_DEFAULT_GENCOMPILERS_CRAY_x86_64=8.6 PE_TRILINOS_DEFAULT_GENCOMPILERS_GNU_x86_64='5.1 4.9' PE_TRILINOS_DEFAULT_GENCOMPILERS_INTEL_x86_64=16.0 PE_TRILINOS_DEFAULT_REQUIRED_PRODUCTS=PE_MPICH:PE_HDF5_PARALLEL:PE_NETCDF_HDF5PARALLEL:PE_LIBSCI:PE_TPSL SETUP_MPI='mpi 0.0.1+3 -f Linux64 -Z /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6' SETUP_MPI4PY='mpi4py 2.0.0+6 -f Linux64 -Z /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6' SETUP_MPICH='mpich 3.2.1 -f Linux64 -Z /opt/lsst/software/stack/stack/miniconda3-4.5.4-10a4fa6' On Fri, Aug 31, 2018 at 3:47 PM Kandes, Martin <mkandes(a)sdsc.edu> wrote: > Hi Heather, > > > Can you copy and paste your Slurm batch job script here to give us an > overview of what the job looks like? It'd also be helpful if you could > provided the definition (or recipe) file for this container and a list of > the software modules available at the site you are running the container. > > > Marty > > > P.S. In general, you want to have the same MPI implementation and version > installed within the Singularity container as-is the one available on the > host systems where the container will run. > ------------------------------ > *From:* Heather Kelly <heather999kelly(a)gmail.com> > *Sent:* Friday, August 31, 2018 12:34:25 PM > *To:* discuss(a)mpich.org > *Subject:* [mpich-discuss] using mpi4py in a Singluarity container run at > a large computing center with Slurm installed > > Hi, > Complete newbie here. > I have a Singularity container (created by someone else), that includes a > python script that uses mpi4py which is installed in the image, as is mpi. > I'm trying to run this at a large computing center where SLURM is > installed. The code in the container, wants to utilize its own > installation of mpi4py and mpi. The code provides flags that allow a user > to specify if they want to allow the code to use Slurm, SMP, or nothing at > all. The default is SMP. > > When I attempt to run this code in the image, on a compute node of this > computing center, I receive an error: > HYDU_create_process (utils/launch/launch.c:75): execvp error on file srun > (No such file or directory > even though I have specified to the program that I want to use SMP, yet, > it appears mpiexec is trying to submit a job to Slurm. Checking the > environment, it appears mpiexec is pointing to the version installed in the > container, and not the one available at the computing center. > > Is there an env variable or some way to set the process management system > to avoid using Slurm? > > In other contexts this code works just fine, my problem seems specific to > running this in a container at these large computing centers where Slurm is > available. It's as if the local computing center's mpi install is taking > precedence..and perhaps that's just how it works, but I'd like to find a > way around that. > > Take care, > Heather > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] using mpi4py in a Singluarity container run at a large computing center with Slurm installed
by Kandes, Martin 31 Aug '18

31 Aug '18

Hi Heather, Can you copy and paste your Slurm batch job script here to give us an overview of what the job looks like? It'd also be helpful if you could provided the definition (or recipe) file for this container and a list of the software modules available at the site you are running the container. Marty P.S. In general, you want to have the same MPI implementation and version installed within the Singularity container as-is the one available on the host systems where the container will run. ________________________________ From: Heather Kelly <heather999kelly(a)gmail.com> Sent: Friday, August 31, 2018 12:34:25 PM To: discuss(a)mpich.org Subject: [mpich-discuss] using mpi4py in a Singluarity container run at a large computing center with Slurm installed Hi, Complete newbie here. I have a Singularity container (created by someone else), that includes a python script that uses mpi4py which is installed in the image, as is mpi. I'm trying to run this at a large computing center where SLURM is installed. The code in the container, wants to utilize its own installation of mpi4py and mpi. The code provides flags that allow a user to specify if they want to allow the code to use Slurm, SMP, or nothing at all. The default is SMP. When I attempt to run this code in the image, on a compute node of this computing center, I receive an error: HYDU_create_process (utils/launch/launch.c:75): execvp error on file srun (No such file or directory even though I have specified to the program that I want to use SMP, yet, it appears mpiexec is trying to submit a job to Slurm. Checking the environment, it appears mpiexec is pointing to the version installed in the container, and not the one available at the computing center. Is there an env variable or some way to set the process management system to avoid using Slurm? In other contexts this code works just fine, my problem seems specific to running this in a container at these large computing centers where Slurm is available. It's as if the local computing center's mpi install is taking precedence..and perhaps that's just how it works, but I'd like to find a way around that. Take care, Heather _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

using mpi4py in a Singluarity container run at a large computing center with Slurm installed
by Heather Kelly 31 Aug '18

31 Aug '18

Hi, Complete newbie here. I have a Singularity container (created by someone else), that includes a python script that uses mpi4py which is installed in the image, as is mpi. I'm trying to run this at a large computing center where SLURM is installed. The code in the container, wants to utilize its own installation of mpi4py and mpi. The code provides flags that allow a user to specify if they want to allow the code to use Slurm, SMP, or nothing at all. The default is SMP. When I attempt to run this code in the image, on a compute node of this computing center, I receive an error: HYDU_create_process (utils/launch/launch.c:75): execvp error on file srun (No such file or directory even though I have specified to the program that I want to use SMP, yet, it appears mpiexec is trying to submit a job to Slurm. Checking the environment, it appears mpiexec is pointing to the version installed in the container, and not the one available at the computing center. Is there an env variable or some way to set the process management system to avoid using Slurm? In other contexts this code works just fine, my problem seems specific to running this in a container at these large computing centers where Slurm is available. It's as if the local computing center's mpi install is taking precedence..and perhaps that's just how it works, but I'd like to find a way around that. Take care, Heather _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] problem in execution
by Hodgess, Erin 28 Aug '18

28 Aug '18

Hi Haider Mpiexec doesn’t like the -f option. Just for fun, did you try it without the hosts file, please? Erin M. Hodgess Associate Professor Department of Mathematics and Statistics University of Houston - Downtown mailto: hodgesse(a)uhd.edu ________________________________ From: Haider Abbas <haiderabbasphy(a)gmail.com> Sent: Tuesday, August 28, 2018 10:09:27 PM To: discuss(a)mpich.org Subject: [mpich-discuss] problem in execution Dear all, I have prepared a 10 node ubuntu based cluster, following the guidelines described in "https://www-users.cs.york.ac.uk/~mjf/pi_cluster/src/Building_a_simple_Beowu…<https://www-users.cs.york.ac.uk/%7Emjf/pi_cluster/src/Building_a_simple_Beo…>" I have followed each and everthing as mentioned in the above documents. everything is Ok, now I try try to check the cluster by executing the command mpiuser@master:~$ mpiexec -f hosts -n 3 /home/mpiuser/bin/cpi then this return with the error. mpiexec was unable to launch the specified application as it could not find an executable: Executable: -f Node: master I could not solve this. I don't know why this happening. However, executable "cpi" is in /home/mpiuser/bin. I request you all to kindly help me to overcome this problem. thanking in advance. yours sincerely Haider Abbas [https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif] _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

problem in execution
by Haider Abbas 28 Aug '18

28 Aug '18

Dear all, I have prepared a 10 node ubuntu based cluster, following the guidelines described in " https://www-users.cs.york.ac.uk/~mjf/pi_cluster/src/Building_a_simple_Beowu… " I have followed each and everthing as mentioned in the above documents. everything is Ok, now I try try to check the cluster by executing the command mpiuser@master:~$ mpiexec -f hosts -n 3 /home/mpiuser/bin/cpi then this return with the error. mpiexec was unable to launch the specified application as it could not find an executable: Executable: -f Node: master I could not solve this. I don't know why this happening. However, executable "cpi" is in /home/mpiuser/bin. I request you all to kindly help me to overcome this problem. thanking in advance. yours sincerely Haider Abbas _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

MPICH Wiki page is down
by Appel, Thibaut 21 Aug '18

21 Aug '18

Hi, https://wiki.mpich.org/mpich/ is down. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH does not work for quad precision
by Jeff Hammond 12 Aug '18

12 Aug '18

Is there a reason you do not want to use mpi_reduce with a user-defined reduction? Perhaps there is something that can be improved about that feature set. Jeff On Thu, Aug 9, 2018 at 5:14 PM Jiancang Zhuang <zhuangjc(a)ism.ac.jp> wrote: > Thanks a lot. > > In fact, I have tried MPI_REAL16. The same thing happened. mpi_gather > does not work for it either. > I have decided to rewrite a new subroutine to replace mpi_reduce. > > Best regards, > > > > On Fri, Aug 10, 2018 at 12:38 AM Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> The specific issue here is that MPI_LONG_DOUBLE does not map to REAL*16. >> You should try MPI_REAL16. I suspect it generates an error telling you >> that it is not supported. >> >> MPI_LONG_DOUBLE corresponds to C "long double", which is an awful type >> that means a bunch of different things depending on your hardware and >> compiler. If you are using x86 Linux, it should be the x87 80-bit type. >> See https://en.wikipedia.org/wiki/Long_double for details. >> >> Jeff >> >> On Thu, Aug 9, 2018 at 8:33 AM, Jeff Hammond <jeff.science(a)gmail.com> >> wrote: >> >>> There is a challenge here because there is no standard type in C >>> corresponding to REAL*16. Either the reduction operation needs to be >>> written in Fortran or MPICH needs to figure out the compiler-dependent >>> equivalent of REAL*16 that works in C. While GCC __float128 and Intel >>> _Quad might be equivalent, this is not a rigorous assumption. >>> >>> I recommend that you write your own user-defined reduction for REAL*16 >>> with the reduction operation callback in Fortran. >>> >>> Jeff >>> >>> On Thu, Aug 9, 2018 at 2:31 AM, Jiancang Zhuang <zhuangjc(a)ism.ac.jp> >>> wrote: >>> >>>> I have found that the fortran version mpi_reduce does not work for >>>> real*18. This can be shown by the following program. I have not test the C >>>> version of mpi_reduce. >>>> >>>> >>>> c-----------------------Fortran code begins >>>> ----------------------------- >>>> implicit real*8 (a-h, o-z) >>>> include 'mpif.h' >>>> real*16 h1, h >>>> >>>> >>>> call mpi_init(ierr) >>>> call mpi_comm_size(mpi_comm_world, nprocs, ierr) >>>> call mpi_comm_rank(mpi_comm_world, myrank, ierr) >>>> >>>> >>>> h1 = (myrank+4) *2.00000000000000 >>>> write(*,'(''before reduce --'', i4,2f12.8)')myrank, h1,h >>>> >>>> >>>> call mpi_reduce(h1,h,1,mpi_long_double,mpi_sum,0, >>>> & mpi_comm_world,ierr) >>>> write(*,'(''after reduce --'',i4,2f12.8)')myrank, h1,h >>>> >>>> call mpi_bcast(h,1,mpi_long_double,0, >>>> & mpi_comm_world,ierr) >>>> >>>> write(*,'(''bcastvalue -- '',i4,2f12.8)')myrank, h1,h >>>> >>>> call mpi_finalize(ierr) >>>> end >>>> >>>> c-----------------------Fortran code begins >>>> ----------------------------- >>>> >>>> >>>> $ mpif77 a.f -o a.out >>>> $ mpirun -np 3 ./a.out >>>> before reduce -- 1 10.00000000 0.00000000 >>>> after reduce -- 1 10.00000000 0.00000000 >>>> before reduce -- 2 12.00000000 0.00000000 >>>> before reduce -- 0 8.00000000 0.00000000 >>>> after reduce -- 2 12.00000000 0.00000000 >>>> after reduce -- 0 8.00000000 8.00000000 >>>> bcastvalue -- 0 8.00000000 8.00000000 >>>> bcastvalue -- 1 10.00000000 8.00000000 >>>> bcastvalue -- 2 12.00000000 8.00000000 >>>> >>>> >>>> >>>> _______________________________________________ >>>> discuss mailing list discuss(a)mpich.org >>>> To manage subscription options or unsubscribe: >>>> https://lists.mpich.org/mailman/listinfo/discuss >>>> >>>> >>> >>> >>> -- >>> Jeff Hammond >>> jeff.science(a)gmail.com >>> http://jeffhammond.github.io/ >>> >> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> > > > Address: 10-3 Midori-cho, Tachikawa, Tokyo 190-8562, Japan > Phone: +81-50-5533-8532 > Fax: +81-42-526-4335 > email: zhuangjc(a)ism.ac.jp > homepage: http://bemlar.ism.ac.jp/zhuang > > ---------------------------------------------------------------------------------------- > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH does not work for quad precision
by Jiancang Zhuang 09 Aug '18

09 Aug '18

Thanks a lot. In fact, I have tried MPI_REAL16. The same thing happened. mpi_gather does not work for it either. I have decided to rewrite a new subroutine to replace mpi_reduce. Best regards, On Fri, Aug 10, 2018 at 12:38 AM Jeff Hammond <jeff.science(a)gmail.com> wrote: > The specific issue here is that MPI_LONG_DOUBLE does not map to REAL*16. > You should try MPI_REAL16. I suspect it generates an error telling you > that it is not supported. > > MPI_LONG_DOUBLE corresponds to C "long double", which is an awful type > that means a bunch of different things depending on your hardware and > compiler. If you are using x86 Linux, it should be the x87 80-bit type. > See https://en.wikipedia.org/wiki/Long_double for details. > > Jeff > > On Thu, Aug 9, 2018 at 8:33 AM, Jeff Hammond <jeff.science(a)gmail.com> > wrote: > >> There is a challenge here because there is no standard type in C >> corresponding to REAL*16. Either the reduction operation needs to be >> written in Fortran or MPICH needs to figure out the compiler-dependent >> equivalent of REAL*16 that works in C. While GCC __float128 and Intel >> _Quad might be equivalent, this is not a rigorous assumption. >> >> I recommend that you write your own user-defined reduction for REAL*16 >> with the reduction operation callback in Fortran. >> >> Jeff >> >> On Thu, Aug 9, 2018 at 2:31 AM, Jiancang Zhuang <zhuangjc(a)ism.ac.jp> >> wrote: >> >>> I have found that the fortran version mpi_reduce does not work for >>> real*18. This can be shown by the following program. I have not test the C >>> version of mpi_reduce. >>> >>> >>> c-----------------------Fortran code begins ----------------------------- >>> implicit real*8 (a-h, o-z) >>> include 'mpif.h' >>> real*16 h1, h >>> >>> >>> call mpi_init(ierr) >>> call mpi_comm_size(mpi_comm_world, nprocs, ierr) >>> call mpi_comm_rank(mpi_comm_world, myrank, ierr) >>> >>> >>> h1 = (myrank+4) *2.00000000000000 >>> write(*,'(''before reduce --'', i4,2f12.8)')myrank, h1,h >>> >>> >>> call mpi_reduce(h1,h,1,mpi_long_double,mpi_sum,0, >>> & mpi_comm_world,ierr) >>> write(*,'(''after reduce --'',i4,2f12.8)')myrank, h1,h >>> >>> call mpi_bcast(h,1,mpi_long_double,0, >>> & mpi_comm_world,ierr) >>> >>> write(*,'(''bcastvalue -- '',i4,2f12.8)')myrank, h1,h >>> >>> call mpi_finalize(ierr) >>> end >>> >>> c-----------------------Fortran code begins ----------------------------- >>> >>> >>> $ mpif77 a.f -o a.out >>> $ mpirun -np 3 ./a.out >>> before reduce -- 1 10.00000000 0.00000000 >>> after reduce -- 1 10.00000000 0.00000000 >>> before reduce -- 2 12.00000000 0.00000000 >>> before reduce -- 0 8.00000000 0.00000000 >>> after reduce -- 2 12.00000000 0.00000000 >>> after reduce -- 0 8.00000000 8.00000000 >>> bcastvalue -- 0 8.00000000 8.00000000 >>> bcastvalue -- 1 10.00000000 8.00000000 >>> bcastvalue -- 2 12.00000000 8.00000000 >>> >>> >>> >>> _______________________________________________ >>> discuss mailing list discuss(a)mpich.org >>> To manage subscription options or unsubscribe: >>> https://lists.mpich.org/mailman/listinfo/discuss >>> >>> >> >> >> -- >> Jeff Hammond >> jeff.science(a)gmail.com >> http://jeffhammond.github.io/ >> > > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > Address: 10-3 Midori-cho, Tachikawa, Tokyo 190-8562, Japan Phone: +81-50-5533-8532 Fax: +81-42-526-4335 email: zhuangjc(a)ism.ac.jp homepage: http://bemlar.ism.ac.jp/zhuang ---------------------------------------------------------------------------------------- _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] MPICH does not work for quad precision
by Jeff Hammond 09 Aug '18

09 Aug '18

The specific issue here is that MPI_LONG_DOUBLE does not map to REAL*16. You should try MPI_REAL16. I suspect it generates an error telling you that it is not supported. MPI_LONG_DOUBLE corresponds to C "long double", which is an awful type that means a bunch of different things depending on your hardware and compiler. If you are using x86 Linux, it should be the x87 80-bit type. See https://en.wikipedia.org/wiki/Long_double for details. Jeff On Thu, Aug 9, 2018 at 8:33 AM, Jeff Hammond <jeff.science(a)gmail.com> wrote: > There is a challenge here because there is no standard type in C > corresponding to REAL*16. Either the reduction operation needs to be > written in Fortran or MPICH needs to figure out the compiler-dependent > equivalent of REAL*16 that works in C. While GCC __float128 and Intel > _Quad might be equivalent, this is not a rigorous assumption. > > I recommend that you write your own user-defined reduction for REAL*16 > with the reduction operation callback in Fortran. > > Jeff > > On Thu, Aug 9, 2018 at 2:31 AM, Jiancang Zhuang <zhuangjc(a)ism.ac.jp> > wrote: > >> I have found that the fortran version mpi_reduce does not work for >> real*18. This can be shown by the following program. I have not test the C >> version of mpi_reduce. >> >> >> c-----------------------Fortran code begins ----------------------------- >> implicit real*8 (a-h, o-z) >> include 'mpif.h' >> real*16 h1, h >> >> >> call mpi_init(ierr) >> call mpi_comm_size(mpi_comm_world, nprocs, ierr) >> call mpi_comm_rank(mpi_comm_world, myrank, ierr) >> >> >> h1 = (myrank+4) *2.00000000000000 >> write(*,'(''before reduce --'', i4,2f12.8)')myrank, h1,h >> >> >> call mpi_reduce(h1,h,1,mpi_long_double,mpi_sum,0, >> & mpi_comm_world,ierr) >> write(*,'(''after reduce --'',i4,2f12.8)')myrank, h1,h >> >> call mpi_bcast(h,1,mpi_long_double,0, >> & mpi_comm_world,ierr) >> >> write(*,'(''bcastvalue -- '',i4,2f12.8)')myrank, h1,h >> >> call mpi_finalize(ierr) >> end >> >> c-----------------------Fortran code begins ----------------------------- >> >> >> $ mpif77 a.f -o a.out >> $ mpirun -np 3 ./a.out >> before reduce -- 1 10.00000000 0.00000000 >> after reduce -- 1 10.00000000 0.00000000 >> before reduce -- 2 12.00000000 0.00000000 >> before reduce -- 0 8.00000000 0.00000000 >> after reduce -- 2 12.00000000 0.00000000 >> after reduce -- 0 8.00000000 8.00000000 >> bcastvalue -- 0 8.00000000 8.00000000 >> bcastvalue -- 1 10.00000000 8.00000000 >> bcastvalue -- 2 12.00000000 8.00000000 >> >> >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >> > > > -- > Jeff Hammond > jeff.science(a)gmail.com > http://jeffhammond.github.io/ > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0