- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Installing MPICH on clusters
by Raffenetti, Kenneth J. 21 Sep '21

21 Sep '21

MPICH does not need to use Spectrum MPI's libraries. Are you sure your application isn't actually linked against Spectrum MPI and not MPICH? Ken On 9/20/21, 6:33 PM, "Feimi Yu via discuss" <discuss(a)mpich.org> wrote: Thank you for the hint! I finally got my job running on my own MPI build. Just out of curious: Actually, I searched this libmpiprofilesupport library before and discovered that it is specific to Spectrum MPI so I didn't go further. But MPICH just ran magically after I load Spectrum MPI and have the Spectrum MPI lib path in $LD_LIBRARY_PATH. Why does MPICH have to use Spectrum MPI's libraries? Thanks! Feimi On 9/19/2021 1:57 PM, Zhou, Hui wrote: > ./step-17.release: error while loading shared libraries: libmpiprofilesupport.so.3: cannot open shared object file: No such file or directory This is complaining about cannot finding a dynamic library (that is linked in your binary) on the compute node. Make sure the path to that library is in the LD_LIBRARY_PATH. ________________________________________ From: Feimi Yu <yuf2(a)rpi.edu> <mailto:[email protected]> Sent: Saturday, September 18, 2021 9:13 PM To: Zhou, Hui <zhouh(a)anl.gov> <mailto:[email protected]>; discuss(a)mpich.org <discuss(a)mpich.org> <mailto:[email protected]> Subject: Re: [mpich-discuss] Installing MPICH on clusters Hi Hui, Thank you for the response! Here is the Slurm batch file I used to run a program with MPICH configured with Hydra: #!/bin/bash #SBATCH --job-name=5e-7 #SBATCH --partition=el8 #SBATCH --time 6:00:00 #SBATCH --ntasks 40 #SBATCH --nodes 1 #SBATCH --gres=gpu:4 date export LD_LIBRARY_PATH=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/lib:$MPI_ROOT:$LD_LIBRARY_PATH srun --mpi=mpichmx hostname -s | sort -u > /tmp/hosts.$SLURM_JOB_ID awk "{ print \$0 \":40\"; }" /tmp/hosts.$SLURM_JOB_ID >/tmp/tmp.$SLURM_JOB_ID mv /tmp/tmp.$SLURM_JOB_ID ./hosts.$SLURM_JOB_ID /gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/bin/mpiexec -f ./hosts.$SLURM_JOB_ID -np $SLURM_NPROCS ./step-17.release date And the error message is: ./step-17.release: error while loading shared libraries: libmpiprofilesupport.so.3: cannot open shared object file: No such file or directory I was not sure if this is related to a network problem because the clusters use Infiniband. Running "/sbin/ifconfig" gives ib0, ib1, ib2 and ib3. I tried the option "-iface ib0" and the error message became: [mpiexec@dcs176] HYDU_sock_get_iface_ip (utils/sock/sock.c:451): unable to find interface ib0 [mpiexec@dcs176] HYDU_sock_create_and_listen_portstr (utils/sock/sock.c:496): unable to get network interface IP [mpiexec@dcs176] HYD_pmci_launch_procs (pm/pmiserv/pmiserv_pmci.c:79): unable to create PMI port [mpiexec@dcs176] main (ui/mpich/mpiexec.c:322): process manager returned error launching processes Specifying ib1-ib3 gives similar results. Thanks! Feimi On 9/17/21 7:57 PM, Zhou, Hui wrote: Hi Feimi, Hydra should be able to work with slurm. How are you launching the job and what is the failure message? -- Hui Zhou ________________________________________ From: Feimi Yu via discuss <discuss(a)mpich.org> <mailto:[email protected]> Sent: Friday, September 17, 2021 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> <mailto:[email protected]> Cc: Feimi Yu <yuf2(a)rpi.edu> <mailto:[email protected]> Subject: [mpich-discuss] Installing MPICH on clusters Hi, I'm working on a supercomputer which only provides Spectrum MPI implementation in modules. Since our code does not perform well with Spectrum MPI I decided to install an MPICH build on our own partition (I'm not an administrator.) The supercomputer has a rhel8 system on ppc64le architecture with Slurm as the process manager. I tried several building options according to the user guide but could not run a job so I have a few questions. Here are things I tried: 1. Build with Hydra PM. I could not launch a job with Hydra at all. 2. Then I decided to use ``--with-pm=none`` option to build and use srun + ``mpiexec -f hostfile`` to launch my job. But what confuses me is the PMI setting: srun --mpi=list gives following: srun: mpi/mpichgm srun: mpi/mpichmx srun: mpi/none srun: mpi/mvapich srun: mpi/openmpi srun: mpi/pmi2 srun: mpi/lam srun: mpi/mpich1_p4 srun: mpi/mpich1_shmem At first I tried use pmix since I found pmix libraries. But it didn't do the trick. It segfaults on PMPI_Init_thread(). The error message is: [dcs135:2312190] PMIX ERROR: NOT-FOUND in file client/pmix_client.c at line 562 Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(159): MPID_Init(509).......: MPIR_pmi_init(92)....: PMIX_Init returned -46 [dcs135:2312190:0:2312190] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil)) Then I switched to pmi2 but make keeps telling me undefined reference to PMI2 library. (actually I couldn't find the pmi2 libraries either.) Then I used ``--with-pmi=slurm``, and it turned out that I couldn't locate the Slurm header files. I guess I don't have the permission to access them. I was wondering if it is still possible for me to build a usable MPICH as a user? If yes, how can I do to have the PMI work? Thanks! Feimi

1 0

Re: [mpich-discuss] Installing MPICH on clusters
by Feimi Yu 20 Sep '21

20 Sep '21

Thank you for the hint! I finally got my job running on my own MPI build. Just out of curious: Actually, I searched this libmpiprofilesupport library before and discovered that it is specific to Spectrum MPI so I didn't go further. But MPICH just ran magically after I load Spectrum MPI and have the Spectrum MPI lib path in $LD_LIBRARY_PATH. Why does MPICH have to use Spectrum MPI's libraries? Thanks! Feimi On 9/19/2021 1:57 PM, Zhou, Hui wrote: > > ./step-17.release: error while loading shared libraries: > libmpiprofilesupport.so.3: cannot open shared object file: No such > file or directory > > This is complaining about cannot finding a dynamic library (that is > linked in your binary) on the compute node. Make sure the path to that > library is in the LD_LIBRARY_PATH. > > ------------------------------------------------------------------------ > *From:* Feimi Yu <yuf2(a)rpi.edu> > *Sent:* Saturday, September 18, 2021 9:13 PM > *To:* Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> > *Subject:* Re: [mpich-discuss] Installing MPICH on clusters > > Hi Hui, > > > Thank you for the response! Here is the Slurm batch file I used to run > a program with MPICH configured with Hydra: > > /#!/bin/bash// > //#SBATCH --job-name=5e-7// > //#SBATCH --partition=el8// > //#SBATCH --time 6:00:00// > //#SBATCH --ntasks 40// > //#SBATCH --nodes 1// > //#SBATCH --gres=gpu:4// > // > //date// > //export > LD_LIBRARY_PATH=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/lib:$MPI_ROOT:$LD_LIBRARY_PATH// > //srun --mpi=mpichmx hostname -s | sort -u > /tmp/hosts.$SLURM_JOB_ID// > //awk "{ print \$0 \":40\"; }" /tmp/hosts.$SLURM_JOB_ID > >/tmp/tmp.$SLURM_JOB_ID// > //mv /tmp/tmp.$SLURM_JOB_ID ./hosts.$SLURM_JOB_ID// > // > ///gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/bin/mpiexec > -f ./hosts.$SLURM_JOB_ID -np $SLURM_NPROCS ./step-17.release// > // > //date/ > > > And the error message is: > > ./step-17.release: error while loading shared libraries: > libmpiprofilesupport.so.3: cannot open shared object file: No such > file or directory > > > I was not sure if this is related to a network problem because the > clusters use Infiniband. Running "/sbin/ifconfig" gives ib0, ib1, ib2 > and ib3. I tried the option "-iface ib0" and the error message became: > > [mpiexec@dcs176] HYDU_sock_get_iface_ip (utils/sock/sock.c:451): > unable to find interface ib0 > [mpiexec@dcs176] HYDU_sock_create_and_listen_portstr > (utils/sock/sock.c:496): unable to get network interface IP > [mpiexec@dcs176] HYD_pmci_launch_procs (pm/pmiserv/pmiserv_pmci.c:79): > unable to create PMI port > [mpiexec@dcs176] main (ui/mpich/mpiexec.c:322): process manager > returned error launching processes > > > Specifying ib1-ib3 gives similar results. > > > Thanks! > > Feimi > > > On 9/17/21 7:57 PM, Zhou, Hui wrote: >> Hi Feimi, >> >> Hydra should be able to work with slurm. How are you launching the >> job and what is the failure message? >> >> -- >> Hui Zhou >> ------------------------------------------------------------------------ >> *From:* Feimi Yu via discuss <discuss(a)mpich.org> >> <mailto:[email protected]> >> *Sent:* Friday, September 17, 2021 10:55 AM >> *To:* discuss(a)mpich.org <mailto:[email protected]> >> <discuss(a)mpich.org> <mailto:[email protected]> >> *Cc:* Feimi Yu <yuf2(a)rpi.edu> <mailto:[email protected]> >> *Subject:* [mpich-discuss] Installing MPICH on clusters >> >> Hi, >> >> I'm working on a supercomputer which only provides Spectrum MPI >> implementation in modules. Since our code does not perform well with >> Spectrum MPI I decided to install an MPICH build on our own partition >> (I'm not an administrator.) The supercomputer has a rhel8 system on >> ppc64le architecture with Slurm as the process manager. I tried >> several building options according to the user guide but could not >> run a job so I have a few questions. Here are things I tried: >> >> 1. Build with Hydra PM. I could not launch a job with Hydra at all. >> >> 2. Then I decided to use ``--with-pm=none`` option to build and use >> srun + ``mpiexec -f hostfile`` to launch my job. But what confuses me >> is the PMI setting: >> >> srun --mpi=list gives following: >> >> srun: mpi/mpichgm >> srun: mpi/mpichmx >> srun: mpi/none >> srun: mpi/mvapich >> srun: mpi/openmpi >> srun: mpi/pmi2 >> srun: mpi/lam >> srun: mpi/mpich1_p4 >> srun: mpi/mpich1_shmem >> >> At first I tried use pmix since I found pmix libraries. But it didn't >> do the trick. It segfaults on PMPI_Init_thread(). The error message is: >> >> /[dcs135:2312190] PMIX ERROR: NOT-FOUND in file client/pmix_client.c >> at line 562/ >> >> /Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in >> PMPI_Init_thread: Other MPI error, error stack:// >> //MPIR_Init_thread(159): // >> //MPID_Init(509).......: // >> //MPIR_pmi_init(92)....: PMIX_Init returned -46 // >> //[dcs135:2312190:0:2312190] Caught signal 11 (Segmentation fault: >> address not mapped to object at address (nil))// >> / >> >> Then I switched to pmi2 but make keeps telling me undefined reference >> to PMI2 library. (actually I couldn't find the pmi2 libraries either.) >> >> Then I used ``--with-pmi=slurm``, and it turned out that I couldn't >> locate the Slurm header files. I guess I don't have the permission to >> access them. >> >> I was wondering if it is still possible for me to build a usable >> MPICH as a user? If yes, how can I do to have the PMI work? >> >> Thanks! >> >> Feimi

1 0

Re: [mpich-discuss] Installing MPICH on clusters
by Zhou, Hui 19 Sep '21

19 Sep '21

> ./step-17.release: error while loading shared libraries: libmpiprofilesupport.so.3: cannot open shared object file: No such file or directory This is complaining about cannot finding a dynamic library (that is linked in your binary) on the compute node. Make sure the path to that library is in the LD_LIBRARY_PATH. ________________________________ From: Feimi Yu <yuf2(a)rpi.edu> Sent: Saturday, September 18, 2021 9:13 PM To: Zhou, Hui <zhouh(a)anl.gov>; discuss(a)mpich.org <discuss(a)mpich.org> Subject: Re: [mpich-discuss] Installing MPICH on clusters Hi Hui, Thank you for the response! Here is the Slurm batch file I used to run a program with MPICH configured with Hydra: #!/bin/bash #SBATCH --job-name=5e-7 #SBATCH --partition=el8 #SBATCH --time 6:00:00 #SBATCH --ntasks 40 #SBATCH --nodes 1 #SBATCH --gres=gpu:4 date export LD_LIBRARY_PATH=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/lib:$MPI_ROOT:$LD_LIBRARY_PATH srun --mpi=mpichmx hostname -s | sort -u > /tmp/hosts.$SLURM_JOB_ID awk "{ print \$0 \":40\"; }" /tmp/hosts.$SLURM_JOB_ID >/tmp/tmp.$SLURM_JOB_ID mv /tmp/tmp.$SLURM_JOB_ID ./hosts.$SLURM_JOB_ID /gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/bin/mpiexec -f ./hosts.$SLURM_JOB_ID -np $SLURM_NPROCS ./step-17.release date And the error message is: ./step-17.release: error while loading shared libraries: libmpiprofilesupport.so.3: cannot open shared object file: No such file or directory I was not sure if this is related to a network problem because the clusters use Infiniband. Running "/sbin/ifconfig" gives ib0, ib1, ib2 and ib3. I tried the option "-iface ib0" and the error message became: [mpiexec@dcs176] HYDU_sock_get_iface_ip (utils/sock/sock.c:451): unable to find interface ib0 [mpiexec@dcs176] HYDU_sock_create_and_listen_portstr (utils/sock/sock.c:496): unable to get network interface IP [mpiexec@dcs176] HYD_pmci_launch_procs (pm/pmiserv/pmiserv_pmci.c:79): unable to create PMI port [mpiexec@dcs176] main (ui/mpich/mpiexec.c:322): process manager returned error launching processes Specifying ib1-ib3 gives similar results. Thanks! Feimi On 9/17/21 7:57 PM, Zhou, Hui wrote: Hi Feimi, Hydra should be able to work with slurm. How are you launching the job and what is the failure message? -- Hui Zhou ________________________________ From: Feimi Yu via discuss <discuss(a)mpich.org><mailto:[email protected]> Sent: Friday, September 17, 2021 10:55 AM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org><mailto:[email protected]> Cc: Feimi Yu <yuf2(a)rpi.edu><mailto:[email protected]> Subject: [mpich-discuss] Installing MPICH on clusters Hi, I'm working on a supercomputer which only provides Spectrum MPI implementation in modules. Since our code does not perform well with Spectrum MPI I decided to install an MPICH build on our own partition (I'm not an administrator.) The supercomputer has a rhel8 system on ppc64le architecture with Slurm as the process manager. I tried several building options according to the user guide but could not run a job so I have a few questions. Here are things I tried: 1. Build with Hydra PM. I could not launch a job with Hydra at all. 2. Then I decided to use ``--with-pm=none`` option to build and use srun + ``mpiexec -f hostfile`` to launch my job. But what confuses me is the PMI setting: srun --mpi=list gives following: srun: mpi/mpichgm srun: mpi/mpichmx srun: mpi/none srun: mpi/mvapich srun: mpi/openmpi srun: mpi/pmi2 srun: mpi/lam srun: mpi/mpich1_p4 srun: mpi/mpich1_shmem At first I tried use pmix since I found pmix libraries. But it didn't do the trick. It segfaults on PMPI_Init_thread(). The error message is: [dcs135:2312190] PMIX ERROR: NOT-FOUND in file client/pmix_client.c at line 562 Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(159): MPID_Init(509).......: MPIR_pmi_init(92)....: PMIX_Init returned -46 [dcs135:2312190:0:2312190] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil)) Then I switched to pmi2 but make keeps telling me undefined reference to PMI2 library. (actually I couldn't find the pmi2 libraries either.) Then I used ``--with-pmi=slurm``, and it turned out that I couldn't locate the Slurm header files. I guess I don't have the permission to access them. I was wondering if it is still possible for me to build a usable MPICH as a user? If yes, how can I do to have the PMI work? Thanks! Feimi

1 0

Re: [mpich-discuss] Installing MPICH on clusters
by Feimi Yu 18 Sep '21

18 Sep '21

Hi Hui, Thank you for the response! Here is the Slurm batch file I used to run a program with MPICH configured with Hydra: /#!/bin/bash// //#SBATCH --job-name=5e-7// //#SBATCH --partition=el8// //#SBATCH --time 6:00:00// //#SBATCH --ntasks 40// //#SBATCH --nodes 1// //#SBATCH --gres=gpu:4// // //date// //export LD_LIBRARY_PATH=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/lib:$MPI_ROOT:$LD_LIBRARY_PATH// //srun --mpi=mpichmx hostname -s | sort -u > /tmp/hosts.$SLURM_JOB_ID// //awk "{ print \$0 \":40\"; }" /tmp/hosts.$SLURM_JOB_ID >/tmp/tmp.$SLURM_JOB_ID// //mv /tmp/tmp.$SLURM_JOB_ID ./hosts.$SLURM_JOB_ID// // ///gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/bin/mpiexec -f ./hosts.$SLURM_JOB_ID -np $SLURM_NPROCS ./step-17.release// // //date/ And the error message is: ./step-17.release: error while loading shared libraries: libmpiprofilesupport.so.3: cannot open shared object file: No such file or directory I was not sure if this is related to a network problem because the clusters use Infiniband. Running "/sbin/ifconfig" gives ib0, ib1, ib2 and ib3. I tried the option "-iface ib0" and the error message became: [mpiexec@dcs176] HYDU_sock_get_iface_ip (utils/sock/sock.c:451): unable to find interface ib0 [mpiexec@dcs176] HYDU_sock_create_and_listen_portstr (utils/sock/sock.c:496): unable to get network interface IP [mpiexec@dcs176] HYD_pmci_launch_procs (pm/pmiserv/pmiserv_pmci.c:79): unable to create PMI port [mpiexec@dcs176] main (ui/mpich/mpiexec.c:322): process manager returned error launching processes Specifying ib1-ib3 gives similar results. Thanks! Feimi On 9/17/21 7:57 PM, Zhou, Hui wrote: > Hi Feimi, > > Hydra should be able to work with slurm. How are you launching the job > and what is the failure message? > > -- > Hui Zhou > ------------------------------------------------------------------------ > *From:* Feimi Yu via discuss <discuss(a)mpich.org> > *Sent:* Friday, September 17, 2021 10:55 AM > *To:* discuss(a)mpich.org <discuss(a)mpich.org> > *Cc:* Feimi Yu <yuf2(a)rpi.edu> > *Subject:* [mpich-discuss] Installing MPICH on clusters > > Hi, > > I'm working on a supercomputer which only provides Spectrum MPI > implementation in modules. Since our code does not perform well with > Spectrum MPI I decided to install an MPICH build on our own partition > (I'm not an administrator.) The supercomputer has a rhel8 system on > ppc64le architecture with Slurm as the process manager. I tried > several building options according to the user guide but could not run > a job so I have a few questions. Here are things I tried: > > 1. Build with Hydra PM. I could not launch a job with Hydra at all. > > 2. Then I decided to use ``--with-pm=none`` option to build and use > srun + ``mpiexec -f hostfile`` to launch my job. But what confuses me > is the PMI setting: > > srun --mpi=list gives following: > > srun: mpi/mpichgm > srun: mpi/mpichmx > srun: mpi/none > srun: mpi/mvapich > srun: mpi/openmpi > srun: mpi/pmi2 > srun: mpi/lam > srun: mpi/mpich1_p4 > srun: mpi/mpich1_shmem > > At first I tried use pmix since I found pmix libraries. But it didn't > do the trick. It segfaults on PMPI_Init_thread(). The error message is: > > /[dcs135:2312190] PMIX ERROR: NOT-FOUND in file client/pmix_client.c > at line 562/ > > /Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in > PMPI_Init_thread: Other MPI error, error stack:// > //MPIR_Init_thread(159): // > //MPID_Init(509).......: // > //MPIR_pmi_init(92)....: PMIX_Init returned -46 // > //[dcs135:2312190:0:2312190] Caught signal 11 (Segmentation fault: > address not mapped to object at address (nil))// > / > > Then I switched to pmi2 but make keeps telling me undefined reference > to PMI2 library. (actually I couldn't find the pmi2 libraries either.) > > Then I used ``--with-pmi=slurm``, and it turned out that I couldn't > locate the Slurm header files. I guess I don't have the permission to > access them. > > I was wondering if it is still possible for me to build a usable MPICH > as a user? If yes, how can I do to have the PMI work? > > Thanks! > > Feimi

1 0

Re: [mpich-discuss] Installing MPICH on clusters
by Zhou, Hui 17 Sep '21

17 Sep '21

Hi Feimi, Hydra should be able to work with slurm. How are you launching the job and what is the failure message? -- Hui Zhou ________________________________ From: Feimi Yu via discuss <discuss(a)mpich.org> Sent: Friday, September 17, 2021 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Feimi Yu <yuf2(a)rpi.edu> Subject: [mpich-discuss] Installing MPICH on clusters Hi, I'm working on a supercomputer which only provides Spectrum MPI implementation in modules. Since our code does not perform well with Spectrum MPI I decided to install an MPICH build on our own partition (I'm not an administrator.) The supercomputer has a rhel8 system on ppc64le architecture with Slurm as the process manager. I tried several building options according to the user guide but could not run a job so I have a few questions. Here are things I tried: 1. Build with Hydra PM. I could not launch a job with Hydra at all. 2. Then I decided to use ``--with-pm=none`` option to build and use srun + ``mpiexec -f hostfile`` to launch my job. But what confuses me is the PMI setting: srun --mpi=list gives following: srun: mpi/mpichgm srun: mpi/mpichmx srun: mpi/none srun: mpi/mvapich srun: mpi/openmpi srun: mpi/pmi2 srun: mpi/lam srun: mpi/mpich1_p4 srun: mpi/mpich1_shmem At first I tried use pmix since I found pmix libraries. But it didn't do the trick. It segfaults on PMPI_Init_thread(). The error message is: [dcs135:2312190] PMIX ERROR: NOT-FOUND in file client/pmix_client.c at line 562 Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(159): MPID_Init(509).......: MPIR_pmi_init(92)....: PMIX_Init returned -46 [dcs135:2312190:0:2312190] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil)) Then I switched to pmi2 but make keeps telling me undefined reference to PMI2 library. (actually I couldn't find the pmi2 libraries either.) Then I used ``--with-pmi=slurm``, and it turned out that I couldn't locate the Slurm header files. I guess I don't have the permission to access them. I was wondering if it is still possible for me to build a usable MPICH as a user? If yes, how can I do to have the PMI work? Thanks! Feimi

1 0

Installing MPICH on clusters
by Feimi Yu 17 Sep '21

17 Sep '21

Hi, I'm working on a supercomputer which only provides Spectrum MPI implementation in modules. Since our code does not perform well with Spectrum MPI I decided to install an MPICH build on our own partition (I'm not an administrator.) The supercomputer has a rhel8 system on ppc64le architecture with Slurm as the process manager. I tried several building options according to the user guide but could not run a job so I have a few questions. Here are things I tried: 1. Build with Hydra PM. I could not launch a job with Hydra at all. 2. Then I decided to use ``--with-pm=none`` option to build and use srun + ``mpiexec -f hostfile`` to launch my job. But what confuses me is the PMI setting: srun --mpi=list gives following: srun: mpi/mpichgm srun: mpi/mpichmx srun: mpi/none srun: mpi/mvapich srun: mpi/openmpi srun: mpi/pmi2 srun: mpi/lam srun: mpi/mpich1_p4 srun: mpi/mpich1_shmem At first I tried use pmix since I found pmix libraries. But it didn't do the trick. It segfaults on PMPI_Init_thread(). The error message is: /[dcs135:2312190] PMIX ERROR: NOT-FOUND in file client/pmix_client.c at line 562/ /Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack:// //MPIR_Init_thread(159): // //MPID_Init(509).......: // //MPIR_pmi_init(92)....: PMIX_Init returned -46 // //[dcs135:2312190:0:2312190] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))// / Then I switched to pmi2 but make keeps telling me undefined reference to PMI2 library. (actually I couldn't find the pmi2 libraries either.) Then I used ``--with-pmi=slurm``, and it turned out that I couldn't locate the Slurm header files. I guess I don't have the permission to access them. I was wondering if it is still possible for me to build a usable MPICH as a user? If yes, how can I do to have the PMI work? Thanks! Feimi

1 0

Re: [mpich-discuss] configure: error: '' is empty
by Zhou, Hui 07 Sep '21

07 Sep '21

Hi Jeff, Try supply CROSS_F77_SIZEOF_REAL and CROSS_F77_SIZEOF_DOUBLE_PRECISION. -- Hui Zhou From: Jeff Hammond via discuss <discuss(a)mpich.org> Date: Tuesday, September 7, 2021 at 6:36 AM To: MPICH <discuss(a)mpich.org> Cc: Jeff Hammond <jeff.science(a)gmail.com> Subject: [mpich-discuss] configure: error: '' is empty I'm trying to cross-compile MPICH for RISC-V with CROSS_F77_TRUE_VALUE=1 CROSS_F77_FALSE_VALUE=0 CROSS_F77_SIZEOF_INTEGER=4 ../configure --host=riscv64-linux-gnu --prefix=/tmp/riscv/mpich CC=riscv64-linux-gnu-gcc-9 CXX=riscv64-linux-gnu-g++-9 FC=riscv64-linux-gnu-gfortran-9 --with-device=ch4:ofi This results in the following error, which I cannot interpret. checking for size of Fortran type integer... 4 checking for size of Fortran type real... 0 checking for size of Fortran type double precision... 0 checking whether integer*1 is supported... yes checking whether integer*2 is supported... yes checking whether integer*4 is supported... yes checking whether integer*8 is supported... yes checking whether integer*16 is supported... yes checking whether real*4 is supported... yes checking whether real*8 is supported... yes checking whether real*16 is supported... yes configure: WARNING: Unable to determine size of a DOUBLE PRECISION type; using 8 checking for C type matching Fortran real... unavailable checking for C type matching Fortran double... unavailable configure: WARNING: Unable to determine size of a DOUBLE PRECISION type; using 8 checking for C type matching Fortran integer... int checking for values of Fortran logicals... True is 1 and False is 0 configure: error: '' is empty Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com<mailto:[email protected]> http://jeffhammond.github.io/

1 0

configure: error: '' is empty
by Jeff Hammond 07 Sep '21

07 Sep '21

I'm trying to cross-compile MPICH for RISC-V with CROSS_F77_TRUE_VALUE=1 CROSS_F77_FALSE_VALUE=0 CROSS_F77_SIZEOF_INTEGER=4 ../configure --host=riscv64-linux-gnu --prefix=/tmp/riscv/mpich CC=riscv64-linux-gnu-gcc-9 CXX=riscv64-linux-gnu-g++-9 FC=riscv64-linux-gnu-gfortran-9 --with-device=ch4:ofi This results in the following error, which I cannot interpret. checking for size of Fortran type integer... 4 checking for size of Fortran type real... 0 checking for size of Fortran type double precision... 0 checking whether integer*1 is supported... yes checking whether integer*2 is supported... yes checking whether integer*4 is supported... yes checking whether integer*8 is supported... yes checking whether integer*16 is supported... yes checking whether real*4 is supported... yes checking whether real*8 is supported... yes checking whether real*16 is supported... yes configure: WARNING: Unable to determine size of a DOUBLE PRECISION type; using 8 checking for C type matching Fortran real... unavailable checking for C type matching Fortran double... unavailable configure: WARNING: Unable to determine size of a DOUBLE PRECISION type; using 8 checking for C type matching Fortran integer... int checking for values of Fortran logicals... True is 1 and False is 0 *configure: error: '' is empty* Thanks, Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-discuss] Build failed with Undefined symbols
by Raffenetti, Kenneth J. 20 Jul '21

20 Jul '21

Issue was the configure version check only applied to MPICH and not Hydra. Pushed a fix in https://github.com/pmodels/mpich/pull/5449. Ken On 7/20/21, 10:28 AM, "Raffenetti, Kenneth J. via discuss" <discuss(a)mpich.org> wrote: We did add a minimum version check during configure, but it may have fallen out of date. We can take a look. Ken On 7/20/21, 8:18 AM, "Junchao Zhang via discuss" <discuss(a)mpich.org> wrote: Adding --with-hwloc=embedded fixed the problem. Yes, I have preinstalled hwloc (it is common since I also found hwloc on another Linux cluster that had the same kind of hw errors). I guess it was because mpich is not compatible with the system's preinstalled hwloc.Perhaps a better approach is for mpich to check the hwloc at configure time (e.g., many petsc third party libraries are checked in this way), then report any compatibility issue and give clear messages to users asking them to either upgrade their hwloc or use the embedded one. --Junchao Zhang On Mon, Jul 19, 2021 at 6:16 PM Tony Curtis <anthony.curtis(a)stonybrook.edu> wrote: > On Jul 19, 2021, at 3:59 PM, Junchao Zhang via discuss <discuss(a)mpich.org> wrote: > > Hello, > > I configured mpich/main (or 4.0a2) with > ./configure CC=gcc CXX=g++ --enable-fortran=no > on Mac with Apple clang version 12.0.5 (clang-1205.0.22.11). The build failed with > > ... > CCLD hydra_pmi_proxy > Undefined symbols for architecture x86_64: > "_hwloc_topology_set_io_types_filter", referenced from: > _HYDT_topo_hwloc_init in libhydra.a(topo_hwloc.o) > Do you have hwloc installed (e.g. via brew)? Or if so, is it new enough? Tony _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Build failed with Undefined symbols
by Raffenetti, Kenneth J. 20 Jul '21

20 Jul '21

We did add a minimum version check during configure, but it may have fallen out of date. We can take a look. Ken On 7/20/21, 8:18 AM, "Junchao Zhang via discuss" <discuss(a)mpich.org> wrote: Adding --with-hwloc=embedded fixed the problem. Yes, I have preinstalled hwloc (it is common since I also found hwloc on another Linux cluster that had the same kind of hw errors). I guess it was because mpich is not compatible with the system's preinstalled hwloc.Perhaps a better approach is for mpich to check the hwloc at configure time (e.g., many petsc third party libraries are checked in this way), then report any compatibility issue and give clear messages to users asking them to either upgrade their hwloc or use the embedded one. --Junchao Zhang On Mon, Jul 19, 2021 at 6:16 PM Tony Curtis <anthony.curtis(a)stonybrook.edu> wrote: > On Jul 19, 2021, at 3:59 PM, Junchao Zhang via discuss <discuss(a)mpich.org> wrote: > > Hello, > > I configured mpich/main (or 4.0a2) with > ./configure CC=gcc CXX=g++ --enable-fortran=no > on Mac with Apple clang version 12.0.5 (clang-1205.0.22.11). The build failed with > > ... > CCLD hydra_pmi_proxy > Undefined symbols for architecture x86_64: > "_hwloc_topology_set_io_types_filter", referenced from: > _HYDT_topo_hwloc_init in libhydra.a(topo_hwloc.o) > Do you have hwloc installed (e.g. via brew)? Or if so, is it new enough? Tony

1 0