- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] HPC cluster network utilisation monitoring tool ?
by Raffenetti, Ken 26 Oct '21

26 Oct '21

In a previous life, I used NAGIOS (https://www.nagios.com/solutions/network-monitoring/) to monitor all kinds of things on servers. It should be capable of telling you port-level network utilization. As for your switch, you might need to find something specific to the hardware. On the MPI side, profilers like mpiP (https://software.llnl.gov/mpiP/) can capture usage statistics and produce a report which you can look to for insights on how MPI is performing for your applications. Ken On 10/19/21, 12:52 PM, "Nicholas Yue via discuss" <discuss(a)mpich.org> wrote: Hi, I am mainly using MPICH via the mpiexec that ships with Paraview I have a small test cluster and a 1Gbit switch. What is the recommended way to determine and record the network utilization for a given MPI run? I was hoping to gather such information over time and be in a proactive position to plan for network equipment update should I find that I start running into a situation where my network is becoming the performance bottleneck. Cheers -- Nicholas Yue https://www.linkedin.com/in/nicholasyue/

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 26 Oct '21

26 Oct '21

Hui, Please disregard my last message. I got MPICH to build with at newer version of pgc++, namely nvc++. Thanks Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Tuesday, October 26, 2021 2:55 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: Maximum number of inter-communicators? Hui, I'm trying to build 4.0a2 with the Portland Group compiler pgc++ 19.5-0. configure seems to finish without problems. ../mpich-4.0a2/configure -prefix=/home/kmccall/mpich-install-4.0a2 CC=pgcc CXX=pgc++ --with-pbs=/opt/torque --with-device=ch3:nemesis --disable-fortran --with-pm=hydra -enable-debuginfo 2>&1 | tee c.txt but when I run make it ends with this error (actually, many of the same error): /home/kmccall/mpich-build-4.0a2/src/pm/hydra/.libs/libhydra.a(args.o): In function `MPL_gpu_query_pointer_attr': /home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: multiple definition of `MPL_gpu_query_pointer_attr' tools/bootstrap/persist/hydra_persist-persist_server.o:/home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: first defined here I've attached c.txt and m.txt from configure and make. Thanks for any help. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I'm exclusively connecting my processes with inter-communicators. I've found that if I increase the number of processes beyond a certain point, many processes don't get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors - it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager's own node. This is the largest job I've been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I'm using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 26 Oct '21

26 Oct '21

Hui, I'm trying to build 4.0a2 with the Portland Group compiler pgc++ 19.5-0. configure seems to finish without problems. ../mpich-4.0a2/configure -prefix=/home/kmccall/mpich-install-4.0a2 CC=pgcc CXX=pgc++ --with-pbs=/opt/torque --with-device=ch3:nemesis --disable-fortran --with-pm=hydra -enable-debuginfo 2>&1 | tee c.txt but when I run make it ends with this error (actually, many of the same error): /home/kmccall/mpich-build-4.0a2/src/pm/hydra/.libs/libhydra.a(args.o): In function `MPL_gpu_query_pointer_attr': /home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: multiple definition of `MPL_gpu_query_pointer_attr' tools/bootstrap/persist/hydra_persist-persist_server.o:/home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: first defined here I've attached c.txt and m.txt from configure and make. Thanks for any help. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I'm exclusively connecting my processes with inter-communicators. I've found that if I increase the number of processes beyond a certain point, many processes don't get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors - it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager's own node. This is the largest job I've been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I'm using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Zhou, Hui 24 Oct '21

24 Oct '21

Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 24 Oct '21

24 Oct '21

Hi, Based on a paper I read about giving an MPI job some fault tolerance, I'm exclusively connecting my processes with inter-communicators. I've found that if I increase the number of processes beyond a certain point, many processes don't get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors - it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager's own node. This is the largest job I've been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I'm using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

HPC cluster network utilisation monitoring tool ?
by Nicholas Yue 19 Oct '21

19 Oct '21

Hi, I am mainly using MPICH via the mpiexec that ships with Paraview I have a small test cluster and a 1Gbit switch. What is the recommended way to determine and record the network utilization for a given MPI run? I was hoping to gather such information over time and be in a proactive position to plan for network equipment update should I find that I start running into a situation where my network is becoming the performance bottleneck. Cheers -- Nicholas Yue https://www.linkedin.com/in/nicholasyue/

1 0

ROMIO: add a new romio driver Oceanfs. Looking foward to receiving feedbacks
by xiafei (E) 29 Sep '21

29 Sep '21

Hi everyone, We implement a new romio driver that supports the OceanFs, which is a parallel file system developed by Huawei. The Oceanfs supports mpi/posix using a distributed parallel client(DPC). More info about the Oceanfs can be found in https://e.huawei.com/en/products/storage/distributed-storage/oceanstor-paci… Please help to review the pull request: https://github.com/pmodels/mpich/pull/5576 Looking forward to receiving feedbacks. Thanks. Best regards, Xia Fei

1 0

Re: [mpich-discuss] How to generate the configure script using the latest mpich?
by Zhou, Hui 28 Sep '21

28 Sep '21

Hi Xiafei, Run `git submodule update –init` to download all the submodules. -- Hui Zhou From: xiafei (E) via discuss <discuss(a)mpich.org> Date: Tuesday, September 28, 2021 at 9:39 AM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: xiafei (E) <xiafei6(a)huawei.com> Subject: [mpich-discuss] How to generate the configure script using the latest mpich? Hi everyone, I’m trying to compile mpich following the guide. [cid:[email protected]] [cid:[email protected]] However, the latest mpich (main branch) doesn’t contain a configure script. How to generate the generate the configure script before compiling mpich? When I execute the autogen.sh, it fails as the following figure shows. [cid:[email protected]] Thanks! XiaFei

1 0

How to generate the configure script using the latest mpich?
by xiafei (E) 28 Sep '21

28 Sep '21

Hi everyone, I'm trying to compile mpich following the guide. [cid:[email protected]] [cid:[email protected]] However, the latest mpich (main branch) doesn't contain a configure script. How to generate the generate the configure script before compiling mpich? When I execute the autogen.sh, it fails as the following figure shows. [cid:[email protected]] Thanks! XiaFei

1 0

Re: [mpich-discuss] Installing MPICH on clusters
by Zhou, Hui 22 Sep '21

22 Sep '21

Feimi Yu, The profiling library is a separate library that sits between MPI and user application so the system can gather statistics over applications MPI usage. The use of this profiling library is optional and typically transparent to users. It is probably is set by the system linker scripts or preloader scripts by your system. It is not surprising that this profiling library has a hard coded libmpi dependency set during its build, in your case, it is linked to the system MPI, which is the spectrum MPI. There usually is an option to not load this profiling library. Please consult your system admin for how to skip the profiling library. -- Hui Zhou -----Original Message----- From: Raffenetti, Kenneth J. <raffenet(a)mcs.anl.gov> Sent: Tuesday, September 21, 2021 11:33 AM To: discuss(a)mpich.org; Zhou, Hui <zhouh(a)anl.gov> Cc: Feimi Yu <yuf2(a)rpi.edu> Subject: Re: [mpich-discuss] Installing MPICH on clusters MPICH does not need to use Spectrum MPI's libraries. Are you sure your application isn't actually linked against Spectrum MPI and not MPICH? Ken On 9/20/21, 6:33 PM, "Feimi Yu via discuss" <discuss(a)mpich.org> wrote: Thank you for the hint! I finally got my job running on my own MPI build. Just out of curious: Actually, I searched this libmpiprofilesupport library before and discovered that it is specific to Spectrum MPI so I didn't go further. But MPICH just ran magically after I load Spectrum MPI and have the Spectrum MPI lib path in $LD_LIBRARY_PATH. Why does MPICH have to use Spectrum MPI's libraries? Thanks! Feimi On 9/19/2021 1:57 PM, Zhou, Hui wrote: > ./step-17.release: error while loading shared libraries: libmpiprofilesupport.so.3: cannot open shared object file: No such file or directory This is complaining about cannot finding a dynamic library (that is linked in your binary) on the compute node. Make sure the path to that library is in the LD_LIBRARY_PATH. ________________________________________ From: Feimi Yu <yuf2(a)rpi.edu> <mailto:[email protected]> Sent: Saturday, September 18, 2021 9:13 PM To: Zhou, Hui <zhouh(a)anl.gov> <mailto:[email protected]>; discuss(a)mpich.org <discuss(a)mpich.org> <mailto:[email protected]> Subject: Re: [mpich-discuss] Installing MPICH on clusters Hi Hui, Thank you for the response! Here is the Slurm batch file I used to run a program with MPICH configured with Hydra: #!/bin/bash #SBATCH --job-name=5e-7 #SBATCH --partition=el8 #SBATCH --time 6:00:00 #SBATCH --ntasks 40 #SBATCH --nodes 1 #SBATCH --gres=gpu:4 date export LD_LIBRARY_PATH=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/lib:$MPI_ROOT:$LD_LIBRARY_PATH srun --mpi=mpichmx hostname -s | sort -u > /tmp/hosts.$SLURM_JOB_ID awk "{ print \$0 \":40\"; }" /tmp/hosts.$SLURM_JOB_ID >/tmp/tmp.$SLURM_JOB_ID mv /tmp/tmp.$SLURM_JOB_ID ./hosts.$SLURM_JOB_ID /gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/mpich-build/bin/mpiexec -f ./hosts.$SLURM_JOB_ID -np $SLURM_NPROCS ./step-17.release date And the error message is: ./step-17.release: error while loading shared libraries: libmpiprofilesupport.so.3: cannot open shared object file: No such file or directory I was not sure if this is related to a network problem because the clusters use Infiniband. Running "/sbin/ifconfig" gives ib0, ib1, ib2 and ib3. I tried the option "-iface ib0" and the error message became: [mpiexec@dcs176] HYDU_sock_get_iface_ip (utils/sock/sock.c:451): unable to find interface ib0 [mpiexec@dcs176] HYDU_sock_create_and_listen_portstr (utils/sock/sock.c:496): unable to get network interface IP [mpiexec@dcs176] HYD_pmci_launch_procs (pm/pmiserv/pmiserv_pmci.c:79): unable to create PMI port [mpiexec@dcs176] main (ui/mpich/mpiexec.c:322): process manager returned error launching processes Specifying ib1-ib3 gives similar results. Thanks! Feimi On 9/17/21 7:57 PM, Zhou, Hui wrote: Hi Feimi, Hydra should be able to work with slurm. How are you launching the job and what is the failure message? -- Hui Zhou ________________________________________ From: Feimi Yu via discuss <discuss(a)mpich.org> <mailto:[email protected]> Sent: Friday, September 17, 2021 10:55 AM To: discuss(a)mpich.org <discuss(a)mpich.org> <mailto:[email protected]> Cc: Feimi Yu <yuf2(a)rpi.edu> <mailto:[email protected]> Subject: [mpich-discuss] Installing MPICH on clusters Hi, I'm working on a supercomputer which only provides Spectrum MPI implementation in modules. Since our code does not perform well with Spectrum MPI I decided to install an MPICH build on our own partition (I'm not an administrator.) The supercomputer has a rhel8 system on ppc64le architecture with Slurm as the process manager. I tried several building options according to the user guide but could not run a job so I have a few questions. Here are things I tried: 1. Build with Hydra PM. I could not launch a job with Hydra at all. 2. Then I decided to use ``--with-pm=none`` option to build and use srun + ``mpiexec -f hostfile`` to launch my job. But what confuses me is the PMI setting: srun --mpi=list gives following: srun: mpi/mpichgm srun: mpi/mpichmx srun: mpi/none srun: mpi/mvapich srun: mpi/openmpi srun: mpi/pmi2 srun: mpi/lam srun: mpi/mpich1_p4 srun: mpi/mpich1_shmem At first I tried use pmix since I found pmix libraries. But it didn't do the trick. It segfaults on PMPI_Init_thread(). The error message is: [dcs135:2312190] PMIX ERROR: NOT-FOUND in file client/pmix_client.c at line 562 Abort(1090831) on node 0 (rank 0 in comm 0): Fatal error in PMPI_Init_thread: Other MPI error, error stack: MPIR_Init_thread(159): MPID_Init(509).......: MPIR_pmi_init(92)....: PMIX_Init returned -46 [dcs135:2312190:0:2312190] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil)) Then I switched to pmi2 but make keeps telling me undefined reference to PMI2 library. (actually I couldn't find the pmi2 libraries either.) Then I used ``--with-pmi=slurm``, and it turned out that I couldn't locate the Slurm header files. I guess I don't have the permission to access them. I was wondering if it is still possible for me to build a usable MPICH as a user? If yes, how can I do to have the PMI work? Thanks! Feimi

1 0