October 2021 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] HPC cluster network utilisation monitoring tool ?
by Raffenetti, Ken 26 Oct '21

26 Oct '21

Apologies, I meant to link to the Nagios Core product, which is free, as an entry point to Nagios monitoring. https://www.nagios.com/products/nagios-core/ Ken On 10/26/21, 10:01 AM, "Raffenetti, Ken via discuss" <discuss(a)mpich.org> wrote: In a previous life, I used NAGIOS (https://www.nagios.com/solutions/network-monitoring/) to monitor all kinds of things on servers. It should be capable of telling you port-level network utilization. As for your switch, you might need to find something specific to the hardware. On the MPI side, profilers like mpiP (https://software.llnl.gov/mpiP/) can capture usage statistics and produce a report which you can look to for insights on how MPI is performing for your applications. Ken On 10/19/21, 12:52 PM, "Nicholas Yue via discuss" <discuss(a)mpich.org> wrote: Hi, I am mainly using MPICH via the mpiexec that ships with Paraview I have a small test cluster and a 1Gbit switch. What is the recommended way to determine and record the network utilization for a given MPI run? I was hoping to gather such information over time and be in a proactive position to plan for network equipment update should I find that I start running into a situation where my network is becoming the performance bottleneck. Cheers -- Nicholas Yue https://www.linkedin.com/in/nicholasyue/ _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] HPC cluster network utilisation monitoring tool ?
by Benson Muite 26 Oct '21

26 Oct '21

May also be of interest: https://github.com/octoshell/octoshell-v2 https://github.com/srcc-msu/job_statistics There are related papers on JobDigest and OctoShell Will probably want to determine what data to measure and keep, as many of the MPI performance monitoring tools may give significantly more information than you need. On 10/26/21 6:00 PM, Raffenetti, Ken via discuss wrote: > In a previous life, I used NAGIOS (https://www.nagios.com/solutions/network-monitoring/) to monitor all kinds of things on servers. It should be capable of telling you port-level network utilization. As for your switch, you might need to find something specific to the hardware. > > On the MPI side, profilers like mpiP (https://software.llnl.gov/mpiP/) can capture usage statistics and produce a report which you can look to for insights on how MPI is performing for your applications. > > Ken > > On 10/19/21, 12:52 PM, "Nicholas Yue via discuss" <discuss(a)mpich.org> wrote: > > Hi, > > I am mainly using MPICH via the mpiexec that ships with Paraview > > I have a small test cluster and a 1Gbit switch. > > What is the recommended way to determine and record the network utilization for a given MPI run? > > I was hoping to gather such information over time and be in a proactive position to plan for network equipment update should I find that I start running into a situation where my network is becoming the performance bottleneck. > > Cheers > -- > Nicholas Yue > https://www.linkedin.com/in/nicholasyue/ > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Re: [mpich-discuss] HPC cluster network utilisation monitoring tool ?
by Raffenetti, Ken 26 Oct '21

26 Oct '21

In a previous life, I used NAGIOS (https://www.nagios.com/solutions/network-monitoring/) to monitor all kinds of things on servers. It should be capable of telling you port-level network utilization. As for your switch, you might need to find something specific to the hardware. On the MPI side, profilers like mpiP (https://software.llnl.gov/mpiP/) can capture usage statistics and produce a report which you can look to for insights on how MPI is performing for your applications. Ken On 10/19/21, 12:52 PM, "Nicholas Yue via discuss" <discuss(a)mpich.org> wrote: Hi, I am mainly using MPICH via the mpiexec that ships with Paraview I have a small test cluster and a 1Gbit switch. What is the recommended way to determine and record the network utilization for a given MPI run? I was hoping to gather such information over time and be in a proactive position to plan for network equipment update should I find that I start running into a situation where my network is becoming the performance bottleneck. Cheers -- Nicholas Yue https://www.linkedin.com/in/nicholasyue/

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 26 Oct '21

26 Oct '21

Hui, Please disregard my last message. I got MPICH to build with at newer version of pgc++, namely nvc++. Thanks Kurt From: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Sent: Tuesday, October 26, 2021 2:55 AM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: Re: Maximum number of inter-communicators? Hui, I'm trying to build 4.0a2 with the Portland Group compiler pgc++ 19.5-0. configure seems to finish without problems. ../mpich-4.0a2/configure -prefix=/home/kmccall/mpich-install-4.0a2 CC=pgcc CXX=pgc++ --with-pbs=/opt/torque --with-device=ch3:nemesis --disable-fortran --with-pm=hydra -enable-debuginfo 2>&1 | tee c.txt but when I run make it ends with this error (actually, many of the same error): /home/kmccall/mpich-build-4.0a2/src/pm/hydra/.libs/libhydra.a(args.o): In function `MPL_gpu_query_pointer_attr': /home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: multiple definition of `MPL_gpu_query_pointer_attr' tools/bootstrap/persist/hydra_persist-persist_server.o:/home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: first defined here I've attached c.txt and m.txt from configure and make. Thanks for any help. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov<mailto:[email protected]>> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org<mailto:[email protected]> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I'm exclusively connecting my processes with inter-communicators. I've found that if I increase the number of processes beyond a certain point, many processes don't get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors - it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager's own node. This is the largest job I've been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I'm using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 26 Oct '21

26 Oct '21

Hui, I'm trying to build 4.0a2 with the Portland Group compiler pgc++ 19.5-0. configure seems to finish without problems. ../mpich-4.0a2/configure -prefix=/home/kmccall/mpich-install-4.0a2 CC=pgcc CXX=pgc++ --with-pbs=/opt/torque --with-device=ch3:nemesis --disable-fortran --with-pm=hydra -enable-debuginfo 2>&1 | tee c.txt but when I run make it ends with this error (actually, many of the same error): /home/kmccall/mpich-build-4.0a2/src/pm/hydra/.libs/libhydra.a(args.o): In function `MPL_gpu_query_pointer_attr': /home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: multiple definition of `MPL_gpu_query_pointer_attr' tools/bootstrap/persist/hydra_persist-persist_server.o:/home/kmccall/mpich-4.0a2/src/pm/hydra/mpl/include/mpl_gpu.h:44: first defined here I've attached c.txt and m.txt from configure and make. Thanks for any help. Thanks, Kurt From: Zhou, Hui <zhouh(a)anl.gov> Sent: Sunday, October 24, 2021 6:46 PM To: discuss(a)mpich.org Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [EXTERNAL] Re: Maximum number of inter-communicators? Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz<https://gcc02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.mpich.…>, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org<mailto:[email protected]>> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org<mailto:[email protected]> <discuss(a)mpich.org<mailto:[email protected]>> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov<mailto:[email protected]>> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I'm exclusively connecting my processes with inter-communicators. I've found that if I increase the number of processes beyond a certain point, many processes don't get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors - it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager's own node. This is the largest job I've been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I'm using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Re: [mpich-discuss] Maximum number of inter-communicators?
by Zhou, Hui 24 Oct '21

24 Oct '21

Hi Kurt, There is indeed a limit on maximum number of communicators that you can have, including both intra communicators and inter-communicators. Try free the communicators that you no longer need. In older version of MPICH, there may be additional limit on how many dynamic processes one can connect. If you still hit crash after making sure there isn't too many simultaneous active communicators, could you try the latest release -- http://www.mpich.org/static/downloads/4.0a2/mpich-4.0a2.tar.gz, and see if the issue persist? -- Hui ________________________________ From: Mccall, Kurt E. (MSFC-EV41) via discuss <discuss(a)mpich.org> Sent: Sunday, October 24, 2021 2:37 PM To: discuss(a)mpich.org <discuss(a)mpich.org> Cc: Mccall, Kurt E. (MSFC-EV41) <kurt.e.mccall(a)nasa.gov> Subject: [mpich-discuss] Maximum number of inter-communicators? Hi, Based on a paper I read about giving an MPI job some fault tolerance, I’m exclusively connecting my processes with inter-communicators. I’ve found that if I increase the number of processes beyond a certain point, many processes don’t get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors – it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager’s own node. This is the largest job I’ve been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I’m using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

Maximum number of inter-communicators?
by Mccall, Kurt E. (MSFC-EV41) 24 Oct '21

24 Oct '21

Hi, Based on a paper I read about giving an MPI job some fault tolerance, I'm exclusively connecting my processes with inter-communicators. I've found that if I increase the number of processes beyond a certain point, many processes don't get created at all and the whole job crashes. Am I running up against an operating system limit (like the number of open file descriptors - it is set at 1024), or some sort of MPICH limit? If it matters, my process architecture (a tree) is as follows: one master process connected to 21 manager processes on 21 other nodes, and each manager connected to 8 worker processes on the manager's own node. This is the largest job I've been able to create without it crashing. Attempting to increase the number of workers beyond 8 results in a crash. I'm using MPICH 3.3.2 on Centos 3.10.0. MPICH was compiled with the Portland Group compiler pgc++ 19.5-0. Thanks, Kurt

1 0

HPC cluster network utilisation monitoring tool ?
by Nicholas Yue 19 Oct '21

19 Oct '21

Hi, I am mainly using MPICH via the mpiexec that ships with Paraview I have a small test cluster and a 1Gbit switch. What is the recommended way to determine and record the network utilization for a given MPI run? I was hoping to gather such information over time and be in a proactive position to plan for network equipment update should I find that I start running into a situation where my network is becoming the performance bottleneck. Cheers -- Nicholas Yue https://www.linkedin.com/in/nicholasyue/

1 0