- discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Run mpich on localhost
by Tony Curtis 07 May '20

07 May '20

> On May 7, 2020, at 2:30 PM, Zhou, Hui via discuss <discuss(a)mpich.org> wrote: > > Hi Martin, > > Could you try the latest version https://www.mpich.org/downloads/ <https://www.mpich.org/downloads/> ? > > -- > Hui Zhou > > > From: Martin Ivanov via discuss <discuss(a)mpich.org> > Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> > Date: Thursday, May 7, 2020 at 11:35 AM > To: "discuss(a)mpich.org" <discuss(a)mpich.org> > Cc: Martin Ivanov <marto1980(a)gmail.com> > Subject: [mpich-discuss] Run mpich on localhost > > Hello, > I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on this single machine, which has 8 cores. When I run the cpi example, I get: > > marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi > ssh: Could not resolve hostname : hostname nor servname provided, or not known > Also, how did you configure MPICH? What is your hostname (uname -n), and what is in /etc/hosts? Tony

1 0

Re: [mpich-discuss] Run mpich on localhost
by Zhou, Hui 07 May '20

07 May '20

Hi Martin, Could you try the latest version https://www.mpich.org/downloads/ ? -- Hui Zhou From: Martin Ivanov via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, May 7, 2020 at 11:35 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Martin Ivanov <marto1980(a)gmail.com> Subject: [mpich-discuss] Run mpich on localhost Hello, I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on this single machine, which has 8 cores. When I run the cpi example, I get: marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi ssh: Could not resolve hostname : hostname nor servname provided, or not known If I make a hosts file with a single line content like this: marto@% cat hosts localhost:3 and again try the cpi example, I get: marto@% mpiexec -f hosts -n 3 mpich-3.0.4/examples/cpi [mpiexec@] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:517): assert (!closed) failed [mpiexec@] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Any suggestions will be appreciated. Best regards, Martin

1 0

Re: [mpich-discuss] MPICH configure
by Zhou, Hui 07 May '20

07 May '20

Hi Bruce, Thanks for the details. Since you already instructed `sbatch` with `-N 2 -n 6` I thought you could simply run with `srun`, i.e. without further `-n`? I believe hydra also reads environment so you could also try `mpiexec` without `-n`. Did you also try `-N 1` and `-N 6`? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, May 7, 2020 at 11:16 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, Sorry for the late reply, I keep getting pulled off on other projects. I’m actually running a test suite most of the time, so the job scripts look like #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list #make check-ga MPIEXEC="mpirun -n 6 " make check-ga MPIEXEC="srun -n 6 " I’ve tried using mpirun, srun and mpiexec in the MPIEXEC variable. If I run a test standalone, then the job submission script is #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list srun -n 6 test.x > test.out Again, I’ve tried running with mpirun, srun, and mpiexec. The environment in the set_mpich file is module purge module load gcc/5.2.0 module load python/2.7.8 module load cmake/3.8.2 module load git module load mkl setenv CC gcc setenv CFLAGS "-pthread" setenv CXX g++ setenv CXXFLAGS "-pthread" setenv FC gfortran setenv FCFLAGS "-pthread" unsetenv F90 unsetenv F90FLAGS setenv PATH /people/d3g293/mpich/mpich-3.3.2/install/bin:${PATH} setenv MANPATH /people/d3g293/mpich/mpich-3.3.2/install/share/man:${MANPATH} setenv LD_LIBRARY_PATH /people/d3g293/mpich/mpich-3.3.2/install/lib:${LD_LIBRARY_PATH} Bruce

1 0

Run mpich on localhost
by Martin Ivanov 07 May '20

07 May '20

Hello, I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on this single machine, which has 8 cores. When I run the cpi example, I get: marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi ssh: Could not resolve hostname : hostname nor servname provided, or not known If I make a hosts file with a single line content like this: marto@% cat hosts localhost:3 and again try the cpi example, I get: marto@% mpiexec -f hosts -n 3 mpich-3.0.4/examples/cpi [mpiexec@] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:517): assert (!closed) failed [mpiexec@] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Any suggestions will be appreciated. Best regards, Martin

1 0

Re: [mpich-discuss] MPICH configure
by Palmer, Bruce J 07 May '20

07 May '20

Hi Hui, Sorry for the late reply, I keep getting pulled off on other projects. I’m actually running a test suite most of the time, so the job scripts look like #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list #make check-ga MPIEXEC="mpirun -n 6 " make check-ga MPIEXEC="srun -n 6 " I’ve tried using mpirun, srun and mpiexec in the MPIEXEC variable. If I run a test standalone, then the job submission script is #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list srun -n 6 test.x > test.out Again, I’ve tried running with mpirun, srun, and mpiexec. The environment in the set_mpich file is module purge module load gcc/5.2.0 module load python/2.7.8 module load cmake/3.8.2 module load git module load mkl setenv CC gcc setenv CFLAGS "-pthread" setenv CXX g++ setenv CXXFLAGS "-pthread" setenv FC gfortran setenv FCFLAGS "-pthread" unsetenv F90 unsetenv F90FLAGS setenv PATH /people/d3g293/mpich/mpich-3.3.2/install/bin:${PATH} setenv MANPATH /people/d3g293/mpich/mpich-3.3.2/install/share/man:${MANPATH} setenv LD_LIBRARY_PATH /people/d3g293/mpich/mpich-3.3.2/install/lib:${LD_LIBRARY_PATH} Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Thursday, April 30, 2020 at 11:25 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Could you share your job scripts with us? It’ll be helpful to understand how you exactly launches jobs. -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, April 30, 2020 at 9:25 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I checked with the system admins and they say there are no firewall restrictions between nodes. According to them they can talk to any port on any machine. I’ve also followed up on their suggestion that I link with PMI and verified with ldd that the PMI libraries are showing up before MPI in the executables. I still only get 1 process when running with srun. This is a summary of what I am seeing when I run on 2 nodes (after configuring and building with slurm). MPICH-3.3.1 Launch with mpiexec: runs okay Launch with mpirun: runs okay Link with PMI and Launch with srun: only get 1 process (from MPI_Comm_size on MPI_COMM_WORLD) MPICH-3.3.2 Launch with mpiexec: hangs Launch with mpirun: hangs Link with PMI and Launch with srun: only get 1 process (from MPI_Comm_size on MPI_COMM_WORLD) For what it’s worth, it looks like the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!) doesn’t show up immediately. It looks like it appears (if it appears) after the system has been hung up for a while. Bruce

1 0

Re: [mpich-discuss] Better alternative to MPI_Allreduce() and avoiding deadlock with MPI_Neighbor_alltoallw().
by Benson Muite 06 May '20

06 May '20

On Tue, May 5, 2020, at 10:19 PM, hritikesh semwal via discuss wrote: > > > On Tue, 5 May, 2020, 10:30 PM , <discuss-request(a)mpich.org> wrote: >> >> > 1. I am using MPI_Neighbor_alltoallw() for exchanging the data by generating a distributed graph topology communicator. My concern is that most of the time my code is working fine but sometimes I guess it is going into deadlock (as it is not showing any output). But MPI_Neighbor_alltoallw uses MPI_Waitall inside it so I am not getting why exactly this is happening. >> >> >> >> May want to check sending and receiving correct data. Perhaps also try MPI_Neighbor_alltoallw >> >> >> >> > 2. Is it possible that every time I run the code the processors times for completion of the task may vary? For example, for one run it all processors takes around 100 seconds and for another run, all processors take 110 seconds. >> >> >> >> There is usually some variability. Do you solve the same system each time? What is the method of solution? If your code is available it can sometimes be easier to give suggestions. >> >> >> > Yes, the system of equations are the same. I am using the finite volume method for solving Navier stokes equations. By first sentence you mean to say it is possible. >> >> Is the method implicit or explicit? > >> > Its an explicit method. Ok >> >> > >> >> > >> >> > Please help in above two matters. >> >> > >> >> > On Tue, May 5, 2020 at 4:28 PM hritikesh semwal <hritikesh.semwal(a)gmail.com> wrote: >> >> >> Thanks for your response. >> >> >> >> >> >> Yes, you are right. I have put barrier just before Allreduce and out of the total time consumed by Allreduce, 79% time is consumed by the barrier. But my computational work is balanced. Right now, I have distributed 97336 cells among 24 processors and maximum and minimum cell distribution among all processors is 4057 and 4055 respectively which is not too bad. Is there any solution to get rid of this. >> >> >> >> Try profiling your code not just looking at cell distribution. Are any profling tools already installed on your cluster? >> > >> > gprof and valgrind are there. >> >> While not ideal GPROF may be helpful. Perhaps initial try running on 12 processors. With GPROF you will get 12 files to examine. Check if all subroutines take similar times on each processor. You can also time the subroutines individually using MPI_WTIME to get the same information. > >> > Yes, I have already timed my code before posting this question. I will try with gprof. > Great. Some documentation on Gprof: http://shwina.github.io/2014/11/profiling-parallel https://cluster.earlham.edu/wiki/index.php/Cluster:Gprof https://portal.tacc.utexas.edu/documents/13601/1041435/29-Overview_of_Profi… https://hpc.llnl.gov/software/development-environment-software/gprof https://support.pawsey.org.au/documentation/display/US/Profiling+with+gprof https://stackoverflow.com/questions/39041871/missing-function-from-gprof-ou… >> >> Also, try not to reply to the digest -, or if you do, change the subject of the message. This is useful in deciding what to read. > >> > Is it fine this time? I have changed the subject line. Is that what you want to say? That is ok if you cannot reply to the message directly.

1 0

Better alternative to MPI_Allreduce() and avoiding deadlock with MPI_Neighbor_alltoallw().
by hritikesh semwal 05 May '20

05 May '20

On Tue, 5 May, 2020, 10:30 PM , <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. Better alternatives of MPI_Allreduce() (Benson Muite) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 05 May 2020 15:17:08 +0300 > From: "Benson Muite" <benson_muite(a)emailplus.org> > To: "Benson Muite via discuss" <discuss(a)mpich.org> > Subject: [mpich-discuss] Better alternatives of MPI_Allreduce() > Message-ID: <ef00483f-25a7-48d8-a04c-964e8001def7(a)www.fastmail.com> > Content-Type: text/plain; charset="us-ascii" > > > 1. I am using MPI_Neighbor_alltoallw() for exchanging the data by > generating a distributed graph topology communicator. My concern is that > most of the time my code is working fine but sometimes I guess it is going > into deadlock (as it is not showing any output). But MPI_Neighbor_alltoallw > uses MPI_Waitall inside it so I am not getting why exactly this is > happening. > >> > >> May want to check sending and receiving correct data. Perhaps also try > MPI_Neighbor_alltoallw > >> > >> > 2. Is it possible that every time I run the code the processors > times for completion of the task may vary? For example, for one run it all > processors takes around 100 seconds and for another run, all processors > take 110 seconds. > >> > >> There is usually some variability. Do you solve the same system each > time? What is the method of solution? If your code is available it can > sometimes be easier to give suggestions. > >> > > Yes, the system of equations are the same. I am using the finite volume > method for solving Navier stokes equations. By first sentence you mean to > say it is possible. > > Is the method implicit or explicit? > Its an explicit method. > > > > >> > > >> > Please help in above two matters. > >> > > >> > On Tue, May 5, 2020 at 4:28 PM hritikesh semwal < > hritikesh.semwal(a)gmail.com> wrote: > >> >> Thanks for your response. > >> >> > >> >> Yes, you are right. I have put barrier just before Allreduce and > out of the total time consumed by Allreduce, 79% time is consumed by the > barrier. But my computational work is balanced. Right now, I have > distributed 97336 cells among 24 processors and maximum and minimum cell > distribution among all processors is 4057 and 4055 respectively which is > not too bad. Is there any solution to get rid of this. > >> > >> Try profiling your code not just looking at cell distribution. Are any > profling tools already installed on your cluster? > > > > gprof and valgrind are there. > > While not ideal GPROF may be helpful. Perhaps initial try running on 12 > processors. With GPROF you will get 12 files to examine. Check if all > subroutines take similar times on each processor. You can also time the > subroutines individually using MPI_WTIME to get the same information. > Yes, I have already timed my code before posting this question. I will try with gprof. > > Also, try not to reply to the digest -, or if you do, change the subject > of the message. This is useful in deciding what to read. > Is it fine this time? I have changed the subject line. Is that what you want to say?

1 0

Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

> 1. I am using MPI_Neighbor_alltoallw() for exchanging the data by generating a distributed graph topology communicator. My concern is that most of the time my code is working fine but sometimes I guess it is going into deadlock (as it is not showing any output). But MPI_Neighbor_alltoallw uses MPI_Waitall inside it so I am not getting why exactly this is happening. >> >> May want to check sending and receiving correct data. Perhaps also try MPI_Neighbor_alltoallw >> >> > 2. Is it possible that every time I run the code the processors times for completion of the task may vary? For example, for one run it all processors takes around 100 seconds and for another run, all processors take 110 seconds. >> >> There is usually some variability. Do you solve the same system each time? What is the method of solution? If your code is available it can sometimes be easier to give suggestions. >> > Yes, the system of equations are the same. I am using the finite volume method for solving Navier stokes equations. By first sentence you mean to say it is possible. Is the method implicit or explicit? > >> > >> > Please help in above two matters. >> > >> > On Tue, May 5, 2020 at 4:28 PM hritikesh semwal <hritikesh.semwal(a)gmail.com> wrote: >> >> Thanks for your response. >> >> >> >> Yes, you are right. I have put barrier just before Allreduce and out of the total time consumed by Allreduce, 79% time is consumed by the barrier. But my computational work is balanced. Right now, I have distributed 97336 cells among 24 processors and maximum and minimum cell distribution among all processors is 4057 and 4055 respectively which is not too bad. Is there any solution to get rid of this. >> >> Try profiling your code not just looking at cell distribution. Are any profling tools already installed on your cluster? > > gprof and valgrind are there. While not ideal GPROF may be helpful. Perhaps initial try running on 12 processors. With GPROF you will get 12 files to examine. Check if all subroutines take similar times on each processor. You can also time the subroutines individually using MPI_WTIME to get the same information. Also, try not to reply to the digest -, or if you do, change the subject of the message. This is useful in deciding what to read.

1 0

Re: [mpich-discuss] discuss Digest, Vol 91, Issue 5
by hritikesh semwal 05 May '20

05 May '20

On Tue, May 5, 2020 at 5:17 PM <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. Better alternatives of MPI_Allreduce() (Benson Muite) > 2. Re: Better alternatives of MPI_Allreduce() (Benson Muite) > 3. Re: Better alternatives of MPI_Allreduce() (hritikesh semwal) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 05 May 2020 14:20:57 +0300 > From: "Benson Muite" <benson_muite(a)emailplus.org> > To: "Benson Muite via discuss" <discuss(a)mpich.org> > Cc: hritikesh.semwal(a)gmail.com > Subject: [mpich-discuss] Better alternatives of MPI_Allreduce() > Message-ID: <a38d489f-25a6-40a1-9640-42bd2f52a80f(a)www.fastmail.com> > Content-Type: text/plain; charset="us-ascii" > > > >> > > >> > Hi Hitesh, > >> > > >> > What hardware are you running on and what is the interconnect? > > > > Right now I am using a cluster. > > What is the interconnect? > > > >> > Have you tried changing any of the MPI settings? > > > > What do you mean by MPI settings? > Given your comment on the barrier, this is probably not so useful at the > moment. > > > >> > Can the reduction be done asynchronously? > > > > I did not get your question. > > For example using a non blocking all reduce: > https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node135.htm > > > > >> > > >> > Regards, > >> > Benson > >> > >> Also, is your work load balanced? One way to check this might be to > place a barrier just before the all-reduce call. If the barrier ends up > taking most of your time, then it is likely you will need to determine a > better way to distribute the computational work. > > > > Thanks for your response. > > > > Yes, you are right. I have put barrier just before Allreduce and out of > the total time consumed by Allreduce, 79% time is consumed by the barrier. > But my computational work is balanced. Right now, I have distributed 97336 > cells among 24 processors and maximum and minimum cell distribution among > all processors is 4057 and 4055 respectively which is not too bad. Is there > any solution to get rid of this. >

1 0

Re: [mpich-discuss] Better alternatives of MPI_Allreduce()
by Benson Muite 05 May '20

05 May '20

On Tue, May 5, 2020, at 2:46 PM, hritikesh semwal wrote: > > > On Tue, May 5, 2020 at 4:51 PM Benson Muite <benson_muite(a)emailplus.org> wrote: >> __ >> >>>> > >>>> > Hi Hitesh, >>>> > >>>> > What hardware are you running on and what is the interconnect? >>> >>> Right now I am using a cluster. >> >> What is the interconnect? > > I don't know about this. Is it relevant? It can affect performance - but expect it may not be the most important factor on 24 processors. Most common is Infiniband (https://en.wikipedia.org/wiki/InfiniBand) > >>> >>>> > Have you tried changing any of the MPI settings? >>> >>> What do you mean by MPI settings? >> Given your comment on the barrier, this is probably not so useful at the moment. >>> >>>> > Can the reduction be done asynchronously? >>> >>> I did not get your question. >> >> For example using a non blocking all reduce: >> https://www.mpi-forum.org/docs/mpi-3.1/mpi31-report/node135.htm >> > > I tried using a non-blocking call but after this code is not working correctly. Ok. Change back to blocking call. It is likely you have poor load balancing. >> >>> >>>> > >>>> > Regards, >>>> > Benson >>>> >>>> Also, is your work load balanced? One way to check this might be to place a barrier just before the all-reduce call. If the barrier ends up taking most of your time, then it is likely you will need to determine a better way to distribute the computational work. >>> >>> Thanks for your response. >>> >>> Yes, you are right. I have put barrier just before Allreduce and out of the total time consumed by Allreduce, 79% time is consumed by the barrier. But my computational work is balanced. Right now, I have distributed 97336 cells among 24 processors and maximum and minimum cell distribution among all processors is 4057 and 4055 respectively which is not too bad. Is there any solution to get rid of this? > Please help me in this regard. If you cannot profile your code, time the section before the all reduce on each processor using MPI_WTIME and check if it is even across all 24 processors. If using more processors, you will likely want to use a profiling tool, but if expect to run on about 24 processors, setting up a profiling tool if not already available may take some time.

1 0