May 2020 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] Run mpich on localhost
by Tony Curtis 08 May '20

08 May '20

> On May 8, 2020, at 1:41 AM, Martin Ivanov <marto1980(a)gmail.com> wrote: > > Hello, > Thank you for your replies. I think another version of MPICH may not work as I am following a tutorial (https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php <https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php>) that requires specific library versions. > My /etc/hosts file contains only these two lines: > > " > ::1 localhost localhost.my.domain > 127.0.0.1 localhost localhost.my.domain > " > > uname -n returns an empty string as I have not set a hostname. I guess I do not need this because I need to run MPICH only on this machine, that is on localhost? uname -a gives the following output: > Thing is, though, your initial post showed ssh being used as launcher. So in that case you would definitely need a hostname so you can resolve back to yourself (your shell prompt was a big clue, by the way). Setting a hostname and making it an alias of localhost in /etc/hosts would probably do wonders no matter the underlying launch. The -info output is pretty much the same as I see on FreeBSD. If you run a hello world with a sleep in it to allow you to react, what does “ps x” show? (Not familiar much with DragonFly beyond “poking at it” so I don’t have any particular insight there.) Tony

1 0

Re: [mpich-discuss] Run mpich on localhost
by Martin Ivanov 08 May '20

08 May '20

Hello, sorry for cross-posting, here is some more info: " marto@% mpiexec -info HYDRA build details: Version: 3.0.4 Release Date: Wed Apr 24 10:08:10 CDT 2013 CC: gcc -I/home/marto/WRF/Build_WRF/LIBRARIES/grib2/include -L/home/marto/WRF/Build_WRF/LIBRARIES/grib2/lib CXX: g++ -I/home/marto/WRF/Build_WRF/LIBRARIES/grib2/include -L/home/marto/WRF/Build_WRF/LIBRARIES/grib2/lib F77: gfortran -m64 -L/home/marto/WRF/Build_WRF/LIBRARIES/grib2/lib F90: gfortran -m64 -L/home/marto/WRF/Build_WRF/LIBRARIES/grib2/lib Configure options: '--disable-option-checking' '--prefix=/home/marto/WRF/Build_WRF/LIBRARIES/mpich' 'CC=gcc' 'LDFLAGS=-L/home/marto/WRF/Build_WRF/LIBRARIES/grib2/lib ' 'CPPFLAGS=-I/home/marto/WRF/Build_WRF/LIBRARIES/grib2/include -I/home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/mpl/include -I/home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/mpl/include -I/home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/openpa/src -I/home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/openpa/src -I/home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/mpi/romio/include' 'CXX=g++' 'F77=gfortran' 'FFLAGS=-m64 -O2' 'FC=gfortran' 'FCFLAGS=-m64 -O2' '--cache-file=/dev/null' '--srcdir=.' 'CFLAGS= -O2' 'LIBS=-lpthread ' Process Manager: pmi Launchers available: ssh rsh fork slurm ll lsf sge manual persist Topology libraries available: hwloc Resource management kernels available: user slurm ll lsf sge pbs cobalt Checkpointing libraries available: Demux engines available: poll select " Best regards, Martin On Fri, May 8, 2020 at 7:41 AM Martin Ivanov <marto1980(a)gmail.com> wrote: > Hello, > Thank you for your replies. I think another version of MPICH may not work > as I am following a tutorial ( > https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php) > that requires specific library versions. > My /etc/hosts file contains only these two lines: > > " > ::1 localhost localhost.my.domain > 127.0.0.1 localhost localhost.my.domain > " > > uname -n returns an empty string as I have not set a hostname. I guess I > do not need this because I need to run MPICH only on this machine, that is > on localhost? uname -a gives the following output: > > " > marto@% uname -a > DragonFly 5.8-RELEASE DragonFly v5.8.1-RELEASE #1: Thu May 7 08:26:21 > CEST 2020 root@:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64 > " > > Apart from providing a --prefix, I configured MPICH with the default > options. During the configure step I got some interesting messages though: > > " > checking for architecture... x86_64 > configure: WARNING: Unknown architecture x86_64... proceeding anyway > ... > ### > ### Configuring hwloc core > ### > checking hwloc building mode... embedded > configure: hwloc builddir: > /home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/pm/hydra/tools/topo/hwloc/hwloc > configure: hwloc srcdir: > /home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/pm/hydra/tools/topo/hwloc/hwloc > checking if want hwloc maintainer support... disabled (embedded mode) > checking for hwloc directory prefix... tools/topo/hwloc/hwloc/ > checking for hwloc symbol prefix... hwloc_ > checking size of void *... 8 > checking which OS support to include... Unsupported! > (x86_64-unknown-dragonfly5.8) > configure: WARNING: > *********************************************************** > configure: WARNING: *** hwloc does not support this system. > configure: WARNING: *** hwloc will *attempt* to build (but it may not > work). > configure: WARNING: *** hwloc run-time results may be reduced to showing > just one processor. > configure: WARNING: *** You have been warned. > configure: WARNING: *** Pausing to give you time to read this message... > configure: WARNING: > *********************************************************** > checking which CPU support to include... x86_64 > ... > configure: creating ./config.status > config.status: creating tools/topo/hwloc/hwloc/Makefile > config.status: creating tools/topo/hwloc/hwloc/include/Makefile > config.status: creating tools/topo/hwloc/hwloc/src/Makefile > config.status: creating Makefile > config.status: creating tools/bootstrap/src/bsci_init.c > config.status: creating hydra-doxygen.cfg > config.status: creating include/hydra_config.h > config.status: include/hydra_config.h is unchanged > config.status: creating > tools/topo/hwloc/hwloc/include/private/autogen/config.h > config.status: tools/topo/hwloc/hwloc/include/private/autogen/config.h is > unchanged > config.status: creating > tools/topo/hwloc/hwloc/include/hwloc/autogen/config.h > config.status: tools/topo/hwloc/hwloc/include/hwloc/autogen/config.h is > unchanged > " > I am looking forward to your replies. > > Best regards, > Martin > > On Thu, May 7, 2020 at 8:59 PM Tony Curtis via discuss <discuss(a)mpich.org> > wrote: > >> >> >> On May 7, 2020, at 2:30 PM, Zhou, Hui via discuss <discuss(a)mpich.org> >> wrote: >> >> Hi Martin, >> >> Could you try the latest version https://www.mpich.org/downloads/ ? >> >> -- >> Hui Zhou >> >> >> *From: *Martin Ivanov via discuss <discuss(a)mpich.org> >> *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> >> *Date: *Thursday, May 7, 2020 at 11:35 AM >> *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> >> *Cc: *Martin Ivanov <marto1980(a)gmail.com> >> *Subject: *[mpich-discuss] Run mpich on localhost >> >> Hello, >> I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on >> this single machine, which has 8 cores. When I run the cpi example, I get: >> >> marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi >> ssh: Could not resolve hostname : hostname nor servname provided, or not >> known >> >> >> >> Also, how did you configure MPICH? What is your hostname (uname -n), and >> what is in /etc/hosts? >> >> Tony >> >> _______________________________________________ >> discuss mailing list discuss(a)mpich.org >> To manage subscription options or unsubscribe: >> https://lists.mpich.org/mailman/listinfo/discuss >> >

1 0

Re: [mpich-discuss] Run mpich on localhost
by Martin Ivanov 08 May '20

08 May '20

Hello, Thank you for your replies. I think another version of MPICH may not work as I am following a tutorial ( https://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php) that requires specific library versions. My /etc/hosts file contains only these two lines: " ::1 localhost localhost.my.domain 127.0.0.1 localhost localhost.my.domain " uname -n returns an empty string as I have not set a hostname. I guess I do not need this because I need to run MPICH only on this machine, that is on localhost? uname -a gives the following output: " marto@% uname -a DragonFly 5.8-RELEASE DragonFly v5.8.1-RELEASE #1: Thu May 7 08:26:21 CEST 2020 root@:/usr/obj/usr/src/sys/X86_64_GENERIC x86_64 " Apart from providing a --prefix, I configured MPICH with the default options. During the configure step I got some interesting messages though: " checking for architecture... x86_64 configure: WARNING: Unknown architecture x86_64... proceeding anyway ... ### ### Configuring hwloc core ### checking hwloc building mode... embedded configure: hwloc builddir: /home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/pm/hydra/tools/topo/hwloc/hwloc configure: hwloc srcdir: /home/marto/WRF/Build_WRF/LIBRARIES/mpich-3.0.4/src/pm/hydra/tools/topo/hwloc/hwloc checking if want hwloc maintainer support... disabled (embedded mode) checking for hwloc directory prefix... tools/topo/hwloc/hwloc/ checking for hwloc symbol prefix... hwloc_ checking size of void *... 8 checking which OS support to include... Unsupported! (x86_64-unknown-dragonfly5.8) configure: WARNING: *********************************************************** configure: WARNING: *** hwloc does not support this system. configure: WARNING: *** hwloc will *attempt* to build (but it may not work). configure: WARNING: *** hwloc run-time results may be reduced to showing just one processor. configure: WARNING: *** You have been warned. configure: WARNING: *** Pausing to give you time to read this message... configure: WARNING: *********************************************************** checking which CPU support to include... x86_64 ... configure: creating ./config.status config.status: creating tools/topo/hwloc/hwloc/Makefile config.status: creating tools/topo/hwloc/hwloc/include/Makefile config.status: creating tools/topo/hwloc/hwloc/src/Makefile config.status: creating Makefile config.status: creating tools/bootstrap/src/bsci_init.c config.status: creating hydra-doxygen.cfg config.status: creating include/hydra_config.h config.status: include/hydra_config.h is unchanged config.status: creating tools/topo/hwloc/hwloc/include/private/autogen/config.h config.status: tools/topo/hwloc/hwloc/include/private/autogen/config.h is unchanged config.status: creating tools/topo/hwloc/hwloc/include/hwloc/autogen/config.h config.status: tools/topo/hwloc/hwloc/include/hwloc/autogen/config.h is unchanged " I am looking forward to your replies. Best regards, Martin On Thu, May 7, 2020 at 8:59 PM Tony Curtis via discuss <discuss(a)mpich.org> wrote: > > > On May 7, 2020, at 2:30 PM, Zhou, Hui via discuss <discuss(a)mpich.org> > wrote: > > Hi Martin, > > Could you try the latest version https://www.mpich.org/downloads/ ? > > -- > Hui Zhou > > > *From: *Martin Ivanov via discuss <discuss(a)mpich.org> > *Reply-To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Date: *Thursday, May 7, 2020 at 11:35 AM > *To: *"discuss(a)mpich.org" <discuss(a)mpich.org> > *Cc: *Martin Ivanov <marto1980(a)gmail.com> > *Subject: *[mpich-discuss] Run mpich on localhost > > Hello, > I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on this > single machine, which has 8 cores. When I run the cpi example, I get: > > marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi > ssh: Could not resolve hostname : hostname nor servname provided, or not > known > > > > Also, how did you configure MPICH? What is your hostname (uname -n), and > what is in /etc/hosts? > > Tony > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss >

1 0

Speed up issue with CFD solver.
by hritikesh semwal 07 May '20

07 May '20

Hello, I am developing a parallel CFD solver for unstructured grids with finite volume explicit time integration method and the global mesh count is 97336 cells. I am trying to get the speed up results of my code but something strange is happening. Following are my timing results after the second time step with the different number of processors; 2 processors- 182.284 seconds 4 processors- 251.7035 seconds 8 processors- 190.8578 seconds 16 processors- 129.7989 seconds 24 processors- 121.0303 seconds 32 processors- 111.3772 seconds I am not getting why there is a sharp rise in time from 2 processors to 4 processors and after that, it is decreasing. I had run the same code 2 weeks ago and at that time the results were different from the present results and time was continuously decreasing at that time. Could someone please help me with this matter by suggesting the probable cause/s of this behaviour? Thank you.

1 0

Re: [mpich-discuss] Run mpich on localhost
by Tony Curtis 07 May '20

07 May '20

> On May 7, 2020, at 2:30 PM, Zhou, Hui via discuss <discuss(a)mpich.org> wrote: > > Hi Martin, > > Could you try the latest version https://www.mpich.org/downloads/ <https://www.mpich.org/downloads/> ? > > -- > Hui Zhou > > > From: Martin Ivanov via discuss <discuss(a)mpich.org> > Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> > Date: Thursday, May 7, 2020 at 11:35 AM > To: "discuss(a)mpich.org" <discuss(a)mpich.org> > Cc: Martin Ivanov <marto1980(a)gmail.com> > Subject: [mpich-discuss] Run mpich on localhost > > Hello, > I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on this single machine, which has 8 cores. When I run the cpi example, I get: > > marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi > ssh: Could not resolve hostname : hostname nor servname provided, or not known > Also, how did you configure MPICH? What is your hostname (uname -n), and what is in /etc/hosts? Tony

1 0

Re: [mpich-discuss] Run mpich on localhost
by Zhou, Hui 07 May '20

07 May '20

Hi Martin, Could you try the latest version https://www.mpich.org/downloads/ ? -- Hui Zhou From: Martin Ivanov via discuss <discuss(a)mpich.org> Reply-To: "discuss(a)mpich.org" <discuss(a)mpich.org> Date: Thursday, May 7, 2020 at 11:35 AM To: "discuss(a)mpich.org" <discuss(a)mpich.org> Cc: Martin Ivanov <marto1980(a)gmail.com> Subject: [mpich-discuss] Run mpich on localhost Hello, I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on this single machine, which has 8 cores. When I run the cpi example, I get: marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi ssh: Could not resolve hostname : hostname nor servname provided, or not known If I make a hosts file with a single line content like this: marto@% cat hosts localhost:3 and again try the cpi example, I get: marto@% mpiexec -f hosts -n 3 mpich-3.0.4/examples/cpi [mpiexec@] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:517): assert (!closed) failed [mpiexec@] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Any suggestions will be appreciated. Best regards, Martin

1 0

Re: [mpich-discuss] MPICH configure
by Zhou, Hui 07 May '20

07 May '20

Hi Bruce, Thanks for the details. Since you already instructed `sbatch` with `-N 2 -n 6` I thought you could simply run with `srun`, i.e. without further `-n`? I believe hydra also reads environment so you could also try `mpiexec` without `-n`. Did you also try `-N 1` and `-N 6`? -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, May 7, 2020 at 11:16 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, Sorry for the late reply, I keep getting pulled off on other projects. I’m actually running a test suite most of the time, so the job scripts look like #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list #make check-ga MPIEXEC="mpirun -n 6 " make check-ga MPIEXEC="srun -n 6 " I’ve tried using mpirun, srun and mpiexec in the MPIEXEC variable. If I run a test standalone, then the job submission script is #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list srun -n 6 test.x > test.out Again, I’ve tried running with mpirun, srun, and mpiexec. The environment in the set_mpich file is module purge module load gcc/5.2.0 module load python/2.7.8 module load cmake/3.8.2 module load git module load mkl setenv CC gcc setenv CFLAGS "-pthread" setenv CXX g++ setenv CXXFLAGS "-pthread" setenv FC gfortran setenv FCFLAGS "-pthread" unsetenv F90 unsetenv F90FLAGS setenv PATH /people/d3g293/mpich/mpich-3.3.2/install/bin:${PATH} setenv MANPATH /people/d3g293/mpich/mpich-3.3.2/install/share/man:${MANPATH} setenv LD_LIBRARY_PATH /people/d3g293/mpich/mpich-3.3.2/install/lib:${LD_LIBRARY_PATH} Bruce

1 0

Run mpich on localhost
by Martin Ivanov 07 May '20

07 May '20

Hello, I installed mpich-3.0.4 on a x86_64 BSD machine. I need to run MPI on this single machine, which has 8 cores. When I run the cpi example, I get: marto@% mpiexec -n 3 mpich-3.0.4/examples/cpi ssh: Could not resolve hostname : hostname nor servname provided, or not known If I make a hosts file with a single line content like this: marto@% cat hosts localhost:3 and again try the cpi example, I get: marto@% mpiexec -f hosts -n 3 mpich-3.0.4/examples/cpi [mpiexec@] HYD_pmcd_pmiserv_proxy_init_cb (./pm/pmiserv/pmiserv_cb.c:517): assert (!closed) failed [mpiexec@] HYDT_dmxu_poll_wait_for_event (./tools/demux/demux_poll.c:77): callback returned error status [mpiexec@] HYD_pmci_wait_for_completion (./pm/pmiserv/pmiserv_pmci.c:197): error waiting for event [mpiexec@] main (./ui/mpich/mpiexec.c:331): process manager error waiting for completion Any suggestions will be appreciated. Best regards, Martin

1 0

Re: [mpich-discuss] MPICH configure
by Palmer, Bruce J 07 May '20

07 May '20

Hi Hui, Sorry for the late reply, I keep getting pulled off on other projects. I’m actually running a test suite most of the time, so the job scripts look like #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list #make check-ga MPIEXEC="mpirun -n 6 " make check-ga MPIEXEC="srun -n 6 " I’ve tried using mpirun, srun and mpiexec in the MPIEXEC variable. If I run a test standalone, then the job submission script is #!/bin/csh #SBATCH -t 02:30:00 #SBATCH -A XGA #SBATCH -p short,slurm,gpu #SBATCH -N 2 #SBATCH -n 6 #SBATCH -o ./test.out #SBATCH -e ./test.err source /etc/profile.d/modules.csh source ~/set_mpich env | grep PATH module list srun -n 6 test.x > test.out Again, I’ve tried running with mpirun, srun, and mpiexec. The environment in the set_mpich file is module purge module load gcc/5.2.0 module load python/2.7.8 module load cmake/3.8.2 module load git module load mkl setenv CC gcc setenv CFLAGS "-pthread" setenv CXX g++ setenv CXXFLAGS "-pthread" setenv FC gfortran setenv FCFLAGS "-pthread" unsetenv F90 unsetenv F90FLAGS setenv PATH /people/d3g293/mpich/mpich-3.3.2/install/bin:${PATH} setenv MANPATH /people/d3g293/mpich/mpich-3.3.2/install/share/man:${MANPATH} setenv LD_LIBRARY_PATH /people/d3g293/mpich/mpich-3.3.2/install/lib:${LD_LIBRARY_PATH} Bruce From: "Zhou, Hui" <zhouh(a)anl.gov> Date: Thursday, April 30, 2020 at 11:25 AM To: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Bruce, Could you share your job scripts with us? It’ll be helpful to understand how you exactly launches jobs. -- Hui Zhou From: "Palmer, Bruce J" <Bruce.Palmer(a)pnnl.gov> Date: Thursday, April 30, 2020 at 9:25 AM To: "Zhou, Hui" <zhouh(a)anl.gov>, "discuss(a)mpich.org" <discuss(a)mpich.org> Subject: Re: [mpich-discuss] MPICH configure Hi Hui, I checked with the system admins and they say there are no firewall restrictions between nodes. According to them they can talk to any port on any machine. I’ve also followed up on their suggestion that I link with PMI and verified with ldd that the PMI libraries are showing up before MPI in the executables. I still only get 1 process when running with srun. This is a summary of what I am seeing when I run on 2 nodes (after configuring and building with slurm). MPICH-3.3.1 Launch with mpiexec: runs okay Launch with mpirun: runs okay Link with PMI and Launch with srun: only get 1 process (from MPI_Comm_size on MPI_COMM_WORLD) MPICH-3.3.2 Launch with mpiexec: hangs Launch with mpirun: hangs Link with PMI and Launch with srun: only get 1 process (from MPI_Comm_size on MPI_COMM_WORLD) For what it’s worth, it looks like the error message [proxy:0:[email protected]] HYDU_sock_connect (utils/sock/sock.c:145): unable to connect from "node168.local" to "node100.local" (Connection refused) [proxy:0:[email protected]] main (pm/pmiserv/pmip.c:183): unable to connect to server node100.local at port 54762 (check for firewalls!) doesn’t show up immediately. It looks like it appears (if it appears) after the system has been hung up for a while. Bruce

1 0

Re: [mpich-discuss] Better alternative to MPI_Allreduce() and avoiding deadlock with MPI_Neighbor_alltoallw().
by Benson Muite 06 May '20

06 May '20

On Tue, May 5, 2020, at 10:19 PM, hritikesh semwal via discuss wrote: > > > On Tue, 5 May, 2020, 10:30 PM , <discuss-request(a)mpich.org> wrote: >> >> > 1. I am using MPI_Neighbor_alltoallw() for exchanging the data by generating a distributed graph topology communicator. My concern is that most of the time my code is working fine but sometimes I guess it is going into deadlock (as it is not showing any output). But MPI_Neighbor_alltoallw uses MPI_Waitall inside it so I am not getting why exactly this is happening. >> >> >> >> May want to check sending and receiving correct data. Perhaps also try MPI_Neighbor_alltoallw >> >> >> >> > 2. Is it possible that every time I run the code the processors times for completion of the task may vary? For example, for one run it all processors takes around 100 seconds and for another run, all processors take 110 seconds. >> >> >> >> There is usually some variability. Do you solve the same system each time? What is the method of solution? If your code is available it can sometimes be easier to give suggestions. >> >> >> > Yes, the system of equations are the same. I am using the finite volume method for solving Navier stokes equations. By first sentence you mean to say it is possible. >> >> Is the method implicit or explicit? > >> > Its an explicit method. Ok >> >> > >> >> > >> >> > Please help in above two matters. >> >> > >> >> > On Tue, May 5, 2020 at 4:28 PM hritikesh semwal <hritikesh.semwal(a)gmail.com> wrote: >> >> >> Thanks for your response. >> >> >> >> >> >> Yes, you are right. I have put barrier just before Allreduce and out of the total time consumed by Allreduce, 79% time is consumed by the barrier. But my computational work is balanced. Right now, I have distributed 97336 cells among 24 processors and maximum and minimum cell distribution among all processors is 4057 and 4055 respectively which is not too bad. Is there any solution to get rid of this. >> >> >> >> Try profiling your code not just looking at cell distribution. Are any profling tools already installed on your cluster? >> > >> > gprof and valgrind are there. >> >> While not ideal GPROF may be helpful. Perhaps initial try running on 12 processors. With GPROF you will get 12 files to examine. Check if all subroutines take similar times on each processor. You can also time the subroutines individually using MPI_WTIME to get the same information. > >> > Yes, I have already timed my code before posting this question. I will try with gprof. > Great. Some documentation on Gprof: http://shwina.github.io/2014/11/profiling-parallel https://cluster.earlham.edu/wiki/index.php/Cluster:Gprof https://portal.tacc.utexas.edu/documents/13601/1041435/29-Overview_of_Profi… https://hpc.llnl.gov/software/development-environment-software/gprof https://support.pawsey.org.au/documentation/display/US/Profiling+with+gprof https://stackoverflow.com/questions/39041871/missing-function-from-gprof-ou… >> >> Also, try not to reply to the digest -, or if you do, change the subject of the message. This is useful in deciding what to read. > >> > Is it fine this time? I have changed the subject line. Is that what you want to say? That is ok if you cannot reply to the message directly.

1 0