August 2017 - discuss - mailman3.cels.anl.gov

Re: [mpich-discuss] question about -disable-auto-cleanup
by Kenneth Raffenetti 31 Aug '17

31 Aug '17

Correct, there is no MPI-level query mechanism without the ULFM extensions. Ken On 08/31/2017 11:49 AM, Zaak Beekman wrote: > Ken, > > Thanks so much for your concise response! > > Is my understanding correct that: If one MPI rank, or all the ranks on a > given node are to fail, from, say, a hardware issue, there is no means > of determining that this has happened unless the MPI library supports > the user level failure mitigation (ULFM) features? So if one were to > pass --disable-auto-cleanup when the runtime didn't have support for > ULFM, and processes on remote MPI ranks died, there'd be no way to query > them and call MPI_abort from a different, still running rank? > > Thanks again, > Zaak > > On Thu, Aug 31, 2017 at 12:19 PM <discuss-request(a)mpich.org > <mailto:[email protected]>> wrote: > > Send discuss mailing list submissions to > discuss(a)mpich.org <mailto:[email protected]> > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org <mailto:[email protected]> > > You can reach the person managing the list at > discuss-owner(a)mpich.org <mailto:[email protected]> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. question about -disable-auto-cleanup (Zaak Beekman) > 2. Re: Torque MPICH jobs stuck (Souparno Adhikary) > 3. Re: question about -disable-auto-cleanup (Kenneth Raffenetti) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 30 Aug 2017 17:29:11 +0000 > From: Zaak Beekman <zbeekman(a)gmail.com <mailto:[email protected]>> > To: discuss(a)mpich.org <mailto:[email protected]> > Subject: [mpich-discuss] question about -disable-auto-cleanup > Message-ID: > > <CAAbnBwb8L2VtXqsbbGjZvSOHpTkd7mK6SeQuhHKTEG0brOPSBQ(a)mail.gmail.com > <mailto:CAAbnBwb8L2VtXqsbbGjZvSOHpTkd7mK6SeQuhHKTEG0brOPSBQ@mail.gmail.com>> > Content-Type: text/plain; charset="utf-8" > > OK, since there were no responses here to my previous email, perhaps a > better question would be: > > What is a good resource to learn about the impact of passing > `--disable-auto-cleanup` at runtime? > > Some google searches bring up discussions of what appear to be bugs > in the > standard and/or implementation, but I'm not sure where to look to > find out > about even the intended runtime semantics. > > Any and all help pointing me in the right direction would be much > appreciated. > > Thanks, > Zaak > > On Wed, Aug 30, 2017 at 1:00 PM <discuss-request(a)mpich.org > <mailto:[email protected]>> wrote: > > > Send discuss mailing list submissions to > > discuss(a)mpich.org <mailto:[email protected]> > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.mpich.org/mailman/listinfo/discuss > > or, via email, send a message with subject or body 'help' to > > discuss-request(a)mpich.org <mailto:[email protected]> > > > > You can reach the person managing the list at > > discuss-owner(a)mpich.org <mailto:[email protected]> > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of discuss digest..." > > > > > > Today's Topics: > > > > 1. question about -disable-auto-cleanup (Zaak Beekman) > > 2. Torque MPICH jobs stuck (Souparno Adhikary) > > 3. Re: Torque MPICH jobs stuck (Halim Amer) > > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Tue, 29 Aug 2017 21:22:49 +0000 > > From: Zaak Beekman <zbeekman(a)gmail.com <mailto:[email protected]>> > > To: discuss(a)mpich.org <mailto:[email protected]> > > Subject: [mpich-discuss] question about -disable-auto-cleanup > > Message-ID: > > < > > > CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ(a)mail.gmail.com > <mailto:CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ@mail.gmail.com>> > > Content-Type: text/plain; charset="utf-8" > > > > I know that --disable-auto-cleanup is required to enable the > fault-tolerant > > MPI features, but are there downsides to passing this? Performance > > implications? > > > > I ask, because over at > https://github.com/sourceryinstitute/OpenCoarrays > > we've > > implemented much of the Fortran 2015 failed images feature on top > of MPICH > > and other MPI implementations. But to use this, > --disable-auto-cleanup must > > be passed to mpiexec. We provide wrapper scripts to try to > abstract the > > back end (GASNet, MPI, OpenSHMEM etc.) in the form of a Fortran > compiler > > wrapper, and an executable launcher. So I'm wondering, since > failed images > > are part of the standard (2015) would it be dumb if we always pass > > --disable-auto-cleanup to mpiexec and only turn off support when > explicitly > > asked for by the user, or is it safer/more performant to default to > > requiring the user to pass an additional flag to our wrapper > script that > > results in --disable-auto-cleanup getting passed to mpiexec? > > > > Feedback would be much appreciated. Feel free to post responses at > > https://github.com/sourceryinstitute/OpenCoarrays/issues/401 as > well.. > > > > Thanks, > > Zaak > >

1 0

Re: [mpich-discuss] question about -disable-auto-cleanup
by Zaak Beekman 31 Aug '17

31 Aug '17

Ken, Thanks so much for your concise response! Is my understanding correct that: If one MPI rank, or all the ranks on a given node are to fail, from, say, a hardware issue, there is no means of determining that this has happened unless the MPI library supports the user level failure mitigation (ULFM) features? So if one were to pass --disable-auto-cleanup when the runtime didn't have support for ULFM, and processes on remote MPI ranks died, there'd be no way to query them and call MPI_abort from a different, still running rank? Thanks again, Zaak On Thu, Aug 31, 2017 at 12:19 PM <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. question about -disable-auto-cleanup (Zaak Beekman) > 2. Re: Torque MPICH jobs stuck (Souparno Adhikary) > 3. Re: question about -disable-auto-cleanup (Kenneth Raffenetti) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 30 Aug 2017 17:29:11 +0000 > From: Zaak Beekman <zbeekman(a)gmail.com> > To: discuss(a)mpich.org > Subject: [mpich-discuss] question about -disable-auto-cleanup > Message-ID: > < > CAAbnBwb8L2VtXqsbbGjZvSOHpTkd7mK6SeQuhHKTEG0brOPSBQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > OK, since there were no responses here to my previous email, perhaps a > better question would be: > > What is a good resource to learn about the impact of passing > `--disable-auto-cleanup` at runtime? > > Some google searches bring up discussions of what appear to be bugs in the > standard and/or implementation, but I'm not sure where to look to find out > about even the intended runtime semantics. > > Any and all help pointing me in the right direction would be much > appreciated. > > Thanks, > Zaak > > On Wed, Aug 30, 2017 at 1:00 PM <discuss-request(a)mpich.org> wrote: > > > Send discuss mailing list submissions to > > discuss(a)mpich.org > > > > To subscribe or unsubscribe via the World Wide Web, visit > > https://lists.mpich.org/mailman/listinfo/discuss > > or, via email, send a message with subject or body 'help' to > > discuss-request(a)mpich.org > > > > You can reach the person managing the list at > > discuss-owner(a)mpich.org > > > > When replying, please edit your Subject line so it is more specific > > than "Re: Contents of discuss digest..." > > > > > > Today's Topics: > > > > 1. question about -disable-auto-cleanup (Zaak Beekman) > > 2. Torque MPICH jobs stuck (Souparno Adhikary) > > 3. Re: Torque MPICH jobs stuck (Halim Amer) > > > > > > ---------------------------------------------------------------------- > > > > Message: 1 > > Date: Tue, 29 Aug 2017 21:22:49 +0000 > > From: Zaak Beekman <zbeekman(a)gmail.com> > > To: discuss(a)mpich.org > > Subject: [mpich-discuss] question about -disable-auto-cleanup > > Message-ID: > > < > > CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ(a)mail.gmail.com> > > Content-Type: text/plain; charset="utf-8" > > > > I know that --disable-auto-cleanup is required to enable the > fault-tolerant > > MPI features, but are there downsides to passing this? Performance > > implications? > > > > I ask, because over at https://github.com/sourceryinstitute/OpenCoarrays > > we've > > implemented much of the Fortran 2015 failed images feature on top of > MPICH > > and other MPI implementations. But to use this, --disable-auto-cleanup > must > > be passed to mpiexec. We provide wrapper scripts to try to abstract the > > back end (GASNet, MPI, OpenSHMEM etc.) in the form of a Fortran compiler > > wrapper, and an executable launcher. So I'm wondering, since failed > images > > are part of the standard (2015) would it be dumb if we always pass > > --disable-auto-cleanup to mpiexec and only turn off support when > explicitly > > asked for by the user, or is it safer/more performant to default to > > requiring the user to pass an additional flag to our wrapper script that > > results in --disable-auto-cleanup getting passed to mpiexec? > > > > Feedback would be much appreciated. Feel free to post responses at > > https://github.com/sourceryinstitute/OpenCoarrays/issues/401 as well.. > > > > Thanks, > > Zaak > > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] question about -disable-auto-cleanup
by Kenneth Raffenetti 31 Aug '17

31 Aug '17

Hi Zaak, I'll try my best to explain here. There are a few things to consider. 1. Hydra: -disable-auto-cleanup means if an MPI process dies, let other processes in the job continue running. Since Hydra (mpiexec) is already monitoring MPI processes to detect when one dies, there is no impact inside Hydra from passing this option. 2. Application behavior: Since the default error handler in MPI is MPI_ERRORS_ARE_FATAL, some applications may rely on that fact and expect a running job to be aborted/cleaned up if a process quits. With -disable-auto-cleanup this will no longer be the case. An application can call MPI_Abort() to force the old behavior, however. Ken On 08/30/2017 12:29 PM, Zaak Beekman wrote: > OK, since there were no responses here to my previous email, perhaps a > better question would be: > > What is a good resource to learn about the impact of passing > `--disable-auto-cleanup` at runtime? > > Some google searches bring up discussions of what appear to be bugs in > the standard and/or implementation, but I'm not sure where to look to > find out about even the intended runtime semantics. > > Any and all help pointing me in the right direction would be much > appreciated. > > Thanks, > Zaak > > On Wed, Aug 30, 2017 at 1:00 PM <discuss-request(a)mpich.org > <mailto:[email protected]>> wrote: > > Send discuss mailing list submissions to > discuss(a)mpich.org <mailto:[email protected]> > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org <mailto:[email protected]> > > You can reach the person managing the list at > discuss-owner(a)mpich.org <mailto:[email protected]> > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. question about -disable-auto-cleanup (Zaak Beekman) > 2. Torque MPICH jobs stuck (Souparno Adhikary) > 3. Re: Torque MPICH jobs stuck (Halim Amer) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Aug 2017 21:22:49 +0000 > From: Zaak Beekman <zbeekman(a)gmail.com <mailto:[email protected]>> > To: discuss(a)mpich.org <mailto:[email protected]> > Subject: [mpich-discuss] question about -disable-auto-cleanup > Message-ID: > > <CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ(a)mail.gmail.com > <mailto:CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ@mail.gmail.com>> > Content-Type: text/plain; charset="utf-8" > > I know that --disable-auto-cleanup is required to enable the > fault-tolerant > MPI features, but are there downsides to passing this? Performance > implications? > > I ask, because over at > https://github.com/sourceryinstitute/OpenCoarrays we've > implemented much of the Fortran 2015 failed images feature on top of > MPICH > and other MPI implementations. But to use this, > --disable-auto-cleanup must > be passed to mpiexec. We provide wrapper scripts to try to abstract the > back end (GASNet, MPI, OpenSHMEM etc.) in the form of a Fortran compiler > wrapper, and an executable launcher. So I'm wondering, since failed > images > are part of the standard (2015) would it be dumb if we always pass > --disable-auto-cleanup to mpiexec and only turn off support when > explicitly > asked for by the user, or is it safer/more performant to default to > requiring the user to pass an additional flag to our wrapper script that > results in --disable-auto-cleanup getting passed to mpiexec? > > Feedback would be much appreciated. Feel free to post responses at > https://github.com/sourceryinstitute/OpenCoarrays/issues/401 as well.. > > Thanks, > Zaak >

1 0

Re: [mpich-discuss] Torque MPICH jobs stuck
by Souparno Adhikary 31 Aug '17

31 Aug '17

We are using mpich2-1.4.1p1. I can give a try with the latest version. My job script is as follows: #!/bin/sh #PBS -N asyn #PBS -q batch #PBS -l nodes=4:ppn=4 #PBS -l walltime=120:00:00 #PBS -V cd $PBS_O_WORKDIR mpirun -np 16 gmx_mpi mdrun -deffnm asyn_10ns Souparno Adhikary, CHPC Lab, Department of Microbiology, University of Calcutta. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

question about -disable-auto-cleanup
by Zaak Beekman 30 Aug '17

30 Aug '17

OK, since there were no responses here to my previous email, perhaps a better question would be: What is a good resource to learn about the impact of passing `--disable-auto-cleanup` at runtime? Some google searches bring up discussions of what appear to be bugs in the standard and/or implementation, but I'm not sure where to look to find out about even the intended runtime semantics. Any and all help pointing me in the right direction would be much appreciated. Thanks, Zaak On Wed, Aug 30, 2017 at 1:00 PM <discuss-request(a)mpich.org> wrote: > Send discuss mailing list submissions to > discuss(a)mpich.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://lists.mpich.org/mailman/listinfo/discuss > or, via email, send a message with subject or body 'help' to > discuss-request(a)mpich.org > > You can reach the person managing the list at > discuss-owner(a)mpich.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of discuss digest..." > > > Today's Topics: > > 1. question about -disable-auto-cleanup (Zaak Beekman) > 2. Torque MPICH jobs stuck (Souparno Adhikary) > 3. Re: Torque MPICH jobs stuck (Halim Amer) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 29 Aug 2017 21:22:49 +0000 > From: Zaak Beekman <zbeekman(a)gmail.com> > To: discuss(a)mpich.org > Subject: [mpich-discuss] question about -disable-auto-cleanup > Message-ID: > < > CAAbnBwZrQ03YmmmayhcHEywh8bEFMZ_AycBydOqZFB023KeJZQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > I know that --disable-auto-cleanup is required to enable the fault-tolerant > MPI features, but are there downsides to passing this? Performance > implications? > > I ask, because over at https://github.com/sourceryinstitute/OpenCoarrays > we've > implemented much of the Fortran 2015 failed images feature on top of MPICH > and other MPI implementations. But to use this, --disable-auto-cleanup must > be passed to mpiexec. We provide wrapper scripts to try to abstract the > back end (GASNet, MPI, OpenSHMEM etc.) in the form of a Fortran compiler > wrapper, and an executable launcher. So I'm wondering, since failed images > are part of the standard (2015) would it be dumb if we always pass > --disable-auto-cleanup to mpiexec and only turn off support when explicitly > asked for by the user, or is it safer/more performant to default to > requiring the user to pass an additional flag to our wrapper script that > results in --disable-auto-cleanup getting passed to mpiexec? > > Feedback would be much appreciated. Feel free to post responses at > https://github.com/sourceryinstitute/OpenCoarrays/issues/401 as well.. > > Thanks, > Zaak > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Torque MPICH jobs stuck
by Halim Amer 30 Aug '17

30 Aug '17

Which MPICH version are you using? Have you tried the latest 3.2 version? If it still fails, can you attach your simple Torque job script here? Halim www.mcs.anl.gov/~aamer On 8/30/17 3:18 AM, Souparno Adhikary wrote: > I know this is not a proper place to discuss this, but, as the > Torque-mpich list seems dead, I can't think of any other place to post this. > > MPICH2 was installed in the servers. I installed Torque afterwards. I > opened the ports including them in the iptables file. > > Torque mpi jobs (even the simple jobs like hostname) remains stuck. But, > the jobs are properly distributed in the nodes and pbsnodes -a showing > them in order. > > The sched_log files and server_logs do not yield anything different. > Therefore, it might be a problem with the mpich2. > > Can you please suggest me from where I can start troubleshooting??? > > Thanks, > > Souparno Adhikary, > CHPC Lab, > Department of Microbiology, > University of Calcutta. > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Torque MPICH jobs stuck
by Souparno Adhikary 30 Aug '17

30 Aug '17

I know this is not a proper place to discuss this, but, as the Torque-mpich list seems dead, I can't think of any other place to post this. MPICH2 was installed in the servers. I installed Torque afterwards. I opened the ports including them in the iptables file. Torque mpi jobs (even the simple jobs like hostname) remains stuck. But, the jobs are properly distributed in the nodes and pbsnodes -a showing them in order. The sched_log files and server_logs do not yield anything different. Therefore, it might be a problem with the mpich2. Can you please suggest me from where I can start troubleshooting??? Thanks, Souparno Adhikary, CHPC Lab, Department of Microbiology, University of Calcutta. _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

question about -disable-auto-cleanup
by Zaak Beekman 29 Aug '17

29 Aug '17

I know that --disable-auto-cleanup is required to enable the fault-tolerant MPI features, but are there downsides to passing this? Performance implications? I ask, because over at https://github.com/sourceryinstitute/OpenCoarrays we've implemented much of the Fortran 2015 failed images feature on top of MPICH and other MPI implementations. But to use this, --disable-auto-cleanup must be passed to mpiexec. We provide wrapper scripts to try to abstract the back end (GASNet, MPI, OpenSHMEM etc.) in the form of a Fortran compiler wrapper, and an executable launcher. So I'm wondering, since failed images are part of the standard (2015) would it be dumb if we always pass --disable-auto-cleanup to mpiexec and only turn off support when explicitly asked for by the user, or is it safer/more performant to default to requiring the user to pass an additional flag to our wrapper script that results in --disable-auto-cleanup getting passed to mpiexec? Feedback would be much appreciated. Feel free to post responses at https://github.com/sourceryinstitute/OpenCoarrays/issues/401 as well.. Thanks, Zaak _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Need help to run hybrid code.
by Jahanzeb Maqbool Hashmi 29 Aug '17

29 Aug '17

Hi, You are using -bind-to socket and -map-by socket with mpirun when running with MVAPICH2. These will not result in correct process mapping with MVAPICH2+OpenMP. You needs to use the options mentioned in the userguide. (http://mvapich.cse.ohio-state.edu/static/media/mvapich/mvapich2-2.3b-usergu…) regards Jahanzeb On Aug 28, 2017, 9:39 AM -0400, Pasha Pashaei <pasha.313(a)hotmail.com>, wrote: > Dear friends > I am going to run a hybrid MPI+OPENMP code. > As you can see in the below I played with the number of threads in various cases while running my main code with (Openmpi,Mpich,Mpich2). > As you can see in OpenMPi and Mpich it seems that openmp did not work at all as Total time did not change considerably. But in Mpich2 Total computational time increased with increasing the number of threads. It could be because of using virtual threads instead of physical threads(or you said that over-subscribing). > > Hybrid code result (MPI + OpenMP) : > Your suggestions: > mvapich > mpirun -np 4 -genv OMP_NUM_THREADS 1 --bind-to hwthread:1 ./pjet.gfortran > output.txt > Total time = 7.290E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 ./pjet.gfortran > output.txt > Total time = 4.940E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 -map-by hwthread:8 ./pjet.gfortran > output.txt > Total time = 4.960E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 --bind-to hwthread:16 ./pjet.gfortran > output.txt > Total time = 4.502E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to core:16 -map-by core:16 ./pjet.gfortran > output.txt > Total time = 4.628E+02 > > Pervios commands > OpenMPI 1.8.1 > mpirun -np 4 -x OMP_NUM_THREADS=1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.475E+02 > mpirun -np 4 -x OMP_NUM_THREADS=8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.525E+02 > mpirun -np 4 -x OMP_NUM_THREADS=16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.611E+02 > > mvapich > mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.441E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.535E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.552E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.591E+02 > > mvapich2 > mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.935E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 5.562E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 6.392E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 8.170E+02 > > Then I used a simple "hybrid.f90" code and its result which I used to check whether the computer can recognize correct value of cores and threads or not. It showed that the correct values in all three in different cases. > here is its result: > Starting omp_dotprod_hybrid. Using 4 Cores... > Core 3 using 16 threads > Core 0 using 16 threads > Core 2 using 16 threads > Core 1 using 16 threads > Core 1 thread 0 partial sum = 0.0000000000000000 > Core 3 thread 0 partial sum = 0.0000000000000000 > Core 1 thread 4 partial sum = 0.0000000000000000 > Core 3 thread 7 partial sum = 200.00000000000000 > Core 1 thread 8 partial sum = 200.00000000000000 > Core 3 thread 9 partial sum = 200.00000000000000 > Core 1 thread 11 partial sum = 200.00000000000000 > Core 3 thread 3 partial sum = 200.00000000000000 > Core 1 thread 2 partial sum = 0.0000000000000000 > Core 3 thread 5 partial sum = 0.0000000000000000 > Core 1 thread 3 partial sum = 200.00000000000000 > Core 3 thread 2 partial sum = 0.0000000000000000 > Core 1 thread 13 partial sum = 200.00000000000000 > Core 3 thread 12 partial sum = 200.00000000000000 > Core 1 thread 1 partial sum = 200.00000000000000 > Core 3 thread 1 partial sum = 200.00000000000000 > Core 3 thread 8 partial sum = 0.0000000000000000 > Core 1 thread 7 partial sum = 0.0000000000000000 > Core 3 thread 11 partial sum = 200.00000000000000 > Core 1 thread 15 partial sum = 0.0000000000000000 > Core 3 thread 15 partial sum = 0.0000000000000000 > Core 1 thread 10 partial sum = 200.00000000000000 > Core 1 thread 9 partial sum = 200.00000000000000 > Core 3 thread 13 partial sum = 0.0000000000000000 > Core 1 thread 5 partial sum = 200.00000000000000 > Core 3 thread 6 partial sum = 0.0000000000000000 > Core 3 thread 4 partial sum = 0.0000000000000000 > Core 1 thread 6 partial sum = 0.0000000000000000 > Core 3 thread 14 partial sum = 200.00000000000000 > Core 1 thread 12 partial sum = 0.0000000000000000 > Core 3 thread 10 partial sum = 200.00000000000000 > Core 0 thread 0 partial sum = 0.0000000000000000 > Core 0 thread 14 partial sum = 200.00000000000000 > Core 0 thread 8 partial sum = 200.00000000000000 > Core 0 thread 7 partial sum = 0.0000000000000000 > Core 0 thread 15 partial sum = 200.00000000000000 > Core 0 thread 5 partial sum = 200.00000000000000 > Core 0 thread 9 partial sum = 200.00000000000000 > Core 0 thread 11 partial sum = 0.0000000000000000 > Core 0 thread 10 partial sum = 200.00000000000000 > Core 0 thread 6 partial sum = 200.00000000000000 > Core 0 thread 3 partial sum = 200.00000000000000 > Core 0 thread 4 partial sum = 0.0000000000000000 > Core 0 thread 2 partial sum = 0.0000000000000000 > Core 0 thread 13 partial sum = 0.0000000000000000 > Core 0 thread 12 partial sum = 0.0000000000000000 > Core 0 thread 1 partial sum = 0.0000000000000000 > Core 0 partial sum = 1600.0000000000000 > Core 2 thread 3 partial sum = 0.0000000000000000 > Core 2 thread 15 partial sum = 0.0000000000000000 > Core 2 thread 0 partial sum = 0.0000000000000000 > Core 2 thread 2 partial sum = 200.00000000000000 > Core 2 thread 4 partial sum = 0.0000000000000000 > Core 2 thread 5 partial sum = 0.0000000000000000 > Core 2 thread 9 partial sum = 200.00000000000000 > Core 2 thread 7 partial sum = 0.0000000000000000 > Core 2 thread 14 partial sum = 200.00000000000000 > Core 2 thread 8 partial sum = 200.00000000000000 > Core 2 thread 12 partial sum = 200.00000000000000 > Core 2 thread 10 partial sum = 200.00000000000000 > Core 2 thread 6 partial sum = 200.00000000000000 > Core 2 thread 1 partial sum = 0.0000000000000000 > Core 2 thread 13 partial sum = 0.0000000000000000 > Core 2 thread 11 partial sum = 200.00000000000000 > Core 2 partial sum = 1600.0000000000000 > Core 3 partial sum = 1600.0000000000000 > Core 1 thread 14 partial sum = 0.0000000000000000 > Core 1 partial sum = 1600.0000000000000 > Done. Hybrid version: global sum = 6400.0000000000000 > > > > Please tell me If I should check something. I still getting nowhere. > Best regards > Pasha Pashaei > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0

Re: [mpich-discuss] Need help to run hybrid code.
by Halim Amer 28 Aug '17

28 Aug '17

First of all, you are using Open MPI and MVAPICH below; I don't see MPICH. If you are using MPICH, then please tell us the version (show the output of mpichversion). > Starting omp_dotprod_hybrid. Using 4 Cores... > Core 3 using 16 threads > Core 0 using 16 threads > Core 2 using 16 threads > Core 1 using 16 threads Second, you are using 16 threads per core; naturally you won't scale. Third, even if you were using 1 OpenMP thread per hardware thread, there is no guarantee that your program will scale (as in, time reduces while increasing the number of threads). You program might have scalability bottlenecks that have nothing to do with MPI. My first advice is to forget about MPI and make sure you program scales with only OpenMP. If your machine is truly a 64-way threads one, then use 1 MPI process per node and investigate the issue. Finally, if you still suspect bad thread bindings, then check how they are bound to hardware threads instead of measuring execution time. Use sched_getcpu() to query thread placement on Linux and tools like hwloc to understand the machine topology. I doubt your problems are related to issues in MPICH, which is the purpose of this list. Thus, I suggest to move this discuss out of the list and reply to me personally instead of replying to the list. Halim www.mcs.anl.gov/~aamer On 8/28/17 8:39 AM, Pasha Pashaei wrote: > Dear friends > I am going to run a hybrid MPI+OPENMP code. > As you can see in the below I played with the number of threads in various cases while running my main code with (Openmpi,Mpich,Mpich2). > As you can see in OpenMPi and Mpich it seems that openmp did not work at all as Total time did not change considerably. But in Mpich2 Total computational time increased with increasing the number of threads. It could be because of using virtual threads instead of physical threads(or you said that over-subscribing). > > > Hybrid code result (MPI + OpenMP) : > Your suggestions: > mvapich > mpirun -np 4 -genv OMP_NUM_THREADS 1 --bind-to hwthread:1 ./pjet.gfortran > output.txt > Total time = 7.290E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 ./pjet.gfortran > output.txt > Total time = 4.940E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 --bind-to hwthread:8 -map-by hwthread:8 ./pjet.gfortran > output.txt > Total time = 4.960E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 --bind-to hwthread:16 ./pjet.gfortran > output.txt > Total time = 4.502E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to core:16 -map-by core:16 ./pjet.gfortran > output.txt > Total time = 4.628E+02 > > Pervios commands > OpenMPI 1.8.1 > mpirun -np 4 -x OMP_NUM_THREADS=1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.475E+02 > mpirun -np 4 -x OMP_NUM_THREADS=8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.525E+02 > mpirun -np 4 -x OMP_NUM_THREADS=16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.611E+02 > > mvapich > mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.441E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.535E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.552E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.591E+02 > > mvapich2 > mpirun -np 4 -genv OMP_NUM_THREADS 1 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 4.935E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 4 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 5.562E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 8 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 6.392E+02 > mpirun -np 4 -genv OMP_NUM_THREADS 16 -bind-to socket -map-by socket ./pjet.gfortran > output.txt > Total time = 8.170E+02 > > Then I used a simple "hybrid.f90" code and its result which I used to check whether the computer can recognize correct value of cores and threads or not. It showed that the correct values in all three in different cases. > here is its result: > > Starting omp_dotprod_hybrid. Using 4 Cores... > Core 3 using 16 threads > Core 0 using 16 threads > Core 2 using 16 threads > Core 1 using 16 threads > Core 1 thread 0 partial sum = 0.0000000000000000 > Core 3 thread 0 partial sum = 0.0000000000000000 > Core 1 thread 4 partial sum = 0.0000000000000000 > Core 3 thread 7 partial sum = 200.00000000000000 > Core 1 thread 8 partial sum = 200.00000000000000 > Core 3 thread 9 partial sum = 200.00000000000000 > Core 1 thread 11 partial sum = 200.00000000000000 > Core 3 thread 3 partial sum = 200.00000000000000 > Core 1 thread 2 partial sum = 0.0000000000000000 > Core 3 thread 5 partial sum = 0.0000000000000000 > Core 1 thread 3 partial sum = 200.00000000000000 > Core 3 thread 2 partial sum = 0.0000000000000000 > Core 1 thread 13 partial sum = 200.00000000000000 > Core 3 thread 12 partial sum = 200.00000000000000 > Core 1 thread 1 partial sum = 200.00000000000000 > Core 3 thread 1 partial sum = 200.00000000000000 > Core 3 thread 8 partial sum = 0.0000000000000000 > Core 1 thread 7 partial sum = 0.0000000000000000 > Core 3 thread 11 partial sum = 200.00000000000000 > Core 1 thread 15 partial sum = 0.0000000000000000 > Core 3 thread 15 partial sum = 0.0000000000000000 > Core 1 thread 10 partial sum = 200.00000000000000 > Core 1 thread 9 partial sum = 200.00000000000000 > Core 3 thread 13 partial sum = 0.0000000000000000 > Core 1 thread 5 partial sum = 200.00000000000000 > Core 3 thread 6 partial sum = 0.0000000000000000 > Core 3 thread 4 partial sum = 0.0000000000000000 > Core 1 thread 6 partial sum = 0.0000000000000000 > Core 3 thread 14 partial sum = 200.00000000000000 > Core 1 thread 12 partial sum = 0.0000000000000000 > Core 3 thread 10 partial sum = 200.00000000000000 > Core 0 thread 0 partial sum = 0.0000000000000000 > Core 0 thread 14 partial sum = 200.00000000000000 > Core 0 thread 8 partial sum = 200.00000000000000 > Core 0 thread 7 partial sum = 0.0000000000000000 > Core 0 thread 15 partial sum = 200.00000000000000 > Core 0 thread 5 partial sum = 200.00000000000000 > Core 0 thread 9 partial sum = 200.00000000000000 > Core 0 thread 11 partial sum = 0.0000000000000000 > Core 0 thread 10 partial sum = 200.00000000000000 > Core 0 thread 6 partial sum = 200.00000000000000 > Core 0 thread 3 partial sum = 200.00000000000000 > Core 0 thread 4 partial sum = 0.0000000000000000 > Core 0 thread 2 partial sum = 0.0000000000000000 > Core 0 thread 13 partial sum = 0.0000000000000000 > Core 0 thread 12 partial sum = 0.0000000000000000 > Core 0 thread 1 partial sum = 0.0000000000000000 > Core 0 partial sum = 1600.0000000000000 > Core 2 thread 3 partial sum = 0.0000000000000000 > Core 2 thread 15 partial sum = 0.0000000000000000 > Core 2 thread 0 partial sum = 0.0000000000000000 > Core 2 thread 2 partial sum = 200.00000000000000 > Core 2 thread 4 partial sum = 0.0000000000000000 > Core 2 thread 5 partial sum = 0.0000000000000000 > Core 2 thread 9 partial sum = 200.00000000000000 > Core 2 thread 7 partial sum = 0.0000000000000000 > Core 2 thread 14 partial sum = 200.00000000000000 > Core 2 thread 8 partial sum = 200.00000000000000 > Core 2 thread 12 partial sum = 200.00000000000000 > Core 2 thread 10 partial sum = 200.00000000000000 > Core 2 thread 6 partial sum = 200.00000000000000 > Core 2 thread 1 partial sum = 0.0000000000000000 > Core 2 thread 13 partial sum = 0.0000000000000000 > Core 2 thread 11 partial sum = 200.00000000000000 > Core 2 partial sum = 1600.0000000000000 > Core 3 partial sum = 1600.0000000000000 > Core 1 thread 14 partial sum = 0.0000000000000000 > Core 1 partial sum = 1600.0000000000000 > Done. Hybrid version: global sum = 6400.0000000000000 > > > > Please tell me If I should check something. I still getting nowhere. > Best regards > > Pasha Pashaei > > > > > _______________________________________________ > discuss mailing list discuss(a)mpich.org > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss(a)mpich.org To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/discuss

1 0