How to confirm the performance of asynchronous computations
Hello Petsc developers and supporters, I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code. PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one); ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr); ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr); ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr); On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation. The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100. Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation? Thank you very much for anything you can provide. Sincerely, Viet.
On , 2021Jan21, at 06:52, Viet H.Q.H. <[email protected]<mailto:[email protected]>> wrote: The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 Your runtime is so short that you probably only measure a slight increase in overhead. How many processors and what was the problem size? Victor.
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u. comb.c has /* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only The idea for this and much of the initial code is contributed by Victor Eijkhout. Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *); Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */ #include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/ static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr; PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); } So first check if $PETSC_DIR/include/petscconf.h has PETSC_HAVE_MPI_IALLREDUCE if it does not then the standard MPI reduce is called. If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using. Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected]> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
Dear Victor and Berry, Thank you so much for your answers. I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave. ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u)); It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000. Asynchronous computation still takes a longer time. I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h I built Petsc by using the following script #!/usr/bin/bash set -e DATE="21.01.18" MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc" cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native' make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install Intel 2018 also complies with the MPI-3 standard. Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function? Sincerely, Viet. On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected]> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected]> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel... <https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html> It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected]> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18" MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
Dear Barry, Thank you very much for your information. It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI. $ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list> https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp... I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet. On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected]> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel...
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected]> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18"
MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64"
MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected]> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected]> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
Sorry about the delay in responding, but I'll add a couple of points here: 1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case). 2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation. For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system). I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.
Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <[email protected]>:
Dear Barry,
Thank you very much for your information.
It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.
$ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list>
https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp... <https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/>
I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet.
On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel... <https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html>
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18" MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
Dear Patrick Sanan, Thank you very much for your answer, especially for your code. I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result Testing with 160 MPI ranks reducing an array of size 32 (256 bytes) Running 5 burnin runs and 100 tests ... Done. For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: MPI timer resolution: 1.0000e-06 seconds MPI timer resolution/#trials: 1.0000e-08 seconds B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden It appears that the non-blocking computation with this test is no faster than the blocking computation. I think I am missing some suitable Intel MPI environment settings. I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests? Thanks again. Viet On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan <[email protected]> wrote:
Sorry about the delay in responding, but I'll add a couple of points here:
1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case).
2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation.
For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system).
I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.
Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <[email protected]>:
Dear Barry,
Thank you very much for your information.
It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.
$ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list>
https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp...
I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet.
On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected]> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel...
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected]> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18"
MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64"
MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected]> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected]> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. <[email protected]> wrote:
Dear Patrick Sanan,
Thank you very much for your answer, especially for your code. I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result
Testing with 160 MPI ranks reducing an array of size 32 (256 bytes) Running 5 burnin runs and 100 tests ... Done. For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: MPI timer resolution: 1.0000e-06 seconds MPI timer resolution/#trials: 1.0000e-08 seconds B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden
It appears that the non-blocking computation with this test is no faster than the blocking computation. I think I am missing some suitable Intel MPI environment settings. I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests?
Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely larger in your application? Thanks, Matt
Thanks again. Viet
On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan <[email protected]> wrote:
Sorry about the delay in responding, but I'll add a couple of points here:
1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case).
2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation.
For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system).
I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.
Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <[email protected]>:
Dear Barry,
Thank you very much for your information.
It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.
$ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list>
https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp...
I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet.
On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected]> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel...
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected]> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18"
MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64"
MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected]> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected]> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
Am 26.01.2021 um 12:01 schrieb Matthew Knepley <[email protected]>:
On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote: Dear Patrick Sanan,
Thank you very much for your answer, especially for your code. I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result
Testing with 160 MPI ranks reducing an array of size 32 (256 bytes) Running 5 burnin runs and 100 tests ... Done. For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: MPI timer resolution: 1.0000e-06 seconds MPI timer resolution/#trials: 1.0000e-08 seconds B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden
It appears that the non-blocking computation with this test is no faster than the blocking computation. I think I am missing some suitable Intel MPI environment settings. I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests? I used Cray MPI, which is based on MPICH.
Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely larger in your application? This is very important! Note that the reductions are quite fast compared to the local work. Even on a much larger number of ranks, they might still be quite fast. The local work here is just some useless computation (here tuned to be comparable to how long it took to do a reduction on a few thousand ranks at the time). As Matt says, having some estimate of how long it takes to apply your operator (matrix) and preconditioner is essential. Do you know how to estimate that for your application?
Thanks,
Matt
Thanks again. Viet
On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan <[email protected] <mailto:[email protected]>> wrote: Sorry about the delay in responding, but I'll add a couple of points here:
1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case).
2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation.
For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system).
I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.
Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <[email protected] <mailto:[email protected]>>:
Dear Barry,
Thank you very much for your information.
It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.
$ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list>
https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp... <https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/>
I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet.
On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel... <https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html>
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18" MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
Dear Patrick and Matthew, Thank you very much for your answers. My test code is just the code that contains the inner product calculation and the calculation of the matrix-vector multiplication. It is easy to measure the time of each calculation. I'm going to revise the code to measure the time as it's done in Patrick's code. I will also switch to Mpich so as not to worry about the MPI environment settings. With the current number of nodes (8), the reduction time for calculating the inner product is relatively small, but I think it would get bigger if I increased the number of nodes to 100 or more. In this case, the latency of the inner product calculation would increase, since the reduced values are transmitted via a complex network structure with cables and switches. Then, hopefully, the non-blocking communication can show its effectiveness. Best, Viet On Wed, Jan 27, 2021 at 2:53 AM Patrick Sanan <[email protected]> wrote:
Am 26.01.2021 um 12:01 schrieb Matthew Knepley <[email protected]>:
On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. <[email protected]> wrote:
Dear Patrick Sanan,
Thank you very much for your answer, especially for your code. I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result
Testing with 160 MPI ranks reducing an array of size 32 (256 bytes) Running 5 burnin runs and 100 tests ... Done. For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: MPI timer resolution: 1.0000e-06 seconds MPI timer resolution/#trials: 1.0000e-08 seconds B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden
It appears that the non-blocking computation with this test is no faster than the blocking computation. I think I am missing some suitable Intel MPI environment settings. I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests?
I used Cray MPI, which is based on MPICH.
Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely larger in your application?
This is very important! Note that the reductions are quite fast compared to the local work. Even on a much larger number of ranks, they might still be quite fast. The local work here is just some useless computation (here tuned to be comparable to how long it took to do a reduction on a few thousand ranks at the time). As Matt says, having some estimate of how long it takes to apply your operator (matrix) and preconditioner is essential. Do you know how to estimate that for your application?
Thanks,
Matt
Thanks again. Viet
On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan <[email protected]> wrote:
Sorry about the delay in responding, but I'll add a couple of points here:
1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case).
2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation.
For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system).
I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.
Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <[email protected]>:
Dear Barry,
Thank you very much for your information.
It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.
$ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list>
https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp...
I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet.
On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected]> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel...
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected]> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18"
MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64"
MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected]> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected]> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On Wed, Jan 27, 2021 at 10:51 AM Viet H.Q.H. <[email protected]> wrote:
Dear Patrick and Matthew,
Thank you very much for your answers. My test code is just the code that contains the inner product calculation and the calculation of the matrix-vector multiplication. It is easy to measure the time of each calculation. I'm going to revise the code to measure the time as it's done in Patrick's code. I will also switch to Mpich so as not to worry about the MPI environment settings.
With the current number of nodes (8), the reduction time for calculating the inner product is relatively small, but I think it would get bigger if I increased the number of nodes to 100 or more. In this case, the latency of the inner product calculation would increase, since the reduced values are transmitted via a complex network structure with cables and switches. Then, hopefully, the non-blocking communication can show its effectiveness.
This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on. Thanks Matt
Best, Viet
On Wed, Jan 27, 2021 at 2:53 AM Patrick Sanan <[email protected]> wrote:
Am 26.01.2021 um 12:01 schrieb Matthew Knepley <[email protected]>:
On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. <[email protected]> wrote:
Dear Patrick Sanan,
Thank you very much for your answer, especially for your code. I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result
Testing with 160 MPI ranks reducing an array of size 32 (256 bytes) Running 5 burnin runs and 100 tests ... Done. For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: MPI timer resolution: 1.0000e-06 seconds MPI timer resolution/#trials: 1.0000e-08 seconds B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden
It appears that the non-blocking computation with this test is no faster than the blocking computation. I think I am missing some suitable Intel MPI environment settings. I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests?
I used Cray MPI, which is based on MPICH.
Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely larger in your application?
This is very important! Note that the reductions are quite fast compared to the local work. Even on a much larger number of ranks, they might still be quite fast. The local work here is just some useless computation (here tuned to be comparable to how long it took to do a reduction on a few thousand ranks at the time). As Matt says, having some estimate of how long it takes to apply your operator (matrix) and preconditioner is essential. Do you know how to estimate that for your application?
Thanks,
Matt
Thanks again. Viet
On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan <[email protected]> wrote:
Sorry about the delay in responding, but I'll add a couple of points here:
1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case).
2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation.
For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system).
I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.
Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <[email protected]>:
Dear Barry,
Thank you very much for your information.
It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.
$ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list>
https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp...
I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet.
On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected]> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel...
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected]> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18"
MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64"
MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected]> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected]> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
Am 27.01.2021 um 17:30 schrieb Matthew Knepley <[email protected]>:
On Wed, Jan 27, 2021 at 10:51 AM Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote: Dear Patrick and Matthew,
Thank you very much for your answers. My test code is just the code that contains the inner product calculation and the calculation of the matrix-vector multiplication. It is easy to measure the time of each calculation. I'm going to revise the code to measure the time as it's done in Patrick's code. I will also switch to Mpich so as not to worry about the MPI environment settings.
With the current number of nodes (8), the reduction time for calculating the inner product is relatively small, but I think it would get bigger if I increased the number of nodes to 100 or more. In this case, the latency of the inner product calculation would increase, since the reduced values are transmitted via a complex network structure with cables and switches. Then, hopefully, the non-blocking communication can show its effectiveness.
This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on.
Seconded! A very simple performance model is to expect reductions to take C * log(P) time on P ranks, which is of course a slowly increasing function.
Thanks
Matt
Best, Viet
On Wed, Jan 27, 2021 at 2:53 AM Patrick Sanan <[email protected] <mailto:[email protected]>> wrote:
Am 26.01.2021 um 12:01 schrieb Matthew Knepley <[email protected] <mailto:[email protected]>>:
On Mon, Jan 25, 2021 at 11:31 PM Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote: Dear Patrick Sanan,
Thank you very much for your answer, especially for your code. I was able to compile and run your code on 8 nodes with 20 processes per node. Below is the result
Testing with 160 MPI ranks reducing an array of size 32 (256 bytes) Running 5 burnin runs and 100 tests ... Done. For 100 runs with 5 burnin runs, on 160 MPI processes, min/max times over all ranks: MPI timer resolution: 1.0000e-06 seconds MPI timer resolution/#trials: 1.0000e-08 seconds B. Red. Only (min/max): 8.850098e-06 / 8.890629e-06 seconds N.B. Red. Only (min/max): 1.725912e-05 / 1.733065e-05 seconds Loc. Only (min/max): 2.364278e-04 / 2.374697e-04 seconds Blocking (min/max): 2.650309e-04 / 2.650595e-04 seconds Non-Blocking (min/max): 2.673984e-04 / 2.674508e-04 seconds Observe to see if the local time is enough to hide the reduction, and see if the reduction is indeed hidden
It appears that the non-blocking computation with this test is no faster than the blocking computation. I think I am missing some suitable Intel MPI environment settings. I am now thinking about using MPICH, which does not require any environment settings for non-blocking computation. Could you please let me know which MPI (MPICH or OpenMPI) you used in your tests? I used Cray MPI, which is based on MPICH.
Note that in the test, the cost of the reduction is only 3% of the total cost. Is that worth putting time into? Is the cost definitely larger in your application? This is very important! Note that the reductions are quite fast compared to the local work. Even on a much larger number of ranks, they might still be quite fast. The local work here is just some useless computation (here tuned to be comparable to how long it took to do a reduction on a few thousand ranks at the time). As Matt says, having some estimate of how long it takes to apply your operator (matrix) and preconditioner is essential. Do you know how to estimate that for your application?
Thanks,
Matt
Thanks again. Viet
On Mon, Jan 25, 2021 at 7:47 PM Patrick Sanan <[email protected] <mailto:[email protected]>> wrote: Sorry about the delay in responding, but I'll add a couple of points here:
1) It's important to have some reason to believe that pipelining will actually help your problem. Pipelined Krylov methods work by overlapping reductions with operator and preconditioner applications. So, to see speedup, the time for a reduction needs to be comparable to the time for the operator/preconditioner application. This will only be true in some cases - typical cases are when you have a large number of ranks/nodes, a slow network, or very fast operator/preconditioner applications (assuming that these require the same time on each rank - it's an interesting case when they don't, but unless you say otherwise I'll assume this doesn't apply to your use case).
2) As you're discovering, simply ensuring that asynchronous progress works, at the pure MPI level, isn't as easy as it might be, as it's so dependent on the MPI implementation.
For both of these reasons, I suggest setting up a test that just directly uses MPI (which you can of course do from a PETSc-style code) and allows you to compare times for blocking and non-blocking reductions, overlapping some (useless) local work. You should make sure to run multiple iterations within the script, and also run the script multiple times on the cluster (bearing in mind that it's possible that the performance will be affected by other users of the system).
I attach an old script I found that I used to test some of these things, to give a more concrete idea of what I mean. Note that this was used early on in our own exploration of these topics so I'm only offering it to give an idea, not as a meaningful benchmark in its own right.
Am 25.01.2021 um 09:17 schrieb Viet H.Q.H. <[email protected] <mailto:[email protected]>>:
Dear Barry,
Thank you very much for your information.
It seems complicated to set environment variables to allow asynchronous progress and pinning threads to cores when using Intel MPI.
$ export I_MPI_ASYNC_PROGRESS = 1 $ export I_MPI_ASYNC_PROGRESS_PIN = <CPU list>
https://techdecoded.intel.io/resources/hiding-communication-latency-using-mp... <https://techdecoded.intel.io/resources/hiding-communication-latency-using-mpi-3-non-blocking-collectives/>
I'm still not sure how to get an appropriate "CPU list" when running MPI with multiple nodes and multiple processes on one node. Best, Viet.
On Sat, Jan 23, 2021 at 3:01 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
https://software.intel.com/content/www/us/en/develop/documentation/mpi-devel... <https://software.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/additional-supported-features/asynchronous-progress-control.html>
It states "and a partial support for non-blocking collectives ( MPI_Ibcas t, MPI_Ireduce , and MPI_Iallreduce )." I do not know what partial support means but you can try setting the variables and see if that helps.
On Jan 22, 2021, at 11:20 AM, Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote:
Dear Victor and Berry,
Thank you so much for your answers.
I fixed the code with the bug in the PetscCommSplitReductionBegin function as commented by Brave.
ierr = PetscCommSplitReductionBegin (PetscObjectComm ((PetscObject) u));
It was also a mistake to set the vector size too small. I just set a vector size of 100000000 and ran the code on 4 nodes with 2 processors per node. The result is as follows
The time used for the asynchronous calculation: 0.022043 + | u | = 10000. The time used for the synchronous calculation: 0.016188 + | b | = 10000.
Asynchronous computation still takes a longer time.
I also confirmed that PETSC_HAVE_MPI_IALLREDUCE is defined in the file $PETSC_DIR/include/petscconf.h
I built Petsc by using the following script
#!/usr/bin/bash set -e DATE="21.01.18" MPIIT_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mpi/intel64" MKL_DIR="/work/A/intel/2018_update2/compilers_and_libraries_2018.2.199/linux/mkl" INSTL_DIR="${HOME}/local/petsc-3.14.3" BUILD_DIR="${HOME}/tmp/petsc/build_${DATE}" PETSC_DIR="${HOME}/tmp/petsc"
cd ${PETSC_DIR} ./configure --force --prefix=${INSTL_DIR} --with-mpi-dir=${MPIIT_DIR} --with-fortran-bindings=0 --with-mpiexe=${MPIIT_DIR}/bin/mpiexec --with-valgrind-dir=${HOME}/local/valgrind --with-blaslapack-dir=${MKL_DIR} --download-make --with-debugging=0 COPTFLAGS='-O3 -march=native -mtune=native' CXXOPTFLAGS='-O3 -march=native -mtune=native' FOPTFLAGS='-O3 -march=native -mtune=native'
make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt all make PETSC_DIR=${HOME}/tmp/petsc PETSC_ARCH=arch-linux2-c-opt install
Intel 2018 also complies with the MPI-3 standard.
Are there specific settings for Intel MPI to obtain the performance of the MPI_IALLREDUCE function?
Sincerely, Viet.
On Fri, Jan 22, 2021 at 11:20 AM Barry Smith <[email protected] <mailto:[email protected]>> wrote:
ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax));
How come you call this on Ax and not on u? For clarity, if nothing else, I think you should call it on u.
comb.c has
/* Split phase global vector reductions with support for combining the communication portion of several operations. Using MPI-1.1 support only
The idea for this and much of the initial code is contributed by Victor Eijkhout.
Usage: VecDotBegin(Vec,Vec,PetscScalar *); VecNormBegin(Vec,NormType,PetscReal *); .... VecDotEnd(Vec,Vec,PetscScalar *); VecNormEnd(Vec,NormType,PetscReal *);
Limitations: - The order of the xxxEnd() functions MUST be in the same order as the xxxBegin(). There is extensive error checking to try to insure that the user calls the routines in the correct order */
#include <petsc/private/vecimpl.h> /*I "petscvec.h" I*/
static PetscErrorCode MPIPetsc_Iallreduce(void *sendbuf,void *recvbuf,PetscMPIInt count,MPI_Datatype datatype,MPI_Op op,MPI_Comm comm,MPI_Request *request) { PETSC_UNUSED PetscErrorCode ierr;
PetscFunctionBegin; #if defined(PETSC_HAVE_MPI_IALLREDUCE) ierr = MPI_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRMPI(ierr); #elif defined(PETSC_HAVE_MPIX_IALLREDUCE) ierr = MPIX_Iallreduce(sendbuf,recvbuf,count,datatype,op,comm,request);CHKERRQ(ierr); #else ierr = MPIU_Allreduce(sendbuf,recvbuf,count,datatype,op,comm);CHKERRQ(ierr); *request = MPI_REQUEST_NULL; #endif PetscFunctionReturn(0); }
So first check if $PETSC_DIR/include/petscconf.h has
PETSC_HAVE_MPI_IALLREDUCE
if it does not then the standard MPI reduce is called.
If this is set then any improvement depends on the implementation of iallreduce inside the MPI you are using.
Barry
On Jan 21, 2021, at 6:52 AM, Viet H.Q.H. <[email protected] <mailto:[email protected]>> wrote:
Hello Petsc developers and supporters,
I would like to confirm the performance of asynchronous computations of inner product computation overlapping with matrix-vector multiplication computation by the below code.
PetscLogDouble tt1,tt2; KSP ksp; //ierr = VecSet(c,one); ierr = VecSet(c,one); ierr = VecSet(u,one); ierr = VecSet(b,one);
ierr = KSPCreate(PETSC_COMM_WORLD,&ksp); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,x,Ax); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNormBegin(u,NORM_2,&norm1); ierr = PetscCommSplitReductionBegin(PetscObjectComm((PetscObject)Ax)); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = VecNormEnd(u,NORM_2,&norm1); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the asynchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |u| = %g\n",(double) norm1); CHKERRQ(ierr);
ierr = PetscTime(&tt1);CHKERRQ(ierr); ierr = VecNorm(b,NORM_2,&norm2); CHKERRQ(ierr); ierr = KSP_MatMult(ksp,A,c,Ac); ierr = PetscTime(&tt2);CHKERRQ(ierr);
ierr = PetscPrintf(PETSC_COMM_WORLD, "The time used for the synchronous calculation: %f\n",tt2-tt1); CHKERRQ(ierr); ierr = PetscPrintf(PETSC_COMM_WORLD,"+ |b| = %g\n",(double) norm2); CHKERRQ(ierr);
On a cluster with two or four nodes, the asynchronous computation is always much slower than synchronous computation.
The time used for the asynchronous calculation: 0.000203 + |u| = 100. The time used for the synchronous calculation: 0.000006 + |b| = 100.
Are there any necessary settings on MPI or Petsc to gain performance of asynchronous computation?
Thank you very much for anything you can provide. Sincerely, Viet.
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
On 27 Jan 2021, at 16:30, Matthew Knepley <[email protected]> wrote:
This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on.
Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me. Although I guess Figure 2 from here https://www.mcs.anl.gov/papers/P5347-0515.pdf probably gives me a clue. Viet, I suspect that Matt thinks you should try and produce a figure like Figure 3 from that linked paper. Lawrence
Lawrence Mitchell <[email protected]> writes:
On 27 Jan 2021, at 16:30, Matthew Knepley <[email protected]> wrote:
This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on.
Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me.
I've been asking for it for years. They say if you care, you should just go run it. Never mind how wasteful that is, and the time commitment to doing so. I think they often avoid making a commitment because latency is super variable (depending on the partition you get and what other jobs are running elsewhere on the machine; Blue Gene famously didn't have that problem). Meanwhile, latency on cloud providers keeps dropping and they're sure to beat conventional HPC centers to publishing a dashboard of expected latency for different configurations. This page illustrates how hardware reductions scale much better than log(P). https://www.mcs.anl.gov/~fischer/gop/
Although I guess Figure 2 from here https://www.mcs.anl.gov/papers/P5347-0515.pdf probably gives me a clue.
Viet, I suspect that Matt thinks you should try and produce a figure like Figure 3 from that linked paper.
Lawrence
Thank you everyone for your valuable materials and comments. Currently, I can use a maximum of 8 nodes on a computer system with a 10 Gb InfiniBand network. I am applying to use all the nodes in this computer system (about 300 nodes). It will take some time. I also hope 300 nodes are enough to check the effectiveness of a simple nonblocking computation test where the inner product overlaps the matrix-vector multiplication. Have a great weekend!! Viet On Fri, Jan 29, 2021 at 1:35 AM Jed Brown <[email protected]> wrote:
Lawrence Mitchell <[email protected]> writes:
On 27 Jan 2021, at 16:30, Matthew Knepley <[email protected]> wrote:
This is very important to do _first_. It would probably only take you a day to measure the Allreduce time on your target, say the whole machine you run on.
Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me.
I've been asking for it for years. They say if you care, you should just go run it. Never mind how wasteful that is, and the time commitment to doing so. I think they often avoid making a commitment because latency is super variable (depending on the partition you get and what other jobs are running elsewhere on the machine; Blue Gene famously didn't have that problem).
Meanwhile, latency on cloud providers keeps dropping and they're sure to beat conventional HPC centers to publishing a dashboard of expected latency for different configurations.
This page illustrates how hardware reductions scale much better than log(P).
https://www.mcs.anl.gov/~fischer/gop/
Although I guess Figure 2 from here https://www.mcs.anl.gov/papers/P5347-0515.pdf probably gives me a clue.
Viet, I suspect that Matt thinks you should try and produce a figure like Figure 3 from that linked paper.
Lawrence
On , 2021Jan28, at 10:35, Jed Brown <[email protected]<mailto:[email protected]>> wrote: Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me. I've been asking for it for years. Curious. Why? What would it tell you? I mean, other than “42”. I’ll ask around how this is done in TX. Considering all the tinkering Intel does to their MPI and what with all the UCX crap, we keep a close eye on MPI performance, especially collectives. latency is super variable (depending on the partition you get and what other jobs are running elsewhere on the machine; Blue Gene famously didn't have that problem). The harsh truth of fat-trees with static routing: https://arxiv.org/abs/1909.12195 Btw, I think we’ve turned off dynamic routing for now because of too many problems. Victor.
Victor Eijkhout <[email protected]> writes:
On , 2021Jan28, at 10:35, Jed Brown <[email protected]<mailto:[email protected]>> wrote:
Why plots like this are not _absolutely standard_ on all HPC sites' webpages is a source of continuing mystery to me.
I've been asking for it for years.
Curious. Why? What would it tell you? I mean, other than “42”.
Quite a bit about what sort of performance and variability can be expected for latency-sensitive jobs. Facilities publish HPL numbers despite far less relevance to application needs. People currently write proposals for Supercomputer X, and only find post-award that it delivers latency regressions compared to Humblecomputer Y and thus isn't suitable for the planned workload, so they fabricate a different workload to use the allocated hours.
participants (7)
-
Barry Smith -
Jed Brown -
Lawrence Mitchell -
Matthew Knepley -
Patrick Sanan -
Victor Eijkhout -
Viet H.Q.H.