MKL BLAS interface inconsistency
Dear petsc team, I've been struggling with this problem for a long time and seeking for your advice now. Let's take simple petsc program: #include <petscksp.h> int main(int argc,char **args) { Vec b; PetscReal norm; PetscInitialize(&argc,&args,(char *)0,NULL); VecCreateSeq(PETSC_COMM_SELF,100,&b); VecSet(b,2.0); VecNorm(b,NORM_1_AND_2,&norm); return 0; } This program works well if I compile petsc with non-mkl blas/lapack. However, if I compile petsc with mkl blas/lapack this program crashes: zdotc, FP=7fff8f121e30 VecNorm_Seq, FP=7fff8f121f90 VecNorm_Seq, FP=7fff8f1220f0 VecNorm, FP=7fff8f122230 main, FP=7fff8f122290 It crashes in zdotc. You call this routine as following: *z = BLASdot_(&bn,xx,&one,xx,&one); When I look at the zdotc interface in mkl.h I see: void ZDOTC(MKL_Complex16 *pres, const MKL_INT *n, const MKL_Complex16 *x, const MKL_INT *incx, const MKL_Complex16 *y, const MKL_INT *incy); I also found example here: http://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl... There are 6 input parameters. But in "classical" BLAS implementation 5 input and 1 output. For example, NORM_1 works well since interface to dzasum is the same. Any ideas? -- Regards, Alexander
This (dirty) patch solves the problem: diff -r a3e9ca59ab58 src/vec/vec/impls/seq/bvec2.c --- a/src/vec/vec/impls/seq/bvec2.c Tue Jan 17 22:04:05 2012 -0600 +++ b/src/vec/vec/impls/seq/bvec2.c Thu Jan 19 17:28:39 2012 +0100 @@ -232,12 +232,13 @@ PetscErrorCode ierr; PetscInt n = xin->map->n; PetscBLASInt one = 1, bn = PetscBLASIntCast(n); + PetscScalar cnorm; PetscFunctionBegin; if (type == NORM_2 || type == NORM_FROBENIUS) { ierr = VecGetArrayRead(xin,&xx);CHKERRQ(ierr); - *z = BLASdot_(&bn,xx,&one,xx,&one); - *z = PetscSqrtReal(*z); + zdotc(&cnorm,&bn,xx,&one,xx,&one); + *z = PetscSqrtReal(PetscAbsScalar(cnorm)); ierr = VecRestoreArrayRead(xin,&xx);CHKERRQ(ierr); ierr = PetscLogFlops(PetscMax(2.0*n-1,0.0));CHKERRQ(ierr); } else if (type == NORM_INFINITY) { The same is applied to mpi vector implementation from /petsc-dev/src/vec/vec/impls/mpi/pvec2.c Of course it works only if one uses Intel MKL BLAS/LAPACK and double complex arithmetics. Which is my case. On 19.01.2012 15:30, Alexander Grayver wrote:
Dear petsc team,
I've been struggling with this problem for a long time and seeking for your advice now. Let's take simple petsc program:
#include <petscksp.h> int main(int argc,char **args) { Vec b; PetscReal norm;
PetscInitialize(&argc,&args,(char *)0,NULL);
VecCreateSeq(PETSC_COMM_SELF,100,&b); VecSet(b,2.0); VecNorm(b,NORM_1_AND_2,&norm);
return 0; }
This program works well if I compile petsc with non-mkl blas/lapack. However, if I compile petsc with mkl blas/lapack this program crashes:
zdotc, FP=7fff8f121e30 VecNorm_Seq, FP=7fff8f121f90 VecNorm_Seq, FP=7fff8f1220f0 VecNorm, FP=7fff8f122230 main, FP=7fff8f122290
It crashes in zdotc. You call this routine as following:
*z = BLASdot_(&bn,xx,&one,xx,&one);
When I look at the zdotc interface in mkl.h I see: void ZDOTC(MKL_Complex16 *pres, const MKL_INT *n, const MKL_Complex16 *x, const MKL_INT *incx, const MKL_Complex16 *y, const MKL_INT *incy);
I also found example here: http://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl... There are 6 input parameters. But in "classical" BLAS implementation 5 input and 1 output.
For example, NORM_1 works well since interface to dzasum is the same.
Any ideas?
-- Regards, Alexander
Hello, The inconsistency is still there. I pulled petsc-dev today and came across with the same problem: src/vec/vec/impls/seq/bvec2.c 239: *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one)); 240: *z = PetscSqrtReal(*z); This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. This check didn't work for me apparently. Any ideas why? Regards, Alexander On 19.01.2012 17:55, Alexander Grayver wrote:
This (dirty) patch solves the problem:
diff -r a3e9ca59ab58 src/vec/vec/impls/seq/bvec2.c --- a/src/vec/vec/impls/seq/bvec2.c Tue Jan 17 22:04:05 2012 -0600 +++ b/src/vec/vec/impls/seq/bvec2.c Thu Jan 19 17:28:39 2012 +0100 @@ -232,12 +232,13 @@ PetscErrorCode ierr; PetscInt n = xin->map->n; PetscBLASInt one = 1, bn = PetscBLASIntCast(n); + PetscScalar cnorm;
PetscFunctionBegin; if (type == NORM_2 || type == NORM_FROBENIUS) { ierr = VecGetArrayRead(xin,&xx);CHKERRQ(ierr); - *z = BLASdot_(&bn,xx,&one,xx,&one); - *z = PetscSqrtReal(*z); + zdotc(&cnorm,&bn,xx,&one,xx,&one); + *z = PetscSqrtReal(PetscAbsScalar(cnorm)); ierr = VecRestoreArrayRead(xin,&xx);CHKERRQ(ierr); ierr = PetscLogFlops(PetscMax(2.0*n-1,0.0));CHKERRQ(ierr); } else if (type == NORM_INFINITY) {
The same is applied to mpi vector implementation from /petsc-dev/src/vec/vec/impls/mpi/pvec2.c Of course it works only if one uses Intel MKL BLAS/LAPACK and double complex arithmetics. Which is my case.
On 19.01.2012 15:30, Alexander Grayver wrote:
Dear petsc team,
I've been struggling with this problem for a long time and seeking for your advice now. Let's take simple petsc program:
#include <petscksp.h> int main(int argc,char **args) { Vec b; PetscReal norm;
PetscInitialize(&argc,&args,(char *)0,NULL);
VecCreateSeq(PETSC_COMM_SELF,100,&b); VecSet(b,2.0); VecNorm(b,NORM_1_AND_2,&norm);
return 0; }
This program works well if I compile petsc with non-mkl blas/lapack. However, if I compile petsc with mkl blas/lapack this program crashes:
zdotc, FP=7fff8f121e30 VecNorm_Seq, FP=7fff8f121f90 VecNorm_Seq, FP=7fff8f1220f0 VecNorm, FP=7fff8f122230 main, FP=7fff8f122290
It crashes in zdotc. You call this routine as following:
*z = BLASdot_(&bn,xx,&one,xx,&one);
When I look at the zdotc interface in mkl.h I see: void ZDOTC(MKL_Complex16 *pres, const MKL_INT *n, const MKL_Complex16 *x, const MKL_INT *incx, const MKL_Complex16 *y, const MKL_INT *incy);
I also found example here: http://software.intel.com/en-us/articles/intel-math-kernel-library-intel-mkl... There are 6 input parameters. But in "classical" BLAS implementation 5 input and 1 output.
For example, NORM_1 works well since interface to dzasum is the same.
Any ideas?
-- Regards, Alexander
On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <[email protected]>wrote:
The inconsistency is still there. I pulled petsc-dev today and came across with the same problem:
src/vec/vec/impls/seq/bvec2.c 239: *z = PetscRealPart(BLASdot_(&bn,xx,**&one,xx,&one)); 240: *z = PetscSqrtReal(*z);
This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. This check didn't work for me apparently. Any ideas why?
As always, we need configure.log. In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure. #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT EXTERN_C_BEGIN extern void zdotc_(PetscScalar *,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*); PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) { PetscScalar tmpz; zdotc_(&tmpz,n,x,sx,y,sy); return tmpz; } EXTERN_C_END #else # define BLASdot_ zdotc_ #endif
On 05.04.2012 14:13, Jed Brown wrote:
On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <[email protected] <mailto:[email protected]>> wrote:
The inconsistency is still there. I pulled petsc-dev today and came across with the same problem:
src/vec/vec/impls/seq/bvec2.c 239: *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one)); 240: *z = PetscSqrtReal(*z);
This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. This check didn't work for me apparently. Any ideas why?
As always, we need configure.log.
Please find it attached.
In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
This I can see...
#ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT EXTERN_C_BEGIN extern void zdotc_(PetscScalar *,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*); PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) { PetscScalar tmpz; zdotc_(&tmpz,n,x,sx,y,sy); return tmpz; } EXTERN_C_END #else # define BLASdot_ zdotc_ #endif
-- Regards, Alexander
On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver <[email protected]>wrote:
** On 05.04.2012 14:13, Jed Brown wrote:
On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <[email protected]>wrote:
The inconsistency is still there. I pulled petsc-dev today and came across with the same problem:
src/vec/vec/impls/seq/bvec2.c 239: *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one)); 240: *z = PetscSqrtReal(*z);
This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. This check didn't work for me apparently. Any ideas why?
As always, we need configure.log.
Please find it attached.
Please ALWAYS ALWAYS ALWAYS send configure.log to [email protected]. They are too big for the public list. Second, your problem is that we get a link error tying to test this so it looks like your BLAS have no complex implementations: sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl sh: Possible ERROR while running linker: /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' output: ret = 256 error message = {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' } Matt
In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
This I can see...
#ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT EXTERN_C_BEGIN extern void zdotc_(PetscScalar *,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*); PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) { PetscScalar tmpz; zdotc_(&tmpz,n,x,sx,y,sy); return tmpz; } EXTERN_C_END #else # define BLASdot_ zdotc_ #endif
-- Regards, Alexander
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
On 05.04.2012 14:42, Matthew Knepley wrote:
On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver <[email protected] <mailto:[email protected]>> wrote:
On 05.04.2012 14:13, Jed Brown wrote:
On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <[email protected] <mailto:[email protected]>> wrote:
The inconsistency is still there. I pulled petsc-dev today and came across with the same problem:
src/vec/vec/impls/seq/bvec2.c 239: *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one)); 240: *z = PetscSqrtReal(*z);
This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. This check didn't work for me apparently. Any ideas why?
As always, we need configure.log.
Please find it attached.
Please ALWAYS ALWAYS ALWAYS send configure.log to [email protected] <mailto:[email protected]>. They are too big for the public list.
Sorry, next time I will.
Second, your problem is that we get a link error tying to test this so it looks like your BLAS have no complex implementations:
It does have zdotc. Old petsc-dev works with it fine (after I patched vec implementaton myself to get around this issue). home> nm /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a | grep zdotc cblas_zdotci_lp64.o: 0000000000000000 T cblas_zdotci_sub cblas_zdotc_lp64.o: 0000000000000000 T cblas_zdotc_sub U zdotc _zdotci_lp64.o: U mkl_blas_zdotci 0000000000000000 T zdotci 0000000000000000 T zdotci_ _zdotc_lp64.o: U mkl_blas_zdotc 0000000000000000 T zdotc 0000000000000000 T zdotc_ home>
sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl sh: Possible ERROR while running linker: /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' output: ret = 256 error message = {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' }
Matt
In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
This I can see...
#ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT EXTERN_C_BEGIN extern void zdotc_(PetscScalar *,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*); PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) { PetscScalar tmpz; zdotc_(&tmpz,n,x,sx,y,sy); return tmpz; } EXTERN_C_END #else # define BLASdot_ zdotc_ #endif
-- Regards, Alexander
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- Regards, Alexander
Matt, It doesn't look like that link line is even linking against the MKL libraries? Barry On Apr 5, 2012, at 7:54 AM, Alexander Grayver wrote:
On 05.04.2012 14:42, Matthew Knepley wrote:
On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver <[email protected]> wrote: On 05.04.2012 14:13, Jed Brown wrote:
On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <[email protected]> wrote: The inconsistency is still there. I pulled petsc-dev today and came across with the same problem:
src/vec/vec/impls/seq/bvec2.c 239: *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one)); 240: *z = PetscSqrtReal(*z);
This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. This check didn't work for me apparently. Any ideas why?
As always, we need configure.log.
Please find it attached.
Please ALWAYS ALWAYS ALWAYS send configure.log to [email protected]. They are too big for the public list.
Sorry, next time I will.
Second, your problem is that we get a link error tying to test this so it looks like your BLAS have no complex implementations:
It does have zdotc. Old petsc-dev works with it fine (after I patched vec implementaton myself to get around this issue).
home> nm /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a | grep zdotc cblas_zdotci_lp64.o: 0000000000000000 T cblas_zdotci_sub cblas_zdotc_lp64.o: 0000000000000000 T cblas_zdotc_sub U zdotc _zdotci_lp64.o: U mkl_blas_zdotci 0000000000000000 T zdotci 0000000000000000 T zdotci_ _zdotc_lp64.o: U mkl_blas_zdotc 0000000000000000 T zdotc 0000000000000000 T zdotc_ home>
sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl sh: Possible ERROR while running linker: /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' output: ret = 256 error message = {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' }
Matt
In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
This I can see...
#ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT EXTERN_C_BEGIN extern void zdotc_(PetscScalar *,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*); PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) { PetscScalar tmpz; zdotc_(&tmpz,n,x,sx,y,sy); return tmpz; } EXTERN_C_END #else # define BLASdot_ zdotc_ #endif
-- Regards, Alexander
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- Regards, Alexander
On Thu, Apr 5, 2012 at 8:05 AM, Barry Smith <[email protected]> wrote:
Matt,
It doesn't look like that link line is even linking against the MKL libraries?
Dang, this is a bug with specifying --with-blas-lapack-lib, since it all gets dumped in self.lapackLib, then self.blasLib is empty. Pushed a fix to BuildSystem. Testing now. Matt
Barry
On Apr 5, 2012, at 7:54 AM, Alexander Grayver wrote:
On 05.04.2012 14:42, Matthew Knepley wrote:
On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver < [email protected]> wrote: On 05.04.2012 14:13, Jed Brown wrote:
On Thu, Apr 5, 2012 at 04:54, Alexander Grayver < [email protected]> wrote: The inconsistency is still there. I pulled petsc-dev today and came across with the same problem:
src/vec/vec/impls/seq/bvec2.c 239: *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one)); 240: *z = PetscSqrtReal(*z);
This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. This check didn't work for me apparently. Any ideas why?
As always, we need configure.log.
Please find it attached.
Please ALWAYS ALWAYS ALWAYS send configure.log to [email protected]. They are too big for the public list.
Sorry, next time I will.
Second, your problem is that we get a link error tying to test this so
it looks like
your BLAS have no complex implementations:
It does have zdotc. Old petsc-dev works with it fine (after I patched vec implementaton myself to get around this issue).
home> nm /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a | grep zdotc cblas_zdotci_lp64.o: 0000000000000000 T cblas_zdotci_sub cblas_zdotc_lp64.o: 0000000000000000 T cblas_zdotc_sub U zdotc _zdotci_lp64.o: U mkl_blas_zdotci 0000000000000000 T zdotci 0000000000000000 T zdotci_ _zdotc_lp64.o: U mkl_blas_zdotc 0000000000000000 T zdotc 0000000000000000 T zdotc_ home>
sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o
/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl
Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl sh: Possible ERROR while running linker: /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' output: ret = 256 error message = {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' }
Matt
In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure.
This I can see...
#ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT EXTERN_C_BEGIN extern void zdotc_(PetscScalar *,const PetscBLASInt*,const
PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*);
PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) { PetscScalar tmpz; zdotc_(&tmpz,n,x,sx,y,sy); return tmpz; } EXTERN_C_END #else # define BLASdot_ zdotc_ #endif
-- Regards, Alexander
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- Regards, Alexander
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
Now everything works! Thanks for such a quick fix. On 05.04.2012 15:53, Matthew Knepley wrote:
On Thu, Apr 5, 2012 at 8:05 AM, Barry Smith <[email protected] <mailto:[email protected]>> wrote:
Matt,
It doesn't look like that link line is even linking against the MKL libraries?
Dang, this is a bug with specifying --with-blas-lapack-lib, since it all gets dumped in self.lapackLib, then self.blasLib is empty. Pushed a fix to BuildSystem. Testing now.
Matt
Barry
On Apr 5, 2012, at 7:54 AM, Alexander Grayver wrote:
> On 05.04.2012 14:42, Matthew Knepley wrote: >> On Thu, Apr 5, 2012 at 7:22 AM, Alexander Grayver <[email protected] <mailto:[email protected]>> wrote: >> On 05.04.2012 14:13, Jed Brown wrote: >>> On Thu, Apr 5, 2012 at 04:54, Alexander Grayver <[email protected] <mailto:[email protected]>> wrote: >>> The inconsistency is still there. I pulled petsc-dev today and came across with the same problem: >>> >>> src/vec/vec/impls/seq/bvec2.c >>> 239: *z = PetscRealPart(BLASdot_(&bn,xx,&one,xx,&one)); >>> 240: *z = PetscSqrtReal(*z); >>> >>> This doesn't work with Intel MKL. However in one of the mailing list discussions it was said there is now check for that. >>> This check didn't work for me apparently. Any ideas why? >>> >>> As always, we need configure.log. >>> >> Please find it attached. >> >> Please ALWAYS ALWAYS ALWAYS send configure.log to [email protected] <mailto:[email protected]>. They are too >> big for the public list. > > Sorry, next time I will. > >> >> Second, your problem is that we get a link error tying to test this so it looks like >> your BLAS have no complex implementations: > > It does have zdotc. Old petsc-dev works with it fine (after I patched vec implementaton myself to get around this issue). > > home> nm /opt/intel/Compiler/11.1/072/mkl/lib/em64t/libmkl_intel_lp64.a | grep zdotc > cblas_zdotci_lp64.o: > 0000000000000000 T cblas_zdotci_sub > cblas_zdotc_lp64.o: > 0000000000000000 T cblas_zdotc_sub > U zdotc > _zdotci_lp64.o: > U mkl_blas_zdotci > 0000000000000000 T zdotci > 0000000000000000 T zdotci_ > _zdotc_lp64.o: > U mkl_blas_zdotc > 0000000000000000 T zdotc > 0000000000000000 T zdotc_ > home> > >> >> sh: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl >> Executing: /opt/mpi/intel/openmpi-1.4.2/bin/mpicc -o /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest -wd1572 -Qoption,cpp,--extended_float_type -g /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o -ldl -L/opt/mpi/intel/openmpi-1.4.2/lib -lmpi -lopen-rte -lopen-pal -lnsl -lutil -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -limf -lsvml -lipgo -ldecimal -lgcc_s -lirc -lpthread -lirc_s -L/opt/mpi/intel/openmpi-1.4.2/lib -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L//opt/intel/Compiler/11.1/072/lib/intel64 -L/usr/lib64/gcc/x86_64-suse-linux/4.3 -L/opt/intel/Compiler/11.1/072/ipp/em64t/lib -L/opt/intel/Compiler/11.1/072/mkl/interfaces/fftw3xf -L/usr/x86_64-suse-linux/lib -ldl >> sh: >> Possible ERROR while running linker: /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': >> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' >> output: ret = 256 >> error message = {/tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.o: In function `main': >> /tmp/petsc-8AEO1f/config.packages.BlasLapack/conftest.c:14: undefined reference to `zdotc_' >> } >> >> >> Matt >> >> In include/petscblaslapack_uscore.h (and other places), you will find the following. So we need to know why PETSC_COMPLEX_DOT_RESULT_ARGUMENT was not found by configure. >> >> This I can see... >> >> >>> >>> #ifdef PETSC_COMPLEX_DOT_RESULT_ARGUMENT >>> EXTERN_C_BEGIN >>> extern void zdotc_(PetscScalar *,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*,const PetscScalar*,const PetscBLASInt*); >>> PETSC_STATIC_INLINE PetscScalar BLASdot_(const PetscBLASInt *n,const PetscScalar *x,const PetscBLASInt *sx,const PetscScalar *y,const PetscBLASInt *sy) { >>> PetscScalar tmpz; >>> zdotc_(&tmpz,n,x,sx,y,sy); >>> return tmpz; >>> } >>> EXTERN_C_END >>> #else >>> # define BLASdot_ zdotc_ >>> #endif >> >> >> >> -- >> Regards, >> Alexander >> >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener > > > -- > Regards, > Alexander >
-- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
-- Regards, Alexander
participants (4)
-
Alexander Grayver -
Barry Smith -
Jed Brown -
Matthew Knepley