Re: [JLSE-Intel-NDA] ScaLAPACK on KNL?
Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania
Hi Jeff, Thanks a lot for your help. I am trying to figure it out, but currently I am getting an error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. ?when I run my program. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 5:35 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania
I have not seen that one before but I have a good idea where it comes from. Have you looked at https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/5903... or https://software.intel.com/en-us/articles/using-intel-mkl-mpi-wrapper-with-t... yet? Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:19 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks a lot for your help. I am trying to figure it out, but currently I am getting an error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. when I run my program. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 5:35 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania
Hi Jeff, Thanks for the links. So, its the MPI wrapper that needs to be resolved. I tried to follow the option that talks about adding -Wl,-Bstatic/dynamic and currently have this: MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,-Bstatic -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group -Wl,-Bdynamic Still I am not able to get the blacs_library* symbols below to appear with my executable: 21 00000000036a5f60 b blacs_library_loader_lock 22 00000000036a5f64 b blacs_library_lock_flag? Beyond that it still is confusing. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 6:25 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? I have not seen that one before but I have a good idea where it comes from. Have you looked at https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/5903... or https://software.intel.com/en-us/articles/using-intel-mkl-mpi-wrapper-with-t... yet? Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:19 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks a lot for your help. I am trying to figure it out, but currently I am getting an error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. ?when I run my program. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 5:35 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania
You might post on the Intel Forum about this. It's going to be at least two weeks before I have time to do any debugging on JLSE. Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:59 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks for the links. So, its the MPI wrapper that needs to be resolved. I tried to follow the option that talks about adding -Wl,-Bstatic/dynamic and currently have this: MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,-Bstatic -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group -Wl,-Bdynamic Still I am not able to get the blacs_library* symbols below to appear with my executable: 21 00000000036a5f60 b blacs_library_loader_lock 22 00000000036a5f64 b blacs_library_lock_flag Beyond that it still is confusing. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 6:25 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? I have not seen that one before but I have a good idea where it comes from. Have you looked at https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/5903... or https://software.intel.com/en-us/articles/using-intel-mkl-mpi-wrapper-with-t... yet? Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:19 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks a lot for your help. I am trying to figure it out, but currently I am getting an error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. when I run my program. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 5:35 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania
Thanks Jeff. It worked for me when I dynamically linked the blacs library. So my compile line looked as: mpiicc -mkl ex.c /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,-Bstatic -Wl,--start-group /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_intel_lp64.a /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_core.a /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -Wl,-Bdynamic -o mult -lmkl_blacs_intelmpi_lp64 Best regards, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 7:01 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? You might post on the Intel Forum about this. It's going to be at least two weeks before I have time to do any debugging on JLSE. Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:59 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks for the links. So, its the MPI wrapper that needs to be resolved. I tried to follow the option that talks about adding -Wl,-Bstatic/dynamic and currently have this: MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,-Bstatic -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group -Wl,-Bdynamic Still I am not able to get the blacs_library* symbols below to appear with my executable: 21 00000000036a5f60 b blacs_library_loader_lock 22 00000000036a5f64 b blacs_library_lock_flag Beyond that it still is confusing. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 6:25 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? I have not seen that one before but I have a good idea where it comes from. Have you looked at https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/5903... or https://software.intel.com/en-us/articles/using-intel-mkl-mpi-wrapper-with-t... yet? Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:19 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks a lot for your help. I am trying to figure it out, but currently I am getting an error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. when I run my program. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 5:35 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania
Thanks for letting me know. I'll try to remember this link line in case someone else asks about it in the future. Best, Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 4:59 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Thanks Jeff. It worked for me when I dynamically linked the blacs library. So my compile line looked as: mpiicc -mkl ex.c /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,-Bstatic -Wl,--start-group /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_intel_lp64.a /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_core.a /soft/compilers/intel/compilers_and_libraries_2017.1.132/linux/mkl/lib/intel64/libmkl_intel_thread.a -Wl,--end-group -Wl,-Bdynamic -o mult -lmkl_blacs_intelmpi_lp64 Best regards, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 7:01 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? You might post on the Intel Forum about this. It's going to be at least two weeks before I have time to do any debugging on JLSE. Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:59 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks for the links. So, its the MPI wrapper that needs to be resolved. I tried to follow the option that talks about adding -Wl,-Bstatic/dynamic and currently have this: MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,-Bstatic -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group -Wl,-Bdynamic Still I am not able to get the blacs_library* symbols below to appear with my executable: 21 00000000036a5f60 b blacs_library_loader_lock 22 00000000036a5f64 b blacs_library_lock_flag Beyond that it still is confusing. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 6:25 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? I have not seen that one before but I have a good idea where it comes from. Have you looked at https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/5903... or https://software.intel.com/en-us/articles/using-intel-mkl-mpi-wrapper-with-t... yet? Jeff From: "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 3:19 PM To: "Hammond, Jeff R" <[email protected]>, "[email protected]" <[email protected]> Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi Jeff, Thanks a lot for your help. I am trying to figure it out, but currently I am getting an error: Intel MKL FATAL ERROR: Cannot load symbol MKLMPI_Get_wrappers. when I run my program. Thanks, Tania ________________________________ From: Hammond, Jeff R <[email protected]> Sent: Monday, February 20, 2017 5:35 PM To: Banerjee,Tania; [email protected] Subject: Re: [JLSE-Intel-NDA] ScaLAPACK on KNL? Yes. It is in $MKLROOT if you source the Intel toolchain script (e.g. /soft/compilers/intel/bin/compilervars.sh). You can link single-threaded ScaLAPACK with -mkl=cluster or link it manually as follows (example is ScaLAPACK with threaded BLAS/LAPACK): MKL_STUFF = ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a \ -Wl,--start-group \ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \ ${MKLROOT}/lib/intel64/libmkl_core.a \ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a \ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.a \ -Wl,--end-group FFTW_INCL = -I${MKLROOT}/include/fftw/ FFTW_LIBS = ${MKL_STUFF} BLAS_LIBS = ${MKL_STUFF} SCALAPACK_LIBS = ${MKL_STUFF} Change lp64 to ilp64 as appropriate. Jeff From: <[email protected]> on behalf of "Banerjee,Tania" <[email protected]> Date: Monday, February 20, 2017 at 2:12 PM To: "[email protected]" <[email protected]> Subject: [JLSE-Intel-NDA] ScaLAPACK on KNL? Hi, Is MKL ScaLAPACK installed on JLSE for use on KNL? Thanks, Tania
participants (2)
-
Banerjee,Tania -
Hammond, Jeff R