February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Intel compilers + CUDA
by John Fettig 24 Feb '12

24 Feb '12

On Fri, Feb 24, 2012 at 11:56 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > for one I mistyped the option - it should be: --with-cudac='nvcc -ccbin > icc' > > But then - should it not be c++? [as cuda internally uses g++ - not gcc]. > So > > --with-cudac='nvcc -ccbin icpc'. > > But this gives me all kinds of wierd errors in configure.log.. Maybe > it won't work [with intel compilers].. > CUDA .cu files are written in C with some extensions (that are handled by nvcc and not the passed compiler), so I think you have to use a C compiler. You can't write C++ in a .cu file. John

1 0

Re: [petsc-dev] Intel compilers + CUDA
by Satish Balay 24 Feb '12

24 Feb '12

On Fri, 24 Feb 2012, Jed Brown wrote: > On Fri, Feb 24, 2012 at 10:51, John Fettig <john.fettig(a)gmail.com> wrote: > > > Unfortunately it doesn't work. I'll send the logs to petsc-maint, but > > here's the error. If you remove these lines: > > > > #ifndef PETSC_HAVE___INT64 > > #define PETSC_HAVE___INT64 1 > > #endif > > > > What system uses __int64? If we have stdint.h, we should use int64_t > because that is the standard. We added that for windows [ms compilers]. But then - intel compilers also support it [and configure sets this flag for icc]. When nvcc is used [with gcc] - this flag conflicts with it. Satish > > > > > > from $PETSC_ARCH/include/petscconf.h, it builds ok (and make test > > passes). Here's the error: > > > > nvcc -O -arch=sm_20 -c --compiler-options=-fPIC > > -I/home/jfe/local/petsc-dev/include > > -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include > > -I/usr/local/cuda/include > > -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include/txpetscgpu/include > > -I/usr/local/encap/platform_mpi-8.01/include > > -D__INSDIR__=src/vec/vec/impls/seq/seqcusp/ veccusp.cu > > /home/jfe/local/petsc-dev/include/petscsys.h(219): error: identifier > > "__int64" is undefined > > > > John > > > > > > On Fri, Feb 24, 2012 at 10:53 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > >> >>>>>>>>>> > >> --compiler-bindir <directory> (-ccbin) > >> Specify the directory in which the compiler executable (Microsoft > >> Visual Studion cl, or a gcc derivative) resides. By default, this > >> executable is expected in the current executable search path. > >> <<<<<<<< > >> > >> It says it will look for the default compiler in the specified path - > >> and doesn't mention that it can switch the compiler. Ah - the manpage > >> is misleading.. Looks like '--ccbin icc' works.. > >> > >> balay@bb30:~/junk>nvcc -v -ccbin icc sizeof.c > >> #$ _SPACE_= > >> #$ _CUDART_=cudart > >> #$ _HERE_=/usr/local/cuda/bin > >> #$ _THERE_=/usr/local/cuda/bin > >> #$ _TARGET_SIZE_=64 > >> #$ TOP=/usr/local/cuda/bin/.. > >> #$ > >> LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/local/cuda/bin/../extools/lib:/soft/intel/11.0.081/lib/intel64:/soft/intel/11.0.081/lib/intel64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 > >> #$ > >> PATH=/usr/local/cuda/bin/../open64/bin:/usr/local/cuda/bin:/soft/intel/11.0.081/bin/intel64:/soft/intel/11.0.081/bin/intel64:/usr/mpi/gcc/mvapich2-1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/cuda/bin:/usr/local/cuda/bin > >> #$ INCLUDES="-I/usr/local/cuda/bin/../include" > >> "-I/usr/local/cuda/bin/../include/cudart" > >> #$ LIBRARIES= "-L/usr/local/cuda/bin/../lib64" -lcudart > >> #$ CUDAFE_FLAGS= > >> #$ OPENCC_FLAGS= > >> #$ PTXAS_FLAGS= > >> #$ icc -c -x c "-I/usr/local/cuda/bin/../include" > >> "-I/usr/local/cuda/bin/../include/cudart" -m64 -o > >> "/tmp/tmpxft_00000199_00000000-1_sizeof.o" "sizeof.c" > >> #$ icpc -m64 -o "a.out" -Wl,--start-group > >> "/tmp/tmpxft_00000199_00000000-1_sizeof.o" > >> "-L/usr/local/cuda/bin/../lib64" -lcudart -Wl,--end-group > >> balay@bb30:~/junk> > >> > >> > >> So '--with-cc=icc' --with-nvcc='nvcc --ccbin icc' should work.. Will > >> check.. > >> > >> Satish > >> > >> On Fri, 24 Feb 2012, John Fettig wrote: > >> > >> > Some time ago I asked about using icc with nvcc. You can use icc with > >> nvcc > >> > by passing it as an argument: > >> > > >> > nvcc -ccbin /path/to/icc > >> > > >> > Has anybody tried building PETSc with CUDA this way? > >> > > >> > John > >> > > >> > >> > > >

1 0

Re: [petsc-dev] Intel compilers + CUDA
by Satish Balay 24 Feb '12

24 Feb '12

for one I mistyped the option - it should be: --with-cudac='nvcc -ccbin icc' But then - should it not be c++? [as cuda internally uses g++ - not gcc]. So --with-cudac='nvcc -ccbin icpc'. But this gives me all kinds of wierd errors in configure.log.. Maybe it won't work [with intel compilers].. Satish On Fri, 24 Feb 2012, John Fettig wrote: > Unfortunately it doesn't work. I'll send the logs to petsc-maint, but > here's the error. If you remove these lines: > > #ifndef PETSC_HAVE___INT64 > #define PETSC_HAVE___INT64 1 > #endif > > from $PETSC_ARCH/include/petscconf.h, it builds ok (and make test passes). > Here's the error: > > nvcc -O -arch=sm_20 -c --compiler-options=-fPIC > -I/home/jfe/local/petsc-dev/include > -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include > -I/usr/local/cuda/include > -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include/txpetscgpu/include > -I/usr/local/encap/platform_mpi-8.01/include > -D__INSDIR__=src/vec/vec/impls/seq/seqcusp/ veccusp.cu > /home/jfe/local/petsc-dev/include/petscsys.h(219): error: identifier > "__int64" is undefined > > John > > On Fri, Feb 24, 2012 at 10:53 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > > > >>>>>>>>>> > > --compiler-bindir <directory> (-ccbin) > > Specify the directory in which the compiler executable (Microsoft > > Visual Studion cl, or a gcc derivative) resides. By default, this > > executable is expected in the current executable search path. > > <<<<<<<< > > > > It says it will look for the default compiler in the specified path - > > and doesn't mention that it can switch the compiler. Ah - the manpage > > is misleading.. Looks like '--ccbin icc' works.. > > > > balay@bb30:~/junk>nvcc -v -ccbin icc sizeof.c > > #$ _SPACE_= > > #$ _CUDART_=cudart > > #$ _HERE_=/usr/local/cuda/bin > > #$ _THERE_=/usr/local/cuda/bin > > #$ _TARGET_SIZE_=64 > > #$ TOP=/usr/local/cuda/bin/.. > > #$ > > LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/local/cuda/bin/../extools/lib:/soft/intel/11.0.081/lib/intel64:/soft/intel/11.0.081/lib/intel64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 > > #$ > > PATH=/usr/local/cuda/bin/../open64/bin:/usr/local/cuda/bin:/soft/intel/11.0.081/bin/intel64:/soft/intel/11.0.081/bin/intel64:/usr/mpi/gcc/mvapich2-1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/cuda/bin:/usr/local/cuda/bin > > #$ INCLUDES="-I/usr/local/cuda/bin/../include" > > "-I/usr/local/cuda/bin/../include/cudart" > > #$ LIBRARIES= "-L/usr/local/cuda/bin/../lib64" -lcudart > > #$ CUDAFE_FLAGS= > > #$ OPENCC_FLAGS= > > #$ PTXAS_FLAGS= > > #$ icc -c -x c "-I/usr/local/cuda/bin/../include" > > "-I/usr/local/cuda/bin/../include/cudart" -m64 -o > > "/tmp/tmpxft_00000199_00000000-1_sizeof.o" "sizeof.c" > > #$ icpc -m64 -o "a.out" -Wl,--start-group > > "/tmp/tmpxft_00000199_00000000-1_sizeof.o" > > "-L/usr/local/cuda/bin/../lib64" -lcudart -Wl,--end-group > > balay@bb30:~/junk> > > > > > > So '--with-cc=icc' --with-nvcc='nvcc --ccbin icc' should work.. Will > > check.. > > > > Satish > > > > On Fri, 24 Feb 2012, John Fettig wrote: > > > > > Some time ago I asked about using icc with nvcc. You can use icc with > > nvcc > > > by passing it as an argument: > > > > > > nvcc -ccbin /path/to/icc > > > > > > Has anybody tried building PETSc with CUDA this way? > > > > > > John > > > > > > > >

1 0

Re: [petsc-dev] Intel compilers + CUDA
by Jed Brown 24 Feb '12

24 Feb '12

On Fri, Feb 24, 2012 at 10:51, John Fettig <john.fettig(a)gmail.com> wrote: > Unfortunately it doesn't work. I'll send the logs to petsc-maint, but > here's the error. If you remove these lines: > > #ifndef PETSC_HAVE___INT64 > #define PETSC_HAVE___INT64 1 > #endif > What system uses __int64? If we have stdint.h, we should use int64_t because that is the standard. > > from $PETSC_ARCH/include/petscconf.h, it builds ok (and make test > passes). Here's the error: > > nvcc -O -arch=sm_20 -c --compiler-options=-fPIC > -I/home/jfe/local/petsc-dev/include > -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include > -I/usr/local/cuda/include > -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include/txpetscgpu/include > -I/usr/local/encap/platform_mpi-8.01/include > -D__INSDIR__=src/vec/vec/impls/seq/seqcusp/ veccusp.cu > /home/jfe/local/petsc-dev/include/petscsys.h(219): error: identifier > "__int64" is undefined > > John > > > On Fri, Feb 24, 2012 at 10:53 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > >> >>>>>>>>>> >> --compiler-bindir <directory> (-ccbin) >> Specify the directory in which the compiler executable (Microsoft >> Visual Studion cl, or a gcc derivative) resides. By default, this >> executable is expected in the current executable search path. >> <<<<<<<< >> >> It says it will look for the default compiler in the specified path - >> and doesn't mention that it can switch the compiler. Ah - the manpage >> is misleading.. Looks like '--ccbin icc' works.. >> >> balay@bb30:~/junk>nvcc -v -ccbin icc sizeof.c >> #$ _SPACE_= >> #$ _CUDART_=cudart >> #$ _HERE_=/usr/local/cuda/bin >> #$ _THERE_=/usr/local/cuda/bin >> #$ _TARGET_SIZE_=64 >> #$ TOP=/usr/local/cuda/bin/.. >> #$ >> LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/local/cuda/bin/../extools/lib:/soft/intel/11.0.081/lib/intel64:/soft/intel/11.0.081/lib/intel64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 >> #$ >> PATH=/usr/local/cuda/bin/../open64/bin:/usr/local/cuda/bin:/soft/intel/11.0.081/bin/intel64:/soft/intel/11.0.081/bin/intel64:/usr/mpi/gcc/mvapich2-1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/cuda/bin:/usr/local/cuda/bin >> #$ INCLUDES="-I/usr/local/cuda/bin/../include" >> "-I/usr/local/cuda/bin/../include/cudart" >> #$ LIBRARIES= "-L/usr/local/cuda/bin/../lib64" -lcudart >> #$ CUDAFE_FLAGS= >> #$ OPENCC_FLAGS= >> #$ PTXAS_FLAGS= >> #$ icc -c -x c "-I/usr/local/cuda/bin/../include" >> "-I/usr/local/cuda/bin/../include/cudart" -m64 -o >> "/tmp/tmpxft_00000199_00000000-1_sizeof.o" "sizeof.c" >> #$ icpc -m64 -o "a.out" -Wl,--start-group >> "/tmp/tmpxft_00000199_00000000-1_sizeof.o" >> "-L/usr/local/cuda/bin/../lib64" -lcudart -Wl,--end-group >> balay@bb30:~/junk> >> >> >> So '--with-cc=icc' --with-nvcc='nvcc --ccbin icc' should work.. Will >> check.. >> >> Satish >> >> On Fri, 24 Feb 2012, John Fettig wrote: >> >> > Some time ago I asked about using icc with nvcc. You can use icc with >> nvcc >> > by passing it as an argument: >> > >> > nvcc -ccbin /path/to/icc >> > >> > Has anybody tried building PETSc with CUDA this way? >> > >> > John >> > >> >> >

1 0

Re: [petsc-dev] Intel compilers + CUDA
by John Fettig 24 Feb '12

24 Feb '12

Unfortunately it doesn't work. I'll send the logs to petsc-maint, but here's the error. If you remove these lines: #ifndef PETSC_HAVE___INT64 #define PETSC_HAVE___INT64 1 #endif from $PETSC_ARCH/include/petscconf.h, it builds ok (and make test passes). Here's the error: nvcc -O -arch=sm_20 -c --compiler-options=-fPIC -I/home/jfe/local/petsc-dev/include -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include -I/usr/local/cuda/include -I/home/jfe/local/petsc-dev/intel-opt-cuda-precise/include/txpetscgpu/include -I/usr/local/encap/platform_mpi-8.01/include -D__INSDIR__=src/vec/vec/impls/seq/seqcusp/ veccusp.cu /home/jfe/local/petsc-dev/include/petscsys.h(219): error: identifier "__int64" is undefined John On Fri, Feb 24, 2012 at 10:53 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > >>>>>>>>>> > --compiler-bindir <directory> (-ccbin) > Specify the directory in which the compiler executable (Microsoft > Visual Studion cl, or a gcc derivative) resides. By default, this > executable is expected in the current executable search path. > <<<<<<<< > > It says it will look for the default compiler in the specified path - > and doesn't mention that it can switch the compiler. Ah - the manpage > is misleading.. Looks like '--ccbin icc' works.. > > balay@bb30:~/junk>nvcc -v -ccbin icc sizeof.c > #$ _SPACE_= > #$ _CUDART_=cudart > #$ _HERE_=/usr/local/cuda/bin > #$ _THERE_=/usr/local/cuda/bin > #$ _TARGET_SIZE_=64 > #$ TOP=/usr/local/cuda/bin/.. > #$ > LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/local/cuda/bin/../extools/lib:/soft/intel/11.0.081/lib/intel64:/soft/intel/11.0.081/lib/intel64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 > #$ > PATH=/usr/local/cuda/bin/../open64/bin:/usr/local/cuda/bin:/soft/intel/11.0.081/bin/intel64:/soft/intel/11.0.081/bin/intel64:/usr/mpi/gcc/mvapich2-1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/cuda/bin:/usr/local/cuda/bin > #$ INCLUDES="-I/usr/local/cuda/bin/../include" > "-I/usr/local/cuda/bin/../include/cudart" > #$ LIBRARIES= "-L/usr/local/cuda/bin/../lib64" -lcudart > #$ CUDAFE_FLAGS= > #$ OPENCC_FLAGS= > #$ PTXAS_FLAGS= > #$ icc -c -x c "-I/usr/local/cuda/bin/../include" > "-I/usr/local/cuda/bin/../include/cudart" -m64 -o > "/tmp/tmpxft_00000199_00000000-1_sizeof.o" "sizeof.c" > #$ icpc -m64 -o "a.out" -Wl,--start-group > "/tmp/tmpxft_00000199_00000000-1_sizeof.o" > "-L/usr/local/cuda/bin/../lib64" -lcudart -Wl,--end-group > balay@bb30:~/junk> > > > So '--with-cc=icc' --with-nvcc='nvcc --ccbin icc' should work.. Will > check.. > > Satish > > On Fri, 24 Feb 2012, John Fettig wrote: > > > Some time ago I asked about using icc with nvcc. You can use icc with > nvcc > > by passing it as an argument: > > > > nvcc -ccbin /path/to/icc > > > > Has anybody tried building PETSc with CUDA this way? > > > > John > > > >

1 0

Re: [petsc-dev] Intel compilers + CUDA
by Satish Balay 24 Feb '12

24 Feb '12

>>>>>>>>>> --compiler-bindir <directory> (-ccbin) Specify the directory in which the compiler executable (Microsoft Visual Studion cl, or a gcc derivative) resides. By default, this executable is expected in the current executable search path. <<<<<<<< It says it will look for the default compiler in the specified path - and doesn't mention that it can switch the compiler. Ah - the manpage is misleading.. Looks like '--ccbin icc' works.. balay@bb30:~/junk>nvcc -v -ccbin icc sizeof.c #$ _SPACE_= #$ _CUDART_=cudart #$ _HERE_=/usr/local/cuda/bin #$ _THERE_=/usr/local/cuda/bin #$ _TARGET_SIZE_=64 #$ TOP=/usr/local/cuda/bin/.. #$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/local/cuda/bin/../extools/lib:/soft/intel/11.0.081/lib/intel64:/soft/intel/11.0.081/lib/intel64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 #$ PATH=/usr/local/cuda/bin/../open64/bin:/usr/local/cuda/bin:/soft/intel/11.0.081/bin/intel64:/soft/intel/11.0.081/bin/intel64:/usr/mpi/gcc/mvapich2-1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/cuda/bin:/usr/local/cuda/bin #$ INCLUDES="-I/usr/local/cuda/bin/../include" "-I/usr/local/cuda/bin/../include/cudart" #$ LIBRARIES= "-L/usr/local/cuda/bin/../lib64" -lcudart #$ CUDAFE_FLAGS= #$ OPENCC_FLAGS= #$ PTXAS_FLAGS= #$ icc -c -x c "-I/usr/local/cuda/bin/../include" "-I/usr/local/cuda/bin/../include/cudart" -m64 -o "/tmp/tmpxft_00000199_00000000-1_sizeof.o" "sizeof.c" #$ icpc -m64 -o "a.out" -Wl,--start-group "/tmp/tmpxft_00000199_00000000-1_sizeof.o" "-L/usr/local/cuda/bin/../lib64" -lcudart -Wl,--end-group balay@bb30:~/junk> So '--with-cc=icc' --with-nvcc='nvcc --ccbin icc' should work.. Will check.. Satish On Fri, 24 Feb 2012, John Fettig wrote: > Some time ago I asked about using icc with nvcc. You can use icc with nvcc > by passing it as an argument: > > nvcc -ccbin /path/to/icc > > Has anybody tried building PETSc with CUDA this way? > > John >

1 0

Intel compilers + CUDA
by John Fettig 24 Feb '12

24 Feb '12

Some time ago I asked about using icc with nvcc. You can use icc with nvcc by passing it as an argument: nvcc -ccbin /path/to/icc Has anybody tried building PETSc with CUDA this way? John

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Jed Brown 23 Feb '12

23 Feb '12

On Thu, Feb 23, 2012 at 23:41, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > I could also send you my mpi/numactl command lines for gpu and cpu when I > am > back in the office. > Yes, please.

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Jed Brown 23 Feb '12

23 Feb '12

Okay, thanks. What is the exact command line you used to set the affinity correctly? (I want to know the semantics of what you did.) On Thu, Feb 23, 2012 at 23:37, Dave Nystrom <dnystrom1(a)comcast.net> wrote: > Jed Brown writes: > > On Thu, Feb 23, 2012 at 18:53, Nystrom, William D <wdn(a)lanl.gov> wrote: > > > > > Rerunning the CPU case with numactl results in a 25x speedup and > > > log_summary > > > results that look reasonable to me now. > > > > > > > What command are you using for this? We usually use the affinity > options to > > mpiexec instead of using numactl/taskset manually. > > I was using openmpi-1.5.4 as installed by the system admins on our testbed > cluster. I talked to a couple of our openmpi developers and they indicated > that the affinity stuff was broken in that version but should be fixed when > 1.5.5 and 1.6 come out - which should be within the next month. > > I also tried mvapich2-1.7 built with slurm and tried using the affinity > stuff > with srun. That also did not seem to work. But I should probably revisit > that and try to make sure that I really understand how to use srun. > > I was pretty surprised that getting the numa stuff right made such a huge > difference. I'm also wondering if getting the affinity right will make > much > of a difference for the gpu case. > > > Did you also set a specific memory policy? > > I'm not sure what you mean by the above question but I'm kind of new to all > this numa stuff. > > > Which Linux kernel is this? > > The OS was the latest beta of TOSS2. If I remember, I can check next time > I > am in my office. It is probably RHEL6. >

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Dave Nystrom 23 Feb '12

23 Feb '12

I could also send you my mpi/numactl command lines for gpu and cpu when I am back in the office. Dave Nystrom writes: > Jed Brown writes: > > On Thu, Feb 23, 2012 at 18:53, Nystrom, William D <wdn(a)lanl.gov> wrote: > > > > > Rerunning the CPU case with numactl results in a 25x speedup and > > > log_summary > > > results that look reasonable to me now. > > > > > > > What command are you using for this? We usually use the affinity options to > > mpiexec instead of using numactl/taskset manually. > > I was using openmpi-1.5.4 as installed by the system admins on our testbed > cluster. I talked to a couple of our openmpi developers and they indicated > that the affinity stuff was broken in that version but should be fixed when > 1.5.5 and 1.6 come out - which should be within the next month. > > I also tried mvapich2-1.7 built with slurm and tried using the affinity stuff > with srun. That also did not seem to work. But I should probably revisit > that and try to make sure that I really understand how to use srun. > > I was pretty surprised that getting the numa stuff right made such a huge > difference. I'm also wondering if getting the affinity right will make much > of a difference for the gpu case. > > > Did you also set a specific memory policy? > > I'm not sure what you mean by the above question but I'm kind of new to all > this numa stuff. > > > Which Linux kernel is this? > > The OS was the latest beta of TOSS2. If I remember, I can check next time I > am in my office. It is probably RHEL6.

1 0