March 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] CMake-assisted CUDA builds ready
by Karl Rupp 26 Mar '13

26 Mar '13

Hi Jose, which host compiler do you use? nvcc seems to have problems with double-quoted flags such as " -O -arch=sm_13 ". Thanks and best regards, Karli On 03/26/2013 11:03 AM, Jose E. Roman wrote: > > El 26/03/2013, a las 16:19, Karl Rupp escribiï¿½: > >> Hi Jose, >> >> alright, the problem should be fixed in 'next' now. The problem was in CMake not consistently interpreting escaped double quotes. >> >> Best regards, >> Karli > > Not yet solved: > > > /usr/local/cuda/bin/nvcc /home/jroman/soft/petsc/src/mat/impls/aij/mpi/mpicusp/mpiaijAssemble.cu " -O -arch=sm_13 " "--compiler-options ,-fPIC,-Wall,-Wwrite-strings,-Wno-strict-aliasing,-Wno-unknown-pragmas,-O" -m64 -D__INSDIR__= -Dpetsc_EXPORTS -Xcompiler ,\"-fPIC\" -DNVCC -M -o /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/CMakeFiles/petsc_generated_mpiaijAssemble.cu.o.NVCC-depend -I/usr/local/cuda/include -I/home/jroman/soft/petsc/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include -I/usr/local/cuda/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include/txpetscgpu/include -I/home/jroman/soft/cusplibrary -I/home/jroman/soft/cusplibrary/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > nvcc fatal : Unknown option '-compiler-options ,-fPIC,-Wall,-Wwrite-strings,-Wno-strict-aliasing,-Wno-unknown-pragmas,-O' > CMake Error at CMakeFiles/petsc_generated_mpiaijAssemble.cu.o.cmake:173 (message): > Error generating > /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/./petsc_generated_mpiaijAssemble.cu.o >

1 0

Re: [petsc-dev] CMake-assisted CUDA builds ready
by Jose E. Roman 26 Mar '13

26 Mar '13

El 26/03/2013, a las 16:19, Karl Rupp escribió: > Hi Jose, > > alright, the problem should be fixed in 'next' now. The problem was in CMake not consistently interpreting escaped double quotes. > > Best regards, > Karli Not yet solved: /usr/local/cuda/bin/nvcc /home/jroman/soft/petsc/src/mat/impls/aij/mpi/mpicusp/mpiaijAssemble.cu " -O -arch=sm_13 " "--compiler-options ,-fPIC,-Wall,-Wwrite-strings,-Wno-strict-aliasing,-Wno-unknown-pragmas,-O" -m64 -D__INSDIR__= -Dpetsc_EXPORTS -Xcompiler ,\"-fPIC\" -DNVCC -M -o /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/CMakeFiles/petsc_generated_mpiaijAssemble.cu.o.NVCC-depend -I/usr/local/cuda/include -I/home/jroman/soft/petsc/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include -I/usr/local/cuda/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include/txpetscgpu/include -I/home/jroman/soft/cusplibrary -I/home/jroman/soft/cusplibrary/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi nvcc fatal : Unknown option '-compiler-options ,-fPIC,-Wall,-Wwrite-strings,-Wno-strict-aliasing,-Wno-unknown-pragmas,-O' CMake Error at CMakeFiles/petsc_generated_mpiaijAssemble.cu.o.cmake:173 (message): Error generating /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/./petsc_generated_mpiaijAssemble.cu.o

1 0

Re: [petsc-dev] VecMDot_SeqCUSP improved
by Jose E. Roman 26 Mar '13

26 Mar '13

El 26/03/2013, a las 02:41, Karl Rupp escribió: > Hi Jose, Paul, and others, > > I worked today and VecMDot and came up with an implementation which is faster than an iterated application of the standard cusp::blas::dot() (which, if I'm not mistaken, just forwards to CUBLAS) if enough vectors (>~6) are involved. For complex arithmetic, an iterated application of cusp::blas::dotc() is used, since passing complex types to CUDA kernels is fairly tricky within PETSc. Jose, any performance feedback from within SLEPc is appreciated :-) > > The new implementation is based on custom kernels, only allocates a little scratchpad memory and is thus more memory efficient than the old version. Also, any unnecessary copying of data is avoided. This should speed up GMRES quite a bit, yet I haven't run any dedicated GMRES benchmarks. Paul, I guess you have some samples at hand, don't you? > > Best regards, > Karli In my tests, the new implementation is actually slower. I tried src/vec/vec/examples/tests/ex43.c with 200 vectors of length 10000. Time increases from 4.1 to 7.2. Can anyone try to repeat the tests below? I have an Intel Core i7 with two Tesla C2050. Jose master --------------- $ ./ex43 -n 10000 -k 200 -mdot -log_summary VecMDot 3980 1.0 3.6485e+00 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 11100 0 0 0 11100 0 0 0 2182 $ ./ex43 -n 10000 -k 200 -mdot -log_summary -vec_type cusp VecMDot 3980 1.0 4.1368e+00 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 40100 0 0 0 40100 0 0 0 1924 $ ./ex43 -n 10000 -k 200 -log_summary VecDot 398000 1.0 2.1585e+01 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 78100 0 0 0 78100 0 0 0 369 $ ./ex43 -n 10000 -k 200 -log_summary -vec_type cusp VecDot 398000 1.0 2.9228e+01 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 82100 0 0 0 82100 0 0 0 272 next --------------- $ ./ex43 -n 10000 -k 200 -mdot -log_summary VecMDot 3980 1.0 3.6899e+00 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 39100 0 0 0 39100 0 0 0 2157 $ ./ex43 -n 10000 -k 200 -mdot -log_summary -vec_type cusp VecMDot 3980 1.0 7.1823e+00 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 54100 0 0 0 54100 0 0 0 1108 $ ./ex43 -n 10000 -k 200 -log_summary VecDot 398000 1.0 2.1702e+01 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 79100 0 0 0 79100 0 0 0 367 $ ./ex43 -n 10000 -k 200 -log_summary -vec_type cusp VecDot 398000 1.0 2.8953e+01 1.0 7.96e+09 1.0 0.0e+00 0.0e+00 0.0e+00 82100 0 0 0 82100 0 0 0 275

1 0

Re: [petsc-dev] CMake-assisted CUDA builds ready
by Karl Rupp 26 Mar '13

26 Mar '13

Hi Jose, alright, the problem should be fixed in 'next' now. The problem was in CMake not consistently interpreting escaped double quotes. Best regards, Karli On 03/26/2013 09:40 AM, Jose E. Roman wrote: > I get errors in the cmake build (in 'next'): > > /usr/local/cuda/bin/nvcc /home/jroman/soft/petsc/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu " -O -arch=sm_13 " "--compiler-options ," -fPIC\",\"-Wall\",\"-Wwrite-strings\",\"-Wno-strict-aliasing\",\"-Wno-unknown-pragmas\",\"-O\" \" -m64 -D__INSDIR__= -Dpetsc_EXPORTS -Xcompiler ,\"-fPIC\" -DNVCC -M -o /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/CMakeFiles/petsc_generated_mpiaijcusparse.cu.o.NVCC-depend -I/usr/local/cuda/include -I/home/jroman/soft/petsc/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include -I/usr/local/cuda/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include/txpetscgpu/include -I/home/jroman/soft/cusplibrary -I/home/jroman/soft/cusplibrary/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > nvcc fatal : Unknown option '-compiler-options ,' > CMake Error at CMakeFiles/petsc_generated_aijcusparse.cu.o.cmake:173 (message): > Error generating > /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/./petsc_generated_aijcusparse.cu.o > > > I have cmake-2.8 and cuda-4.2. > > Configure options: --download-txpetscgpu --with-cuda=1 --with-cusp-dir=/home/jroman/soft/cusplibrary --with-cusp=1 --with-debugging=0 --with-thrust=1 > > I can send the logs to petsc-maint if you want. > > Jose >

1 0

Re: [petsc-dev] CMake-assisted CUDA builds ready
by Karl Rupp 26 Mar '13

26 Mar '13

Hi Jose, No need to send the logs, I can see the problem: Something is going wrong when dealing with double-quotes. The flag string --compiler-options ,\"-fPIC\",\"-Wall\",\"-Wwrite-strings\",\"-Wno-strict-aliasing\",\"-Wno-unknown-pragmas\",\"-O\" gets passed through BuildSystem, which tries to put double quotes around it. This leads to " --compiler-options ,"-fPIC\",\"-Wall\",\"-Wwrite-strings\",\"-Wno-strict-aliasing\",\"-Wno-unknown-pragmas\",\"-O\" \" instead of " --compiler-options ,\"-fPIC\",\"-Wall\",\"-Wwrite-strings\",\"-Wno-strict-aliasing\",\"-Wno-unknown-pragmas\",\"-O\" " and thus the compiler complains. For some reasons the issue does not show up on my machine, even though I have the same versions of CMake and CUDA. I'll try to track it down. Best regards, Karli On 03/26/2013 09:40 AM, Jose E. Roman wrote: > I get errors in the cmake build (in 'next'): > > /usr/local/cuda/bin/nvcc /home/jroman/soft/petsc/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu " -O -arch=sm_13 " "--compiler-options ," -fPIC\",\"-Wall\",\"-Wwrite-strings\",\"-Wno-strict-aliasing\",\"-Wno-unknown-pragmas\",\"-O\" \" -m64 -D__INSDIR__= -Dpetsc_EXPORTS -Xcompiler ,\"-fPIC\" -DNVCC -M -o /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/CMakeFiles/petsc_generated_mpiaijcusparse.cu.o.NVCC-depend -I/usr/local/cuda/include -I/home/jroman/soft/petsc/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include -I/usr/local/cuda/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include/txpetscgpu/include -I/home/jroman/soft/cusplibrary -I/home/jroman/soft/cusplibrary/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi > nvcc fatal : Unknown option '-compiler-options ,' > CMake Error at CMakeFiles/petsc_generated_aijcusparse.cu.o.cmake:173 (message): > Error generating > /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/./petsc_generated_aijcusparse.cu.o > > > I have cmake-2.8 and cuda-4.2. > > Configure options: --download-txpetscgpu --with-cuda=1 --with-cusp-dir=/home/jroman/soft/cusplibrary --with-cusp=1 --with-debugging=0 --with-thrust=1 > > I can send the logs to petsc-maint if you want. > > Jose >

1 0

Re: [petsc-dev] CMake-assisted CUDA builds ready
by Jose E. Roman 26 Mar '13

26 Mar '13

I get errors in the cmake build (in 'next'): /usr/local/cuda/bin/nvcc /home/jroman/soft/petsc/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu " -O -arch=sm_13 " "--compiler-options ," -fPIC\",\"-Wall\",\"-Wwrite-strings\",\"-Wno-strict-aliasing\",\"-Wno-unknown-pragmas\",\"-O\" \" -m64 -D__INSDIR__= -Dpetsc_EXPORTS -Xcompiler ,\"-fPIC\" -DNVCC -M -o /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/CMakeFiles/petsc_generated_mpiaijcusparse.cu.o.NVCC-depend -I/usr/local/cuda/include -I/home/jroman/soft/petsc/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include -I/usr/local/cuda/include -I/home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/include/txpetscgpu/include -I/home/jroman/soft/cusplibrary -I/home/jroman/soft/cusplibrary/include -I/usr/lib/openmpi/include -I/usr/lib/openmpi/include/openmpi nvcc fatal : Unknown option '-compiler-options ,' CMake Error at CMakeFiles/petsc_generated_aijcusparse.cu.o.cmake:173 (message): Error generating /home/jroman/soft/petsc/arch-linux-gnu-c-opt-cuda/./petsc_generated_aijcusparse.cu.o I have cmake-2.8 and cuda-4.2. Configure options: --download-txpetscgpu --with-cuda=1 --with-cusp-dir=/home/jroman/soft/cusplibrary --with-cusp=1 --with-debugging=0 --with-thrust=1 I can send the logs to petsc-maint if you want. Jose

1 0

Re: [petsc-dev] VecMDot_SeqCUSP improved
by Paul Mullowney 26 Mar '13

26 Mar '13

Karli, Thanks for doing this. Once I get my current PETSc fork into next, I will test this asap. I have a couple examples using GMRES and I hope this will make a big difference. I'll let you know what I find. -Paul > Hi Jose, Paul, and others, > > I worked today and VecMDot and came up with an implementation which is > faster than an iterated application of the standard cusp::blas::dot() > (which, if I'm not mistaken, just forwards to CUBLAS) if enough > vectors (>~6) are involved. For complex arithmetic, an iterated > application of cusp::blas::dotc() is used, since passing complex types > to CUDA kernels is fairly tricky within PETSc. Jose, any performance > feedback from within SLEPc is appreciated :-) > > The new implementation is based on custom kernels, only allocates a > little scratchpad memory and is thus more memory efficient than the > old version. Also, any unnecessary copying of data is avoided. This > should speed up GMRES quite a bit, yet I haven't run any dedicated > GMRES benchmarks. Paul, I guess you have some samples at hand, don't you? > > Best regards, > Karli

1 0

Re: [petsc-dev] VecMDot_SeqCUSP improved
by Karl Rupp 25 Mar '13

25 Mar '13

Forgot to mention: Code is in 'next': https://bitbucket.org/petsc/petsc/commits/78e6257bdd411e017b354225da1226dab… On 03/25/2013 08:41 PM, Karl Rupp wrote: > Hi Jose, Paul, and others, > > I worked today and VecMDot and came up with an implementation which is > faster than an iterated application of the standard cusp::blas::dot() > (which, if I'm not mistaken, just forwards to CUBLAS) if enough vectors > (>~6) are involved. For complex arithmetic, an iterated application of > cusp::blas::dotc() is used, since passing complex types to CUDA kernels > is fairly tricky within PETSc. Jose, any performance feedback from > within SLEPc is appreciated :-) > > The new implementation is based on custom kernels, only allocates a > little scratchpad memory and is thus more memory efficient than the old > version. Also, any unnecessary copying of data is avoided. This should > speed up GMRES quite a bit, yet I haven't run any dedicated GMRES > benchmarks. Paul, I guess you have some samples at hand, don't you? > > Best regards, > Karli

1 0

VecMDot_SeqCUSP improved
by Karl Rupp 25 Mar '13

25 Mar '13

Hi Jose, Paul, and others, I worked today and VecMDot and came up with an implementation which is faster than an iterated application of the standard cusp::blas::dot() (which, if I'm not mistaken, just forwards to CUBLAS) if enough vectors (>~6) are involved. For complex arithmetic, an iterated application of cusp::blas::dotc() is used, since passing complex types to CUDA kernels is fairly tricky within PETSc. Jose, any performance feedback from within SLEPc is appreciated :-) The new implementation is based on custom kernels, only allocates a little scratchpad memory and is thus more memory efficient than the old version. Also, any unnecessary copying of data is avoided. This should speed up GMRES quite a bit, yet I haven't run any dedicated GMRES benchmarks. Paul, I guess you have some samples at hand, don't you? Best regards, Karli

1 0

Re: [petsc-dev] CMake-assisted CUDA builds ready
by Karl Rupp 25 Mar '13

25 Mar '13

Hi Dave, it was another timeout issue again. The combination of various external packages as well as the optimization flags provided led to a CMake boot time of 72 seconds on my machine, which is more than the 60 seconds timeout we used so far. The same timeout must have appeared on your machine. The new timeout is now set to 300 seconds and should also be sufficient for weaker machines^1. Waiting in 'next' for another test run :-) Thanks for all the testing and best regards, Karli ^1 Except some supercomputers with OpenMP enabled... ;-) On 03/23/2013 07:48 PM, Dave Nystrom wrote: > Thanks. I mistakenly sent the log files to petsc-dev. Sorry about that. I > did a resend and sent them to petsc-maint. > > Dave > > Karl Rupp writes: > > Hi Dave, > > > > thanks for the log-files and the testing. CMake hangs during it's > > startup and thus runs into a timeout. Maybe this is related to the bug > > Jed found in CMake 2.8.10.2, I have no other explanation why this hangs. > > I'll try to reproduce this as soon as I' at a CUDA-enabled machine again. > > > > Best regards, > > Karli > > > > > > > > On 03/23/2013 06:49 PM, Dave Nystrom wrote: > > > Hi Karli, > > > > > > Still does not seem to work. Attached are the logs from the latest attempt. > > > > > > Cheers, > > > > > > Dave > > > > > > > > > > > > > > > Karl Rupp writes: > > > > Hi again, > > > > > > > > I've found the cause of the issue, namely an overly tight timeout during > > > > the CMake initialization. Dave, could you please rerun reconfigure and > > > > make in the latest `next`? > > > > > > > > Thanks and best regards, > > > > Karli > > > > > > > > > > > > > > > > On 03/20/2013 07:31 PM, Dave Nystrom wrote: > > > > > Hi Karli, > > > > > > > > > > Just now seeing this. I did my build from a clean slate. > > > > > > > > > > Thanks, > > > > > > > > > > Dave > > > > > > > > > > Karl Rupp writes: > > > > > > Hi Dave, > > > > > > > > > > > > I also found that in some occasions one should issue > > > > > > $> rm -rf PETSC_ARCH/CMake* > > > > > > prior to running a reconfigure in order to circumvent a bug [1] in CMake. > > > > > > > > > > > > @Jed, Matt: The current cmakeboot.py removes > > > > > > PETSC_ARCH/CMakeFiles/{version_string}, which does not seem to even > > > > > > exist on my machine (CMake 2.8.7). Can we instead just delete the whole > > > > > > CMakeFiles-folder and probably CMakeCache.txt? > > > > > > > > > > > > Best regards, > > > > > > Karli > > > > > > > > > > > > [1] http://www.mail-archive.com/[email protected]/msg44765.html > > > > > > > > > > > > > > > > > > > > > > > > On 03/20/2013 10:17 AM, Nystrom, William D wrote: > > > > > > > Hi Karli, > > > > > > > > > > > > > > I ran a test build last night and it failed. It was a relatively simple configuration where > > > > > > > I was downloading mpich and cmake and building with threads, cuda and the txpetscgpu > > > > > > > package. I was building on a Fedora 17 machine with Cuda 5. The failure seem to be > > > > > > > associated with cmake. When I get home this evening, I could send you the configure > > > > > > > and make logs. I did not have time this morning before leaving for work. I wanted to > > > > > > > give you some early feedback even though the logs will have to wait till this evening. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Dave > > > > > > > > > > > > > > -- > > > > > > > Dave Nystrom > > > > > > > LANL HPC-5 > > > > > > > Phone: 505-667-7913 > > > > > > > Email: wdn(a)lanl.gov > > > > > > > Smail: Mail Stop B272 > > > > > > > Group HPC-5 > > > > > > > Los Alamos National Laboratory > > > > > > > Los Alamos, NM 87545 > > > > > > > > > > > > > > > > > > > > > ________________________________________ > > > > > > > From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Karl Rupp [rupp(a)mcs.anl.gov] > > > > > > > Sent: Tuesday, March 19, 2013 4:04 PM > > > > > > > To: petsc-dev(a)mcs.anl.gov > > > > > > > Subject: Re: [petsc-dev] CMake-assisted CUDA builds ready > > > > > > > > > > > > > > Hi Dave, > > > > > > > > > > > > > > perfect, thanks! > > > > > > > I should add that a reconfigure is needed. It won't work with re-running > > > > > > > make only. > > > > > > > > > > > > > > Best regards, > > > > > > > Karli > > > > > > > > > > > > > > > > > > > > > On 03/19/2013 04:59 PM, Nystrom, William D wrote: > > > > > > >> Hi Karli, > > > > > > >> > > > > > > >> I will try this soon and report back when I have results. > > > > > > >> > > > > > > >> Thanks, > > > > > > >> > > > > > > >> Dave > > > > > > >> > > > > > > >> ________________________________________ > > > > > > >> From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Karl Rupp [rupp(a)mcs.anl.gov] > > > > > > >> Sent: Tuesday, March 19, 2013 3:58 PM > > > > > > >> To: For users of the development version of PETSc > > > > > > >> Subject: [petsc-dev] CMake-assisted CUDA builds ready > > > > > > >> > > > > > > >> Dear all, > > > > > > >> > > > > > > >> I just merged the CMake-based builds for CUDA-packages into 'next', > > > > > > >> reducing compilation times considerably compared to the old legacy > > > > > > >> builds. While the legacy builds print something like > > > > > > >> 'libfast in <DIR>' > > > > > > >> the new CMake-based builds print the usual percentage counters in front. > > > > > > >> > > > > > > >> Any feedback on the proper functionality (both positive and negative) > > > > > > >> are appreciated. > > > > > > >> > > > > > > >> Thanks and best regards, > > > > > > >> Karli > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > >

1 0