February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] DMComplex and PCFieldSplit
by Matthew Knepley 24 Feb '12

24 Feb '12

On Fri, Feb 24, 2012 at 1:58 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > http://petsc.cs.iit.edu/petsc/petsc-dev/rev/518ff70e8a0a > > Matt, I want to get rid of these conditionals, not add more. We should > have a DM base interface for getting fields on which to split, that common > interface should _replace_ the DMComposite specialization. > I agree. However, I won't put anything in until I do it by hand once. Also, I was tempted to just promote DMGetGlobalISes(), but its not quite right. I am planning to put the IS method in PetscSection. I am hoping eventually DMDA uses PetscSection for layout. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

DMComplex and PCFieldSplit
by Jed Brown 24 Feb '12

24 Feb '12

http://petsc.cs.iit.edu/petsc/petsc-dev/rev/518ff70e8a0a Matt, I want to get rid of these conditionals, not add more. We should have a DM base interface for getting fields on which to split, that common interface should _replace_ the DMComposite specialization.

1 0

-log_summary should print the number of GPUs being used
by Barry Smith 24 Feb '12

24 Feb '12

-log_summary should be printing the number of GPUs being used

1 0

Re: [petsc-dev] Understanding Some Parallel Results with PETSc
by Nystrom, William D 24 Feb '12

24 Feb '12

Hi Jed, Attached is a gzipped tarball of the stuff I used to run these two test problems with numactl. Actually, I hacked them a bit this morning because I was running them in our test framework for doing acceptance testing of new systems. But the scripts in the tarball should give you all the info you need. There is a top level script called runPetsc that just invokes mpirun from openmpi and calls the wrapper scripts for using numactl. You could actually dispense with the top level script and just invoke the mpirun commands yourself. I include it as an easy way to document what I did. The runPetscProb_1 script runs petsc on the gpus using numactl to control the affinities of the gpus to the cpu numa nodes. The runPetscProb_2 script runs petsc on the cpus using numactl. Note that both of those wrapper scripts are using openmpi variables. I'm not sure how one would do the same thing with another flavor of mpi. But I imagine it is possible. Also, I'm not sure if there are other more elegant ways to run with numactl than using the wrapper script approach. Perhaps there is but this is what we have been doing. I've also included a Perl script called numa-maps that is useful for actually checking the affinities that you get while running in order to make sure that numactl is doing what you think it is doing. I'm not sure where this script comes from. I find it on some systems and not on others. I've also include logs with the output of cpuinfo, nvidia-smi and uname -a to answer any questions you had about the system I was running on. Finally, I've included runPetscProb_1.log and runPetscProb_2.log which contains the log_summary output for my latest runs on the gpu and cpu respectively. Using numactl reduced the runtime for the gpu case as well but not as much as for the cpu case. So the final result was that running the same problem while using all of the gpu resources on a node was about 2.5x times faster than using all of the cpu resources on the same number of nodes. Let me know if you need more any more info. I'm planning to use this stuff to help test a new gpu cluster that we have just started acceptance testing on. It has the same basic hardware as the testbed cluster for these results but has 308 nodes. That should be interesting and fun. Thanks, Dave -- Dave Nystrom LANL HPC-5 Phone: 505-667-7913 Email: wdn(a)lanl.gov Smail: Mail Stop B272 Group HPC-5 Los Alamos National Laboratory Los Alamos, NM 87545 ________________________________ From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Jed Brown [jedbrown(a)mcs.anl.gov] Sent: Thursday, February 23, 2012 10:43 PM To: For users of the development version of PETSc Cc: Dave Nystrom Subject: Re: [petsc-dev] Understanding Some Parallel Results with PETSc On Thu, Feb 23, 2012 at 23:41, Dave Nystrom <dnystrom1(a)comcast.net<mailto:[email protected]>> wrote: I could also send you my mpi/numactl command lines for gpu and cpu when I am back in the office. Yes, please.

1 0

Re: [petsc-dev] Intel compilers + CUDA
by John Fettig 24 Feb '12

24 Feb '12

On Fri, Feb 24, 2012 at 11:56 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > for one I mistyped the option - it should be: --with-cudac='nvcc -ccbin > icc' > When I try this, configure fails because nvcc gives a warning on my machine when used with icc: icc: command line warning #10156: ignoring option '-fp'; no argument required Honestly, I really don't care if gcc is used for nvcc. I guess the main issue for configure is determining what are the flags that can be passed through in this case. I guess I'll just continue modifying the makefiles by hand after configure (and commenting out the check for gcc inside conf/PETSc/packages/cuda.py) Thanks, John

1 0

Re: [petsc-dev] Intel compilers + CUDA
by John Fettig 24 Feb '12

24 Feb '12

On Fri, Feb 24, 2012 at 12:12 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Fri, Feb 24, 2012 at 11:06 AM, John Fettig <john.fettig(a)gmail.com>wrote: > >> On Fri, Feb 24, 2012 at 11:56 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: >> >>> for one I mistyped the option - it should be: --with-cudac='nvcc -ccbin >>> icc' >>> >>> But then - should it not be c++? [as cuda internally uses g++ - not >>> gcc]. So >>> >>> --with-cudac='nvcc -ccbin icpc'. >>> >>> But this gives me all kinds of wierd errors in configure.log.. Maybe >>> it won't work [with intel compilers].. >>> >> >> CUDA .cu files are written in C with some extensions (that are handled by >> nvcc and not the passed compiler), so I think you have to use a C >> compiler. You can't write C++ in a .cu file. >> > > I do not think this is correct. You can have C++. > Oops, you are right. There are restrictions on what can be on the device side, but not on the host side. John

1 0

Re: [petsc-dev] Intel compilers + CUDA
by Matthew Knepley 24 Feb '12

24 Feb '12

On Fri, Feb 24, 2012 at 11:06 AM, John Fettig <john.fettig(a)gmail.com> wrote: > On Fri, Feb 24, 2012 at 11:56 AM, Satish Balay <balay(a)mcs.anl.gov> wrote: > >> for one I mistyped the option - it should be: --with-cudac='nvcc -ccbin >> icc' >> >> But then - should it not be c++? [as cuda internally uses g++ - not gcc]. >> So >> >> --with-cudac='nvcc -ccbin icpc'. >> >> But this gives me all kinds of wierd errors in configure.log.. Maybe >> it won't work [with intel compilers].. >> > > CUDA .cu files are written in C with some extensions (that are handled by > nvcc and not the passed compiler), so I think you have to use a C > compiler. You can't write C++ in a .cu file. > I do not think this is correct. You can have C++. Matt > > John > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Intel compilers + CUDA
by Satish Balay 24 Feb '12

24 Feb '12

On Fri, 24 Feb 2012, Satish Balay wrote: > for one I mistyped the option - it should be: --with-cudac='nvcc -ccbin icc' > > But then - should it not be c++? [as cuda internally uses g++ - not gcc]. So > > --with-cudac='nvcc -ccbin icpc'. > > But this gives me all kinds of wierd errors in configure.log.. Maybe > it won't work [with intel compilers].. I see 2 primary issues: 1. incompatible include files: >>>>>>>>>>>>>>> balay@bb30:~/petsc-dev/src/ksp/pc/impls/bicgstabcusp>nvcc -v -ccbin icc -g -arch=sm_13 -c --compiler-options="-wd1572 -g -I/home/balay/petsc-dev/include -I/home/balay/petsc-dev/arch-intel-cuda-double/include -I/usr/local/cuda/include -D__INSDIR__=src/ksp/pc/impls/bicgstabcusp/" bicgstabcusp.cu#$ _SPACE_= #$ _CUDART_=cudart #$ _HERE_=/usr/local/cuda/bin #$ _THERE_=/usr/local/cuda/bin #$ _TARGET_SIZE_=64 #$ TOP=/usr/local/cuda/bin/.. #$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/local/cuda/bin/../extools/lib:/soft/intel/11.0.081/lib/intel64:/soft/intel/11.0.081/lib/intel64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 #$ PATH=/usr/local/cuda/bin/../open64/bin:/usr/local/cuda/bin:/soft/intel/11.0.081/bin/intel64:/soft/intel/11.0.081/bin/intel64:/usr/mpi/gcc/mvapich2-1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/cuda/bin:/usr/local/cuda/bin #$ INCLUDES="-I/usr/local/cuda/bin/../include" "-I/usr/local/cuda/bin/../include/cudart" #$ LIBRARIES= "-L/usr/local/cuda/bin/../lib64" -lcudart #$ CUDAFE_FLAGS= #$ OPENCC_FLAGS= #$ PTXAS_FLAGS= #$ icc -D__CUDA_ARCH__=130 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -C -wd1572 -g -I/home/balay/petsc-dev/include -I/home/balay/petsc-dev/arch-intel-cuda-double/include -I/usr/local/cuda/include -D__INSDIR__=src/ksp/pc/impls/bicgstabcusp/ "-I/usr/local/cuda/bin/../include" "-I/usr/local/cuda/bin/../include/cudart" -include "cuda_runtime.h" -m64 -g -o "/tmp/tmpxft_0000340c_00000000-4_bicgstabcusp.cpp1.ii" "bicgstabcusp.cu" #$ cudafe --m64 --gnu_version=40403 -tused --no_remove_unneeded_entities --gen_c_file_name "/tmp/tmpxft_0000340c_00000000-1_bicgstabcusp.cudafe1.c" --stub_file_name "/tmp/tmpxft_0000340c_00000000-1_bicgstabcusp.cudafe1.stub.c" --gen_device_file_name "/tmp/tmpxft_0000340c_00000000-1_bicgstabcusp.cudafe1.gpu" --include_file_name "/tmp/tmpxft_0000340c_00000000-3_bicgstabcusp.fatbin.c" "/tmp/tmpxft_0000340c_00000000-4_bicgstabcusp.cpp1.ii" /usr/local/cuda/include/math_functions.h(389): warning: exception specification is incompatible with that of previous function "__isinf" /soft/intel/11.0.081/include/math.h(236): here <<<<<<<<<<<<<<< 2. command line options not progegated properly to the compiler by nvcc [option '-Qoption,cpp,--extended_float_type' is munged with additional spaces] >>>>>>>>>>>>>>> balay@bb30:~/petsc-dev/src/ksp/pc/impls/bicgstabcusp>nvcc -v -ccbin icc -g -arch=sm_13 -c --compiler-options="-wd1572 -Qoption,cpp,--extended_float_type -g -I/home/balay/petsc-dev/include -I/home/balay/petsc-dev/arch-intel-cuda-double/include -I/usr/local/cuda/include -D__INSDIR__=src/ksp/pc/impls/bicgstabcusp/" bicgstabcusp.cu #$ _SPACE_= #$ _CUDART_=cudart #$ _HERE_=/usr/local/cuda/bin #$ _THERE_=/usr/local/cuda/bin #$ _TARGET_SIZE_=64 #$ TOP=/usr/local/cuda/bin/.. #$ LD_LIBRARY_PATH=/usr/local/cuda/bin/../lib:/usr/local/cuda/bin/../extools/lib:/soft/intel/11.0.081/lib/intel64:/soft/intel/11.0.081/lib/intel64:/usr/local/cuda/lib64:/usr/local/cuda/lib64 #$ PATH=/usr/local/cuda/bin/../open64/bin:/usr/local/cuda/bin:/soft/intel/11.0.081/bin/intel64:/soft/intel/11.0.081/bin/intel64:/usr/mpi/gcc/mvapich2-1.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/cuda/bin:/usr/local/cuda/bin #$ INCLUDES="-I/usr/local/cuda/bin/../include" "-I/usr/local/cuda/bin/../include/cudart" #$ LIBRARIES= "-L/usr/local/cuda/bin/../lib64" -lcudart #$ CUDAFE_FLAGS= #$ OPENCC_FLAGS= #$ PTXAS_FLAGS= #$ icc -D__CUDA_ARCH__=130 -E -x c++ -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -C -wd1572 -Qoption cpp --extended_float_type -g -I/home/balay/petsc-dev/include -I/home/balay/petsc-dev/arch-intel-cuda-double/include -I/usr/local/cuda/include -D__INSDIR__=src/ksp/pc/impls/bicgstabcusp/ "-I/usr/local/cuda/bin/../include" "-I/usr/local/cuda/bin/../include/cudart" -include "cuda_runtime.h" -m64 -g -o "/tmp/tmpxft_00003430_00000000-4_bicgstabcusp.cpp1.ii" "bicgstabcusp.cu" icc: command line warning #10006: ignoring unknown option '-Qoption' icc: command line warning #10006: ignoring unknown option '-fextended_float_type' Catastrophic error: could not open source file "cpp" # --error 0x4 -- balay@bb30:~/petsc-dev/src/ksp/pc/impls/bicgstabcusp> <snip> <<<<<<<<<<<<<<< Satish

1 0

Re: [petsc-dev] BuildSystem clean-up
by Sean Farley 24 Feb '12

24 Feb '12

> > BTW: externalpackages/package/PETSC_ARCH is more like a default prefix > for the package. Ideally if the package supported prefix - we would > use it and put the stuff directly in PETSC_DIR/PETSC_ARCH - and > externalpackages/package/PETSC_ARCH won't have any stuff in it. > Ah, ok, that makes sense now. > Perhaps some package.py files are doing things manually and using > externalpackages/package/PETSC_ARCH as temporary location - and later > coping things over to PETSC_DIR/PETSC_ARCH - and not cleaning things > up. [in externalpackages/package/PETSC_ARCH]. Right, but what should we do about it? It could potentially cause a bunch of headaches.

1 0

Re: [petsc-dev] BuildSystem clean-up
by Sean Farley 24 Feb '12

24 Feb '12

> > The assumption was that each external package had an appropriate clean > that would be called by the XXXX.py BEFORE doing a build to clean it up; > this did require each XXXX.py knowing how to call the clean and doing this. > Clearly this assumption was wilding optimistic. > Way too optimistic. > The draw back to having buildsystem (note the lack of caps) doing the > clean AFTER doing the building is important generated stuff useful for > debugging may be lost. So what about doing the clean out of that directory > at the beginning, not at the end? > Actually, I was advocating cleaning it out at the beginning but in packages.py it doesn't seem easy to tell if self.installNeeded is set yet. Do we just blindly blow away this directory? Or should we (Matt) define a preInstall concept? How buildsystem knowns if the package should be rebuilt is NOT stored in > that directory so making your change should not affect the current somewhat > "smart" build again only if needed logic. > Righto.

1 0