March 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] git errors when trying to push to a fork.
by Jed Brown 25 Mar '13

25 Mar '13

Looking here: https://bitbucket.org/pmullowney/petsc-aijcusparse-icc/commits/all you're probably better off pushing on branch pm/aijcusparse-icc (or another named branch). If you amended your work (e.g., using 'git rebase -i' or 'git commit --amend') then you would need to use the '--force' option to push. Force pushes are acceptable *only* when nobody depends on your repository/branch, which is the case here. On Mon, Mar 25, 2013 at 10:38 AM, Paul Mullowney <paulm(a)txcorp.com> wrote: > Hi, > Here's an error with git I don't know how to resolve. > > [paulm@ivy petsc]$ git push https://pmullowney@bitbucket.** > org/pmullowney/petsc-**aijcusparse-icc.git<https://[email protected]/pmullowney/petsc-aijcusparse-icc.git> Password: > To https://pmullowney@bitbucket.**org/pmullowney/petsc-** > aijcusparse-icc.git<https://[email protected]/pmullowney/petsc-aijcusparse-icc.git> > ! [rejected] master -> master (non-fast-forward) > error: failed to push some refs to 'https://pmullowney@bitbucket.** > org/pmullowney/petsc-**aijcusparse-icc.git<https://[email protected]/pmullowney/petsc-aijcusparse-icc.git> > ' > To prevent you from losing history, non-fast-forward updates were rejected > Merge the remote changes before pushing again. See the 'Note about > fast-forwards' section of 'git push --help' for details. > > Should I do: > git pull -u > And then try to push? Presumably, this will pull from my fork and not > 'next' or 'master'? > > -Paul > >

1 0

git errors when trying to push to a fork.
by Paul Mullowney 25 Mar '13

25 Mar '13

Hi, Here's an error with git I don't know how to resolve. [paulm@ivy petsc]$ git push https://[email protected]/pmullowney/petsc-aijcusparse-icc.git Password: To https://[email protected]/pmullowney/petsc-aijcusparse-icc.git ! [rejected] master -> master (non-fast-forward) error: failed to push some refs to 'https://[email protected]/pmullowney/petsc-aijcusparse-icc.git' To prevent you from losing history, non-fast-forward updates were rejected Merge the remote changes before pushing again. See the 'Note about fast-forwards' section of 'git push --help' for details. Should I do: git pull -u And then try to push? Presumably, this will pull from my fork and not 'next' or 'master'? -Paul

1 0

more MPI/OpenMP experimentation
by Gerard Gorman 25 Mar '13

25 Mar '13

Hi We have been doing more experiments with threaded algorithms for MPI/OpenMP on our branch of petsc-3.3: https://bitbucket.org/ggorman/petsc-3.3-omp (all the usual health warnings etc - more than a few times we thought "Jed isn't going to like this") We just got a paper accepted to ISC which goes through the general benchmarking, preprint: http://arxiv.org/abs/1303.5275 It a nut shell the message seems to be that for strong scaling: - Task thread based approach is good as you can use one thread to do the mpi communication (so real asynchronous communication). - It is important to balance the load per thread as it's load imbalance that begins to limit scalability. We still have quite a bit of profiling to do. In particular, we need to look at the hardware counters and really make sure we understand what's happening on the machine. Nevertheless we think the results are interesting. We would welcome any suggestions/feedback on this. It seems like the same approach could be implemented in petsc-dev since threadcomms has most of the support required for this. Cheers Gerard

1 0

Re: [petsc-dev] ViennaCL-bindings in next
by Karl Rupp 24 Mar '13

24 Mar '13

Hi Dave, > I'm very interested in trying out your new ViennaCL interface in the near > future. Looking at the documentation of ViennaCL online, it looks like there > are preconditioners available within ViennaCL. Are there plans to provide > petsc interfaces to the ViennaCL preconditioners in the near future? Yes, there are a few preconditioners in ViennaCL already, most of them using the CPU for the setup phase. I'm planning to make them available in PETSc as well, plus adding more GPU-based preconditioners. Stay tuned :-) Best regards, Karli > > Karl Rupp writes: > > Hi, > > > > the interface to ViennaCL [1] has just been merged to 'next', enabling > > PETSc to use OpenCL-enabled devices (most notably Intel's Xeon Phi and > > GPUs from AMD in addition to the existing NVIDIA bindings). I tested it > > with Vector-, Matrix-, and KSP- examples. > > > > The new bindings are enabled by the configure flags > > --download-viennacl > > --with-opencl-include=/path/to/OpenCL-Headers > > --with-opencl-lib=/path/to/libOpenCL.so > > > > The OpenCL headers can be obtained on the Khronos webpage [2] and are > > often available via package managers, e.g. > > $> sudo apt-get install opencl-headers > > or > > $> sudo yam install opencl-headers > > In such case, the OpenCL include directory is typically /usr/include > > (BuildSystem may not detect these headers correctly on Mac OS, I'm > > working on this). The OpenCL library comes with your graphics driver and > > is usually located at /usr/lib/. There is no separate compiler as for > > CUDA needed! > > > > FAQ: > > > > - What is the performance difference between OpenCL and CUDA for GPUs? > > My experience is that for large problem sizes they are equivalent for > > almost all linear algebra kernels. For small problem sizes (kernel > > execution less than 1ms), CUDA is currently superior because of lower > > latency. > > > > - Which solvers can I use with the ViennaCL-bindings? > > The same as with CUSP. Most notably the usual CG, BiCGS, GMRES. > > > > - Which preconditioners are provided in the ViennaCL-bindings? > > Currently none. You can, however, use all the standard CPU-based > > preconditioners in PETSc together with ViennaCL matrices and vectors. > > > > - Which command line options do I need to use? > > Use -vec_type viennacl -mat_type aijviennacl > > On Intel's Xeon Phi you also need to pass -viennacl_device_accelerator. > > (If you read this email in an archive: Check the manual for current flags). > > > > Best regards, > > Karli > > > > [1] http://viennacl.sourceforge.net/ > > [2] http://www.khronos.org/registry/cl/ >

1 0

Re: [petsc-dev] ViennaCL-bindings in next
by Dave Nystrom 24 Mar '13

24 Mar '13

Hi Karli, I'm very interested in trying out your new ViennaCL interface in the near future. Looking at the documentation of ViennaCL online, it looks like there are preconditioners available within ViennaCL. Are there plans to provide petsc interfaces to the ViennaCL preconditioners in the near future? Thanks, Dave Karl Rupp writes: > Hi, > > the interface to ViennaCL [1] has just been merged to 'next', enabling > PETSc to use OpenCL-enabled devices (most notably Intel's Xeon Phi and > GPUs from AMD in addition to the existing NVIDIA bindings). I tested it > with Vector-, Matrix-, and KSP- examples. > > The new bindings are enabled by the configure flags > --download-viennacl > --with-opencl-include=/path/to/OpenCL-Headers > --with-opencl-lib=/path/to/libOpenCL.so > > The OpenCL headers can be obtained on the Khronos webpage [2] and are > often available via package managers, e.g. > $> sudo apt-get install opencl-headers > or > $> sudo yam install opencl-headers > In such case, the OpenCL include directory is typically /usr/include > (BuildSystem may not detect these headers correctly on Mac OS, I'm > working on this). The OpenCL library comes with your graphics driver and > is usually located at /usr/lib/. There is no separate compiler as for > CUDA needed! > > FAQ: > > - What is the performance difference between OpenCL and CUDA for GPUs? > My experience is that for large problem sizes they are equivalent for > almost all linear algebra kernels. For small problem sizes (kernel > execution less than 1ms), CUDA is currently superior because of lower > latency. > > - Which solvers can I use with the ViennaCL-bindings? > The same as with CUSP. Most notably the usual CG, BiCGS, GMRES. > > - Which preconditioners are provided in the ViennaCL-bindings? > Currently none. You can, however, use all the standard CPU-based > preconditioners in PETSc together with ViennaCL matrices and vectors. > > - Which command line options do I need to use? > Use -vec_type viennacl -mat_type aijviennacl > On Intel's Xeon Phi you also need to pass -viennacl_device_accelerator. > (If you read this email in an archive: Check the manual for current flags). > > Best regards, > Karli > > [1] http://viennacl.sourceforge.net/ > [2] http://www.khronos.org/registry/cl/

1 0

User manual converted to/for Doxygen
by Karl Rupp 24 Mar '13

24 Mar '13

Dear all, I've finally completed my conversion of the user manual to Doxygen. An online copy can be found here: http://krupp.iue.tuwien.ac.at/petsc-doxygen/usermanual.html This completes the raw conversion of the existing documentation to a doxygen-based system. Further polishing is certainly required, but certain parts may not be possible without actually touching the existing sowing-centered code comments in the source tree. The not-so-ideal thing is that the LaTeX manual and the doxygen pages exist in parallel. Converting from either of them to the other is not possible in an automatic way. Thus, I suggest to introduce a unified Markdown-like format, from which we can generate both types of documentation. Some customizations is certainly necessary because of the way the text needs to be written. For example, in a PDF it is natural to write "Further explanation is given in Chapter 42", while in an HTML documentation one ideally provides a link of the type "[More information on Foo]". Similarly, some large code snippets included directly in the PDF manual can be linked easily within an HTML documentation. Best regards, Karli PS: To avoid confusion: PETSc will keep its existing documentation system for a while. We did not yet agree on the/any future of this doxygen-based approach.

1 0

Re: [petsc-dev] patches for ICC/Cholesky preconditioner on GPUs
by Jed Brown 24 Mar '13

24 Mar '13

On Sun, Mar 24, 2013 at 4:52 PM, Mark F. Adams <mark.adams(a)columbia.edu>wrote: > There is currently a 'data' directory in ksp/ksp/examples/tutorials that > has data files for some tests. > This is *not* an acceptable place to put data files that are not trivially small. As a general rule, unless it is small enough that you either wrote it by hand or can check its integrity by hand, don't check it directly into the repository.

1 0

Re: [petsc-dev] patches for ICC/Cholesky preconditioner on GPUs
by Mark F. Adams 24 Mar '13

24 Mar '13

> (d) Where do I put binary matrix files that these tests rely upon? > > Put them somewhere and we'll add them here: > > ftp://ftp.mcs.anl.gov/pub/petsc/matrices/ > > (We could consider nicer ways to maintain these.) > There is currently a 'data' directory in ksp/ksp/examples/tutorials that has data files for some tests.

1 0

Re: [petsc-dev] patches for ICC/Cholesky preconditioner on GPUs
by Jed Brown 24 Mar '13

24 Mar '13

On Sun, Mar 24, 2013 at 3:17 PM, Paul Mullowney <paulm(a)txcorp.com> wrote: > Jed, > > I think I've resolved all of your comments. I have some questions. > > 1) Tests for the nightly test harness > > (a) Is there a naming convention for the tests? > I have written two examples named aijcusparse_ex_ilu.c and > aijcusparse_ex_icc.c. Is this ok? > They source code files should be numbered unless we change policy. If we were to change, we would still need some systematic naming so that .gitignore would not be interrupted, perhaps 'ex071-aijcusp-icc.c'. Note that some projects use sequential numbering to indicate "level" and priority of tests (lower numbers run first). For now, just use numbers. > (b) What directory should they live in? > I assume these go in ksp/ksp/examples/tutorials > If they are well-documented, thus suitable as _tutorials_, they can go in the tutorials directory. If they are _tests_, they should be in examples/tests/. > > (c) How do I ensure that they tests are built and run every night with > txpetscgpu package? > Use TESTEXAMPLES_TXPETSCGPU in conf/test and in the examples makefile. You'll see lots of other packages. > (d) Where do I put binary matrix files that these tests rely upon? > Put them somewhere and we'll add them here: ftp://ftp.mcs.anl.gov/pub/petsc/matrices/ (We could consider nicer ways to maintain these.) > 2) Doing > > git status > shows > Your branch is ahead of 'origin/pm/aijcusparse-icc' by 2 commits. > > I assume that when I push to the fork, these changes will be uploaded > remotely? > Yes, you can use 'git push --dry-run' to see what would happen if you push. After pushing to your fork, open a pull request.

1 0

Re: [petsc-dev] patches for ICC/Cholesky preconditioner on GPUs
by Paul Mullowney 24 Mar '13

24 Mar '13

Jed, I think I've resolved all of your comments. I have some questions. 1) Tests for the nightly test harness (a) Is there a naming convention for the tests? I have written two examples named aijcusparse_ex_ilu.c and aijcusparse_ex_icc.c. Is this ok? (b) What directory should they live in? I assume these go in ksp/ksp/examples/tutorials (c) How do I ensure that they tests are built and run every night with txpetscgpu package? (d) Where do I put binary matrix files that these tests rely upon? 2) Doing > git status shows Your branch is ahead of 'origin/pm/aijcusparse-icc' by 2 commits. I assume that when I push to the fork, these changes will be uploaded remotely? Thanks, -Paul > Paul Mullowney<paulm(a)txcorp.com> writes: > >> Hi, >> >> I've gotten GPU based icc preconditioners working with the following set >> of patches. One uses ICC by loading an upper triangular matrix into the >> cusparse class, and then setting the symmetry option, i.e. >> >> ierr = >> MatCreateSeqAIJCUSPARSE(comm,n,m,PETSC_NULL,num_entries_per_row,&A);CHKERRQ(ierr); >> ierr = MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE); >> >> I've organized the implementations into 2 files: >> 1) icc-cholesky-cusparse-organization.patch : This patch has the basic >> infrastructure for calling the symbolic and numeric factorizations (both >> ICC and Cholesky). Currently these routines make scoping calls to the >> implementations in aij/seq/aijfact.c > Can you write context like this into the commit messages? They are > pretty opaque right now, but they are more likely to be read in the > future than this email. > > This patch series does not have any tests. Can you add tests so that we > can have a reasonable chance of not breaking it in the future? > > We can review on the mailing list, but I think we would prefer to use > pull requests, mostly because comments there sort by context rather than > by author. Barry, Matt, and others, do you prefer the granularity of > email or bitbucket comments? > > A few comments now: > >> exporting patch: > This first line should not be in the patch file. It should start with the lines below: > >> # HG changeset patch >> # User Paul Mullowney<paulm(a)txcorp.com> >> # Date 1362859983 25200 >> # Node ID 6be0d0f06ad18794377015b5ec468e11105b58b1 >> # Parent 6d60df3aadfe49f0bf640fa0a41fe3a423b88cf6 >> Implementation of ICC in this patch. > Include the component (aijcusparse) in the commit message, otherwise we > don't have a clue from the commit message what is actually being done. > >> diff -r 6d60df3aadfe -r 6be0d0f06ad1 src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu >> --- a/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu Mon Feb 04 11:42:47 2013 -0700 >> +++ b/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu Sat Mar 09 13:13:03 2013 -0700 >> @@ -1,12 +1,13 @@ >> /* >> - Defines the basic matrix operations for the AIJ (compressed row) >> + Defines the basic matrix operations for the AIJ (compressed row) >> matrix storage format. >> */ >> >> #include "petscconf.h" >> PETSC_CUDA_EXTERN_C_BEGIN >> #include "../src/mat/impls/aij/seq/aij.h" /*I "petscmat.h" I*/ >> -//#include "petscbt.h" >> +#include<../src/mat/impls/sbaij/seq/sbaij.h> >> + >> #include "../src/vec/vec/impls/dvecimpl.h" >> #include "petsc-private/vecimpl.h" >> PETSC_CUDA_EXTERN_C_END > Not part of this patch, but PETSC_CUDA_EXTERN_C_BEGIN/END should be > replaced with appropriate use of PETSC_EXTERN (as shown below). We are > no longer mangling any public function names, so we can now compile > PETSc using --with-clanguage=C++ and then call it from plain C. > >> @@ -53,12 +54,28 @@ >> EXTERN_C_BEGIN >> extern PetscErrorCode MatGetFactor_seqaij_petsc(Mat,MatFactorType,Mat*); >> EXTERN_C_END > Use this instead: > > PETSC_EXTERN PetscErrorCode MatGetFactor_seqaij_petsc(Mat,MatFactorType,Mat*); > >> @@ -404,12 +417,119 @@ >> >> >> >> + >> + > Isn't this a lot of blank lines? > >> +#undef __FUNCT__ >> +#define __FUNCT__ "MatSeqAIJCUSPARSEBuildICCTriMatrices" >> +PetscErrorCode MatSeqAIJCUSPARSEBuildICCTriMatrices(Mat A) > Please use 'static' if the function is really only used in this one > file. This identifier is more than 31 characters, thus too long for a > public symbol. > >> +{ >> + Mat_SeqAIJ *a = (Mat_SeqAIJ*)A->data; >> + Mat_SeqAIJCUSPARSETriFactors *cusparseTriFactors = (Mat_SeqAIJCUSPARSETriFactors*)A->spptr; >> + GPU_Matrix_Ifc * cusparseMatLo = (GPU_Matrix_Ifc*)cusparseTriFactors->loTriFactorPtr; >> + GPU_Matrix_Ifc * cusparseMatUp = (GPU_Matrix_Ifc*)cusparseTriFactors->upTriFactorPtr; > Either line up consecutive lines or don't, but don't go half way. PETSc > style guide says 'type *ptr', not 'type * ptr'. > >> + cusparseStatus_t stat; >> + PetscErrorCode ierr; >> + PetscInt *AiUp, *AjUp; >> + PetscScalar *AAUp; >> + PetscScalar *AALo; >> + PetscInt nzUpper=a->nz,n=A->rmap->n,i,offset,nz,j; >> + Mat_SeqSBAIJ *b = (Mat_SeqSBAIJ*)A->data; >> + const PetscInt *ai=b->i,*aj=b->j,*vj; >> + const MatScalar *aa=b->a,*v; > Again, line up or don't line up. > >> + /* set this flag ... for performance logging */ >> + ((Mat_SeqAIJCUSPARSETriFactors*)A->spptr)->isSymmOrHerm = PETSC_TRUE; > Does this have a MatSetOption() interface? > >> #undef __FUNCT__ >> #define __FUNCT__ "MatSeqAIJCUSPARSEICCAnalysisAndCopyToGPU" >> PetscErrorCode MatSeqAIJCUSPARSEICCAnalysisAndCopyToGPU(Mat A) > This can have static linkage, right? > >> @@ -706,8 +848,28 @@ >> ierr = MPI_Comm_size(PETSC_COMM_WORLD,&size);CHKERRQ(ierr); > [Different patch] PETSC_COMM_WORLD cannot be used outside of global > registration.

1 0