March 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Errors in ThreadComm stuff
by Jed Brown 21 Mar '13

21 Mar '13

On Thu, Mar 21, 2013 at 7:00 PM, Mark F. Adams <mark.adams(a)columbia.edu>wrote: > No, not intentionally, we don't touch threads. > I'm not sure, maybe funny business with thread-local storage. Based on the stack trace above and the intent of this code, you can apply this for now. diff --git i/src/sys/objects/tagm.c w/src/sys/objects/tagm.c index 46e32c1..cc399ab 100644 --- i/src/sys/objects/tagm.c +++ w/src/sys/objects/tagm.c @@ -198,9 +198,7 @@ PetscErrorCode PetscCommDuplicate(MPI_Comm comm_in,MPI_Comm *comm_out,PetscMPII ierr = PetscCommGetThreadComm(*comm_out,&tcomm);CHKERRQ(ierr); ierr = PetscThreadCommAttach(*comm_out,tcomm);CHKERRQ(ierr); } - /* Only the main thread updates counter->refcount */ - ierr = PetscThreadCommGetRank(tcomm,&trank);CHKERRQ(ierr); - if (!trank) counter->refcount++; /* number of references to this comm */ + counter->refcount++; /* number of references to this comm */ PetscFunctionReturn(0); } > > On Mar 21, 2013, at 7:45 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > Did you enable threadcomm? I better finish this work I started and get my > branch out there... > > > On Thu, Mar 21, 2013 at 6:42 PM, Mark F. Adams <mark.adams(a)columbia.edu>wrote: > >> I'm getting segv in at the end of runs on the Cray XC30. in optimized >> builds it segv with a stack like >> >> PetscFinalize >> PetscViewerASCIIGetStdout >> PetscCommDuplicate >> PetscThreadCommGetRank >> >> In debug builds it does not segv but there are a bunch of un freeded >> memory in ThreadComm stuff. >> >> Mark > > > >

1 0

Re: [petsc-dev] Errors in ThreadComm stuff
by Mark F. Adams 21 Mar '13

21 Mar '13

No, not intentionally, we don't touch threads. On Mar 21, 2013, at 7:45 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > Did you enable threadcomm? I better finish this work I started and get my branch out there... > > > On Thu, Mar 21, 2013 at 6:42 PM, Mark F. Adams <mark.adams(a)columbia.edu> wrote: > I'm getting segv in at the end of runs on the Cray XC30. in optimized builds it segv with a stack like > > PetscFinalize > PetscViewerASCIIGetStdout > PetscCommDuplicate > PetscThreadCommGetRank > > In debug builds it does not segv but there are a bunch of un freeded memory in ThreadComm stuff. > > Mark >

1 0

ViennaCL-bindings in next
by Karl Rupp 21 Mar '13

21 Mar '13

Hi, the interface to ViennaCL [1] has just been merged to 'next', enabling PETSc to use OpenCL-enabled devices (most notably Intel's Xeon Phi and GPUs from AMD in addition to the existing NVIDIA bindings). I tested it with Vector-, Matrix-, and KSP- examples. The new bindings are enabled by the configure flags --download-viennacl --with-opencl-include=/path/to/OpenCL-Headers --with-opencl-lib=/path/to/libOpenCL.so The OpenCL headers can be obtained on the Khronos webpage [2] and are often available via package managers, e.g. $> sudo apt-get install opencl-headers or $> sudo yam install opencl-headers In such case, the OpenCL include directory is typically /usr/include (BuildSystem may not detect these headers correctly on Mac OS, I'm working on this). The OpenCL library comes with your graphics driver and is usually located at /usr/lib/. There is no separate compiler as for CUDA needed! FAQ: - What is the performance difference between OpenCL and CUDA for GPUs? My experience is that for large problem sizes they are equivalent for almost all linear algebra kernels. For small problem sizes (kernel execution less than 1ms), CUDA is currently superior because of lower latency. - Which solvers can I use with the ViennaCL-bindings? The same as with CUSP. Most notably the usual CG, BiCGS, GMRES. - Which preconditioners are provided in the ViennaCL-bindings? Currently none. You can, however, use all the standard CPU-based preconditioners in PETSc together with ViennaCL matrices and vectors. - Which command line options do I need to use? Use -vec_type viennacl -mat_type aijviennacl On Intel's Xeon Phi you also need to pass -viennacl_device_accelerator. (If you read this email in an archive: Check the manual for current flags). Best regards, Karli [1] http://viennacl.sourceforge.net/ [2] http://www.khronos.org/registry/cl/

1 0

Re: [petsc-dev] Errors in ThreadComm stuff
by Jed Brown 21 Mar '13

21 Mar '13

Did you enable threadcomm? I better finish this work I started and get my branch out there... On Thu, Mar 21, 2013 at 6:42 PM, Mark F. Adams <mark.adams(a)columbia.edu>wrote: > I'm getting segv in at the end of runs on the Cray XC30. in optimized > builds it segv with a stack like > > PetscFinalize > PetscViewerASCIIGetStdout > PetscCommDuplicate > PetscThreadCommGetRank > > In debug builds it does not segv but there are a bunch of un freeded > memory in ThreadComm stuff. > > Mark

1 0

Errors in ThreadComm stuff
by Mark F. Adams 21 Mar '13

21 Mar '13

I'm getting segv in at the end of runs on the Cray XC30. in optimized builds it segv with a stack like PetscFinalize PetscViewerASCIIGetStdout PetscCommDuplicate PetscThreadCommGetRank In debug builds it does not segv but there are a bunch of un freeded memory in ThreadComm stuff. Mark

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Karl Rupp 21 Mar '13

21 Mar '13

Hey, > Alright, here we go: > https://bitbucket.org/petsc/__petsc/commits/__9200956088015849f17bb1ff07731… > <https://bitbucket.org/petsc/petsc/commits/9200956088015849f17bb1ff07731f2d9…> > > Note that it fixes the problem by removing code only :-) > > > I get a warm fuzzy feeling when I see only red in the diff. I think the > same change can be made to BAIJ and SBAIJ. MOAR DELETIONS! as you wish, Milord: https://bitbucket.org/petsc/petsc/commits/547bc4d4d2815e4285fa0d952804660ec… Feels indeed a bit weird, but I like it :-) Best regards, Karli

1 0

Re: [petsc-dev] petsc/petsc added -pc_use_amat option to PCSetFromOptions() (commit b9ee023)
by Barry Smith 21 Mar '13

21 Mar '13

On Mar 21, 2013, at 1:15 PM, Jed Brown <commits-noreply(a)bitbucket.org> wrote: > New comment on changeset: > > https://bitbucket.org/petsc/petsc/commits/b9ee023e1977adecb25629462ec8ac3b6… > > Jed Brown (jedbrown) said: > > Why isn't this `PetscOptionsList()` so that it shows up in `-help`? My mistake, I cut and pasted the PetscOptionsGetInt() from the line above and figure that it mustn't be in a PetscOptionsBegin{} phase. You can fix it if you like. Barry This is why wrong code must be removed from PETSc, otherwise it grows by cut and paste. > > -- > This is an comment notification from bitbucket.org. > You are receiving this because you are participating > in a comment thread.

1 0

Re: [petsc-dev] CMake-assisted CUDA builds ready
by Karl Rupp 21 Mar '13

21 Mar '13

Hi again, I've found the cause of the issue, namely an overly tight timeout during the CMake initialization. Dave, could you please rerun reconfigure and make in the latest `next`? Thanks and best regards, Karli On 03/20/2013 07:31 PM, Dave Nystrom wrote: > Hi Karli, > > Just now seeing this. I did my build from a clean slate. > > Thanks, > > Dave > > Karl Rupp writes: > > Hi Dave, > > > > I also found that in some occasions one should issue > > $> rm -rf PETSC_ARCH/CMake* > > prior to running a reconfigure in order to circumvent a bug [1] in CMake. > > > > @Jed, Matt: The current cmakeboot.py removes > > PETSC_ARCH/CMakeFiles/{version_string}, which does not seem to even > > exist on my machine (CMake 2.8.7). Can we instead just delete the whole > > CMakeFiles-folder and probably CMakeCache.txt? > > > > Best regards, > > Karli > > > > [1] http://www.mail-archive.com/[email protected]/msg44765.html > > > > > > > > On 03/20/2013 10:17 AM, Nystrom, William D wrote: > > > Hi Karli, > > > > > > I ran a test build last night and it failed. It was a relatively simple configuration where > > > I was downloading mpich and cmake and building with threads, cuda and the txpetscgpu > > > package. I was building on a Fedora 17 machine with Cuda 5. The failure seem to be > > > associated with cmake. When I get home this evening, I could send you the configure > > > and make logs. I did not have time this morning before leaving for work. I wanted to > > > give you some early feedback even though the logs will have to wait till this evening. > > > > > > Thanks, > > > > > > Dave > > > > > > -- > > > Dave Nystrom > > > LANL HPC-5 > > > Phone: 505-667-7913 > > > Email: wdn(a)lanl.gov > > > Smail: Mail Stop B272 > > > Group HPC-5 > > > Los Alamos National Laboratory > > > Los Alamos, NM 87545 > > > > > > > > > ________________________________________ > > > From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Karl Rupp [rupp(a)mcs.anl.gov] > > > Sent: Tuesday, March 19, 2013 4:04 PM > > > To: petsc-dev(a)mcs.anl.gov > > > Subject: Re: [petsc-dev] CMake-assisted CUDA builds ready > > > > > > Hi Dave, > > > > > > perfect, thanks! > > > I should add that a reconfigure is needed. It won't work with re-running > > > make only. > > > > > > Best regards, > > > Karli > > > > > > > > > On 03/19/2013 04:59 PM, Nystrom, William D wrote: > > >> Hi Karli, > > >> > > >> I will try this soon and report back when I have results. > > >> > > >> Thanks, > > >> > > >> Dave > > >> > > >> ________________________________________ > > >> From: petsc-dev-bounces(a)mcs.anl.gov [petsc-dev-bounces(a)mcs.anl.gov] on behalf of Karl Rupp [rupp(a)mcs.anl.gov] > > >> Sent: Tuesday, March 19, 2013 3:58 PM > > >> To: For users of the development version of PETSc > > >> Subject: [petsc-dev] CMake-assisted CUDA builds ready > > >> > > >> Dear all, > > >> > > >> I just merged the CMake-based builds for CUDA-packages into 'next', > > >> reducing compilation times considerably compared to the old legacy > > >> builds. While the legacy builds print something like > > >> 'libfast in <DIR>' > > >> the new CMake-based builds print the usual percentage counters in front. > > >> > > >> Any feedback on the proper functionality (both positive and negative) > > >> are appreciated. > > >> > > >> Thanks and best regards, > > >> Karli > > >> > > > > > >

1 0

Re: [petsc-dev] PETSc port to OpenCL
by Reymond, Antoine 21 Mar '13

21 Mar '13

Customer is saying: We are performing nonlinear SNES solvers with linear KSP solves inside of them. The KSP solver types that we use are CG, CGS, BiCG, BiCGS, GMRES, Richardson, and LSQR; though Richardson and LSQR are very infrequently used. We use a block Jacobi preconditioner with the following sub-preconditioner types: None, ILU, LU, ICC, Cholesky, and Jacobi. The global system sizes range from 10k x 10k to 20M x 20M depending on model size. We typically run the smaller models on 4-core workstations and the larger models with between 32 and 128 cores on an HPC. The workstations will only have a single GPU, but the HPC may have one GPU per 16 cores (1 GPU per HPC node). Antoine From: petsc-dev-bounces(a)mcs.anl.gov [mailto:[email protected]] On Behalf Of Paul Mullowney Sent: Wednesday, March 13, 2013 1:43 PM To: For users of the development version of PETSc Subject: Re: [petsc-dev] PETSc port to OpenCL Hi Antoine, I don't think one can claim that PETSc supports OpenCL right now. However, I think progress has definitely been made in getting core vector computations (for a Krylov algorithm) to run via OpenCL. Karl, do you have any comment on this? Once those are in place, it wouldn't be too difficult to get a prototype matrix class into PETSc where the other core pieces of a solve, mainly SpMV (and potentially the triangular solves) run on AMD devices. The real question is, what algorithms does your customer want to run? i.e. is it a KSP solve? What is the preconditioner? How big is the system? Are multiple GPUs neeeded? -Paul We (AMD FirePro) have a customer using PETSc and trying to port their thermal/ablation solver code to OpenCL. Do you support AMD GPUs and OpenCL? Ref: Comparing the AMD FirePro S10000 to the Nvidia Tesla K10, K20 and K20X<http://fireuser.com/blog/comparing_the_amd_firepro_s10000_to_the_tesla_k10_…> AMD FirePro(tm) S10000 Server Graphics<http://www.amd.com/us/products/workstation/graphics/firepro-remote-graphics…> Heterogeneous Compute - Tools<http://developer.amd.com/tools/> Regards, Antoine Reymond Sr. Strategic Alliances Manager AMD Professional Graphics O: +(1) 949 336 6244 M: +(1) 949 870 2200 [cid:[email protected]] [52324A_FireProTechLogo_S_E s] Visit us at: Twitter<https://twitter.com/AMDFirePro> | amd.com<http://www.fireprographics.com/>

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Karl Rupp 21 Mar '13

21 Mar '13

Hey, >> Alright, here we go: >> https://bitbucket.org/petsc/petsc/commits/9200956088015849f17bb1ff07731f2d9… >> >> Note that it fixes the problem by removing code only :-) >> >> Karl wins the Barry Smith Code Removal Award :) > > +1 Cheers :-D As Jed noted, I've got even more to delete in BAIJ and SBAIJ! :-) Best regards, Karli

1 0