February 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Jed Brown 28 Feb '12

28 Feb '12

On Mon, Feb 27, 2012 at 23:31, Matthew Knepley <knepley(a)gmail.com> wrote: > I like this because we are using DM for our equation abstraction. I think > the dependency loops are due to refusal to > decompose the DM. > What do you mean by "decompose the DM"?

1 0

Regarding an error
by Chen S 28 Feb '12

28 Feb '12

Hi , I'm building petsc using cygwin (on a windows machine), and I got the following errors while running "make all", (this is the latest build, i've updated on Feb 28th. ): C:\cygwin\home\PETSC-~4\src\mat\coarsen\impls\mis\mis.c(352) : error C2016: C requires that a struct or union has at least one member C:\cygwin\home\PETSC-~4\src\mat\coarsen\impls\hem\hem.c(322) : error C2016: C requires that a struct or union has at least one member C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(421) : error C2143: syntax error : missing ';' before 'type' C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(424) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(424) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(424) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(432) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(432) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(432) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(434) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(434) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(434) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(436) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(436) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(436) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(437) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(437) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(437) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(438) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(438) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(438) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(439) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(439) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(439) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(440) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(440) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(440) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(442) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(442) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\snes\impls\ms\ms.c(442) : error C2065: 'ierr' : undeclared identifier C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(516) : error C2143: syntax error : missing ';' before 'type' C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : error C2065: 'selected' : undeclared identifier C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4047: 'function' : 'IS *' differs in levels of indirection from 'int *' C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4024: 'function through pointer' : different types for formal and actual parameter 3 C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : error C2065: 'llist' : undeclared identifier C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4047: 'function' : 'IS *' differs in levels of indirection from 'int *' C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(519) : warning C4024: 'function through pointer' : different types for formal and actual parameter 4 C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : error C2065: 'selected' : undeclared identifier C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4047: 'function' : 'IS' differs in levels of indirection from 'int' C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4024: 'function through pointer' : different types for formal and actual parameter 4 C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : error C2065: 'llist' : undeclared identifier C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4047: 'function' : 'IS' differs in levels of indirection from 'int' C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(520) : warning C4024: 'function through pointer' : different types for formal and actual parameter 5 C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(538) : error C2065: 'selected' : undeclared identifier C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(538) : warning C4047: 'function' : 'IS *' differs in levels of indirection from 'int *' C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(538) : warning C4024: 'ISDestroy' : different types for formal and actual parameter 1 C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(539) : error C2065: 'llist' : undeclared identifier C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(539) : warning C4047: 'function' : 'IS *' differs in levels of indirection from 'int *' C:\cygwin\home\PETSC-~4\src\ksp\pc\impls\gamg\gamg.c(539) : warning C4024: 'ISDestroy' : different types for formal and actual parameter 1

1 0

Re: [petsc-dev] PETSc GPU capabilities
by Chetan Jhurani 28 Feb '12

28 Feb '12

John, Paul, I ran the example with the same options and the code aborts at a different location in cusp. Although still called by PCSetUp_SACUSP. The example works fine if txpetscgpu is not used. Valgrind does not show any relevant issues prior to the std::terminate. My best guess based on this and some investigation is that this is happening because of inconsistent C style casts in the code (which are #ifdefed out when txpetscgpu is not used). They could be related to different code paths taken in calling MatCUSPCopyToGPU in sacusp.cu depending on txpetscgpu macro. I'm busy with other stuff, but I'll let you know when this gets fixed. Chetan From: petsc-dev-bounces(a)mcs.anl.gov [mailto:[email protected]] On Behalf Of John Fettig Sent: Monday, February 27, 2012 2:02 PM To: For users of the development version of PETSc Subject: Re: [petsc-dev] PETSc GPU capabilities It finally finished running through cuda-gdb. Here's a backtrace. new_size=46912574500784 in the call to thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::resize looks suspicious. #0 0x0000003e1c832885 in raise () from /lib64/libc.so.6 #1 0x0000003e1c834065 in abort () from /lib64/libc.so.6 #2 0x0000003e284bea7d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6 #3 0x0000003e284bcc06 in ?? () from /usr/lib64/libstdc++.so.6 #4 0x0000003e284bcc33 in std::terminate() () from /usr/lib64/libstdc++.so.6 #5 0x0000003e284bcd2e in __cxa_throw () from /usr/lib64/libstdc++.so.6 #6 0x00002aaaab45ad71 in thrust::detail::backend::cuda::malloc<0u> (n=375300596006272) at malloc.inl:50 #7 0x00002aaaab454322 in thrust::detail::backend::dispatch::malloc<0u> (n=375300596006272) at malloc.h:56 #8 0x00002aaaab453555 in thrust::device_malloc (n=375300596006272) at device_malloc.inl:32 #9 0x00002aaaab46477d in thrust::device_malloc<double> (n=46912574500784) at device_malloc.inl:38 #10 0x00002aaaab461fce in thrust::device_malloc_allocator<double>::allocate ( this=0x7fffffff9880, cnt=46912574500784) at device_malloc_allocator.h:101 #11 0x00002aaaab45ee91 in thrust::detail::contiguous_storage<double, thrust::device_malloc_allocator<double> >::allocate (this=0x7fffffff9880, n=46912574500784) at contiguous_storage.inl:134 #12 0x00002aaaab46ebba in thrust::detail::contiguous_storage<double, thrust::device_malloc_allocator<double> >::contiguous_storage (this=0x7fffffff9880, n=46912574500784) at contiguous_storage.inl:46 #13 0x00002aaaab46cd1e in thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::fill_insert (this=0x13623990, position=..., n=46912574500784, x=@0x7fffffff9f18) at vector_base.inl:792 #14 0x00002aaaab46b058 in thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::insert (this=0x13623990, position=..., n=46912574500784, x=@0x7fffffff9f18) at vector_base.inl:561 #15 0x00002aaaab4692a3 in thrust::detail::vector_base<double, thrust::device_malloc_allocator<double> >::resize (this=0x13623990, new_size=46912574500784, x=@0x7fffffff9f18) at vector_base.inl:222 #16 0x00002aaaac2c3d9b in cusp::precond::smoothed_aggregation<int, double, thrust::detail::cuda_device_space_tag>::smoothed_aggregation<cusp::csr_matrix<int, double, thrust::detail::cuda_device_space_tag> > (this=0x136182b0, A=..., theta=0) at smoothed_aggregation.inl:210 #17 0x00002aaaac27cf84 in PCSetUp_SACUSP (pc=0x1360f330) at sacusp.cu:76 #18 0x00002aaaac1f0024 in PCSetUp (pc=0x1360f330) at precon.c:832 #19 0x00002aaaabd02144 in KSPSetUp (ksp=0x135d2a00) at itfunc.c:261 #20 0x00002aaaabd0396e in KSPSolve (ksp=0x135d2a00, b=0x135a0fa0, x=0x135a2b50) at itfunc.c:385 #21 0x0000000000403619 in main (argc=17, args=0x7fffffffc538) at ex2.c:217 On Mon, Feb 27, 2012 at 4:48 PM, John Fettig <john.fettig(a)gmail.com> wrote: Hi Paul, This is very interesting. I tried building the code with --download-txpetscgpu and it doesn't work for me. It runs out of memory, no matter how small the problem (this is ex2 from src/ksp/ksp/examples/tutorials): mpirun -np 1 ./ex2 -n 10 -m 10 -ksp_type cg -pc_type sacusp -mat_type aijcusp -vec_type cusp -cusp_storage_format csr -use_cusparse 0 terminate called after throwing an instance of 'thrust::system::detail::bad_alloc' what(): std::bad_alloc: out of memory MPI Application rank 0 killed before MPI_Finalize() with signal 6 This example works fine when I build without your gpu additions (and for much larger problems too). Am I doing something wrong? For reference, I'm using CUDA 4.1, CUSP 0.3, and Thrust 1.5.1 John On Fri, Feb 10, 2012 at 5:04 PM, Paul Mullowney <paulm(a)txcorp.com> wrote: Hi All, I've been developing GPU capabilities for PETSc. The development has focused mostly on (1) An efficient multi-GPU SpMV, i.e. MatMult. This is working well. (2) Triangular Solve used in ILU preconditioners; i.e. MatSolve. The performance of this ... is what it is :| This code is in beta mode. Keep that in mind, if you decide to use it. It supports single and double precision, real numbers only! Complex will be supported at some point in the future, but not any time soon. To build with these capabilities, add the following to your configure line. --download-txpetscgpu=yes The capabilities of the SpMV code are accessed with the following 2 command line flags -cusp_storage_format csr (other options are coo (coordinate), ell (ellpack), dia (diagonal). hyb (hybrid) is not yet supported) -use_cusparse (this is a boolean and at the moment is only supported with csr format matrices. In the future, cusparse will work with ell, coo, and hyb formats). Regarding the number of GPUs to run on: Imagine a system with P nodes, N cores per node, and M GPUs per node. Then, to use only the GPUs, I would run with M ranks per node over P nodes. As an example, I have a system with 2 nodes. Each node has 8 cores, and 4 GPUs attached to each node (P=2, N=8, M=4). In a PBS queue script, one would use 2 nodes at 4 processors per node. Each mpi rank (CPU processor) will be attached to a GPU. You do not need to explicitly manage the GPUs, apart from understanding what type of system you are running on. To learn how many devices are available per node, use the command line flag: -cuda_show_devices -Paul

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Matthew Knepley 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 11:12 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > So we still haven't resolved the issue. Let's start by deciding who owns > the function pointers. Are we putting all the function pointers for SNES > and TS into DM? If this is the right route, I can make a simple container > DM that SNES and TS automatically allocate to store the pointers, so the > current interface would continue to work, at least for a while. I like this because we are using DM for our equation abstraction. I think the dependency loops are due to refusal to decompose the DM. Matt > What is the alternative? > > > On Mon, Feb 27, 2012 at 11:55, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > >> On Mon, Feb 27, 2012 at 11:48, Matthew Knepley <knepley(a)gmail.com> wrote: >> >>> ? My signature for FormJacobianLocal(DM dm, Vec X, Mat Jac, AppCtx >>> *user) definitely has a context, which is >>> required to have DM as the first member. We could require the solver as >>> the second member. >>> >> >> "My" is not the one in DM base. We are suffering some fragmentation >> because different DMs implement the local problem differently. I think it >> would mostly resolve the issues if the DM functions forwarded the SNES and >> TS on to the user, but that makes a dependency loop (unless that code does >> not live in DM; we can hack around it with void*). >> >> Alternatively, we can make TS and SNES have PushDM() and PopDM() methods >> so the callback can be (SNES, Vec, Mat, void *ctx), but SNESGetDM() returns >> the correct DM. Maybe this is too confusing. >> >> In any case, if we fix the DM callbacks, I would prefer to get rid of the >> ability for SNES and TS to hold their own callbacks. We can keep >> essentially the same user interface (though perhaps the DM argument would >> become explicit). >> > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] SNES, Coloring, DM, and TS
by Jed Brown 27 Feb '12

27 Feb '12

So we still haven't resolved the issue. Let's start by deciding who owns the function pointers. Are we putting all the function pointers for SNES and TS into DM? If this is the right route, I can make a simple container DM that SNES and TS automatically allocate to store the pointers, so the current interface would continue to work, at least for a while. What is the alternative? On Mon, Feb 27, 2012 at 11:55, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Mon, Feb 27, 2012 at 11:48, Matthew Knepley <knepley(a)gmail.com> wrote: > >> ? My signature for FormJacobianLocal(DM dm, Vec X, Mat Jac, AppCtx *user) >> definitely has a context, which is >> required to have DM as the first member. We could require the solver as >> the second member. >> > > "My" is not the one in DM base. We are suffering some fragmentation > because different DMs implement the local problem differently. I think it > would mostly resolve the issues if the DM functions forwarded the SNES and > TS on to the user, but that makes a dependency loop (unless that code does > not live in DM; we can hack around it with void*). > > Alternatively, we can make TS and SNES have PushDM() and PopDM() methods > so the callback can be (SNES, Vec, Mat, void *ctx), but SNESGetDM() returns > the correct DM. Maybe this is too confusing. > > In any case, if we fix the DM callbacks, I would prefer to get rid of the > ability for SNES and TS to hold their own callbacks. We can keep > essentially the same user interface (though perhaps the DM argument would > become explicit). >

1 0

Re: [petsc-dev] OpenMP/Vec
by Jed Brown 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 16:31, Gerard Gorman <g.gorman(a)imperial.ac.uk>wrote: > I had a quick go at trying to get some sensible benchmarks for this but > I was getting too much system noise. I am particularly interested in > seeing if the overhead goes to zero if num_threads(1) is used. > What timing method did you use? I did not see overhead going to zero when num_threads goes to 1 when using GCC compilers, but Intel seems to do fairly well. > > I'm surprised by this. I not aware of any compiler that doesn't have > OpenMP support - and then you do not actually enable OpenMP compilers > generally just ignore the pragma. Do you know of any compiler that does > not have OpenMP support which will complain? > Sean points out that omp.h might not be available, but that misses the point. As far as I know, recent mainstream compilers have enough sense to at least ignore these directives, but I'm sure there are still cases where it would be an issue. More importantly, #pragma was a misfeature that should never be used now that _Pragma() exists. The latter is better not just because it can be turned off, but because it can be manipulated using macros and can be explicitly compiled out. > This may not be flexible enough. You frequently want to have a parallel > region, and then have multiple omp for's within that one region. > PetscPragmaOMPObject(obj, parallel) { PetscPragmaOMP(whetever you normally write for this loop) for (....) { } ... and so on } > I think what you describe is close to Fig 3 of this paper written by > your neighbours: > http://greg.bronevetsky.com/papers/2008IWOMP.pdf > However, before making the implementation more complex, it would be good > to benchmark the current approach and use a tool like likwid to measure > the NUMA traffic so we can get a good handle on the costs. > Sure. > Well this is where the implementation details get richer and there are > many options - they also become less portable. For example, what does > all this mean for the sparc64 processors which are UMA. > Delay to runtime, use an ignorant partition for UMA. (Blue Gene/Q is also essentially uniform.) But note that even with uniform memory, cache still makes it somewhat hierarchical. > Not to mention > Intel MIC which also supports OpenMP. I guess I am cautious about > getting too bogged down with very invasive optimisations until we have > benchmarked the basic approach which in a wide range of use cases will > achieve good thread/page locality as illustrated previously. > I guess I'm just interested in exposing enough semantic information to be able to schedule a few different ways using run-time (or, if absolutely necessary, configure-time) options. I don't want to have to revisit individual loops.

1 0

Re: [petsc-dev] the complex dot business
by Matthew Knepley 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 10:55 AM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > I'm afraid the sudden appearance of the blas dot for complex numbers > problem is my fault. > > Traditionally since 199x we never used the complex dot product from blas > because of the return complex type problem. VecDot_Seq() used #ifdef for > complex and a simple loop. > > Since some blas norm suck I off-hand told Shri to use dot for norm and > wasn't thinking the complex case. > > Since then, of course, the complex dot problem has been coming up. > > Maybe it is best to continue to never use complex blas dot and just > change the VecNorm_ routines to use norm for complex and not dot.? Just > throw away this silly problem and time sink? I just pushed a fix for this. The pthread stuff will still have to be fixed since I can't build it. Matt > > Barry > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] OpenMP/Vec
by Sean Farley 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 4:31 PM, Gerard Gorman <g.gorman(a)imperial.ac.uk>wrote: > I'm surprised by this. I not aware of any compiler that doesn't have > OpenMP support - and then you do not actually enable OpenMP compilers > generally just ignore the pragma. Do you know of any compiler that does > not have OpenMP support which will complain? > clang (which is on any mac with xcode >= 4.0) will not compile any OpenMP (but gcc will compile just fine): >>>>>>>>>>>> fatal error: 'omp.h' file not found #include <omp.h> ^ 1 error generated. <<<<<<<<<<<<< It's not in any llvm stuff as of last month: http://www.phoronix.com/scan.php?page=news_item&px=MTA0Mzc Putting this layer of abstraction is really needed (especially if there are any calls to omp_ functions)

1 0

Re: [petsc-dev] OpenMP/Vec
by Gerard Gorman 27 Feb '12

27 Feb '12

Jed Brown emailed the following on 27/02/12 00:39: > On Sun, Feb 26, 2012 at 04:07, Gerard Gorman <g.gorman(a)imperial.ac.uk > <mailto:[email protected]>> wrote: > > > Did you post a repository yet? I'd like to have a look at the code. > > It's on Barry's favourite collaborative software development site of > course ;-) > > https://bitbucket.org/wence/petsc-dev-omp/overview > > > I looked through the code and I'm concerned that all the OpenMP code > is inlined into vec/impls/seq and mat/impls/aij/seq with, as far as I > can tell, no way to use OpenMP for some objects in a simulation, but > not others. I think that all the pragmas should have > num_threads(x->nthreads) clauses. We can compute the correct number of > threads based on sizes when memory is allocated (or specified through > command line options, inherited from related objects, etc). The num_threads(x->nthreads) is worth investigating. However, in the benchmarks I have done so far it would seem that the only two sensible values for nthreads would be 1 or the total number of threads. When you set num_threads(1) it seems that OpenMP (for some implementations at least) is sensible enough not to do anything silly that would introduce scheduling/synchronisation overheads. Once you use more than one thread then you incur the scheduling/synchronisation overheads. As you increase the number of threads you may for small arrays see parallel efficiency decreasing, but I have not seen the actual time increasing. If this is a general result then it might not be a big deal that OpenMP is inlined into vec/impls/seq and mat/impls/aij/seq so long as the num_threads was used as suggested. For determining where the cut off array size for using all or one thread - as interesting option would be to determine this at run time, i.e. learn at run time what the appropriate is. > > I don't think we can get away from OpenMP schedule overhead (several > hundred cycles) even for those objects that we choose not to use > threads for, but (at least with my gcc tests), that overhead is only > about a third of the cost of actually starting a parallel region. I had a quick go at trying to get some sensible benchmarks for this but I was getting too much system noise. I am particularly interested in seeing if the overhead goes to zero if num_threads(1) is used. The next step is to take a look at the EPCC OpenMP microbenchmarks to see if they have tied down these issues. > > It's really not acceptable to insert unguarded > > #pragma omp ... > > into the code because this will generate tons of warnings or errors > with compilers that don't know about OpenMP. It would be better to > test for _Pragma and use I'm surprised by this. I not aware of any compiler that doesn't have OpenMP support - and then you do not actually enable OpenMP compilers generally just ignore the pragma. Do you know of any compiler that does not have OpenMP support which will complain? > > #define PetscPragmatize(x) _Pragma(#x) > #if defined(PETSC_USE_OPENMP) > # define PetscPragmaOMP(x) PetscPragmatize(omp x) > #else > # define PetscPragmaOMP(x) > #endif > > then use > > PetscPragmaOMP(parallel for ...) > > We should probably use a variant for object-based threading > > #define PetscPragmaOMPObject(obj,x) PetscPragmaOMP(x > num_threads((obj)->nthreads)) This may not be flexible enough. You frequently want to have a parallel region, and then have multiple omp for's within that one region. > > In the case of multiple objects, I think you usually want the object > being modified to control the number of threads. I take this point. > > In many cases, I would prefer more control over the partition of the > loop. For example, in many cases, I'd be willing to tolerate a slight > computational imbalance between threads in exchange for working > exclusively within my page. Note that the arithmetic to compute such > things is orders of magnitude less expensive than the > schedule/distribution to threads. I don't know how to do that except to > > PragmaOMP(parallel) { > int nthreads = omp_get_num_threads(); > int tnum = omp_get_thread_num(); > int start,end; > // compute start and end > for (int i=start; i<end; i++) { > // the work > } > } > > We could perhaps capture some of this common logic in a macro: > > #define VecOMPParallelBegin(X,args) do { \ > PragmaOMPObject(X,parallel args) { \ > PetscInt _start, _end; \ > VecOMPGetThreadLocalPart(X,&_start,&_end); \ > { do {} while(0) > > #define VecOMPParallelEnd() }}} while(0) > > VecOMPParallelBegin(X, shared/private ...); > { > PetscInt i; > for (i=_start; i<_end; i++) { > // the work > } > } > VecOMPParallelEnd(); > > That should reasonably give us complete run-time control of the number > of parallel threads per object and their distribution, within the > constraints of contiguous thread partition. I think what you describe is close to Fig 3 of this paper written by your neighbours: http://greg.bronevetsky.com/papers/2008IWOMP.pdf However, before making the implementation more complex, it would be good to benchmark the current approach and use a tool like likwid to measure the NUMA traffic so we can get a good handle on the costs. > That also leaves open the possibility of using libnuma to query and > migrate pages. (For example, a short vector that needs to be accessed > from multiple NUMA nodes might intentionally be faulted with pages > spread apart even though other vectors of similar size might be > accessed from within one NUMA nodes and thus not use threads at all. > (One 4 KiB page is only 512 doubles, but if the memory is local to a > single NUMA node, we wouldn't use threads until the vector length was > 4 to 8 times larger.) Well this is where the implementation details get richer and there are many options - they also become less portable. For example, what does all this mean for the sparc64 processors which are UMA. Not to mention Intel MIC which also supports OpenMP. I guess I am cautious about getting too bogged down with very invasive optimisations until we have benchmarked the basic approach which in a wide range of use cases will achieve good thread/page locality as illustrated previously. Cheers Gerard

1 0

Re: [petsc-dev] petsc-dev on bitbucket
by Jed Brown 27 Feb '12

27 Feb '12

On Mon, Feb 27, 2012 at 16:07, Stefano Zampini <stefano.zampini(a)gmail.com>wrote: > > I just pushed to bitbucket and my write permissions were revoked. Nobody is pushing directly to bitbucket. We all push to petsc.cs.iit.eduand it gets mirrored to bitbucket. Your old credentials should still work there.

1 0