March 2011 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] what happened to information about setting -display DISPLAY etc and getting it to work?
by Jed Brown 31 Mar '11

31 Mar '11

On Thu, Mar 31, 2011 at 16:48, Jed Brown <jed(a)59a2.org> wrote: > no need to always -display :0 to sign in umm, typed in wrong text box. No need to use -display :0 on most systems, and DISPLAY from rank 0 is propagated if not set otherwise. Do you have a case where it is failing?

1 0

Re: [petsc-dev] what happened to information about setting -display DISPLAY etc and getting it to work?
by Jed Brown 31 Mar '11

31 Mar '11

On Thu, Mar 31, 2011 at 16:45, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > what happened to information about setting -display DISPLAY etc and getting > it to work under various senerios? I cannot find it in faq.html is it > somewhere else or did it get lost over the years? > I don't know. I fixed the common cases a while back so it should automatically work in most common configurations (no need to always -display :0 to sign in).

1 0

what happened to information about setting -display DISPLAY etc and getting it to work?
by Barry Smith 31 Mar '11

31 Mar '11

what happened to information about setting -display DISPLAY etc and getting it to work under various senerios? I cannot find it in faq.html is it somewhere else or did it get lost over the years? Thanks Barry

1 0

Re: [petsc-dev] [petsc-maint #66751] Re: [petsc-users] Which preconditioners are scalable?
by Jed Brown 31 Mar '11

31 Mar '11

On Thu, Mar 31, 2011 at 02:50, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > Don't ask me, ask US! I know Matt and Jed did a couple of things but > don't know what is left to be done. > > Everyone please chip in what needs to be done and if they can contribute. > I think Hong finished the AO so she might be interested in tackling the rest > of what needs to be resolved here and Satish can help here since she did > much of the original ctable stuff. > I fixed a non-scalable row map in MatIncreaseOverlap_MPIAIJ_Once, there is a new PetscLayoutFindOwner (cheap, binary searches the row map) which makes it very easy. This change needs to be replicated for BAIJ: changeset: 18476:59612080e9b0 user: Jed Brown <jed(a)59A2.org> date: Sun Mar 13 20:34:14 2011 +0100 summary: Add PetscLayoutFindOwner and eliminate non-scalable "rtable" from MatIncreaseOverlap_MPIAIJ_Once The column map allocated at mpiov.c:1133,1134 and row map allocated at mpiov.c:1177,1178 need to be replaced with a hash (PetscTable). I think those are all the large allocations, can look more carefully another time.

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Jed Brown 31 Mar '11

31 Mar '11

On Tue, Mar 29, 2011 at 15:29, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > Is there any example which makes use of it? src/snes/examples/tutorials/ex28.c is a (contrived) multi-physics prototype that uses MatNest tranparently if you run with -pack_dm_mat_type nest (currently, you should add -pc_type fieldsplit -pc_fieldsplit_type additive because off-diagonal blocks are not automatically preallocated). There are also two tests that use the low-level API: $ grep -l MatNest src/**/tests/*.c src/ksp/ksp/examples/tests/ex22.c src/snes/examples/tests/ex17.c

1 0

Re: [petsc-dev] [petsc-maint #66751] Re: [petsc-users] Which preconditioners are scalable?
by Barry Smith 30 Mar '11

30 Mar '11

Sebastian, Don't ask me, ask US! I know Matt and Jed did a couple of things but don't know what is left to be done. Everyone please chip in what needs to be done and if they can contribute. I think Hong finished the AO so she might be interested in tackling the rest of what needs to be resolved here and Satish can help here since she did much of the original ctable stuff. Barry On Mar 30, 2011, at 9:32 AM, Sebastian Steiger wrote: > Barry, > > Have you by any chance had the time to look into this issue? > > Sebastian > > > > On 03/12/2011 07:48 PM, Barry Smith wrote: >> >> On Mar 12, 2011, at 6:42 PM, Matthew Knepley wrote: >> >>> On Sat, Mar 12, 2011 at 6:36 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> >>> >>> I thought all that code is there and turned on. I have PETSC_USE_CTABLE defined. Why doesn't he? >> >> Only some things in parallel matrices has the memory scalable version, like putting values into an assembled matrix so that code is scattered with PETSC_USE_CTABLE. >> >> Other code like the get overlap and get submatrices was never made memory scalable using ctable. That is the code that is screwing him. I had completely forgotten/not thought about the non memory scaleble issues in these routines for 15 years. >> >> >> Barry >> >>> >>> Matt >>> >>> Sebastian, >>> >>> Now you see the awful truth. Satish and I wrote this code 15 years ago and using the array colmap was ok then. I got old and the machines slowly got larger making what use to be ok no longer scalable. >>> >>> >>> Barry >>> >>>> >>>> I don't know the meaning of these variables, but the latter two look >>>> suspicious to me :-) >>>> >>>> >>>> >>>>> We'll get this figured out and fixed even if it burns all the >>>> bandwidth between purdue and ANL. >>>> >>>> I don't know what that means, but I'm glad you are willing to help me. >>>> >>>> >>>> >>>> Best >>>> Sebastian >>>> >>>> >>>> >>>> >>>> >>>> On 03/12/2011 03:23 PM, Barry Smith wrote: >>>>> >>>>> On Mar 12, 2011, at 2:12 PM, Sebastian Steiger wrote: >>>>> >>>>>> Barry, >>>>>> >>>>>> I have checked out the development version and was able to compile it - >>>>>> however, there is a major problem because my code also uses SLEPc and >>>>>> that one isn't compatible anymore with the development version of PETSc >>>>>> ( seems like you guys changed EXTERN->extern, and also >>>>>> INSTALLDIR->DESTDIR in petscvariables - I suppose there are more changes >>>>>> like this). >>>>>> >>>>>> Hence I cannot link my application against petsc-dev. However, I >>>>>> compared src/mat/impls/aij/mpi/miov.c in petsc-3.1-p4 (which I am using) >>>>>> and petsc-dev, and I presume you inserted >>>>>> >>>>>> ierr = PetscInfo2(C,"Number of outgoing messages %D Total message length >>>>>> %D\n",nrqs,msz);CHKERRQ(ierr); >>>>>> >>>>>> I have inserted this line manually into petsc-3.1-p4 and am now >>>>>> recompiling it. Then I rerun the stuff using -info. >>>>>> >>>>> OK >>>>> >>>>>> Best >>>>>> Sebastian >>>>>> >>>>>> >>>>>> >>>>>> On 03/11/2011 11:00 PM, Barry Smith wrote: >>>>>>> >>>>>>> Sebastian >>>>>>> >>>>>>> I've gone through all the mallocs in MatGetSubMatrices_MPIAIJ_Local() and don't see why there should be a problem with "normal" matrices. >>>>>>> >>>>>>> I checked also in your data the memory used in the local part of the original matrix and then in the overlapping matrix and that memory scaled, so the problem is not that the memory of the matrices grows, making it more likely your matrices are "normal". >>>>>>> >>>>>>> I have added a PetscInfo() call to MatGetSubMatrices_MPIAIJ_Local to provide useful information to further debug the problem. So if you could hg pull ; recompile in the src/mat/impls/aij/mpi and relink the program than run the 300 and 600 cases with the option -info and send use the two resulting outputs. From this information I can decide what needs to be done next. >>>>>>> >>>>>>> Once we track down the issue in atGetSubMatrices_MPIAIJ_Local() I think it is likely the same cause is there for the get overlap and that will be easy to find. >>>>>>> >>>>>>> Barry >>>>>>> >>>>>>> Unfortunately you have to be using petsc-dev to do this. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Mar 11, 2011, at 2:55 PM, Sebastian Steiger wrote: >>>>>>> >>>>>>>> Matt, Barry, >>>>>>>> >>>>>>>> Attached you find some detailed function-resolved information on memory, >>>>>>>> time and flops for my simulation that uses the ASM preconditioner, for >>>>>>>> 300, 600 and 1200 cores. >>>>>>>> >>>>>>>> For this run I have set the ASM overlap to be 0. Seems to me that the >>>>>>>> memory consumption is dominated by MatGetSubMatrices_MPIAIJ_Local() for >>>>>>>> this case. >>>>>>>> >>>>>>>> Also look for the following lines (here for N=300): >>>>>>>> >>>>>>>> [StrainVFF] total problem size (DOFs): 6899904 >>>>>>>> [StrainVFF] p0 stores 8008 atoms >>>>>>>> Avg line fill in Hessian: 46.259490509491 on same process, >>>>>>>> 3.4547952047952 on other processes. >>>>>>>> p0 number of ghost points: 6204 >>>>>>>> >>>>>>>> The fact that all these numbers stay roughly constant except for the >>>>>>>> total problem size tells me that my problem should be scalable. >>>>>>>> >>>>>>>> My simulation actually employs the PETSc newton solver which needs 4 >>>>>>>> iterations to reach convergence. So every run solves Ax=b four times. >>>>>>>> >>>>>>>> I appreciate you looking into the memory scalability issue. >>>>>>>> >>>>>>>> Best >>>>>>>> Sebastian >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 03/11/2011 11:56 AM, Matthew Knepley wrote: >>>>>>>>> On Fri, Mar 11, 2011 at 9:52 AM, Sebastian Steiger <steiger(a)purdue.edu >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> Hello PETSc developers >>>>>>>>> >>>>>>>>> I'm doing some scaling benchmarks and I found that the parallel asm >>>>>>>>> preconditioner, my favorite preconditioner, has a limit in the number of >>>>>>>>> cores it can handle. >>>>>>>>> >>>>>>>>> I am doing a numerical experiment where I scale up the size of my matrix >>>>>>>>> by roughly the same factor as the number of CPUs employed. When I look >>>>>>>>> at which function used how much memory using PETSc's routine >>>>>>>>> PetscMallocDumpLog, I see the following: >>>>>>>>> >>>>>>>>> Function name N=300 N=600 increase >>>>>>>>> ====================================================================== >>>>>>>>> MatGetSubMatrices_MPIAIJ_Local 75'912'016 134'516'928 1.77 >>>>>>>>> MatIncreaseOverlap_MPIAIJ_Once 168'288'288 346'870'832 2.06 >>>>>>>>> MatIncreaseOverlap_MPIAIJ_Receive 2'918'960 5'658'160 1.94 >>>>>>>>> >>>>>>>>> The matrix sizes are 6'899'904 and 14'224'896, respectively. Above >>>>>>>>> N~5000 CPUs I am running out of memory. >>>>>>>>> >>>>>>>>> >>>>>>>>> We have run ASM on 224,000 processors of the XT5 at ORNL, so something else >>>>>>>>> is going on here. The best thing to do here is send us -log_summary. For >>>>>>>>> attachments, >>>>>>>>> we usually recommend petsc-maint(a)mcs.anl.gov >>>>>>>>> <mailto:[email protected]>. >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>> Here's my question now: Is the asm preconditioner limited from the >>>>>>>>> algorithm point of view, or is it the implementation? I thought that >>>>>>>>> 'only' the local matrices, plus some constant overlap with neighbors, >>>>>>>>> are solved, so that memory consumption should stay constant when I scale >>>>>>>>> up with a constant number of rows per process. >>>>>>>>> >>>>>>>>> Best >>>>>>>>> Sebastian >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>> their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> >>>>>>>> <qd_300.log><qd_600.log><qd_1200.log> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >> >

1 0

Re: [petsc-dev] [petsc-maint #68651] Re: [petsc-users] compile petsc error about mmintrin.h
by Barry Smith 29 Mar '11

29 Mar '11

Matt and Satish, How about on non-Windows systems configure refuses to use Intel compilers if the appropriate environmental variables (that are obtaining by sourcing ccvars.sh and friends) are not set? For example in setCompilers.py at the lines if vendor == 'intel' or not vendor: yield 'icc' yield 'ecc' yield 'win32fe icl' if vendor == 'intel' or not vendor: yield 'icpc' yield 'ccpc' yield 'icc' yield 'ecc' yield 'win32fe icl' check for the existence of icc in the path, if it exists {check the needed environmental variables are set; if not set bitch and moan with a useful error message pointing to faq.html#libimf telling the person to do the proper sourcing. If the environmental variables are set then do the yield.} Without these checks we will continue to be haunted by this type of problem, Barry On Mar 28, 2011, at 11:45 PM, Gong Ding wrote: > Sorry, its my problem. > I didn't source iccvars.sh which set the include and lib path for intel compiler correctly. > For long time, I just include bin directory of intel compiler in my PATH, and use icc directly. > So stupid. > > >> This is very strange. >> >> >> >> 1) The Intel compiler is making understanding this difficult. Normally when a compiler detects an error it gives the exact location where it found the problem, if it is in an include file then it shows where the include file was included from. Here it just prints the error message and goes on. I don't see why it is even including that include file. And why doesn't this problem get triggered in compiling other files???? Other files have the exact same includes so should have the same problem. >> >> >> >> I think there may be a conflict between your machines setup and the Intel compiler. >> >> >> >> 2) Could you please try the following, move the file /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h to something like /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h.backup and then run the make again, what happens? >> >> >> >> You might need to start removing chunks of factimpl.c to see where it is triggering the error. You can change that file and then just run make in that directory until the compiler stops failing and that leads to the problem >> >> >> >> >> >> Do you have access to some other Linux system you can compile on with the Intel compilers. >> >> >> >> Barry >> >> >> >> >> >> On Mar 27, 2011, at 12:48 AM, Gong Ding wrote: >> >> >> >>> The attachment is configure.log and make.log. >> >>> >> >>> >> >>>> >> >>>> >> >>>> We need configure.log and make.log sent to petsc-maint(a)mcs.anl.gov Always error on the side of sending too much information to petsc-maint(a)mcs.anl.gov rather than too few. >> >>>> >> >>>> >> >>>> >> >>>> Barry >> >>>> >> >>>> >> >>>> >> >>>> On Mar 25, 2011, at 1:57 AM, Gong Ding wrote: >> >>>> >> >>>> >> >>>> >> >>>>> On centos5 with icc version 11.1, factimpl.c produced many errors such as >> >>>> >> >>>>> >> >>>> >> >>>>> /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h(78): error: cast to type "__m64" is not allowed >> >>>> >> >>>>> return (__m64) __i; >> >>>> >> >>>>> >> >>>> >> >>>>> This problem exists for a long time. In the petsc version of 3.1, I just comment out >> >>>> >> >>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >> >>>> >> >>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >> >>>> >> >>>>> #endif >> >>>> >> >>>>> in the petscconf.h, and compile it. >> >>>> >> >>>>> >> >>>> >> >>>>> However, petsc-dev has other dependence on mmintrin.h, comment out PETSC_HAVE_XMMINTRIN_H will product other problems. >> >>>> >> >>>>> How can I solve this problem? >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> BTW, it seems gcc works well. >> >>>> >> >>>>> >> >>>> >> >>>>> Gong Ding >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>> <configure.log.gz><make.log> >> >> >> >> >

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Thomas Witkowski 29 Mar '11

29 Mar '11

Jed Brown wrote: > On Tue, Mar 29, 2011 at 13:12, Thomas Witkowski > <thomas.witkowski(a)tu-dresden.de > <mailto:[email protected]>> wrote: > > PCFieldsplitSchurPrecondition seems to do what I want. But how to > create the matrix for the preconditioner? I'm a little bit > confused about it because the matrix/operator S acts on an index > range that does not start from 0 and is not continuous, because > each rank owns only some parts of the interior domain boundaries. > > > We do not currently have a way to set attributes on a "submatrix" that > you will extract in the future. You can implement MatGetSubMatrix to > return your special matrix for that block when called with that pair > of ISs. Or you can use MatNest for the preconditioning matrix, it is > frequently a good storage format when doing fieldsplit and you can use > it unintrusively. MatNest sounds good. Is there any example which makes use of it? Thomas

1 0

Re: [petsc-dev] Preconditioner for Schur complement (pcfieldsplit)
by Jed Brown 29 Mar '11

29 Mar '11

On Tue, Mar 29, 2011 at 13:12, Thomas Witkowski < thomas.witkowski(a)tu-dresden.de> wrote: > PCFieldsplitSchurPrecondition seems to do what I want. But how to create > the matrix for the preconditioner? I'm a little bit confused about it > because the matrix/operator S acts on an index range that does not start > from 0 and is not continuous, because each rank owns only some parts of the > interior domain boundaries. We do not currently have a way to set attributes on a "submatrix" that you will extract in the future. You can implement MatGetSubMatrix to return your special matrix for that block when called with that pair of ISs. Or you can use MatNest for the preconditioning matrix, it is frequently a good storage format when doing fieldsplit and you can use it unintrusively.

1 0

Re: [petsc-dev] [petsc-maint #68651] Re: [petsc-users] compile petsc error about mmintrin.h
by Barry Smith 29 Mar '11

29 Mar '11

Matt and Satish, How about on non-Windows systems configure refuses to use Intel compilers if the appropriate environmental variables (that are obtaining by sourcing ccvars.sh and friends) are not set? For example in setCompilers.py at the lines if vendor == 'intel' or not vendor: yield 'icc' yield 'ecc' yield 'win32fe icl' if vendor == 'intel' or not vendor: yield 'icpc' yield 'ccpc' yield 'icc' yield 'ecc' yield 'win32fe icl' check for the existence of icc in the path, if it exists {check the needed environmental variables are set; if not set bitch and moan with a useful error message pointing to faq.html#libimf telling the person to do the proper sourcing. If the environmental variables are set then do the yield.} Without these checks we will continue to be haunted by this type of problem, Barry On Mar 28, 2011, at 11:45 PM, Gong Ding wrote: > Sorry, its my problem. > I didn't source iccvars.sh which set the include and lib path for intel compiler correctly. > For long time, I just include bin directory of intel compiler in my PATH, and use icc directly. > So stupid. > > >> This is very strange. >> >> >> >> 1) The Intel compiler is making understanding this difficult. Normally when a compiler detects an error it gives the exact location where it found the problem, if it is in an include file then it shows where the include file was included from. Here it just prints the error message and goes on. I don't see why it is even including that include file. And why doesn't this problem get triggered in compiling other files???? Other files have the exact same includes so should have the same problem. >> >> >> >> I think there may be a conflict between your machines setup and the Intel compiler. >> >> >> >> 2) Could you please try the following, move the file /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h to something like /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h.backup and then run the make again, what happens? >> >> >> >> You might need to start removing chunks of factimpl.c to see where it is triggering the error. You can change that file and then just run make in that directory until the compiler stops failing and that leads to the problem >> >> >> >> >> >> Do you have access to some other Linux system you can compile on with the Intel compilers. >> >> >> >> Barry >> >> >> >> >> >> On Mar 27, 2011, at 12:48 AM, Gong Ding wrote: >> >> >> >>> The attachment is configure.log and make.log. >> >>> >> >>> >> >>>> >> >>>> >> >>>> We need configure.log and make.log sent to petsc-maint(a)mcs.anl.gov Always error on the side of sending too much information to petsc-maint(a)mcs.anl.gov rather than too few. >> >>>> >> >>>> >> >>>> >> >>>> Barry >> >>>> >> >>>> >> >>>> >> >>>> On Mar 25, 2011, at 1:57 AM, Gong Ding wrote: >> >>>> >> >>>> >> >>>> >> >>>>> On centos5 with icc version 11.1, factimpl.c produced many errors such as >> >>>> >> >>>>> >> >>>> >> >>>>> /usr/lib/gcc/x86_64-redhat-linux/4.1.2/include/mmintrin.h(78): error: cast to type "__m64" is not allowed >> >>>> >> >>>>> return (__m64) __i; >> >>>> >> >>>>> >> >>>> >> >>>>> This problem exists for a long time. In the petsc version of 3.1, I just comment out >> >>>> >> >>>>> #ifndef PETSC_HAVE_XMMINTRIN_H >> >>>> >> >>>>> #define PETSC_HAVE_XMMINTRIN_H 1 >> >>>> >> >>>>> #endif >> >>>> >> >>>>> in the petscconf.h, and compile it. >> >>>> >> >>>>> >> >>>> >> >>>>> However, petsc-dev has other dependence on mmintrin.h, comment out PETSC_HAVE_XMMINTRIN_H will product other problems. >> >>>> >> >>>>> How can I solve this problem? >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> BTW, it seems gcc works well. >> >>>> >> >>>>> >> >>>> >> >>>>> Gong Ding >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>>> >> >>>> >> >>>> >> >>>> >> >>>> >> >>> <configure.log.gz><make.log> >> >> >> >> >

1 0