June 2012 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] segv on Cray with 64K processes
by Mark F. Adams 05 Jun '12

05 Jun '12

On Jun 5, 2012, at 10:30 AM, Jed Brown wrote: > On Tue, Jun 5, 2012 at 8:59 AM, Mark F. Adams <mark.adams(a)columbia.edu> wrote: > I've got a consistent segv on the Cray at NERSC with 64K cores. No problems with smaller jobs. > > It seems to happen in here: > > /* Done after init due to a bug in MPICH-GM? */ > ierr = PetscErrorPrintfInitialize();CHKERRQ(ierr); > > Do you have any more information? How did you identify that this was the point of error? printf surrounded by barriers. There are a lot of system calls in PetscErrorPrintfInitialize. I've added printfs to this and submitted the job so I should have that info tomorrow. > > > I can keep diving in, but anyone have any ideas on this? > > Mark >

1 0

Re: [petsc-dev] segv on Cray with 64K processes
by Jed Brown 05 Jun '12

05 Jun '12

On Tue, Jun 5, 2012 at 8:59 AM, Mark F. Adams <mark.adams(a)columbia.edu>wrote: > I've got a consistent segv on the Cray at NERSC with 64K cores. No > problems with smaller jobs. > > It seems to happen in here: > > /* Done after init due to a bug in MPICH-GM? */ > ierr = PetscErrorPrintfInitialize();CHKERRQ(ierr); > Do you have any more information? How did you identify that this was the point of error? > > I can keep diving in, but anyone have any ideas on this? > > Mark

1 0

segv on Cray with 64K processes
by Mark F. Adams 05 Jun '12

05 Jun '12

I've got a consistent segv on the Cray at NERSC with 64K cores. No problems with smaller jobs. It seems to happen in here: /* Done after init due to a bug in MPICH-GM? */ ierr = PetscErrorPrintfInitialize();CHKERRQ(ierr); I can keep diving in, but anyone have any ideas on this? Mark

1 0

Re: [petsc-dev] [petsc-users] use mpic++ instead of mpicc
by Jed Brown 05 Jun '12

05 Jun '12

We should really fix this. We detect a C++ compiler sometimes just for external packages. What about having SOURCECXX and a rule for files with a C++ extension?s We should get this working in the next week so that we can do the Elemental interface without requiring all of PETSc to be built with C++. On Jun 5, 2012 8:29 AM, "Jose E. Roman" <jroman(a)dsic.upv.es> wrote: > > El 05/06/2012, a las 11:59, Degang Wu escribió: > > > Hi, > > > > I am new to Petsc and Slepc, and I rely on the makefile provided in the > example code folders to compile. For example, the makefile in > src/eps/examples/tutorials/makefile. How should I modify the makefile so > that make will use mpic++ rather than mpicc to compile and link? > > > > Wu Degang > > You must configure PETSc --with-clanguage=c++ > > Jose > >

1 0

Re: [petsc-dev] API changes in MatIS
by Jed Brown 05 Jun '12

05 Jun '12

On Tue, Jun 5, 2012 at 4:40 AM, Stefano Zampini <stefano.zampini(a)gmail.com>wrote: > As I understood well, "second approach" is similar to dohrmann's except > for a further reordering which should reduce the computational costs of > factorizations? Such approach sounds to me over-complicated. I'm more > inclined to solve directly the local saddle point problems (either with a > change of basis or not). Are you planning to add interfaces to LDL^T > solvers to PETSc? > Cholmod and Pastix do LDL^T, maybe others. > > >> >> >>> You said you would have the new matrix class to support either more >>>> subdomains per core, or more cores per subdomain. In the latter case, >>>> threaded or mpi matrices (on subcomms)? >>>> >>> >> I would plan to make it support any combination. >> >> > It would be great. I'm wondering how you will accomplish the > communications. I think there should be an intermediate step, before > multiplying by each subdomain matrix owned by the proc (either multiproc or > sequential), where you need to scatter values from the global vector to a > "local" vector containing all the dofs (in the case of more subdomains) > using some information generated by analyzing what the user has passed in > when requesting matrix creation. Then a number of additional scatters to > realize each single "subdomain" multiplication. How do you do this with > mutliprocessors subdomains? PETSc does support scatters between two > different comms (with size > 1)? I think that mixing multisubdomain procs > and multiproc subdomain will be nasty to get it work but definitely an > interesting challenge. > VecScatter should live on the "larger" comm when it needs to scatter to a vector on a subcomm. > Do you mean assembling the matrix on preselected vertices during > MatAssemblyBegin/End? Note that this will imply that standard > Neumann-Neumann methods will not work (they need the unassembled matrix to > solve for the local Schur complements). I'm not too concerned about that since I consider the classic N-N and original FETI methods to be rather special-purpose compared to the newer generation. I would like to limit the number of copies of a matrix to control peak memory usage.

1 0

Re: [petsc-dev] API changes in MatIS
by Stefano Zampini 05 Jun '12

05 Jun '12

Hi Jed, sorry for late reply, I was very busy 2012/5/26 Jed Brown <jedbrown(a)mcs.anl.gov> > On Sat, May 26, 2012 at 9:39 AM, Stefano Zampini < > stefano.zampini(a)gmail.com> wrote: > >> One we have the constraint matrix, we can easily obtain the change of >> basis matrix T (as in Klawonn-Widlund papers). >> Note that the change of basis approach will be very effective for exact >> applications with reduced iterations. I think we should include in the new >> matrix class the possibility of doing iterations on the reduced space >> instead of the whole space of dofs. >> > > Indeed. You might be aware of the "second approach" described here (CPAM > 2006) > > https://ftp.cs.nyu.edu/web/Research/TechReports/TR2004-855/TR2004-855.pdf > > I think it's worth testing, but since the number of integral Lagrange > multipliers is reasonably small, I would not a priori be concerned by > solving the saddle point problem in which they are ordered last. If there > are enough vertex constraints to keep the subdomains nonsingular, you can > easily handle the problem manually (as Dohrmann does), but I wouldn't > expect modern LDL^T packages to be slowed greatly by a handful of Lagrange > multipliers and you want it to reorder anyway to reduce fill during the > factorization. > As I understood well, "second approach" is similar to dohrmann's except for a further reordering which should reduce the computational costs of factorizations? Such approach sounds to me over-complicated. I'm more inclined to solve directly the local saddle point problems (either with a change of basis or not). Are you planning to add interfaces to LDL^T solvers to PETSc? > > >> You said you would have the new matrix class to support either more >>> subdomains per core, or more cores per subdomain. In the latter case, >>> threaded or mpi matrices (on subcomms)? >>> >> > I would plan to make it support any combination. > > It would be great. I'm wondering how you will accomplish the communications. I think there should be an intermediate step, before multiplying by each subdomain matrix owned by the proc (either multiproc or sequential), where you need to scatter values from the global vector to a "local" vector containing all the dofs (in the case of more subdomains) using some information generated by analyzing what the user has passed in when requesting matrix creation. Then a number of additional scatters to realize each single "subdomain" multiplication. How do you do this with mutliprocessors subdomains? PETSc does support scatters between two different comms (with size > 1)? I think that mixing multisubdomain procs and multiproc subdomain will be nasty to get it work but definitely an interesting challenge. > Tell me if you think it's a bad idea, but my thought was to pre-select a > few coarse vertices (ideally enough to prevent floating subdomains, but the > method should be able to tolerate if not). Then the form that gets > assembled due to normal MatSetValuesLocal() would have nonsingular > subdomains. We can factor those subdomains and then adaptively select > additional vertex or integral constraints to enrich the coarse space. As > soon as we have enough constraints to make the subdomain nonsingular (i.e. > we would usually start nonsingular), we can enforce all additional > constraints with Lagrange multipliers so that the factorization does not > need to be repeated. > > Once we have enriched the coarse space, we have a choice of whether to > apply a change of basis and re-factor or to reuse the current factorization > and keep enforcing the enrichment by Lagrange multipliers. > Do you mean assembling the matrix on preselected vertices during MatAssemblyBegin/End? Note that this will imply that standard Neumann-Neumann methods will not work (they need the unassembled matrix to solve for the local Schur complements). -- Stefano

1 0

builder broken
by Barry Smith 04 Jun '12

04 Jun '12

File "setup.py", line 180, in run build(self.dry_run) File "setup.py", line 132, in build import builder File "/Users/barrysmith/Src/petsc-dev/config/builder.py", line 571 except KeyError as e: builder doesn't work with python 2.5 (as it should) since it is apparently used by setup.py Barry

1 0

Re: [petsc-dev] What needs to be done before a release?
by Barry Smith 03 Jun '12

03 Jun '12

On Jun 3, 2012, at 8:15 AM, Blaise Bourdin wrote: >> > > yes, I was trying to get rid of the static maximum size for farray. There are two hard-coded static sizes in PetscBagRegisterEnum: that the number of entries is less than 97, and that the length of each of the enum name is at most 98 char. The error message you get with an enum with more than 97 entries is not very informative. > I don't know how to get the size of the list of names without using the size function. > > Are these static length a big deal? probably no. I increased the sizes to 256 and also added checking if either one is violated; it will set the ierr argument (no printing error message stuff because I don't want to do that from Fortran). Barry > > >> >> >>> I would understand in the situation of pure f77 compatibility for instance, but we threw this away with the iso_c_binding module anyway... >> >> I want the user experience to be as close to f77 (what they are used to) as possible and not introduced unnecessary flater crap. > > That's fair. > > Blaise > -- > Department of Mathematics and Center for Computation & Technology > Louisiana State University, Baton Rouge, LA 70803, USA > Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin > > > > > > >

1 0

Re: [petsc-dev] What needs to be done before a release?
by Blaise Bourdin 03 Jun '12

03 Jun '12

On Jun 1, 2012, at 6:35 PM, Barry Smith wrote: > > On Jun 1, 2012, at 1:34 PM, Blaise Bourdin wrote: > >> >> On Jun 1, 2012, at 12:47 PM, Barry Smith wrote: >> >>> >>> On Jun 1, 2012, at 12:37 PM, Blaise Bourdin wrote: >>>> >>>> What compiler is this? I don't have access to any solaris box, and have pretty bad memories of sun's fortran compilers. My understanding of the C interoperability is that these are perfectly legal look at the example at http://fortranwiki.org/fortran/show/c_loc >>>> >>>> I pushed a change to bagenum.F Could you have a look and see if you get the same error in PetscBagRegisterEnum? >>>> >>>> Also, this binding needs an explicit interface (technically, the old one too, but Barry's trick for getting farray length was a workaround the lack of explicit interface). >>> >>> Huh, what? We don't want to require an explicit interface! Where is this change? What is the url of the change you made? Did you put it in petsc-dev or petsc-3.3? Why do you need to change it to require an explicit interface. >>> >>> Be like Jed and when you say you made some change include the url business so people can jump immediately to the change you are talking about instead of wasting 1/2 and hour trying to find something they cannot find. >> >> Apologies, will do next time. That would also have helped me realize that I had pushed to my own clone, not petsc-3.3 or petsc-dev repository... I am attaching the patch I was planing on submitting. >> >> Out of curiosity, what is the rationale for not wanting an explicit interface? As far as I understand, the fortran standard mandates an explicit interface in this situation (array of strings). > > I'm confused. F77 supported passing an array of strings but doesn't even have the concept of an explicit interface so how can flater require an explicit interface for the exact same call? > > Is it because you want to use size(FArray) instead of looking for the '' string? yes, I was trying to get rid of the static maximum size for farray. There are two hard-coded static sizes in PetscBagRegisterEnum: that the number of entries is less than 97, and that the length of each of the enum name is at most 98 char. The error message you get with an enum with more than 97 entries is not very informative. I don't know how to get the size of the list of names without using the size function. Are these static length a big deal? probably no. > > >> I would understand in the situation of pure f77 compatibility for instance, but we threw this away with the iso_c_binding module anyway... > > I want the user experience to be as close to f77 (what they are used to) as possible and not introduced unnecessary flater crap. That's fair. Blaise -- Department of Mathematics and Center for Computation & Technology Louisiana State University, Baton Rouge, LA 70803, USA Tel. +1 (225) 578 1612, Fax +1 (225) 578 4276 http://www.math.lsu.edu/~bourdin

1 0

Re: [petsc-dev] MatInvertBlockDiagonal
by Jed Brown 02 Jun '12

02 Jun '12

On Sat, Jun 2, 2012 at 9:27 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > This routine currently returns PetscScalar**, but I don't think it was > ever intended to be writable. I'm changing it to const PetscScalar** unless > someone speaks up. Also, the auto Fortran interface was incorrect, thus > clearly unused. Specifically, tell me if you want this reverted. http://petsc.cs.iit.edu/petsc/releases/petsc-3.3/rev/a86c8fe2e184

1 0