March 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Barry Smith 21 Mar '13

21 Mar '13

On Mar 21, 2013, at 11:04 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Thu, Mar 21, 2013 at 11:59 AM, Karl Rupp <rupp(a)mcs.anl.gov> wrote: > Hey, > > > I can confirm that this fixes the issues with GPUs. If, on the other > hand, we pick a), then code gets substantially simpler even than it > is now. As noted, the only drawback is that the default > preallocation for on- and off-processor values is the same. I can > live with that. > > > Still want simpler. > > Matt > > Alright, here we go: > https://bitbucket.org/petsc/petsc/commits/9200956088015849f17bb1ff07731f2d9… > > Note that it fixes the problem by removing code only :-) > > Karl wins the Barry Smith Code Removal Award :) +1 > > Matt > > Best regards, > Karli > > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Matthew Knepley 21 Mar '13

21 Mar '13

On Thu, Mar 21, 2013 at 11:59 AM, Karl Rupp <rupp(a)mcs.anl.gov> wrote: > Hey, > > > I can confirm that this fixes the issues with GPUs. If, on the other >> hand, we pick a), then code gets substantially simpler even than it >> is now. As noted, the only drawback is that the default >> preallocation for on- and off-processor values is the same. I can >> live with that. >> >> >> Still want simpler. >> >> Matt >> > > Alright, here we go: > https://bitbucket.org/petsc/**petsc/commits/** > 9200956088015849f17bb1ff07731f**2d9638da35<https://bitbucket.org/petsc/petsc/commits/9200956088015849f17bb1ff07731f2d9…> > > Note that it fixes the problem by removing code only :-) > Karl wins the Barry Smith Code Removal Award :) Matt > Best regards, > Karli > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Jed Brown 21 Mar '13

21 Mar '13

On Thu, Mar 21, 2013 at 10:59 AM, Karl Rupp <rupp(a)mcs.anl.gov> wrote: > Alright, here we go: > https://bitbucket.org/petsc/**petsc/commits/** > 9200956088015849f17bb1ff07731f**2d9638da35<https://bitbucket.org/petsc/petsc/commits/9200956088015849f17bb1ff07731f2d9…> > > Note that it fixes the problem by removing code only :-) > I get a warm fuzzy feeling when I see only red in the diff. I think the same change can be made to BAIJ and SBAIJ. MOAR DELETIONS!

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Karl Rupp 21 Mar '13

21 Mar '13

Hey, > I can confirm that this fixes the issues with GPUs. If, on the other > hand, we pick a), then code gets substantially simpler even than it > is now. As noted, the only drawback is that the default > preallocation for on- and off-processor values is the same. I can > live with that. > > > Still want simpler. > > Matt Alright, here we go: https://bitbucket.org/petsc/petsc/commits/9200956088015849f17bb1ff07731f2d9… Note that it fixes the problem by removing code only :-) Best regards, Karli

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Matthew Knepley 21 Mar '13

21 Mar '13

On Thu, Mar 21, 2013 at 11:25 AM, Karl Rupp <rupp(a)mcs.anl.gov> wrote: > Hi guys, > > > > I vote a). No new function, gets rid of a stupid optimization that > > does no one any good, and is the simplest. > > > > > > I also think (b) is becoming over-complicated. If someone knows their > > stencil and cares about memory usage, they can just pass the correct > > values. The defaults are terribly for anything but a 5-point stencil > anyway. > > > for completeness, here's how b) would look like: > > https://bitbucket.org/petsc/**petsc/commits/** > 814b6bb1ec0ea67e05a42f7f07cab1**169a9be913<https://bitbucket.org/petsc/petsc/commits/814b6bb1ec0ea67e05a42f7f07cab1169…> > > This has the additional freedom of being able to deal with different > defaults for on- and off-processor blocks correctly at the expense of a > separate function MatSeqAIJSetPreallocation_**SeqAIJAlloc() > > I can confirm that this fixes the issues with GPUs. If, on the other hand, > we pick a), then code gets substantially simpler even than it is now. As > noted, the only drawback is that the default preallocation for on- and > off-processor values is the same. I can live with that. > Still want simpler. Matt > Best regards, > Karli > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Karl Rupp 21 Mar '13

21 Mar '13

Hi guys, > I vote a). No new function, gets rid of a stupid optimization that > does no one any good, and is the simplest. > > > I also think (b) is becoming over-complicated. If someone knows their > stencil and cares about memory usage, they can just pass the correct > values. The defaults are terribly for anything but a 5-point stencil anyway. for completeness, here's how b) would look like: https://bitbucket.org/petsc/petsc/commits/814b6bb1ec0ea67e05a42f7f07cab1169… This has the additional freedom of being able to deal with different defaults for on- and off-processor blocks correctly at the expense of a separate function MatSeqAIJSetPreallocation_SeqAIJAlloc() I can confirm that this fixes the issues with GPUs. If, on the other hand, we pick a), then code gets substantially simpler even than it is now. As noted, the only drawback is that the default preallocation for on- and off-processor values is the same. I can live with that. Best regards, Karli

1 0

Re: [petsc-dev] PETSc/Cygwin and Intel Visual Studio
by Satish Balay 21 Mar '13

21 Mar '13

On Thu, 21 Mar 2013, Chen Shapira wrote: > Hi Everyone, > > While building PETSc in Cygwin, I've noticed the following warning : > "You are using an Intel supplied intrinsic header file with a > third-party compiler." Must be intel compiler issue [when mixing icl with ifort] > It seems that the configure and 'make' were completed well, > but when I'm testing the petsc tutorials/examples, I sometimes get > strange errors/warnings such as : > In ex2 and ex3 : "Error : Window Not Successfuly Created" > In ex7 , ex10 and others : "Warning: win32fe: Library Path Not Found: > /home/petsc-dev/PROGRA~2IntelCOMPOS~3binia32" I'm not sure what the error means. Does the compile go through fine? Is this with PETSc makefiles - with the same compiler env setup [as [per the installation instructions?] You can send us the complete compile log Satish > > Any ideas on what to do ? > > Thanks > Chen >

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Jed Brown 21 Mar '13

21 Mar '13

On Thu, Mar 21, 2013 at 9:21 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > I vote a). No new function, gets rid of a stupid optimization that does no > one any good, and is the simplest. I also think (b) is becoming over-complicated. If someone knows their stencil and cares about memory usage, they can just pass the correct values. The defaults are terribly for anything but a 5-point stencil anyway.

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Matthew Knepley 21 Mar '13

21 Mar '13

On Thu, Mar 21, 2013 at 10:08 AM, Karl Rupp <rupp(a)mcs.anl.gov> wrote: > Hi, > > > > Yes, I want to preserve the behavior Barry described in 2), i.e. > >> only error if the user specified an allocation pattern explicitly. >> I'm only questioning the way it is currently implemented: >> >> If the number of nonzeros is specified as PETSC_DECIDE to >> MatSeqAIJSetPreallocation___**SeqAIJ(), >> it does not set MAT_NEW_NONZERO_ALLOCATION___**ERR. If the number of >> >> nonzeros is larger than zero, it does. So far, so good. >> >> >> If nnz is nonnegative, yes. >> > > right, it's actually nz >= 0. > > > MatMPIAIJSetPreallocation___**MPIAIJ(), however, first checks for the >> >> number of nonzeros. If it is PETSC_DECIDE, it sets defaults and then >> calls MatSeqAIJSetPreallocation___**SeqAIJ() with the defaults. >> Because MatSeqAIJSetPreallocation___**SeqAIJ() no longer sees >> PETSC_DECIDE, it sets MAT_NEW_NONZERO_ALLOCATION___**ERR, which is >> then reset right after in MatMPIAIJSetPreallocation___**MPIAIJ(). The >> >> CUSP-implementation does not reset the flag and thus leads to >> errors, even though the implementation of the CUSP-Preallocation >> routine looks perfectly fine. I rather want to fix the hack in >> MatMPIAIJSetPreallocation___**MPIAIJ() than to propagate it to the >> GPU >> >> implementations as well. >> >> >> I thought perhaps you were going to leave it to the _SeqAIJ parts to >> store the nonew state. If the _MPIAIJ part resolves PETSC_DECIDE >> independently from the _SeqAIJ part, then they have to *always* coincide >> exactly, otherwise we'll have very confusing bugs. You can likely >> forward PETSC_DECIDE down to the _SeqAIJ parts, then query a->nonew and >> b->nonew to set the mpiaij->nonew flag. (I'm not sure it's consistent >> now, but we want it to be consistent.) >> > > There is no mpiaij->nonew flag in Mat_MPIAIJ, this is all handled in the > two diagonal/off-processor matrices of type Mat_SeqAIJ. I don't want to > change this ;-) > I just want to resolve the way way a->nonew and b->nonew is set, so that > it does not require a hack in MatMPIAIJSetPreallocation_**MPIAIJ(). > > A closer look showed three options: > a) just forward PETSC_DECIDE to MatSeqAIJSetPreallocation_**SeqAIJ(), > don't modify the nonew-flags from _MPIAIJ(). This almost fixes the problem, > the only drawback is that the number of nonzeros on the off-processor > blocks is then also 5 instead of 2 when using PETSC_DECIDE. > I vote a). No new function, gets rid of a stupid optimization that does no one any good, and is the simplest. Matt > b) Allow to pass default values to MatSeqAIJSetPreallocation_**SeqAIJ(). > The cause of the issue is that MatSeqAIJSetPreallocation_**SeqAIJ() does > two things: It interprets PETSC_DECIDE and it does the preallocation. If we > split these two tasks into two functions, i.e. > > - MatSeqAIJSetPreallocation_**SeqAIJ() for interpreting PETSC_DECIDE and > dealing with nonew, then calling > - MatSeqAIJSetPreallocation_**SeqAIJAlloc() for the actual preallocation > > we can directly call MatSeqAIJSetPreallocation_**SeqAIJAlloc() from the > _MPIAIJ() part and cleanly set MAT_NEW_NONZERO_ALLOCATION_ERR for the on- > and off-processor matrices. > > c) Improve the existing hack by querying the state of a->nonew and > b->nonew before calling MatSeqAIJSetPreallocation_**SeqAIJ() and after > that reset a user-defined state if necessary. > > I'll create a pull request for b), as it is consistent with the current > behavior and appears to be the cleanest solution to this. > > Best regards, > Karli > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Dealing with off-processor aij-matrices not having preallocation information
by Karl Rupp 21 Mar '13

21 Mar '13

Hi, > Yes, I want to preserve the behavior Barry described in 2), i.e. > only error if the user specified an allocation pattern explicitly. > I'm only questioning the way it is currently implemented: > > If the number of nonzeros is specified as PETSC_DECIDE to > MatSeqAIJSetPreallocation___SeqAIJ(), > it does not set MAT_NEW_NONZERO_ALLOCATION___ERR. If the number of > nonzeros is larger than zero, it does. So far, so good. > > > If nnz is nonnegative, yes. right, it's actually nz >= 0. > MatMPIAIJSetPreallocation___MPIAIJ(), however, first checks for the > number of nonzeros. If it is PETSC_DECIDE, it sets defaults and then > calls MatSeqAIJSetPreallocation___SeqAIJ() with the defaults. > Because MatSeqAIJSetPreallocation___SeqAIJ() no longer sees > PETSC_DECIDE, it sets MAT_NEW_NONZERO_ALLOCATION___ERR, which is > then reset right after in MatMPIAIJSetPreallocation___MPIAIJ(). The > CUSP-implementation does not reset the flag and thus leads to > errors, even though the implementation of the CUSP-Preallocation > routine looks perfectly fine. I rather want to fix the hack in > MatMPIAIJSetPreallocation___MPIAIJ() than to propagate it to the GPU > implementations as well. > > > I thought perhaps you were going to leave it to the _SeqAIJ parts to > store the nonew state. If the _MPIAIJ part resolves PETSC_DECIDE > independently from the _SeqAIJ part, then they have to *always* coincide > exactly, otherwise we'll have very confusing bugs. You can likely > forward PETSC_DECIDE down to the _SeqAIJ parts, then query a->nonew and > b->nonew to set the mpiaij->nonew flag. (I'm not sure it's consistent > now, but we want it to be consistent.) There is no mpiaij->nonew flag in Mat_MPIAIJ, this is all handled in the two diagonal/off-processor matrices of type Mat_SeqAIJ. I don't want to change this ;-) I just want to resolve the way way a->nonew and b->nonew is set, so that it does not require a hack in MatMPIAIJSetPreallocation_MPIAIJ(). A closer look showed three options: a) just forward PETSC_DECIDE to MatSeqAIJSetPreallocation_SeqAIJ(), don't modify the nonew-flags from _MPIAIJ(). This almost fixes the problem, the only drawback is that the number of nonzeros on the off-processor blocks is then also 5 instead of 2 when using PETSC_DECIDE. b) Allow to pass default values to MatSeqAIJSetPreallocation_SeqAIJ(). The cause of the issue is that MatSeqAIJSetPreallocation_SeqAIJ() does two things: It interprets PETSC_DECIDE and it does the preallocation. If we split these two tasks into two functions, i.e. - MatSeqAIJSetPreallocation_SeqAIJ() for interpreting PETSC_DECIDE and dealing with nonew, then calling - MatSeqAIJSetPreallocation_SeqAIJAlloc() for the actual preallocation we can directly call MatSeqAIJSetPreallocation_SeqAIJAlloc() from the _MPIAIJ() part and cleanly set MAT_NEW_NONZERO_ALLOCATION_ERR for the on- and off-processor matrices. c) Improve the existing hack by querying the state of a->nonew and b->nonew before calling MatSeqAIJSetPreallocation_SeqAIJ() and after that reset a user-defined state if necessary. I'll create a pull request for b), as it is consistent with the current behavior and appears to be the cleanest solution to this. Best regards, Karli

1 0