May 2015 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Jed Brown 29 May '15

29 May '15

Mark Adams <mfadams(a)lbl.gov> writes: > Thanks, this code does not have any communication in VecAssembly so I added > the INGNORE stuff and added a barrier before this section of code. I am > suspecting that VecAssembly is catching load imbalance but not reporting it > for some reason. If you put a barrier before, it starts at about the same time on all processes. The operation contains synchronization followed by (scalable) point-to-point messaging. The slowness is almost certainly caused by the non-scalable algorithm -- I've seen it with other applications. So try my branch.

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Mark Adams 29 May '15

29 May '15

Thanks, this code does not have any communication in VecAssembly so I added the INGNORE stuff and added a barrier before this section of code. I am suspecting that VecAssembly is catching load imbalance but not reporting it for some reason. On Fri, May 29, 2015 at 10:54 AM, Jed Brown <jed(a)jedbrown.org> wrote: > Mark Adams <mfadams(a)lbl.gov> writes: > > > I am suspecting that it is catching load imbalance and just not reporting > > it correctly. I've added a barrier in the code. > > > > Here are the two log files. > > Mark, there has always been a worst-case O(n*p) algorithm in > VecStashScatterBegin_Private: > > for (i=0; i<stash->n; i++) { > /* if indices are NOT locally sorted, need to start search at the > beginning */ > if (lastidx > (idx = stash->idx[i])) j = 0; > lastidx = idx; > for (; j<size; j++) { > if (idx >= owners[j] && idx < owners[j+1]) { > nprocs[2*j]++; nprocs[2*j+1] = 1; owner[i] = j; break; > } > } > } > > The branch jed/mat-assembly-perf has a scalable implementation. Can you > try it (either in that branch or in 'next')? >

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Jed Brown 29 May '15

29 May '15

Mark Adams <mfadams(a)lbl.gov> writes: > I am suspecting that it is catching load imbalance and just not reporting > it correctly. I've added a barrier in the code. > > Here are the two log files. Mark, there has always been a worst-case O(n*p) algorithm in VecStashScatterBegin_Private: for (i=0; i<stash->n; i++) { /* if indices are NOT locally sorted, need to start search at the beginning */ if (lastidx > (idx = stash->idx[i])) j = 0; lastidx = idx; for (; j<size; j++) { if (idx >= owners[j] && idx < owners[j+1]) { nprocs[2*j]++; nprocs[2*j+1] = 1; owner[i] = j; break; } } } The branch jed/mat-assembly-perf has a scalable implementation. Can you try it (either in that branch or in 'next')?

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Mark Adams 29 May '15

29 May '15

I am suspecting that it is catching load imbalance and just not reporting it correctly. I've added a barrier in the code. Here are the two log files. On Thu, May 28, 2015 at 7:48 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > VecAssemblyBegin() serves as a barrier unless you set the vector option > VEC_IGNORE_OFF_PROC_ENTRIES so I am not surprised that it "appears" to take > a lot of time. BUT the balance between the fastest and slowest is listed in > your table below is 1.0 which is very surprising; indicating every process > supposedly spent the same amount of time within the VecAssemblyBegin(). > Note that for VecAssemblyEnd() the balance is 2.3 which is what I commonly > would expect. Please send me ALL the output for -log_summary for these > cases. Version of PETSc shouldn't matter for this issue. > > > On May 28, 2015, at 4:59 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > We are seeing some large times spent in VecAssemblyBegin: > > > > VecAssemblyBegin 242 1.0 7.9796e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 7.3e+02 12 0 0 0 5 76 0 0 0 10 0 > > VecAssemblyEnd 242 1.0 5.6624e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > > This is with 64K cores on Edison. On 128K cores (weak speedup) we see: > > > > VecAssemblyBegin 248 1.0 2.3615e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 > 7.4e+02 17 0 0 0 4 87 0 0 0 10 0 > > VecAssemblyEnd 248 1.0 6.8855e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 > 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > > > We are working on using older versions of PETSc to make sure this is a > PETSc issue but does anyone have any thoughts on this? > > > > Mark > >

1 0

Re: [petsc-dev] I have a question about PETSc example.
by Jung, Ki Won 28 May '15

28 May '15

Dear Barry, Thank you for forwarding this e-mail to other users. Regarding the version of PETSc, we would not adhere to a specific version of PETSc as you suggested. The only reason why PETSc 3.2p6 has been used for studying and evaluating another code (2D Helmholtz) because it utilizes 3.2p6. However, since it is a non-native PETSc code and does not work as it should be, I and my colleague do not need to stick to 3.2p6. As you mentioned, by Helmholtz equation, I meant the equation that you showed: -delta u - sigma1*u + i*sigma2*u = f, where sigma1 would be roughly k^2 (i.e., by ignoring -alpha^2) and sigma2 would be 2*k*alpha. u, alpha, and k are the acoustic pressure, the absorption coefficient, and the wavenumber, respectively. Assuming that the frequency range that we are interested in lies between 100 and 1000 Hz, sigma1 would be between 0.175 and 1.754E1 (i.e., assuming the speed of sound is 1500 m/s). I haven't calculated sigma2, but it would be much smaller than sigma1 due to the fact that alpha is much smaller than k. Another thing to be noted is that what we are trying to do is to incorporate the speed of sound profile as a function of position for the problem. For now, we are not thinking about any interaction of sound with ocean bottom yet, although it is near a shoreline. Except the shoreline, all the sides would be open boundary condition by using radiating boundary condition such as PML. The top should be the pressure release boundary condition (i.e., the acoustic pressure would be zero at the surface) and the bottom could be a rigid boundary condition due to lack of density information in the equation. However, all of these are vague ideas for now and could be changed in the future. Please let me know if something is missing. Thank you, Ki Won -----Original Message----- From: Barry Smith [mailto:[email protected]] Sent: Thursday, May 28, 2015 5:31 PM To: Jung, Ki Won; For users of the development version of PETSc Subject: Re: I have a question about PETSc example. I am forwarding this to the other PETSc developers since they know more about these types of problems then I do. > On May 28, 2015, at 6:58 PM, Jung, Ki Won <KiWon.Jung(a)pnnl.gov> wrote: > > Dear Dr. Barry Smith, > > My name is KiWon Jung working as a postdoctoral research associate at Pacific Northwest National Laboratory located in Richland, WA. > My specialty is physical acoustics and I earned my Ph.D. in Acoustics at Pennsylvania State University. > I have used several finite element modeling softwares for years, but I started building a code that will propagate sound wave in ocean using 3D Helmholtz equation approach based on the use of PETSc. (specifically we are trying to use PETSc 3.2p6 which is readily available on our computers for now.) Please DO NOT DO THIS. It is straightforward to install the latest version of PETSc; it will just lead to headaches to use an older version that we do not support and then have to change your code to work with a newer version. Just get the latest release and install it. > > Because I am new to PETSc, I am trying to learn how to formulate my problem using different preconditioners, parallel computing, or multigrid scheme by reading and running examples available in tutorial directory. Luckily, I was able to find some relevant examples such as 2D Helmholtz problem, but I hope that I could get some example of 3D Helmholtz problem. I have been looking for an example using Google, but I was unable to find one. > > My colleague suggests that I should contact you and that is why I am sending you this e-mail. > If you have some examples of 3D Helmholtz using PETSc or know someone who has done the same thing using PETSc, I would very much appreciate if you kindly let me know. If you provide additional information it will help us to provide guidance. By Helmholtz do you mean -delta u - sigma1*u + i*sigma2*u = f, and what range of values of sigma1 and sigma2 matter for your "propagate sound wave in ocean"? What geometry do you need to handle? The ocean bottom? the shore line? objects in the ocean such as ships? the ocean surface? Do you have any "far field boundary conditions? Ideally you could use a higher order (spectral?) method and not just a second order method like in our examples. Depending on the problem Helmholtz can be notoriously difficult for multigrid and is, I would say, a research area. Once we have more information about the details above we can make suggestions as to particular publications to look at initially. Barry > > Thank you for your time. > > Sincerely, > > KiWon Jung

1 0

Re: [petsc-dev] I have a question about PETSc example.
by Barry Smith 28 May '15

28 May '15

I am forwarding this to the other PETSc developers since they know more about these types of problems then I do. > On May 28, 2015, at 6:58 PM, Jung, Ki Won <KiWon.Jung(a)pnnl.gov> wrote: > > Dear Dr. Barry Smith, > > My name is KiWon Jung working as a postdoctoral research associate at Pacific Northwest National Laboratory located in Richland, WA. > My specialty is physical acoustics and I earned my Ph.D. in Acoustics at Pennsylvania State University. > I have used several finite element modeling softwares for years, but I started building a code that will propagate sound wave in ocean using 3D Helmholtz equation approach based on the use of PETSc. (specifically we are trying to use PETSc 3.2p6 which is readily available on our computers for now.) Please DO NOT DO THIS. It is straightforward to install the latest version of PETSc; it will just lead to headaches to use an older version that we do not support and then have to change your code to work with a newer version. Just get the latest release and install it. > > Because I am new to PETSc, I am trying to learn how to formulate my problem using different preconditioners, parallel computing, or multigrid scheme by reading and running examples available in tutorial directory. Luckily, I was able to find some relevant examples such as 2D Helmholtz problem, but I hope that I could get some example of 3D Helmholtz problem. I have been looking for an example using Google, but I was unable to find one. > > My colleague suggests that I should contact you and that is why I am sending you this e-mail. > If you have some examples of 3D Helmholtz using PETSc or know someone who has done the same thing using PETSc, I would very much appreciate if you kindly let me know. If you provide additional information it will help us to provide guidance. By Helmholtz do you mean -delta u - sigma1*u + i*sigma2*u = f, and what range of values of sigma1 and sigma2 matter for your "propagate sound wave in ocean"? What geometry do you need to handle? The ocean bottom? the shore line? objects in the ocean such as ships? the ocean surface? Do you have any "far field boundary conditions? Ideally you could use a higher order (spectral?) method and not just a second order method like in our examples. Depending on the problem Helmholtz can be notoriously difficult for multigrid and is, I would say, a research area. Once we have more information about the details above we can make suggestions as to particular publications to look at initially. Barry > > Thank you for your time. > > Sincerely, > > KiWon Jung

1 0

Re: [petsc-dev] git bisection failure
by Barry Smith 28 May '15

28 May '15

This snapshot of PETSc is broken for MPI that don't explicitly list a mpi include directory. File "/global/u2/m/madams/petsc/config/BuildSystem/config/packages/parmetis.py", line 26, in formCMakeConfigureArgs args.append('-DMPI_INCLUDE_PATH='+self.mpi.include[0]) To get passed it add a -with-mpi-include=/pathtoincludedirectorywhereyouknowmpi.his to the configure arguments. Barry > On May 28, 2015, at 5:33 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > My code is hanging PCFieldSplit and maint seems to work. I started git bisection bit it is failing to download Parmetis. Could there be a missing tarball on the server? > Mark > <configure.log>

1 0

Re: [petsc-dev] new P^1.5 algorithm in VecAssembleBegin?
by Barry Smith 28 May '15

28 May '15

VecAssemblyBegin() serves as a barrier unless you set the vector option VEC_IGNORE_OFF_PROC_ENTRIES so I am not surprised that it "appears" to take a lot of time. BUT the balance between the fastest and slowest is listed in your table below is 1.0 which is very surprising; indicating every process supposedly spent the same amount of time within the VecAssemblyBegin(). Note that for VecAssemblyEnd() the balance is 2.3 which is what I commonly would expect. Please send me ALL the output for -log_summary for these cases. Version of PETSc shouldn't matter for this issue. > On May 28, 2015, at 4:59 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > We are seeing some large times spent in VecAssemblyBegin: > > VecAssemblyBegin 242 1.0 7.9796e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.3e+02 12 0 0 0 5 76 0 0 0 10 0 > VecAssemblyEnd 242 1.0 5.6624e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > This is with 64K cores on Edison. On 128K cores (weak speedup) we see: > > VecAssemblyBegin 248 1.0 2.3615e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 7.4e+02 17 0 0 0 4 87 0 0 0 10 0 > VecAssemblyEnd 248 1.0 6.8855e-04 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > > We are working on using older versions of PETSc to make sure this is a PETSc issue but does anyone have any thoughts on this? > > Mark

1 0

git bisection failure
by Mark Adams 28 May '15

28 May '15

My code is hanging PCFieldSplit and maint seems to work. I started git bisection bit it is failing to download Parmetis. Could there be a missing tarball on the server? Mark

1 0

Re: [petsc-dev] Requiring $prefix/lib for --with-pkg-dir=
by Satish Balay 28 May '15

28 May '15

Here is my slightly modified change [to avoid checking '' twice] Satish ------- $ git diff |cat diff --git a/config/BuildSystem/config/package.py b/config/BuildSystem/config/package.py index 4ff326a..3993c05 100644 --- a/config/BuildSystem/config/package.py +++ b/config/BuildSystem/config/package.py @@ -320,9 +320,10 @@ class Package(config.base.Configure): return os.path.join(prefix, includeDir) def generateGuesses(self): + libdirs = list(set([self.libdir, self.altlibdir, ''])) d = self.checkDownload() if d: - for libdir in [self.libdir, self.altlibdir]: + for libdir in libdirs: libdirpath = os.path.join(d, libdir) if not os.path.isdir(libdirpath): self.logPrint(self.PACKAGE+': Downloaded DirPath not found.. skipping: '+libdirpath) @@ -358,7 +359,7 @@ class Package(config.base.Configure): fakeExternalPackagesDir+' is reserved for --download-package scratch space. \n'+ 'Do not install software in this location nor use software in this directory.') - for libdir in [self.libdir, self.altlibdir]: + for libdir in libdirs: libdirpath = os.path.join(d, libdir) if not os.path.isdir(libdirpath): self.logPrint(self.PACKAGE+': UserSpecified DirPath not found.. skipping: '+libdirpath) @@ -405,7 +406,7 @@ class Package(config.base.Configure): self.logPrint(self.PACKAGE+': SearchDir DirPath not found.. skipping: '+d) continue includedir = self.getIncludeDirs(d, self.includedir) - for libdir in [self.libdir, self.altlibdir]: + for libdir in libdirs: libdirpath = os.path.join(d, libdir) if not os.path.isdir(libdirpath): self.logPrint(self.PACKAGE+': DirPath not found.. skipping: '+libdirpath) diff --git a/config/BuildSystem/config/packages/MPI.py b/config/BuildSystem/config/packages/MPI.py index 72da4cf..bcd621c 100644 --- a/config/BuildSystem/config/packages/MPI.py +++ b/config/BuildSystem/config/packages/MPI.py @@ -68,6 +68,7 @@ class Configure(config.package.Package): def generateLibList(self, directory): if self.setCompilers.usedMPICompilers: self.liblist = [] + self.libdir = [] return config.package.Package.generateLibList(self,directory) # search many obscure locations for MPI On Thu, 28 May 2015, Satish Balay wrote: > Sorry - haven't checked on this - and replying late.. > > On Mon, 11 May 2015, Jed Brown wrote: > > > Satish Balay <balay(a)mcs.anl.gov> writes: > > > > > For one - I want to get rid of 'self.libdir,self.altlibdir' usage [but > > > self.libdir is used allover configure - so I should look at this usage > > > more closely before I make this change]. > > > > > > But to adress your issue - I expected packages that don't have 'lib' to > > > set: [as in config/BuildSystem/config/packages/cusp.py] > > > > > > self.libdir = ''. > > > > > > But to get libdir check working for mpicc etc - perhaps the following would > > > suffice? > > > > > >>>>>>>> > > > $ git diff config/BuildSystem/config/packages/MPI.py > > > diff --git a/config/BuildSystem/config/packages/MPI.py b/config/BuildSystem/config/packages/MPI.py > > > index 0a0cb2a..312de0c 100644 > > > --- a/config/BuildSystem/config/packages/MPI.py > > > +++ b/config/BuildSystem/config/packages/MPI.py > > > @@ -68,6 +68,7 @@ class Configure(config.package.Package): > > > def generateLibList(self, directory): > > > if self.setCompilers.usedMPICompilers: > > > self.liblist = [] > > > + self.libdir = [] > > > return config.package.Package.generateLibList(self,directory) > > > > > > # search many obscure locations for MPI > > > <<<<<<<<< > > > > Why should MPI be special? > > You mean wrt 'self.libdir = []' change? In that piece of code - we are > using MPICompilers [and not a library]. So this is the way to indicate > it. [similar to 'cusp.py' code thats already present] > > > > > > To get this working more generally - perhaps the following change? > > > > > >>>>>>>>> > > > $ git diff config/BuildSystem/config/package.py |cat > > > diff --git a/config/BuildSystem/config/package.py b/config/BuildSystem/config/package.py > > > index 4c1f2c4..7a7da1b 100644 > > > --- a/config/BuildSystem/config/package.py > > > +++ b/config/BuildSystem/config/package.py > > > @@ -322,7 +322,7 @@ class Package(config.base.Configure): > > > def generateGuesses(self): > > > d = self.checkDownload() > > > if d: > > > - for libdir in [self.libdir, self.altlibdir]: > > > + for libdir in [self.libdir, self.altlibdir, '']: > > > libdirpath = os.path.join(d, libdir) > > > if not os.path.isdir(libdirpath): > > > self.logPrint(self.PACKAGE+': Downloaded DirPath not found.. skipping: '+libdirpath) > > > @@ -358,7 +358,7 @@ class Package(config.base.Configure): > > > fakeExternalPackagesDir+' is reserved for --download-package scratch space. \n'+ > > > 'Do not install software in this location nor use software in this directory.') > > > > > > - for libdir in [self.libdir, self.altlibdir]: > > > + for libdir in [self.libdir, self.altlibdir, '']: > > > libdirpath = os.path.join(d, libdir) > > > if not os.path.isdir(libdirpath): > > > self.logPrint(self.PACKAGE+': UserSpecified DirPath not found.. skipping: '+libdirpath) > > > @@ -405,7 +405,7 @@ class Package(config.base.Configure): > > > self.logPrint(self.PACKAGE+': SearchDir DirPath not found.. skipping: '+d) > > > continue > > > includedir = self.getIncludeDirs(d, self.includedir) > > > - for libdir in [self.libdir, self.altlibdir]: > > > + for libdir in [self.libdir, self.altlibdir, '']: > > > libdirpath = os.path.join(d, libdir) > > > if not os.path.isdir(libdirpath): > > > self.logPrint(self.PACKAGE+': DirPath not found.. skipping: '+libdirpath) > > > <<<<<<<<<<<<< > > > > Was there a reason to check in the first place? > > You mean - check for existance of the dir? Its an optimization to > avoid numerous compilers [that would fail anyway] in configure. > > > Having an explicit check for a $prefix/lib directory and then > > circumventing the check just adds complexity. > > I think its still a net gain - and reduces the number of compiles invoked. > > And the check for libdir = '' is explicitly looking at default > compiler paths. [so a more correct thingy - than a successful link > test with a nonexistant path?] > > At some point someone could rewrite this whole thing [with each > package listing its posible libdir list] - but do you object to this > change as a fix to the current code? > > Satish >

1 0