August 2019 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT
by Smith, Barry F. 31 Aug '19

31 Aug '19

Any explanation for why the scaling is much better for CPUs and than GPUs? Is it the "extra" time needed for communication from the GPUs? Perhaps you could try the GPU version with Junchao's new MPI-aware CUDA branch (in the gitlab merge requests) that can speed up the communication from GPUs? Barry > On Aug 30, 2019, at 11:56 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > Here is some more weak scaling data with a fixed number of iterations (I have given a test with the numerical problems to ORNL and they said they would give it to Nvidia). > > I implemented an option to "spread" the reduced coarse grids across the whole machine as opposed to a "compact" layout where active processes are laid out in simple lexicographical order. This spread approach looks a little better. > > Mark > > On Wed, Aug 14, 2019 at 10:46 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > Ahh, PGI compiler, that explains it :-) > > Ok, thanks. Don't worry about the runs right now. We'll figure out the fix. The code is just > > *a = (PetscReal)strtod(name,endptr); > > could be a compiler bug. > > > > > > On Aug 14, 2019, at 9:23 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > I am getting this error with single: > > > > 22:21 /gpfs/alpine/geo127/scratch/adams$ jsrun -n 1 -a 1 -c 1 -g 1 ./ex56_single -cells 2,2,2 -ex56_dm_vec_type cuda -ex56_dm_mat_type aijcusparse -fp_trap > > [0] 81 global equations, 27 vertices > > [0]PETSC ERROR: *** unknown floating point error occurred *** > > [0]PETSC ERROR: The specific exception can be determined by running in a debugger. When the > > [0]PETSC ERROR: debugger traps the signal, the exception can be found with fetestexcept(0x3e000000) > > [0]PETSC ERROR: where the result is a bitwise OR of the following flags: > > [0]PETSC ERROR: FE_INVALID=0x20000000 FE_DIVBYZERO=0x4000000 FE_OVERFLOW=0x10000000 FE_UNDERFLOW=0x8000000 FE_INEXACT=0x2000000 > > [0]PETSC ERROR: Try option -start_in_debugger > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] PetscDefaultFPTrap line 355 /autofs/nccs-svm1_home1/adams/petsc/src/sys/error/fp.c > > [0]PETSC ERROR: [0] PetscStrtod line 1964 /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/options.c > > [0]PETSC ERROR: [0] PetscOptionsStringToReal line 2021 /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/options.c > > [0]PETSC ERROR: [0] PetscOptionsGetReal line 2321 /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/options.c > > [0]PETSC ERROR: [0] PetscOptionsReal_Private line 1015 /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/aoptions.c > > [0]PETSC ERROR: [0] KSPSetFromOptions line 329 /autofs/nccs-svm1_home1/adams/petsc/src/ksp/ksp/interface/itcl.c > > [0]PETSC ERROR: [0] SNESSetFromOptions line 869 /autofs/nccs-svm1_home1/adams/petsc/src/snes/interface/snes.c > > [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- > > [0]PETSC ERROR: Floating point exception > > [0]PETSC ERROR: trapped floating point error > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1685-gd3eb2e1 GIT Date: 2019-08-13 06:33:29 -0400 > > [0]PETSC ERROR: ./ex56_single on a arch-summit-dbg-single-pgi-cuda named h36n11 by adams Wed Aug 14 22:21:56 2019 > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpiCC --with-fc=mpif90 COPTFLAGS="-g -Mfcon" CXXOPTFLAGS="-g -Mfcon" FOPTFLAGS="-g -Mfcon" --with-precision=single --with-ssl=0 --with-batch=0 --with-mpiexec="jsrun -g 1" --with-cuda=1 --with-cudac=nvcc CUDAFLAGS="-ccbin pgc++" --download-metis --download-parmetis --download-fblaslapack --with-x=0 --with-64-bit-indices=0 --with-debugging=1 PETSC_ARCH=arch-summit-dbg-single-pgi-cuda > > [0]PETSC ERROR: #1 User provided function() line 0 in Unknown file > > -------------------------------------------------------------------------- > > > > On Wed, Aug 14, 2019 at 9:51 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > > > Oh, doesn't even have to be that large. We just need to be able to look at the flop rates (as a surrogate for run times) and compare with the previous runs. So long as the size per process is pretty much the same that is good enough. > > > > Barry > > > > > > > On Aug 14, 2019, at 8:45 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > I can run single, I just can't scale up. But I can use like 1500 processors. > > > > > > On Wed, Aug 14, 2019 at 9:31 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > > > > > Oh, are all your integers 8 bytes? Even on one node? > > > > > > Once Karl's new middleware is in place we should see about reducing to 4 bytes on the GPU. > > > > > > Barry > > > > > > > > > > On Aug 14, 2019, at 7:44 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > OK, I'll run single. It a bit perverse to run with 4 byte floats and 8 byte integers ... I could use 32 bit ints and just not scale out. > > > > > > > > On Wed, Aug 14, 2019 at 6:48 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > > > > > > > Mark, > > > > > > > > Oh, I don't even care if it converges, just put in a fixed number of iterations. The idea is to just get a baseline of the possible improvement. > > > > > > > > ECP is literally dropping millions into research on "multi precision" computations on GPUs, we need to have some actual numbers for the best potential benefit to determine how much we invest in further investigating it, or not. > > > > > > > > I am not expressing any opinions on the approach, we are just in the fact gathering stage. > > > > > > > > > > > > Barry > > > > > > > > > > > > > On Aug 14, 2019, at 2:27 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > > > > > > > > > On Wed, Aug 14, 2019 at 2:35 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > > > > > > > > > Mark, > > > > > > > > > > Would you be able to make one run using single precision? Just single everywhere since that is all we support currently? > > > > > > > > > > > > > > > Experience in engineering at least is single does not work for FE elasticity. I have tried it many years ago and have heard this from others. This problem is pretty simple other than using Q2. I suppose I could try it, but just be aware the FE people might say that single sucks. > > > > > > > > > > The results will give us motivation (or anti-motivation) to have support for running KSP (or PC (or Mat) in single precision while the simulation is double. > > > > > > > > > > Thanks. > > > > > > > > > > Barry > > > > > > > > > > For example if the GPU speed on KSP is a factor of 3 over the double on GPUs this is serious motivation. > > > > > > > > > > > > > > > > On Aug 14, 2019, at 12:45 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > FYI, Here is some scaling data of GAMG on SUMMIT. Getting about 4x GPU speedup with 98K dof/proc (3D Q2 elasticity). > > > > > > > > > > > > This is weak scaling of a solve. There is growth in iteration count folded in here. I should put rtol in the title and/or run a fixed number of iterations and make it clear in the title. > > > > > > > > > > > > Comments welcome. > > > > > > <out_cpu_012288><out_cpu_001536><out_cuda_012288><out_cpu_000024><out_cpu_000192><out_cuda_001536><out_cuda_000192><out_cuda_000024><weak_scaling_cpu.png><weak_scaling_cuda.png> > > > > > > > > > > > > > > > > <weak_scaling_gpu_compact_spread.png><weak_scaling_cpu.png><spread.tar><compact.tar>

1 0

Re: [petsc-dev] PETSC_HAVE_METIS
by Tabrez Ali 30 Aug '19

30 Aug '19

Thanks Barry. It works now. Regards, Tabrez On Wed, Aug 28, 2019 at 1:39 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > What do you mean by latest? I sent email where I said it was turned on > in the branch barry/2019-08-27/revert-have-c99-have-nonpetsc-used-package > and will be in master once it passed testing so it is not yet in master but > is in that branch and could be used there until it gets to master. We are > still switching to a new CI so updates to master are slower than they will > be in a few days. > > Barry > > > > On Aug 25, 2019, at 7:53 AM, Tabrez Ali via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > > > > Hello > > > > I recently configured/built PETSc (latest) with the '--download-metis' > option. However, PETSC_HAVE_METIS doesn't appear to be defined. > > > > Any ideas, as to what could be wrong? The configure/make log files are > attached. > > > > Regards, > > > > Tabrez > > <configure.log.gz><make.log.gz> > >

1 0

Re: [petsc-dev] [petsc-maint] running CUDA on SUMMIT
by Mark Adams 30 Aug '19

30 Aug '19

Here is some more weak scaling data with a fixed number of iterations (I have given a test with the numerical problems to ORNL and they said they would give it to Nvidia). I implemented an option to "spread" the reduced coarse grids across the whole machine as opposed to a "compact" layout where active processes are laid out in simple lexicographical order. This spread approach looks a little better. Mark On Wed, Aug 14, 2019 at 10:46 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > Ahh, PGI compiler, that explains it :-) > > Ok, thanks. Don't worry about the runs right now. We'll figure out the > fix. The code is just > > *a = (PetscReal)strtod(name,endptr); > > could be a compiler bug. > > > > > > On Aug 14, 2019, at 9:23 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > I am getting this error with single: > > > > 22:21 /gpfs/alpine/geo127/scratch/adams$ jsrun -n 1 -a 1 -c 1 -g 1 > ./ex56_single -cells 2,2,2 -ex56_dm_vec_type cuda -ex56_dm_mat_type > aijcusparse -fp_trap > > [0] 81 global equations, 27 vertices > > [0]PETSC ERROR: *** unknown floating point error occurred *** > > [0]PETSC ERROR: The specific exception can be determined by running in a > debugger. When the > > [0]PETSC ERROR: debugger traps the signal, the exception can be found > with fetestexcept(0x3e000000) > > [0]PETSC ERROR: where the result is a bitwise OR of the following flags: > > [0]PETSC ERROR: FE_INVALID=0x20000000 FE_DIVBYZERO=0x4000000 > FE_OVERFLOW=0x10000000 FE_UNDERFLOW=0x8000000 FE_INEXACT=0x2000000 > > [0]PETSC ERROR: Try option -start_in_debugger > > [0]PETSC ERROR: likely location of problem given in stack below > > [0]PETSC ERROR: --------------------- Stack Frames > ------------------------------------ > > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not > available, > > [0]PETSC ERROR: INSTEAD the line number of the start of the > function > > [0]PETSC ERROR: is given. > > [0]PETSC ERROR: [0] PetscDefaultFPTrap line 355 > /autofs/nccs-svm1_home1/adams/petsc/src/sys/error/fp.c > > [0]PETSC ERROR: [0] PetscStrtod line 1964 > /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/options.c > > [0]PETSC ERROR: [0] PetscOptionsStringToReal line 2021 > /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/options.c > > [0]PETSC ERROR: [0] PetscOptionsGetReal line 2321 > /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/options.c > > [0]PETSC ERROR: [0] PetscOptionsReal_Private line 1015 > /autofs/nccs-svm1_home1/adams/petsc/src/sys/objects/aoptions.c > > [0]PETSC ERROR: [0] KSPSetFromOptions line 329 > /autofs/nccs-svm1_home1/adams/petsc/src/ksp/ksp/interface/itcl.c > > [0]PETSC ERROR: [0] SNESSetFromOptions line 869 > /autofs/nccs-svm1_home1/adams/petsc/src/snes/interface/snes.c > > [0]PETSC ERROR: --------------------- Error Message > -------------------------------------------------------------- > > [0]PETSC ERROR: Floating point exception > > [0]PETSC ERROR: trapped floating point error > > [0]PETSC ERROR: See https://www.mcs.anl.gov/petsc/documentation/faq.html > for trouble shooting. > > [0]PETSC ERROR: Petsc Development GIT revision: v3.11.3-1685-gd3eb2e1 > GIT Date: 2019-08-13 06:33:29 -0400 > > [0]PETSC ERROR: ./ex56_single on a arch-summit-dbg-single-pgi-cuda named > h36n11 by adams Wed Aug 14 22:21:56 2019 > > [0]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpiCC > --with-fc=mpif90 COPTFLAGS="-g -Mfcon" CXXOPTFLAGS="-g -Mfcon" > FOPTFLAGS="-g -Mfcon" --with-precision=single --with-ssl=0 --with-batch=0 > --with-mpiexec="jsrun -g 1" --with-cuda=1 --with-cudac=nvcc > CUDAFLAGS="-ccbin pgc++" --download-metis --download-parmetis > --download-fblaslapack --with-x=0 --with-64-bit-indices=0 > --with-debugging=1 PETSC_ARCH=arch-summit-dbg-single-pgi-cuda > > [0]PETSC ERROR: #1 User provided function() line 0 in Unknown file > > > -------------------------------------------------------------------------- > > > > On Wed, Aug 14, 2019 at 9:51 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> > wrote: > > > > Oh, doesn't even have to be that large. We just need to be able to > look at the flop rates (as a surrogate for run times) and compare with the > previous runs. So long as the size per process is pretty much the same that > is good enough. > > > > Barry > > > > > > > On Aug 14, 2019, at 8:45 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > I can run single, I just can't scale up. But I can use like 1500 > processors. > > > > > > On Wed, Aug 14, 2019 at 9:31 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> > wrote: > > > > > > Oh, are all your integers 8 bytes? Even on one node? > > > > > > Once Karl's new middleware is in place we should see about reducing > to 4 bytes on the GPU. > > > > > > Barry > > > > > > > > > > On Aug 14, 2019, at 7:44 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > OK, I'll run single. It a bit perverse to run with 4 byte floats and > 8 byte integers ... I could use 32 bit ints and just not scale out. > > > > > > > > On Wed, Aug 14, 2019 at 6:48 PM Smith, Barry F. <bsmith(a)mcs.anl.gov> > wrote: > > > > > > > > Mark, > > > > > > > > Oh, I don't even care if it converges, just put in a fixed number > of iterations. The idea is to just get a baseline of the possible > improvement. > > > > > > > > ECP is literally dropping millions into research on "multi > precision" computations on GPUs, we need to have some actual numbers for > the best potential benefit to determine how much we invest in further > investigating it, or not. > > > > > > > > I am not expressing any opinions on the approach, we are just in > the fact gathering stage. > > > > > > > > > > > > Barry > > > > > > > > > > > > > On Aug 14, 2019, at 2:27 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > > > > > > > > > > > > > > > > > > > On Wed, Aug 14, 2019 at 2:35 PM Smith, Barry F. < > bsmith(a)mcs.anl.gov> wrote: > > > > > > > > > > Mark, > > > > > > > > > > Would you be able to make one run using single precision? Just > single everywhere since that is all we support currently? > > > > > > > > > > > > > > > Experience in engineering at least is single does not work for FE > elasticity. I have tried it many years ago and have heard this from others. > This problem is pretty simple other than using Q2. I suppose I could try > it, but just be aware the FE people might say that single sucks. > > > > > > > > > > The results will give us motivation (or anti-motivation) to > have support for running KSP (or PC (or Mat) in single precision while the > simulation is double. > > > > > > > > > > Thanks. > > > > > > > > > > Barry > > > > > > > > > > For example if the GPU speed on KSP is a factor of 3 over the > double on GPUs this is serious motivation. > > > > > > > > > > > > > > > > On Aug 14, 2019, at 12:45 PM, Mark Adams <mfadams(a)lbl.gov> > wrote: > > > > > > > > > > > > FYI, Here is some scaling data of GAMG on SUMMIT. Getting about > 4x GPU speedup with 98K dof/proc (3D Q2 elasticity). > > > > > > > > > > > > This is weak scaling of a solve. There is growth in iteration > count folded in here. I should put rtol in the title and/or run a fixed > number of iterations and make it clear in the title. > > > > > > > > > > > > Comments welcome. > > > > > > > <out_cpu_012288><out_cpu_001536><out_cuda_012288><out_cpu_000024><out_cpu_000192><out_cuda_001536><out_cuda_000192><out_cuda_000024><weak_scaling_cpu.png><weak_scaling_cuda.png> > > > > > > > > > > > > > > > >

1 0

Re: [petsc-dev] PETSC_HAVE_METIS
by Alexander Lindsay 28 Aug '19

28 Aug '19

Thanks Barry! On Wed, Aug 28, 2019 at 11:39 AM Smith, Barry F. via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > > What do you mean by latest? I sent email where I said it was turned on > in the branch barry/2019-08-27/revert-have-c99-have-nonpetsc-used-package > and will be in master once it passed testing so it is not yet in master but > is in that branch and could be used there until it gets to master. We are > still switching to a new CI so updates to master are slower than they will > be in a few days. > > Barry > > > > On Aug 25, 2019, at 7:53 AM, Tabrez Ali via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > > > > Hello > > > > I recently configured/built PETSc (latest) with the '--download-metis' > option. However, PETSC_HAVE_METIS doesn't appear to be defined. > > > > Any ideas, as to what could be wrong? The configure/make log files are > attached. > > > > Regards, > > > > Tabrez > > <configure.log.gz><make.log.gz> > >

1 0

Re: [petsc-dev] PETSC_HAVE_METIS
by Smith, Barry F. 28 Aug '19

28 Aug '19

What do you mean by latest? I sent email where I said it was turned on in the branch barry/2019-08-27/revert-have-c99-have-nonpetsc-used-package and will be in master once it passed testing so it is not yet in master but is in that branch and could be used there until it gets to master. We are still switching to a new CI so updates to master are slower than they will be in a few days. Barry > On Aug 25, 2019, at 7:53 AM, Tabrez Ali via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > Hello > > I recently configured/built PETSc (latest) with the '--download-metis' option. However, PETSC_HAVE_METIS doesn't appear to be defined. > > Any ideas, as to what could be wrong? The configure/make log files are attached. > > Regards, > > Tabrez > <configure.log.gz><make.log.gz>

1 0

Re: [petsc-dev] PETSC_HAVE_METIS
by Alexander Lindsay 28 Aug '19

28 Aug '19

We use PETSC_HAVE_METIS in libmesh configuration so we would like for it to come back. On Mon, Aug 26, 2019 at 8:42 AM Matthew Knepley via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > On Mon, Aug 26, 2019 at 10:08 AM Tabrez Ali via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > >> Hello >> >> I recently configured/built PETSc (latest) with the '--download-metis' >> option. However, PETSC_HAVE_METIS doesn't appear to be defined. >> >> Any ideas, as to what could be wrong? The configure/make log files are >> attached. >> > > It was turned off > > > https://gitlab.com/petsc/petsc/commit/2475b7ca256cea2a4b7cbf2d8babcda14e5fa… > > I think we should turn it back on. > > Thanks, > > Matt > > >> Regards, >> >> Tabrez >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] Scotch 6.0.8
by Smith, Barry F. 28 Aug '19

28 Aug '19

> On Aug 28, 2019, at 3:30 AM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > > > > On Wed, 28 Aug 2019 at 06:26, Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > > > On Aug 27, 2019, at 3:22 AM, Lisandro Dalcin via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > > > There is a new scotch release. We should update configure to this version. I'm a bit confused about the current status PTscotch.py in BuildSystem, it seems to be using the scotch/master from the official repo. > > It is using > > self.version = '6.0.7' > self.versionname = 'SCOTCH_VERSION.SCOTCH_RELEASE.SCOTCH_PATCHLEVEL' > self.gitcommit = 'v'+self.version > self.download = ['git:https://gitlab.inria.fr/scotch/scotch', > 'https://gitlab.inria.fr/scotch/scotch/-/archive/'+self.gitcommit+'/scotch-'+self.gitcommit+'.tar.gz', > 'http://ftp.mcs.anl.gov/pub/petsc/externalpackages/scotch-'+self.gitcommit+'.tar.gz'] > > which means it is using the tag v6.0.7 from the repo which in a sane world corresponds to release 6.0.7 exactly. > > > Well, yes, but I'm still wondering why we default to git rather than an official release tarball. Why not? It makes it trivial to switch to other branches/packages for testing. Is there some inherent reason that using "official release tarball" for any package instead of the the appropriate git branch is better? We don't have an official policy on which to use (maybe we should) but my feeling is to use the git repository unless for a particular package there is a reason not too. Personally I'd like to see the idea of "official release tarballs" get lost in the mists of time and everyone just make tags in their repository for releases. There is the question of built documentation in the tarball but I won't even put docs in the releases, just put them on the web. It is 2019. Barry > > To switch to the new release just change the 7 to an 8; test and make a merge request. We are not official accepting merge requests yet but despite this people are shoving them in like crazy, you might as well join the crowd. > > > No pressure, I'll do it once merge request acceptance is back to normal. > > -- > Lisandro Dalcin > ============ > Research Scientist > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/

1 0

Re: [petsc-dev] petsc-dev post from [email protected] requires approval
by Smith, Barry F. 28 Aug '19

28 Aug '19

None of this matters because PetscPartitioner is due to be gutted and its functionality provided by calls to MatPartitioning. All improvements to managing calls to the underly partitioners should be made to MatPartioning. Barry > On Aug 28, 2019, at 7:53 AM, Jed Brown <jed(a)jedbrown.org> wrote: > > Lisandro Dalcin via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > >>> If this line was protected with #if defined(PETSC_HAVE_METIS) and >>> PETSc was not installed with ParMetis, but only Metis would the code run >>> correctly? Or is it somehow that even though you are only using metis here >>> you still need parmetis? For what reason? >>> >>> >> With some code changes, it would be possible to support PETSc with METIS >> and no ParMETIS, but that would only cover a special (though frequent) case >> (partitioning an initially sequential mesh), and the PetscPartitioner type >> name is "parmetis", so that would be confusing. I think it is not really >> worth it to make these changes. > > I recall Jack observing that sequential METIS often outperformed (in > time) ParMETIS for nested dissection ordering in Elemental's sparse > solvers. This isn't trivial since we'd need to implement gathering to > one process and scattering the results back, but it's something to > consider if we add support for calling METIS directly.

1 0

Re: [petsc-dev] petsc-dev post from [email protected] requires approval
by Jed Brown 28 Aug '19

28 Aug '19

Lisandro Dalcin via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: >> If this line was protected with #if defined(PETSC_HAVE_METIS) and >> PETSc was not installed with ParMetis, but only Metis would the code run >> correctly? Or is it somehow that even though you are only using metis here >> you still need parmetis? For what reason? >> >> > With some code changes, it would be possible to support PETSc with METIS > and no ParMETIS, but that would only cover a special (though frequent) case > (partitioning an initially sequential mesh), and the PetscPartitioner type > name is "parmetis", so that would be confusing. I think it is not really > worth it to make these changes. I recall Jack observing that sequential METIS often outperformed (in time) ParMETIS for nested dissection ordering in Elemental's sparse solvers. This isn't trivial since we'd need to implement gathering to one process and scattering the results back, but it's something to consider if we add support for calling METIS directly.

1 0

Re: [petsc-dev] petsc-dev post from [email protected] requires approval
by Lisandro Dalcin 28 Aug '19

28 Aug '19

On Wed, 28 Aug 2019 at 06:45, Smith, Barry F. <bsmith(a)mcs.anl.gov> wrote: > > > > On Aug 27, 2019, at 3:27 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > > > > On Tue, Aug 27, 2019 at 4:11 AM Lisandro Dalcin <dalcinl(a)gmail.com> > wrote: > > On Mon, 26 Aug 2019 at 21:26, Smith, Barry F. <bsmith(a)mcs.anl.gov> > wrote: > > > > > > > On Aug 26, 2019, at 10:11 AM, Lisandro Dalcin <dalcinl(a)gmail.com> > wrote: > > > > > > > > > > > > On Sun, 25 Aug 2019 at 18:37, Smith, Barry F. via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > > > > > > Metis is installed. > > > > > > config/PETSc/Configure.py: > self.addDefine('HAVE_'+i.PACKAGE.replace('-','_'), 1) # ONLY list package > if it is used directly by PETSc (and not only by another package) > > > > > > Since metis is not used by PETSc we don't set the PETSC_HAVE_METIS > flag because PETSc doesn't need it. > > > > > > > > > This is not true, in DMPlex we have an explicit call to > METIS_PartGraphKway(). Of course, this code is protected with just > HAVE_PARMETIS, because if you have parmetis, then you have metis. > > > > Hmm, I think is should be protected with HAVE_METIS. There is nothing > that says PETSc can only be built with metis plus parmetis; it could be > built with just metis. > > > > > > Sorry, maybe I was not clear enough. > > > > The DMPlex code I'm talkinga about uses both ParMETIS and METIS, > depending on the input graph being sequential or parallel. > > > https://gitlab.com/petsc/petsc/blob/master/src/dm/impls/plex/plexpartition.… > > So this code requires ParMETIS, and if you have it, then you have METIS, > the a HAVE_METIS check seems a bit redundant to me. > > Do you still think we should change it? > > > > Or maybe we should use `#if defined(PETSC_HAVE_METIS) && > defined(PETSC_HAVE_PARMETIS)`? > > > > Anything to add, Matthew? > > > > 1) I think we turn on PETSC_HAVE_METIS > > > > 2) I think we leave the guard as PARMETIS since we need it for the > parallel case and it implies Metis > > If this line was protected with #if defined(PETSC_HAVE_METIS) and > PETSc was not installed with ParMetis, but only Metis would the code run > correctly? Or is it somehow that even though you are only using metis here > you still need parmetis? For what reason? > > With some code changes, it would be possible to support PETSc with METIS and no ParMETIS, but that would only cover a special (though frequent) case (partitioning an initially sequential mesh), and the PetscPartitioner type name is "parmetis", so that would be confusing. I think it is not really worth it to make these changes. -- Lisandro Dalcin ============ Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/

1 0