April 2018 - petsc-dev - mailman3.cels.anl.gov

*unwind* next branch
by Satish Balay 08 Apr '18

08 Apr '18

All (next users), ref: https://lists.mcs.anl.gov/pipermail/petsc-dev/2017-September/021361.html I've recreated 'next' branch - and have a backup for current 'next' at 'next-apr-2018'. So if you have/use 'next' branch in your clone - please do the following (in all your git clones): git checkout master git fetch git branch -D next git checkout next Please *do not push* from a stale 'next' branch in your clone. If this is attempted, the merge step should get a *conflict* on include/petscversion.h. i.e this merge conflict indicates you have to recreate next - and *not* resolve/push. $ git fetch $ git merge origin/next Auto-merging include/petscversion.h CONFLICT (content): Merge conflict in include/petscversion.h Automatic merge failed; fix conflicts and then commit the result. Satish ------ Ref:currently merged branches in old next $ comm -12 <(git branch -r --merged origin/next-apr-2018 | sort) <(git branch -r --no-merged origin/master | sort) origin/knepley/fix-dm-gmg origin/knepley/fix-ex62-tests origin/knepley/fix-fe-bd-integral origin/knepley/fix-fe-vector-spaces origin/knepley/fix-snes-ex69 origin/tisaac/feature-dmfield

1 0

Re: [petsc-dev] nested timers transfers and reductions
by Smith, Barry F. 07 Apr '18

07 Apr '18

> On Apr 6, 2018, at 5:22 AM, Klaij, Christiaan <C.Klaij(a)marin.nl> wrote: > > Hi Barry, > > In our version of the nested timers, we had the functions > PetscAdminSend, PetscAdminRecv and PetscAdminReduce. So I was > expecting to find them at the end of the file > petsc-3.8.4/src/sys/logging/xmllogevent.c but apparently > you (re)moved them? I do not remember why I would have removed these functions. If you can send a patch that would return them I would gladly try to get them working with PETSc again. Barry > > We basically used these functions to fill the transfer(GiB/s) and > reductions/s columns of the log just like we use PetscLogFlops to > fill the compute(Mflops) column for our registered events. > > How should I do this with the petsc version of the nested timers? > > Chris > > > dr. ir. Christiaan Klaij | Senior Researcher | Research & Development > MARIN | T +31 317 49 33 44 | mailto:[email protected] | http://www.marin.nl > > MARIN news: http://www.marin.nl/web/News/News-items/Application-of-potential-flow-metho… >

1 0

Re: [petsc-dev] More CMake hatred
by Jed Brown 06 Apr '18

06 Apr '18

Jeff Hammond <jeff.science(a)gmail.com> writes: >> /util/common/openmpi/3.0.0/gcc-4.8.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/dell/srvadmin/bin:/user/knepley/bin >> > >> > I cannot see why it would say this. >> >> It's complaining that the CC is two words instead of a single path. You >> can make a link >> >> ln -s `which ccache` ~/bin/mpicc >> >> and then CC=$HOME/bin/mpicc will work. Yes, it's irritating. > > > While we are proposing features that place no value on users’ time, why not > just ask them to implement their own MPI wrapper scripts with ccache on the > inside? Make no mistake: I am not making excuses for CMake, merely providing Matt with a workable solution.

1 0

Re: [petsc-dev] More CMake hatred
by Jeff Hammond 06 Apr '18

06 Apr '18

On Thu, Apr 5, 2018 at 8:44 PM Jed Brown <jed(a)jedbrown.org> wrote: > Matthew Knepley <knepley(a)gmail.com> writes: > > > Error configuring EIGEN with cmake Could not execute "cd > > > /projects/academic/knepley/PETSc3/petsc/arch-master-debug/externalpackages/eigen-eigen-5a0156e40feb/build > > && /projects/academic/knepley/PETSc3/petsc/arch-master-debug/bin/cmake .. > > > -DCMAKE_INSTALL_PREFIX=/projects/academic/knepley/PETSc3/petsc/arch-master-debug > > -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="/user/knepley/bin/ccache > > mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib > > -DCMAKE_C_FLAGS:STRING="-fPIC -fstack-protector -g3" > > -DCMAKE_C_FLAGS_DEBUG:STRING="-fPIC -fstack-protector -g3" > > -DCMAKE_C_FLAGS_RELEASE:STRING="-fPIC -fstack-protector -g3" > > -DCMAKE_CXX_COMPILER="/user/knepley/bin/ccache mpicxx" > > -DCMAKE_CXX_FLAGS:STRING="-fstack-protector -g -fPIC" > > -DCMAKE_CXX_FLAGS_DEBUG:STRING="-fstack-protector -g -fPIC" > > -DCMAKE_CXX_FLAGS_RELEASE:STRING="-fstack-protector -g -fPIC" > > -DCMAKE_Fortran_COMPILER="/user/knepley/bin/ccache mpif90" > > -DCMAKE_Fortran_FLAGS:STRING="-fPIC -ffree-line-length-0 -g" > > -DCMAKE_Fortran_FLAGS_DEBUG:STRING="-fPIC -ffree-line-length-0 -g" > > -DCMAKE_Fortran_FLAGS_RELEASE:STRING="-fPIC -ffree-line-length-0 -g" > > -DBUILD_SHARED_LIBS=on -DENABLE_OPENMP=OFF": > > > > -- The C compiler identification is unknown > > > > -- The CXX compiler identification is unknown > > > > -- Configuring incomplete, errors occurred! > > > > See also > > > "/projects/academic/knepley/PETSc3/petsc/arch-master-debug/externalpackages/eigen-eigen-5a0156e40feb/build/CMakeFiles/CMakeOutput.log". > > > > See also > > > "/projects/academic/knepley/PETSc3/petsc/arch-master-debug/externalpackages/eigen-eigen-5a0156e40feb/build/CMakeFiles/CMakeError.log".CMake > > Error at CMakeLists.txt:1 (project): > > > > The CMAKE_C_COMPILER: > > > > > > /user/knepley/bin/ccache mpicc > > > > > > is not a full path to an existing compiler tool. > > > > > > Tell CMake where to find the compiler by setting either the environment > > > > variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full > path > > to > > > > the compiler, or to the compiler name if it is in the PATH. > > > > > > > > CMake Error at CMakeLists.txt:1 (project): > > > > The CMAKE_CXX_COMPILER: > > > > > > /user/knepley/bin/ccache mpicxx > > > > > > is not a full path to an existing compiler tool. > > > > > > Tell CMake where to find the compiler by setting either the environment > > > > variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full > > path > > > > to the compiler, or to the compiler name if it is in the PATH. > > > > > > > > > ******************************************************************************* > > > > > > [knepley@rush:/projects/academic/knepley/PETSc3/petsc]$ which mpicc > > > > /util/common/openmpi/3.0.0/gcc-4.8.5/bin/mpicc > > > > [knepley@rush:/projects/academic/knepley/PETSc3/petsc]$ echo $PATH > > > > > /util/common/openmpi/3.0.0/gcc-4.8.5/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/dell/srvadmin/bin:/user/knepley/bin > > > > I cannot see why it would say this. > > It's complaining that the CC is two words instead of a single path. You > can make a link > > ln -s `which ccache` ~/bin/mpicc > > and then CC=$HOME/bin/mpicc will work. Yes, it's irritating. While we are proposing features that place no value on users’ time, why not just ask them to implement their own MPI wrapper scripts with ccache on the inside? Jeff -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] upcoming release and testing [v2]
by Satish Balay 06 Apr '18

06 Apr '18

The tests are run in parallel - but most other tests are sequential. When I tried to reproduce I ran sequentially [as in 'make -f gmakefile' without -j] - and some of them hung.. Satish On Fri, 6 Apr 2018, Karl Rupp wrote: > Hi again, > > the reason for the higher number of timeouts is likely to be due to the higher > number of GPU tests. GPU builds that formerly only used CUSP now run against > the CUDA backend, which has a higher number of tests. Also, the CUDA backend > uses CUBLAS and CUSPARSE, whereas CUSP used its own kernels. As far as I know, > CUBLAS and CUSPARSE initialization is fairly slow on the M2090. > > Best regards, > Karli > > > On 04/06/2018 09:13 PM, Karl Rupp wrote: > > Hi, > > > >> The CUDA tests are hanging/timing-out more often now. For eg: > >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/06/examples_ne… > >> > >> > >> And I did see some build where they didn't get killed due to timeout. For > >> eg: > >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/05/examples_ne… > >> > >> > >> This is on M2090.ï¿œ I can see them getting stuck on es.mcs [when I run > >> manually - and check with nvidia-smi] > >> > >> When i run these tests manually on GTX1050 (frog.mcs) - they zip through.. > >> Any idea why they get stuck on M2090? [more frequently than random hangs..] > > > > no, I don't know why this is the case. All my local tests finish quickly, > > too. I noticed last summer that there is higher startup overhead on the > > M2090 than on more recent GPUs, but that was in the seconds regime, not in > > minutes. > > > > Are the tests run in parallel? If so, then maybe the parallel initialization > > of GPUs is slowing things down. > > > > Best regards, > > Karli >

1 0

Re: [petsc-dev] upcoming release and testing [v2]
by Karl Rupp 06 Apr '18

06 Apr '18

Hi again, the reason for the higher number of timeouts is likely to be due to the higher number of GPU tests. GPU builds that formerly only used CUSP now run against the CUDA backend, which has a higher number of tests. Also, the CUDA backend uses CUBLAS and CUSPARSE, whereas CUSP used its own kernels. As far as I know, CUBLAS and CUSPARSE initialization is fairly slow on the M2090. Best regards, Karli On 04/06/2018 09:13 PM, Karl Rupp wrote: > Hi, > >> The CUDA tests are hanging/timing-out more often now. For eg: >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/06/examples_ne… >> >> >> And I did see some build where they didn't get killed due to timeout. >> For eg: >> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/05/examples_ne… >> >> >> This is on M2090. I can see them getting stuck on es.mcs [when I run >> manually - and check with nvidia-smi] >> >> When i run these tests manually on GTX1050 (frog.mcs) - they zip >> through.. >> Any idea why they get stuck on M2090? [more frequently than random >> hangs..] > > no, I don't know why this is the case. All my local tests finish > quickly, too. I noticed last summer that there is higher startup > overhead on the M2090 than on more recent GPUs, but that was in the > seconds regime, not in minutes. > > Are the tests run in parallel? If so, then maybe the parallel > initialization of GPUs is slowing things down. > > Best regards, > Karli

1 0

Re: [petsc-dev] upcoming release and testing [v2]
by Karl Rupp 06 Apr '18

06 Apr '18

Hi, > The CUDA tests are hanging/timing-out more often now. For eg: > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/06/examples_ne… > > And I did see some build where they didn't get killed due to timeout. For eg: > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/05/examples_ne… > > This is on M2090. I can see them getting stuck on es.mcs [when I run manually - and check with nvidia-smi] > > When i run these tests manually on GTX1050 (frog.mcs) - they zip through.. > Any idea why they get stuck on M2090? [more frequently than random hangs..] no, I don't know why this is the case. All my local tests finish quickly, too. I noticed last summer that there is higher startup overhead on the M2090 than on more recent GPUs, but that was in the seconds regime, not in minutes. Are the tests run in parallel? If so, then maybe the parallel initialization of GPUs is slowing things down. Best regards, Karli

1 0

Re: [petsc-dev] upcoming release and testing [v2]
by Richard Tran Mills 06 Apr '18

06 Apr '18

Satish: Oops, I noticed that the order of the developer photos got messed up (Hong and Stefano's photos switched) when I updated the website. Have pushed a fix to master. --Richard On Thu, Apr 5, 2018 at 7:07 PM, Balay, Satish <balay(a)mcs.anl.gov> wrote: > On Fri, 6 Apr 2018, Richard Tran Mills wrote: > > > Thanks, Satish! I notice also that, in the stanza beginning "When citing > > PETSc, please do not refer to this document", Dave May is still missing. > > pushed to master. > > > Also, that stanza still refers to version 3.7 and 2016. > > updated in balay/release-3.9 > > Thanks! > Satish >

1 0

Re: [petsc-dev] upcoming release and testing [v2]
by Satish Balay 06 Apr '18

06 Apr '18

I have the following branches for today's next-tmp test. origin/knepley/fix-dm-gmg origin/knepley/fix-ex62-tests origin/knepley/fix-fe-bd-integral origin/knepley/fix-fe-vector-spaces origin/knepley/fix-snes-ex69 origin/tisaac/feature-dmfield origin/hzhang/iscoloring-testvalid origin/hzhang/superlu_dist-matinertia-hermitian origin/scott/test-harness-timing2 Karl, The CUDA tests are hanging/timing-out more often now. For eg: http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/06/examples_ne… And I did see some build where they didn't get killed due to timeout. For eg: http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/05/examples_ne… This is on M2090. I can see them getting stuck on es.mcs [when I run manually - and check with nvidia-smi] When i run these tests manually on GTX1050 (frog.mcs) - they zip through.. Any idea why they get stuck on M2090? [more frequently than random hangs..] thanks, Satish

1 0

nested timers transfers and reductions
by Klaij, Christiaan 06 Apr '18

06 Apr '18

Hi Barry, In our version of the nested timers, we had the functions PetscAdminSend, PetscAdminRecv and PetscAdminReduce. So I was expecting to find them at the end of the file petsc-3.8.4/src/sys/logging/xmllogevent.c but apparently you (re)moved them? We basically used these functions to fill the transfer(GiB/s) and reductions/s columns of the log just like we use PetscLogFlops to fill the compute(Mflops) column for our registered events. How should I do this with the petsc version of the nested timers? Chris dr. ir. Christiaan Klaij | Senior Researcher | Research & Development MARIN | T +31 317 49 33 44 | mailto:[email protected] | http://www.marin.nl MARIN news: http://www.marin.nl/web/News/News-items/Application-of-potential-flow-metho…

1 0