July 2020 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] [petsc-users] PETSc Make error
by Satish Balay 31 Jul '20

31 Jul '20

Yes - [mentioned earlier in this thread] this check is in petsc-3.13.3 Satish On Fri, 31 Jul 2020, Barry Smith wrote: > > If PETSc requires distutils.sysconfig then ./Configure should check for it and print a very useful error message if it does not find it. > > We can't have python crashes off at build time with confusing error messages that users cannot decipher. > > Barry > > > > On Jul 31, 2020, at 11:00 AM, Satish Balay via petsc-users <petsc-users(a)mcs.anl.gov> wrote: > > > > You still get this error message - so its likely not installed. To confirm you [and the admin] can run: > > > > python3 -c "import distutils.sysconfig" > > > > [and see if you get an error or not] > > > > You can also try python2 as I suggested in my earlier e-mail [if distutils is installed for python2] > > > > python2 -c "import distutils.sysconfig" > > > > python2 ./configure ... > > > > Satish > > > > > > On Fri, 31 Jul 2020, Eda Oktay wrote: > > > >> Dear Matt, > >> > >> Administrator installed python3-dib-utils but still I am getting the > >> same error message. I attached configure.log and make.log again. > >> > >> Thanks! > >> > >> Eda > >> > >> Matthew Knepley <knepley(a)gmail.com>, 31 Tem 2020 Cum, 13:20 tarihinde > >> şunu yazdı: > >>> > >>> On Fri, Jul 31, 2020 at 4:15 AM Eda Oktay <eda.oktay(a)metu.edu.tr> wrote: > >>>> > >>>> Dear Satish, > >>>> > >>>> I configured petsc-3.13.2. Is this still the same problem? > >>> > >>> > >>> Yes, you need the distutils package installed. > >>> > >>> Thanks, > >>> > >>> Matt > >>> > >>>> > >>>> Thanks! > >>>> > >>>> Eda > >>>> > >>>> On Fri, Jul 31, 2020, 10:50 AM Satish Balay <balay(a)mcs.anl.gov> wrote: > >>>>> > >>>>> > >>>>>>>>>>>>> > >>>>> File "./config/gmakegen.py", line 4, in <module> > >>>>> from distutils.sysconfig import parse_makefile > >>>>> ModuleNotFoundError: No module named 'distutils.sysconfig' > >>>>> <<<<< > >>>>> > >>>>> For one - if you used petsc-3.13.3 - it would have also tried python2 - or given a reasonable error message. > >>>>> > >>>>> If you have python2 - you can try: > >>>>> > >>>>> python2 ./configure ... > >>>>> > >>>>> Or you can ask sysadmin on this machine to install python3-distutils or equivalent package. > >>>>> > >>>>> Satish > >>>>> > >>>>> On Fri, 31 Jul 2020, Eda Oktay wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> I configured petsc-3.13.2 with following options: > >>>>>> > >>>>>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran > >>>>>> --download-mpich --download-openblas --download-metis > >>>>>> --download-parmetis --download-chaco --download-slepc --with-X=1 > >>>>>> > >>>>>> However, I got a make error. > >>>>>> > >>>>>> I attached make.log and configure.log, how can I fix that problem? > >>>>>> > >>>>>> Thanks! > >>>>>> > >>>>>> Eda > >>>>>> > >>>>> > >>> > >>> > >>> -- > >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > >>> -- Norbert Wiener > >>> > >>> https://www.cse.buffalo.edu/~knepley/ > >> >

1 0

Re: [petsc-dev] [petsc-users] PETSc Make error
by Barry Smith 31 Jul '20

31 Jul '20

If PETSc requires distutils.sysconfig then ./Configure should check for it and print a very useful error message if it does not find it. We can't have python crashes off at build time with confusing error messages that users cannot decipher. Barry > On Jul 31, 2020, at 11:00 AM, Satish Balay via petsc-users <petsc-users(a)mcs.anl.gov> wrote: > > You still get this error message - so its likely not installed. To confirm you [and the admin] can run: > > python3 -c "import distutils.sysconfig" > > [and see if you get an error or not] > > You can also try python2 as I suggested in my earlier e-mail [if distutils is installed for python2] > > python2 -c "import distutils.sysconfig" > > python2 ./configure ... > > Satish > > > On Fri, 31 Jul 2020, Eda Oktay wrote: > >> Dear Matt, >> >> Administrator installed python3-dib-utils but still I am getting the >> same error message. I attached configure.log and make.log again. >> >> Thanks! >> >> Eda >> >> Matthew Knepley <knepley(a)gmail.com>, 31 Tem 2020 Cum, 13:20 tarihinde >> şunu yazdı: >>> >>> On Fri, Jul 31, 2020 at 4:15 AM Eda Oktay <eda.oktay(a)metu.edu.tr> wrote: >>>> >>>> Dear Satish, >>>> >>>> I configured petsc-3.13.2. Is this still the same problem? >>> >>> >>> Yes, you need the distutils package installed. >>> >>> Thanks, >>> >>> Matt >>> >>>> >>>> Thanks! >>>> >>>> Eda >>>> >>>> On Fri, Jul 31, 2020, 10:50 AM Satish Balay <balay(a)mcs.anl.gov> wrote: >>>>> >>>>> >>>>>>>>>>>>> >>>>> File "./config/gmakegen.py", line 4, in <module> >>>>> from distutils.sysconfig import parse_makefile >>>>> ModuleNotFoundError: No module named 'distutils.sysconfig' >>>>> <<<<< >>>>> >>>>> For one - if you used petsc-3.13.3 - it would have also tried python2 - or given a reasonable error message. >>>>> >>>>> If you have python2 - you can try: >>>>> >>>>> python2 ./configure ... >>>>> >>>>> Or you can ask sysadmin on this machine to install python3-distutils or equivalent package. >>>>> >>>>> Satish >>>>> >>>>> On Fri, 31 Jul 2020, Eda Oktay wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I configured petsc-3.13.2 with following options: >>>>>> >>>>>> ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran >>>>>> --download-mpich --download-openblas --download-metis >>>>>> --download-parmetis --download-chaco --download-slepc --with-X=1 >>>>>> >>>>>> However, I got a make error. >>>>>> >>>>>> I attached make.log and configure.log, how can I fix that problem? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> Eda >>>>>> >>>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>

1 0

Argonne National Laboratory hiring for staff position in Numerical PDEs and Scientific Computing
by Mills, Richard Tran 29 Jul '20

29 Jul '20

Dear PETSc Users and Developers, The Laboratory for Applied Mathematics, Numerical Software, and Statistics (LANS, https://www.anl.gov/mcs/lans) in the Mathematics and Computer Science Division at Argonne National Laboratory -- which has served as the "home" for PETSc development for over two decades -- is seeking a research staff member in the area of partial differential equations (PDEs), numerical software, and scientific computing. Because this position could provide an excellent opportunity for someone interested in pursuing a research program furthering the development and applications of PETSc, I want to make members of the PETSc user and developer community aware of it. Applicants at junior or senior career levels will be considered. For further details and to apply, please visit the job listings posted at Senior applicants: https://bit.ly/2Otr7De Earlier-career applicants: https://bit.ly/2B1Rxch Best regards, Richard

1 0

Re: [petsc-dev] Question about MPICH device we use
by Jed Brown 26 Jul '20

26 Jul '20

Jeff Hammond <jeff.science(a)gmail.com> writes: > On Thu, Jul 23, 2020 at 9:35 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > >> On Thu, 23 Jul 2020, Jeff Hammond wrote: >> >> > Open-MPI refuses to let users over subscribe without an extra flag to >> > mpirun. >> >> Yes - and when using this flag - it lets the run through - but there is >> still performance degradation in oversubscribe mode. >> >> > I think Intel MPI has an option for blocking poll that supports >> > oversubscription “nicely”. >> >> What option is this? Is it compile time option or something for mpiexec? >> > > https://software.intel.com/content/www/us/en/develop/articles/tuning-the-in… > > Apply wait mode to oversubscribed jobs > > This option is particularly relevant for oversubscribed MPI jobs. The goal > is to enable the wait mode of the progress engine in order to wait for > messages without polling the fabric(s). This can save CPU cycles but > decreases the message-response rate (latency), so it should be used with > caution. To enable wait mode simply use: > > I_MPI_WAIT_MODE=1 Has anyone tested ch4:ucx? $ rg UCX_PERF_WAIT_MODE src/mpid/ch4/netmod/ucx/ucx/test/gtest/common/test_perf.cc 190: params.wait_mode = UCX_PERF_WAIT_MODE_LAST; src/mpid/ch4/netmod/ucx/ucx/src/tools/perf/perftest.c 513: params->wait_mode = UCX_PERF_WAIT_MODE_LAST; src/mpid/ch4/netmod/ucx/ucx/src/tools/perf/api/libperf.h 70: UCX_PERF_WAIT_MODE_PROGRESS, /* Repeatedly call progress */ 71: UCX_PERF_WAIT_MODE_SLEEP, /* Go to sleep */ 72: UCX_PERF_WAIT_MODE_SPIN, /* Spin without calling progress */ 73: UCX_PERF_WAIT_MODE_LAST modules/ucx/test/gtest/common/test_perf.cc 189: params.wait_mode = UCX_PERF_WAIT_MODE_LAST; modules/ucx/src/tools/perf/perftest.c 553: params->wait_mode = UCX_PERF_WAIT_MODE_LAST; modules/ucx/src/tools/perf/api/libperf.h 71: UCX_PERF_WAIT_MODE_PROGRESS, /* Repeatedly call progress */ 72: UCX_PERF_WAIT_MODE_SLEEP, /* Go to sleep */ 73: UCX_PERF_WAIT_MODE_SPIN, /* Spin without calling progress */ 74: UCX_PERF_WAIT_MODE_LAST

1 0

Re: [petsc-dev] Question about MPICH device we use
by Jeff Hammond 26 Jul '20

26 Jul '20

On Thu, Jul 23, 2020 at 9:35 PM Satish Balay <balay(a)mcs.anl.gov> wrote: > On Thu, 23 Jul 2020, Jeff Hammond wrote: > > > Open-MPI refuses to let users over subscribe without an extra flag to > > mpirun. > > Yes - and when using this flag - it lets the run through - but there is > still performance degradation in oversubscribe mode. > > > I think Intel MPI has an option for blocking poll that supports > > oversubscription “nicely”. > > What option is this? Is it compile time option or something for mpiexec? > https://software.intel.com/content/www/us/en/develop/articles/tuning-the-in… Apply wait mode to oversubscribed jobs This option is particularly relevant for oversubscribed MPI jobs. The goal is to enable the wait mode of the progress engine in order to wait for messages without polling the fabric(s). This can save CPU cycles but decreases the message-response rate (latency), so it should be used with caution. To enable wait mode simply use: I_MPI_WAIT_MODE=1 Jeff > Satish > > > MPICH might have a “no local” option that > > disables shared memory, in which case nemesis over libfabric with the > > sockets or TCP provider _might_ do the right thing. But you should ask > > MPICH people for details. > > > > Jeff > > > > On Thu, Jul 23, 2020 at 12:40 PM Jed Brown <jed(a)jedbrown.org> wrote: > > > > > I think we should default to ch3:nemesis when --download-mpich, and > only > > > do ch3:sock when requested (which we would do in CI). > > > > > > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > > > > > Primarily because ch3:sock performance does not degrade in > oversubscribe > > > mode - which is developer friendly - i.e on your laptop. > > > > > > > > And folks doing optimized runs should use a properly tuned MPI for > their > > > setup anyway. > > > > > > > > In this case --download-mpich-device=ch3:nemesis is likely > appropriate > > > if using --download-mpich [and not using a separate/optimized MPI] > > > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > > > Satish > > > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on > the > > > >> Firedrake list where it ended up that reconfiguring using > ch3:nemesis > > > had a > > > >> 2x performance boost on his 16-core proc, and noticeable effect on > the 4 > > > >> core speedup. > > > >> > > > >> Why do we default to sock? > > > >> > > > >> Thanks, > > > >> > > > >> Matt > > > >> > > > >> > > > > > > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [petsc-dev] Question about MPICH device we use
by Satish Balay 24 Jul '20

24 Jul '20

I have this change at: https://gitlab.com/petsc/petsc/-/merge_requests/2990 Satish On Thu, 23 Jul 2020, Satish Balay via petsc-dev wrote: > The primary reason is for users - developing on laptops/desktop and doing development runs in oversubscribed mode. > > The choice was few percent loss in performance for sock vs exponential cost for oversubscribed usage of nemesis [so we defaulted to sock]. > > I think we should preserve this behavior for at-least the debug builds. [i.e switch only optimized builds to nemesis] > > In CI we do pay this extra cost for some of the builds [that explicitly test with wemesis, openmpi etc..] > > Satish > > On Thu, 23 Jul 2020, Jed Brown wrote: > > > I think we should default to ch3:nemesis when --download-mpich, and only do ch3:sock when requested (which we would do in CI). > > > > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > > > Primarily because ch3:sock performance does not degrade in oversubscribe mode - which is developer friendly - i.e on your laptop. > > > > > > And folks doing optimized runs should use a properly tuned MPI for their setup anyway. > > > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if using --download-mpich [and not using a separate/optimized MPI] > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > Satish > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis had a > > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > >> core speedup. > > >> > > >> Why do we default to sock? > > >> > > >> Thanks, > > >> > > >> Matt > > >> > > >> > > >

1 0

Re: [petsc-dev] Question about MPICH device we use
by Satish Balay 24 Jul '20

24 Jul '20

Should also note: the test suite is also run by users - not just CI. Only yesterday I suggested Oana to try nemesis for a different issues [on WSL] - and the response was 'test suite is slow' so reverted back to sock [and tried a different workaround for that issue] Satish On Thu, 23 Jul 2020, Satish Balay via petsc-dev wrote: > The primary reason is for users - developing on laptops/desktop and doing development runs in oversubscribed mode. > > The choice was few percent loss in performance for sock vs exponential cost for oversubscribed usage of nemesis [so we defaulted to sock]. > > I think we should preserve this behavior for at-least the debug builds. [i.e switch only optimized builds to nemesis] > > In CI we do pay this extra cost for some of the builds [that explicitly test with wemesis, openmpi etc..] > > Satish > > On Thu, 23 Jul 2020, Jed Brown wrote: > > > I think we should default to ch3:nemesis when --download-mpich, and only do ch3:sock when requested (which we would do in CI). > > > > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > > > Primarily because ch3:sock performance does not degrade in oversubscribe mode - which is developer friendly - i.e on your laptop. > > > > > > And folks doing optimized runs should use a properly tuned MPI for their setup anyway. > > > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if using --download-mpich [and not using a separate/optimized MPI] > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > Satish > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis had a > > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > >> core speedup. > > >> > > >> Why do we default to sock? > > >> > > >> Thanks, > > >> > > >> Matt > > >> > > >> > > >

1 0

Re: [petsc-dev] Question about MPICH device we use
by Junchao Zhang 24 Jul '20

24 Jul '20

On Thu, Jul 23, 2020 at 11:35 PM Satish Balay via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > On Thu, 23 Jul 2020, Jeff Hammond wrote: > > > Open-MPI refuses to let users over subscribe without an extra flag to > > mpirun. > > Yes - and when using this flag - it lets the run through - but there is > still performance degradation in oversubscribe mode. > > > I think Intel MPI has an option for blocking poll that supports > > oversubscription “nicely”. > > What option is this? Is it compile time option or something for mpiexec? > I only found configure time options, --enable-nemesis-dbg-nolocal, alias for --enable-dbg-nolocal --enable-dbg-nolocal enables debugging mode where shared-memory communication is disabled Satish > > > MPICH might have a “no local” option that > > disables shared memory, in which case nemesis over libfabric with the > > sockets or TCP provider _might_ do the right thing. But you should ask > > MPICH people for details. > > > > Jeff > > > > On Thu, Jul 23, 2020 at 12:40 PM Jed Brown <jed(a)jedbrown.org> wrote: > > > > > I think we should default to ch3:nemesis when --download-mpich, and > only > > > do ch3:sock when requested (which we would do in CI). > > > > > > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > > > > > Primarily because ch3:sock performance does not degrade in > oversubscribe > > > mode - which is developer friendly - i.e on your laptop. > > > > > > > > And folks doing optimized runs should use a properly tuned MPI for > their > > > setup anyway. > > > > > > > > In this case --download-mpich-device=ch3:nemesis is likely > appropriate > > > if using --download-mpich [and not using a separate/optimized MPI] > > > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > > > Satish > > > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on > the > > > >> Firedrake list where it ended up that reconfiguring using > ch3:nemesis > > > had a > > > >> 2x performance boost on his 16-core proc, and noticeable effect on > the 4 > > > >> core speedup. > > > >> > > > >> Why do we default to sock? > > > >> > > > >> Thanks, > > > >> > > > >> Matt > > > >> > > > >> > > > > > >

1 0

Re: [petsc-dev] Question about MPICH device we use
by Satish Balay 24 Jul '20

24 Jul '20

On Thu, 23 Jul 2020, Jeff Hammond wrote: > Open-MPI refuses to let users over subscribe without an extra flag to > mpirun. Yes - and when using this flag - it lets the run through - but there is still performance degradation in oversubscribe mode. > I think Intel MPI has an option for blocking poll that supports > oversubscription “nicely”. What option is this? Is it compile time option or something for mpiexec? Satish > MPICH might have a “no local” option that > disables shared memory, in which case nemesis over libfabric with the > sockets or TCP provider _might_ do the right thing. But you should ask > MPICH people for details. > > Jeff > > On Thu, Jul 23, 2020 at 12:40 PM Jed Brown <jed(a)jedbrown.org> wrote: > > > I think we should default to ch3:nemesis when --download-mpich, and only > > do ch3:sock when requested (which we would do in CI). > > > > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > > > Primarily because ch3:sock performance does not degrade in oversubscribe > > mode - which is developer friendly - i.e on your laptop. > > > > > > And folks doing optimized runs should use a properly tuned MPI for their > > setup anyway. > > > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate > > if using --download-mpich [and not using a separate/optimized MPI] > > > > > > Having defaults that satisfy all use cases is not practical. > > > > > > Satish > > > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis > > had a > > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > > >> core speedup. > > >> > > >> Why do we default to sock? > > >> > > >> Thanks, > > >> > > >> Matt > > >> > > >> > > >

1 0

Re: [petsc-dev] Question about MPICH device we use
by Satish Balay 24 Jul '20

24 Jul '20

The primary reason is for users - developing on laptops/desktop and doing development runs in oversubscribed mode. The choice was few percent loss in performance for sock vs exponential cost for oversubscribed usage of nemesis [so we defaulted to sock]. I think we should preserve this behavior for at-least the debug builds. [i.e switch only optimized builds to nemesis] In CI we do pay this extra cost for some of the builds [that explicitly test with wemesis, openmpi etc..] Satish On Thu, 23 Jul 2020, Jed Brown wrote: > I think we should default to ch3:nemesis when --download-mpich, and only do ch3:sock when requested (which we would do in CI). > > Satish Balay via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > > > Primarily because ch3:sock performance does not degrade in oversubscribe mode - which is developer friendly - i.e on your laptop. > > > > And folks doing optimized runs should use a properly tuned MPI for their setup anyway. > > > > In this case --download-mpich-device=ch3:nemesis is likely appropriate if using --download-mpich [and not using a separate/optimized MPI] > > > > Having defaults that satisfy all use cases is not practical. > > > > Satish > > > > On Wed, 22 Jul 2020, Matthew Knepley wrote: > > > >> We default to ch3:sock. Scott MacLachlan just had a long thread on the > >> Firedrake list where it ended up that reconfiguring using ch3:nemesis had a > >> 2x performance boost on his 16-core proc, and noticeable effect on the 4 > >> core speedup. > >> > >> Why do we default to sock? > >> > >> Thanks, > >> > >> Matt > >> > >> >

1 0