November 2023 - petsc-users - mailman3.cels.anl.gov

Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI
by Barry Smith 15 Nov '23

15 Nov '23

It is successfully installed and you can start using it. Barry > On Nov 15, 2023, at 6:49 AM, 程奔 <ctchengben(a)mail.scut.edu.cn> wrote: > > Hi Barry, > > I follow your suggestion and the result showed: > mpiexec -n 1 ./ex19 -snes_monitor > lid velocity = 0.0625, prandtl # = 1., grashof # = 1. > 0 SNES Function norm 2.391552133017e-01 > 1 SNES Function norm 6.839858507066e-05 > 2 SNES Function norm 8.558777232425e-11 > Number of SNES iterations = 2 > > > It seems worked, shall I try the more examples or PETSc had been installed successfully on my computer. > > sinserely, > > Cheng. > > > > > > > -----原始邮件----- > 发件人: "Barry Smith" <bsmith(a)petsc.dev <mailto:[email protected]>> > 发送时间: 2023-11-15 01:17:42 (星期三) > 收件人: 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> > 抄送: petsc-users <petsc-users(a)mcs.anl.gov <mailto:[email protected]>> > 主题: Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI > > > > You can check the build was successful manually. > > cd src/snes/tutorials > make ex19 > mpiexec -n 1 ./ex19 -snes_monitor > > > >> On Nov 14, 2023, at 11:46 AM, 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> wrote: >> >> Hi Barry, >> >> Thanks for the suggestion. It seems good after I change complier to english.Then I begin to configure and make it. >> But unfortunate, when I try to make PETSC_DIR=/cygdrive/g/mypetsc/petsc-3.19.2 PETSC_ARCH=arch-mswin-c-debug check >> It just happen to an error: >> Running check examples to verify correct installation >> Using PETSC_DIR=/cygdrive/g/mypetsc/petsc-3.19.2 and PETSC_ARCH=arch-mswin-c-debug >> /usr/bin/bash: -c: line 9: unexpected EOF while looking for matching `"' >> make[1]: *** [makefile:123: check] Error 2 >> make: *** [GNUmakefile:17: check] Error 2 >> >> >> So I send email to look for you help, and configure.log is attached. >> >> >> sinserely, >> Cheng. >> >> >> >> >> -----原始邮件----- >> 发件人: "Barry Smith" <bsmith(a)petsc.dev <mailto:[email protected]>> >> 发送时间: 2023-11-13 11:19:23 (星期一) >> 收件人: 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> >> 抄送: petsc-users <petsc-users(a)mcs.anl.gov <mailto:[email protected]>> >> 主题: Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI >> >> >> The configure is failing while testing the Intel Fortran compiler >> >> Executing: /cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win_ifort -c -o /tmp/petsc-61cxbt4e/config.setCompilers/conftest.o -I/tmp/petsc-61cxbt4e/config.compilers -I/tmp/petsc-61cxbt4e/config.setCompilers -MT -Z7 -Od /tmp/petsc-61cxbt4e/config.setCompilers/conftest.F90 >> Successful compile: >> Source: >> program main >> >> end >> >> Executing: /cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win_ifort -o /tmp/petsc-61cxbt4e/config.setCompilers/conftest.exe -MT -Z7 -Od /tmp/petsc-61cxbt4e/config.setCompilers/conftest.o >> stdout: LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� >> Linker output before filtering: >> LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� >> Linker output after filtering: >> LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� >> >> Can you please try changing the compiler to print its messages in English and then attempt to compile the trivial program above as indicated and send all the output. >> >> Barry >> >> >>> On Nov 12, 2023, at 9:02 PM, 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> wrote: >>> >>> Sorry for replying to your email so late, and I find configure.log >>> >>> >>> -----原始邮件----- >>> 发件人: "Barry Smith" <bsmith(a)petsc.dev <mailto:[email protected]>> >>> 发送时间: 2023-11-08 23:42:30 (星期三) >>> 收件人: 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> >>> 抄送: petsc-users(a)mcs.anl.gov <mailto:[email protected]> >>> 主题: Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI >>> >>> >>> Send the file $PETSC_ARCH/lib/petsc/conf/configure.log >>> >>> >>> >>>> On Nov 8, 2023, at 12:20 AM, 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> wrote: >>>> >>>> Hello, >>>> Recently I try to install PETSc with Cygwin since I'd like to use PETSc with Visual Studio on Windows10 plateform.For the sake of clarity, I firstly list the softwares/packages used below: >>>> 1. PETSc: version 3.19.2 >>>> 2. VS: version 2022 >>>> 3. Intel MPI: download Intel oneAPI Base Toolkit and HPC Toolkit >>>> 4. Cygwin: see the picture attatched (see picture cygwin) >>>> >>>> >>>> And the compiler option in configuration is: >>>> ./configure --prefix=/cygdrive/g/mypetsc/petsc2023 --with-cc='win32fe cl' --with-fc='win32fe ifort' --with-cxx='win32fe cl' --with-shared-libraries=0 >>>> --with-mpi-include=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/include >>>> --with-mpi-lib=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/libTherefore, I write this e-mail to look for your help. >>>> /release/impi.lib >>>> --with-mpiexec=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/bin/mpiexec -localonly >>>> >>>> >>>> >>>> >>>> but there return an error: >>>> ********************************************************************************************* >>>> OSError while running ./configure >>>> --------------------------------------------------------------------------------------------- >>>> Cannot run executables created with FC. If this machine uses a batch system >>>> to submit jobs you will need to configure using ./configure with the additional option >>>> --with-batch. >>>> Otherwise there is problem with the compilers. Can you compile and run code with your >>>> compiler '/cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win32fe ifort'? >>>> See https://petsc.org/release/faq/#error-libimf >>>> ********************************************************************************************* >>>> >>>> >>>> >>>> Then I try to open configure.log in petsc, but there turnout an error that I can't open it.(see picture 1) >>>> >>>> And then I right click on properties and click safety,it just turnout "The permissions on test directory are incorrectly ordered.which may cause some entries to be ineffective." (see picture 2) >>>> >>>> And it also likely seen “NULL SID” as the top entry in permission lists.(see picture 3) >>>> >>>> Then i follow this blog(https://blog.dhampir.no/content/forcing-cygwin-to-create-sane-permissions-on-windows) to edit /etc/fstab in Cygwin, and add “noacl” to the mount options for /cygdrive. >>>> >>>> But it's not working. >>>> >>>> So I can't sent configure.log to you guys, it seems cygwin that installed in my computer happened to some problem. >>>> >>>> Mayebe the error happened in the configure on petsc just because of this reason. >>>> >>>> >>>> >>>> So I wrrit this email to report my problem and ask for your help. >>>> >>>> >>>> Looking forward your reply! >>>> >>>> >>>> sinserely, >>>> Cheng. >>>> <cygwin.png><picture1.jpg><picture2.png><picture3.jpg>

1 0

Inquiry Regarding Callback Implementation for SNES in PETSc
by Rene Chenard 14 Nov '23

14 Nov '23

Dear PETSc Developers, I hope this message finds you well. My name is René Chenard, and I am a Research Professional at Université Laval. We are currently working on implementing a wrapper for the SNES in our project and are seeking guidance on incorporating callbacks at different stages of the resolution process. Specifically, our objective is to implement a callback that triggers at the initiation of every nonlinear iteration and another that activates at the conclusion of each nonlinear iteration. In our exploration, we discovered the potential use of SNESSetUpdate and SNESSetConvergenceTest for this purpose. However, we encountered a challenge with SNESSetUpdate, as it seems to be ineffective for the ngmres and anderson solver types, the latter of which appears to be based on the implementation of ngmres. We are reaching out to seek clarification on whether this behavior is intentional and to explore alternative approaches that might better suit our needs. To facilitate a clearer understanding of our observations, we have prepared a reproduction example in the file named test_snes.c. This file outlines the specific scenarios where we encountered challenges with SNESSetUpdate and provides a context for our inquiries. Here are our specific questions: 1. Is SNESSetUpdate designed to function uniformly across all types of SNES solvers? 2. What would be the recommended approach to implement custom callbacks around every nonlinear iteration, especially considering the apparent limitations with SNESSetUpdate for certain solver types? 3. We observed a discrepancy in the iteration/step numbering between the update function (set by SNESSetUpdate) and the convergence function (set by SNESSetConvergenceTest). Could you provide clarification on this, considering the documentation's description of SNESSetUpdate as the function "called at the beginning of every iteration of the nonlinear solve"? We genuinely appreciate your expertise in this matter, and your insights will be invaluable in guiding our implementation. Thank you in advance for your consideration and support. Warm regards, —René Chenard Research Professional at Université Laval rene.chenard.1(a)ulaval.ca <mailto:[email protected]>

2 1

Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI
by Barry Smith 14 Nov '23

14 Nov '23

You can check the build was successful manually. cd src/snes/tutorials make ex19 mpiexec -n 1 ./ex19 -snes_monitor > On Nov 14, 2023, at 11:46 AM, 程奔 <ctchengben(a)mail.scut.edu.cn> wrote: > > Hi Barry, > > Thanks for the suggestion. It seems good after I change complier to english.Then I begin to configure and make it. > But unfortunate, when I try to make PETSC_DIR=/cygdrive/g/mypetsc/petsc-3.19.2 PETSC_ARCH=arch-mswin-c-debug check > It just happen to an error: > Running check examples to verify correct installation > Using PETSC_DIR=/cygdrive/g/mypetsc/petsc-3.19.2 and PETSC_ARCH=arch-mswin-c-debug > /usr/bin/bash: -c: line 9: unexpected EOF while looking for matching `"' > make[1]: *** [makefile:123: check] Error 2 > make: *** [GNUmakefile:17: check] Error 2 > > > So I send email to look for you help, and configure.log is attached. > > > sinserely, > Cheng. > > > > > -----原始邮件----- > 发件人: "Barry Smith" <bsmith(a)petsc.dev <mailto:[email protected]>> > 发送时间: 2023-11-13 11:19:23 (星期一) > 收件人: 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> > 抄送: petsc-users <petsc-users(a)mcs.anl.gov <mailto:[email protected]>> > 主题: Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI > > > The configure is failing while testing the Intel Fortran compiler > > Executing: /cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win_ifort -c -o /tmp/petsc-61cxbt4e/config.setCompilers/conftest.o -I/tmp/petsc-61cxbt4e/config.compilers -I/tmp/petsc-61cxbt4e/config.setCompilers -MT -Z7 -Od /tmp/petsc-61cxbt4e/config.setCompilers/conftest.F90 > Successful compile: > Source: > program main > > end > > Executing: /cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win_ifort -o /tmp/petsc-61cxbt4e/config.setCompilers/conftest.exe -MT -Z7 -Od /tmp/petsc-61cxbt4e/config.setCompilers/conftest.o > stdout: LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� > Linker output before filtering: > LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� > Linker output after filtering: > LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� > > Can you please try changing the compiler to print its messages in English and then attempt to compile the trivial program above as indicated and send all the output. > > Barry > > >> On Nov 12, 2023, at 9:02 PM, 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> wrote: >> >> Sorry for replying to your email so late, and I find configure.log >> >> >> -----原始邮件----- >> 发件人: "Barry Smith" <bsmith(a)petsc.dev <mailto:[email protected]>> >> 发送时间: 2023-11-08 23:42:30 (星期三) >> 收件人: 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> >> 抄送: petsc-users(a)mcs.anl.gov <mailto:[email protected]> >> 主题: Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI >> >> >> Send the file $PETSC_ARCH/lib/petsc/conf/configure.log >> >> >> >>> On Nov 8, 2023, at 12:20 AM, 程奔 <ctchengben(a)mail.scut.edu.cn <mailto:[email protected]>> wrote: >>> >>> Hello, >>> Recently I try to install PETSc with Cygwin since I'd like to use PETSc with Visual Studio on Windows10 plateform.For the sake of clarity, I firstly list the softwares/packages used below: >>> 1. PETSc: version 3.19.2 >>> 2. VS: version 2022 >>> 3. Intel MPI: download Intel oneAPI Base Toolkit and HPC Toolkit >>> 4. Cygwin: see the picture attatched (see picture cygwin) >>> >>> >>> And the compiler option in configuration is: >>> ./configure --prefix=/cygdrive/g/mypetsc/petsc2023 --with-cc='win32fe cl' --with-fc='win32fe ifort' --with-cxx='win32fe cl' --with-shared-libraries=0 >>> --with-mpi-include=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/include >>> --with-mpi-lib=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/libTherefore, I write this e-mail to look for your help. >>> /release/impi.lib >>> --with-mpiexec=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/bin/mpiexec -localonly >>> >>> >>> >>> >>> but there return an error: >>> ********************************************************************************************* >>> OSError while running ./configure >>> --------------------------------------------------------------------------------------------- >>> Cannot run executables created with FC. If this machine uses a batch system >>> to submit jobs you will need to configure using ./configure with the additional option >>> --with-batch. >>> Otherwise there is problem with the compilers. Can you compile and run code with your >>> compiler '/cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win32fe ifort'? >>> See https://petsc.org/release/faq/#error-libimf >>> ********************************************************************************************* >>> >>> >>> >>> Then I try to open configure.log in petsc, but there turnout an error that I can't open it.(see picture 1) >>> >>> And then I right click on properties and click safety,it just turnout "The permissions on test directory are incorrectly ordered.which may cause some entries to be ineffective." (see picture 2) >>> >>> And it also likely seen “NULL SID” as the top entry in permission lists.(see picture 3) >>> >>> Then i follow this blog(https://blog.dhampir.no/content/forcing-cygwin-to-create-sane-permissions-on-windows) to edit /etc/fstab in Cygwin, and add “noacl” to the mount options for /cygdrive. >>> >>> But it's not working. >>> >>> So I can't sent configure.log to you guys, it seems cygwin that installed in my computer happened to some problem. >>> >>> Mayebe the error happened in the configure on petsc just because of this reason. >>> >>> >>> >>> So I wrrit this email to report my problem and ask for your help. >>> >>> >>> Looking forward your reply! >>> >>> >>> sinserely, >>> Cheng. >>> <cygwin.png><picture1.jpg><picture2.png><picture3.jpg> >> > > >

1 0

Re: [petsc-users] [petsc-maint] small problems
by Matthew Knepley 13 Nov '23

13 Nov '23

On Mon, Nov 13, 2023 at 8:23 AM edgar via petsc-maint < petsc-maint(a)mcs.anl.gov> wrote: > Dear list, > > Some weeks ago, someone kindly shared a patch to improve the performance > on small problems (less than or around 1 000 dofs). I deleted the > e-mail, thinking that I would be able to find it back on the archives. > It may have been in the petsc-maint list, which means that there is no > archive. I would like to ask if someone could redirect the thread, a > link in https://lists.mcs.anl.gov/pipermail or tell me where I can find > the pull request or patch. Thank you very much in advance. > I cannot find it. What search string could we use? Thanks, Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

small problems
by edgar 13 Nov '23

13 Nov '23

Dear list, Some weeks ago, someone kindly shared a patch to improve the performance on small problems (less than or around 1 000 dofs). I deleted the e-mail, thinking that I would be able to find it back on the archives. It may have been in the petsc-maint list, which means that there is no archive. I would like to ask if someone could redirect the thread, a link in https://lists.mcs.anl.gov/pipermail or tell me where I can find the pull request or patch. Thank you very much in advance.

1 0

Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI
by Barry Smith 12 Nov '23

12 Nov '23

The configure is failing while testing the Intel Fortran compiler Executing: /cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win_ifort -c -o /tmp/petsc-61cxbt4e/config.setCompilers/conftest.o -I/tmp/petsc-61cxbt4e/config.compilers -I/tmp/petsc-61cxbt4e/config.setCompilers -MT -Z7 -Od /tmp/petsc-61cxbt4e/config.setCompilers/conftest.F90 Successful compile: Source: program main end Executing: /cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win_ifort -o /tmp/petsc-61cxbt4e/config.setCompilers/conftest.exe -MT -Z7 -Od /tmp/petsc-61cxbt4e/config.setCompilers/conftest.o stdout: LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� Linker output before filtering: LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� Linker output after filtering: LINK : û��ҵ� G:\cygwin\tmp\PE9718~1\CONFIG~1.SET\conftest.exe ��һ��û��ִ��ȫ�� Can you please try changing the compiler to print its messages in English and then attempt to compile the trivial program above as indicated and send all the output. Barry > On Nov 12, 2023, at 9:02 PM, 程奔 <ctchengben(a)mail.scut.edu.cn> wrote: > > Sorry for replying to your email so late, and I find configure.log > > > -----原始邮件----- > 发件人: "Barry Smith" <bsmith(a)petsc.dev> > 发送时间: 2023-11-08 23:42:30 (星期三) > 收件人: 程奔 <ctchengben(a)mail.scut.edu.cn> > 抄送: petsc-users(a)mcs.anl.gov > 主题: Re: [petsc-users] Error in configuring PETSc with Cygwin on Windows by using Intel MPI > > > Send the file $PETSC_ARCH/lib/petsc/conf/configure.log > > > >> On Nov 8, 2023, at 12:20 AM, 程奔 <ctchengben(a)mail.scut.edu.cn> wrote: >> >> Hello, >> Recently I try to install PETSc with Cygwin since I'd like to use PETSc with Visual Studio on Windows10 plateform.For the sake of clarity, I firstly list the softwares/packages used below: >> 1. PETSc: version 3.19.2 >> 2. VS: version 2022 >> 3. Intel MPI: download Intel oneAPI Base Toolkit and HPC Toolkit >> 4. Cygwin: see the picture attatched (see picture cygwin) >> >> >> And the compiler option in configuration is: >> ./configure --prefix=/cygdrive/g/mypetsc/petsc2023 --with-cc='win32fe cl' --with-fc='win32fe ifort' --with-cxx='win32fe cl' --with-shared-libraries=0 >> --with-mpi-include=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/include >> --with-mpi-lib=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/libTherefore, I write this e-mail to look for your help. >> /release/impi.lib >> --with-mpiexec=/cygdrive/g/Intel/oneAPI/mpi/2021.10.0/bin/mpiexec -localonly >> >> >> >> >> but there return an error: >> ********************************************************************************************* >> OSError while running ./configure >> --------------------------------------------------------------------------------------------- >> Cannot run executables created with FC. If this machine uses a batch system >> to submit jobs you will need to configure using ./configure with the additional option >> --with-batch. >> Otherwise there is problem with the compilers. Can you compile and run code with your >> compiler '/cygdrive/g/mypetsc/petsc-3.19.2/lib/petsc/bin/win32fe/win32fe ifort'? >> See https://petsc.org/release/faq/#error-libimf >> ********************************************************************************************* >> >> >> >> Then I try to open configure.log in petsc, but there turnout an error that I can't open it.(see picture 1) >> >> And then I right click on properties and click safety,it just turnout "The permissions on test directory are incorrectly ordered.which may cause some entries to be ineffective." (see picture 2) >> >> And it also likely seen “NULL SID” as the top entry in permission lists.(see picture 3) >> >> Then i follow this blog(https://blog.dhampir.no/content/forcing-cygwin-to-create-sane-permissions-on-windows) to edit /etc/fstab in Cygwin, and add “noacl” to the mount options for /cygdrive. >> >> But it's not working. >> >> So I can't sent configure.log to you guys, it seems cygwin that installed in my computer happened to some problem. >> >> Mayebe the error happened in the configure on petsc just because of this reason. >> >> >> >> So I wrrit this email to report my problem and ask for your help. >> >> >> Looking forward your reply! >> >> >> sinserely, >> Cheng. >> <cygwin.png><picture1.jpg><picture2.png><picture3.jpg> >

1 0

Re: [petsc-users] Petsc4py Simulation: Mat.axpy() slow
by Barry Smith 11 Nov '23

11 Nov '23

The flop rate of 123 megaflops is absurdly small, something unexected must be happening. Any chance you can send me small code that reproduces the below? > On Nov 11, 2023, at 9:19 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu> wrote: > > I've isolated the profiling to only run during the addition of the two matrices. The output is > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > MatAssemblyBegin 1 1.0 4.5000e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAssemblyEnd 1 1.0 7.4370e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0 > MatAXPY 1 1.0 6.6962e-01 1.0 8.24e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 31 0 0 0 100 100 0 0 0 123 > ------------------------------------------------------------------------------------------------------------------------ > > On Sat, Nov 11, 2023 at 6:20 PM Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >> My apologies I missed your other question, >> >> Yes I imagine most of those setValue and getRow are due to the construction of the matrices in question. I Ended up writing my own "semi-parallel" kronecker product function which involves setting and getting >> a lot of values. I imagine I should optimize that much more, but I consider it an upfront cost to time propagation, so I haven't touched it since I got it working. >> >> Thank you for the quick replies >> >> >> >> On Sat, Nov 11, 2023 at 6:12 PM Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >>> My apologies, that was run with 7 cores, this is the result for n = 1. >>> >>> ------------------------------------------------------------------------------------------------------------------------ >>> Event Count Time (sec) Flop --- Global --- --- Stage ---- Total >>> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> --- Event Stage 0: Main Stage >>> >>> BuildTwoSided 13 1.0 1.4770e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> BuildTwoSidedF 13 1.0 2.3150e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecView 24 1.0 3.1525e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecCopy 12 1.0 4.1000e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyBegin 13 1.0 5.1290e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecAssemblyEnd 13 1.0 7.0200e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> VecSetRandom 2 1.0 6.2500e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatMult 112 1.0 1.7071e-03 1.0 1.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 8268 >>> MatSolve 112 1.0 8.3160e-04 1.0 7.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 8486 >>> MatLUFactorSym 2 1.0 1.2660e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatLUFactorNum 2 1.0 2.2895e-04 1.0 1.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 4646 >>> MatScale 2 1.0 4.4903e-02 1.0 1.65e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 61 0 0 0 0 61 0 0 0 3671 >>> MatAssemblyBegin 169 1.0 4.5040e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAssemblyEnd 169 1.0 2.2126e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 40 0 0 0 0 40 0 0 0 0 0 >>> MatSetValues 9078 1.0 6.1440e-01 1.0 8.24e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 31 0 0 0 1 31 0 0 0 134 >>> MatGetRow 9078 1.0 5.8936e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatGetRowIJ 2 1.0 9.0900e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatGetOrdering 2 1.0 3.6880e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> MatAXPY 1 1.0 6.4112e-01 1.0 8.24e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 31 0 0 0 1 31 0 0 0 129 >>> PCSetUp 2 1.0 4.6308e-04 1.0 1.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2297 >>> PCApply 112 1.0 8.4632e-04 1.0 7.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 8339 >>> KSPSetUp 2 1.0 4.5000e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> KSPSolve 112 1.0 8.9241e-04 1.0 7.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 7908 >>> EPSSetUp 2 1.0 6.5904e-04 1.0 1.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1614 >>> EPSSolve 2 1.0 4.4982e-03 1.0 2.11e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 8 0 0 0 0 8 0 0 0 4687 >>> STSetUp 2 1.0 5.0441e-04 1.0 1.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2109 >>> STComputeOperatr 2 1.0 2.5700e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> STApply 112 1.0 1.6852e-03 1.0 1.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 8375 >>> STMatSolve 112 1.0 9.1930e-04 1.0 7.06e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 7677 >>> BVCopy 20 1.0 1.7930e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> BVMultVec 219 1.0 2.3467e-04 1.0 2.31e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 9861 >>> BVMultInPlace 12 1.0 1.1088e-04 1.0 1.09e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 9857 >>> BVDotVec 219 1.0 2.9956e-04 1.0 2.47e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 8245 >>> BVOrthogonalizeV 114 1.0 5.8952e-04 1.0 4.82e+06 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 8181 >>> BVScale 114 1.0 2.9150e-05 1.0 4.06e+04 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1392 >>> BVSetRandom 2 1.0 9.6900e-06 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> BVMatMultVec 112 1.0 1.7464e-03 1.0 1.41e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 5 0 0 0 0 5 0 0 0 8082 >>> DSSolve 10 1.0 9.1354e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> DSVectors 24 1.0 1.1908e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> DSOther 30 1.0 1.2798e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>> ------------------------------------------------------------------------------------------------------------------------ >>> >>> On Sat, Nov 11, 2023 at 5:59 PM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>> >>>> How many MPI processes did you use? Please try with just one to get a base line >>>> >>>> MatSetValues 1298 1.0 1.2313e-01 1.1 1.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 32 0 0 0 1 32 0 0 0 669 >>>> MatGetRow 1298 1.0 1.6896e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>> MatAXPY 1 1.0 1.7225e-01 1.0 1.18e+07 1.0 8.4e+01 1.3e+03 6.0e+00 1 32 5 14 0 1 32 5 14 0 478 >>>> >>>> This is concerning: there are 1298 MatSetValues and MatGetRow, are you calling them? If not that means the MatAXPY is calling them (if SAME_NONZERO_PATTERN is not used these are used in the MatAXPY). >>>> >>>>> I'm still somewhat new to parallel computing, so I'm not sure what you mean by binding. Does this lock in certain processes to certain cores? >>>> >>>> >>>> Yes, it is usually the right thing to do for numerical computing. I included a link to some discussion of it on petsc.org <http://petsc.org/> >>>> >>>>> On Nov 11, 2023, at 6:38 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >>>>> >>>>> Hello again, >>>>> >>>>> I've run the simulation with profiling again. In this setup I only ran the necessary methods to construct the matrices, and then at the end I added H_mix and H_ang using the Mataxpy method. Below are the results >>>>> >>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>> Event Count Time (sec) Flop --- Global --- --- Stage ---- Total >>>>> Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>> >>>>> --- Event Stage 0: Main Stage >>>>> >>>>> BuildTwoSided 350 1.0 6.9605e-01 1.9 0.00e+00 0.0 3.9e+02 5.6e+00 3.5e+02 3 0 24 0 22 3 0 24 0 22 0 >>>>> BuildTwoSidedF 236 1.0 6.9569e-01 1.9 0.00e+00 0.0 2.3e+02 1.1e+01 2.4e+02 3 0 14 0 15 3 0 14 0 15 0 >>>>> SFSetGraph 114 1.0 2.0555e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> SFSetUp 116 1.0 1.2123e-03 1.1 0.00e+00 0.0 6.2e+02 8.9e+02 1.1e+02 0 0 38 71 7 0 0 38 71 7 0 >>>>> SFPack 312 1.0 6.2350e-05 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> SFUnpack 312 1.0 2.2470e-05 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecView 26 1.0 4.0207e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecCopy 13 1.0 7.6100e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecAssemblyBegin 14 1.0 2.9690e-04 2.1 0.00e+00 0.0 2.3e+02 1.1e+01 1.4e+01 0 0 14 0 1 0 0 14 0 1 0 >>>>> VecAssemblyEnd 14 1.0 2.9400e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecScatterBegin 312 1.0 6.0288e-04 1.6 0.00e+00 0.0 7.3e+02 3.1e+02 1.0e+02 0 0 45 29 6 0 0 45 29 7 0 >>>>> VecScatterEnd 312 1.0 9.1886e-04 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> VecSetRandom 2 1.0 3.6500e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatMult 104 1.0 2.5355e-02 1.0 2.65e+05 1.1 8.0e+02 2.8e+02 2.2e+02 0 1 49 29 14 0 1 49 29 14 70 >>>>> MatSolve 104 1.0 2.4431e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 1.1e+02 0 0 49 29 7 0 0 49 29 7 36 >>>>> MatLUFactorSym 2 1.0 1.2584e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatLUFactorNum 2 1.0 7.7804e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatScale 2 1.0 4.5068e-02 1.0 2.36e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 65 0 0 0 0 65 0 0 0 3657 >>>>> MatAssemblyBegin 169 1.0 4.2296e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02 2 0 0 0 7 2 0 0 0 7 0 >>>>> MatAssemblyEnd 169 1.0 3.6979e+00 1.0 0.00e+00 0.0 5.9e+02 9.4e+02 7.8e+02 22 0 36 71 48 22 0 36 71 49 0 >>>>> MatSetValues 1298 1.0 1.2313e-01 1.1 1.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 32 0 0 0 1 32 0 0 0 669 >>>>> MatGetRow 1298 1.0 1.6896e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> MatAXPY 1 1.0 1.7225e-01 1.0 1.18e+07 1.0 8.4e+01 1.3e+03 6.0e+00 1 32 5 14 0 1 32 5 14 0 478 >>>>> PCSetUp 2 1.0 2.0848e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0 >>>>> PCApply 104 1.0 2.4459e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 1.1e+02 0 0 49 29 7 0 0 49 29 7 36 >>>>> KSPSetUp 2 1.0 1.8400e-06 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> KSPSolve 104 1.0 2.5008e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 2.2e+02 0 0 49 29 14 0 0 49 29 14 35 >>>>> EPSSetUp 2 1.0 2.4027e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 1 0 0 0 0 1 0 >>>>> EPSSolve 2 1.0 3.0789e-02 1.0 1.09e+06 1.1 8.0e+02 2.8e+02 4.5e+02 0 3 49 29 28 0 3 49 29 28 242 >>>>> STSetUp 2 1.0 2.1294e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0 >>>>> STComputeOperatr 2 1.0 2.1100e-06 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> STApply 104 1.0 2.5290e-02 1.0 2.65e+05 1.1 8.0e+02 2.8e+02 2.2e+02 0 1 49 29 14 0 1 49 29 14 70 >>>>> STMatSolve 104 1.0 2.5081e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 2.2e+02 0 0 49 29 14 0 0 49 29 14 35 >>>>> BVCopy 20 1.0 2.8770e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> BVMultVec 206 1.0 1.8736e-04 1.5 3.13e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 11442 >>>>> BVMultInPlace 11 1.0 9.2730e-05 1.2 1.49e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 11026 >>>>> BVDotVec 206 1.0 1.0389e-03 1.2 3.35e+05 1.1 0.0e+00 0.0e+00 2.1e+02 0 1 0 0 13 0 1 0 0 13 2205 >>>>> BVOrthogonalizeV 106 1.0 1.2649e-03 1.1 6.84e+05 1.1 0.0e+00 0.0e+00 2.1e+02 0 2 0 0 13 0 2 0 0 13 3706 >>>>> BVScale 106 1.0 9.0329e-05 1.9 5.51e+03 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 418 >>>>> BVSetRandom 2 1.0 1.4080e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> BVMatMultVec 104 1.0 2.5468e-02 1.0 2.65e+05 1.1 8.0e+02 2.8e+02 2.2e+02 0 1 49 29 14 0 1 49 29 14 70 >>>>> DSSolve 9 1.0 1.2452e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> DSVectors 24 1.0 1.5345e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> DSOther 27 1.0 1.6735e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>> ------------------------------------------------------------------------------------------------------------------------ >>>>> >>>>> I believe that the only matrix addition I did is shown there to take about 0.17 seconds. Doing so twice per timestep in addition to scaling, this ends up taking about 0.4 seconds >>>>> during the full calculation so the timestep is far too slow. >>>>> >>>>> However it seems that the number of flops is on the order of 10^7. Would this imply that >>>>> I am perhaps memory bandwidth bound on my home pc? >>>>> >>>>> I'm still somewhat new to parallel computing, so I'm not sure what you mean by binding. Does this lock in certain processes to certain cores? >>>>> >>>>> Thank you for your time and assistance >>>>> >>>>> On Sat, Nov 11, 2023 at 2:54 PM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>> >>>>>> Here is the code for MPIBAIJ with SAME_NONZERO_STRUCTURE. (the other formats are similar) >>>>>> >>>>>> PetscErrorCode MatAXPY_SeqBAIJ(Mat Y, PetscScalar a, Mat X, MatStructure str) >>>>>> { >>>>>> Mat_SeqBAIJ *x = (Mat_SeqBAIJ *)X->data, *y = (Mat_SeqBAIJ *)Y->data; >>>>>> PetscInt bs = Y->rmap->bs, bs2 = bs * bs; >>>>>> PetscBLASInt one = 1; >>>>>> .... >>>>>> if (str == SAME_NONZERO_PATTERN) { >>>>>> PetscScalar alpha = a; >>>>>> PetscBLASInt bnz; >>>>>> PetscCall(PetscBLASIntCast(x->nz * bs2, &bnz)); >>>>>> PetscCallBLAS("BLASaxpy", BLASaxpy_(&bnz, &alpha, x->a, &one, y->a, &one)); >>>>>> PetscCall(PetscObjectStateIncrease((PetscObject)Y)); >>>>>> >>>>>> It directly adds the nonzero values from the two matrices together (the nonzero structure plays no role) and it uses the BLAS so it should perform as "fast as possible" given the hardware (are you configured --with-debugging=0 ?, are you using binding with mpiexec to ensure MPI is using the best combination of cores? https://petsc.org/release/faq/#what-kind-of-parallel-computers-or-clusters-… <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org…>) >>>>>> >>>>>> Can you run with -log_view and send the output so we can see directly the performance? >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Nov 11, 2023, at 4:25 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hello, and thank you for the quick reply. >>>>>>> >>>>>>> For some context, all of these matrices are produced from a kronecker product in parallel between an "angular" and a "radial" matrix. In this case S_total and H_atom use the same angular matrix which is the identity, while their >>>>>>> radial matrices obey different sparsity. >>>>>>> >>>>>>> Meanwhile H_mix and H_ang are constructed from two different angular matrices, however both are only nonzero along the two off diagonals, and the radial matrices have the same sparse structure. >>>>>>> >>>>>>> So S_total and H_atom have the same block sparse structure, while H_mix and H_ang have the same block sparse structure. However even just adding H_mix and H_ang for the simplest case takes around half a second which is still too slow even when using SAME_NONZERO_PATTERN . >>>>>>> >>>>>>> This confuses me the most because another researcher wrote a similar code a few years back using finite difference, so his matrices are on the order 250kx250k and larger, yet the matrix additions are far faster for him. >>>>>>> >>>>>>> If you would like I can show more code snippets but im not sure what you would want to see. >>>>>>> >>>>>>> Thank you for your time >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Nov 11, 2023 at 11:22 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> DIFFERENT_NONZERO_PATTERN will always be slow because it needs to determine the nonzero pattern of the result for each computation. >>>>>>>> >>>>>>>> SAME_NONZERO_PATTERN can obviously run fast but may not be practical for your problem. >>>>>>>> >>>>>>>> SUBSET_NONZERO_PATTERN (which means in A += b*B we know that B has NO nonzero locations that A does not have) does not need to reallocate anything in A so it can be reasonably fast (we can do more optimizations on the current code to improve it for you). >>>>>>>> >>>>>>>> So, can you construct the S nonzero structure initially to have the union of the nonzero structures of all the matrices that accumulate into it? This is the same >>>>>>>> nonzero structure that it "ends up with" in the current code. With this nonzero structure, SUBSET_NONZERO_PATTERN can be used. Depending on how sparse all the matrices that get accumulated into S are, you could perhaps go further and make sure they all have the same nonzero structure (sure, it uses more memory and will do extra operations, but the actual computation will be so fast it may be worth the extra computations. >>>>>>>> >>>>>>>> Barry >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Nov 10, 2023, at 9:53 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> I am trying to use petsc4py to conduct a quantum mechanics simulation. I've been able to construct all of the relevant matrices, however I am reaching a gigantic bottleneck. >>>>>>>>> >>>>>>>>> For the simplest problem I am running I have a few matrices each about 5000x5000. In order to begin time propagation I need to add these matrices together. However, on 6 cores of my local machine it is taking approximately 1-2 seconds per addition. Since I need to do this for each time step in my simulation it is prohibitively slow since there could be upwards of 10K time steps. >>>>>>>>> >>>>>>>>> Below is the relevant code: >>>>>>>>> >>>>>>>>> structure = structure=PETSc.Mat.Structure.DIFFERENT_NONZERO_PATTERN >>>>>>>>> if test2: >>>>>>>>> def makeLeft(S,MIX,ANG,ATOM,i): >>>>>>>>> >>>>>>>>> >>>>>>>>> S.axpy(-Field.pulse[i],MIX,structure) >>>>>>>>> S.axpy(-Field.pulse[i],ANG,structure) >>>>>>>>> S.axpy(-1,ATOM,structure) >>>>>>>>> return S >>>>>>>>> def makeRight(S,MIX,ANG,ATOM,i): >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> S.axpy(Field.pulse[i],MIX,structure) >>>>>>>>> S.axpy(Field.pulse[i],ANG,structure) >>>>>>>>> S.axpy(1,ATOM,structure) >>>>>>>>> >>>>>>>>> return S >>>>>>>>> >>>>>>>>> H_mix = Int.H_mix >>>>>>>>> H_mix.scale(1j * dt /2) >>>>>>>>> >>>>>>>>> H_ang = Int.H_ang >>>>>>>>> H_ang.scale(1j * dt /2) >>>>>>>>> >>>>>>>>> H_atom = Int.H_atom >>>>>>>>> H_atom.scale(1j * dt /2) >>>>>>>>> >>>>>>>>> S = Int.S_total >>>>>>>>> >>>>>>>>> psi_initial = psi.psi_initial.copy() >>>>>>>>> ksp = PETSc.KSP().create(PETSc.COMM_WORLD) >>>>>>>>> >>>>>>>>> >>>>>>>>> for i,t in enumerate(box.t): >>>>>>>>> print(i,L) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> O_L = makeLeft(S,H_mix,H_ang,H_atom,i) >>>>>>>>> O_R = makeRight(S,H_mix,H_ang,H_atom,i) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> if i == 0: >>>>>>>>> known = O_R.getVecRight() >>>>>>>>> sol = O_L.getVecRight() >>>>>>>>> >>>>>>>>> O_R.mult(psi_initial,known) >>>>>>>>> >>>>>>>>> ksp.setOperators(O_L) >>>>>>>>> >>>>>>>>> >>>>>>>>> ksp.solve(known,sol) >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> psi_initial.copy(sol) >>>>>>>>> >>>>>>>>> >>>>>>>>> I need to clean it up a bit, but the main point is that I need to add the matrices many times for a single time step. I can't preallocate memory very well since some of the matrices aren't the most sparse either. It seems if I cannot speed up the addition it will be difficult to continue so I was wondering if you had any insights. >>>>>>>>> >>>>>>>>> Thank you for your time >>>>>>>> >>>>>> >>>>

1 0

Re: [petsc-users] Petsc4py Simulation: Mat.axpy() slow
by Barry Smith 11 Nov '23

11 Nov '23

How many MPI processes did you use? Please try with just one to get a base line MatSetValues 1298 1.0 1.2313e-01 1.1 1.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 32 0 0 0 1 32 0 0 0 669 MatGetRow 1298 1.0 1.6896e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 MatAXPY 1 1.0 1.7225e-01 1.0 1.18e+07 1.0 8.4e+01 1.3e+03 6.0e+00 1 32 5 14 0 1 32 5 14 0 478 This is concerning: there are 1298 MatSetValues and MatGetRow, are you calling them? If not that means the MatAXPY is calling them (if SAME_NONZERO_PATTERN is not used these are used in the MatAXPY). > I'm still somewhat new to parallel computing, so I'm not sure what you mean by binding. Does this lock in certain processes to certain cores? Yes, it is usually the right thing to do for numerical computing. I included a link to some discussion of it on petsc.org > On Nov 11, 2023, at 6:38 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu> wrote: > > Hello again, > > I've run the simulation with profiling again. In this setup I only ran the necessary methods to construct the matrices, and then at the end I added H_mix and H_ang using the Mataxpy method. Below are the results > > ------------------------------------------------------------------------------------------------------------------------ > Event Count Time (sec) Flop --- Global --- --- Stage ---- Total > Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s > ------------------------------------------------------------------------------------------------------------------------ > > --- Event Stage 0: Main Stage > > BuildTwoSided 350 1.0 6.9605e-01 1.9 0.00e+00 0.0 3.9e+02 5.6e+00 3.5e+02 3 0 24 0 22 3 0 24 0 22 0 > BuildTwoSidedF 236 1.0 6.9569e-01 1.9 0.00e+00 0.0 2.3e+02 1.1e+01 2.4e+02 3 0 14 0 15 3 0 14 0 15 0 > SFSetGraph 114 1.0 2.0555e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SFSetUp 116 1.0 1.2123e-03 1.1 0.00e+00 0.0 6.2e+02 8.9e+02 1.1e+02 0 0 38 71 7 0 0 38 71 7 0 > SFPack 312 1.0 6.2350e-05 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > SFUnpack 312 1.0 2.2470e-05 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecView 26 1.0 4.0207e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecCopy 13 1.0 7.6100e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecAssemblyBegin 14 1.0 2.9690e-04 2.1 0.00e+00 0.0 2.3e+02 1.1e+01 1.4e+01 0 0 14 0 1 0 0 14 0 1 0 > VecAssemblyEnd 14 1.0 2.9400e-05 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecScatterBegin 312 1.0 6.0288e-04 1.6 0.00e+00 0.0 7.3e+02 3.1e+02 1.0e+02 0 0 45 29 6 0 0 45 29 7 0 > VecScatterEnd 312 1.0 9.1886e-04 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > VecSetRandom 2 1.0 3.6500e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatMult 104 1.0 2.5355e-02 1.0 2.65e+05 1.1 8.0e+02 2.8e+02 2.2e+02 0 1 49 29 14 0 1 49 29 14 70 > MatSolve 104 1.0 2.4431e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 1.1e+02 0 0 49 29 7 0 0 49 29 7 36 > MatLUFactorSym 2 1.0 1.2584e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatLUFactorNum 2 1.0 7.7804e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatScale 2 1.0 4.5068e-02 1.0 2.36e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 65 0 0 0 0 65 0 0 0 3657 > MatAssemblyBegin 169 1.0 4.2296e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 1.1e+02 2 0 0 0 7 2 0 0 0 7 0 > MatAssemblyEnd 169 1.0 3.6979e+00 1.0 0.00e+00 0.0 5.9e+02 9.4e+02 7.8e+02 22 0 36 71 48 22 0 36 71 49 0 > MatSetValues 1298 1.0 1.2313e-01 1.1 1.18e+07 1.0 0.0e+00 0.0e+00 0.0e+00 1 32 0 0 0 1 32 0 0 0 669 > MatGetRow 1298 1.0 1.6896e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > MatAXPY 1 1.0 1.7225e-01 1.0 1.18e+07 1.0 8.4e+01 1.3e+03 6.0e+00 1 32 5 14 0 1 32 5 14 0 478 > PCSetUp 2 1.0 2.0848e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0 > PCApply 104 1.0 2.4459e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 1.1e+02 0 0 49 29 7 0 0 49 29 7 36 > KSPSetUp 2 1.0 1.8400e-06 5.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > KSPSolve 104 1.0 2.5008e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 2.2e+02 0 0 49 29 14 0 0 49 29 14 35 > EPSSetUp 2 1.0 2.4027e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 1 0 0 0 0 1 0 > EPSSolve 2 1.0 3.0789e-02 1.0 1.09e+06 1.1 8.0e+02 2.8e+02 4.5e+02 0 3 49 29 28 0 3 49 29 28 242 > STSetUp 2 1.0 2.1294e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0 > STComputeOperatr 2 1.0 2.1100e-06 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > STApply 104 1.0 2.5290e-02 1.0 2.65e+05 1.1 8.0e+02 2.8e+02 2.2e+02 0 1 49 29 14 0 1 49 29 14 70 > STMatSolve 104 1.0 2.5081e-02 1.0 1.31e+05 1.2 8.0e+02 2.8e+02 2.2e+02 0 0 49 29 14 0 0 49 29 14 35 > BVCopy 20 1.0 2.8770e-05 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > BVMultVec 206 1.0 1.8736e-04 1.5 3.13e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 11442 > BVMultInPlace 11 1.0 9.2730e-05 1.2 1.49e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 11026 > BVDotVec 206 1.0 1.0389e-03 1.2 3.35e+05 1.1 0.0e+00 0.0e+00 2.1e+02 0 1 0 0 13 0 1 0 0 13 2205 > BVOrthogonalizeV 106 1.0 1.2649e-03 1.1 6.84e+05 1.1 0.0e+00 0.0e+00 2.1e+02 0 2 0 0 13 0 2 0 0 13 3706 > BVScale 106 1.0 9.0329e-05 1.9 5.51e+03 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 418 > BVSetRandom 2 1.0 1.4080e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > BVMatMultVec 104 1.0 2.5468e-02 1.0 2.65e+05 1.1 8.0e+02 2.8e+02 2.2e+02 0 1 49 29 14 0 1 49 29 14 70 > DSSolve 9 1.0 1.2452e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > DSVectors 24 1.0 1.5345e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > DSOther 27 1.0 1.6735e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 > ------------------------------------------------------------------------------------------------------------------------ > > I believe that the only matrix addition I did is shown there to take about 0.17 seconds. Doing so twice per timestep in addition to scaling, this ends up taking about 0.4 seconds > during the full calculation so the timestep is far too slow. > > However it seems that the number of flops is on the order of 10^7. Would this imply that > I am perhaps memory bandwidth bound on my home pc? > > I'm still somewhat new to parallel computing, so I'm not sure what you mean by binding. Does this lock in certain processes to certain cores? > > Thank you for your time and assistance > > On Sat, Nov 11, 2023 at 2:54 PM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> Here is the code for MPIBAIJ with SAME_NONZERO_STRUCTURE. (the other formats are similar) >> >> PetscErrorCode MatAXPY_SeqBAIJ(Mat Y, PetscScalar a, Mat X, MatStructure str) >> { >> Mat_SeqBAIJ *x = (Mat_SeqBAIJ *)X->data, *y = (Mat_SeqBAIJ *)Y->data; >> PetscInt bs = Y->rmap->bs, bs2 = bs * bs; >> PetscBLASInt one = 1; >> .... >> if (str == SAME_NONZERO_PATTERN) { >> PetscScalar alpha = a; >> PetscBLASInt bnz; >> PetscCall(PetscBLASIntCast(x->nz * bs2, &bnz)); >> PetscCallBLAS("BLASaxpy", BLASaxpy_(&bnz, &alpha, x->a, &one, y->a, &one)); >> PetscCall(PetscObjectStateIncrease((PetscObject)Y)); >> >> It directly adds the nonzero values from the two matrices together (the nonzero structure plays no role) and it uses the BLAS so it should perform as "fast as possible" given the hardware (are you configured --with-debugging=0 ?, are you using binding with mpiexec to ensure MPI is using the best combination of cores? https://petsc.org/release/faq/#what-kind-of-parallel-computers-or-clusters-… <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpetsc.org…>) >> >> Can you run with -log_view and send the output so we can see directly the performance? >> >> >> >> >>> On Nov 11, 2023, at 4:25 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >>> >>> Hello, and thank you for the quick reply. >>> >>> For some context, all of these matrices are produced from a kronecker product in parallel between an "angular" and a "radial" matrix. In this case S_total and H_atom use the same angular matrix which is the identity, while their >>> radial matrices obey different sparsity. >>> >>> Meanwhile H_mix and H_ang are constructed from two different angular matrices, however both are only nonzero along the two off diagonals, and the radial matrices have the same sparse structure. >>> >>> So S_total and H_atom have the same block sparse structure, while H_mix and H_ang have the same block sparse structure. However even just adding H_mix and H_ang for the simplest case takes around half a second which is still too slow even when using SAME_NONZERO_PATTERN . >>> >>> This confuses me the most because another researcher wrote a similar code a few years back using finite difference, so his matrices are on the order 250kx250k and larger, yet the matrix additions are far faster for him. >>> >>> If you would like I can show more code snippets but im not sure what you would want to see. >>> >>> Thank you for your time >>> >>> >>> >>> On Sat, Nov 11, 2023 at 11:22 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>> >>>> >>>> DIFFERENT_NONZERO_PATTERN will always be slow because it needs to determine the nonzero pattern of the result for each computation. >>>> >>>> SAME_NONZERO_PATTERN can obviously run fast but may not be practical for your problem. >>>> >>>> SUBSET_NONZERO_PATTERN (which means in A += b*B we know that B has NO nonzero locations that A does not have) does not need to reallocate anything in A so it can be reasonably fast (we can do more optimizations on the current code to improve it for you). >>>> >>>> So, can you construct the S nonzero structure initially to have the union of the nonzero structures of all the matrices that accumulate into it? This is the same >>>> nonzero structure that it "ends up with" in the current code. With this nonzero structure, SUBSET_NONZERO_PATTERN can be used. Depending on how sparse all the matrices that get accumulated into S are, you could perhaps go further and make sure they all have the same nonzero structure (sure, it uses more memory and will do extra operations, but the actual computation will be so fast it may be worth the extra computations. >>>> >>>> Barry >>>> >>>> >>>> >>>>> On Nov 10, 2023, at 9:53 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >>>>> >>>>> Hello, >>>>> >>>>> I am trying to use petsc4py to conduct a quantum mechanics simulation. I've been able to construct all of the relevant matrices, however I am reaching a gigantic bottleneck. >>>>> >>>>> For the simplest problem I am running I have a few matrices each about 5000x5000. In order to begin time propagation I need to add these matrices together. However, on 6 cores of my local machine it is taking approximately 1-2 seconds per addition. Since I need to do this for each time step in my simulation it is prohibitively slow since there could be upwards of 10K time steps. >>>>> >>>>> Below is the relevant code: >>>>> >>>>> structure = structure=PETSc.Mat.Structure.DIFFERENT_NONZERO_PATTERN >>>>> if test2: >>>>> def makeLeft(S,MIX,ANG,ATOM,i): >>>>> >>>>> >>>>> S.axpy(-Field.pulse[i],MIX,structure) >>>>> S.axpy(-Field.pulse[i],ANG,structure) >>>>> S.axpy(-1,ATOM,structure) >>>>> return S >>>>> def makeRight(S,MIX,ANG,ATOM,i): >>>>> >>>>> >>>>> >>>>> S.axpy(Field.pulse[i],MIX,structure) >>>>> S.axpy(Field.pulse[i],ANG,structure) >>>>> S.axpy(1,ATOM,structure) >>>>> >>>>> return S >>>>> >>>>> H_mix = Int.H_mix >>>>> H_mix.scale(1j * dt /2) >>>>> >>>>> H_ang = Int.H_ang >>>>> H_ang.scale(1j * dt /2) >>>>> >>>>> H_atom = Int.H_atom >>>>> H_atom.scale(1j * dt /2) >>>>> >>>>> S = Int.S_total >>>>> >>>>> psi_initial = psi.psi_initial.copy() >>>>> ksp = PETSc.KSP().create(PETSc.COMM_WORLD) >>>>> >>>>> >>>>> for i,t in enumerate(box.t): >>>>> print(i,L) >>>>> >>>>> >>>>> >>>>> O_L = makeLeft(S,H_mix,H_ang,H_atom,i) >>>>> O_R = makeRight(S,H_mix,H_ang,H_atom,i) >>>>> >>>>> >>>>> >>>>> if i == 0: >>>>> known = O_R.getVecRight() >>>>> sol = O_L.getVecRight() >>>>> >>>>> O_R.mult(psi_initial,known) >>>>> >>>>> ksp.setOperators(O_L) >>>>> >>>>> >>>>> ksp.solve(known,sol) >>>>> >>>>> >>>>> >>>>> >>>>> psi_initial.copy(sol) >>>>> >>>>> >>>>> I need to clean it up a bit, but the main point is that I need to add the matrices many times for a single time step. I can't preallocate memory very well since some of the matrices aren't the most sparse either. It seems if I cannot speed up the addition it will be difficult to continue so I was wondering if you had any insights. >>>>> >>>>> Thank you for your time >>>> >>

1 0

Re: [petsc-users] Petsc4py Simulation: Mat.axpy() slow
by Barry Smith 11 Nov '23

11 Nov '23

Here is the code for MPIBAIJ with SAME_NONZERO_STRUCTURE. (the other formats are similar) PetscErrorCode MatAXPY_SeqBAIJ(Mat Y, PetscScalar a, Mat X, MatStructure str) { Mat_SeqBAIJ *x = (Mat_SeqBAIJ *)X->data, *y = (Mat_SeqBAIJ *)Y->data; PetscInt bs = Y->rmap->bs, bs2 = bs * bs; PetscBLASInt one = 1; .... if (str == SAME_NONZERO_PATTERN) { PetscScalar alpha = a; PetscBLASInt bnz; PetscCall(PetscBLASIntCast(x->nz * bs2, &bnz)); PetscCallBLAS("BLASaxpy", BLASaxpy_(&bnz, &alpha, x->a, &one, y->a, &one)); PetscCall(PetscObjectStateIncrease((PetscObject)Y)); It directly adds the nonzero values from the two matrices together (the nonzero structure plays no role) and it uses the BLAS so it should perform as "fast as possible" given the hardware (are you configured --with-debugging=0 ?, are you using binding with mpiexec to ensure MPI is using the best combination of cores? https://petsc.org/release/faq/#what-kind-of-parallel-computers-or-clusters-…) Can you run with -log_view and send the output so we can see directly the performance? > On Nov 11, 2023, at 4:25 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu> wrote: > > Hello, and thank you for the quick reply. > > For some context, all of these matrices are produced from a kronecker product in parallel between an "angular" and a "radial" matrix. In this case S_total and H_atom use the same angular matrix which is the identity, while their > radial matrices obey different sparsity. > > Meanwhile H_mix and H_ang are constructed from two different angular matrices, however both are only nonzero along the two off diagonals, and the radial matrices have the same sparse structure. > > So S_total and H_atom have the same block sparse structure, while H_mix and H_ang have the same block sparse structure. However even just adding H_mix and H_ang for the simplest case takes around half a second which is still too slow even when using SAME_NONZERO_PATTERN . > > This confuses me the most because another researcher wrote a similar code a few years back using finite difference, so his matrices are on the order 250kx250k and larger, yet the matrix additions are far faster for him. > > If you would like I can show more code snippets but im not sure what you would want to see. > > Thank you for your time > > > > On Sat, Nov 11, 2023 at 11:22 AM Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >> >> >> DIFFERENT_NONZERO_PATTERN will always be slow because it needs to determine the nonzero pattern of the result for each computation. >> >> SAME_NONZERO_PATTERN can obviously run fast but may not be practical for your problem. >> >> SUBSET_NONZERO_PATTERN (which means in A += b*B we know that B has NO nonzero locations that A does not have) does not need to reallocate anything in A so it can be reasonably fast (we can do more optimizations on the current code to improve it for you). >> >> So, can you construct the S nonzero structure initially to have the union of the nonzero structures of all the matrices that accumulate into it? This is the same >> nonzero structure that it "ends up with" in the current code. With this nonzero structure, SUBSET_NONZERO_PATTERN can be used. Depending on how sparse all the matrices that get accumulated into S are, you could perhaps go further and make sure they all have the same nonzero structure (sure, it uses more memory and will do extra operations, but the actual computation will be so fast it may be worth the extra computations. >> >> Barry >> >> >> >>> On Nov 10, 2023, at 9:53 PM, Donald Rex Planalp <Donald.Planalp(a)colorado.edu <mailto:[email protected]>> wrote: >>> >>> Hello, >>> >>> I am trying to use petsc4py to conduct a quantum mechanics simulation. I've been able to construct all of the relevant matrices, however I am reaching a gigantic bottleneck. >>> >>> For the simplest problem I am running I have a few matrices each about 5000x5000. In order to begin time propagation I need to add these matrices together. However, on 6 cores of my local machine it is taking approximately 1-2 seconds per addition. Since I need to do this for each time step in my simulation it is prohibitively slow since there could be upwards of 10K time steps. >>> >>> Below is the relevant code: >>> >>> structure = structure=PETSc.Mat.Structure.DIFFERENT_NONZERO_PATTERN >>> if test2: >>> def makeLeft(S,MIX,ANG,ATOM,i): >>> >>> >>> S.axpy(-Field.pulse[i],MIX,structure) >>> S.axpy(-Field.pulse[i],ANG,structure) >>> S.axpy(-1,ATOM,structure) >>> return S >>> def makeRight(S,MIX,ANG,ATOM,i): >>> >>> >>> >>> S.axpy(Field.pulse[i],MIX,structure) >>> S.axpy(Field.pulse[i],ANG,structure) >>> S.axpy(1,ATOM,structure) >>> >>> return S >>> >>> H_mix = Int.H_mix >>> H_mix.scale(1j * dt /2) >>> >>> H_ang = Int.H_ang >>> H_ang.scale(1j * dt /2) >>> >>> H_atom = Int.H_atom >>> H_atom.scale(1j * dt /2) >>> >>> S = Int.S_total >>> >>> psi_initial = psi.psi_initial.copy() >>> ksp = PETSc.KSP().create(PETSc.COMM_WORLD) >>> >>> >>> for i,t in enumerate(box.t): >>> print(i,L) >>> >>> >>> >>> O_L = makeLeft(S,H_mix,H_ang,H_atom,i) >>> O_R = makeRight(S,H_mix,H_ang,H_atom,i) >>> >>> >>> >>> if i == 0: >>> known = O_R.getVecRight() >>> sol = O_L.getVecRight() >>> >>> O_R.mult(psi_initial,known) >>> >>> ksp.setOperators(O_L) >>> >>> >>> ksp.solve(known,sol) >>> >>> >>> >>> >>> psi_initial.copy(sol) >>> >>> >>> I need to clean it up a bit, but the main point is that I need to add the matrices many times for a single time step. I can't preallocate memory very well since some of the matrices aren't the most sparse either. It seems if I cannot speed up the addition it will be difficult to continue so I was wondering if you had any insights. >>> >>> Thank you for your time >>

1 0

Petsc4py Simulation: Mat.axpy() slow
by Donald Rex Planalp 11 Nov '23

11 Nov '23

Hello, I am trying to use petsc4py to conduct a quantum mechanics simulation. I've been able to construct all of the relevant matrices, however I am reaching a gigantic bottleneck. For the simplest problem I am running I have a few matrices each about 5000x5000. In order to begin time propagation I need to add these matrices together. However, on 6 cores of my local machine it is taking approximately 1-2 seconds per addition. Since I need to do this for each time step in my simulation it is prohibitively slow since there could be upwards of 10K time steps. Below is the relevant code: structure = structure=PETSc.Mat.Structure.DIFFERENT_NONZERO_PATTERN if test2: def makeLeft(S,MIX,ANG,ATOM,i): S.axpy(-Field.pulse[i],MIX,structure) S.axpy(-Field.pulse[i],ANG,structure) S.axpy(-1,ATOM,structure) return S def makeRight(S,MIX,ANG,ATOM,i): S.axpy(Field.pulse[i],MIX,structure) S.axpy(Field.pulse[i],ANG,structure) S.axpy(1,ATOM,structure) return S H_mix = Int.H_mix H_mix.scale(1j * dt /2) H_ang = Int.H_ang H_ang.scale(1j * dt /2) H_atom = Int.H_atom H_atom.scale(1j * dt /2) S = Int.S_total psi_initial = psi.psi_initial.copy() ksp = PETSc.KSP().create(PETSc.COMM_WORLD) for i,t in enumerate(box.t): print(i,L) O_L = makeLeft(S,H_mix,H_ang,H_atom,i) O_R = makeRight(S,H_mix,H_ang,H_atom,i) if i == 0: known = O_R.getVecRight() sol = O_L.getVecRight() O_R.mult(psi_initial,known) ksp.setOperators(O_L) ksp.solve(known,sol) psi_initial.copy(sol) I need to clean it up a bit, but the main point is that I need to add the matrices many times for a single time step. I can't preallocate memory very well since some of the matrices aren't the most sparse either. It seems if I cannot speed up the addition it will be difficult to continue so I was wondering if you had any insights. Thank you for your time

2 1