March 2018 - petsc-users - mailman3.cels.anl.gov

Nodes coordinates in distributed dmplex
by Alejandro D Otero 16 Jul '22

16 Jul '22

Hi, sorry if this is an obvious question, but I cannot figure out how to recover finite element nodes coordinates once I have distributed a mesh stored as a dmplex. I am using petsc4py as interface to petsc rutines. I first created a dmplex using: dm.createFromCellList() In a sequential run I got the coordinates with: Coords = dm.getCoordinates() which gave a sequential vector with the coordinates of the mesh nodes. When I distribute the mesh with: dm.distribute() each mpi process has it own dm but the indexing of the vector resulting from getCoordinates() or getCoordinatesLocal() seems not consistent with the local numbering of the cells and nodes. Which is the correct way of doing this in PETSc philosophy? Thanks in advance, Alejandro

4 9

MatGetDiagonal
by Nachiket Gokhale 09 Dec '19

09 Dec '19

Any chance of making this work in serial? http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetDiago… Not a show stopper, I am trying to get the diagonal of some small projected, dense matrices (which come from large sparse matrices). I am running in serial because 1) Since my projected matrices are small, and 2) PETSc does not do certain matrix multiplications involving a dense matrix in parallel, -Nachiket

4 5

GAMG scaling
by Mark Adams 24 Dec '18

24 Dec '18

(Hong), what is the current state of optimizing RAP for scaling? Nate, is driving 3D elasticity problems at scaling with GAMG and we are working out performance problems. They are hitting problems at ~1.5B dof problems on a basic Cray (XC30 I think). Thanks, Mark

7 28

monitoring the convergence of fieldsplit 0 and 1
by Klaij, Christiaan 15 May '18

15 May '18

A question when using PCFieldSplit: for each linear iteration of the system, how many iterations for fielsplit 0 and 1? One way to find out is to run with -ksp_monitor, -fieldsplit_0_ksp_monitor and -fieldsplit_0_ksp_monitor. This gives the complete convergence history. Another way, suggested by Matt, is to use -ksp_monitor, -fieldsplit_0_ksp_converged_reason and -fieldsplit_1_ksp_converged_reason. This gives only the totals for fieldsplit 0 and 1 (but without saying for which one). Both ways require to somehow process the output, which is a bit inconvenient. Could KSPGetResidualHistory perhaps return (some) information on the subsystems' convergence for processing inside the code? Chris dr. ir. Christiaan Klaij CFD Researcher Research & Development E mailto:[email protected] T +31 317 49 33 44 MARIN 2, Haagsteeg, P.O. Box 28, 6700 AA Wageningen, The Netherlands T +31 317 49 39 11, F +31 317 49 32 45, I www.marin.nl

4 19

Re: [petsc-users] Scaling problem when cores > 600
by Smith, Barry F. 22 Apr '18

22 Apr '18

What are you using for Poisson log. If it is a Poisson problem then almost for sure you should be using Hypre BoomerAMG?. It sounds like your matrix does not change. You will need to discuss the scaling with the hypre people. Barry > On Mar 7, 2018, at 5:38 AM, TAY wee-beng <zonexo(a)gmail.com> wrote: > > > On 7/3/2018 6:22 AM, Smith, Barry F. wrote: >> The speed up for "Poisson log" is 1.6425364214878704 = 5.0848e+02/3.0957e+02 >> >> This is lower than I would expect for Hypre BoomerAMG? >> >> Are you doing multiple solves with the same matrix with hypre or is each solve a new matrix? If each solve is a new matrix then you may be getting expected behavior since the multigrid AMG construction process does not scale as well as the application of AMG once it is constructed. >> >> I am forwarding to the hypre team since this is their expertise not ours >> >> Barry >> > Hi, > > My LHS of the eqn does not change. Only the RHS changes at each time step. So should this be expected? > > So maybe I should change to BoomerAMG and compare? > > Will PETSc GAMG give better performance? > > Also, I must add that I only partition in the x and y direction. Will this be a factor? > > Thanks. > >>> On Mar 5, 2018, at 11:19 PM, TAY wee-beng <zonexo(a)gmail.com> wrote: >>> >>> >>> On 5/3/2018 11:43 AM, Smith, Barry F. wrote: >>>> 360 process >>>> >>>> KSPSolve 99 1.0 2.6403e+02 1.0 6.67e+10 1.1 2.7e+05 9.9e+05 5.1e+02 15100 17 42 19 15100 17 42 19 87401 >>>> >>>> 1920 processes >>>> >>>> KSPSolve 99 1.0 2.3184e+01 1.0 1.32e+10 1.2 1.5e+06 4.3e+05 5.1e+02 4100 17 42 19 4100 17 42 19 967717 >>>> >>>> >>>> Ratio of number of processes 5.33 ratio of time for KSPSolve 11.388 so the time for the solve is scaling very well (extremely well actually). The problem is >>>> due to "other time" that is not in KSP solve. Note that the percentage of the total time in KSPSolve went from 15 percent of the runtime to 4 percent. This means something outside of KSPSolve is scaling very poorly. You will need to profile the rest of the code to determine where the time is being spent. PetscLogEventRegister() and PetscLogEventBegin/End() will be needed in your code. Already with 360 processes the linear solver is only taking 15 percent of the time. >>>> >>>> Barry >>>> >>> Hi, >>> >>> I have attached the new logging results with the HYPRE Poisson eqn solver. However, due to some problems, I am now using Intel 2018. Should be quite similar to 2016 in terms of runtime. Using 360 processes can't work this time, and I'm not sure why though. >>>> >>>>> On Mar 4, 2018, at 9:23 PM, TAY wee-beng <zonexo(a)gmail.com> wrote: >>>>> >>>>> >>>>> On 1/3/2018 12:14 PM, Smith, Barry F. wrote: >>>>>>> On Feb 28, 2018, at 8:01 PM, TAY wee-beng <zonexo(a)gmail.com> wrote: >>>>>>> >>>>>>> >>>>>>> On 1/3/2018 12:10 AM, Matthew Knepley wrote: >>>>>>>> On Wed, Feb 28, 2018 at 10:45 AM, TAY wee-beng <zonexo(a)gmail.com> wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have a CFD code which uses PETSc and HYPRE. I found that for a certain case with grid size of 192,570,048, I encounter scaling problem when my cores > 600. At 600 cores, the code took 10min for 100 time steps. At 960, 1440 and 2880 cores, it still takes around 10min. At 360 cores, it took 15min. >>>>>>>> >>>>>>>> So how can I find the bottleneck? Any recommended steps? >>>>>>>> >>>>>>>> For any performance question, we need to see the output of -log_view for all test cases. >>>>>>> Hi, >>>>>>> >>>>>>> To be more specific, I use PETSc KSPBCGS and HYPRE geometric multigrid (entirely based on HYPRE, no PETSc) for the momentum and Poisson eqns in my code. >>>>>>> >>>>>>> So can log_view be used in this case to give a meaningful? Since part of the code uses HYPRE? >>>>>> Yes, just send the logs. >>>>>> >>>>> Hi, >>>>> >>>>> I have attached the logs, with the number indicating the no. of cores used. Some of the new results are different from the previous runs, although I'm using the same cluster. >>>>> >>>>> Thanks for the help. >>>>>>> I also program another subroutine in the past which uses PETSc to solve the Poisson eqn. It uses either HYPRE's boomeramg, KSPBCGS or KSPGMRES. >>>>>>> >>>>>>> If I use boomeramg, can log_view be used in this case? >>>>>>> >>>>>>> Or do I have to use KSPBCGS or KSPGMRES, which is directly from PETSc? However, I ran KSPGMRES yesterday with the Poisson eqn and my ans didn't converge. >>>>>>> >>>>>>> Thanks. >>>>>>>> I must also mention that I partition my grid only in the x and y direction. There is no partitioning in the z direction due to limited code development. I wonder if there is a strong effect in this case. >>>>>>>> >>>>>>>> Maybe. Usually what happens is you fill up memory with a z-column and cannot scale further. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> -- >>>>>>>> Thank you very much >>>>>>>> >>>>>>>> Yours sincerely, >>>>>>>> >>>>>>>> ================================================ >>>>>>>> TAY Wee-Beng 郑伟明 (Zheng Weiming) >>>>>>>> Personal research webpage: http://tayweebeng.wixsite.com/website >>>>>>>> Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA >>>>>>>> linkedin: www.linkedin.com/in/tay-weebeng >>>>>>>> ================================================ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <log960.txt><log600.txt><log360.txt><log1920.txt> >>> <log1920_2.txt><log600_2.txt><log960_2.txt><log1440_2.txt>

3 5

ftp.mcs.anl.gov is down - breaking petsc builds with --download-package
by Satish Balay 05 Apr '18

05 Apr '18

We've been informed by the site admin that ftp.mcs.anl.gov is taken down due to 'cyber incident' - there is no ETA on when it will be back. We've been using it the primary host for distributing petsc tarballs and hosting some externalpackage tarballs - so some of the petsc buils with '--download-package' option are failing. Obtaining petsc [and some most externalpackages] via git should work. I will update when ftp.mcs.anl.gov is brought back up. Satish

2 4

Could not execute "['git', 'rev-parse', '--git-dir']"
by Kong, Fande 05 Apr '18

05 Apr '18

Hi PETSc team: What is the possible reason for this? The log file is attached. Fande,

5 23

--download-hdf5 and then compiling cgns
by Klaij, Christiaan 05 Apr '18

05 Apr '18

Satish, I've build petsc-3.8.3 with --download-hdf5 and this works fine. Now, I'm trying to compile cgns-3.3.1 and have it use the hdf5 library from petsc. Therefore, in the cmake configure I have set this option -D HDF5_DIR=/path/to/petsc/hdf5 but cmake gives this message: CMake Warning at CMakeLists.txt:207 (find_package): Could not find a package configuration file provided by "HDF5" with any of the following names: hdf5Config.cmake hdf5-config.cmake and then falls back to the system's hdf5. Indeed there are no such files in /path/to/petsc/hdf5. Any idea on how to proceed from here? Chris dr. ir. Christiaan Klaij | Senior Researcher | Research & Development MARIN | T +31 317 49 33 44 | mailto:[email protected] | http://www.marin.nl MARIN news: http://www.marin.nl/web/News/News-items/Comfort-draft-for-semisubmersible-y…

3 5

Obtaining compiling and building information from a.out
by TAY wee-beng 04 Apr '18

04 Apr '18

Hi, I have been compiling and building different version of my CFD with the intel 2016, 2018 compilers, and also different compiling options. I tested a version of my a.out and it is much faster than the other a.out, using only 3 min instead of more than 10min to solve a certain case using GAMG. However, I can't recall how it was compiled. I only know that I used the intel 2016 compiler. So is there any way I can find out how the a.out was compiled? Like what options were used? -- Thank you very much. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) 郑伟明 Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================

3 6

Compilation error with PETSc and intel options
by TAY wee-beng 30 Mar '18

30 Mar '18

Hi, I have an error compiling PETSc with Intel Fortran and its options. I have attached the configure.log -- Thank you very much. Yours sincerely, ================================================ TAY Wee-Beng (Zheng Weiming) 郑伟明 Personal research webpage: http://tayweebeng.wixsite.com/website Youtube research showcase: https://www.youtube.com/channel/UC72ZHtvQNMpNs2uRTSToiLA linkedin: www.linkedin.com/in/tay-weebeng ================================================

2 1