December 2018 - petsc-dev - mailman3.cels.anl.gov

Fwd: [petsc-maint] Can't install latest version using spack, complains about superludist cxx clash
by Matthew Knepley 20 Dec '18

20 Dec '18

This thread is instructive for people who think Spack is a good system. Matt ---------- Forwarded message --------- From: Balay, Satish via petsc-maint <petsc-maint(a)mcs.anl.gov> Date: Thu, Dec 20, 2018 at 3:01 PM Subject: Re: [petsc-maint] Can't install latest version using spack, complains about superludist cxx clash To: Syed, Sajid Ali <sajidsyed2021(a)u.northwestern.edu> Cc: Balay, Satish <balay(a)mcs.anl.gov>, petsc-maint <petsc-maint(a)mcs.anl.gov> Can you send the relavent config files? Try installing with spack install --keep-stage petsc [options] [from the build location] send: spack-build.env spack-build.out configure.log Satish On Thu, 20 Dec 2018, Sajid Ali via petsc-maint wrote: > Looking closely, I don’t understand why petsc was ever linked to libblas > and liblapack at lib64. > > Does this have anything to do with the F77 warning ? > > [sajid@xrm lib]$ ldd libpetsc.so.3.10.3 > linux-vdso.so.1 => (0x00007ffc2cdef000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6385ee8000) > libm.so.6 => /lib64/libm.so.6 (0x00007f6385be6000) > libdl.so.2 => /lib64/libdl.so.2 (0x00007f63859e2000) > libsuperlu_dist.so.6 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/superlu-dist-6.1.0-gi3hicljihkga7x3ew66q5xdyunmmqc3/lib/libsuperlu_dist > .so.6 (0x00007f6385724000) > libmkl_intel_lp64.so => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_lib > raries_2019.0.117/linux/mkl/lib/intel64/libmkl_intel_lp64.so > (0x00007f6384bf3000) > libmkl_sequential.so => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_lib > raries_2019.0.117/linux/mkl/lib/intel64/libmkl_sequential.so > (0x00007f638365a000) > libmkl_core.so => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/intel-mkl-2019.0.117-wzqlcijwx7odz2x5chembudo5leqpfh2/compilers_and_libraries > _2019.0.117/linux/mkl/lib/intel64/libmkl_core.so (0x00007f637f521000) > libhdf5hl_fortran.so.100 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/hdf5-1.10.4-ltstvsxvyjue2gxfegi4nvr6c5xg3zww/lib/libhdf5hl_fortran. > so.100 (0x00007f637f304000) > libhdf5_fortran.so.100 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/hdf5-1.10.4-ltstvsxvyjue2gxfegi4nvr6c5xg3zww/lib/libhdf5_fortran.so.1 > 00 (0x00007f637f0bc000) > libhdf5_hl.so.100 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/hdf5-1.10.4-ltstvsxvyjue2gxfegi4nvr6c5xg3zww/lib/libhdf5_hl.so.100 > (0x0000 > 7f637ee97000) > libhdf5.so.103 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/hdf5-1.10.4-ltstvsxvyjue2gxfegi4nvr6c5xg3zww/lib/libhdf5.so.103 > (0x00007f637e > 8a4000) > libparmetis.so => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/parmetis-4.0.3-hw3j2ss7mjsc5x5f2gaflirnuufzptil/lib/libparmetis.so > (0x00007f6 > 37e66c000) > libmetis.so => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/metis-5.1.0-nhgzn4kjskctzmzv35mstvd34nj2ugek/lib/libmetis.so > (0x00007f637e3fc000 > ) > libz.so.1 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/zlib-1.2.11-ldu43taplg2nbkxtem346zq4ibhad64i/lib/libz.so.1 > (0x00007f637e1e5000) > libX11.so.6 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/libx11-1.6.5-fq4iuwdxjyckf5eqsvnistset6p4hrzm/lib/libX11.so.6 > (0x00007f637deaa00 > 0) > libstdc++.so.6 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libstdc++.so.6 > (0x00007f637d > b28000) > libmpifort.so.12 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/lib/libmpifort.so.12 > (0x00007f63 > 7d8ef000) > libmpi.so.12 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/lib/libmpi.so.12 > (0x00007f637d39c000 > ) > libgfortran.so.4 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libgfortran.so.4 > (0x00007f > 637cfc9000) > libgcc_s.so.1 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libgcc_s.so.1 > (0x00007f637cdb > 1000) > libquadmath.so.0 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libquadmath.so.0 > (0x00007f > 637cb72000) > libc.so.6 => /lib64/libc.so.6 (0x00007f637c7af000) > /lib64/ld-linux-x86-64.so.2 (0x0000562660743000) > liblapack.so.3 => /lib64/liblapack.so.3 (0x00007f637c051000) > libblas.so.3 => /lib64/libblas.so.3 (0x00007f637bdf7000) > libmpicxx.so.12 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/mpich-3.3-z5uiwmx24jylnivuhlnqjjmm674ozj6x/lib/libmpicxx.so.12 > (0x00007f637b > bd6000) > libgomp.so.1 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-4.8.5/gcc-7.3.0-qrjpi76aeo4bysagruwwfii6oneh56lj/lib64/libgomp.so.1 > (0x00007f637b9a80 > 00) > libxcb.so.1 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/libxcb-1.13-w3g6d3ulogwz3cxf5brw3ymiw22apkax/lib/libxcb.so.1 > (0x00007f637b780000 > ) > libXau.so.6 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/libxau-1.0.8-kv6fes6sr7z5mu6gkmpanmzx2tmbaroc/lib/libXau.so.6 > (0x00007f637b57c00 > 0) > libXdmcp.so.6 => > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/libxdmcp-1.1.2-pogqqdhqufmv567ch6n4eipdw7wgpdui/lib/libXdmcp.so.6 > (0x00007f637 > b375000) > libudev.so.1 => /lib64/libudev.so.1 (0x00007f637b360000) > libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f637aff6000) > librt.so.1 => /lib64/librt.so.1 (0x00007f637aded000) > libgfortran.so.3 => /lib64/libgfortran.so.3 > (0x00007f637aacb000) > libcap.so.2 => /lib64/libcap.so.2 (0x00007f637a8c5000) > libdw.so.1 => /lib64/libdw.so.1 (0x00007f637a67e000) > liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f637a458000) > libattr.so.1 => /lib64/libattr.so.1 (0x00007f637a252000) > libelf.so.1 => /lib64/libelf.so.1 (0x00007f637a03a000) > libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f6379e2a000) > [sajid@xrm lib]$ pwd > /raid/home/sajid/packages/spack/opt/spack/linux-rhel7-x86_64/gcc-7.3.0/petsc-3.10.3-7fyzhapqnl6qkcuidwugoizr6yt4ujil/lib > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Jed Brown 19 Dec '18

19 Dec '18

Hapla Vaclav via petsc-dev <petsc-dev(a)mcs.anl.gov> writes: > So my overall idea (which I presented also at this year's User Meeting > in London and nobody has objected yet), is that some FE codes could > potentially use only this for both checkpointing and > viewing. Advantages would include removing the redundancy in storage > (reducing the file size) and increased interoperability with 3rd party > tools. This is great in principle, but note that visualization files often contain diagnostic quantities like pressure, temperature, and vorticity while checkpoint files contain only state variables like density, momentum, and total energy. Although one can be derived from the other, it requires an equation of state (an arbitrarily complex constitutive model) and/or compatible derivatives, both of which are so cumbersome to provide to offline visualization that almost nobody has an appetite for it. It certainly makes sense to use the same format that can include arbitrary diagnostic quantities. I think that should be provided by an extensible Viewer filter so the caller (e.g., a TS monitor) can just call VecView and the viewer's configuration determines which diagnostic and/or state quantities to include. Note that Vec holds a reference to its DM so it can determine whether the topology is already present in the output file and link itself appropriately. There should be no assumption that there is only one DM or that it has already been written.

1 0

Re: [petsc-dev] MatNest and inner KSP
by Pierre Jolivet 19 Dec '18

19 Dec '18

> On 19 Dec 2018, at 9:46 AM, Stefano Zampini <stefano.zampini(a)gmail.com> wrote: > > Pierre, > > The details of the inner solvers for PCFIELDSPLIT are not known if you don’t call PCSetUp, so the cascade approach should be preferred. OK, great! It was actually not that big of a deal to implement. Thanks, Pierre > After PCSetUp, you can access the inner solves, see e.g. https://bitbucket.org/petsc/petsc/src/4ddd364d454d7bce974e92e0b5bd4f4516dbf… >> On Dec 19, 2018, at 9:01 AM, Pierre Jolivet via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: >> >> Hello, >> I have a PCFIELDSPLIT where the fields are defined by a MATNEST “decomposition”. >> Right now, I am first defining the KSP of the diagonal blocks of the MATNEST, which are themselves PCFIELDSPLIT, then creating the Mat itself, and eventually I’d like to solve systems with the complete MATNEST. >> However, the KSP and PC options of the “first level" inner solvers are not propagated to the global outer solver. >> Can this be achieved somehow, or do I always need to set the options in the opposite direction (first outer solver, then inner solvers, e.g., through a cascade of PCFieldSplitGetSubKSP)? >> >> Thanks in advance, >> Pierre >

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Václav Hapla 19 Dec '18

19 Dec '18

Thanks Karli! That should be it. Vaclav On 18 December 2018 16:48:30 GMT+01:00, Karl Rupp <rupp(a)iue.tuwien.ac.at> wrote: > > >> >> I have not quickly found how is that "VTK ordering" defined but I >> hopefully it's a well-defined unambiguous cell-local numbering. I >will >> try to find it out soon and get back to you. > >Hope this helps: >https://www.vtk.org/wp-content/uploads/2015/04/file-formats.pdf >(page 9) > >Best regards, >Karli

1 0

Re: [petsc-dev] MatNest and inner KSP
by Stefano Zampini 19 Dec '18

19 Dec '18

Pierre, The details of the inner solvers for PCFIELDSPLIT are not known if you don’t call PCSetUp, so the cascade approach should be preferred. After PCSetUp, you can access the inner solves, see e.g. https://bitbucket.org/petsc/petsc/src/4ddd364d454d7bce974e92e0b5bd4f4516dbf… > On Dec 19, 2018, at 9:01 AM, Pierre Jolivet via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > Hello, > I have a PCFIELDSPLIT where the fields are defined by a MATNEST “decomposition”. > Right now, I am first defining the KSP of the diagonal blocks of the MATNEST, which are themselves PCFIELDSPLIT, then creating the Mat itself, and eventually I’d like to solve systems with the complete MATNEST. > However, the KSP and PC options of the “first level" inner solvers are not propagated to the global outer solver. > Can this be achieved somehow, or do I always need to set the options in the opposite direction (first outer solver, then inner solvers, e.g., through a cascade of PCFieldSplitGetSubKSP)? > > Thanks in advance, > Pierre

1 0

MatNest and inner KSP
by Pierre Jolivet 19 Dec '18

19 Dec '18

Hello, I have a PCFIELDSPLIT where the fields are defined by a MATNEST “decomposition”. Right now, I am first defining the KSP of the diagonal blocks of the MATNEST, which are themselves PCFIELDSPLIT, then creating the Mat itself, and eventually I’d like to solve systems with the complete MATNEST. However, the KSP and PC options of the “first level" inner solvers are not propagated to the global outer solver. Can this be achieved somehow, or do I always need to set the options in the opposite direction (first outer solver, then inner solvers, e.g., through a cascade of PCFieldSplitGetSubKSP)? Thanks in advance, Pierre

1 0

Re: [petsc-dev] [SPAM?] Re: Checkpoint-restart with DMPlex objects
by Karl Rupp 18 Dec '18

18 Dec '18

> > I have not quickly found how is that "VTK ordering" defined but I > hopefully it's a well-defined unambiguous cell-local numbering. I will > try to find it out soon and get back to you. Hope this helps: https://www.vtk.org/wp-content/uploads/2015/04/file-formats.pdf (page 9) Best regards, Karli

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Hapla Vaclav 18 Dec '18

18 Dec '18

On 18 Dec 2018, at 14:30, Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: On Tue, Dec 18, 2018 at 8:28 AM Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: On Tue, Dec 18, 2018 at 6:54 AM Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch<mailto:[email protected]>> wrote: On 17 Dec 2018, at 20:36, Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: On Mon, Dec 17, 2018 at 12:11 PM Lawrence Mitchell <wence(a)gmx.li<mailto:[email protected]>> wrote: > On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch<mailto:[email protected]>> wrote: > > Matt, great that your reminded this email. I actually completely missed it that time. > >> On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: [...] >> I would like: >> >> - To be able to dump the DMPlex, and fields, on N processes >> >> I think the current HDF5 does what you want. >> >> - To be able to load the DMPlex, and fields, on P processes. In the first instance, to get things going, I am happy if P=1. >> >> I think this also works with arbitrary P, although the testing can be described as extremely thin. > > I think we need to be much more precise here. First off, there are now two HDF5 formats: > 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization > 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of vertices and cells > 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for visualization, you perhaps understand it better > > PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I think what Lawrence calls Native should be 1). > > The format 1) is currently written in parallel but loaded sequentially > https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… > > I don't understand, how it can work correctly for a distributed mesh while the Point SF (connecting partitions) is not stored FWICS. I think there's even no PetscSFView_HDF5(). I will check it more deeply soon. > > The format 2) is for which I implemented parallel DMLoad(). > Unfortunately, I can't declare it bulletproof until we declare parallel DMPlexInterpolate() as 100% working. I did quite some work towards it in > https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… > but as stated in the PR summary, there are still some examples failing because of the wrong Point SF which is partly fixed in knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, is there any chance you could look at it at some point in near future? > > I think for Lawrence's purposes, 2) can be used to read the initial mesh file but for checkpointing 1) seems to be better ATM because it dumps everything including interpolated edges & faces, labels and perhaps some more additional information. OK, so I guess there are two different things going on here: 1. Store the data you need to reconstruct a DMPlex 2. Store the data you need to have a DMPlex viewable via XDMF. 3. Store the data you need to reconstruct a DMPlex AND have it viewable via XDMF. For checkpointing only purposes, I only really need 1; for viz purposes, one only needs 2; ideally, one would not separate viz and checkpointing files if there is sufficient overlap of data (I think there is), which needs 3. > I will nevertheless keep on working to improve 2) so that it can store edges & faces & labels in the XDMF-compatible way. > >> >> For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering. >> >> I would use HDF5. >> >> For the field coefficients, I can just VecView(vec). But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh. >> >> Hmm, I will check this right now. If it does not exist, I will write it. > > No, it certainly doesn't exist. There is only ASCII view implemented. > >> >> I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering? >> >> No, all Sections use local points. We do not use global point numbers anywhere in Plex. > > True. DMPlex is partition-wise sequential. The only thing which connects the submeshes is the Point SF. OK, so I think I misunderstood what the dump format looks like then. For parallel store/load cycle when I go from N to P processes what must I do? If I understand correctly the dump on N processes contains: For each process, in process-local numbering - The DMPlex topology on that process Now, given that the only thing that connects these local pieces of the DM together is the point SF, as Vaclav says, it must be the case that a reloadable dump file contains that information. No, the dump contains a completely consistent serial DM. Now I remember why parallel load is not implemented :) We demand that the dump look identical from any number of procs for all PETSc stuff. Thus we get a global renumbering and dump with that for all things. Oh I see. I misunderstood this. More clear to me know - DMPlexCreatePointNumbering() is employed in DMPlexView_HDF5_Internal(). Now, when we load in parallel, we need to use the new parallel loading from Michael. What exactly do you mean? If I remember well Michael implemented a parallel MED loader and it uses DMPlexCreateFromCellListParallel() just as my XDMF-HDF5 reader does. Is this function what you mean by "the new parallel loading"? Yes exactly. You are doing it right. We just need to extend that. Actually, I think we should probably just store an attribute for interpolated meshes, and interpolate on load. This is much simpler, less storage, and makes everything uniform. What do you think? Hmm, the problematic part is the labels. How do we make sure we are labeling the right edge/face? I think we should adopt a cell-local numbering. If we did that the same way as XDMF, it would really make my life easier. Unfortunately, http://www.xdmf.org/index.php/XDMF_Model_and_Format#Attribute only states that "Edge and Face values are defined for each Cell in turn, using VTK ordering." and http://www.xdmf.org/index.php/XDMF_Model_and_Format#Set shows a stupid example which "defines values on Face 0 of Cells 1,2,3 and 4." which doesn't really make clear anything. I have not quickly found how is that "VTK ordering" defined but I hopefully it's a well-defined unambiguous cell-local numbering. I will try to find it out soon and get back to you. Thanks Vaclav Matt Thanks, Matt Thanks Vaclav I have not yet written that, but it should be straightforward :) So the below is not really right. We need to call parallel load for the topology. Then we need code that loads the labels and uses the migration SF to redistribute them, but I think that code already exists for redistribution, so we just hijack it. Matt OK, so to dump a field so that we can reload it we must need: - topology (in local numbering) - point SF (connecting the local pieces of the topology together) - Vector (dofs), presumably in local layout to make things easier - Section describing the vector layout (local numbering) So to load, I do: 1. load and distribute topology, and construct new point SF (this presumably gives me a "migration SF" that maps from old points to new points 2. Broadcast the Section over migration SF so that we know how many dofs belong to each point in the new topology 3. Broadcast the Vec over the migration SF to get the dofs to the right place. Whenever I think of this on paper it seems "easy", but then I occasionally try and sit down and do it and immediately get lost, so I am normally missing something. What am I missing this time? Lawrence -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Matthew Knepley 18 Dec '18

18 Dec '18

On Tue, Dec 18, 2018 at 8:28 AM Matthew Knepley <knepley(a)gmail.com> wrote: > On Tue, Dec 18, 2018 at 6:54 AM Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> > wrote: > >> >> >> On 17 Dec 2018, at 20:36, Matthew Knepley <knepley(a)gmail.com> wrote: >> >> On Mon, Dec 17, 2018 at 12:11 PM Lawrence Mitchell <wence(a)gmx.li> wrote: >> >>> >>> > On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> >>> wrote: >>> > >>> > Matt, great that your reminded this email. I actually completely >>> missed it that time. >>> > >>> >> On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev < >>> petsc-dev(a)mcs.anl.gov> wrote: >>> >>> [...] >>> >>> >> I would like: >>> >> >>> >> - To be able to dump the DMPlex, and fields, on N processes >>> >> >>> >> I think the current HDF5 does what you want. >>> >> >>> >> - To be able to load the DMPlex, and fields, on P processes. In the >>> first instance, to get things going, I am happy if P=1. >>> >> >>> >> I think this also works with arbitrary P, although the testing can be >>> described as extremely thin. >>> > >>> > I think we need to be much more precise here. First off, there are now >>> two HDF5 formats: >>> > 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization >>> > 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of >>> vertices and cells >>> > 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for >>> visualization, you perhaps understand it better >>> > >>> > PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I >>> think what Lawrence calls Native should be 1). >>> > >>> > The format 1) is currently written in parallel but loaded sequentially >>> > >>> https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… >>> > >>> > I don't understand, how it can work correctly for a distributed mesh >>> while the Point SF (connecting partitions) is not stored FWICS. I think >>> there's even no PetscSFView_HDF5(). I will check it more deeply soon. >>> > >>> > The format 2) is for which I implemented parallel DMLoad(). >>> > Unfortunately, I can't declare it bulletproof until we declare >>> parallel DMPlexInterpolate() as 100% working. I did quite some work towards >>> it in >>> > >>> https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… >>> > but as stated in the PR summary, there are still some examples failing >>> because of the wrong Point SF which is partly fixed in >>> knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, >>> is there any chance you could look at it at some point in near future? >>> > >>> > I think for Lawrence's purposes, 2) can be used to read the initial >>> mesh file but for checkpointing 1) seems to be better ATM because it dumps >>> everything including interpolated edges & faces, labels and perhaps some >>> more additional information. >>> >>> OK, so I guess there are two different things going on here: >>> >>> 1. Store the data you need to reconstruct a DMPlex >>> >>> 2. Store the data you need to have a DMPlex viewable via XDMF. >>> >>> 3. Store the data you need to reconstruct a DMPlex AND have it viewable >>> via XDMF. >>> >>> For checkpointing only purposes, I only really need 1; for viz purposes, >>> one only needs 2; ideally, one would not separate viz and checkpointing >>> files if there is sufficient overlap of data (I think there is), which >>> needs 3. >>> >>> > I will nevertheless keep on working to improve 2) so that it can store >>> edges & faces & labels in the XDMF-compatible way. >>> > >>> >> >>> >> For dumping, I think I can do DMView(dm) in PETSc "native" format, >>> and that will write out the topology in a global numbering. >>> >> >>> >> I would use HDF5. >>> >> >>> >> For the field coefficients, I can just VecView(vec). But there does >>> not appear to be any way of saving the Section so that I can actually >>> attach those coefficients to points in the mesh. >>> >> >>> >> Hmm, I will check this right now. If it does not exist, I will write >>> it. >>> > >>> > No, it certainly doesn't exist. There is only ASCII view implemented. >>> > >>> >> >>> >> I can do PetscSectionCreateGlobalSection(section), so that I have a >>> the global numbering for offsets, but presumably for the point numbering, I >>> need to convert the local chart into global point numbers using >>> DMPlexCreatePointNumbering? >>> >> >>> >> No, all Sections use local points. We do not use global point numbers >>> anywhere in Plex. >>> > >>> > True. DMPlex is partition-wise sequential. The only thing which >>> connects the submeshes is the Point SF. >>> >>> OK, so I think I misunderstood what the dump format looks like then. For >>> parallel store/load cycle when I go from N to P processes what must I do? >>> >>> If I understand correctly the dump on N processes contains: >>> >>> For each process, in process-local numbering >>> >>> - The DMPlex topology on that process >>> >>> Now, given that the only thing that connects these local pieces of the >>> DM together is the point SF, as Vaclav says, it must be the case that a >>> reloadable dump file contains that information. >>> >> >> No, the dump contains a completely consistent serial DM. Now I remember >> why parallel load is not implemented :) >> We demand that the dump look identical from any number of procs for all >> PETSc stuff. Thus we get a global renumbering >> and dump with that for all things. >> >> >> Oh I see. I misunderstood this. More clear to me know - >> DMPlexCreatePointNumbering() is employed in DMPlexView_HDF5_Internal(). >> >> >> Now, when we load in parallel, we need to use the new parallel loading >> from Michael. >> >> >> What exactly do you mean? >> If I remember well Michael implemented a parallel MED loader and it uses DMPlexCreateFromCellListParallel() >> just as my XDMF-HDF5 reader does. >> Is this function what you mean by "the new parallel loading"? >> > > Yes exactly. You are doing it right. We just need to extend that. > Actually, I think we should probably just store an attribute for > interpolated meshes, and interpolate on load. This is much simpler, less > storage, and makes everything uniform. What do you think? > Hmm, the problematic part is the labels. How do we make sure we are labeling the right edge/face? Matt > Thanks, > > Matt > > >> Thanks >> >> Vaclav >> >> I have not yet written that, but it should >> be straightforward :) So the below is not really right. We need to call >> parallel load for the topology. Then we need code that >> loads the labels and uses the migration SF to redistribute them, but I >> think that code already exists for redistribution, so we >> just hijack it. >> >> Matt >> >> >>> OK, so to dump a field so that we can reload it we must need: >>> >>> - topology (in local numbering) >>> - point SF (connecting the local pieces of the topology together) >>> - Vector (dofs), presumably in local layout to make things easier >>> - Section describing the vector layout (local numbering) >>> >>> So to load, I do: >>> >>> 1. load and distribute topology, and construct new point SF (this >>> presumably gives me a "migration SF" that maps from old points to new points >>> >>> 2. Broadcast the Section over migration SF so that we know how many dofs >>> belong to each point in the new topology >>> >>> 3. Broadcast the Vec over the migration SF to get the dofs to the right >>> place. >>> >>> Whenever I think of this on paper it seems "easy", but then I >>> occasionally try and sit down and do it and immediately get lost, so I am >>> normally missing something. >>> >>> What am I missing this time? >>> >>> Lawrence >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Matthew Knepley 18 Dec '18

18 Dec '18

On Tue, Dec 18, 2018 at 6:54 AM Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> wrote: > > > On 17 Dec 2018, at 20:36, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Mon, Dec 17, 2018 at 12:11 PM Lawrence Mitchell <wence(a)gmx.li> wrote: > >> >> > On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> >> wrote: >> > >> > Matt, great that your reminded this email. I actually completely missed >> it that time. >> > >> >> On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev < >> petsc-dev(a)mcs.anl.gov> wrote: >> >> [...] >> >> >> I would like: >> >> >> >> - To be able to dump the DMPlex, and fields, on N processes >> >> >> >> I think the current HDF5 does what you want. >> >> >> >> - To be able to load the DMPlex, and fields, on P processes. In the >> first instance, to get things going, I am happy if P=1. >> >> >> >> I think this also works with arbitrary P, although the testing can be >> described as extremely thin. >> > >> > I think we need to be much more precise here. First off, there are now >> two HDF5 formats: >> > 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization >> > 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of >> vertices and cells >> > 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for >> visualization, you perhaps understand it better >> > >> > PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I >> think what Lawrence calls Native should be 1). >> > >> > The format 1) is currently written in parallel but loaded sequentially >> > >> https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… >> > >> > I don't understand, how it can work correctly for a distributed mesh >> while the Point SF (connecting partitions) is not stored FWICS. I think >> there's even no PetscSFView_HDF5(). I will check it more deeply soon. >> > >> > The format 2) is for which I implemented parallel DMLoad(). >> > Unfortunately, I can't declare it bulletproof until we declare parallel >> DMPlexInterpolate() as 100% working. I did quite some work towards it in >> > >> https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… >> > but as stated in the PR summary, there are still some examples failing >> because of the wrong Point SF which is partly fixed in >> knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, >> is there any chance you could look at it at some point in near future? >> > >> > I think for Lawrence's purposes, 2) can be used to read the initial >> mesh file but for checkpointing 1) seems to be better ATM because it dumps >> everything including interpolated edges & faces, labels and perhaps some >> more additional information. >> >> OK, so I guess there are two different things going on here: >> >> 1. Store the data you need to reconstruct a DMPlex >> >> 2. Store the data you need to have a DMPlex viewable via XDMF. >> >> 3. Store the data you need to reconstruct a DMPlex AND have it viewable >> via XDMF. >> >> For checkpointing only purposes, I only really need 1; for viz purposes, >> one only needs 2; ideally, one would not separate viz and checkpointing >> files if there is sufficient overlap of data (I think there is), which >> needs 3. >> >> > I will nevertheless keep on working to improve 2) so that it can store >> edges & faces & labels in the XDMF-compatible way. >> > >> >> >> >> For dumping, I think I can do DMView(dm) in PETSc "native" format, and >> that will write out the topology in a global numbering. >> >> >> >> I would use HDF5. >> >> >> >> For the field coefficients, I can just VecView(vec). But there does >> not appear to be any way of saving the Section so that I can actually >> attach those coefficients to points in the mesh. >> >> >> >> Hmm, I will check this right now. If it does not exist, I will write >> it. >> > >> > No, it certainly doesn't exist. There is only ASCII view implemented. >> > >> >> >> >> I can do PetscSectionCreateGlobalSection(section), so that I have a >> the global numbering for offsets, but presumably for the point numbering, I >> need to convert the local chart into global point numbers using >> DMPlexCreatePointNumbering? >> >> >> >> No, all Sections use local points. We do not use global point numbers >> anywhere in Plex. >> > >> > True. DMPlex is partition-wise sequential. The only thing which >> connects the submeshes is the Point SF. >> >> OK, so I think I misunderstood what the dump format looks like then. For >> parallel store/load cycle when I go from N to P processes what must I do? >> >> If I understand correctly the dump on N processes contains: >> >> For each process, in process-local numbering >> >> - The DMPlex topology on that process >> >> Now, given that the only thing that connects these local pieces of the DM >> together is the point SF, as Vaclav says, it must be the case that a >> reloadable dump file contains that information. >> > > No, the dump contains a completely consistent serial DM. Now I remember > why parallel load is not implemented :) > We demand that the dump look identical from any number of procs for all > PETSc stuff. Thus we get a global renumbering > and dump with that for all things. > > > Oh I see. I misunderstood this. More clear to me know - > DMPlexCreatePointNumbering() is employed in DMPlexView_HDF5_Internal(). > > > Now, when we load in parallel, we need to use the new parallel loading > from Michael. > > > What exactly do you mean? > If I remember well Michael implemented a parallel MED loader and it uses DMPlexCreateFromCellListParallel() > just as my XDMF-HDF5 reader does. > Is this function what you mean by "the new parallel loading"? > Yes exactly. You are doing it right. We just need to extend that. Actually, I think we should probably just store an attribute for interpolated meshes, and interpolate on load. This is much simpler, less storage, and makes everything uniform. What do you think? Thanks, Matt > Thanks > > Vaclav > > I have not yet written that, but it should > be straightforward :) So the below is not really right. We need to call > parallel load for the topology. Then we need code that > loads the labels and uses the migration SF to redistribute them, but I > think that code already exists for redistribution, so we > just hijack it. > > Matt > > >> OK, so to dump a field so that we can reload it we must need: >> >> - topology (in local numbering) >> - point SF (connecting the local pieces of the topology together) >> - Vector (dofs), presumably in local layout to make things easier >> - Section describing the vector layout (local numbering) >> >> So to load, I do: >> >> 1. load and distribute topology, and construct new point SF (this >> presumably gives me a "migration SF" that maps from old points to new points >> >> 2. Broadcast the Section over migration SF so that we know how many dofs >> belong to each point in the new topology >> >> 3. Broadcast the Vec over the migration SF to get the dofs to the right >> place. >> >> Whenever I think of this on paper it seems "easy", but then I >> occasionally try and sit down and do it and immediately get lost, so I am >> normally missing something. >> >> What am I missing this time? >> >> Lawrence >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0