December 2018 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Hapla Vaclav 18 Dec '18

18 Dec '18

On 17 Dec 2018, at 20:36, Matthew Knepley <knepley(a)gmail.com<mailto:[email protected]>> wrote: On Mon, Dec 17, 2018 at 12:11 PM Lawrence Mitchell <wence(a)gmx.li<mailto:[email protected]>> wrote: > On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch<mailto:[email protected]>> wrote: > > Matt, great that your reminded this email. I actually completely missed it that time. > >> On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: [...] >> I would like: >> >> - To be able to dump the DMPlex, and fields, on N processes >> >> I think the current HDF5 does what you want. >> >> - To be able to load the DMPlex, and fields, on P processes. In the first instance, to get things going, I am happy if P=1. >> >> I think this also works with arbitrary P, although the testing can be described as extremely thin. > > I think we need to be much more precise here. First off, there are now two HDF5 formats: > 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization > 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of vertices and cells > 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for visualization, you perhaps understand it better > > PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I think what Lawrence calls Native should be 1). > > The format 1) is currently written in parallel but loaded sequentially > https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… > > I don't understand, how it can work correctly for a distributed mesh while the Point SF (connecting partitions) is not stored FWICS. I think there's even no PetscSFView_HDF5(). I will check it more deeply soon. > > The format 2) is for which I implemented parallel DMLoad(). > Unfortunately, I can't declare it bulletproof until we declare parallel DMPlexInterpolate() as 100% working. I did quite some work towards it in > https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… > but as stated in the PR summary, there are still some examples failing because of the wrong Point SF which is partly fixed in knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, is there any chance you could look at it at some point in near future? > > I think for Lawrence's purposes, 2) can be used to read the initial mesh file but for checkpointing 1) seems to be better ATM because it dumps everything including interpolated edges & faces, labels and perhaps some more additional information. OK, so I guess there are two different things going on here: 1. Store the data you need to reconstruct a DMPlex 2. Store the data you need to have a DMPlex viewable via XDMF. 3. Store the data you need to reconstruct a DMPlex AND have it viewable via XDMF. For checkpointing only purposes, I only really need 1; for viz purposes, one only needs 2; ideally, one would not separate viz and checkpointing files if there is sufficient overlap of data (I think there is), which needs 3. > I will nevertheless keep on working to improve 2) so that it can store edges & faces & labels in the XDMF-compatible way. > >> >> For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering. >> >> I would use HDF5. >> >> For the field coefficients, I can just VecView(vec). But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh. >> >> Hmm, I will check this right now. If it does not exist, I will write it. > > No, it certainly doesn't exist. There is only ASCII view implemented. > >> >> I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering? >> >> No, all Sections use local points. We do not use global point numbers anywhere in Plex. > > True. DMPlex is partition-wise sequential. The only thing which connects the submeshes is the Point SF. OK, so I think I misunderstood what the dump format looks like then. For parallel store/load cycle when I go from N to P processes what must I do? If I understand correctly the dump on N processes contains: For each process, in process-local numbering - The DMPlex topology on that process Now, given that the only thing that connects these local pieces of the DM together is the point SF, as Vaclav says, it must be the case that a reloadable dump file contains that information. No, the dump contains a completely consistent serial DM. Now I remember why parallel load is not implemented :) We demand that the dump look identical from any number of procs for all PETSc stuff. Thus we get a global renumbering and dump with that for all things. Oh I see. I misunderstood this. More clear to me know - DMPlexCreatePointNumbering() is employed in DMPlexView_HDF5_Internal(). Now, when we load in parallel, we need to use the new parallel loading from Michael. What exactly do you mean? If I remember well Michael implemented a parallel MED loader and it uses DMPlexCreateFromCellListParallel() just as my XDMF-HDF5 reader does. Is this function what you mean by "the new parallel loading"? Thanks Vaclav I have not yet written that, but it should be straightforward :) So the below is not really right. We need to call parallel load for the topology. Then we need code that loads the labels and uses the migration SF to redistribute them, but I think that code already exists for redistribution, so we just hijack it. Matt OK, so to dump a field so that we can reload it we must need: - topology (in local numbering) - point SF (connecting the local pieces of the topology together) - Vector (dofs), presumably in local layout to make things easier - Section describing the vector layout (local numbering) So to load, I do: 1. load and distribute topology, and construct new point SF (this presumably gives me a "migration SF" that maps from old points to new points 2. Broadcast the Section over migration SF so that we know how many dofs belong to each point in the new topology 3. Broadcast the Vec over the migration SF to get the dofs to the right place. Whenever I think of this on paper it seems "easy", but then I occasionally try and sit down and do it and immediately get lost, so I am normally missing something. What am I missing this time? Lawrence -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Hapla Vaclav 18 Dec '18

18 Dec '18

On 18 Dec 2018, at 12:16, Hapla Vaclav via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: On 17 Dec 2018, at 18:11, Lawrence Mitchell <wence(a)gmx.li<mailto:[email protected]>> wrote: On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch<mailto:[email protected]>> wrote: Matt, great that your reminded this email. I actually completely missed it that time. On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: [...] I would like: - To be able to dump the DMPlex, and fields, on N processes I think the current HDF5 does what you want. - To be able to load the DMPlex, and fields, on P processes. In the first instance, to get things going, I am happy if P=1. I think this also works with arbitrary P, although the testing can be described as extremely thin. I think we need to be much more precise here. First off, there are now two HDF5 formats: 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of vertices and cells 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for visualization, you perhaps understand it better PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I think what Lawrence calls Native should be 1). The format 1) is currently written in parallel but loaded sequentially https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… I don't understand, how it can work correctly for a distributed mesh while the Point SF (connecting partitions) is not stored FWICS. I think there's even no PetscSFView_HDF5(). I will check it more deeply soon. The format 2) is for which I implemented parallel DMLoad(). Unfortunately, I can't declare it bulletproof until we declare parallel DMPlexInterpolate() as 100% working. I did quite some work towards it in https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… but as stated in the PR summary, there are still some examples failing because of the wrong Point SF which is partly fixed in knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, is there any chance you could look at it at some point in near future? I think for Lawrence's purposes, 2) can be used to read the initial mesh file but for checkpointing 1) seems to be better ATM because it dumps everything including interpolated edges & faces, labels and perhaps some more additional information. OK, so I guess there are two different things going on here: 1. Store the data you need to reconstruct a DMPlex 2. Store the data you need to have a DMPlex viewable via XDMF. 3. Store the data you need to reconstruct a DMPlex AND have it viewable via XDMF. For checkpointing only purposes, I only really need 1; for viz purposes, one only needs 2; ideally, one would not separate viz and checkpointing files if there is sufficient overlap of data (I think there is), which needs 3. Yes. Coordinates can be shared, topology storage is completely different, so with 3, it's just stored in two different ways redundantly. Note I try to explain it's not that sharp that the XDMF format itself is only for viewing. The format itself is virtually capable to store the whole DMPlex representation. As I say, if you create HDF5 viewer and push PETSC_VIEWER_HDF5_XDMF format, DMLoad() loads _just_ the XDMF-related data in parallel (using DMPlexCreateFromCellListParallel()). For example, src/dm/impls/plex/examples/tutorials/ex5.c shows that. The current interface is limited in that it only stores/loads vertices/cells, and edges/faces must be made on-the-fly. However, XDMF itself allows to store also these, so the need to call DMPlexInterpolate() could be eliminated. Sorry, this is not entirely correct. Edges/faces are not explicitly stored but can be referred to using cell-local numbering. So labels or fields can still refer to them. But we can't avoid DMPlexInterpolate() after each load. For me it's not such a problem because parallel DMPlexInterpolate() scales almost perfectly (it's only not entirely correct ATM as I keep reminding). Vaclav There are also means to store labels using XDMF attributes/sets, and I'm planning to implement this in DMPlex. So my overall idea (which I presented also at this year's User Meeting in London and nobody has objected yet), is that some FE codes could potentially use only this for both checkpointing and viewing. Advantages would include removing the redundancy in storage (reducing the file size) and increased interoperability with 3rd party tools. Anyway, as XDMF refers to HDF5 heavy data, one is free to store any XDMF-unrelated data there as well, which is what happens with PETSC_VIEWER_DEFAULT. The "native PETSc" data, the "PETSc-native" topology, is then stored as well. _At the moment_ it's a good idea because of the limitations of the _current_ XDMF-HDF5 I/O implementation in PETSc. Of course, XDMF + corresponding HDF5 part can never be as general as DMPlex itself and the "native PETSc" storage. For instance just because it supports "only" 3 dimensions of a normal world :-) whereas DMPlex support virtually any spatial dimension (a big deal of functionality relies on spatial dimension <= 3, though). Anyone knows what e.g. Fenics uses for checkpointing and/or final viewable datafile? Thanks Vaclav I will nevertheless keep on working to improve 2) so that it can store edges & faces & labels in the XDMF-compatible way. For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering. I would use HDF5. For the field coefficients, I can just VecView(vec). But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh. Hmm, I will check this right now. If it does not exist, I will write it. No, it certainly doesn't exist. There is only ASCII view implemented. I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering? No, all Sections use local points. We do not use global point numbers anywhere in Plex. True. DMPlex is partition-wise sequential. The only thing which connects the submeshes is the Point SF. OK, so I think I misunderstood what the dump format looks like then. For parallel store/load cycle when I go from N to P processes what must I do? If I understand correctly the dump on N processes contains: For each process, in process-local numbering - The DMPlex topology on that process Now, given that the only thing that connects these local pieces of the DM together is the point SF, as Vaclav says, it must be the case that a reloadable dump file contains that information. OK, so to dump a field so that we can reload it we must need: - topology (in local numbering) - point SF (connecting the local pieces of the topology together) - Vector (dofs), presumably in local layout to make things easier - Section describing the vector layout (local numbering) So to load, I do: 1. load and distribute topology, and construct new point SF (this presumably gives me a "migration SF" that maps from old points to new points 2. Broadcast the Section over migration SF so that we know how many dofs belong to each point in the new topology 3. Broadcast the Vec over the migration SF to get the dofs to the right place. Whenever I think of this on paper it seems "easy", but then I occasionally try and sit down and do it and immediately get lost, so I am normally missing something. What am I missing this time? Lawrence

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Hapla Vaclav 18 Dec '18

18 Dec '18

> On 17 Dec 2018, at 18:11, Lawrence Mitchell <wence(a)gmx.li> wrote: > > >> On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> wrote: >> >> Matt, great that your reminded this email. I actually completely missed it that time. >> >>> On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > [...] > >>> I would like: >>> >>> - To be able to dump the DMPlex, and fields, on N processes >>> >>> I think the current HDF5 does what you want. >>> >>> - To be able to load the DMPlex, and fields, on P processes. In the first instance, to get things going, I am happy if P=1. >>> >>> I think this also works with arbitrary P, although the testing can be described as extremely thin. >> >> I think we need to be much more precise here. First off, there are now two HDF5 formats: >> 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization >> 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of vertices and cells >> 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for visualization, you perhaps understand it better >> >> PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I think what Lawrence calls Native should be 1). >> >> The format 1) is currently written in parallel but loaded sequentially >> https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… >> >> I don't understand, how it can work correctly for a distributed mesh while the Point SF (connecting partitions) is not stored FWICS. I think there's even no PetscSFView_HDF5(). I will check it more deeply soon. >> >> The format 2) is for which I implemented parallel DMLoad(). >> Unfortunately, I can't declare it bulletproof until we declare parallel DMPlexInterpolate() as 100% working. I did quite some work towards it in >> https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… >> but as stated in the PR summary, there are still some examples failing because of the wrong Point SF which is partly fixed in knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, is there any chance you could look at it at some point in near future? >> >> I think for Lawrence's purposes, 2) can be used to read the initial mesh file but for checkpointing 1) seems to be better ATM because it dumps everything including interpolated edges & faces, labels and perhaps some more additional information. > > OK, so I guess there are two different things going on here: > > 1. Store the data you need to reconstruct a DMPlex > > 2. Store the data you need to have a DMPlex viewable via XDMF. > > 3. Store the data you need to reconstruct a DMPlex AND have it viewable via XDMF. > > For checkpointing only purposes, I only really need 1; for viz purposes, one only needs 2; ideally, one would not separate viz and checkpointing files if there is sufficient overlap of data (I think there is), which needs 3. Yes. Coordinates can be shared, topology storage is completely different, so with 3, it's just stored in two different ways redundantly. Note I try to explain it's not that sharp that the XDMF format itself is only for viewing. The format itself is virtually capable to store the whole DMPlex representation. As I say, if you create HDF5 viewer and push PETSC_VIEWER_HDF5_XDMF format, DMLoad() loads _just_ the XDMF-related data in parallel (using DMPlexCreateFromCellListParallel()). For example, src/dm/impls/plex/examples/tutorials/ex5.c shows that. The current interface is limited in that it only stores/loads vertices/cells, and edges/faces must be made on-the-fly. However, XDMF itself allows to store also these, so the need to call DMPlexInterpolate() could be eliminated. There are also means to store labels using XDMF attributes/sets, and I'm planning to implement this in DMPlex. So my overall idea (which I presented also at this year's User Meeting in London and nobody has objected yet), is that some FE codes could potentially use only this for both checkpointing and viewing. Advantages would include removing the redundancy in storage (reducing the file size) and increased interoperability with 3rd party tools. Anyway, as XDMF refers to HDF5 heavy data, one is free to store any XDMF-unrelated data there as well, which is what happens with PETSC_VIEWER_DEFAULT. The "native PETSc" data, the "PETSc-native" topology, is then stored as well. _At the moment_ it's a good idea because of the limitations of the _current_ XDMF-HDF5 I/O implementation in PETSc. Of course, XDMF + corresponding HDF5 part can never be as general as DMPlex itself and the "native PETSc" storage. For instance just because it supports "only" 3 dimensions of a normal world :-) whereas DMPlex support virtually any spatial dimension (a big deal of functionality relies on spatial dimension <= 3, though). Anyone knows what e.g. Fenics uses for checkpointing and/or final viewable datafile? Thanks Vaclav > >> I will nevertheless keep on working to improve 2) so that it can store edges & faces & labels in the XDMF-compatible way. >> >>> >>> For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering. >>> >>> I would use HDF5. >>> >>> For the field coefficients, I can just VecView(vec). But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh. >>> >>> Hmm, I will check this right now. If it does not exist, I will write it. >> >> No, it certainly doesn't exist. There is only ASCII view implemented. >> >>> >>> I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering? >>> >>> No, all Sections use local points. We do not use global point numbers anywhere in Plex. >> >> True. DMPlex is partition-wise sequential. The only thing which connects the submeshes is the Point SF. > > OK, so I think I misunderstood what the dump format looks like then. For parallel store/load cycle when I go from N to P processes what must I do? > > If I understand correctly the dump on N processes contains: > > For each process, in process-local numbering > > - The DMPlex topology on that process > > Now, given that the only thing that connects these local pieces of the DM together is the point SF, as Vaclav says, it must be the case that a reloadable dump file contains that information. > > > OK, so to dump a field so that we can reload it we must need: > > - topology (in local numbering) > - point SF (connecting the local pieces of the topology together) > - Vector (dofs), presumably in local layout to make things easier > - Section describing the vector layout (local numbering) > > So to load, I do: > > 1. load and distribute topology, and construct new point SF (this presumably gives me a "migration SF" that maps from old points to new points > > 2. Broadcast the Section over migration SF so that we know how many dofs belong to each point in the new topology > > 3. Broadcast the Vec over the migration SF to get the dofs to the right place. > > Whenever I think of this on paper it seems "easy", but then I occasionally try and sit down and do it and immediately get lost, so I am normally missing something. > > What am I missing this time? > > Lawrence >

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Matthew Knepley 17 Dec '18

17 Dec '18

On Mon, Dec 17, 2018 at 12:11 PM Lawrence Mitchell <wence(a)gmx.li> wrote: > > > On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> > wrote: > > > > Matt, great that your reminded this email. I actually completely missed > it that time. > > > >> On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev < > petsc-dev(a)mcs.anl.gov> wrote: > > [...] > > >> I would like: > >> > >> - To be able to dump the DMPlex, and fields, on N processes > >> > >> I think the current HDF5 does what you want. > >> > >> - To be able to load the DMPlex, and fields, on P processes. In the > first instance, to get things going, I am happy if P=1. > >> > >> I think this also works with arbitrary P, although the testing can be > described as extremely thin. > > > > I think we need to be much more precise here. First off, there are now > two HDF5 formats: > > 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization > > 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of > vertices and cells > > 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for > visualization, you perhaps understand it better > > > > PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I > think what Lawrence calls Native should be 1). > > > > The format 1) is currently written in parallel but loaded sequentially > > > https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… > > > > I don't understand, how it can work correctly for a distributed mesh > while the Point SF (connecting partitions) is not stored FWICS. I think > there's even no PetscSFView_HDF5(). I will check it more deeply soon. > > > > The format 2) is for which I implemented parallel DMLoad(). > > Unfortunately, I can't declare it bulletproof until we declare parallel > DMPlexInterpolate() as 100% working. I did quite some work towards it in > > > https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… > > but as stated in the PR summary, there are still some examples failing > because of the wrong Point SF which is partly fixed in > knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, > is there any chance you could look at it at some point in near future? > > > > I think for Lawrence's purposes, 2) can be used to read the initial mesh > file but for checkpointing 1) seems to be better ATM because it dumps > everything including interpolated edges & faces, labels and perhaps some > more additional information. > > OK, so I guess there are two different things going on here: > > 1. Store the data you need to reconstruct a DMPlex > > 2. Store the data you need to have a DMPlex viewable via XDMF. > > 3. Store the data you need to reconstruct a DMPlex AND have it viewable > via XDMF. > > For checkpointing only purposes, I only really need 1; for viz purposes, > one only needs 2; ideally, one would not separate viz and checkpointing > files if there is sufficient overlap of data (I think there is), which > needs 3. > > > I will nevertheless keep on working to improve 2) so that it can store > edges & faces & labels in the XDMF-compatible way. > > > >> > >> For dumping, I think I can do DMView(dm) in PETSc "native" format, and > that will write out the topology in a global numbering. > >> > >> I would use HDF5. > >> > >> For the field coefficients, I can just VecView(vec). But there does > not appear to be any way of saving the Section so that I can actually > attach those coefficients to points in the mesh. > >> > >> Hmm, I will check this right now. If it does not exist, I will write it. > > > > No, it certainly doesn't exist. There is only ASCII view implemented. > > > >> > >> I can do PetscSectionCreateGlobalSection(section), so that I have a the > global numbering for offsets, but presumably for the point numbering, I > need to convert the local chart into global point numbers using > DMPlexCreatePointNumbering? > >> > >> No, all Sections use local points. We do not use global point numbers > anywhere in Plex. > > > > True. DMPlex is partition-wise sequential. The only thing which connects > the submeshes is the Point SF. > > OK, so I think I misunderstood what the dump format looks like then. For > parallel store/load cycle when I go from N to P processes what must I do? > > If I understand correctly the dump on N processes contains: > > For each process, in process-local numbering > > - The DMPlex topology on that process > > Now, given that the only thing that connects these local pieces of the DM > together is the point SF, as Vaclav says, it must be the case that a > reloadable dump file contains that information. > No, the dump contains a completely consistent serial DM. Now I remember why parallel load is not implemented :) We demand that the dump look identical from any number of procs for all PETSc stuff. Thus we get a global renumbering and dump with that for all things. Now, when we load in parallel, we need to use the new parallel loading from Michael. I have not yet written that, but it should be straightforward :) So the below is not really right. We need to call parallel load for the topology. Then we need code that loads the labels and uses the migration SF to redistribute them, but I think that code already exists for redistribution, so we just hijack it. Matt > OK, so to dump a field so that we can reload it we must need: > > - topology (in local numbering) > - point SF (connecting the local pieces of the topology together) > - Vector (dofs), presumably in local layout to make things easier > - Section describing the vector layout (local numbering) > > So to load, I do: > > 1. load and distribute topology, and construct new point SF (this > presumably gives me a "migration SF" that maps from old points to new points > > 2. Broadcast the Section over migration SF so that we know how many dofs > belong to each point in the new topology > > 3. Broadcast the Vec over the migration SF to get the dofs to the right > place. > > Whenever I think of this on paper it seems "easy", but then I occasionally > try and sit down and do it and immediately get lost, so I am normally > missing something. > > What am I missing this time? > > Lawrence > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] GPU web page out of date
by Smith, Barry F. 17 Dec '18

17 Dec '18

Fixed the link in master. > On Dec 17, 2018, at 5:15 AM, Mark Adams via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: > > The GPU web page looks like it is 8 years old ... this link is dead (but ex47cu seems to be in the repo): > > • Example that uses CUDA directly in the user function evaluation > Thanks, > Mark

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Lawrence Mitchell 17 Dec '18

17 Dec '18

> On 17 Dec 2018, at 11:56, Hapla Vaclav <vaclav.hapla(a)erdw.ethz.ch> wrote: > > Matt, great that your reminded this email. I actually completely missed it that time. > >> On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev <petsc-dev(a)mcs.anl.gov> wrote: [...] >> I would like: >> >> - To be able to dump the DMPlex, and fields, on N processes >> >> I think the current HDF5 does what you want. >> >> - To be able to load the DMPlex, and fields, on P processes. In the first instance, to get things going, I am happy if P=1. >> >> I think this also works with arbitrary P, although the testing can be described as extremely thin. > > I think we need to be much more precise here. First off, there are now two HDF5 formats: > 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization > 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of vertices and cells > 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for visualization, you perhaps understand it better > > PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I think what Lawrence calls Native should be 1). > > The format 1) is currently written in parallel but loaded sequentially > https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… > > I don't understand, how it can work correctly for a distributed mesh while the Point SF (connecting partitions) is not stored FWICS. I think there's even no PetscSFView_HDF5(). I will check it more deeply soon. > > The format 2) is for which I implemented parallel DMLoad(). > Unfortunately, I can't declare it bulletproof until we declare parallel DMPlexInterpolate() as 100% working. I did quite some work towards it in > https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… > but as stated in the PR summary, there are still some examples failing because of the wrong Point SF which is partly fixed in knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, is there any chance you could look at it at some point in near future? > > I think for Lawrence's purposes, 2) can be used to read the initial mesh file but for checkpointing 1) seems to be better ATM because it dumps everything including interpolated edges & faces, labels and perhaps some more additional information. OK, so I guess there are two different things going on here: 1. Store the data you need to reconstruct a DMPlex 2. Store the data you need to have a DMPlex viewable via XDMF. 3. Store the data you need to reconstruct a DMPlex AND have it viewable via XDMF. For checkpointing only purposes, I only really need 1; for viz purposes, one only needs 2; ideally, one would not separate viz and checkpointing files if there is sufficient overlap of data (I think there is), which needs 3. > I will nevertheless keep on working to improve 2) so that it can store edges & faces & labels in the XDMF-compatible way. > >> >> For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering. >> >> I would use HDF5. >> >> For the field coefficients, I can just VecView(vec). But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh. >> >> Hmm, I will check this right now. If it does not exist, I will write it. > > No, it certainly doesn't exist. There is only ASCII view implemented. > >> >> I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering? >> >> No, all Sections use local points. We do not use global point numbers anywhere in Plex. > > True. DMPlex is partition-wise sequential. The only thing which connects the submeshes is the Point SF. OK, so I think I misunderstood what the dump format looks like then. For parallel store/load cycle when I go from N to P processes what must I do? If I understand correctly the dump on N processes contains: For each process, in process-local numbering - The DMPlex topology on that process Now, given that the only thing that connects these local pieces of the DM together is the point SF, as Vaclav says, it must be the case that a reloadable dump file contains that information. OK, so to dump a field so that we can reload it we must need: - topology (in local numbering) - point SF (connecting the local pieces of the topology together) - Vector (dofs), presumably in local layout to make things easier - Section describing the vector layout (local numbering) So to load, I do: 1. load and distribute topology, and construct new point SF (this presumably gives me a "migration SF" that maps from old points to new points 2. Broadcast the Section over migration SF so that we know how many dofs belong to each point in the new topology 3. Broadcast the Vec over the migration SF to get the dofs to the right place. Whenever I think of this on paper it seems "easy", but then I occasionally try and sit down and do it and immediately get lost, so I am normally missing something. What am I missing this time? Lawrence

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Lawrence Mitchell 17 Dec '18

17 Dec '18

Hi Matt, all, > On 14 Dec 2018, at 19:54, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Fri, Jul 20, 2018 at 5:34 AM Lawrence Mitchell <wence(a)gmx.li> wrote: > Dear petsc-dev, > > I'm once again revisiting doing "proper" checkpoint-restart cycles. I would like to leverage the existing PETSc stuff for this as much as possible, but I am a bit lost as to what is implemented, and what is missing. > > I have: > > - A (distributed) DMPlex defining the topology > > - Some number of fields defined on this topology, each described by: > > - a Section (with a point permutation set) > - a Vec of coefficients > - Some extra information that describes what the coefficients mean, but let's assume I know how to handle that. > > (Aside, for Vecs with a block size > 1, I actually have a section that indexes the blocks, which probably means I need to unroll into an unblocked version first). > > Sections usually are {point --> {dofs}}. Not sure how it uses blocks instead of points Note that I use sections to construct a function space numbering, but I don't use them directly in any assembly. So I have some extra information that the Section doesn't have. IOW, I map {point -> {# nodes}} and then separately say "there are 3 dofs at each node". Hence the section for P1 and Vector(P1) that I get from PETSc are identical. We can argue about whether or not this is the right choice, but it's easy to unroll. [...] Elided, local or global numbering for Sections/DMPlex objects. > Hmm, I have to think a little bit more. Where are you at with this now? Exactly where I was in July when I wrote the email (which I had forgotten I wrote!). Thanks, Lawrence

1 0

Re: [petsc-dev] GPU web page out of date
by Matthew Knepley 17 Dec '18

17 Dec '18

You can kill that on the webpage. Thats not how I would do it anymore. Matt On Mon, Dec 17, 2018 at 6:18 AM Mark Adams via petsc-dev < petsc-dev(a)mcs.anl.gov> wrote: > The GPU web page looks like it is 8 years old ... this link is dead (but > ex47cu seems to be in the repo): > > > - Example that uses CUDA directly in the user function evaluation > <http://www.mcs.anl.gov/petsc-master/src/snes/examples/tutorials/ex47cu.cu.h…> > > Thanks, > Mark > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

GPU web page out of date
by Mark Adams 17 Dec '18

17 Dec '18

The GPU web page looks like it is 8 years old ... this link is dead (but ex47cu seems to be in the repo): - Example that uses CUDA directly in the user function evaluation <http://www.mcs.anl.gov/petsc-master/src/snes/examples/tutorials/ex47cu.cu.h…> Thanks, Mark

1 0

Re: [petsc-dev] Checkpoint-restart with DMPlex objects
by Hapla Vaclav 17 Dec '18

17 Dec '18

Matt, great that your reminded this email. I actually completely missed it that time. On 14 Dec 2018, at 19:54, Matthew Knepley via petsc-dev <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>> wrote: On Fri, Jul 20, 2018 at 5:34 AM Lawrence Mitchell <wence(a)gmx.li<mailto:[email protected]>> wrote: Dear petsc-dev, I'm once again revisiting doing "proper" checkpoint-restart cycles. I would like to leverage the existing PETSc stuff for this as much as possible, but I am a bit lost as to what is implemented, and what is missing. I have: - A (distributed) DMPlex defining the topology - Some number of fields defined on this topology, each described by: - a Section (with a point permutation set) - a Vec of coefficients - Some extra information that describes what the coefficients mean, but let's assume I know how to handle that. (Aside, for Vecs with a block size > 1, I actually have a section that indexes the blocks, which probably means I need to unroll into an unblocked version first). Sections usually are {point --> {dofs}}. Not sure how it uses blocks instead of points. I would like: - To be able to dump the DMPlex, and fields, on N processes I think the current HDF5 does what you want. - To be able to load the DMPlex, and fields, on P processes. In the first instance, to get things going, I am happy if P=1. I think this also works with arbitrary P, although the testing can be described as extremely thin. I think we need to be much more precise here. First off, there are now two HDF5 formats: 1) PETSC_VIEWER_HDF5_PETSC - store Plex graph serialization 2) PETSC_VIEWER_HDF5_XDMF - store XDMF-compatible representation of vertices and cells 3) PETSC_VIEWER_HDF5_VIZ slightly extends 2) with some stuff for visualization, you perhaps understand it better PETSC_VIEWER_DEFAULT/PETSC_VIEWER_NATIVE mean store all three above. I think what Lawrence calls Native should be 1). The format 1) is currently written in parallel but loaded sequentially https://bitbucket.org/petsc/petsc/src/fbb1886742ac2bbe3b4d1df09bff9724d3fee… I don't understand, how it can work correctly for a distributed mesh while the Point SF (connecting partitions) is not stored FWICS. I think there's even no PetscSFView_HDF5(). I will check it more deeply soon. The format 2) is for which I implemented parallel DMLoad(). Unfortunately, I can't declare it bulletproof until we declare parallel DMPlexInterpolate() as 100% working. I did quite some work towards it in https://bitbucket.org/petsc/petsc/pull-requests/1227/dmplexintepolate-fix-o… but as stated in the PR summary, there are still some examples failing because of the wrong Point SF which is partly fixed in knepley/fix-plex-interpolate-sf but it seems it's not yet finished. Matt, is there any chance you could look at it at some point in near future? I think for Lawrence's purposes, 2) can be used to read the initial mesh file but for checkpointing 1) seems to be better ATM because it dumps everything including interpolated edges & faces, labels and perhaps some more additional information. I will nevertheless keep on working to improve 2) so that it can store edges & faces & labels in the XDMF-compatible way. For dumping, I think I can do DMView(dm) in PETSc "native" format, and that will write out the topology in a global numbering. I would use HDF5. For the field coefficients, I can just VecView(vec). But there does not appear to be any way of saving the Section so that I can actually attach those coefficients to points in the mesh. Hmm, I will check this right now. If it does not exist, I will write it. No, it certainly doesn't exist. There is only ASCII view implemented. I can do PetscSectionCreateGlobalSection(section), so that I have a the global numbering for offsets, but presumably for the point numbering, I need to convert the local chart into global point numbers using DMPlexCreatePointNumbering? No, all Sections use local points. We do not use global point numbers anywhere in Plex. True. DMPlex is partition-wise sequential. The only thing which connects the submeshes is the Point SF. For load, I can do DMLoad(dm), which only loads on rank-0 for now. I do not think that is true. Actually it is true if we talk about PETSC_VIEWER_HDF5_PETSC. Then VecLoad for the coefficients, and (presumably) a putative PetscSectionLoad so that I can associate coefficients to points in the topology. Okay, I am remembering more now. I just use the PetscDS to automatically create the Section, and pray that it matches the Vec that we saved. This is a real design limitation, since the Section is associated with the DM not the Vec, you have to assume all stored Vecs were created with the default Section in the DM. Now, if this is true, then you could just load the Section, set it as the default for the DM, and load the Vec. So it feels like the crucial part of this is a native (HDF5) based viewing for PetscSection. Yep. I thought this worked, but maybe I only decided it should work. For each section (I'm ignoring fields, because those are just sections anyway), there are two explicit, and one implicit, pieces of information: 1. The number of dofs per point Yep. 2. The offset of the dof (this is the global numbering) No, you should store the local Section. You use the SF for the DM to automatically get the global section. True, see above. 3 (implicit). The association of the local chart numbering to the global chart. Above. Saving the first two is easy, how best to save the last so that I can load easily. My thought is to, for each point in the section chart, save a three-tuple: (global-point-number, dofs-per-point, global-dof-number) Hmm, I have to think a little bit more. Where are you at with this now? Matt Then, I can easily read this data in and correctly map from the on-disk coefficients into any section I might have built on the newly loaded DM. Does this all sound reasonable? Or have I missed either an existing implementation, or other issues? Cheers, Lawrence I think there should be no big problem implementing PetscSection{View,Load}_HDF5(). I would suggest to do make use of ISView() to store the integer arrays (as I did with matrices). The same for PetscSF. Note it seems to me XDMF could possibly store also those coefficients (mapped by PetscSection on PETSc side) http://www.xdmf.org/index.php/XDMF_Model_and_Format#Attribute I would be thankful for feedback on this. Vaclav -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>

1 0