June 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 25 Jun '21

25 Jun '21

I am not sure how to proceed. What I think is the current recommended practice of setting the sequence number that I usei n Landau/ex1 (src/ts/utils/dmplexlandau/tutorials): ierr = DMSetOutputSequenceNumber(dm, 0, time);CHKERRQ(ierr); ierr = VecViewFromOptions(X,NULL,"-vec_view");CHKERRQ(ierr); .... ierr = DMSetOutputSequenceNumber(dm, 1, time);CHKERRQ(ierr); ierr = VecViewFromOptions(X,NULL,"-vec_view");CHKERRQ(ierr); Does not work after d7dd068b. Only the last View's data is in the .h5 file. The example in this commit (ex19) is hardwired for HDF5 and ex1 is not. Ex1 is tiny (40 lines) I assume we don't want to abandon the abstract interface and this idea of letting the Viewer keep track of the sequence number is a great idea. So I'm not sure how to proceed. We could a) abandon the abstract interface and I will move to HDF5, b) someone fix this, or c) extend/abstract this idea to not hardwire for HDF5. (c) would be ideal and I could see making it work for HDF5 (only) for now and fix other viewers later. I could then move my ex1 and ex2 to this and just use HDF5. Thanks, Mark On Thu, Jun 24, 2021 at 5:30 PM Mark Adams <mfadams(a)lbl.gov> wrote: > OK, I see that this change is aimed at _not_ having the application keep > track of the time step and allow for incrementing the step number. > That is definitely a better design. > I will change these two tests to use this new system. > ex19 uses a Viewer. I would prefer the ViewFromOptions method. Not sure > this works. > > Thanks, > Mark > > On Thu, Jun 24, 2021 at 5:19 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> >> >> On Thu, Jun 24, 2021 at 3:05 PM Matthew Knepley <knepley(a)gmail.com> >> wrote: >> >>> On Thu, Jun 24, 2021 at 9:58 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> OK, I tried again and deleted the arch directory every time: >>>> >>> >>> Okay, this changes the interface for OutputSequenceNumber. It _should_ >>> be backwards compatible. The likely error I would think >>> is that it thinks you have a timestep at first but not later, however >>> that should be an HDF5 error. Maybe it think the timestep is always >>> the same and overwrites? >>> >> >> * The symptom seems to be that there is only one time step in the file >> and it is the last one. So yes, it seems to overwrite one file location >> over and over. >> >> * I commented out the 2nd View call in ex1 and the file was only 8 bytes >> smaller (eg, the time step). The .xmf file was a bit under 2x bigger. >> >> * Visit "sees" all the time steps. That is, I can step through or run a >> movie of all the time steps, but the data is exactly the same at all steps. >> I've run longer problems with ex2, for each time step. >> >> Mark >> >> >> >> >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> (base) 09:56 (84933e400a...)|BISECTING ~/Codes/petsc2$ git bisect good >>>> d7dd068b66a8daa3a37c9e1556b34bd8def54922 is the first bad commit >>>> commit d7dd068b66a8daa3a37c9e1556b34bd8def54922 >>>> Author: Vaclav Hapla <vaclav.hapla(a)erdw.ethz.ch> >>>> Date: Wed Apr 14 21:22:37 2021 +0200 >>>> >>>> HDF5: Improve timestepping. >>>> >>>> * add PetscViewerHDF5{Push,Pop,Is}Timestepping to control >>>> timestepping mode >>>> * write timestepping attribute for datasets and check it on reading >>>> * fail gracefully if trying to read non-timestepped dataset in >>>> timestepping mode and vice-versa (fix #425) >>>> * rewrite src/vec/vec/tutorials/ex19.c to improve coverage for >>>> timestepping testing >>>> >>>> doc/documentation/changes/dev.rst | 19 ++- >>>> include/petsc/private/viewerhdf5impl.h | 2 + >>>> include/petscviewerhdf5.h | 4 + >>>> src/dm/impls/da/gr2.c | 21 ++-- >>>> src/dm/impls/plex/plexhdf5.c | 25 +++- >>>> src/sys/classes/viewer/impls/hdf5/hdf5v.c | 124 ++++++++++++++++++-- >>>> src/vec/is/is/impls/general/general.c | 10 +- >>>> src/vec/is/utils/hdf5io.c | 45 +++++--- >>>> src/vec/vec/impls/mpi/pdvec.c | 14 ++- >>>> src/vec/vec/tutorials/ex19.c | 186 >>>> +++++++++++++++++------------- >>>> src/vec/vec/tutorials/output/ex19_2.out | 0 >>>> src/vec/vec/tutorials/output/ex19_3.out | 0 >>>> src/vec/vec/tutorials/output/ex19_4.out | 0 >>>> 13 files changed, 320 insertions(+), 130 deletions(-) >>>> delete mode 100644 src/vec/vec/tutorials/output/ex19_2.out >>>> delete mode 100644 src/vec/vec/tutorials/output/ex19_3.out >>>> delete mode 100644 src/vec/vec/tutorials/output/ex19_4.out >>>> >>>> On Thu, Jun 24, 2021 at 7:04 AM Matthew Knepley <knepley(a)gmail.com> >>>> wrote: >>>> >>>>> On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> OK, this is what I get with bisect: >>>>>> >>>>>> (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good >>>>>> The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. >>>>>> This means the bug has been fixed between >>>>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and >>>>>> [09da24df01e50defd94bc4f7396f866a808ecea5 >>>>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. >>>>>> >>>>>> (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout >>>>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 >>>>>> Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 >>>>>> HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' >>>>>> into 'release' >>>>>> (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ >>>>>> >>>>> >>>>> I do not understand. Neither of those commits do anything. >>>>> >>>>> Matt >>>>> >>>>> >>>>>> On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang < >>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>> >>>>>>> Mark, >>>>>>> I am not sure what your problem is. If it is a regression, can >>>>>>> you bisect it? >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> I also tried commenting out the second VecView, so there is just >>>>>>>> one step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>>>>>>> file goes from 5373 bytes to 3090 bytes. >>>>>>>> >>>>>>>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>>> >>>>>>>>> It is not a device issue but it is a regression. >>>>>>>>> >>>>>>>>> Landau ex1 is tiny and just calls VecView before and after the >>>>>>>>> TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 >>>>>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 >>>>>>>>> (see below), you get an h5 file with two time steps, as it should be. >>>>>>>>> This is a huge electric field, Ez=10, which makes the electron >>>>>>>>> distribution (u_e) get visibly pulled off center. >>>>>>>>> In Visit, both time steps have identical data that is >>>>>>>>> clearly after the solve and not the initial condition (see attached). >>>>>>>>> >>>>>>>>> I ran this again with -ex1_ts_max_steps 0 and get the expected >>>>>>>>> result of two steps/frames with the symmetric initial condition in both. >>>>>>>>> THis is correct behavior. >>>>>>>>> >>>>>>>>> Any ideas? >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>>> index 9e4c8f1b61..31dfda2fad 100644 >>>>>>>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>>>>>>> test: >>>>>>>>> suffix: 0 >>>>>>>>> requires: p4est !complex >>>>>>>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>>>>>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>>>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>>>>>>> >>>>>>>>> TEST*/ >>>>>>>>> >>>>>>>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Landau ex1 should work. I will test. >>>>>>>>>> >>>>>>>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley < >>>>>>>>>> knepley(a)gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to >>>>>>>>>>>> the vector array. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I think Mark is saying that those are not working. We do call >>>>>>>>>>> VecGetArrayRead() in the HDF5 code. >>>>>>>>>>> >>>>>>>>>>> Mark, it seem like a small broken code is necessary. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> --Junchao Zhang >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType >>>>>>>>>>>>> (one has a pointer to the other). >>>>>>>>>>>>> >>>>>>>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken >>>>>>>>>>>>> for devices. >>>>>>>>>>>>> >>>>>>>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>>>>>>> >>>>>>>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer >>>>>>>>>>>>> when "mtype==NULL"? >>>>>>>>>>>>> >>>>>>>>>>>>> Mark >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>>>> >>>>>>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 24 Jun '21

24 Jun '21

OK, I see that this change is aimed at _not_ having the application keep track of the time step and allow for incrementing the step number. That is definitely a better design. I will change these two tests to use this new system. ex19 uses a Viewer. I would prefer the ViewFromOptions method. Not sure this works. Thanks, Mark On Thu, Jun 24, 2021 at 5:19 PM Mark Adams <mfadams(a)lbl.gov> wrote: > > > On Thu, Jun 24, 2021 at 3:05 PM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Thu, Jun 24, 2021 at 9:58 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> OK, I tried again and deleted the arch directory every time: >>> >> >> Okay, this changes the interface for OutputSequenceNumber. It _should_ be >> backwards compatible. The likely error I would think >> is that it thinks you have a timestep at first but not later, however >> that should be an HDF5 error. Maybe it think the timestep is always >> the same and overwrites? >> > > * The symptom seems to be that there is only one time step in the file and > it is the last one. So yes, it seems to overwrite one file location over > and over. > > * I commented out the 2nd View call in ex1 and the file was only 8 bytes > smaller (eg, the time step). The .xmf file was a bit under 2x bigger. > > * Visit "sees" all the time steps. That is, I can step through or run a > movie of all the time steps, but the data is exactly the same at all steps. > I've run longer problems with ex2, for each time step. > > Mark > > > > >> >> Thanks, >> >> Matt >> >> >>> (base) 09:56 (84933e400a...)|BISECTING ~/Codes/petsc2$ git bisect good >>> d7dd068b66a8daa3a37c9e1556b34bd8def54922 is the first bad commit >>> commit d7dd068b66a8daa3a37c9e1556b34bd8def54922 >>> Author: Vaclav Hapla <vaclav.hapla(a)erdw.ethz.ch> >>> Date: Wed Apr 14 21:22:37 2021 +0200 >>> >>> HDF5: Improve timestepping. >>> >>> * add PetscViewerHDF5{Push,Pop,Is}Timestepping to control >>> timestepping mode >>> * write timestepping attribute for datasets and check it on reading >>> * fail gracefully if trying to read non-timestepped dataset in >>> timestepping mode and vice-versa (fix #425) >>> * rewrite src/vec/vec/tutorials/ex19.c to improve coverage for >>> timestepping testing >>> >>> doc/documentation/changes/dev.rst | 19 ++- >>> include/petsc/private/viewerhdf5impl.h | 2 + >>> include/petscviewerhdf5.h | 4 + >>> src/dm/impls/da/gr2.c | 21 ++-- >>> src/dm/impls/plex/plexhdf5.c | 25 +++- >>> src/sys/classes/viewer/impls/hdf5/hdf5v.c | 124 ++++++++++++++++++-- >>> src/vec/is/is/impls/general/general.c | 10 +- >>> src/vec/is/utils/hdf5io.c | 45 +++++--- >>> src/vec/vec/impls/mpi/pdvec.c | 14 ++- >>> src/vec/vec/tutorials/ex19.c | 186 >>> +++++++++++++++++------------- >>> src/vec/vec/tutorials/output/ex19_2.out | 0 >>> src/vec/vec/tutorials/output/ex19_3.out | 0 >>> src/vec/vec/tutorials/output/ex19_4.out | 0 >>> 13 files changed, 320 insertions(+), 130 deletions(-) >>> delete mode 100644 src/vec/vec/tutorials/output/ex19_2.out >>> delete mode 100644 src/vec/vec/tutorials/output/ex19_3.out >>> delete mode 100644 src/vec/vec/tutorials/output/ex19_4.out >>> >>> On Thu, Jun 24, 2021 at 7:04 AM Matthew Knepley <knepley(a)gmail.com> >>> wrote: >>> >>>> On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> OK, this is what I get with bisect: >>>>> >>>>> (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good >>>>> The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. >>>>> This means the bug has been fixed between >>>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and >>>>> [09da24df01e50defd94bc4f7396f866a808ecea5 >>>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. >>>>> >>>>> (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout >>>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 >>>>> Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 >>>>> HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into >>>>> 'release' >>>>> (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ >>>>> >>>> >>>> I do not understand. Neither of those commits do anything. >>>> >>>> Matt >>>> >>>> >>>>> On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang < >>>>> junchao.zhang(a)gmail.com> wrote: >>>>> >>>>>> Mark, >>>>>> I am not sure what your problem is. If it is a regression, can you >>>>>> bisect it? >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> I also tried commenting out the second VecView, so there is just one >>>>>>> step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>>>>>> file goes from 5373 bytes to 3090 bytes. >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> It is not a device issue but it is a regression. >>>>>>>> >>>>>>>> Landau ex1 is tiny and just calls VecView before and after the >>>>>>>> TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 >>>>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see >>>>>>>> below), you get an h5 file with two time steps, as it should be. >>>>>>>> This is a huge electric field, Ez=10, which makes the electron >>>>>>>> distribution (u_e) get visibly pulled off center. >>>>>>>> In Visit, both time steps have identical data that is clearly after >>>>>>>> the solve and not the initial condition (see attached). >>>>>>>> >>>>>>>> I ran this again with -ex1_ts_max_steps 0 and get the expected >>>>>>>> result of two steps/frames with the symmetric initial condition in both. >>>>>>>> THis is correct behavior. >>>>>>>> >>>>>>>> Any ideas? >>>>>>>> Thanks >>>>>>>> >>>>>>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>> index 9e4c8f1b61..31dfda2fad 100644 >>>>>>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>>>>>> test: >>>>>>>> suffix: 0 >>>>>>>> requires: p4est !complex >>>>>>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>>>>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>>>>>> >>>>>>>> TEST*/ >>>>>>>> >>>>>>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>>> >>>>>>>>> Landau ex1 should work. I will test. >>>>>>>>> >>>>>>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley < >>>>>>>>> knepley(a)gmail.com> wrote: >>>>>>>>> >>>>>>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to >>>>>>>>>>> the vector array. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I think Mark is saying that those are not working. We do call >>>>>>>>>> VecGetArrayRead() in the HDF5 code. >>>>>>>>>> >>>>>>>>>> Mark, it seem like a small broken code is necessary. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Matt >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> --Junchao Zhang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType >>>>>>>>>>>> (one has a pointer to the other). >>>>>>>>>>>> >>>>>>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken >>>>>>>>>>>> for devices. >>>>>>>>>>>> >>>>>>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>>>>>> >>>>>>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer >>>>>>>>>>>> when "mtype==NULL"? >>>>>>>>>>>> >>>>>>>>>>>> Mark >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>>> experiments lead. >>>>>>>>>> -- Norbert Wiener >>>>>>>>>> >>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>>> >>>>>>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 24 Jun '21

24 Jun '21

On Thu, Jun 24, 2021 at 3:05 PM Matthew Knepley <knepley(a)gmail.com> wrote: > On Thu, Jun 24, 2021 at 9:58 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> OK, I tried again and deleted the arch directory every time: >> > > Okay, this changes the interface for OutputSequenceNumber. It _should_ be > backwards compatible. The likely error I would think > is that it thinks you have a timestep at first but not later, however that > should be an HDF5 error. Maybe it think the timestep is always > the same and overwrites? > * The symptom seems to be that there is only one time step in the file and it is the last one. So yes, it seems to overwrite one file location over and over. * I commented out the 2nd View call in ex1 and the file was only 8 bytes smaller (eg, the time step). The .xmf file was a bit under 2x bigger. * Visit "sees" all the time steps. That is, I can step through or run a movie of all the time steps, but the data is exactly the same at all steps. I've run longer problems with ex2, for each time step. Mark > > Thanks, > > Matt > > >> (base) 09:56 (84933e400a...)|BISECTING ~/Codes/petsc2$ git bisect good >> d7dd068b66a8daa3a37c9e1556b34bd8def54922 is the first bad commit >> commit d7dd068b66a8daa3a37c9e1556b34bd8def54922 >> Author: Vaclav Hapla <vaclav.hapla(a)erdw.ethz.ch> >> Date: Wed Apr 14 21:22:37 2021 +0200 >> >> HDF5: Improve timestepping. >> >> * add PetscViewerHDF5{Push,Pop,Is}Timestepping to control >> timestepping mode >> * write timestepping attribute for datasets and check it on reading >> * fail gracefully if trying to read non-timestepped dataset in >> timestepping mode and vice-versa (fix #425) >> * rewrite src/vec/vec/tutorials/ex19.c to improve coverage for >> timestepping testing >> >> doc/documentation/changes/dev.rst | 19 ++- >> include/petsc/private/viewerhdf5impl.h | 2 + >> include/petscviewerhdf5.h | 4 + >> src/dm/impls/da/gr2.c | 21 ++-- >> src/dm/impls/plex/plexhdf5.c | 25 +++- >> src/sys/classes/viewer/impls/hdf5/hdf5v.c | 124 ++++++++++++++++++-- >> src/vec/is/is/impls/general/general.c | 10 +- >> src/vec/is/utils/hdf5io.c | 45 +++++--- >> src/vec/vec/impls/mpi/pdvec.c | 14 ++- >> src/vec/vec/tutorials/ex19.c | 186 >> +++++++++++++++++------------- >> src/vec/vec/tutorials/output/ex19_2.out | 0 >> src/vec/vec/tutorials/output/ex19_3.out | 0 >> src/vec/vec/tutorials/output/ex19_4.out | 0 >> 13 files changed, 320 insertions(+), 130 deletions(-) >> delete mode 100644 src/vec/vec/tutorials/output/ex19_2.out >> delete mode 100644 src/vec/vec/tutorials/output/ex19_3.out >> delete mode 100644 src/vec/vec/tutorials/output/ex19_4.out >> >> On Thu, Jun 24, 2021 at 7:04 AM Matthew Knepley <knepley(a)gmail.com> >> wrote: >> >>> On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> OK, this is what I get with bisect: >>>> >>>> (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good >>>> The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. >>>> This means the bug has been fixed between >>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and >>>> [09da24df01e50defd94bc4f7396f866a808ecea5 >>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. >>>> >>>> (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout >>>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 >>>> Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 >>>> HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into >>>> 'release' >>>> (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ >>>> >>> >>> I do not understand. Neither of those commits do anything. >>> >>> Matt >>> >>> >>>> On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>>> wrote: >>>> >>>>> Mark, >>>>> I am not sure what your problem is. If it is a regression, can you >>>>> bisect it? >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> I also tried commenting out the second VecView, so there is just one >>>>>> step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>>>>> file goes from 5373 bytes to 3090 bytes. >>>>>> >>>>>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> It is not a device issue but it is a regression. >>>>>>> >>>>>>> Landau ex1 is tiny and just calls VecView before and after the >>>>>>> TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 >>>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see >>>>>>> below), you get an h5 file with two time steps, as it should be. >>>>>>> This is a huge electric field, Ez=10, which makes the electron >>>>>>> distribution (u_e) get visibly pulled off center. >>>>>>> In Visit, both time steps have identical data that is clearly after >>>>>>> the solve and not the initial condition (see attached). >>>>>>> >>>>>>> I ran this again with -ex1_ts_max_steps 0 and get the expected >>>>>>> result of two steps/frames with the symmetric initial condition in both. >>>>>>> THis is correct behavior. >>>>>>> >>>>>>> Any ideas? >>>>>>> Thanks >>>>>>> >>>>>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>> index 9e4c8f1b61..31dfda2fad 100644 >>>>>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>>>>> test: >>>>>>> suffix: 0 >>>>>>> requires: p4est !complex >>>>>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>>>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>>>>> >>>>>>> TEST*/ >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> Landau ex1 should work. I will test. >>>>>>>> >>>>>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to >>>>>>>>>> the vector array. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I think Mark is saying that those are not working. We do call >>>>>>>>> VecGetArrayRead() in the HDF5 code. >>>>>>>>> >>>>>>>>> Mark, it seem like a small broken code is necessary. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> --Junchao Zhang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one >>>>>>>>>>> has a pointer to the other). >>>>>>>>>>> >>>>>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken >>>>>>>>>>> for devices. >>>>>>>>>>> >>>>>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>>>>> >>>>>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer >>>>>>>>>>> when "mtype==NULL"? >>>>>>>>>>> >>>>>>>>>>> Mark >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>>> >>>>>>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Matthew Knepley 24 Jun '21

24 Jun '21

On Thu, Jun 24, 2021 at 9:58 AM Mark Adams <mfadams(a)lbl.gov> wrote: > OK, I tried again and deleted the arch directory every time: > Okay, this changes the interface for OutputSequenceNumber. It _should_ be backwards compatible. The likely error I would think is that it thinks you have a timestep at first but not later, however that should be an HDF5 error. Maybe it think the timestep is always the same and overwrites? Thanks, Matt > (base) 09:56 (84933e400a...)|BISECTING ~/Codes/petsc2$ git bisect good > d7dd068b66a8daa3a37c9e1556b34bd8def54922 is the first bad commit > commit d7dd068b66a8daa3a37c9e1556b34bd8def54922 > Author: Vaclav Hapla <vaclav.hapla(a)erdw.ethz.ch> > Date: Wed Apr 14 21:22:37 2021 +0200 > > HDF5: Improve timestepping. > > * add PetscViewerHDF5{Push,Pop,Is}Timestepping to control timestepping > mode > * write timestepping attribute for datasets and check it on reading > * fail gracefully if trying to read non-timestepped dataset in > timestepping mode and vice-versa (fix #425) > * rewrite src/vec/vec/tutorials/ex19.c to improve coverage for > timestepping testing > > doc/documentation/changes/dev.rst | 19 ++- > include/petsc/private/viewerhdf5impl.h | 2 + > include/petscviewerhdf5.h | 4 + > src/dm/impls/da/gr2.c | 21 ++-- > src/dm/impls/plex/plexhdf5.c | 25 +++- > src/sys/classes/viewer/impls/hdf5/hdf5v.c | 124 ++++++++++++++++++-- > src/vec/is/is/impls/general/general.c | 10 +- > src/vec/is/utils/hdf5io.c | 45 +++++--- > src/vec/vec/impls/mpi/pdvec.c | 14 ++- > src/vec/vec/tutorials/ex19.c | 186 > +++++++++++++++++------------- > src/vec/vec/tutorials/output/ex19_2.out | 0 > src/vec/vec/tutorials/output/ex19_3.out | 0 > src/vec/vec/tutorials/output/ex19_4.out | 0 > 13 files changed, 320 insertions(+), 130 deletions(-) > delete mode 100644 src/vec/vec/tutorials/output/ex19_2.out > delete mode 100644 src/vec/vec/tutorials/output/ex19_3.out > delete mode 100644 src/vec/vec/tutorials/output/ex19_4.out > > On Thu, Jun 24, 2021 at 7:04 AM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> OK, this is what I get with bisect: >>> >>> (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good >>> The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. >>> This means the bug has been fixed between >>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and >>> [09da24df01e50defd94bc4f7396f866a808ecea5 >>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. >>> >>> (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout >>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 >>> Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 >>> HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into >>> 'release' >>> (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ >>> >> >> I do not understand. Neither of those commits do anything. >> >> Matt >> >> >>> On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>> wrote: >>> >>>> Mark, >>>> I am not sure what your problem is. If it is a regression, can you >>>> bisect it? >>>> --Junchao Zhang >>>> >>>> >>>> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> I also tried commenting out the second VecView, so there is just one >>>>> step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>>>> file goes from 5373 bytes to 3090 bytes. >>>>> >>>>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> It is not a device issue but it is a regression. >>>>>> >>>>>> Landau ex1 is tiny and just calls VecView before and after the >>>>>> TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 >>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see >>>>>> below), you get an h5 file with two time steps, as it should be. >>>>>> This is a huge electric field, Ez=10, which makes the electron >>>>>> distribution (u_e) get visibly pulled off center. >>>>>> In Visit, both time steps have identical data that is clearly after >>>>>> the solve and not the initial condition (see attached). >>>>>> >>>>>> I ran this again with -ex1_ts_max_steps 0 and get the expected >>>>>> result of two steps/frames with the symmetric initial condition in both. >>>>>> THis is correct behavior. >>>>>> >>>>>> Any ideas? >>>>>> Thanks >>>>>> >>>>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> index 9e4c8f1b61..31dfda2fad 100644 >>>>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>>>> test: >>>>>> suffix: 0 >>>>>> requires: p4est !complex >>>>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>>>> >>>>>> TEST*/ >>>>>> >>>>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> Landau ex1 should work. I will test. >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>>> >>>>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>>>>>> vector array. >>>>>>>>> >>>>>>>> >>>>>>>> I think Mark is saying that those are not working. We do call >>>>>>>> VecGetArrayRead() in the HDF5 code. >>>>>>>> >>>>>>>> Mark, it seem like a small broken code is necessary. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> --Junchao Zhang >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one >>>>>>>>>> has a pointer to the other). >>>>>>>>>> >>>>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>>>>>> devices. >>>>>>>>>> >>>>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>>>> >>>>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer >>>>>>>>>> when "mtype==NULL"? >>>>>>>>>> >>>>>>>>>> Mark >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>> >>>>>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Jed Brown 24 Jun '21

24 Jun '21

https://gitlab.com/petsc/petsc/-/commit/d7dd068b66a8daa3a37c9e1556b34bd8def… On Thu, Jun 24, 2021, at 11:40 AM, Mark Adams wrote: > How does one view a commit on gitlab? > I would like to look at d7dd068b66a8daa3a37c9e1556b34bd8def54922 > > On Thu, Jun 24, 2021 at 9:58 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> OK, I tried again and deleted the arch directory every time: >> >> (base) 09:56 (84933e400a...)|BISECTING ~/Codes/petsc2$ git bisect good >> d7dd068b66a8daa3a37c9e1556b34bd8def54922 is the first bad commit >> commit d7dd068b66a8daa3a37c9e1556b34bd8def54922 >> Author: Vaclav Hapla <vaclav.hapla(a)erdw.ethz.ch> >> Date: Wed Apr 14 21:22:37 2021 +0200 >> >> HDF5: Improve timestepping. >> >> * add PetscViewerHDF5{Push,Pop,Is}Timestepping to control timestepping mode >> * write timestepping attribute for datasets and check it on reading >> * fail gracefully if trying to read non-timestepped dataset in timestepping mode and vice-versa (fix #425) >> * rewrite src/vec/vec/tutorials/ex19.c to improve coverage for timestepping testing >> >> doc/documentation/changes/dev.rst | 19 ++- >> include/petsc/private/viewerhdf5impl.h | 2 + >> include/petscviewerhdf5.h | 4 + >> src/dm/impls/da/gr2.c | 21 ++-- >> src/dm/impls/plex/plexhdf5.c | 25 +++- >> src/sys/classes/viewer/impls/hdf5/hdf5v.c | 124 ++++++++++++++++++-- >> src/vec/is/is/impls/general/general.c | 10 +- >> src/vec/is/utils/hdf5io.c | 45 +++++--- >> src/vec/vec/impls/mpi/pdvec.c | 14 ++- >> src/vec/vec/tutorials/ex19.c | 186 +++++++++++++++++------------- >> src/vec/vec/tutorials/output/ex19_2.out | 0 >> src/vec/vec/tutorials/output/ex19_3.out | 0 >> src/vec/vec/tutorials/output/ex19_4.out | 0 >> 13 files changed, 320 insertions(+), 130 deletions(-) >> delete mode 100644 src/vec/vec/tutorials/output/ex19_2.out >> delete mode 100644 src/vec/vec/tutorials/output/ex19_3.out >> delete mode 100644 src/vec/vec/tutorials/output/ex19_4.out >> >> On Thu, Jun 24, 2021 at 7:04 AM Matthew Knepley <knepley(a)gmail.com> wrote: >>> On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> OK, this is what I get with bisect: >>>> >>>> (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good >>>> The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. >>>> This means the bug has been fixed between c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and [09da24df01e50defd94bc4f7396f866a808ecea5 c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. >>>> >>>> (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 >>>> Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 >>>> HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into 'release' >>>> (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ >>> >>> I do not understand. Neither of those commits do anything. >>> >>> Matt >>> >>>> On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: >>>>> Mark, >>>>> I am not sure what your problem is. If it is a regression, can you bisect it? >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> I also tried commenting out the second VecView, so there is just one step in the file, and the .h5 file is only 8 bytes smaller and the .xmf file goes from 5373 bytes to 3090 bytes. >>>>>> >>>>>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> It is not a device issue but it is a regression. >>>>>>> >>>>>>> Landau ex1 is tiny and just calls VecView before and after the TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you get an h5 file with two time steps, as it should be. >>>>>>> This is a huge electric field, Ez=10, which makes the electron distribution (u_e) get visibly pulled off center. >>>>>>> In Visit, both time steps have identical data that is clearly after the solve and not the initial condition (see attached). >>>>>>> >>>>>>> I ran this again with -ex1_ts_max_steps 0 and get the expected result of two steps/frames with the symmetric initial condition in both. THis is correct behavior. >>>>>>> >>>>>>> Any ideas? >>>>>>> Thanks >>>>>>> >>>>>>> >>>>>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>> index 9e4c8f1b61..31dfda2fad 100644 >>>>>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>>>>> test: >>>>>>> suffix: 0 >>>>>>> requires: p4est !complex >>>>>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 -ex1_ts_adapt_scale_solve_failed 0.75 -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>>>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 -ex1_ts_adapt_scale_solve_failed 0.75 -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>>>>> >>>>>>> TEST*/ >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>>> Landau ex1 should work. I will test. >>>>>>>> >>>>>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> wrote: >>>>>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: >>>>>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the vector array. >>>>>>>>> >>>>>>>>> I think Mark is saying that those are not working. We do call VecGetArrayRead() in the HDF5 code. >>>>>>>>> >>>>>>>>> Mark, it seem like a small broken code is necessary. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>>> --Junchao Zhang >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one has a pointer to the other). >>>>>>>>>>> >>>>>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for devices. >>>>>>>>>>> >>>>>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>>>>> >>>>>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when "mtype==NULL"? >>>>>>>>>>> >>>>>>>>>>> Mark >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>> >>> >>> -- >>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 24 Jun '21

24 Jun '21

How does one view a commit on gitlab? I would like to look at d7dd068b66a8daa3a37c9e1556b34bd8def54922 On Thu, Jun 24, 2021 at 9:58 AM Mark Adams <mfadams(a)lbl.gov> wrote: > OK, I tried again and deleted the arch directory every time: > > (base) 09:56 (84933e400a...)|BISECTING ~/Codes/petsc2$ git bisect good > d7dd068b66a8daa3a37c9e1556b34bd8def54922 is the first bad commit > commit d7dd068b66a8daa3a37c9e1556b34bd8def54922 > Author: Vaclav Hapla <vaclav.hapla(a)erdw.ethz.ch> > Date: Wed Apr 14 21:22:37 2021 +0200 > > HDF5: Improve timestepping. > > * add PetscViewerHDF5{Push,Pop,Is}Timestepping to control timestepping > mode > * write timestepping attribute for datasets and check it on reading > * fail gracefully if trying to read non-timestepped dataset in > timestepping mode and vice-versa (fix #425) > * rewrite src/vec/vec/tutorials/ex19.c to improve coverage for > timestepping testing > > doc/documentation/changes/dev.rst | 19 ++- > include/petsc/private/viewerhdf5impl.h | 2 + > include/petscviewerhdf5.h | 4 + > src/dm/impls/da/gr2.c | 21 ++-- > src/dm/impls/plex/plexhdf5.c | 25 +++- > src/sys/classes/viewer/impls/hdf5/hdf5v.c | 124 ++++++++++++++++++-- > src/vec/is/is/impls/general/general.c | 10 +- > src/vec/is/utils/hdf5io.c | 45 +++++--- > src/vec/vec/impls/mpi/pdvec.c | 14 ++- > src/vec/vec/tutorials/ex19.c | 186 > +++++++++++++++++------------- > src/vec/vec/tutorials/output/ex19_2.out | 0 > src/vec/vec/tutorials/output/ex19_3.out | 0 > src/vec/vec/tutorials/output/ex19_4.out | 0 > 13 files changed, 320 insertions(+), 130 deletions(-) > delete mode 100644 src/vec/vec/tutorials/output/ex19_2.out > delete mode 100644 src/vec/vec/tutorials/output/ex19_3.out > delete mode 100644 src/vec/vec/tutorials/output/ex19_4.out > > On Thu, Jun 24, 2021 at 7:04 AM Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> OK, this is what I get with bisect: >>> >>> (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good >>> The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. >>> This means the bug has been fixed between >>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and >>> [09da24df01e50defd94bc4f7396f866a808ecea5 >>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. >>> >>> (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout >>> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 >>> Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 >>> HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into >>> 'release' >>> (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ >>> >> >> I do not understand. Neither of those commits do anything. >> >> Matt >> >> >>> On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> >>> wrote: >>> >>>> Mark, >>>> I am not sure what your problem is. If it is a regression, can you >>>> bisect it? >>>> --Junchao Zhang >>>> >>>> >>>> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> I also tried commenting out the second VecView, so there is just one >>>>> step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>>>> file goes from 5373 bytes to 3090 bytes. >>>>> >>>>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> It is not a device issue but it is a regression. >>>>>> >>>>>> Landau ex1 is tiny and just calls VecView before and after the >>>>>> TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 >>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see >>>>>> below), you get an h5 file with two time steps, as it should be. >>>>>> This is a huge electric field, Ez=10, which makes the electron >>>>>> distribution (u_e) get visibly pulled off center. >>>>>> In Visit, both time steps have identical data that is clearly after >>>>>> the solve and not the initial condition (see attached). >>>>>> >>>>>> I ran this again with -ex1_ts_max_steps 0 and get the expected >>>>>> result of two steps/frames with the symmetric initial condition in both. >>>>>> THis is correct behavior. >>>>>> >>>>>> Any ideas? >>>>>> Thanks >>>>>> >>>>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> index 9e4c8f1b61..31dfda2fad 100644 >>>>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>>>> test: >>>>>> suffix: 0 >>>>>> requires: p4est !complex >>>>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>>>> >>>>>> TEST*/ >>>>>> >>>>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> Landau ex1 should work. I will test. >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>>> >>>>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>>>>>> vector array. >>>>>>>>> >>>>>>>> >>>>>>>> I think Mark is saying that those are not working. We do call >>>>>>>> VecGetArrayRead() in the HDF5 code. >>>>>>>> >>>>>>>> Mark, it seem like a small broken code is necessary. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Matt >>>>>>>> >>>>>>>> >>>>>>>>> --Junchao Zhang >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one >>>>>>>>>> has a pointer to the other). >>>>>>>>>> >>>>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>>>>>> devices. >>>>>>>>>> >>>>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>>>> >>>>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer >>>>>>>>>> when "mtype==NULL"? >>>>>>>>>> >>>>>>>>>> Mark >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> What most experimenters take for granted before they begin their >>>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>>> experiments lead. >>>>>>>> -- Norbert Wiener >>>>>>>> >>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>>> >>>>>>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 24 Jun '21

24 Jun '21

OK, I tried again and deleted the arch directory every time: (base) 09:56 (84933e400a...)|BISECTING ~/Codes/petsc2$ git bisect good d7dd068b66a8daa3a37c9e1556b34bd8def54922 is the first bad commit commit d7dd068b66a8daa3a37c9e1556b34bd8def54922 Author: Vaclav Hapla <vaclav.hapla(a)erdw.ethz.ch> Date: Wed Apr 14 21:22:37 2021 +0200 HDF5: Improve timestepping. * add PetscViewerHDF5{Push,Pop,Is}Timestepping to control timestepping mode * write timestepping attribute for datasets and check it on reading * fail gracefully if trying to read non-timestepped dataset in timestepping mode and vice-versa (fix #425) * rewrite src/vec/vec/tutorials/ex19.c to improve coverage for timestepping testing doc/documentation/changes/dev.rst | 19 ++- include/petsc/private/viewerhdf5impl.h | 2 + include/petscviewerhdf5.h | 4 + src/dm/impls/da/gr2.c | 21 ++-- src/dm/impls/plex/plexhdf5.c | 25 +++- src/sys/classes/viewer/impls/hdf5/hdf5v.c | 124 ++++++++++++++++++-- src/vec/is/is/impls/general/general.c | 10 +- src/vec/is/utils/hdf5io.c | 45 +++++--- src/vec/vec/impls/mpi/pdvec.c | 14 ++- src/vec/vec/tutorials/ex19.c | 186 +++++++++++++++++------------- src/vec/vec/tutorials/output/ex19_2.out | 0 src/vec/vec/tutorials/output/ex19_3.out | 0 src/vec/vec/tutorials/output/ex19_4.out | 0 13 files changed, 320 insertions(+), 130 deletions(-) delete mode 100644 src/vec/vec/tutorials/output/ex19_2.out delete mode 100644 src/vec/vec/tutorials/output/ex19_3.out delete mode 100644 src/vec/vec/tutorials/output/ex19_4.out On Thu, Jun 24, 2021 at 7:04 AM Matthew Knepley <knepley(a)gmail.com> wrote: > On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> OK, this is what I get with bisect: >> >> (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good >> The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. >> This means the bug has been fixed between >> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and >> [09da24df01e50defd94bc4f7396f866a808ecea5 >> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. >> >> (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout >> c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 >> Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 >> HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into >> 'release' >> (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ >> > > I do not understand. Neither of those commits do anything. > > Matt > > >> On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> Mark, >>> I am not sure what your problem is. If it is a regression, can you >>> bisect it? >>> --Junchao Zhang >>> >>> >>> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> I also tried commenting out the second VecView, so there is just one >>>> step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>>> file goes from 5373 bytes to 3090 bytes. >>>> >>>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> It is not a device issue but it is a regression. >>>>> >>>>> Landau ex1 is tiny and just calls VecView before and after the >>>>> TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 >>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see >>>>> below), you get an h5 file with two time steps, as it should be. >>>>> This is a huge electric field, Ez=10, which makes the electron >>>>> distribution (u_e) get visibly pulled off center. >>>>> In Visit, both time steps have identical data that is clearly after >>>>> the solve and not the initial condition (see attached). >>>>> >>>>> I ran this again with -ex1_ts_max_steps 0 and get the expected result >>>>> of two steps/frames with the symmetric initial condition in both. THis is >>>>> correct behavior. >>>>> >>>>> Any ideas? >>>>> Thanks >>>>> >>>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>> index 9e4c8f1b61..31dfda2fad 100644 >>>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>>> test: >>>>> suffix: 0 >>>>> requires: p4est !complex >>>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>>> >>>>> TEST*/ >>>>> >>>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> Landau ex1 should work. I will test. >>>>>> >>>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>>>> wrote: >>>>>> >>>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>>> >>>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>>>>> vector array. >>>>>>>> >>>>>>> >>>>>>> I think Mark is saying that those are not working. We do call >>>>>>> VecGetArrayRead() in the HDF5 code. >>>>>>> >>>>>>> Mark, it seem like a small broken code is necessary. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> --Junchao Zhang >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>>> >>>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one >>>>>>>>> has a pointer to the other). >>>>>>>>> >>>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>>>>> devices. >>>>>>>>> >>>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>>> >>>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>>>>>>> "mtype==NULL"? >>>>>>>>> >>>>>>>>> Mark >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>>> >>>>>> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Matthew Knepley 24 Jun '21

24 Jun '21

On Thu, Jun 24, 2021 at 7:00 AM Mark Adams <mfadams(a)lbl.gov> wrote: > OK, this is what I get with bisect: > > (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good > The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. > This means the bug has been fixed between > c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and > [09da24df01e50defd94bc4f7396f866a808ecea5 > c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. > > (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout > c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 > Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 > HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into > 'release' > (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ > I do not understand. Neither of those commits do anything. Matt > On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> Mark, >> I am not sure what your problem is. If it is a regression, can you >> bisect it? >> --Junchao Zhang >> >> >> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> I also tried commenting out the second VecView, so there is just one >>> step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>> file goes from 5373 bytes to 3090 bytes. >>> >>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> It is not a device issue but it is a regression. >>>> >>>> Landau ex1 is tiny and just calls VecView before and after the TSsolve, >>>> which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view >>>> hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you >>>> get an h5 file with two time steps, as it should be. >>>> This is a huge electric field, Ez=10, which makes the electron >>>> distribution (u_e) get visibly pulled off center. >>>> In Visit, both time steps have identical data that is clearly after the >>>> solve and not the initial condition (see attached). >>>> >>>> I ran this again with -ex1_ts_max_steps 0 and get the expected result >>>> of two steps/frames with the symmetric initial condition in both. THis is >>>> correct behavior. >>>> >>>> Any ideas? >>>> Thanks >>>> >>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> index 9e4c8f1b61..31dfda2fad 100644 >>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>> test: >>>> suffix: 0 >>>> requires: p4est !complex >>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>> >>>> TEST*/ >>>> >>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> Landau ex1 should work. I will test. >>>>> >>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>>> wrote: >>>>> >>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>> >>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>>>> vector array. >>>>>>> >>>>>> >>>>>> I think Mark is saying that those are not working. We do call >>>>>> VecGetArrayRead() in the HDF5 code. >>>>>> >>>>>> Mark, it seem like a small broken code is necessary. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one >>>>>>>> has a pointer to the other). >>>>>>>> >>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>>>> devices. >>>>>>>> >>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>> >>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>>>>>> "mtype==NULL"? >>>>>>>> >>>>>>>> Mark >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>> >>>>> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 24 Jun '21

24 Jun '21

> > > > Can you nudge it to the next commit? It looks like I split up the sets > incorrectly for the PetscWeakForm change. > Not sure how to do that but I simply commented out the whole block (in a "hybrid" method that I apparently don't use).

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 24 Jun '21

24 Jun '21

OK, this is what I get with bisect: (base) 06:54 (c3b2925bfb...)|BISECTING ~/Codes/petsc2$ git bisect good The merge base c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 is bad. This means the bug has been fixed between c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 and [09da24df01e50defd94bc4f7396f866a808ecea5 c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47]. (base) 06:56 (v3.15.1) ~/Codes/petsc2$ git checkout c3b2925bfbe1c0a0b3a69c9a76295d0747e33d47 Previous HEAD position was 09da24df01 Increase patchlevel to 3.15.1 HEAD is now at c3b2925bfb Merge branch 'knepley-main-patch-78504' into 'release' (base) 06:56 (c3b2925bfb...) ~/Codes/petsc2$ On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Mark, > I am not sure what your problem is. If it is a regression, can you > bisect it? > --Junchao Zhang > > > On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> I also tried commenting out the second VecView, so there is just one step >> in the file, and the .h5 file is only 8 bytes smaller and the .xmf file >> goes from 5373 bytes to 3090 bytes. >> >> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> It is not a device issue but it is a regression. >>> >>> Landau ex1 is tiny and just calls VecView before and after the TSsolve, >>> which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view >>> hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you >>> get an h5 file with two time steps, as it should be. >>> This is a huge electric field, Ez=10, which makes the electron >>> distribution (u_e) get visibly pulled off center. >>> In Visit, both time steps have identical data that is clearly after the >>> solve and not the initial condition (see attached). >>> >>> I ran this again with -ex1_ts_max_steps 0 and get the expected result >>> of two steps/frames with the symmetric initial condition in both. THis is >>> correct behavior. >>> >>> Any ideas? >>> Thanks >>> >>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> index 9e4c8f1b61..31dfda2fad 100644 >>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>> test: >>> suffix: 0 >>> requires: p4est !complex >>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>> -ex1_ts_adapt_scale_solve_failed 0.75 >>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>> -ex1_ts_adapt_scale_solve_failed 0.75 >>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>> >>> TEST*/ >>> >>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> Landau ex1 should work. I will test. >>>> >>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>> wrote: >>>> >>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>> junchao.zhang(a)gmail.com> wrote: >>>>> >>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>>> vector array. >>>>>> >>>>> >>>>> I think Mark is saying that those are not working. We do call >>>>> VecGetArrayRead() in the HDF5 code. >>>>> >>>>> Mark, it seem like a small broken code is necessary. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one has >>>>>>> a pointer to the other). >>>>>>> >>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>>> devices. >>>>>>> >>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>> >>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>>>>> "mtype==NULL"? >>>>>>> >>>>>>> Mark >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>>

1 0