June 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] VecGetArrayAndMemType
by Matthew Knepley 24 Jun '21

24 Jun '21

On Thu, Jun 24, 2021 at 6:50 AM Mark Adams <mfadams(a)lbl.gov> wrote: > Good idea. > I have it on my laptop and I have 8 bisection steps to do (the most recent > version tag is good, yes!), but the repo seems broken at ea672e62cc. > I am looking at the code and these vars are in fact not defined. I > commented out this block and am pressing on. > > However, this looks like a bug. > You should be able to reproduce this by bisection with main=good and > v3.15.1=bad. > First step fails with this: > > /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1138:11: > error: use of undeclared identifier 'n0' > if (n0) { > ^ > /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1143:11: > error: use of undeclared identifier 'n1' > if (n1) { > ^ > /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1148:11: > error: use of undeclared identifier 'n2' > if (n2) { > ^ > /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1153:11: > error: use of undeclared identifier 'n3' > if (n3) { > ^ > 4 errors generated. > make[3]: *** [arch-macosx-gnu-g/obj/dm/dt/fe/impls/basic/febasic.o] Error 1 > make[3]: *** Waiting for unfinished jobs.... > make[2]: *** [libs] Error 2 > **************************ERROR************************************* > Error during compile, check arch-macosx-gnu-g/lib/petsc/conf/make.log > Send it and arch-macosx-gnu-g/lib/petsc/conf/configure.log to > petsc-maint(a)mcs.anl.gov > ******************************************************************** > make[1]: *** [all] Error 1 > make: *** [all] Error 2 > *(base) 06:32 2 (ea672e62cc...)|BISECTING ~/Codes/petsc2$ rm -fr > arch-macosx-gnu-g/* > Can you nudge it to the next commit? It looks like I split up the sets incorrectly for the PetscWeakForm change. Matt > On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> Mark, >> I am not sure what your problem is. If it is a regression, can you >> bisect it? >> --Junchao Zhang >> >> >> On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> I also tried commenting out the second VecView, so there is just one >>> step in the file, and the .h5 file is only 8 bytes smaller and the .xmf >>> file goes from 5373 bytes to 3090 bytes. >>> >>> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> It is not a device issue but it is a regression. >>>> >>>> Landau ex1 is tiny and just calls VecView before and after the TSsolve, >>>> which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view >>>> hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you >>>> get an h5 file with two time steps, as it should be. >>>> This is a huge electric field, Ez=10, which makes the electron >>>> distribution (u_e) get visibly pulled off center. >>>> In Visit, both time steps have identical data that is clearly after the >>>> solve and not the initial condition (see attached). >>>> >>>> I ran this again with -ex1_ts_max_steps 0 and get the expected result >>>> of two steps/frames with the symmetric initial condition in both. THis is >>>> correct behavior. >>>> >>>> Any ideas? >>>> Thanks >>>> >>>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> index 9e4c8f1b61..31dfda2fad 100644 >>>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>>> test: >>>> suffix: 0 >>>> requires: p4est !complex >>>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>>> -ex1_ts_adapt_scale_solve_failed 0.75 >>>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>>> >>>> TEST*/ >>>> >>>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> Landau ex1 should work. I will test. >>>>> >>>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>>> wrote: >>>>> >>>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>>> junchao.zhang(a)gmail.com> wrote: >>>>>> >>>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>>>> vector array. >>>>>>> >>>>>> >>>>>> I think Mark is saying that those are not working. We do call >>>>>> VecGetArrayRead() in the HDF5 code. >>>>>> >>>>>> Mark, it seem like a small broken code is necessary. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Matt >>>>>> >>>>>> >>>>>>> --Junchao Zhang >>>>>>> >>>>>>> >>>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>>> >>>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one >>>>>>>> has a pointer to the other). >>>>>>>> >>>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>>>> devices. >>>>>>>> >>>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>>> >>>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>>>>>> "mtype==NULL"? >>>>>>>> >>>>>>>> Mark >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> What most experimenters take for granted before they begin their >>>>>> experiments is infinitely more interesting than any results to which their >>>>>> experiments lead. >>>>>> -- Norbert Wiener >>>>>> >>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>>> >>>>> -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 24 Jun '21

24 Jun '21

Good idea. I have it on my laptop and I have 8 bisection steps to do (the most recent version tag is good, yes!), but the repo seems broken at ea672e62cc. I am looking at the code and these vars are in fact not defined. I commented out this block and am pressing on. However, this looks like a bug. You should be able to reproduce this by bisection with main=good and v3.15.1=bad. First step fails with this: /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1138:11: error: use of undeclared identifier 'n0' if (n0) { ^ /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1143:11: error: use of undeclared identifier 'n1' if (n1) { ^ /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1148:11: error: use of undeclared identifier 'n2' if (n2) { ^ /Users/markadams/Codes/petsc2/src/dm/dt/fe/impls/basic/febasic.c:1153:11: error: use of undeclared identifier 'n3' if (n3) { ^ 4 errors generated. make[3]: *** [arch-macosx-gnu-g/obj/dm/dt/fe/impls/basic/febasic.o] Error 1 make[3]: *** Waiting for unfinished jobs.... make[2]: *** [libs] Error 2 **************************ERROR************************************* Error during compile, check arch-macosx-gnu-g/lib/petsc/conf/make.log Send it and arch-macosx-gnu-g/lib/petsc/conf/configure.log to petsc-maint(a)mcs.anl.gov ******************************************************************** make[1]: *** [all] Error 1 make: *** [all] Error 2 *(base) 06:32 2 (ea672e62cc...)|BISECTING ~/Codes/petsc2$ rm -fr arch-macosx-gnu-g/* On Wed, Jun 23, 2021 at 11:14 PM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Mark, > I am not sure what your problem is. If it is a regression, can you > bisect it? > --Junchao Zhang > > > On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> I also tried commenting out the second VecView, so there is just one step >> in the file, and the .h5 file is only 8 bytes smaller and the .xmf file >> goes from 5373 bytes to 3090 bytes. >> >> On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> It is not a device issue but it is a regression. >>> >>> Landau ex1 is tiny and just calls VecView before and after the TSsolve, >>> which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view >>> hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you >>> get an h5 file with two time steps, as it should be. >>> This is a huge electric field, Ez=10, which makes the electron >>> distribution (u_e) get visibly pulled off center. >>> In Visit, both time steps have identical data that is clearly after the >>> solve and not the initial condition (see attached). >>> >>> I ran this again with -ex1_ts_max_steps 0 and get the expected result >>> of two steps/frames with the symmetric initial condition in both. THis is >>> correct behavior. >>> >>> Any ideas? >>> Thanks >>> >>> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> index 9e4c8f1b61..31dfda2fad 100644 >>> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >>> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >>> test: >>> suffix: 0 >>> requires: p4est !complex >>> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>> -ex1_ts_adapt_scale_solve_failed 0.75 >>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>> .1 -dm_refine 1 -dm_landau_gpu_assembly false >>> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 >>> -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >>> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >>> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >>> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >>> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >>> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >>> -ex1_ts_adapt_scale_solve_failed 0.75 >>> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >>> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >>> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >>> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >>> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >>> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >>> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >>> >>> TEST*/ >>> >>> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> Landau ex1 should work. I will test. >>>> >>>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>>> wrote: >>>> >>>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang < >>>>> junchao.zhang(a)gmail.com> wrote: >>>>> >>>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>>> vector array. >>>>>> >>>>> >>>>> I think Mark is saying that those are not working. We do call >>>>> VecGetArrayRead() in the HDF5 code. >>>>> >>>>> Mark, it seem like a small broken code is necessary. >>>>> >>>>> Thanks, >>>>> >>>>> Matt >>>>> >>>>> >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>>> >>>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one has >>>>>>> a pointer to the other). >>>>>>> >>>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>>> devices. >>>>>>> >>>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>>> >>>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>>>>> "mtype==NULL"? >>>>>>> >>>>>>> Mark >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.cse.buffalo.edu/~knepley/> >>>>> >>>>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Junchao Zhang 23 Jun '21

23 Jun '21

Mark, I am not sure what your problem is. If it is a regression, can you bisect it? --Junchao Zhang On Wed, Jun 23, 2021 at 4:04 PM Mark Adams <mfadams(a)lbl.gov> wrote: > I also tried commenting out the second VecView, so there is just one step > in the file, and the .h5 file is only 8 bytes smaller and the .xmf file > goes from 5373 bytes to 3090 bytes. > > On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> It is not a device issue but it is a regression. >> >> Landau ex1 is tiny and just calls VecView before and after the TSsolve, >> which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view >> hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you get >> an h5 file with two time steps, as it should be. >> This is a huge electric field, Ez=10, which makes the electron >> distribution (u_e) get visibly pulled off center. >> In Visit, both time steps have identical data that is clearly after the >> solve and not the initial condition (see attached). >> >> I ran this again with -ex1_ts_max_steps 0 and get the expected result of >> two steps/frames with the symmetric initial condition in both. THis is >> correct behavior. >> >> Any ideas? >> Thanks >> >> diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c >> b/src/ts/utils/dmplexlandau/tutorials/ex1.c >> index 9e4c8f1b61..31dfda2fad 100644 >> --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c >> +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c >> @@ -66,6 +66,6 @@ int main(int argc, char **argv) >> test: >> suffix: 0 >> requires: p4est !complex >> - args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type >> p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >> -ex1_ts_adapt_scale_solve_failed 0.75 >> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >> .1 -dm_refine 1 -dm_landau_gpu_assembly false >> + args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type >> p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 >> -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 >> 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 >> -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex >> -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 >> -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 >> -ex1_ts_adapt_scale_solve_failed 0.75 >> -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type >> lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 >> -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius >> 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 >> -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 >> .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 >> -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* >> >> TEST*/ >> >> On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> Landau ex1 should work. I will test. >>> >>> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >>> wrote: >>> >>>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang <junchao.zhang(a)gmail.com> >>>> wrote: >>>> >>>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>>> vector array. >>>>> >>>> >>>> I think Mark is saying that those are not working. We do call >>>> VecGetArrayRead() in the HDF5 code. >>>> >>>> Mark, it seem like a small broken code is necessary. >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> >>>>> --Junchao Zhang >>>>> >>>>> >>>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>>> >>>>>> First, there seem to be two pages for VecGetArrayAndMemType (one has >>>>>> a pointer to the other). >>>>>> >>>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>>> devices. >>>>>> >>>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>>> >>>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>>>> "mtype==NULL"? >>>>>> >>>>>> Mark >>>>>> >>>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which their >>>> experiments lead. >>>> -- Norbert Wiener >>>> >>>> https://www.cse.buffalo.edu/~knepley/ >>>> <http://www.cse.buffalo.edu/~knepley/> >>>> >>>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 23 Jun '21

23 Jun '21

I also tried commenting out the second VecView, so there is just one step in the file, and the .h5 file is only 8 bytes smaller and the .xmf file goes from 5373 bytes to 3090 bytes. On Wed, Jun 23, 2021 at 4:01 PM Mark Adams <mfadams(a)lbl.gov> wrote: > It is not a device issue but it is a regression. > > Landau ex1 is tiny and just calls VecView before and after the TSsolve, > which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view > hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you get > an h5 file with two time steps, as it should be. > This is a huge electric field, Ez=10, which makes the electron > distribution (u_e) get visibly pulled off center. > In Visit, both time steps have identical data that is clearly after the > solve and not the initial condition (see attached). > > I ran this again with -ex1_ts_max_steps 0 and get the expected result of > two steps/frames with the symmetric initial condition in both. THis is > correct behavior. > > Any ideas? > Thanks > > diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c > b/src/ts/utils/dmplexlandau/tutorials/ex1.c > index 9e4c8f1b61..31dfda2fad 100644 > --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c > +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c > @@ -66,6 +66,6 @@ int main(int argc, char **argv) > test: > suffix: 0 > requires: p4est !complex > - args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type > p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 > -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 > 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 > -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex > -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 > -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 > -ex1_ts_adapt_scale_solve_failed 0.75 > -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type > lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 > -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius > 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 > -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 > .1 -dm_refine 1 -dm_landau_gpu_assembly false > + args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type > p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 > -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 > 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 > -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex > -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 > -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 > -ex1_ts_adapt_scale_solve_failed 0.75 > -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type > lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 > -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius > 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 > -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 > .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 > -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* > > TEST*/ > > On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: > >> Landau ex1 should work. I will test. >> >> On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> >> wrote: >> >>> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang <junchao.zhang(a)gmail.com> >>> wrote: >>> >>>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>>> vector array. >>>> >>> >>> I think Mark is saying that those are not working. We do call >>> VecGetArrayRead() in the HDF5 code. >>> >>> Mark, it seem like a small broken code is necessary. >>> >>> Thanks, >>> >>> Matt >>> >>> >>>> --Junchao Zhang >>>> >>>> >>>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>>> >>>>> First, there seem to be two pages for VecGetArrayAndMemType (one has a >>>>> pointer to the other). >>>>> >>>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>>> devices. >>>>> >>>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>>> >>>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>>> "mtype==NULL"? >>>>> >>>>> Mark >>>>> >>>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ >>> <http://www.cse.buffalo.edu/~knepley/> >>> >>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 23 Jun '21

23 Jun '21

It is not a device issue but it is a regression. Landau ex1 is tiny and just calls VecView before and after the TSsolve, which is one time step. If you add "*-dm_view hdf5:f.h5 -vec_view hdf5:f.h5::append -dm_landau_Ez 10.*" to landau/ex1 (see below), you get an h5 file with two time steps, as it should be. This is a huge electric field, Ez=10, which makes the electron distribution (u_e) get visibly pulled off center. In Visit, both time steps have identical data that is clearly after the solve and not the initial condition (see attached). I ran this again with -ex1_ts_max_steps 0 and get the expected result of two steps/frames with the symmetric initial condition in both. THis is correct behavior. Any ideas? Thanks diff --git a/src/ts/utils/dmplexlandau/tutorials/ex1.c b/src/ts/utils/dmplexlandau/tutorials/ex1.c index 9e4c8f1b61..31dfda2fad 100644 --- a/src/ts/utils/dmplexlandau/tutorials/ex1.c +++ b/src/ts/utils/dmplexlandau/tutorials/ex1.c @@ -66,6 +66,6 @@ int main(int argc, char **argv) test: suffix: 0 requires: p4est !complex - args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 -ex1_ts_adapt_scale_solve_failed 0.75 -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 .1 -dm_refine 1 -dm_landau_gpu_assembly false + args: -petscspace_degree 3 -petscspace_poly_tensor 1 -dm_landau_type p4est -dm_landau_ion_masses 2,4 -dm_landau_ion_charges 1,18 -dm_landau_thermal_temps 5,5,.5 -dm_landau_n 1.00018,1,1e-5 -dm_landau_n_0 1e20 -ex1_ts_monitor -ex1_snes_rtol 1.e-14 -ex1_snes_stol 1.e-14 -ex1_snes_monitor -ex1_snes_converged_reason -ex1_ts_type arkimex -ex1_ts_arkimex_type 1bee -ex1_ts_max_snes_failures -1 -ex1_ts_rtol 1e-1 -ex1_ts_dt 1.e-1 -ex1_ts_max_time 1 -ex1_ts_adapt_clip .5,1.25 -ex1_ts_adapt_scale_solve_failed 0.75 -ex1_ts_adapt_time_step_increase_delay 5 -ex1_ts_max_steps 1 -ex1_pc_type lu -ex1_ksp_type preonly -dm_landau_amr_levels_max 7 -dm_landau_domain_radius 5 -dm_landau_amr_re_levels 0 -dm_landau_re_radius 1 -dm_landau_amr_z_refine1 1 -dm_landau_amr_z_refine2 0 -dm_landau_amr_post_refine 0 -dm_landau_z_radius1 .1 -dm_landau_z_radius2 .1 -dm_refine 1 -dm_landau_gpu_assembly false *-dm_view hdf5:f.h5 -vec_view hdf5:f.h5::append -dm_landau_Ez 10.* TEST*/ On Wed, Jun 23, 2021 at 1:38 PM Mark Adams <mfadams(a)lbl.gov> wrote: > Landau ex1 should work. I will test. > > On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> > wrote: > >> On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang <junchao.zhang(a)gmail.com> >> wrote: >> >>> Use VecGetArrayRead/Write() to get up-to-date host pointers to the >>> vector array. >>> >> >> I think Mark is saying that those are not working. We do call >> VecGetArrayRead() in the HDF5 code. >> >> Mark, it seem like a small broken code is necessary. >> >> Thanks, >> >> Matt >> >> >>> --Junchao Zhang >>> >>> >>> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >>> >>>> First, there seem to be two pages for VecGetArrayAndMemType (one has a >>>> pointer to the other). >>>> >>>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>>> devices. >>>> >>>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>>> >>>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>>> "mtype==NULL"? >>>> >>>> Mark >>>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 23 Jun '21

23 Jun '21

Landau ex1 should work. I will test. On Wed, Jun 23, 2021 at 10:47 AM Matthew Knepley <knepley(a)gmail.com> wrote: > On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang <junchao.zhang(a)gmail.com> > wrote: > >> Use VecGetArrayRead/Write() to get up-to-date host pointers to the vector >> array. >> > > I think Mark is saying that those are not working. We do call > VecGetArrayRead() in the HDF5 code. > > Mark, it seem like a small broken code is necessary. > > Thanks, > > Matt > > >> --Junchao Zhang >> >> >> On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: >> >>> First, there seem to be two pages for VecGetArrayAndMemType (one has a >>> pointer to the other). >>> >>> So I need to get a CPU array for HDF5 viewing. Totally broken for >>> devices. >>> >>> I don't find a VecGetArrayCpu[HOST] that does the right thing. >>> >>> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >>> "mtype==NULL"? >>> >>> Mark >>> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> >

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Mark Adams 23 Jun '21

23 Jun '21

On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Use VecGetArrayRead/Write() to get up-to-date host pointers to the vector > array. > Humm, well HDF5 uses that now. I have to dig deeper. > > --Junchao Zhang > > > On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> First, there seem to be two pages for VecGetArrayAndMemType (one has a >> pointer to the other). >> >> So I need to get a CPU array for HDF5 viewing. Totally broken for devices. >> >> I don't find a VecGetArrayCpu[HOST] that does the right thing. >> >> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >> "mtype==NULL"? >> >> Mark >> >

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Matthew Knepley 23 Jun '21

23 Jun '21

On Wed, Jun 23, 2021 at 10:44 AM Junchao Zhang <junchao.zhang(a)gmail.com> wrote: > Use VecGetArrayRead/Write() to get up-to-date host pointers to the vector > array. > I think Mark is saying that those are not working. We do call VecGetArrayRead() in the HDF5 code. Mark, it seem like a small broken code is necessary. Thanks, Matt > --Junchao Zhang > > > On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: > >> First, there seem to be two pages for VecGetArrayAndMemType (one has a >> pointer to the other). >> >> So I need to get a CPU array for HDF5 viewing. Totally broken for devices. >> >> I don't find a VecGetArrayCpu[HOST] that does the right thing. >> >> Perhaps have VecGetArrayAndMemType return a valid CPU pointer when >> "mtype==NULL"? >> >> Mark >> > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] VecGetArrayAndMemType
by Junchao Zhang 23 Jun '21

23 Jun '21

Use VecGetArrayRead/Write() to get up-to-date host pointers to the vector array. --Junchao Zhang On Wed, Jun 23, 2021 at 9:15 AM Mark Adams <mfadams(a)lbl.gov> wrote: > First, there seem to be two pages for VecGetArrayAndMemType (one has a > pointer to the other). > > So I need to get a CPU array for HDF5 viewing. Totally broken for devices. > > I don't find a VecGetArrayCpu[HOST] that does the right thing. > > Perhaps have VecGetArrayAndMemType return a valid CPU pointer when > "mtype==NULL"? > > Mark >

1 0

VecGetArrayAndMemType
by Mark Adams 23 Jun '21

23 Jun '21

First, there seem to be two pages for VecGetArrayAndMemType (one has a pointer to the other). So I need to get a CPU array for HDF5 viewing. Totally broken for devices. I don't find a VecGetArrayCpu[HOST] that does the right thing. Perhaps have VecGetArrayAndMemType return a valid CPU pointer when "mtype==NULL"? Mark

1 0