April 2018 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] PetscOptions defaults after PetscInitialize
by Smith, Barry F. 22 Apr '18

22 Apr '18

> On Apr 22, 2018, at 7:41 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > "Smith, Barry F." <bsmith(a)mcs.anl.gov> writes: > >> Yuck, I think a far better user API is that PetscOptionsInsertFile() be callable before PetscInitialize(). The problem is that people have shoveled so much post-PetscInitialize() code into PetscOptionsInsertFile() over the years that stripping it all out would be painful. Maybe get a simplier pre- 2005 version of the routine and strip out the post-PetscInitialize() material? > > You want every rank to open the file independently? I forgot about that. > Or > PetscOptionsInsertFile somehow caches the file contents without using > PetscMalloc and broadcasts it after reaching PetscInitialize? That > seems a bit crazy. Maybe have use of pre-PetscInitialize() PetscOptionsInsertFile() cache the __file name(s)__ and then have PetscInitialize() loop over the cached filenames and load those options using simple MPI and PetscOptionsSetValue() before processing the command line etc Post-PetscInitialize() use of PetscOptionsInsertFile() would remain the same as today. We'd still need a stripped down version of PetscOptionsInsertFile(). Your proposed route allows bad decisions made years ago to dictate a bad user API forever. I don't like that. Barry > >> Barry >> >> >>> On Apr 22, 2018, at 5:54 PM, Jed Brown <jed(a)jedbrown.org> wrote: >>> >>> For users that read their own configuration files and/or choose >>> PetscOptionsInsertFile after PetscInitialize, we don't have a good way >>> to avoid overwriting PETSC_OPTIONS or command-line options. The user >>> could manually find argv and the environment variable, but that's a poor >>> abstraction. Should PetscOptionsInsertFile learn how to behave so as to >>> add new entries to the options database, but not supersede any that >>> already exist?

1 0

Re: [petsc-dev] PetscOptions defaults after PetscInitialize
by Matthew Knepley 22 Apr '18

22 Apr '18

On Sun, Apr 22, 2018 at 8:41 PM, Jed Brown <jed(a)jedbrown.org> wrote: > "Smith, Barry F." <bsmith(a)mcs.anl.gov> writes: > > > Yuck, I think a far better user API is that PetscOptionsInsertFile() > be callable before PetscInitialize(). The problem is that people have > shoveled so much post-PetscInitialize() code into PetscOptionsInsertFile() > over the years that stripping it all out would be painful. Maybe get a > simplier pre- 2005 version of the routine and strip out the > post-PetscInitialize() material? > > You want every rank to open the file independently? Or > PetscOptionsInsertFile somehow caches the file contents without using > PetscMalloc and broadcasts it after reaching PetscInitialize? That > seems a bit crazy. I am not for this "before PetcInitialize" strategy at all. However, I do think its far too fat. It initializes everything we can think of without giving the user and entry points in to customize it. That is compiler-level user disempowerment. I would rather see us rearchitect Init() so that you have better control over options, logging, CUDA, and all the other initializations hiding in there. Matt > > Barry > > > > > >> On Apr 22, 2018, at 5:54 PM, Jed Brown <jed(a)jedbrown.org> wrote: > >> > >> For users that read their own configuration files and/or choose > >> PetscOptionsInsertFile after PetscInitialize, we don't have a good way > >> to avoid overwriting PETSC_OPTIONS or command-line options. The user > >> could manually find argv and the environment variable, but that's a poor > >> abstraction. Should PetscOptionsInsertFile learn how to behave so as to > >> add new entries to the options database, but not supersede any that > >> already exist? > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>

1 0

Re: [petsc-dev] PetscOptions defaults after PetscInitialize
by Jed Brown 22 Apr '18

22 Apr '18

"Smith, Barry F." <bsmith(a)mcs.anl.gov> writes: > Yuck, I think a far better user API is that PetscOptionsInsertFile() be callable before PetscInitialize(). The problem is that people have shoveled so much post-PetscInitialize() code into PetscOptionsInsertFile() over the years that stripping it all out would be painful. Maybe get a simplier pre- 2005 version of the routine and strip out the post-PetscInitialize() material? You want every rank to open the file independently? Or PetscOptionsInsertFile somehow caches the file contents without using PetscMalloc and broadcasts it after reaching PetscInitialize? That seems a bit crazy. > Barry > > >> On Apr 22, 2018, at 5:54 PM, Jed Brown <jed(a)jedbrown.org> wrote: >> >> For users that read their own configuration files and/or choose >> PetscOptionsInsertFile after PetscInitialize, we don't have a good way >> to avoid overwriting PETSC_OPTIONS or command-line options. The user >> could manually find argv and the environment variable, but that's a poor >> abstraction. Should PetscOptionsInsertFile learn how to behave so as to >> add new entries to the options database, but not supersede any that >> already exist?

1 0

Re: [petsc-dev] PetscOptions defaults after PetscInitialize
by Smith, Barry F. 22 Apr '18

22 Apr '18

Yuck, I think a far better user API is that PetscOptionsInsertFile() be callable before PetscInitialize(). The problem is that people have shoveled so much post-PetscInitialize() code into PetscOptionsInsertFile() over the years that stripping it all out would be painful. Maybe get a simplier pre- 2005 version of the routine and strip out the post-PetscInitialize() material? Barry > On Apr 22, 2018, at 5:54 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > For users that read their own configuration files and/or choose > PetscOptionsInsertFile after PetscInitialize, we don't have a good way > to avoid overwriting PETSC_OPTIONS or command-line options. The user > could manually find argv and the environment variable, but that's a poor > abstraction. Should PetscOptionsInsertFile learn how to behave so as to > add new entries to the options database, but not supersede any that > already exist?

1 0

HDF5 without H5Pset_fapl_mpio
by Jed Brown 22 Apr '18

22 Apr '18

The current code just charges along if H5Pset_fapl_mpio is not available, but then H5Fcreate fails because ranks are fighting for a new file. #000: H5F.c line 491 in H5Fcreate(): unable to create file major: File accessibilty minor: Unable to open file #001: H5Fint.c line 1305 in H5F_open(): unable to lock the file major: File accessibilty minor: Unable to open file #002: H5FD.c line 1839 in H5FD_lock(): driver lock request failed major: Virtual File Layer minor: Can't update object #003: H5FDsec2.c line 940 in H5FD_sec2_lock(): unable to lock file, errno = 11, error message = 'Resource temporarily unavailable' major: File accessibilty minor: Bad file ID accessed Seems to me that if the size of the communicator is 1, we shouldn't bother trying H5Pset_fapl_mpio and if the comm is larger than 1, we need to *fail* with a sensible error message if H5Pset_fapl_mpio is not available. Does that sound right to the people that use HDF5 more heavily? #if defined(PETSC_HAVE_H5PSET_FAPL_MPIO) PetscStackCallHDF5(H5Pset_fapl_mpio,(plist_id, PetscObjectComm((PetscObject)viewer), info)); #endif /* Create or open the file collectively */ switch (hdf5->btype) { case FILE_MODE_READ: PetscStackCallHDF5Return(hdf5->file_id,H5Fopen,(name, H5F_ACC_RDONLY, plist_id)); break; case FILE_MODE_APPEND: PetscStackCallHDF5Return(hdf5->file_id,H5Fopen,(name, H5F_ACC_RDWR, plist_id)); break; case FILE_MODE_WRITE: PetscStackCallHDF5Return(hdf5->file_id,H5Fcreate,(name, H5F_ACC_TRUNC, H5P_DEFAULT, plist_id)); break; default: SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ORDER, "Must call PetscViewerFileSetMode() before PetscViewerFileSetName()"); }

1 0

PetscOptions defaults after PetscInitialize
by Jed Brown 22 Apr '18

22 Apr '18

For users that read their own configuration files and/or choose PetscOptionsInsertFile after PetscInitialize, we don't have a good way to avoid overwriting PETSC_OPTIONS or command-line options. The user could manually find argv and the environment variable, but that's a poor abstraction. Should PetscOptionsInsertFile learn how to behave so as to add new entries to the options database, but not supersede any that already exist?

1 0

Re: [petsc-dev] broken stuff in next for many days
by Satish Balay 22 Apr '18

22 Apr '18

On Sun, 22 Apr 2018, Smith, Barry F. wrote: > > It would be good to figure out exactly where the difference comes form > > [wrt chaco from trilinos vs separate install of chaco - or some other > > interaction from trilinos] > > Different random numbers being generated? Its trilinos vs non-trilinos build of packages - so I don't think its 'different random number generation' issue. Trilinos install does: if self.libraries.check(self.dlib, "interface"): self.addDefine('HAVE_CHACO',1) self.addDefine('HAVE_CHACO_INT_ASSIGNMENT',1) if self.libraries.check(self.dlib, "ML_Set_PrintLevel"): self.addDefine('HAVE_ML',1) if self.libraries.check(self.dlib, "Zoltan_LB_Partition"): self.addDefine('HAVE_ZOLTAN',1) if self.libraries.check(self.dlib, "ex_close"): self.addDefine('HAVE_EXODUSII',1) For one there is different chaco code in petsc based on this flag HAVE_CHACO_INT_ASSIGNMENT. Its not clear to me if this is causing the difference. --download-chaco is using 'Chaco-2.2-p2.tar.gz'. This matches the latest chaco tarball from their website [Chaco-2.2.tar.gz]. So I do not know if chaco packaged by trilinos has changes that cause this difference [clearly it added some change that resulted in us using the flag HAVE_CHACO_INT_ASSIGNMENT] And then there is also exodus.. --download-exodus uses the latest snapshot from https://github.com/gsjaardema/seacas/ Satish

1 0

Re: [petsc-dev] broken stuff in next for many days
by Smith, Barry F. 22 Apr '18

22 Apr '18

> On Apr 21, 2018, at 6:38 PM, Balay, Satish <balay(a)mcs.anl.gov> wrote: > > On Sat, 21 Apr 2018, Václav Hapla wrote: > >> Sorry Barry, that's my fault. >> >> Thanks a lot Satish for the quick fix with !trilinos. Hope it works now. >> >> I think the chaco partitioner should be definitely replaced by simple it these tests as Matt suggested. But I will rather push it within a separate PR since it extends the tested archs (by removing chaco and !trilinos requirements) so some previously unseen issues might appear theoretically. > > Yes - any changes should be a new PR. > > It would be good to figure out exactly where the difference comes form > [wrt chaco from trilinos vs separate install of chaco - or some other > interaction from trilinos] Different random numbers being generated? > > and we could keep chaco tests - and tailor them appropriately with > '!trilinos' or have multipe outout file [i.e _alt.out format] for > chaco tests - if thats appropriate. If both sets are correct than an _alt file is better than !trilinos in the long run. Barry > > Satish

1 0

Re: [petsc-dev] broken stuff in next for many days
by Satish Balay 21 Apr '18

21 Apr '18

On Sat, 21 Apr 2018, Václav Hapla wrote: > Sorry Barry, that's my fault. > > Thanks a lot Satish for the quick fix with !trilinos. Hope it works now. > > I think the chaco partitioner should be definitely replaced by simple it these tests as Matt suggested. But I will rather push it within a separate PR since it extends the tested archs (by removing chaco and !trilinos requirements) so some previously unseen issues might appear theoretically. Yes - any changes should be a new PR. It would be good to figure out exactly where the difference comes form [wrt chaco from trilinos vs separate install of chaco - or some other interaction from trilinos] and we could keep chaco tests - and tailor them appropriately with '!trilinos' or have multipe outout file [i.e _alt.out format] for chaco tests - if thats appropriate. Satish

1 0

Re: [petsc-dev] broken stuff in next for many days
by Václav Hapla 21 Apr '18

21 Apr '18

Sorry Barry, that's my fault. Thanks a lot Satish for the quick fix with !trilinos. Hope it works now. I think the chaco partitioner should be definitely replaced by simple it these tests as Matt suggested. But I will rather push it within a separate PR since it extends the tested archs (by removing chaco and !trilinos requirements) so some previously unseen issues might appear theoretically. Vaclav 21. dubna 2018 19:17:58 SELČ, Satish Balay <balay(a)mcs.anl.gov> napsal: >https://bitbucket.org/petsc/petsc/pull-requests/896/read-xdmf-style-hdf5-fi… > >Pushed the following [until it gets properly resolved] > >https://bitbucket.org/petsc/petsc/commits/2baebb300dfeb523617ec9e93c0d7046e… > >Satish > > >On Sat, 21 Apr 2018, Smith, Barry F. wrote: > >> >> >ftp://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2018/04/21/examples_nex… >> >> not ok diff-dm_impls_plex_tutorials-ex5_2a_nsize-8_format-hdf5_xdmf >> # 6,9c6,9 >> # < [0]: 7 ----> 3 >> # < [0]: 7 ----> 4 >> # < [0]: 7 ----> 5 >> # < [0]: 7 ----> 6 >> # --- >> # > [0]: 7 ----> 0 >> # > [0]: 7 ----> 1 >> # > [0]: 8 ----> 1 >> # > [0]: 8 ----> 2 >> # 11,13c11,16 >> # < [0]: 8 ----> 6 >> >> Is someone working on fixing this or does the branch need to be >abandoned and next cleaned, this has been haunting next for way to >long. >> >> Barry >> >>

1 0