December 2014 - petsc-dev - mailman3.cels.anl.gov

Fwd: [petsc-maint] [OMPI users] Deadlock in OpenMPI 1.8.3 and PETSc 3.4.5
by Barry Smith 19 Dec '14

19 Dec '14

> Begin forwarded message: > > Date: December 19, 2014 at 9:44:31 AM CST > From: George Bosilca <bosilca(a)icl.utk.edu> > To: Open MPI Users <users(a)open-mpi.org> > Cc: "petsc-maint(a)mcs.anl.gov" <petsc-maint(a)mcs.anl.gov> > Subject: Re: [petsc-maint] [OMPI users] Deadlock in OpenMPI 1.8.3 and PETSc 3.4.5 > > On Fri, Dec 19, 2014 at 8:58 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com <mailto:[email protected]>> wrote: > George: > > (I'm not a member of petsc-maint; I have no idea whether my mail will actually go through to that list) > > TL;DR: I do not think that George's change was correct. PETSC is relying on undefined behavior in the MPI standard and should probably update to use a different scheme. > > I will detail below why the change is correct or at least not worst than what we have today. > > Regarding your second point, while I do tend to agree that such issue is better addressed in the MPI Forum, the last attempt to fix this was certainly not a resounding success. > > > More detail: > > More specifically, George's change can lead to inconsistency/incorrectness in the presence of multiple threads simultaneously executing attribute actions on a single entity. > > Indeed, there is a slight window of opportunity for inconsistencies in the recursive behavior. But the inconsistencies were already in the code, especially in the single threaded case. As we never received any complaints related to this topic I did not deemed interesting to address them with my last commit. Moreover, the specific behavior needed by PETSc is available in Open MPI when compiled without thread support, as the only thing that "protects" the attributes is that global mutex. > > For example, in ompi_attr_delete_all(), it gets the count of all attributes and then loops <count> times to delete each attribute. But each attribute callback can now insert or delete attributes on that entity. This can mean that the loop can either fail to delete an attribute (because some attribute callback already deleted it) or fail to delete *all* attributes (because some attribute callback added more). > > To be extremely precise the deletion part is always correct as it copies the values to be deleted into a temporary array before calling any callbacks (and before releasing the mutex), so we only remove what was in the object attribute hash when the function was called. Don't misunderstand we have an extremely good reason to do it this way, we need to call the callbacks in the order in which they were created (mandated by the MPI standard). > > ompi_attr_copy_all() has similar problems -- in general, the hash that it is looping over can change underneath it. > > For the copy it is a little bit more tricky, as the calling order is not imposed. Our peculiar implementation of the hash table (with array) makes the code work, with a single (possible minor) exception when the hash table itself is grown between 2 calls. However, as stated before this issue was already present in the code in single threaded cases for years. Addressing it is another 2 line patch, but I leave this exercise to an interested reader. > > George. > > > This is, unfortunately, an undefined area of the MPI specification. I do believe that our previous behavior was *correct* -- it just deadlocks with PETSC because PETSC is relying on undefined behavior. > > For those who care, Microsoft/Cisco proposed a new attribute system to the Forum a while ago that removes all these kinds of ambiguities (see http://meetings.mpi-forum.org/secretary/2013/09/slides/jsquyres-attributes-… <http://meetings.mpi-forum.org/secretary/2013/09/slides/jsquyres-attributes-…>). However, we didn't get a huge amount of interest, and therefore lost our window of availability opportunity to be able to advance the proposal. I'd be more than happy to talk anyone through the proposal if they have interest/cycles in taking it over and advancing it with the Forum. > > Two additional points from the PDF listed above: > > - on slide 21, it was decided to no allow the recursive behavior (i.e., you can ignore the "This is under debate" bullet. > - the "destroy" callback was not judged to be useful; you can ignore slides 22 and 23. > > > On Dec 17, 2014, at 11:16 PM, George Bosilca <bosilca(a)icl.utk.edu <mailto:[email protected]>> wrote: > > > Ben, > > > > I can't find anything in the MPI standard suggesting that a recursive behavior of the attribute deletion is enforced/supported by the MPI standard. Thus, the current behavior of Open MPI (a single lock for all attributes), while maybe a little strict, is standard compliant (and thus correct). I guess this is one of these peculiar differences between different MPI libraries that makes other libraries (taking advantage of particular implementation of MPI functions) less portable. > > > > That being said, changing Open MPI to a less restrictive behavior is a 4 lines patch [1]. Unfortunately, this will not make your life easier, you will still have to cope with all the existing versions of Open MPI that do not support this recursive behavior. > > > > I leave the answer to the question of including this patch into the next release to our release managers. > > > > George. > > > > [1] https://github.com/open-mpi/ompi/commit/4d55ae838d5eff2818707da7ba60ffd6401… <https://github.com/open-mpi/ompi/commit/4d55ae838d5eff2818707da7ba60ffd6401…> > > > > > > On Wed, Dec 17, 2014 at 10:11 PM, Howard Pritchard <hppritcha(a)gmail.com <mailto:[email protected]>> wrote: > > Hi Ben, > > > > Would you mind checking if you still observe this deadlock condition if you use > > the 1.8.4 rc4 candidate? > > > > openmpi-1.8.4rc4.tar.gz > > > > I realize the behavior will likely be the same, but this is just to double check. > > > > The Open MPI man page for MPI_Attr_get (hmm.. no MPI_Comm_attr_get man page, > > needs to be fixed) says nothing about issues with recursion with respect to invoking > > this function within an attribute delete callback, so I would treat this as a bug. > > > > Thanks for your patience, > > > > Howard > > > > > > 2014-12-17 17:07 GMT-07:00 Ben Menadue <ben.menadue(a)nci.org.au <mailto:[email protected]>>: > > Hi PETSc and OpenMPI teams, > > > > I'm running into a deadlock in PETSc 3.4.5 with OpenMPI 1.8.3: > > > > 1. PetscCommDestroy calls MPI_Attr_delete > > 2. MPI_Attr_delete acquires a lock > > 3. MPI_Attr_delete calls Petsc_DelComm_Outer (through a callback) > > 4. Petsc_DelComm_Outer calls MPI_Attr_get > > 5. MPI_Attr_get wants to also acquire the lock from step 2. > > > > Looking at the OpenMPI source code, it looks like you can't call an > > MPI_Attr_* function from inside the registered deletion callback. The > > OpenMPI source code notes that all of the functions acquire a global lock, > > which is where the problem is coming from - here are the comments and the > > lock definition, in ompi/attribute/attribute.c of OpenMPI 1.8.3: > > > > 404 /* > > 405 * We used to have multiple locks for semi-fine-grained locking. > > But > > 406 * the code got complex, and we had to spend time looking for subtle > > 407 * bugs. Craziness -- MPI attributes are *not* high performance, so > > 408 * just use a One Big Lock approach: there is *no* concurrent > > access. > > 409 * If you have the lock, you can do whatever you want and no data > > will > > 410 * change/disapear from underneath you. > > 411 */ > > 412 static opal_mutex_t attribute_lock; > > > > To get it to work, I had to modify the definition of this lock to use a > > recursive mutex: > > > > 412 static opal_mutex_t attribute_lock = { .m_lock_pthread = > > PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP }; > > > > but this is non-portable. > > > > Is the behaviour expected from new versions OpenMPI? In which case a new > > approach might be needed in PETSc. Otherwise, maybe a per-attribute lock is > > needed in OpenMPI - but I'm not sure whether the get in the callback is on > > the same attribute as is being deleted. > > > > Thanks, > > Ben > > > > #0 0x00007fd7d5de4264 in __lll_lock_wait () from /lib64/libpthread.so.0 > > #1 0x00007fd7d5ddf508 in _L_lock_854 () from /lib64/libpthread.so.0 > > #2 0x00007fd7d5ddf3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 > > #3 0x00007fd7d27d91bc in ompi_attr_get_c () from > > /apps/openmpi/1.8.3/lib/libmpi.so.1 > > #4 0x00007fd7d2803f03 in PMPI_Attr_get () from > > /apps/openmpi/1.8.3/lib/libmpi.so.1 > > #5 0x00007fd7d7716006 in Petsc_DelComm_Outer (comm=0x7fd7d2a83b30, > > keyval=128, attr_val=0x7fff00a20f00, extra_state=0xffffffffffffffff) at > > pinit.c:406 > > #6 0x00007fd7d27d8cad in ompi_attr_delete_impl () from > > /apps/openmpi/1.8.3/lib/libmpi.so.1 > > #7 0x00007fd7d27d8f2f in ompi_attr_delete () from > > /apps/openmpi/1.8.3/lib/libmpi.so.1 > > #8 0x00007fd7d2803dfc in PMPI_Attr_delete () from > > /apps/openmpi/1.8.3/lib/libmpi.so.1 > > #9 0x00007fd7d78bf5c5 in PetscCommDestroy (comm=0x7fd7d2a83b30) at > > tagm.c:256 > > #10 0x00007fd7d7506f58 in PetscHeaderDestroy_Private (h=0x7fd7d2a83b30) at > > inherit.c:114 > > #11 0x00007fd7d75038a0 in ISDestroy (is=0x7fd7d2a83b30) at index.c:225 > > #12 0x00007fd7d75029b7 in PCReset_ILU (pc=0x7fd7d2a83b30) at ilu.c:42 > > #13 0x00007fd7d77a9baa in PCReset (pc=0x7fd7d2a83b30) at precon.c:81 > > #14 0x00007fd7d77a99ae in PCDestroy (pc=0x7fd7d2a83b30) at precon.c:117 > > #15 0x00007fd7d7557c1a in KSPDestroy (ksp=0x7fd7d2a83b30) at itfunc.c:788 > > #16 0x00007fd7d91cdcca in linearSystemPETSc<double>::~linearSystemPETSc > > (this=0x7fd7d2a83b30) at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Solver/li > > nearSystemPETSc.hpp:73 > > #17 0x00007fd7d8ddb63b in GFaceCompound::parametrize (this=0x7fd7d2a83b30, > > step=128, tom=10620672) at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Geo/GFace > > Compound.cpp:1672 > > #18 0x00007fd7d8dda0fe in GFaceCompound::parametrize (this=0x7fd7d2a83b30) > > at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Geo/GFace > > Compound.cpp:916 > > #19 0x00007fd7d8f98b0e in checkMeshCompound (gf=0x7fd7d2a83b30, edges=...) > > at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/mesh > > GFace.cpp:2588 > > #20 0x00007fd7d8f95c7e in meshGenerator (gf=0xd13020, RECUR_ITER=0, > > repairSelfIntersecting1dMesh=true, onlyInitialMesh=false, debug=false, > > replacement_edges=0x0) > > at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/mesh > > GFace.cpp:1075 > > #21 0x00007fd7d8f9a41e in meshGFace::operator() (this=0x7fd7d2a83b30, > > gf=0x80, print=false) at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/mesh > > GFace.cpp:2562 > > #22 0x00007fd7d8f8c327 in Mesh2D (m=0x7fd7d2a83b30) at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/Gene > > rator.cpp:407 > > #23 0x00007fd7d8f8ad0b in GenerateMesh (m=0x7fd7d2a83b30, ask=128) at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/Gene > > rator.cpp:641 > > #24 0x00007fd7d8e43126 in GModel::mesh (this=0x7fd7d2a83b30, dimension=128) > > at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Geo/GMode > > l.cpp:535 > > #25 0x00007fd7d8c1acd2 in GmshBatch () at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Common/Gm > > sh.cpp:240 > > #26 0x000000000040187a in main (argc=-760726736, argv=0x80) at > > /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Common/Ma > > in.cpp:27 > > > > > > _______________________________________________ > > users mailing list > > users(a)open-mpi.org <mailto:[email protected]> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users <http://www.open-mpi.org/mailman/listinfo.cgi/users> > > Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/26018.php <http://www.open-mpi.org/community/lists/users/2014/12/26018.php> > > > > _______________________________________________ > > users mailing list > > users(a)open-mpi.org <mailto:[email protected]> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users <http://www.open-mpi.org/mailman/listinfo.cgi/users> > > Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/26025.php <http://www.open-mpi.org/community/lists/users/2014/12/26025.php> > > > > _______________________________________________ > > users mailing list > > users(a)open-mpi.org <mailto:[email protected]> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users <http://www.open-mpi.org/mailman/listinfo.cgi/users> > > Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/26028.php <http://www.open-mpi.org/community/lists/users/2014/12/26028.php> > > > -- > Jeff Squyres > jsquyres(a)cisco.com <mailto:[email protected]> > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ <http://www.cisco.com/web/about/doing_business/legal/cri/> > > _______________________________________________ > users mailing list > users(a)open-mpi.org <mailto:[email protected]> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users <http://www.open-mpi.org/mailman/listinfo.cgi/users> > Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/26041.php <http://www.open-mpi.org/community/lists/users/2014/12/26041.php> >

1 0

Fwd: [petsc-maint] [OMPI users] Deadlock in OpenMPI 1.8.3 and PETSc 3.4.5
by Barry Smith 19 Dec '14

19 Dec '14

> Begin forwarded message: > > From: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > To: Open MPI User's List <users(a)open-mpi.org> > Date: December 19, 2014 at 8:05:19 AM CST > Cc: "petsc-maint(a)mcs.anl.gov" <petsc-maint(a)mcs.anl.gov> > Subject: Re: [petsc-maint] [OMPI users] Deadlock in OpenMPI 1.8.3 and PETSc 3.4.5 > > On Dec 19, 2014, at 8:58 AM, Jeff Squyres (jsquyres) <jsquyres(a)cisco.com> wrote: > >> More specifically, George's change can lead to inconsistency/incorrectness in the presence of multiple threads simultaneously executing attribute actions on a single entity. > > Actually -- it's worse than I first thought. This change can lead to inconsistencies/incorrectness even with a single thread, if that thread adds or removes attributes on the same entity from which a callback is currently being invoked. > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >

1 0

Fwd: [petsc-maint] [OMPI users] Deadlock in OpenMPI 1.8.3 and PETSc 3.4.5
by Barry Smith 19 Dec '14

19 Dec '14

> Begin forwarded message: > > From: "Jeff Squyres (jsquyres)" <jsquyres(a)cisco.com> > To: Open MPI User's List <users(a)open-mpi.org> > Date: December 19, 2014 at 7:58:17 AM CST > Cc: "petsc-maint(a)mcs.anl.gov" <petsc-maint(a)mcs.anl.gov> > Subject: Re: [petsc-maint] [OMPI users] Deadlock in OpenMPI 1.8.3 and PETSc 3.4.5 > > George: > > (I'm not a member of petsc-maint; I have no idea whether my mail will actually go through to that list) > > TL;DR: I do not think that George's change was correct. PETSC is relying on undefined behavior in the MPI standard and should probably update to use a different scheme. > > More detail: > > More specifically, George's change can lead to inconsistency/incorrectness in the presence of multiple threads simultaneously executing attribute actions on a single entity. > > For example, in ompi_attr_delete_all(), it gets the count of all attributes and then loops <count> times to delete each attribute. But each attribute callback can now insert or delete attributes on that entity. This can mean that the loop can either fail to delete an attribute (because some attribute callback already deleted it) or fail to delete *all* attributes (because some attribute callback added more). > > ompi_attr_copy_all() has similar problems -- in general, the hash that it is looping over can change underneath it. > > This is, unfortunately, an undefined area of the MPI specification. I do believe that our previous behavior was *correct* -- it just deadlocks with PETSC because PETSC is relying on undefined behavior. > > For those who care, Microsoft/Cisco proposed a new attribute system to the Forum a while ago that removes all these kinds of ambiguities (see http://meetings.mpi-forum.org/secretary/2013/09/slides/jsquyres-attributes-…) However, we didn't get a huge amount of interest, and therefore lost our window of availability opportunity to be able to advance the proposal. I'd be more than happy to talk anyone through the proposal if they have interest/cycles in taking it over and advancing it with the Forum. > > Two additional points from the PDF listed above: > > - on slide 21, it was decided to no allow the recursive behavior (i.e., you can ignore the "This is under debate" bullet. > - the "destroy" callback was not judged to be useful; you can ignore slides 22 and 23. > > > On Dec 17, 2014, at 11:16 PM, George Bosilca <bosilca(a)icl.utk.edu> wrote: > >> Ben, >> >> I can't find anything in the MPI standard suggesting that a recursive behavior of the attribute deletion is enforced/supported by the MPI standard. Thus, the current behavior of Open MPI (a single lock for all attributes), while maybe a little strict, is standard compliant (and thus correct). I guess this is one of these peculiar differences between different MPI libraries that makes other libraries (taking advantage of particular implementation of MPI functions) less portable. >> >> That being said, changing Open MPI to a less restrictive behavior is a 4 lines patch [1]. Unfortunately, this will not make your life easier, you will still have to cope with all the existing versions of Open MPI that do not support this recursive behavior. >> >> I leave the answer to the question of including this patch into the next release to our release managers. >> >> George. >> >> [1] https://github.com/open-mpi/ompi/commit/4d55ae838d5eff2818707da7ba60ffd6401… >> >> >> On Wed, Dec 17, 2014 at 10:11 PM, Howard Pritchard <hppritcha(a)gmail.com> wrote: >> Hi Ben, >> >> Would you mind checking if you still observe this deadlock condition if you use >> the 1.8.4 rc4 candidate? >> >> openmpi-1.8.4rc4.tar.gz >> >> I realize the behavior will likely be the same, but this is just to double check. >> >> The Open MPI man page for MPI_Attr_get (hmm.. no MPI_Comm_attr_get man page, >> needs to be fixed) says nothing about issues with recursion with respect to invoking >> this function within an attribute delete callback, so I would treat this as a bug. >> >> Thanks for your patience, >> >> Howard >> >> >> 2014-12-17 17:07 GMT-07:00 Ben Menadue <ben.menadue(a)nci.org.au>: >> Hi PETSc and OpenMPI teams, >> >> I'm running into a deadlock in PETSc 3.4.5 with OpenMPI 1.8.3: >> >> 1. PetscCommDestroy calls MPI_Attr_delete >> 2. MPI_Attr_delete acquires a lock >> 3. MPI_Attr_delete calls Petsc_DelComm_Outer (through a callback) >> 4. Petsc_DelComm_Outer calls MPI_Attr_get >> 5. MPI_Attr_get wants to also acquire the lock from step 2. >> >> Looking at the OpenMPI source code, it looks like you can't call an >> MPI_Attr_* function from inside the registered deletion callback. The >> OpenMPI source code notes that all of the functions acquire a global lock, >> which is where the problem is coming from - here are the comments and the >> lock definition, in ompi/attribute/attribute.c of OpenMPI 1.8.3: >> >> 404 /* >> 405 * We used to have multiple locks for semi-fine-grained locking. >> But >> 406 * the code got complex, and we had to spend time looking for subtle >> 407 * bugs. Craziness -- MPI attributes are *not* high performance, so >> 408 * just use a One Big Lock approach: there is *no* concurrent >> access. >> 409 * If you have the lock, you can do whatever you want and no data >> will >> 410 * change/disapear from underneath you. >> 411 */ >> 412 static opal_mutex_t attribute_lock; >> >> To get it to work, I had to modify the definition of this lock to use a >> recursive mutex: >> >> 412 static opal_mutex_t attribute_lock = { .m_lock_pthread = >> PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP }; >> >> but this is non-portable. >> >> Is the behaviour expected from new versions OpenMPI? In which case a new >> approach might be needed in PETSc. Otherwise, maybe a per-attribute lock is >> needed in OpenMPI - but I'm not sure whether the get in the callback is on >> the same attribute as is being deleted. >> >> Thanks, >> Ben >> >> #0 0x00007fd7d5de4264 in __lll_lock_wait () from /lib64/libpthread.so.0 >> #1 0x00007fd7d5ddf508 in _L_lock_854 () from /lib64/libpthread.so.0 >> #2 0x00007fd7d5ddf3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0 >> #3 0x00007fd7d27d91bc in ompi_attr_get_c () from >> /apps/openmpi/1.8.3/lib/libmpi.so.1 >> #4 0x00007fd7d2803f03 in PMPI_Attr_get () from >> /apps/openmpi/1.8.3/lib/libmpi.so.1 >> #5 0x00007fd7d7716006 in Petsc_DelComm_Outer (comm=0x7fd7d2a83b30, >> keyval=128, attr_val=0x7fff00a20f00, extra_state=0xffffffffffffffff) at >> pinit.c:406 >> #6 0x00007fd7d27d8cad in ompi_attr_delete_impl () from >> /apps/openmpi/1.8.3/lib/libmpi.so.1 >> #7 0x00007fd7d27d8f2f in ompi_attr_delete () from >> /apps/openmpi/1.8.3/lib/libmpi.so.1 >> #8 0x00007fd7d2803dfc in PMPI_Attr_delete () from >> /apps/openmpi/1.8.3/lib/libmpi.so.1 >> #9 0x00007fd7d78bf5c5 in PetscCommDestroy (comm=0x7fd7d2a83b30) at >> tagm.c:256 >> #10 0x00007fd7d7506f58 in PetscHeaderDestroy_Private (h=0x7fd7d2a83b30) at >> inherit.c:114 >> #11 0x00007fd7d75038a0 in ISDestroy (is=0x7fd7d2a83b30) at index.c:225 >> #12 0x00007fd7d75029b7 in PCReset_ILU (pc=0x7fd7d2a83b30) at ilu.c:42 >> #13 0x00007fd7d77a9baa in PCReset (pc=0x7fd7d2a83b30) at precon.c:81 >> #14 0x00007fd7d77a99ae in PCDestroy (pc=0x7fd7d2a83b30) at precon.c:117 >> #15 0x00007fd7d7557c1a in KSPDestroy (ksp=0x7fd7d2a83b30) at itfunc.c:788 >> #16 0x00007fd7d91cdcca in linearSystemPETSc<double>::~linearSystemPETSc >> (this=0x7fd7d2a83b30) at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Solver/li >> nearSystemPETSc.hpp:73 >> #17 0x00007fd7d8ddb63b in GFaceCompound::parametrize (this=0x7fd7d2a83b30, >> step=128, tom=10620672) at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Geo/GFace >> Compound.cpp:1672 >> #18 0x00007fd7d8dda0fe in GFaceCompound::parametrize (this=0x7fd7d2a83b30) >> at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Geo/GFace >> Compound.cpp:916 >> #19 0x00007fd7d8f98b0e in checkMeshCompound (gf=0x7fd7d2a83b30, edges=...) >> at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/mesh >> GFace.cpp:2588 >> #20 0x00007fd7d8f95c7e in meshGenerator (gf=0xd13020, RECUR_ITER=0, >> repairSelfIntersecting1dMesh=true, onlyInitialMesh=false, debug=false, >> replacement_edges=0x0) >> at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/mesh >> GFace.cpp:1075 >> #21 0x00007fd7d8f9a41e in meshGFace::operator() (this=0x7fd7d2a83b30, >> gf=0x80, print=false) at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/mesh >> GFace.cpp:2562 >> #22 0x00007fd7d8f8c327 in Mesh2D (m=0x7fd7d2a83b30) at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/Gene >> rator.cpp:407 >> #23 0x00007fd7d8f8ad0b in GenerateMesh (m=0x7fd7d2a83b30, ask=128) at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Mesh/Gene >> rator.cpp:641 >> #24 0x00007fd7d8e43126 in GModel::mesh (this=0x7fd7d2a83b30, dimension=128) >> at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Geo/GMode >> l.cpp:535 >> #25 0x00007fd7d8c1acd2 in GmshBatch () at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Common/Gm >> sh.cpp:240 >> #26 0x000000000040187a in main (argc=-760726736, argv=0x80) at >> /short/z00/bjm900/build/fluidity/intel15-ompi183/gmsh-2.8.5-source/Common/Ma >> in.cpp:27 >> >> >> _______________________________________________ >> users mailing list >> users(a)open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/26018.php >> >> _______________________________________________ >> users mailing list >> users(a)open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/26025.php >> >> _______________________________________________ >> users mailing list >> users(a)open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users >> Link to this post: http://www.open-mpi.org/community/lists/users/2014/12/26028.php > > > -- > Jeff Squyres > jsquyres(a)cisco.com > For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ >

1 0

Patch to fix vector allocation in ksp-fcg
by Sascha Schnepp 19 Dec '14

19 Dec '14

Hello developers, I prepared a patch to fix the allocation of direction vectors in KSPAllocateVectors_FCG within ksp-fcg. The fix consists only of replacing a PetscMin with a PetscMax. This prevents vectors being allocated one-by-one instead of chunksize of them once the preallocated ones (nprealloc) are exhausted. Let me know if you prefer a PR. Best, Sascha -------------------------------------- ETH Zurich Sascha Schnepp Institute of Geophysics - Geophysical Fluid Dynamics NO H23, Sonneggstrasse 5, 8092 Zurich, Switzerland schnepps(a)ethz.ch www.gfd.ethz.ch; www.saschaschnepp.net +41 44 632 0244 phone Consider encrypting your mail Public Key ID: 0x67AC24EF --------------------------------------

1 0

Re: [petsc-dev] Using petsc4py in notebooks
by Barry Smith 18 Dec '14

18 Dec '14

> On Nov 16, 2014, at 6:53 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > > Our output needs some overhauling to operate nicely here: > > a) ALL monitors need to take the type:file:format:mode form argument. -snes_monitor > does but -ksp_monitor does not > > b) PetscObjectViewFromOptions() has a static variable to shuts it off after any error > so long running processes are hosed. I would like to make any error in that short > function jump to a label which sets incall = PETSC_FALSE and then calls CHKERRQ fixed in next and barry/petscobjectviewfromoptions-resetincall > > c) The converged_reason options need to take the ::: argument > > Comments? > > Matt > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener

1 0

Fwd: [Bitbucket] Issue #84: pastix compilation error when custom compiler flags are set (petsc/petsc)
by Barry Smith 18 Dec '14

18 Dec '14

Satish, Can you update maint and master to use this new version (after testing in next of course :-) Barry > Begin forwarded message: > > Subject: Re: [Bitbucket] Issue #84: pastix compilation error when custom compiler flags are set (petsc/petsc) > From: XL <issues-reply(a)bitbucket.org> > To: <bsmith(a)mcs.anl.gov> > Date: December 18, 2014 at 8:57:38 AM CST > > > XL commented on issue #84: > pastix compilation error when custom compiler flags are set <https://bitbucket.org/petsc/petsc/issue/84/pastix-compilation-error-when-cu…> > I just released a new version that should fix the issue: https://gforge.inria.fr/frs/download.php/file/34392/pastix_5.2.2.20.tar.bz2 <https://gforge.inria.fr/frs/download.php/file/34392/pastix_5.2.2.20.tar.bz2>. > View this issue <https://bitbucket.org/petsc/petsc/issue/84/pastix-compilation-error-when-cu…> or add a comment by replying to this email. > Unsubscribe from issue emails <https://bitbucket.org/api/1.0/petsc/petsc/issue/84/unsubscribe/BarryFSmith/…> for this repository. <https://bitbucket.org/>

1 0

Re: [petsc-dev] MUMPS and 64 bit indices
by Barry Smith 17 Dec '14

17 Dec '14

Garth, I have found the bug and it is ours. In SuiteSparse.py we set ulong_max = 'ULONG_MAX' which is totally wrong because SuiteSparse is looking for the largest SIGNED long. So if you change the ULONG_MAX to LONG_MAX in SuiteSparse.py and rerun the ./configure (after deleting your $PETSC_DIR/$PETSC_ARCH directory) the 64 bit index version will run the same as the 32 bit version. I have fixed this in the branches maint and master Barry > On Dec 16, 2014, at 2:52 PM, Garth N. Wells <gnw20(a)cam.ac.uk> wrote: > > Attached is test code and matrix/vector files to reproduce a problem I'm seeing with Umfpack and 64-bit indices. It works fine with Umfpack and 32-bit indices. The matrix comes from a mixed formulation of the Poisson equation. The matrix is indefinite. I've computed all eigenvalues and they're nonzero. > > Running the test program with: > > ./test -pc_factor_mat_solver_package umfpack > > the output is > > Solution L2 norm: -nan > Solution L2 norm should be: 2.96824 > > Running the test program with > > ./test -pc_factor_mat_solver_package superlu_dist > > the output is > > Solution L2 norm: 2.968238 > Solution L2 norm should be: 2.96824 > > My PETSc configure line is > > --with-64-bit-indices --download-hypre=yes --download-suitesparse=1 --download-parmetis --download-metis --with-debugging=no COPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native" CXXOPTFLAGS="-O3 -march=native" --prefix=/home/garth/local/packages/petsc-64 --download-ptscotch --download-superlu_dist > > Git log is 7fbfed63fbc5325cf1eefdb967627c4417715c46 > > (message copied to petsc-maint so the attachment doesn't get scrubbed) > > Garth > > On Mon, 15 Dec, 2014 at 9:49 PM, Garth N. Wells <gnw20(a)cam.ac.uk> wrote: >>> On 15 Dec 2014, at 21:42, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>>> On Dec 15, 2014, at 3:16 PM, Garth N. Wells <gnw20(a)cam.ac.uk> wrote: >>>> It's possible to configure PETSc with the options >>>> --with-64-bit-indices --download-mumps >>>> and compile and run, but it doesn't look like the MUMPS interface supports 64 bit indices. Should the above combination throw an error at configure time? >>> Yes, there is commented code in the consistency check, I cannot remember why it was commented. >>> # if self.double and not self.scalartypes.precision.lower() == 'double': >>> # raise RuntimeError('Cannot use '+self.name+' withOUT double precision numbers, it is not coded for this capability') >>> # if not self.complex and self.scalartypes.scalartype.lower() == 'complex': >>> # raise RuntimeError('Cannot use '+self.name+' with complex numbers it is not coded for this capability') >>> # if self.libraryOptions.integerSize == 64 and self.requires32bitint: >>> # raise RuntimeError('Cannot use '+self.name+' with 64 bit integers, it is not coded for this capability') >>>> (On a side note, SuperLU_dist and Umfpack are flaky with 64 bit indices, which seems largely related to calling ParMETIS/METIS). >>> Hmm, never heard such reports. Are you using the --download-parmetis and metis? You should be. Is it reproducible problem? We'd like to fix it. >> Yes, I’m using --download-parmetis and --download-metis. >> Yes, it’s reproducible for a given matrix but hard to know which matrices will be a problem, and for which solver (SuperLU_dist/Umfpack). I’ll try to package up a couple of examples that fail or give wrong results. >> Garth >>> Barry >>>> Garth > <petsc_lu_test.tgz>

1 0

Re: [petsc-dev] Broken example ?
by Patrick Sanan 17 Dec '14

17 Dec '14

I can reproduce (on a laptop, OS X 10.9) Running in the debugger (and looking at a related error when running the second suggested command line option in ex55.c), a possible source of the problem is with the way the IS's are being set up. The direct cause of the segfault on my machine is that, in PCFieldSplitSetDefaults, jac->head is a null pointer. It seems to be unrelated to hypre (based on the fact that the same behavior is observed changing to -fieldsplit_0_pc_type lu). Pasted below: 1) LLDB running the first recommended settings 2) Error from these settings 3) Error from the second set of recommended options ======================================================= 1) $ lldb ./ex55 (lldb) target create "./ex55" Current executable set to './ex55' (x86_64). (lldb) r -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -fieldsplit_1_ksp_type fgmres -fieldsplit_1_pc_type lsc -snes_vi_monitor -ksp_monitor_true_residual -fieldsplit_ksp_monitor -fieldsplit_0_pc_type hypre -da_grid_x 65 -da_grid_y 65 -snes_atol 1.e-11 -ksp_rtol 1.e-8 Process 84592 launched: './ex55' (x86_64) 0 SNES VI Function norm 6.221605446769e-05 Active lower constraints 1928/2241 upper constraints 302/397 Percent of total 0.131953 Percent of bounded 0.175937 Process 84592 stopped * thread #1: tid = 0x8c484, 0x0000000101196734 libpetsc.3.05.dylib`PCFieldSplitSetDefaults(pc=0x0000000104351a60) + 6868 at fieldsplit.c:411, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x48) frame #0: 0x0000000101196734 libpetsc.3.05.dylib`PCFieldSplitSetDefaults(pc=0x0000000104351a60) + 6868 at fieldsplit.c:411 408 ierr = MatFindZeroDiagonals(pc->mat,&zerodiags);CHKERRQ(ierr); 409 ierr = ISComplement(zerodiags,nmin,nmax,&rest);CHKERRQ(ierr); 410 if (jac->reset) { -> 411 jac->head->is = rest; 412 jac->head->next->is = zerodiags; 413 } else { 414 ierr = PCFieldSplitSetIS(pc,"0",rest);CHKERRQ(ierr); (lldb) ======================================================= 2) $ ./ex55 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -fieldsplit_1_ksp_type fgmres -fieldsplit_1_pc_type lsc -snes_vi_monitor -ksp_monitor_true_residual -fieldsplit_ksp_monitor -fieldsplit_0_pc_type hypre -da_grid_x 65 -da_grid_y 65 -snes_atol 1.e-11 -ksp_rtol 1.e-8 0 SNES VI Function norm 6.221605446769e-05 Active lower constraints 1928/2241 upper constraints 302/397 Percent of total 0.131953 Percent of bounded 0.175937 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: likely location of problem given in stack below [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, [0]PETSC ERROR: INSTEAD the line number of the start of the function [0]PETSC ERROR: is given. [0]PETSC ERROR: [0] PCFieldSplitSetDefaults line 321 /Users/patrick/program/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: [0] PCSetUp_FieldSplit line 484 /Users/patrick/program/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: [0] KSPSetUp line 220 /Users/patrick/program/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: [0] SNESSolve_VINEWTONRSLS line 346 /Users/patrick/program/petsc/src/snes/impls/vi/rs/virs.c [0]PETSC ERROR: [0] SNESSolve line 3696 /Users/patrick/program/petsc/src/snes/interface/snes.c [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-1118-g64daeef GIT Date: 2014-12-11 04:32:01 -0600 [0]PETSC ERROR: ./ex55 on a arch-darwin-double-debug named Patricks-MacBook-Pro-3.local by patrick Tue Dec 16 11:19:48 2014 [0]PETSC ERROR: Configure options PETSC_DIR=/Users/patrick/program/petsc PETSC_ARCH=arch-darwin-double-debug --download-mpich --download-fblaslapack --with-cc=clang --with-cxx=clang++ --with-fc=gfortran-mp-4.9 --with-debugging=1 --with-precision=double --download-mumps --download-scalapack --download-metis --download-parmetis --download-spai --download-hypre --download-viennacl --with-opencl [0]PETSC ERROR: #1 User provided function() line 0 in unknown file application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 0 ======================================================= 3) $ ./ex55 -ksp_type fgmres -pc_type mg -mg_levels_ksp_type fgmres -mg_levels_pc_type fieldsplit -mg_levels_pc_fieldsplit_detect_saddle_point -mg_levels_pc_fieldsplit_type schur -mg_levels_pc_fieldsplit_factorization_type full -mg_levels_pc_fieldsplit_schur_precondition user -mg_levels_fieldsplit_1_ksp_type gmres -mg_levels_fieldsplit_1_pc_type none -mg_levels_fieldsplit_0_ksp_type preonly -mg_levels_fieldsplit_0_pc_type sor -mg_levels_fieldsplit_0_pc_sor_forward -snes_vi_monitor -ksp_monitor_true_residual -pc_mg_levels 5 -pc_mg_galerkin -mg_levels_ksp_monitor -mg_levels_fieldsplit_ksp_monitor -mg_levels_ksp_max_it 2 -mg_levels_fieldsplit_ksp_max_it 5 -snes_atol 1.e-11 -mg_coarse_ksp_type preonly -mg_coarse_pc_type svd -da_grid_x 65 -da_grid_y 65 -ksp_rtol 1.e-8 ............. SNESVI solver converged at t = 0.0000 in 8 iterations 0 SNES VI Function norm 6.499999997699e+01 Active lower constraints 12663/12663 upper constraints 0/0 Percent of total 0.74929 Percent of bounded 0.999053 [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Invalid argument [0]PETSC ERROR: Wrong type of object: Parameter # 2 [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.5.2-1118-g64daeef GIT Date: 2014-12-11 04:32:01 -0600 [0]PETSC ERROR: ./ex55 on a arch-darwin-double-debug named Patricks-MacBook-Pro-3.local by patrick Tue Dec 16 11:21:47 2014 [0]PETSC ERROR: Configure options PETSC_DIR=/Users/patrick/program/petsc PETSC_ARCH=arch-darwin-double-debug --download-mpich --download-fblaslapack --with-cc=clang --with-cxx=clang++ --with-fc=gfortran-mp-4.9 --with-debugging=1 --with-precision=double --download-mumps --download-scalapack --download-metis --download-parmetis --download-spai --download-hypre --download-viennacl --with-opencl [0]PETSC ERROR: #1 MatGetSubMatrix() line 7482 in /Users/patrick/program/petsc/src/mat/interface/matrix.c [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 552 in /Users/patrick/program/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c [0]PETSC ERROR: #3 PCSetUp() line 902 in /Users/patrick/program/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #4 KSPSetUp() line 306 in /Users/patrick/program/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #5 PCSetUp_MG() line 758 in /Users/patrick/program/petsc/src/ksp/pc/impls/mg/mg.c [0]PETSC ERROR: #6 PCSetUp() line 902 in /Users/patrick/program/petsc/src/ksp/pc/interface/precon.c [0]PETSC ERROR: #7 KSPSetUp() line 306 in /Users/patrick/program/petsc/src/ksp/ksp/interface/itfunc.c [0]PETSC ERROR: #8 SNESSolve_VINEWTONRSLS() line 506 in /Users/patrick/program/petsc/src/snes/impls/vi/rs/virs.c [0]PETSC ERROR: #9 SNESSolve() line 3743 in /Users/patrick/program/petsc/src/snes/interface/snes.c [0]PETSC ERROR: #10 main() line 108 in /Users/patrick/program/petsc/src/snes/examples/tutorials/ex55.c [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint(a)mcs.anl.gov---------- application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 [unset]: aborting job: application called MPI_Abort(MPI_COMM_WORLD, 62) - process 0 On 16/12/14 05:22 AM, Pierre Jolivet wrote: > When trying to run snes/examples/tutorials/ex55 with the command line given in the source code, i.e. > $ ./ex55 -ksp_type fgmres -pc_type fieldsplit -pc_fieldsplit_detect_saddle_point -pc_fieldsplit_type schur -pc_fieldsplit_schur_precondition self -fieldsplit_1_ksp_type fgmres -fieldsplit_1_pc_type lsc -snes_vi_monitor -ksp_monitor_true_residual -fieldsplit_ksp_monitor -fieldsplit_0_pc_type hypre -da_grid_x 65 -da_grid_y 65 -snes_atol 1.e-11 -ksp_rtol 1.e-8 > I end up with a crash with these lines in the log: > [0]PETSC ERROR: --------------------- Stack Frames ------------------------------------ > [0]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, > [0]PETSC ERROR: INSTEAD the line number of the start of the function > [0]PETSC ERROR: is given. > [0]PETSC ERROR: [0] PCFieldSplitSetDefaults line 321petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: [0] PCSetUp_FieldSplit line 484 petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c > [0]PETSC ERROR: [0] KSPSetUp line 220 petsc/src/ksp/ksp/interface/itfunc.c > [0]PETSC ERROR: [0] SNESSolve_VINEWTONRSLS line 346 petsc/src/snes/impls/vi/rs/virs.c > [0]PETSC ERROR: [0] SNESSolve line 3696 petsc/src/snes/interface/snes.c > > This is happening on two computers (a MBP and a Linux, on the master branch), both have —download-hypre, am I the only one having this error ? Please let me know if so, I’ll send you both configure.log if need be. > > Thank you, > Pierre

1 0

Re: [petsc-dev] MUMPS and 64 bit indices
by Barry Smith 16 Dec '14

16 Dec '14

I've run the 64 bit version under valgrind and it detected no problems. I also ran with -mat_umfpack_prl 10 to get additional information and it appears to be using a different ordering with 64 bit indices (I don't know why it is, need to look at the code in more detail). My current conclusion is that there is nothing "buggy" with the code, it is just using a different ordering that results in zero pivots with 64 bit indices than when running with 32 bit indices. This goes back to your question about sparse factorizations, they can generally always fail with zero pivots (for some matrices) depending on what ordering they use unless they do very aggressive numerical pivoting which can slow them down a great deal. I will try to see why it is using a different ordering in the 64 bit case. Barry > On Dec 16, 2014, at 2:52 PM, Garth N. Wells <gnw20(a)cam.ac.uk> wrote: > > Attached is test code and matrix/vector files to reproduce a problem I'm seeing with Umfpack and 64-bit indices. It works fine with Umfpack and 32-bit indices. The matrix comes from a mixed formulation of the Poisson equation. The matrix is indefinite. I've computed all eigenvalues and they're nonzero. > > Running the test program with: > > ./test -pc_factor_mat_solver_package umfpack > > the output is > > Solution L2 norm: -nan > Solution L2 norm should be: 2.96824 > > Running the test program with > > ./test -pc_factor_mat_solver_package superlu_dist > > the output is > > Solution L2 norm: 2.968238 > Solution L2 norm should be: 2.96824 > > My PETSc configure line is > > --with-64-bit-indices --download-hypre=yes --download-suitesparse=1 --download-parmetis --download-metis --with-debugging=no COPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native" CXXOPTFLAGS="-O3 -march=native" --prefix=/home/garth/local/packages/petsc-64 --download-ptscotch --download-superlu_dist > > Git log is 7fbfed63fbc5325cf1eefdb967627c4417715c46 > > (message copied to petsc-maint so the attachment doesn't get scrubbed) > > Garth > > On Mon, 15 Dec, 2014 at 9:49 PM, Garth N. Wells <gnw20(a)cam.ac.uk> wrote: >>> On 15 Dec 2014, at 21:42, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>>> On Dec 15, 2014, at 3:16 PM, Garth N. Wells <gnw20(a)cam.ac.uk> wrote: >>>> It's possible to configure PETSc with the options >>>> --with-64-bit-indices --download-mumps >>>> and compile and run, but it doesn't look like the MUMPS interface supports 64 bit indices. Should the above combination throw an error at configure time? >>> Yes, there is commented code in the consistency check, I cannot remember why it was commented. >>> # if self.double and not self.scalartypes.precision.lower() == 'double': >>> # raise RuntimeError('Cannot use '+self.name+' withOUT double precision numbers, it is not coded for this capability') >>> # if not self.complex and self.scalartypes.scalartype.lower() == 'complex': >>> # raise RuntimeError('Cannot use '+self.name+' with complex numbers it is not coded for this capability') >>> # if self.libraryOptions.integerSize == 64 and self.requires32bitint: >>> # raise RuntimeError('Cannot use '+self.name+' with 64 bit integers, it is not coded for this capability') >>>> (On a side note, SuperLU_dist and Umfpack are flaky with 64 bit indices, which seems largely related to calling ParMETIS/METIS). >>> Hmm, never heard such reports. Are you using the --download-parmetis and metis? You should be. Is it reproducible problem? We'd like to fix it. >> Yes, I’m using --download-parmetis and --download-metis. >> Yes, it’s reproducible for a given matrix but hard to know which matrices will be a problem, and for which solver (SuperLU_dist/Umfpack). I’ll try to package up a couple of examples that fail or give wrong results. >> Garth >>> Barry >>>> Garth > <petsc_lu_test.tgz>

1 0

Re: [petsc-dev] MPICH no longer reliable
by Satish Balay 16 Dec '14

16 Dec '14

https://trac.mpich.org/projects/mpich/ticket/2213 satish On Tue, 16 Dec 2014, Satish Balay wrote: > I'll report with the current details.. > > Satish > > On Tue, 16 Dec 2014, Barry Smith wrote: > > > > > Are the MPICH folks aware that MPICH doesn't successfully run on over 5 % of the systems users have today? Do they care? > > > > Barry > > > > > >

1 0