March 2021 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Tag name v3.15
by Satish Balay 31 Mar '21

31 Mar '21

>>>> https://semver.org/ Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when you make incompatible API changes, MINOR version when you add functionality in a backwards compatible manner, and PATCH version when you make backwards compatible bug fixes. <<<< For one - we don't strictly use semantic versions. - Our PATCH versions have (some) new functionality [from above that should be MINOR] - Our MINOR versions have lot more new functionality, with some incompatible API changes [this should be MAJOR - but we haven't changed that in a while] Wrt v3.15.0 vs v3.15 I don't remember the exact reason. I suspect we just stuck with this notation for a long time [and didn't want to change it] Also Barry still wants to use 'release' term only with v3.15 - and 3.15.1 etc as updates [but not releases] Satish On Wed, 31 Mar 2021, Lisandro Dalcin wrote: > Satish, if we use semantic versioning for the PETSc version itself and the > tarball, why don't we use the same format for tags? In short, why aren't we > using v3.15.0 instead of v3.15 ? > >

1 0

Re: [petsc-dev] [petsc-users] [EXTERNAL] Re: Question about periodic conditions
by Matthew Knepley 31 Mar '21

31 Mar '21

On Wed, Mar 31, 2021 at 6:40 AM Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > Am 31.03.2021 um 12:11 schrieb Matthew Knepley <knepley(a)gmail.com>: > > On Wed, Mar 31, 2021 at 3:21 AM Patrick Sanan <patrick.sanan(a)gmail.com> > wrote: > >> (moving to petsc-dev) >> >> To follow up further on this, Matt is correct as to what's happening now, >> but periodic coordinates aren't sufficiently supported yet in DMStag, so I >> will add something. >> >> The way things are set up now has a conceptual elegance to it, in that to >> define coordinates, you use another DM which has coordinate information on >> it, instead of other field information. It's periodic iff the primary DM >> is. So there is no point on the right boundary, at 2 * pi in the 1D version >> of this example, because that point would be identical to the point at 0, >> on the left boundary. >> >> The problem with the current implementation (for DMStag) is that the >> right boundary of the domain [0, 2*pi) is never stored. There's no way to >> know the width of the last cell on the right. You need that information for >> at least two important reasons: >> 1. to visualize the mesh, where even though the boundary point is the >> same point on the torus, you are plotting it on the plane and want >> different representations of the point on the left and right. >> 2. to use PIC methods (DMSwarm), where we need a way to determine if a >> particle is in the last cell. >> >> Matt, Mark, Dave, et. al, it'd be very helpful to know if the following >> seems like a good/bad idea to you, since I assume you resolved this same >> issue for DMPlex + DMSwarm: >> >> A tempting way to proceed here is to use the existing DMSetPeriodicity(), >> which allows you to specify that missing piece of information and store it >> in the DM. This could be called from the DMStagSetUniformCoordinatesXXX() >> functions, so the user wouldn't have to worry about it in that case. That >> also makes conceptual sense as that's the stage, after setup, in which you >> specify the "embedding" part of the DM. A next step would be to make >> DMLocalizeCoordinates() work for DMStag (and DMDA if possible, while I'm at >> it). >> > > That is what I added that stuff for. In Plex, in order to generalize to > situations not in a box, we went to a formulation that uses a DG coordinate > field instead. I think that > is overkill here and would not give you any added functionality. > > Ah, cool, I was wondering what the coordinate field function was for. Does > the DMSetPeriodicity() stuff get used anymore, now that you have a > different approach? Quickly looking it seems like it gets passed around as > you create DMs from existing ones, and it's perhaps used in some Plex > output functionality. > We use it to identify that the mesh is periodic and in what directions, and use the length if we have to figure out the coordinates. > Jed may argue that he wants you to retain the far point and use L2G to > eliminate it, but that sounds like a lot more work. > > Computational work or implementation work? > implementation. > I already have logic in DMStag to support something related, which was > required for L2G with INSERT_VALUES in the periodic, 1-rank case, where > multiple local points can respond to the same global point. So building the > local coordinates with the far point and then doing L2G doesn't sound bad > to implement, at least. > > https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMSTAG/DMStagP… > That may work. It is a problem in Plex because that strategy destroys the topological queries. However, it sounds like you do not have that problem. Thanks, Matt > > THanks, > > Matt > > >> Am 25.03.2021 um 00:20 schrieb Matthew Knepley <knepley(a)gmail.com>: >> >> On Wed, Mar 24, 2021 at 7:17 PM Jorti, Zakariae via petsc-users < >> petsc-users(a)mcs.anl.gov> wrote: >> >>> Hi Patrick, >>> >>> >>> Thanks for your responses. >>> >>> As for the code, I was not granted permission to share it yet. So, I >>> cannot send it to you for the moment. I apologize for that. >>> >>> >>> I wanted to let you know that while I was testing my code, I discovered >>> that when the periodic boundary conditions are activated, the coordinates >>> accessed might be incorrect on one side of the boundary. >>> >>> Let me give you an example in cylindrical coordinates with a 3x3x3 >>> DMStag mesh: >>> >>> >>> >>> >>> PetscInt startr,startphi,startz,nr,nphi,nz,d; >>> >>> PetscInt er,ephi,ez,icErmphip[3]; >>> >>> DM dmCoorda, coordDA; >>> >>> Vec coordaLocal; >>> >>> PetscScalar ****arrCoord; >>> >>> PetscScalar surf; >>> >>> >>> >>> DMStagCreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,3,3,3,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,1,1,DMSTAG_STENCIL_BOX,1,NULL,NULL,NULL,&coordDA); >>> >>> >>> DMSetFromOptions(coordDA); >>> >>> DMSetUp(coordDA); >>> >>> >>> >>> DMStagGetCorners(coordDA,&startr,&startphi,&startz,&nr,&nphi,&nz,NULL,NULL,NULL); >>> >>> >>> DMGetCoordinateDM(coordDA,&dmCoorda); >>> >>> DMGetCoordinatesLocal(coordDA,&coordaLocal); >>> >>> DMStagVecGetArrayRead(dmCoorda,coordaLocal,&arrCoord); >>> >>> >>> for (d=0; d< 3; ++d){ >>> >>> DMStagGetLocationSlot(dmCoorda,UP_LEFT,d,&icErmphip[d]); >>> >>> } >>> >>> >>> er = 1; ez = 0; >>> >>> for (ephi=0; ephi< 3; ++ephi){ >>> >>> PetscPrintf(PETSC_COMM_WORLD,"Phi_p(%d,%d,%d) = %E\n",er,ephi,ez >>> ,(double)arrCoord[ez][ephi][er][icErmphip[1]); >>> >>> } >>> >>> >>> When I execute this example, I get this output: >>> >>> Phi_p(1,0,0) = 2.094395E+00 >>> >>> Phi_p(1,1,0) = 4.188790E+00 >>> >>> Phi_p(1,2,0) = 0.000000E+00 >>> >>> >>> Note here that the first two lines correspond to 2π / 3 and 4π / 3 >>> respectively. Thus, nothing is wrong here. >>> >>> But the last line should rather give 2π instead of 0. >>> >>> >>> I understand that degrees of freedom should be the same on both sides of >>> the boundary, but should the coordinates not be preserved? >>> >>> I don't think so. The circle has coordinates in [0, 2\pi), so the point >> at 2\pi is identified with the point at 0 and you must choose >> one, so we choose 0. >> >> Thanks, >> >> Matt >> >>> Thank you. >>> >>> Best regards, >>> >>> >>> Zakariae Jorti >>> ------------------------------ >>> *From:* Patrick Sanan <patrick.sanan(a)gmail.com> >>> *Sent:* Tuesday, March 23, 2021 11:37:04 AM >>> *To:* Jorti, Zakariae >>> *Cc:* petsc-users(a)mcs.anl.gov >>> *Subject:* [EXTERNAL] Re: Question about periodic conditions >>> >>> Hi Zakariae - sorry about the delay - responses inline below. >>> >>> I'd be curious to see your code (which you can send directly to me if >>> you don't want to post it publicly), so I can give you more comments, as >>> DMStag is a new component. >>> >>> >>> Am 23.03.2021 um 00:54 schrieb Jorti, Zakariae <zjorti(a)lanl.gov>: >>> >>> Hi, >>> >>> I implemented a PETSc code to solve Maxwell's equations for the magnetic >>> and electric fields (B and E) in a cylinder: >>> 0 < r_min <= r <= r_max; with r_max > r_min >>> phi_min = 0 <= r <= phi_max = 2 π >>> z_min <= z =< z_max; with z_max > z_min. >>> >>> I am using a PETSc staggered grid with the electric field E defined on >>> edge centers and the magnetic field B defined on face centers. (dof0 = >>> 0, dof1 = 1,dof2 = 1, dof3 = 0;). >>> >>> >>> I have two versions of my code: >>> 1 - A first version in which I set the boundary type to >>> DM_BOUNDARY_NONE in the three directions r, phi and z >>> 2- A second version in which I set the boundary type to >>> DM_BOUNDARY_NONE in the r and z directions, and DM_BOUNDARY_PERIODIC in >>> the phi direction. >>> >>> When I print the solution vector X, which contains both E and B >>> components, I notice that the vector is shorter with the second version >>> compared to the first one. >>> Is it normal? >>> >>> Yes - with the periodic boundary conditions, there will be fewer points >>> since there won't be the "extra" layer of faces and edges at phi = 2 * pi . >>> >>> If you consider a 1-d example with 1 dof on vertices and cells, with >>> three elements, the periodic case looks like this, globally, >>> >>> x ---- x ---- x ---- >>> >>> as opposed to the non-periodic case, >>> >>> x ---- x ---- x ---- x >>> >>> >>> >>> Besides, I was wondering if I have to change the way I define the value >>> of the solution on the boundary. What I am doing so far in both versions is >>> something like: >>> B_phi [phi = 0] = 1.0; >>> B_phi [phi = 2π] = 1.0; >>> E_z [r, phi = 0] = 1/r; >>> E_z [r, phi = 2π] = 1/r; >>> >>> Assuming that values at phi = 0 should be the same as at phi=2π with >>> the periodic boundary conditions, is it sufficient for example to have only >>> the following boundary conditions: >>> B_phi [phi = 0] = 1.0; >>> >>> E_z [r, phi = 0] = 1/r ? >>> >>> >>> Yes - this is the intention, since the boundary at phi = 2 * pi is >>> represented by the same entries in the global vector. >>> >>> Of course, you need to make sure that your continuous problem is >>> well-posed, which in general could change when using different boundary >>> conditions. >>> >>> Thank you. >>> Best regards, >>> >>> Zakariae Jorti >>> >>> >>> >> >> -- >> What most experimenters take for granted before they begin their >> experiments is infinitely more interesting than any results to which their >> experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ >> <http://www.cse.buffalo.edu/~knepley/> >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] [petsc-users] [EXTERNAL] Re: Question about periodic conditions
by Patrick Sanan 31 Mar '21

31 Mar '21

> Am 31.03.2021 um 12:11 schrieb Matthew Knepley <knepley(a)gmail.com>: > > On Wed, Mar 31, 2021 at 3:21 AM Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>> wrote: > (moving to petsc-dev) > > To follow up further on this, Matt is correct as to what's happening now, but periodic coordinates aren't sufficiently supported yet in DMStag, so I will add something. > > The way things are set up now has a conceptual elegance to it, in that to define coordinates, you use another DM which has coordinate information on it, instead of other field information. It's periodic iff the primary DM is. So there is no point on the right boundary, at 2 * pi in the 1D version of this example, because that point would be identical to the point at 0, on the left boundary. > > The problem with the current implementation (for DMStag) is that the right boundary of the domain [0, 2*pi) is never stored. There's no way to know the width of the last cell on the right. You need that information for at least two important reasons: > 1. to visualize the mesh, where even though the boundary point is the same point on the torus, you are plotting it on the plane and want different representations of the point on the left and right. > 2. to use PIC methods (DMSwarm), where we need a way to determine if a particle is in the last cell. > > Matt, Mark, Dave, et. al, it'd be very helpful to know if the following seems like a good/bad idea to you, since I assume you resolved this same issue for DMPlex + DMSwarm: > > A tempting way to proceed here is to use the existing DMSetPeriodicity(), which allows you to specify that missing piece of information and store it in the DM. This could be called from the DMStagSetUniformCoordinatesXXX() functions, so the user wouldn't have to worry about it in that case. That also makes conceptual sense as that's the stage, after setup, in which you specify the "embedding" part of the DM. A next step would be to make DMLocalizeCoordinates() work for DMStag (and DMDA if possible, while I'm at it). > > That is what I added that stuff for. In Plex, in order to generalize to situations not in a box, we went to a formulation that uses a DG coordinate field instead. I think that > is overkill here and would not give you any added functionality. > Ah, cool, I was wondering what the coordinate field function was for. Does the DMSetPeriodicity() stuff get used anymore, now that you have a different approach? Quickly looking it seems like it gets passed around as you create DMs from existing ones, and it's perhaps used in some Plex output functionality. > Jed may argue that he wants you to retain the far point and use L2G to eliminate it, but that sounds like a lot more work. Computational work or implementation work? I already have logic in DMStag to support something related, which was required for L2G with INSERT_VALUES in the periodic, 1-rank case, where multiple local points can respond to the same global point. So building the local coordinates with the far point and then doing L2G doesn't sound bad to implement, at least. https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMSTAG/DMStagP… <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMSTAG/DMStagP…> > > THanks, > > Matt > >> Am 25.03.2021 um 00:20 schrieb Matthew Knepley <knepley(a)gmail.com <mailto:[email protected]>>: >> >> On Wed, Mar 24, 2021 at 7:17 PM Jorti, Zakariae via petsc-users <petsc-users(a)mcs.anl.gov <mailto:[email protected]>> wrote: >> Hi Patrick, >> >> >> >> Thanks for your responses. >> >> As for the code, I was not granted permission to share it yet. So, I cannot send it to you for the moment. I apologize for that. >> >> >> >> I wanted to let you know that while I was testing my code, I discovered that when the periodic boundary conditions are activated, the coordinates accessed might be incorrect on one side of the boundary. >> >> Let me give you an example in cylindrical coordinates with a 3x3x3 DMStag mesh: >> >> >> >> >> >> >> PetscInt startr,startphi,startz,nr,nphi,nz,d; >> >> >> PetscInt er,ephi,ez,icErmphip[3]; >> >> >> DM dmCoorda, coordDA; >> >> Vec coordaLocal; >> >> PetscScalar ****arrCoord; >> >> PetscScalar surf; >> >> >> >> DMStagCreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,3,3,3,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,1,1,DMSTAG_STENCIL_BOX,1,NULL,NULL,NULL,&coordDA); >> >> >> >> DMSetFromOptions(coordDA); >> >> DMSetUp(coordDA); >> >> >> >> DMStagGetCorners(coordDA,&startr,&startphi,&startz,&nr,&nphi,&nz,NULL,NULL,NULL); >> >> >> >> DMGetCoordinateDM(coordDA,&dmCoorda); >> >> DMGetCoordinatesLocal(coordDA,&coordaLocal); >> >> DMStagVecGetArrayRead(dmCoorda,coordaLocal,&arrCoord); >> >> >> >> for (d=0; d< 3; ++d){ >> >> DMStagGetLocationSlot(dmCoorda,UP_LEFT,d,&icErmphip[d]); >> >> } >> >> >> >> er = 1; ez = 0; >> >> for (ephi=0; ephi< 3; ++ephi){ >> >> PetscPrintf(PETSC_COMM_WORLD,"Phi_p(%d,%d,%d) = %E\n",er,ephi,ez,(double)arrCoord[ez][ephi][er][icErmphip[1]); >> >> } >> >> >> >> When I execute this example, I get this output: >> >> Phi_p(1,0,0) = 2.094395E+00 >> >> Phi_p(1,1,0) = 4.188790E+00 >> >> Phi_p(1,2,0) = 0.000000E+00 >> >> >> >> Note here that the first two lines correspond to 2π / 3 and 4π / 3 respectively. Thus, nothing is wrong here. >> >> But the last line should rather give 2π instead of 0. >> >> >> >> I understand that degrees of freedom should be the same on both sides of the boundary, but should the coordinates not be preserved? >> >> >> I don't think so. The circle has coordinates in [0, 2\pi), so the point at 2\pi is identified with the point at 0 and you must choose >> one, so we choose 0. >> >> Thanks, >> >> Matt >> Thank you. >> >> Best regards, >> >> >> >> Zakariae Jorti >> >> From: Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>> >> Sent: Tuesday, March 23, 2021 11:37:04 AM >> To: Jorti, Zakariae >> Cc: petsc-users(a)mcs.anl.gov <mailto:[email protected]> >> Subject: [EXTERNAL] Re: Question about periodic conditions >> >> Hi Zakariae - sorry about the delay - responses inline below. >> >> I'd be curious to see your code (which you can send directly to me if you don't want to post it publicly), so I can give you more comments, as DMStag is a new component. >> >> >>> Am 23.03.2021 um 00:54 schrieb Jorti, Zakariae <zjorti(a)lanl.gov <mailto:[email protected]>>: >>> >>> Hi, >>> >>> I implemented a PETSc code to solve Maxwell's equations for the magnetic and electric fields (B and E) in a cylinder: >>> 0 < r_min <= r <= r_max; with r_max > r_min >>> phi_min = 0 <= r <= phi_max = 2 π >>> z_min <= z =< z_max; with z_max > z_min. >>> >>> I am using a PETSc staggered grid with the electric field E defined on edge centers and the magnetic field B defined on face centers. (dof0 = 0, dof1 = 1,dof2 = 1, dof3 = 0;). >>> >>> I have two versions of my code: >>> 1 - A first version in which I set the boundary type to DM_BOUNDARY_NONE in the three directions r, phi and z >>> 2- A second version in which I set the boundary type to DM_BOUNDARY_NONE in the r and z directions, and DM_BOUNDARY_PERIODIC in the phi direction. >>> >>> When I print the solution vector X, which contains both E and B components, I notice that the vector is shorter with the second version compared to the first one. >>> Is it normal? >> Yes - with the periodic boundary conditions, there will be fewer points since there won't be the "extra" layer of faces and edges at phi = 2 * pi . >> >> If you consider a 1-d example with 1 dof on vertices and cells, with three elements, the periodic case looks like this, globally, >> >> x ---- x ---- x ---- >> >> as opposed to the non-periodic case, >> >> x ---- x ---- x ---- x >> >> >>> >>> Besides, I was wondering if I have to change the way I define the value of the solution on the boundary. What I am doing so far in both versions is something like: >>> B_phi [phi = 0] = 1.0; >>> B_phi [phi = 2π] = 1.0; >>> E_z [r, phi = 0] = 1/r; >>> E_z [r, phi = 2π] = 1/r; >>> >>> Assuming that values at phi = 0 should be the same as at phi=2π with the periodic boundary conditions, is it sufficient for example to have only the following boundary conditions: >>> B_phi [phi = 0] = 1.0; >>> E_z [r, phi = 0] = 1/r ? >> >> Yes - this is the intention, since the boundary at phi = 2 * pi is represented by the same entries in the global vector. >> >> Of course, you need to make sure that your continuous problem is well-posed, which in general could change when using different boundary conditions. >> >>> Thank you. >>> Best regards, >>> >>> Zakariae Jorti >> >> >> >> -- >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. >> -- Norbert Wiener >> >> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/> > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] [petsc-users] [EXTERNAL] Re: Question about periodic conditions
by Matthew Knepley 31 Mar '21

31 Mar '21

On Wed, Mar 31, 2021 at 3:21 AM Patrick Sanan <patrick.sanan(a)gmail.com> wrote: > (moving to petsc-dev) > > To follow up further on this, Matt is correct as to what's happening now, > but periodic coordinates aren't sufficiently supported yet in DMStag, so I > will add something. > > The way things are set up now has a conceptual elegance to it, in that to > define coordinates, you use another DM which has coordinate information on > it, instead of other field information. It's periodic iff the primary DM > is. So there is no point on the right boundary, at 2 * pi in the 1D version > of this example, because that point would be identical to the point at 0, > on the left boundary. > > The problem with the current implementation (for DMStag) is that the right > boundary of the domain [0, 2*pi) is never stored. There's no way to know > the width of the last cell on the right. You need that information for at > least two important reasons: > 1. to visualize the mesh, where even though the boundary point is the same > point on the torus, you are plotting it on the plane and want different > representations of the point on the left and right. > 2. to use PIC methods (DMSwarm), where we need a way to determine if a > particle is in the last cell. > > Matt, Mark, Dave, et. al, it'd be very helpful to know if the following > seems like a good/bad idea to you, since I assume you resolved this same > issue for DMPlex + DMSwarm: > > A tempting way to proceed here is to use the existing DMSetPeriodicity(), > which allows you to specify that missing piece of information and store it > in the DM. This could be called from the DMStagSetUniformCoordinatesXXX() > functions, so the user wouldn't have to worry about it in that case. That > also makes conceptual sense as that's the stage, after setup, in which you > specify the "embedding" part of the DM. A next step would be to make > DMLocalizeCoordinates() work for DMStag (and DMDA if possible, while I'm at > it). > That is what I added that stuff for. In Plex, in order to generalize to situations not in a box, we went to a formulation that uses a DG coordinate field instead. I think that is overkill here and would not give you any added functionality. Jed may argue that he wants you to retain the far point and use L2G to eliminate it, but that sounds like a lot more work. THanks, Matt > Am 25.03.2021 um 00:20 schrieb Matthew Knepley <knepley(a)gmail.com>: > > On Wed, Mar 24, 2021 at 7:17 PM Jorti, Zakariae via petsc-users < > petsc-users(a)mcs.anl.gov> wrote: > >> Hi Patrick, >> >> >> Thanks for your responses. >> >> As for the code, I was not granted permission to share it yet. So, I >> cannot send it to you for the moment. I apologize for that. >> >> >> I wanted to let you know that while I was testing my code, I discovered >> that when the periodic boundary conditions are activated, the coordinates >> accessed might be incorrect on one side of the boundary. >> >> Let me give you an example in cylindrical coordinates with a 3x3x3 >> DMStag mesh: >> >> >> >> >> PetscInt startr,startphi,startz,nr,nphi,nz,d; >> >> PetscInt er,ephi,ez,icErmphip[3]; >> >> DM dmCoorda, coordDA; >> >> Vec coordaLocal; >> >> PetscScalar ****arrCoord; >> >> PetscScalar surf; >> >> >> >> DMStagCreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,3,3,3,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,1,1,DMSTAG_STENCIL_BOX,1,NULL,NULL,NULL,&coordDA); >> >> >> DMSetFromOptions(coordDA); >> >> DMSetUp(coordDA); >> >> >> >> DMStagGetCorners(coordDA,&startr,&startphi,&startz,&nr,&nphi,&nz,NULL,NULL,NULL); >> >> >> DMGetCoordinateDM(coordDA,&dmCoorda); >> >> DMGetCoordinatesLocal(coordDA,&coordaLocal); >> >> DMStagVecGetArrayRead(dmCoorda,coordaLocal,&arrCoord); >> >> >> for (d=0; d< 3; ++d){ >> >> DMStagGetLocationSlot(dmCoorda,UP_LEFT,d,&icErmphip[d]); >> >> } >> >> >> er = 1; ez = 0; >> >> for (ephi=0; ephi< 3; ++ephi){ >> >> PetscPrintf(PETSC_COMM_WORLD,"Phi_p(%d,%d,%d) = %E\n",er,ephi,ez >> ,(double)arrCoord[ez][ephi][er][icErmphip[1]); >> >> } >> >> >> When I execute this example, I get this output: >> >> Phi_p(1,0,0) = 2.094395E+00 >> >> Phi_p(1,1,0) = 4.188790E+00 >> >> Phi_p(1,2,0) = 0.000000E+00 >> >> >> Note here that the first two lines correspond to 2π / 3 and 4π / 3 >> respectively. Thus, nothing is wrong here. >> >> But the last line should rather give 2π instead of 0. >> >> >> I understand that degrees of freedom should be the same on both sides of >> the boundary, but should the coordinates not be preserved? >> >> I don't think so. The circle has coordinates in [0, 2\pi), so the point > at 2\pi is identified with the point at 0 and you must choose > one, so we choose 0. > > Thanks, > > Matt > >> Thank you. >> >> Best regards, >> >> >> Zakariae Jorti >> ------------------------------ >> *From:* Patrick Sanan <patrick.sanan(a)gmail.com> >> *Sent:* Tuesday, March 23, 2021 11:37:04 AM >> *To:* Jorti, Zakariae >> *Cc:* petsc-users(a)mcs.anl.gov >> *Subject:* [EXTERNAL] Re: Question about periodic conditions >> >> Hi Zakariae - sorry about the delay - responses inline below. >> >> I'd be curious to see your code (which you can send directly to me if you >> don't want to post it publicly), so I can give you more comments, as DMStag >> is a new component. >> >> >> Am 23.03.2021 um 00:54 schrieb Jorti, Zakariae <zjorti(a)lanl.gov>: >> >> Hi, >> >> I implemented a PETSc code to solve Maxwell's equations for the magnetic >> and electric fields (B and E) in a cylinder: >> 0 < r_min <= r <= r_max; with r_max > r_min >> phi_min = 0 <= r <= phi_max = 2 π >> z_min <= z =< z_max; with z_max > z_min. >> >> I am using a PETSc staggered grid with the electric field E defined on >> edge centers and the magnetic field B defined on face centers. (dof0 = >> 0, dof1 = 1,dof2 = 1, dof3 = 0;). >> >> >> I have two versions of my code: >> 1 - A first version in which I set the boundary type to DM_BOUNDARY_NONE >> in the three directions r, phi and z >> 2- A second version in which I set the boundary type to DM_BOUNDARY_NONE >> in the r and z directions, and DM_BOUNDARY_PERIODIC in the phi direction. >> >> >> When I print the solution vector X, which contains both E and B >> components, I notice that the vector is shorter with the second version >> compared to the first one. >> Is it normal? >> >> Yes - with the periodic boundary conditions, there will be fewer points >> since there won't be the "extra" layer of faces and edges at phi = 2 * pi . >> >> If you consider a 1-d example with 1 dof on vertices and cells, with >> three elements, the periodic case looks like this, globally, >> >> x ---- x ---- x ---- >> >> as opposed to the non-periodic case, >> >> x ---- x ---- x ---- x >> >> >> >> Besides, I was wondering if I have to change the way I define the value >> of the solution on the boundary. What I am doing so far in both versions is >> something like: >> B_phi [phi = 0] = 1.0; >> B_phi [phi = 2π] = 1.0; >> E_z [r, phi = 0] = 1/r; >> E_z [r, phi = 2π] = 1/r; >> >> Assuming that values at phi = 0 should be the same as at phi=2π with >> the periodic boundary conditions, is it sufficient for example to have only >> the following boundary conditions: >> B_phi [phi = 0] = 1.0; >> >> E_z [r, phi = 0] = 1/r ? >> >> >> Yes - this is the intention, since the boundary at phi = 2 * pi is >> represented by the same entries in the global vector. >> >> Of course, you need to make sure that your continuous problem is >> well-posed, which in general could change when using different boundary >> conditions. >> >> Thank you. >> Best regards, >> >> Zakariae Jorti >> >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Re: [petsc-dev] [petsc-users] [EXTERNAL] Re: Question about periodic conditions
by Patrick Sanan 31 Mar '21

31 Mar '21

(moving to petsc-dev) To follow up further on this, Matt is correct as to what's happening now, but periodic coordinates aren't sufficiently supported yet in DMStag, so I will add something. The way things are set up now has a conceptual elegance to it, in that to define coordinates, you use another DM which has coordinate information on it, instead of other field information. It's periodic iff the primary DM is. So there is no point on the right boundary, at 2 * pi in the 1D version of this example, because that point would be identical to the point at 0, on the left boundary. The problem with the current implementation (for DMStag) is that the right boundary of the domain [0, 2*pi) is never stored. There's no way to know the width of the last cell on the right. You need that information for at least two important reasons: 1. to visualize the mesh, where even though the boundary point is the same point on the torus, you are plotting it on the plane and want different representations of the point on the left and right. 2. to use PIC methods (DMSwarm), where we need a way to determine if a particle is in the last cell. Matt, Mark, Dave, et. al, it'd be very helpful to know if the following seems like a good/bad idea to you, since I assume you resolved this same issue for DMPlex + DMSwarm: A tempting way to proceed here is to use the existing DMSetPeriodicity(), which allows you to specify that missing piece of information and store it in the DM. This could be called from the DMStagSetUniformCoordinatesXXX() functions, so the user wouldn't have to worry about it in that case. That also makes conceptual sense as that's the stage, after setup, in which you specify the "embedding" part of the DM. A next step would be to make DMLocalizeCoordinates() work for DMStag (and DMDA if possible, while I'm at it). > Am 25.03.2021 um 00:20 schrieb Matthew Knepley <knepley(a)gmail.com>: > > On Wed, Mar 24, 2021 at 7:17 PM Jorti, Zakariae via petsc-users <petsc-users(a)mcs.anl.gov <mailto:[email protected]>> wrote: > Hi Patrick, > > > > Thanks for your responses. > > As for the code, I was not granted permission to share it yet. So, I cannot send it to you for the moment. I apologize for that. > > > > I wanted to let you know that while I was testing my code, I discovered that when the periodic boundary conditions are activated, the coordinates accessed might be incorrect on one side of the boundary. > > Let me give you an example in cylindrical coordinates with a 3x3x3 DMStag mesh: > > > > > > > PetscInt startr,startphi,startz,nr,nphi,nz,d; > > > PetscInt er,ephi,ez,icErmphip[3]; > > > DM dmCoorda, coordDA; > > Vec coordaLocal; > > PetscScalar ****arrCoord; > > PetscScalar surf; > > > > DMStagCreate3d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,3,3,3,PETSC_DECIDE,PETSC_DECIDE,PETSC_DECIDE,1,1,1,1,DMSTAG_STENCIL_BOX,1,NULL,NULL,NULL,&coordDA); > > > > DMSetFromOptions(coordDA); > > DMSetUp(coordDA); > > > > DMStagGetCorners(coordDA,&startr,&startphi,&startz,&nr,&nphi,&nz,NULL,NULL,NULL); > > > > DMGetCoordinateDM(coordDA,&dmCoorda); > > DMGetCoordinatesLocal(coordDA,&coordaLocal); > > DMStagVecGetArrayRead(dmCoorda,coordaLocal,&arrCoord); > > > > for (d=0; d< 3; ++d){ > > DMStagGetLocationSlot(dmCoorda,UP_LEFT,d,&icErmphip[d]); > > } > > > > er = 1; ez = 0; > > for (ephi=0; ephi< 3; ++ephi){ > > PetscPrintf(PETSC_COMM_WORLD,"Phi_p(%d,%d,%d) = %E\n",er,ephi,ez,(double)arrCoord[ez][ephi][er][icErmphip[1]); > > } > > > > When I execute this example, I get this output: > > Phi_p(1,0,0) = 2.094395E+00 > > Phi_p(1,1,0) = 4.188790E+00 > > Phi_p(1,2,0) = 0.000000E+00 > > > > Note here that the first two lines correspond to 2π / 3 and 4π / 3 respectively. Thus, nothing is wrong here. > > But the last line should rather give 2π instead of 0. > > > > I understand that degrees of freedom should be the same on both sides of the boundary, but should the coordinates not be preserved? > > > I don't think so. The circle has coordinates in [0, 2\pi), so the point at 2\pi is identified with the point at 0 and you must choose > one, so we choose 0. > > Thanks, > > Matt > Thank you. > > Best regards, > > > > Zakariae Jorti > > From: Patrick Sanan <patrick.sanan(a)gmail.com <mailto:[email protected]>> > Sent: Tuesday, March 23, 2021 11:37:04 AM > To: Jorti, Zakariae > Cc: petsc-users(a)mcs.anl.gov <mailto:[email protected]> > Subject: [EXTERNAL] Re: Question about periodic conditions > > Hi Zakariae - sorry about the delay - responses inline below. > > I'd be curious to see your code (which you can send directly to me if you don't want to post it publicly), so I can give you more comments, as DMStag is a new component. > > >> Am 23.03.2021 um 00:54 schrieb Jorti, Zakariae <zjorti(a)lanl.gov <mailto:[email protected]>>: >> >> Hi, >> >> I implemented a PETSc code to solve Maxwell's equations for the magnetic and electric fields (B and E) in a cylinder: >> 0 < r_min <= r <= r_max; with r_max > r_min >> phi_min = 0 <= r <= phi_max = 2 π >> z_min <= z =< z_max; with z_max > z_min. >> >> I am using a PETSc staggered grid with the electric field E defined on edge centers and the magnetic field B defined on face centers. (dof0 = 0, dof1 = 1,dof2 = 1, dof3 = 0;). >> >> I have two versions of my code: >> 1 - A first version in which I set the boundary type to DM_BOUNDARY_NONE in the three directions r, phi and z >> 2- A second version in which I set the boundary type to DM_BOUNDARY_NONE in the r and z directions, and DM_BOUNDARY_PERIODIC in the phi direction. >> >> When I print the solution vector X, which contains both E and B components, I notice that the vector is shorter with the second version compared to the first one. >> Is it normal? > Yes - with the periodic boundary conditions, there will be fewer points since there won't be the "extra" layer of faces and edges at phi = 2 * pi . > > If you consider a 1-d example with 1 dof on vertices and cells, with three elements, the periodic case looks like this, globally, > > x ---- x ---- x ---- > > as opposed to the non-periodic case, > > x ---- x ---- x ---- x > > >> >> Besides, I was wondering if I have to change the way I define the value of the solution on the boundary. What I am doing so far in both versions is something like: >> B_phi [phi = 0] = 1.0; >> B_phi [phi = 2π] = 1.0; >> E_z [r, phi = 0] = 1/r; >> E_z [r, phi = 2π] = 1/r; >> >> Assuming that values at phi = 0 should be the same as at phi=2π with the periodic boundary conditions, is it sufficient for example to have only the following boundary conditions: >> B_phi [phi = 0] = 1.0; >> E_z [r, phi = 0] = 1/r ? > > Yes - this is the intention, since the boundary at phi = 2 * pi is represented by the same entries in the global vector. > > Of course, you need to make sure that your continuous problem is well-posed, which in general could change when using different boundary conditions. > >> Thank you. >> Best regards, >> >> Zakariae Jorti > > > > -- > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0

Tag name v3.15
by Lisandro Dalcin 31 Mar '21

31 Mar '21

Satish, if we use semantic versioning for the PETSc version itself and the tarball, why don't we use the same format for tags? In short, why aren't we using v3.15.0 instead of v3.15 ? -- Lisandro Dalcin ============ Senior Research Scientist Extreme Computing Research Center (ECRC) King Abdullah University of Science and Technology (KAUST) http://ecrc.kaust.edu.sa/

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Eric Chamberland 30 Mar '21

30 Mar '21

Hi Barry, Here is what I have: 1. The hpddm issues have been all solved (you can't see no more hpddm failures here: https://giref.ulaval.ca/~cmpgiref/petsc-main-debug/2021.03.29.02h00m02s_mak…) 2. For Hypre, I think it is indeed not a bug but a feature, as far as I can see what has been told on the hypre discussion list it is said "It still depends on the number of threads, that can’t be avoided" ( https://github.com/hypre-space/hypre/issues/303#issuecomment-800442755 ) and here https://www.researchgate.net/publication/220411740_Multigrid_Smoothers_for_…, into section 7.3, we have some interesting informations, as: Figure 7.6 clearly illustrates that convergence degrades with the addition of threads for hybrid SGS; .... The 3D sphere problem is the most extreme example because AMG-CG with hybrid SGS no longer converges with the addition of threading. but I might have misunderstood since I am not an expert for that... 3. For SuperLU_Dist, I have tried to build SuperLU_dist out of PETSc to run the tests from superlu itself: sadly the bug is not showing up (see https://github.com/xiaoyeli/superlu_dist/issues/69) I would like to build a reproducer superlu_dist example from what is done in the faulty test: ksp_ksp_tutorials-ex5 that is buggy when called from PETSc: what bugs me, is that many other PETSc tests are running fine with superlu_dist: maybe something is uniquely done in ksp_ksp_tutorials-ex5 ? So I think it worth digging into #3: the simple thing I have not yet done is retreiving the stack when it fails (timeout). And a question: when you state that you upgraded to OpenMPI 4.1 you mean for one of your automated (docker?) compilation into the gitlab pipelines? Thanks for taking news! :) Eric On 2021-03-30 1:47 p.m., Barry Smith wrote: > > Eric, > > How are things going on this OpenMP front? Any bug fixes from > hypre or SuperLU_DIST? > > BTW: we have upgraded to OpenMPI 4.1 perhaps this resolves some > issues? > > Barry > > >> On Mar 22, 2021, at 2:07 PM, Eric Chamberland >> <Eric.Chamberland(a)giref.ulaval.ca >> <mailto:[email protected]>> wrote: >> >> I added some information here: >> >> https://github.com/xiaoyeli/superlu_dist/issues/69#issuecomment-804318719 >> >> Maybe someone can say more than I on what PETSc tries to do with the >> 2 mentioned tutorials that are timing out... >> >> Thanks, >> >> Eric >> >> >> On 2021-03-15 11:31 a.m., Eric Chamberland wrote: >>> >>> Reported timeout bugs to SuperLU_dist too: >>> >>> https://github.com/xiaoyeli/superlu_dist/issues/69 >>> >>> Eric >>> >>> >>> On 2021-03-14 2:18 p.m., Eric Chamberland wrote: >>>> >>>> Done: >>>> >>>> https://github.com/hypre-space/hypre/issues/303 >>>> >>>> Maybe I will need some help about PETSc to answer their questions... >>>> >>>> Eric >>>> >>>> On 2021-03-14 3:44 a.m., Stefano Zampini wrote: >>>>> Eric >>>>> >>>>> You should report these HYPRE issues upstream >>>>> https://github.com/hypre-space/hypre/issues >>>>> <https://github.com/hypre-space/hypre/issues> >>>>> >>>>> >>>>>> On Mar 14, 2021, at 3:44 AM, Eric Chamberland >>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>> <mailto:[email protected]>> wrote: >>>>>> >>>>>> For us it clearly creates problems in real computations... >>>>>> >>>>>> I understand the need to have clean test for PETSc, but for me, >>>>>> it reveals that hypre isn't usable with more than one thread for >>>>>> now... >>>>>> >>>>>> Another solution: force single-threaded configuration for hypre >>>>>> until this is fixed? >>>>>> >>>>>> Eric >>>>>> >>>>>> On 2021-03-13 8:50 a.m., Pierre Jolivet wrote: >>>>>>> -pc_hypre_boomeramg_relax_type_all Jacobi => >>>>>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC >>>>>>> iterations 3 >>>>>>> -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => >>>>>>> OK, independently of the architecture it seems (Eric Docker >>>>>>> image with 1 or 2 threads or my macOS), but contraction factor >>>>>>> is higher >>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 8 >>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 24 >>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 26 >>>>>>> v. currently >>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 9 >>>>>>> Linear solve converged due to CONVERGED_RTOL iterations 10 >>>>>>> >>>>>>> Do we change this? Or should we force OMP_NUM_THREADS=1 for make >>>>>>> test? >>>>>>> >>>>>>> Thanks, >>>>>>> Pierre >>>>>>> >>>>>>>> On 13 Mar 2021, at 2:26 PM, Mark Adams <mfadams(a)lbl.gov >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hypre uses a multiplicative smoother by default. It has a >>>>>>>> chebyshev smoother. That with a Jacobi PC should be thread >>>>>>>> invariant. >>>>>>>> Mark >>>>>>>> >>>>>>>> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> >>>>>>>>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet >>>>>>>>> <pierre(a)joliv.et <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> Hello Eric, >>>>>>>>> I’ve made an “interesting” discovery, so I’ll put back the >>>>>>>>> list in c/c. >>>>>>>>> It appears the following snippet of code which uses >>>>>>>>> Allreduce() + lambda function + MPI_IN_PLACE is: >>>>>>>>> - Valgrind-clean with MPICH; >>>>>>>>> - Valgrind-clean with OpenMPI 4.0.5; >>>>>>>>> - not Valgrind-clean with OpenMPI 4.1.0. >>>>>>>>> I’m not sure who is to blame here, I’ll need to look at >>>>>>>>> the MPI specification for what is required by the >>>>>>>>> implementors and users in that case. >>>>>>>>> >>>>>>>>> In the meantime, I’ll do the following: >>>>>>>>> - update config/BuildSystem/config/packages/OpenMPI.py to >>>>>>>>> use OpenMPI 4.1.0, see if any other error appears; >>>>>>>>> - provide a hotfix to bypass the segfaults; >>>>>>>> >>>>>>>> I can confirm that splitting the single Allreduce with my >>>>>>>> own MPI_Op into two Allreduce with MAX and BAND fixes the >>>>>>>> segfaults with OpenMPI (*). >>>>>>>> >>>>>>>>> - look at the hypre issue and whether they should be >>>>>>>>> deferred to the hypre team. >>>>>>>> >>>>>>>> I don’t know if there is something wrong in hypre threading >>>>>>>> or if it’s just a side effect of threading, but it seems >>>>>>>> that the number of threads has a drastic effect on the >>>>>>>> quality of the PC. >>>>>>>> By default, it looks that there are two threads per process >>>>>>>> with your Docker image. >>>>>>>> If I force OMP_NUM_THREADS=1, then I get the same >>>>>>>> convergence as in the output file. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Pierre >>>>>>>> >>>>>>>> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 >>>>>>>> <https://gitlab.com/petsc/petsc/-/merge_requests/3712> >>>>>>>> >>>>>>>>> Thank you for the Docker files, they were really useful. >>>>>>>>> If you want to avoid oversubscription failures, you can >>>>>>>>> edit the file >>>>>>>>> /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append >>>>>>>>> the line: >>>>>>>>> localhost slots=12 >>>>>>>>> If you want to increase the timeout limit of PETSc test >>>>>>>>> suite for each test, you can add the extra flag in your >>>>>>>>> command line TIMEOUT=180 (default is 60, units are seconds). >>>>>>>>> >>>>>>>>> Thanks, I’ll ping you on GitLab when I’ve got something >>>>>>>>> ready for you to try, >>>>>>>>> Pierre >>>>>>>>> >>>>>>>>> <ompi.cxx> >>>>>>>>> >>>>>>>>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland >>>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>> >>>>>>>>>> Hi Pierre, >>>>>>>>>> >>>>>>>>>> I now have a docker container reproducing the problems here. >>>>>>>>>> >>>>>>>>>> Actually, if I look at >>>>>>>>>> snes_tutorials-ex12_quad_singular_hpddm it fails like this: >>>>>>>>>> >>>>>>>>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error >>>>>>>>>> code: 59 >>>>>>>>>> # Initial guess >>>>>>>>>> # L_2 Error: 0.00803099 >>>>>>>>>> # Initial Residual >>>>>>>>>> # L_2 Residual: 1.09057 >>>>>>>>>> # Au - b = Au + F(0) >>>>>>>>>> # Linear L_2 Residual: 1.09057 >>>>>>>>>> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 >>>>>>>>>> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 >>>>>>>>>> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 >>>>>>>>>> # [3]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> # [3]PETSC ERROR: Caught signal number 11 SEGV: >>>>>>>>>> Segmentation Violation, probably memory access out of range >>>>>>>>>> # [3]PETSC ERROR: Try option -start_in_debugger or >>>>>>>>>> -on_error_attach_debugger >>>>>>>>>> # [3]PETSC ERROR: or see >>>>>>>>>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind >>>>>>>>>> <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >>>>>>>>>> # [3]PETSC ERROR: or try http://valgrind.org >>>>>>>>>> <http://valgrind.org/> on GNU/linux and Apple Mac OS X to >>>>>>>>>> find memory corruption errors >>>>>>>>>> # [3]PETSC ERROR: likely location of problem given in >>>>>>>>>> stack below >>>>>>>>>> # [3]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>> ------------------------------------ >>>>>>>>>> # [3]PETSC ERROR: Note: The EXACT line numbers in the >>>>>>>>>> stack are not available, >>>>>>>>>> # [3]PETSC ERROR: INSTEAD the line number of the start of >>>>>>>>>> the function >>>>>>>>>> # [3]PETSC ERROR: is given. >>>>>>>>>> # [3]PETSC ERROR: [3] buildTwo line 987 >>>>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>>>> # [3]PETSC ERROR: [3] next line 1130 >>>>>>>>>> /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>>>> # [3]PETSC ERROR: --------------------- Error Message >>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>> # [3]PETSC ERROR: Signal received >>>>>>>>>> # [3]PETSC ERROR: [0]PETSC ERROR: >>>>>>>>>> ------------------------------------------------------------------------ >>>>>>>>>> >>>>>>>>>> also ex12_quad_hpddm_reuse_baij fails with a lot more >>>>>>>>>> "Read -1, expected ..." which I don't know where they >>>>>>>>>> come from...? >>>>>>>>>> >>>>>>>>>> Hypre (like in diff-snes_tutorials-ex56_hypre) is also >>>>>>>>>> having DIVERGED_INDEFINITE_PC failures... >>>>>>>>>> >>>>>>>>>> Please see the 3 attached docker files: >>>>>>>>>> >>>>>>>>>> 1) fedora_mkl_and_devtools : the DockerFile which install >>>>>>>>>> fedore 33 with gnu compilers and MKL and everything to >>>>>>>>>> develop. >>>>>>>>>> >>>>>>>>>> 2) openmpi: the DockerFile to bluid OpenMPI >>>>>>>>>> >>>>>>>>>> 3) petsc: The las DockerFile that build/install and test >>>>>>>>>> PETSc >>>>>>>>>> >>>>>>>>>> I build the 3 like this: >>>>>>>>>> >>>>>>>>>> docker build -t fedora_mkl_and_devtools -f >>>>>>>>>> fedora_mkl_and_devtools . >>>>>>>>>> >>>>>>>>>> docker build -t openmpi -f openmpi . >>>>>>>>>> >>>>>>>>>> docker build -t petsc -f petsc . >>>>>>>>>> >>>>>>>>>> Disclaimer: I am not a docker expert, so I may do things >>>>>>>>>> that are not docker-stat-of-the-art but I am opened to >>>>>>>>>> suggestions... ;) >>>>>>>>>> >>>>>>>>>> I have just ran it on my portable (long) which have not >>>>>>>>>> enough cores, so many more tests failed (should force >>>>>>>>>> --oversubscribe but don't know how to). I will relaunch >>>>>>>>>> on my workstation in a few minutes. >>>>>>>>>> >>>>>>>>>> I will now test your branch! (sorry for the delay). >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >>>>>>>>>>> >>>>>>>>>>> Hi Pierre, >>>>>>>>>>> >>>>>>>>>>> ok, that's interesting! >>>>>>>>>>> >>>>>>>>>>> I will try to build a docker image until tomorrow and >>>>>>>>>>> give you the exact recipe to reproduce the bugs. >>>>>>>>>>> >>>>>>>>>>> Eric >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith >>>>>>>>>>>>> <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Eric, >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry about not being more immediate. We still have >>>>>>>>>>>>> this in our active email so you don't need to submit >>>>>>>>>>>>> individual issues. We'll try to get to them as soon as >>>>>>>>>>>>> we can. >>>>>>>>>>>> >>>>>>>>>>>> Indeed, I’m still trying to figure this out. >>>>>>>>>>>> I realized that some of my configure flags were >>>>>>>>>>>> different than yours, e.g., no --with-memalign. >>>>>>>>>>>> I’ve also added SuperLU_DIST to my installation. >>>>>>>>>>>> Still, I can’t reproduce any issue. >>>>>>>>>>>> I will continue looking into this, it appears I’m >>>>>>>>>>>> seeing some valgrind errors, but I don’t know if this >>>>>>>>>>>> is some side effect of OpenMPI not being valgrind-clean >>>>>>>>>>>> (last time I checked, there was no error with MPICH). >>>>>>>>>>>> >>>>>>>>>>>> Thank you for your patience, >>>>>>>>>>>> Pierre >>>>>>>>>>>> >>>>>>>>>>>> /usr/bin/gmake -f gmakefile test test-fail=1 >>>>>>>>>>>> Using MAKEFLAGS: test-fail=1 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>>>> ok >>>>>>>>>>>> diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>>>>>>>>>>> ok ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>>>>>>>>>>> ok snes_tutorials-ex56_hypre >>>>>>>>>>>> ok diff-snes_tutorials-ex56_hypre >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>>>>>>>>>>> ok ksp_ksp_tutorials-ex56_2 >>>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex56_2 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>>>> ok >>>>>>>>>>>> diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>>>>>>>>>>> #srun: error: Unable to create step for job 1426755: >>>>>>>>>>>> More processors requested than permitted >>>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command >>>>>>>>>>>> failed so no diff >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran >>>>>>>>>>>> required for this test >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>>>>>>>>>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>>>>>>>>>>> ok snes_tutorials-ex19_tut_3 >>>>>>>>>>>> ok diff-snes_tutorials-ex19_tut_3 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP >>>>>>>>>>>> Fortran required for this test >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist >>>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>>>>>>>>>>> ok >>>>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>>>> ok >>>>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>>>>>>>>>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>>>>>>>>>>> #srun: error: Unable to create step for job 1426755: >>>>>>>>>>>> More processors requested than permitted >>>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command >>>>>>>>>>>> failed so no diff >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>>>>>>>>>>> ok >>>>>>>>>>>> snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>>>> ok >>>>>>>>>>>> diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>>>>>>>>>>> ok ksp_ksp_tutorials-ex64_1 >>>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex64_1 >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>>>>>>>>>>> #srun: error: Unable to create step for job 1426755: >>>>>>>>>>>> More processors requested than permitted >>>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command >>>>>>>>>>>> failed so no diff >>>>>>>>>>>> TEST >>>>>>>>>>>> arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP >>>>>>>>>>>> Fortran required for this test >>>>>>>>>>>> >>>>>>>>>>>>> Barry >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland >>>>>>>>>>>>>> <Eric.Chamberland(a)giref.ulaval.ca >>>>>>>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Barry, >>>>>>>>>>>>>> >>>>>>>>>>>>>> to get a some follow up on --with-openmp=1 failures, >>>>>>>>>>>>>> shall I open gitlab issues for: >>>>>>>>>>>>>> >>>>>>>>>>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>>>>>>>>>>> >>>>>>>>>>>>>> b) all superlu_dist failures giving different results >>>>>>>>>>>>>> with initia and "Exceeded timeout limit of 60 s" >>>>>>>>>>>>>> >>>>>>>>>>>>>> c) hpddm failures "free(): invalid next size (fast)" >>>>>>>>>>>>>> and "Segmentation Violation" >>>>>>>>>>>>>> >>>>>>>>>>>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>>>>>>>>>>> >>>>>>>>>>>>>> I don't see how I could do all these debugging by >>>>>>>>>>>>>> myself... >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Eric >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>>>> Professionnel de recherche >>>>>>>>>>> GIREF/Université Laval >>>>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>>> -- >>>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>>> Professionnel de recherche >>>>>>>>>> GIREF/Université Laval >>>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>>> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> -- >>>>>> Eric Chamberland, ing., M. Ing >>>>>> Professionnel de recherche >>>>>> GIREF/Université Laval >>>>>> (418) 656-2131 poste 41 22 42 >>>>> >>>> -- >>>> Eric Chamberland, ing., M. Ing >>>> Professionnel de recherche >>>> GIREF/Université Laval >>>> (418) 656-2131 poste 41 22 42 >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Université Laval >>> (418) 656-2131 poste 41 22 42 >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 > -- Eric Chamberland, ing., M. Ing Professionnel de recherche GIREF/Université Laval (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] Petsc "make test" have more failures for --with-openmp=1
by Barry Smith 30 Mar '21

30 Mar '21

Eric, How are things going on this OpenMP front? Any bug fixes from hypre or SuperLU_DIST? BTW: we have upgraded to OpenMPI 4.1 perhaps this resolves some issues? Barry > On Mar 22, 2021, at 2:07 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca> wrote: > > I added some information here: > > https://github.com/xiaoyeli/superlu_dist/issues/69#issuecomment-804318719 <https://github.com/xiaoyeli/superlu_dist/issues/69#issuecomment-804318719> > Maybe someone can say more than I on what PETSc tries to do with the 2 mentioned tutorials that are timing out... > > Thanks, > > Eric > > > > On 2021-03-15 11:31 a.m., Eric Chamberland wrote: >> Reported timeout bugs to SuperLU_dist too: >> >> https://github.com/xiaoyeli/superlu_dist/issues/69 <https://github.com/xiaoyeli/superlu_dist/issues/69> >> Eric >> >> >> >> On 2021-03-14 2:18 p.m., Eric Chamberland wrote: >>> Done: >>> >>> https://github.com/hypre-space/hypre/issues/303 <https://github.com/hypre-space/hypre/issues/303> >>> Maybe I will need some help about PETSc to answer their questions... >>> >>> Eric >>> >>> On 2021-03-14 3:44 a.m., Stefano Zampini wrote: >>>> Eric >>>> >>>> You should report these HYPRE issues upstream https://github.com/hypre-space/hypre/issues <https://github.com/hypre-space/hypre/issues> >>>> >>>> >>>>> On Mar 14, 2021, at 3:44 AM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>>>> >>>>> For us it clearly creates problems in real computations... >>>>> >>>>> I understand the need to have clean test for PETSc, but for me, it reveals that hypre isn't usable with more than one thread for now... >>>>> >>>>> Another solution: force single-threaded configuration for hypre until this is fixed? >>>>> >>>>> Eric >>>>> >>>>> On 2021-03-13 8:50 a.m., Pierre Jolivet wrote: >>>>>> -pc_hypre_boomeramg_relax_type_all Jacobi => >>>>>> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 3 >>>>>> -pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi => >>>>>> OK, independently of the architecture it seems (Eric Docker image with 1 or 2 threads or my macOS), but contraction factor is higher >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 8 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 24 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 26 >>>>>> v. currently >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 7 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 9 >>>>>> Linear solve converged due to CONVERGED_RTOL iterations 10 >>>>>> >>>>>> Do we change this? Or should we force OMP_NUM_THREADS=1 for make test? >>>>>> >>>>>> Thanks, >>>>>> Pierre >>>>>> >>>>>>> On 13 Mar 2021, at 2:26 PM, Mark Adams <mfadams(a)lbl.gov <mailto:[email protected]>> wrote: >>>>>>> >>>>>>> Hypre uses a multiplicative smoother by default. It has a chebyshev smoother. That with a Jacobi PC should be thread invariant. >>>>>>> Mark >>>>>>> >>>>>>> On Sat, Mar 13, 2021 at 8:18 AM Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: >>>>>>> >>>>>>>> On 13 Mar 2021, at 9:17 AM, Pierre Jolivet <pierre(a)joliv.et <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>> Hello Eric, >>>>>>>> I’ve made an “interesting” discovery, so I’ll put back the list in c/c. >>>>>>>> It appears the following snippet of code which uses Allreduce() + lambda function + MPI_IN_PLACE is: >>>>>>>> - Valgrind-clean with MPICH; >>>>>>>> - Valgrind-clean with OpenMPI 4.0.5; >>>>>>>> - not Valgrind-clean with OpenMPI 4.1.0. >>>>>>>> I’m not sure who is to blame here, I’ll need to look at the MPI specification for what is required by the implementors and users in that case. >>>>>>>> >>>>>>>> In the meantime, I’ll do the following: >>>>>>>> - update config/BuildSystem/config/packages/OpenMPI.py to use OpenMPI 4.1.0, see if any other error appears; >>>>>>>> - provide a hotfix to bypass the segfaults; >>>>>>> >>>>>>> I can confirm that splitting the single Allreduce with my own MPI_Op into two Allreduce with MAX and BAND fixes the segfaults with OpenMPI (*). >>>>>>> >>>>>>>> - look at the hypre issue and whether they should be deferred to the hypre team. >>>>>>> >>>>>>> I don’t know if there is something wrong in hypre threading or if it’s just a side effect of threading, but it seems that the number of threads has a drastic effect on the quality of the PC. >>>>>>> By default, it looks that there are two threads per process with your Docker image. >>>>>>> If I force OMP_NUM_THREADS=1, then I get the same convergence as in the output file. >>>>>>> >>>>>>> Thanks, >>>>>>> Pierre >>>>>>> >>>>>>> (*) https://gitlab.com/petsc/petsc/-/merge_requests/3712 <https://gitlab.com/petsc/petsc/-/merge_requests/3712> >>>>>>>> Thank you for the Docker files, they were really useful. >>>>>>>> If you want to avoid oversubscription failures, you can edit the file /opt/openmpi-4.1.0/etc/openmpi-default-hostfile and append the line: >>>>>>>> localhost slots=12 >>>>>>>> If you want to increase the timeout limit of PETSc test suite for each test, you can add the extra flag in your command line TIMEOUT=180 (default is 60, units are seconds). >>>>>>>> >>>>>>>> Thanks, I’ll ping you on GitLab when I’ve got something ready for you to try, >>>>>>>> Pierre >>>>>>>> >>>>>>>> <ompi.cxx> >>>>>>>> >>>>>>>>> On 12 Mar 2021, at 8:54 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>>>>>>>> >>>>>>>>> Hi Pierre, >>>>>>>>> >>>>>>>>> I now have a docker container reproducing the problems here. >>>>>>>>> >>>>>>>>> Actually, if I look at snes_tutorials-ex12_quad_singular_hpddm it fails like this: >>>>>>>>> >>>>>>>>> not ok snes_tutorials-ex12_quad_singular_hpddm # Error code: 59 >>>>>>>>> # Initial guess >>>>>>>>> # L_2 Error: 0.00803099 >>>>>>>>> # Initial Residual >>>>>>>>> # L_2 Residual: 1.09057 >>>>>>>>> # Au - b = Au + F(0) >>>>>>>>> # Linear L_2 Residual: 1.09057 >>>>>>>>> # [d470c54ce086:14127] Read -1, expected 4096, errno = 1 >>>>>>>>> # [d470c54ce086:14128] Read -1, expected 4096, errno = 1 >>>>>>>>> # [d470c54ce086:14129] Read -1, expected 4096, errno = 1 >>>>>>>>> # [3]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>>> # [3]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range >>>>>>>>> # [3]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger >>>>>>>>> # [3]PETSC ERROR: or see https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind <https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind> >>>>>>>>> # [3]PETSC ERROR: or try http://valgrind.org <http://valgrind.org/> on GNU/linux and Apple Mac OS X to find memory corruption errors >>>>>>>>> # [3]PETSC ERROR: likely location of problem given in stack below >>>>>>>>> # [3]PETSC ERROR: --------------------- Stack Frames ------------------------------------ >>>>>>>>> # [3]PETSC ERROR: Note: The EXACT line numbers in the stack are not available, >>>>>>>>> # [3]PETSC ERROR: INSTEAD the line number of the start of the function >>>>>>>>> # [3]PETSC ERROR: is given. >>>>>>>>> # [3]PETSC ERROR: [3] buildTwo line 987 /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>>> # [3]PETSC ERROR: [3] next line 1130 /opt/petsc-main/include/HPDDM_schwarz.hpp >>>>>>>>> # [3]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- >>>>>>>>> # [3]PETSC ERROR: Signal received >>>>>>>>> # [3]PETSC ERROR: [0]PETSC ERROR: ------------------------------------------------------------------------ >>>>>>>>> >>>>>>>>> also ex12_quad_hpddm_reuse_baij fails with a lot more "Read -1, expected ..." which I don't know where they come from...? >>>>>>>>> >>>>>>>>> Hypre (like in diff-snes_tutorials-ex56_hypre) is also having DIVERGED_INDEFINITE_PC failures... >>>>>>>>> >>>>>>>>> Please see the 3 attached docker files: >>>>>>>>> >>>>>>>>> 1) fedora_mkl_and_devtools : the DockerFile which install fedore 33 with gnu compilers and MKL and everything to develop. >>>>>>>>> >>>>>>>>> 2) openmpi: the DockerFile to bluid OpenMPI >>>>>>>>> >>>>>>>>> 3) petsc: The las DockerFile that build/install and test PETSc >>>>>>>>> >>>>>>>>> I build the 3 like this: >>>>>>>>> >>>>>>>>> docker build -t fedora_mkl_and_devtools -f fedora_mkl_and_devtools . >>>>>>>>> >>>>>>>>> docker build -t openmpi -f openmpi . >>>>>>>>> >>>>>>>>> docker build -t petsc -f petsc . >>>>>>>>> >>>>>>>>> Disclaimer: I am not a docker expert, so I may do things that are not docker-stat-of-the-art but I am opened to suggestions... ;) >>>>>>>>> >>>>>>>>> I have just ran it on my portable (long) which have not enough cores, so many more tests failed (should force --oversubscribe but don't know how to). I will relaunch on my workstation in a few minutes. >>>>>>>>> >>>>>>>>> I will now test your branch! (sorry for the delay). >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Eric >>>>>>>>> >>>>>>>>> On 2021-03-11 9:03 a.m., Eric Chamberland wrote: >>>>>>>>>> Hi Pierre, >>>>>>>>>> >>>>>>>>>> ok, that's interesting! >>>>>>>>>> >>>>>>>>>> I will try to build a docker image until tomorrow and give you the exact recipe to reproduce the bugs. >>>>>>>>>> >>>>>>>>>> Eric >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2021-03-11 2:46 a.m., Pierre Jolivet wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On 11 Mar 2021, at 6:16 AM, Barry Smith <bsmith(a)petsc.dev <mailto:[email protected]>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Eric, >>>>>>>>>>>> >>>>>>>>>>>> Sorry about not being more immediate. We still have this in our active email so you don't need to submit individual issues. We'll try to get to them as soon as we can. >>>>>>>>>>> >>>>>>>>>>> Indeed, I’m still trying to figure this out. >>>>>>>>>>> I realized that some of my configure flags were different than yours, e.g., no --with-memalign. >>>>>>>>>>> I’ve also added SuperLU_DIST to my installation. >>>>>>>>>>> Still, I can’t reproduce any issue. >>>>>>>>>>> I will continue looking into this, it appears I’m seeing some valgrind errors, but I don’t know if this is some side effect of OpenMPI not being valgrind-clean (last time I checked, there was no error with MPICH). >>>>>>>>>>> >>>>>>>>>>> Thank you for your patience, >>>>>>>>>>> Pierre >>>>>>>>>>> >>>>>>>>>>> /usr/bin/gmake -f gmakefile test test-fail=1 >>>>>>>>>>> Using MAKEFLAGS: test-fail=1 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_baij.counts >>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_baij >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist_2.counts >>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist_2 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex49_superlu_dist.counts >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-0_conv-1 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-1herm-1_conv-1 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-0_conv-1 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-0 >>>>>>>>>>> ok ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>>> ok diff-ksp_ksp_tests-ex49_superlu_dist+nsize-4herm-1_conv-1 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex50_tut_2.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex50_tut_2 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tests-ex33_superlu_dist.counts >>>>>>>>>>> ok ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>>> ok diff-ksp_ksp_tests-ex33_superlu_dist >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_hypre.counts >>>>>>>>>>> ok snes_tutorials-ex56_hypre >>>>>>>>>>> ok diff-snes_tutorials-ex56_hypre >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex56_2.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex56_2 >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex56_2 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_elas.counts >>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_elas >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij.counts >>>>>>>>>>> ok snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>>> ok diff-snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_3.counts >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_3 # Error code: 1 >>>>>>>>>>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_3 # SKIP Command failed so no diff >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist # SKIP Fortran required for this test >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex12_tri_parmetis_hpddm_baij.counts >>>>>>>>>>> ok snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>>> ok diff-snes_tutorials-ex12_tri_parmetis_hpddm_baij >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_tut_3.counts >>>>>>>>>>> ok snes_tutorials-ex19_tut_3 >>>>>>>>>>> ok diff-snes_tutorials-ex19_tut_3 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex17_3d_q3_trig_vlap.counts >>>>>>>>>>> ok snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>>> ok diff-snes_tutorials-ex17_3d_q3_trig_vlap >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_3.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_3 # SKIP Fortran required for this test >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist.counts >>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist >>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre.counts >>>>>>>>>>> ok snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>>> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-1_bddc_approx_hypre >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex49_hypre_nullspace.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex49_hypre_nullspace >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex19_superlu_dist_2.counts >>>>>>>>>>> ok snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>>> ok diff-snes_tutorials-ex19_superlu_dist_2 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist_2.counts >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist_2 # Error code: 1 >>>>>>>>>>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist_2 # SKIP Command failed so no diff >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre.counts >>>>>>>>>>> ok snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>>> ok diff-snes_tutorials-ex56_attach_mat_nearnullspace-0_bddc_approx_hypre >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex64_1.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex64_1 >>>>>>>>>>> ok diff-ksp_ksp_tutorials-ex64_1 >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5_superlu_dist.counts >>>>>>>>>>> not ok ksp_ksp_tutorials-ex5_superlu_dist # Error code: 1 >>>>>>>>>>> # srun: error: Unable to create step for job 1426755: More processors requested than permitted >>>>>>>>>>> ok ksp_ksp_tutorials-ex5_superlu_dist # SKIP Command failed so no diff >>>>>>>>>>> TEST arch-linux2-c-opt-ompi/tests/counts/ksp_ksp_tutorials-ex5f_superlu_dist_2.counts >>>>>>>>>>> ok ksp_ksp_tutorials-ex5f_superlu_dist_2 # SKIP Fortran required for this test >>>>>>>>>>> >>>>>>>>>>>> Barry >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On Mar 10, 2021, at 11:03 PM, Eric Chamberland <Eric.Chamberland(a)giref.ulaval.ca <mailto:[email protected]>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Barry, >>>>>>>>>>>>> >>>>>>>>>>>>> to get a some follow up on --with-openmp=1 failures, shall I open gitlab issues for: >>>>>>>>>>>>> >>>>>>>>>>>>> a) all hypre failures giving DIVERGED_INDEFINITE_PC >>>>>>>>>>>>> >>>>>>>>>>>>> b) all superlu_dist failures giving different results with initia and "Exceeded timeout limit of 60 s" >>>>>>>>>>>>> >>>>>>>>>>>>> c) hpddm failures "free(): invalid next size (fast)" and "Segmentation Violation" >>>>>>>>>>>>> >>>>>>>>>>>>> d) all tao's "Exceeded timeout limit of 60 s" >>>>>>>>>>>>> >>>>>>>>>>>>> I don't see how I could do all these debugging by myself... >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Eric >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>>> Professionnel de recherche >>>>>>>>>> GIREF/Université Laval >>>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> -- >>>>>>>>> Eric Chamberland, ing., M. Ing >>>>>>>>> Professionnel de recherche >>>>>>>>> GIREF/Université Laval >>>>>>>>> (418) 656-2131 poste 41 22 42 >>>>>>>>> <fedora_mkl_and_devtools.txt><openmpi.txt><petsc.txt> >>>>>>>> >>>>>>> >>>>>> >>>>> -- >>>>> Eric Chamberland, ing., M. Ing >>>>> Professionnel de recherche >>>>> GIREF/Université Laval >>>>> (418) 656-2131 poste 41 22 42 >>>> >>> -- >>> Eric Chamberland, ing., M. Ing >>> Professionnel de recherche >>> GIREF/Université Laval >>> (418) 656-2131 poste 41 22 42 >> -- >> Eric Chamberland, ing., M. Ing >> Professionnel de recherche >> GIREF/Université Laval >> (418) 656-2131 poste 41 22 42 > -- > Eric Chamberland, ing., M. Ing > Professionnel de recherche > GIREF/Université Laval > (418) 656-2131 poste 41 22 42

1 0

Re: [petsc-dev] reproducing crashes in the test harness
by Scott Kruger 30 Mar '21

30 Mar '21

If you run the full test suite and have failures, you get something like this: # # To rerun failed tests: # make -f gmakefile test test-fail=1 This was turned off for non-full-test cases (search and query cases) in the MR that reduced test harness verbosity. I think what you are wanting is to turn it back on for all cases with this message: # # To rerun all failed tests in debugger: # make -f gmakefile test test-fail=1 DEBUG=1 Is that correct? This is a 2 line fix in `config/report_tests.py` if I understand this correctly. Scott P.S. Please don't forget the help target. It's actually helpful: make -f gmakefile.test help On 2021-03-29 23:25, Barry Smith did write: > > # FAILED snes_tutorials-ex12_quad_hpddm_reuse_threshold snes_tutorials-ex12_p4est_nc_singular_2d_hpddm snes_tutorials-ex56_hpddm snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij sys_tests-ex53_2 snes_tutorials-ex12_quad_hpddm_reuse_baij snes_tutorials-ex12_quad_hpddm_reuse snes_tutorials-ex12_p4est_singular_2d_hpddm snes_tutorials-ex12_tri_parmetis_hpddm snes_tutorials-ex12_quad_singular_hpddm sys_tests-ex26_1 sys_tests-ex26_2 snes_tutorials-ex12_tri_parmetis_hpddm_baij snes_tutorials-ex12_tri_hpddm_reuse_baij snes_tutorials-ex12_tri_hpddm_reus > > Scott, > > Any thoughts on how the test harness could tell the developer exactly how to reproduce a problematic cases in the debugger without them digging around in the code to check arguments etc. > > So for example "Run: mpiexec -n N ./xxx args -start_in_debugger" to reproduce this problem? Then one could just cut and paste and be debugging away. > > Thanks > > Barry > -- Scott Kruger Tech-X Corporation kruger(a)txcorp.com 5621 Arapahoe Ave, Suite A Phone: (720) 466-3196 Boulder, CO 80303 Fax: (303) 448-7756

1 0

Re: [petsc-dev] reproducing crashes in the test harness
by Matthew Knepley 30 Mar '21

30 Mar '21

On Tue, Mar 30, 2021 at 6:52 AM Pierre Jolivet <pierre(a)joliv.et> wrote: > > > On 30 Mar 2021, at 12:37 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > On Tue, Mar 30, 2021 at 12:25 AM Barry Smith <bsmith(a)petsc.dev> wrote: > >> >> # FAILED snes_tutorials-ex12_quad_hpddm_reuse_threshold >> snes_tutorials-ex12_p4est_nc_singular_2d_hpddm snes_tutorials-ex56_hpddm >> snes_tutorials-ex12_quad_hpddm_reuse_threshold_baij sys_tests-ex53_2 >> snes_tutorials-ex12_quad_hpddm_reuse_baij >> snes_tutorials-ex12_quad_hpddm_reuse >> snes_tutorials-ex12_p4est_singular_2d_hpddm >> snes_tutorials-ex12_tri_parmetis_hpddm >> snes_tutorials-ex12_quad_singular_hpddm sys_tests-ex26_1 sys_tests-ex26_2 >> snes_tutorials-ex12_tri_parmetis_hpddm_baij >> snes_tutorials-ex12_tri_hpddm_reuse_baij snes_tutorials-ex12_tri_hpddm_reus >> >> Scott, >> >> Any thoughts on how the test harness could tell the developer exactly >> how to reproduce a problematic cases in the debugger without them digging >> around in the code to check arguments etc. >> >> So for example "Run: mpiexec -n N ./xxx args -start_in_debugger" to >> reproduce this problem? Then one could just cut and paste and be debugging >> away. >> > > I always just EXTRA_ARGUMENTS="-start_in_debugger". However, if you really > want to run standalone, V=1 will display the args I think > > > I don’t think it does. > It displays the path of the .sh script being invoked, but then you have to > decipher that as well. > Also, when there are failures on workers such as diff-test+param, you > can’t copy/paste that directly in $PETSC_ARCH/tests/echofailures.sh, you > have to filter out the diff- and +param part, which is not very practical I > think. > True. I am also not able to reproduce just part of a param test. I need to run all of them. THanks, Matt > Thanks, > Pierre > > Thanks, > > Matt > > >> Thanks >> >> Barry >> >> > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ > <http://www.cse.buffalo.edu/~knepley/> > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

1 0