September 2017 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] hypre + OMP and MKL-AIJ
by Kalinkin, Alexander A 14 Sep '17

14 Sep '17

HI Mark, Looks like we can implement this functionality (if everything goes correctly) in MKL 2018u2 which will be published during following January On our side we will test Both MKL triple implementation and wrappers from PETSc to MKL much early so if any petsc user will show interest to try this functionality we can commit PETSc wrapper early and provide engineering build with correspond function. Thanks, Alex From: Mark Adams [mailto:[email protected]] Sent: Thursday, September 14, 2017 1:18 PM To: Kalinkin, Alexander A <alexander.a.kalinkin(a)intel.com> Cc: Richard Tran Mills <rtmills(a)anl.gov>; For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov>; Sokolova, Irina <irina.sokolova(a)intel.com> Subject: Re: [petsc-dev] hypre + OMP and MKL-AIJ On Thu, Sep 14, 2017 at 3:25 PM, Kalinkin, Alexander A <alexander.a.kalinkin(a)intel.com<mailto:[email protected]>> wrote: Hi All, Of course we can if you need it but I see the problem with testing – we need to implement functionality and test it simultaneously with MatMatMult() wrappers in petsc branch. Any thoughts how it can be realized? I've warned my users that this new. If you can get snes/examples/tutorials/ex56 to work then you are probably ready for users. My users will scale up and can work with you to debug. Thanks, Thanks, Alex From: Richard Tran Mills [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, September 14, 2017 11:45 AM To: Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> Cc: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov<mailto:[email protected]>>; Kalinkin, Alexander A <alexander.a.kalinkin(a)intel.com<mailto:[email protected]>>; Sokolova, Irina <irina.sokolova(a)intel.com<mailto:[email protected]>> Subject: Re: [petsc-dev] hypre + OMP and MKL-AIJ Hi Mark, Regarding MatMatMult() support, I think it should be fairly trivial for me to add that to AIJMKL. However, MKL doesn't currently have a matrix triple product routine (though the possibility of adding this has come up before in discussions with Alex Kalinkin from the MKL sparse team), so for now we'd need to implement that using MatMatMult(). Perhaps Alex or Irina (cc'd) can say something about possible support for sparse matrix triple products in MKL? --Richard On Thu, Sep 14, 2017 at 11:32 AM, Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: I recall Barry saying that he updated the hypre interface after the last hypre release, which includes OpenMP. But, I am not finding the email. Can someone tell me the status of this? Note, I have two users that are interested in using threads with AMG. I think we would be interested in testing the new MKL-AIJ matrix but as I understand it we are waiting on the Mat-Mat Mult method(s) to get migrated. I should have at least one user that would be interested in testing this when it is ready. Thanks, Mark

1 0

Re: [petsc-dev] hypre + OMP and MKL-AIJ
by Mark Adams 14 Sep '17

14 Sep '17

On Thu, Sep 14, 2017 at 3:25 PM, Kalinkin, Alexander A < alexander.a.kalinkin(a)intel.com> wrote: > Hi All, > > Of course we can if you need it but I see the problem with testing – we > need to implement functionality and test it simultaneously with > MatMatMult() wrappers in petsc branch. Any thoughts how it can be realized? > I've warned my users that this new. If you can get snes/examples/tutorials/ex56 to work then you are probably ready for users. My users will scale up and can work with you to debug. Thanks, > Thanks, > > Alex > > > > *From:* Richard Tran Mills [mailto:[email protected]] > *Sent:* Thursday, September 14, 2017 11:45 AM > *To:* Mark Adams <mfadams(a)lbl.gov> > *Cc:* For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov>; > Kalinkin, Alexander A <alexander.a.kalinkin(a)intel.com>; Sokolova, Irina < > irina.sokolova(a)intel.com> > *Subject:* Re: [petsc-dev] hypre + OMP and MKL-AIJ > > > > Hi Mark, > > Regarding MatMatMult() support, I think it should be fairly trivial for me > to add that to AIJMKL. However, MKL doesn't currently have a matrix triple > product routine (though the possibility of adding this has come up before > in discussions with Alex Kalinkin from the MKL sparse team), so for now > we'd need to implement that using MatMatMult(). > > Perhaps Alex or Irina (cc'd) can say something about possible support for > sparse matrix triple products in MKL? > > --Richard > > > > On Thu, Sep 14, 2017 at 11:32 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I recall Barry saying that he updated the hypre interface after the last > hypre release, which includes OpenMP. But, I am not finding the email. Can > someone tell me the status of this? > > > > Note, I have two users that are interested in using threads with AMG. I > think we would be interested in testing the new MKL-AIJ matrix but as I > understand it we are waiting on the Mat-Mat Mult method(s) to get migrated. > I should have at least one user that would be interested in testing this > when it is ready. > > > > Thanks, > > Mark > > >

1 0

Re: [petsc-dev] hypre + OMP and MKL-AIJ
by Mark Adams 14 Sep '17

14 Sep '17

On Thu, Sep 14, 2017 at 2:45 PM, Richard Tran Mills <rtmills(a)anl.gov> wrote: > Hi Mark, > > Regarding MatMatMult() support, I think it should be fairly trivial for me > to add that to AIJMKL. However, MKL doesn't currently have a matrix triple > product routine (though the possibility of adding this has come up before > in discussions with Alex Kalinkin from the MKL sparse team), so for now > we'd need to implement that using MatMatMult(). > > Perhaps Alex or Irina (cc'd) can say something about possible support for > sparse matrix triple products in MKL? > I don't mind if you wrap two Mat-Mat into the P'AP. Both of my apps are linear so this is amortized. Thanks, > > > --Richard > > On Thu, Sep 14, 2017 at 11:32 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > >> I recall Barry saying that he updated the hypre interface after the last >> hypre release, which includes OpenMP. But, I am not finding the email. Can >> someone tell me the status of this? >> >> Note, I have two users that are interested in using threads with AMG. I >> think we would be interested in testing the new MKL-AIJ matrix but as I >> understand it we are waiting on the Mat-Mat Mult method(s) to get migrated. >> I should have at least one user that would be interested in testing this >> when it is ready. >> >> Thanks, >> Mark >> > >

1 0

Re: [petsc-dev] hypre + OMP and MKL-AIJ
by Kalinkin, Alexander A 14 Sep '17

14 Sep '17

Hi All, Of course we can if you need it but I see the problem with testing – we need to implement functionality and test it simultaneously with MatMatMult() wrappers in petsc branch. Any thoughts how it can be realized? Thanks, Alex From: Richard Tran Mills [mailto:[email protected]] Sent: Thursday, September 14, 2017 11:45 AM To: Mark Adams <mfadams(a)lbl.gov> Cc: For users of the development version of PETSc <petsc-dev(a)mcs.anl.gov>; Kalinkin, Alexander A <alexander.a.kalinkin(a)intel.com>; Sokolova, Irina <irina.sokolova(a)intel.com> Subject: Re: [petsc-dev] hypre + OMP and MKL-AIJ Hi Mark, Regarding MatMatMult() support, I think it should be fairly trivial for me to add that to AIJMKL. However, MKL doesn't currently have a matrix triple product routine (though the possibility of adding this has come up before in discussions with Alex Kalinkin from the MKL sparse team), so for now we'd need to implement that using MatMatMult(). Perhaps Alex or Irina (cc'd) can say something about possible support for sparse matrix triple products in MKL? --Richard On Thu, Sep 14, 2017 at 11:32 AM, Mark Adams <mfadams(a)lbl.gov<mailto:[email protected]>> wrote: I recall Barry saying that he updated the hypre interface after the last hypre release, which includes OpenMP. But, I am not finding the email. Can someone tell me the status of this? Note, I have two users that are interested in using threads with AMG. I think we would be interested in testing the new MKL-AIJ matrix but as I understand it we are waiting on the Mat-Mat Mult method(s) to get migrated. I should have at least one user that would be interested in testing this when it is ready. Thanks, Mark

1 0

Re: [petsc-dev] hypre + OMP and MKL-AIJ
by Barry Smith 14 Sep '17

14 Sep '17

> On Sep 14, 2017, at 1:32 PM, Mark Adams <mfadams(a)lbl.gov> wrote: > > I recall Barry saying that he updated the hypre interface after the last hypre release, which includes OpenMP. But, I am not finding the email. Can someone tell me the status of this? Just build in master with --with-openmp --download-hypre then at runtime make sure you set the damn OpenMP environmental variable to the number of OpenMP threads you want to use (Best interface design EVER!). Report problems to petsc-maint. Similar quality interface for SuperLU_Dist and presumably for MKL-AIJ matrices. Barry > > Note, I have two users that are interested in using threads with AMG. I think we would be interested in testing the new MKL-AIJ matrix but as I understand it we are waiting on the Mat-Mat Mult method(s) to get migrated. I should have at least one user that would be interested in testing this when it is ready. > > Thanks, > Mark

1 0

Re: [petsc-dev] hypre + OMP and MKL-AIJ
by Richard Tran Mills 14 Sep '17

14 Sep '17

Hi Mark, Regarding MatMatMult() support, I think it should be fairly trivial for me to add that to AIJMKL. However, MKL doesn't currently have a matrix triple product routine (though the possibility of adding this has come up before in discussions with Alex Kalinkin from the MKL sparse team), so for now we'd need to implement that using MatMatMult(). Perhaps Alex or Irina (cc'd) can say something about possible support for sparse matrix triple products in MKL? --Richard On Thu, Sep 14, 2017 at 11:32 AM, Mark Adams <mfadams(a)lbl.gov> wrote: > I recall Barry saying that he updated the hypre interface after the last > hypre release, which includes OpenMP. But, I am not finding the email. Can > someone tell me the status of this? > > Note, I have two users that are interested in using threads with AMG. I > think we would be interested in testing the new MKL-AIJ matrix but as I > understand it we are waiting on the Mat-Mat Mult method(s) to get migrated. > I should have at least one user that would be interested in testing this > when it is ready. > > Thanks, > Mark >

1 0

hypre + OMP and MKL-AIJ
by Mark Adams 14 Sep '17

14 Sep '17

I recall Barry saying that he updated the hypre interface after the last hypre release, which includes OpenMP. But, I am not finding the email. Can someone tell me the status of this? Note, I have two users that are interested in using threads with AMG. I think we would be interested in testing the new MKL-AIJ matrix but as I understand it we are waiting on the Mat-Mat Mult method(s) to get migrated. I should have at least one user that would be interested in testing this when it is ready. Thanks, Mark

1 0

Re: [petsc-dev] TSComputeRHSJacobian
by Stefano Zampini 14 Sep '17

14 Sep '17

Fair enough. Here you have an example. You need to checkout the branch stefano_zampini/feature-continuousadjoint and apply patch_ex23 to src/ts/examples/tutorials/ex23.c The patch just dumps the final RHS jacobian matrix of a standard TSSolve, recomputes it with by calling TSComputeRHSJacobian, and dumps it again. If you run ex23 using the RHS interface and an implicit solver, you see from the first MatView that ts->Arhs is shifted. After a call to TSComputeRHSJacobian it is ok. [szampini@localhost tutorials]$ ./ex23 -t0 0 -tf 1 -dt 0.5 -p 1 -ts_type bdf Mat Object: 1 MPI processes type: seqaij row 0: (0, 13.2008) Mat Object: 1 MPI processes type: seqaij row 0: (0, 0.7) Objective function: time [0.,1.], val 0.725152 (should be 0.724109) Vec Object: 1 MPI processes type: seq 1.45054 0.404627 -0.126697 If you apply patch_ts (which disables my fix at https://bitbucket.org/petsc/petsc/branch/stefano_zampini/feature-continuous…) you see that the matrix is unchanged after TSComputeRHSJacobian [szampini@localhost tutorials]$ ./ex23 -t0 0 -tf 1 -dt 0.5 -p 1 -ts_type bdf Mat Object: 1 MPI processes type: seqaij row 0: (0, 13.2008) Mat Object: 1 MPI processes type: seqaij row 0: (0, 13.2008) Objective function: time [0.,1.], val 0.725152 (should be 0.724109) Vec Object: 1 MPI processes type: seq 1.45054 0.404627 -0.126697 2017-09-10 22:43 GMT+03:00 Barry Smith <bsmith(a)mcs.anl.gov>: > > Stefano, > > Thanks, but I don't really want a fix. I want a code that I can run > that demonstrates a problem (then we can evaluate fixes) > > Barry > > > On Sep 8, 2017, at 11:51 AM, Stefano Zampini <stefano.zampini(a)gmail.com> > wrote: > > > > I encountered the problem while working on the adjoint code. Here is the > fix I have in my branch > > > > https://bitbucket.org/petsc/petsc/branch/stefano_zampini/ > feature-continuousadjoint#Lsrc/ts/interface/ts.cT613 > > > > However, I'm not sure if this is a problem just with my code. This is > why I'm asking on the list. > > > > > > > > 2017-09-08 0:02 GMT+03:00 Barry Smith <bsmith(a)mcs.anl.gov>: > > > > Example that demonstrates the current code is problematic in some > cases? > > > > > On Aug 28, 2017, at 11:59 AM, Stefano Zampini < > stefano.zampini(a)gmail.com> wrote: > > > > > > > > > I was wondering if the following code in TSComputeRHSJacobian is > correct when the user pass the reuse flag via TSRHSJacobianReuse > > > > > > if (ts->rhsjacobian.time == t && (ts->problem_type == TS_LINEAR || > (ts->rhsjacobian.Xid == Uid && ts->rhsjacobian.Xstate == Ustate)) && > (rhsfunction != TSComputeRHSFunctionLinear)) { > > > PetscFunctionReturn(0); > > > } > > > > > > Should the above code be replaced by the following? > > > > > > if (ts->rhsjacobian.time == t && (ts->problem_type == TS_LINEAR || > (ts->rhsjacobian.Xid == Uid && ts->rhsjacobian.Xstate == Ustate)) && > (rhsfunction != TSComputeRHSFunctionLinear)) { > > > if (ts->rhsjacobian.reuse) { > > > ierr = MatShift(A,-ts->rhsjacobian.shift);CHKERRQ(ierr); > > > ierr = MatScale(A,1./ts->rhsjacobian.scale);CHKERRQ(ierr); > > > if (B && A != B) { > > > ierr = MatShift(B,-ts->rhsjacobian.shift);CHKERRQ(ierr); > > > ierr = MatScale(B,1./ts->rhsjacobian.scale);CHKERRQ(ierr); > > > } > > > ts->rhsjacobian.shift = 0; > > > ts->rhsjacobian.scale = 1.; > > > } > > > PetscFunctionReturn(0); > > > } > > > > > > > > > -- > > > Stefano > > > > > > > > > > -- > > Stefano > > -- Stefano

1 0

Re: [petsc-dev] possibility of short term subcontract work with PETSc
by Sean Farley 13 Sep '17

13 Sep '17

Barry Smith <bsmith(a)mcs.anl.gov> writes: > PETSc folks, > > Argonne National Laboratory has recently set up a system that may make it possible for the PETSc group at ANL to subcontract particular PETSc contribution projects to developers in most of the world. These could be from one week projects to months. Nice, that's pretty cool!

1 0

Re: [petsc-dev] ParMETIS change
by Eric Chamberland 13 Sep '17

13 Sep '17

No "big" probleme here... just curious... I changed our code to use real_t and everything is fine now. Just wanted to understand the reasons why it changed! Thanks a lot! Eric On 13/09/17 10:07 AM, Balay, Satish wrote: > What errors do you get? The primary code change was to use real_t > instead of PetscReal in calls to parmetis. > > The change was done to make parmetis built by petsc be same as default > parmetis that others/distros might build. > > And this enables consistent usage from quad build of petsc. > > I can add ï¿½download-metis-double or similar option to configure ï¿½ if useful. > > Satish > > *From: *Eric Chamberland <mailto:[email protected]> > *Sent: *Wednesday, September 13, 2017 8:41 AM > *To: *petsc-dev <mailto:[email protected]> > *Subject: *[petsc-dev] ParMETIS change > > Hi, > > having trouble to compile our code with PETSc-master new defaults > options for PARMEtis. > > Parmetis was using double precision before... > > By curiosity, is there a big performance gain to use float? > > Thanks, > > Eric > >

1 0