April 2016 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] plans for PETSc release
by Karl Rupp 24 Apr '16

24 Apr '16

>>> 1) I don't think having a feature freeze in master for a week is tenable at all. Developers want to move stuff along into it and continue work as they should. Plus developers don't like to think that stuff they are working and just finishing won't be in a release for another year so will lobby (as Lisandro did) to slip into the current release. >> >> If that is a concern, one could still fork a branch off master, which gets stabilized for a week and merged to maint once the release is out. The price to pay is the extra backporting to master, which is just as much a nuisance as a feature freeze. > > Why, all you do is merge this "maint" branch to master every time you fix something in it. Still something that requires extra effort and may (even though unlikely) lead to merge conflicts. >> On the other hand, a feature freeze for a week is a perfect opportunity for improving documentation and other work that will not break the code. And even if one prefers to keep coding, new features can still be put in next for testing. Particularly if communicated in advance ("Hey, we will have a feature freeze in 7 days and plan to release in 10 days"), people can easily arrange things. I think Lisandro's feature would have been ready on time if he had known 7 days in advance. > > Nonsense, he knew weeks ahead. Urgency and deadlines are something people tend to consider in their priority queue to schedule their work. I'm merely suggesting that a better communicated release schedule allows us to provide a better release. >>> 2) You seem to think that if we announce a freeze on master (or some branch) dozens of disparate types of users will jump all over it and do massive testing during that week finding all kinds of issues. Maybe that happens with some package's users but not PETSc; we're lucky if one or two people give it a half-hearted try out. >> >> Even if it's only two, it's better than zero. Providing the possibility for users to test (e.g. against their own set of nightly tests) is better than not providing the option at all. > > The option is ALWAYS there. We even have users who run Jenkins or buildbot against master everyday. It's not like there was not a warning that a release was coming out soon. People who want to do the pretesting (and there are very very few of them) had the opportunity. > > Some people like to have a pre-annouced day for a release (thinking if people are told that day, they will actually finish their feature on time or do some testing before then), I've avoided this because we are reliant on people (both in and outside PETSc) whose schedules are erratic, I think people do crappy work with fixed deadlines, and when we've tried it in the past we've always pushed past it anyways so what meaning did it have? A feature freeze will not magically fix things if we've done a crappy job in the months before the release. However, each bug fixed in a feature freeze stage will help avoid repeated support requests later (some users only update between major releases). >>> You are being a bit pedantically Roscoe in this thread. >> >> ;-) Jed is not proposing something that induces more effort, just a more coordinated plan of moving things when and where. > > Actually he is proposing something that induces more effort. For example for a whole freaking week my work-flow of moving stuff that passes the tests in next into master is broken and this induces later more manual merges because other developers started their features/fixes with outdated code. Plus I am suppose to be running around for a week fixing documentation and testing on odd-ball machines. > > If you have a staff of programmers sitting in an office everyday whose schedule revolves completely around a software package they work on full time on the boss can say to them "ok, next week you will be doing testing stop all development". But we are not a staff of programmers sitting in an office whose schedule revolves completely around a software package they work full time, nor do we have a boss. Matt may be in China that week, Jed climbing the Matterhorn, Lisandro working on some cool new feature of PetIAG, Barry pandering to some third rate project we have to interact with for DOE politics, Emil proving some great global error estimator, Karl having another baby. The idea that there can be this magical week for the testing process where more than one PETSc person is able to exert serious effort on is absurd. What Jed proposes* is idealistic, not practical, and would result in just having a less productive week (this is why I compared it to Ross's blathering). The feature freeze is not meant to be enforced on everybody (i.e. writing documentation, testing on funky machines, etc.), but serves as a coordinated and agreed-upon synchronization point, where the end result is the best software package we could come up with. There are some machine configurations I can test if a release is imminent, but are not available for testing on a regular basis for various reasons. When should I run those tests if not right before the release (maximizing relevance)? Best regards, Karli

1 0

Re: [petsc-dev] plans for PETSc release
by Jed Brown 24 Apr '16

24 Apr '16

Barry Smith <bsmith(a)mcs.anl.gov> writes: > Is it really crucial that all the packages that depend on PETSc > have to make their releases exactly when PETSc makes a release? Definitely not. They make releases when it works for them, or not (some projects don't make releases). The point is that they have a chance to test their development branch with PETSc's RC (i.e., 'master' during feature freeze). Yeah, they could test any time, but if new features that they don't know/care about are continually coming into PETSc, there is no motivation to test now versus next week or next month. And I realize most probably won't pay attention, but I think the odds of checking an RC is much higher for people that have contributed to PETSc.

1 0

Re: [petsc-dev] plans for PETSc release
by Barry Smith 24 Apr '16

24 Apr '16

> On Apr 24, 2016, at 4:11 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Barry Smith <bsmith(a)mcs.anl.gov> writes: >> There is still no excuse for not during regular testing >> there. Maybe not daily but at least once a week and automating it >> as much as possible. For example I can envision each Wednesday >> morning at 9 am Satish receiving an email that says: Log into >> mira.alcf.anl.gov and run the script testmira to start the test >> build. Then that script runs all the compiles and ships off to the >> dashboard all results without requiring further actions by a >> human. Each Thursday Jason gets a similar email for hopper etc. It >> is not perfect but it is better than someone manually doing tests >> when they are inspired. The timing of these pseudo automated >> testing could be increased around releases. > > That would improve reliability, but it doesn't exist yet and given an > open dev community assumption, will never cover all such machines. For > example, we've gotten substantial contributions from Europe during this > release cycle. Those developers have packages that depend on PETSc, but > may not have updated in the final days before an (vaguely announced) > release. Not all of them will test if we announce a feature freeze, but > some probably will. After all, it means a lot to their users to have an > easy install and be functional with the latest PETSc. After our release the developers of each of those packages needs to (possibly update and then) test against against the PETSc release and report problems with the PETSc release. We fix those problems in a patch release and then they can make their package release. Is it really crucial that all the packages that depend on PETSc have to make their releases exactly when PETSc makes a release? Probably they are in the middle of their implementation cycle and are not ready to make a new release anyways. I don't think we should even consider having the goal that other packages that use PETSc have the same date of release as PETSc. Yes we want them to be within a couple of months at most, but I don't see setting a goal of the exact same day.

1 0

Re: [petsc-dev] plans for PETSc release
by Barry Smith 24 Apr '16

24 Apr '16

Agreed, the more bugs found before a release (and by us) the better. Expecting or even desiring that our users help in finding the bugs before a release is IMHO unrealistic. > On Apr 24, 2016, at 4:00 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Barry Smith <bsmith(a)mcs.anl.gov> writes: > >>> On Apr 24, 2016, at 1:48 PM, Karl Rupp <rupp(a)iue.tuwien.ac.at> wrote: >>> On the other hand, a feature freeze for a week is a perfect >>> opportunity for improving documentation and other work that will not >>> break the code. And even if one prefers to keep coding, new features >>> can still be put in next for testing. Particularly if communicated in >>> advance ("Hey, we will have a feature freeze in 7 days and plan to >>> release in 10 days"), people can easily arrange things. I think >>> Lisandro's feature would have been ready on time if he had known 7 >>> days in advance. >> >> Nonsense, he knew weeks ahead. > > He knew that a release was coming, but no particular date. > >> The option is ALWAYS there. We even have users who run Jenkins or >> buildbot against master everyday. It's not like there was not a >> warning that a release was coming out soon. People who want to do >> the pretesting (and there are very very few of them) had the >> opportunity. > > However, there was no guarantee that such testing would be relevant to > the release. Someone could be working on some feature that is going > into the release. > >> Some people like to have a pre-annouced day for a release (thinking >> if people are told that day, they will actually finish their >> feature on time or do some testing before then), I've avoided this >> because we are reliant on people (both in and outside PETSc) whose >> schedules are erratic, I think people do crappy work with fixed >> deadlines, and when we've tried it in the past we've always pushed >> past it anyways so what meaning did it have? > > You have announced specific dates in the past. Every conference has a > deadline for paper submission and many/most extend it by a week or two. > I think there would be mass complaints if the organizers instead > announced that they were going to close submissions at some point in the > coming weeks, let some time go by, then said today's the last day for > submissions. > >> Actually he is proposing something that induces more effort. For >> example for a whole freaking week my work-flow of moving stuff that >> passes the tests in next into master is broken and this induces >> later more manual merges because other developers started their >> features/fixes with outdated code. Plus I am suppose to be running >> around for a week fixing documentation and testing on odd-ball >> machines. > > Our features are rarely so interdependent that the feature you implement > today breaks the feature some other developer starts on Tuesday. That > merge isn't really urgent and the marginal cost of waiting a week is > really low. > > But I think the reminder about fixing documentation and any outstanding > bugs/reports is well worth it in terms of improving the quality of the > release. > > Not every user that encounters a bug reports it. While the cost to us > in fixing a bug is about the same if it is discovered and fixed in > 'master' versus 'maint', it is significantly higher for users. > Moreover, when we ship a release with bugs, users encounter that bug for > months, even after we have patched it (they don't instantly and > effortlessly receive the patch release when we tag it), and then we get > the support mail. So bugs that affect multiple users are much more > costly when discovered after the release. > >> If you have a staff of programmers sitting in an office everyday > > [...] > > I have no expectation that everyone participate actively in the release. > But I also don't think it's true that everyone ignores the release > either.

1 0

Re: [petsc-dev] plans for PETSc release
by Jed Brown 24 Apr '16

24 Apr '16

Barry Smith <bsmith(a)mcs.anl.gov> writes: > There is still no excuse for not during regular testing > there. Maybe not daily but at least once a week and automating it > as much as possible. For example I can envision each Wednesday > morning at 9 am Satish receiving an email that says: Log into > mira.alcf.anl.gov and run the script testmira to start the test > build. Then that script runs all the compiles and ships off to the > dashboard all results without requiring further actions by a > human. Each Thursday Jason gets a similar email for hopper etc. It > is not perfect but it is better than someone manually doing tests > when they are inspired. The timing of these pseudo automated > testing could be increased around releases. That would improve reliability, but it doesn't exist yet and given an open dev community assumption, will never cover all such machines. For example, we've gotten substantial contributions from Europe during this release cycle. Those developers have packages that depend on PETSc, but may not have updated in the final days before an (vaguely announced) release. Not all of them will test if we announce a feature freeze, but some probably will. After all, it means a lot to their users to have an easy install and be functional with the latest PETSc.

1 0

Re: [petsc-dev] plans for PETSc release
by Jed Brown 24 Apr '16

24 Apr '16

Barry Smith <bsmith(a)mcs.anl.gov> writes: >> On Apr 24, 2016, at 1:48 PM, Karl Rupp <rupp(a)iue.tuwien.ac.at> wrote: >> On the other hand, a feature freeze for a week is a perfect >> opportunity for improving documentation and other work that will not >> break the code. And even if one prefers to keep coding, new features >> can still be put in next for testing. Particularly if communicated in >> advance ("Hey, we will have a feature freeze in 7 days and plan to >> release in 10 days"), people can easily arrange things. I think >> Lisandro's feature would have been ready on time if he had known 7 >> days in advance. > > Nonsense, he knew weeks ahead. He knew that a release was coming, but no particular date. > The option is ALWAYS there. We even have users who run Jenkins or > buildbot against master everyday. It's not like there was not a > warning that a release was coming out soon. People who want to do > the pretesting (and there are very very few of them) had the > opportunity. However, there was no guarantee that such testing would be relevant to the release. Someone could be working on some feature that is going into the release. > Some people like to have a pre-annouced day for a release (thinking > if people are told that day, they will actually finish their > feature on time or do some testing before then), I've avoided this > because we are reliant on people (both in and outside PETSc) whose > schedules are erratic, I think people do crappy work with fixed > deadlines, and when we've tried it in the past we've always pushed > past it anyways so what meaning did it have? You have announced specific dates in the past. Every conference has a deadline for paper submission and many/most extend it by a week or two. I think there would be mass complaints if the organizers instead announced that they were going to close submissions at some point in the coming weeks, let some time go by, then said today's the last day for submissions. > Actually he is proposing something that induces more effort. For > example for a whole freaking week my work-flow of moving stuff that > passes the tests in next into master is broken and this induces > later more manual merges because other developers started their > features/fixes with outdated code. Plus I am suppose to be running > around for a week fixing documentation and testing on odd-ball > machines. Our features are rarely so interdependent that the feature you implement today breaks the feature some other developer starts on Tuesday. That merge isn't really urgent and the marginal cost of waiting a week is really low. But I think the reminder about fixing documentation and any outstanding bugs/reports is well worth it in terms of improving the quality of the release. Not every user that encounters a bug reports it. While the cost to us in fixing a bug is about the same if it is discovered and fixed in 'master' versus 'maint', it is significantly higher for users. Moreover, when we ship a release with bugs, users encounter that bug for months, even after we have patched it (they don't instantly and effortlessly receive the patch release when we tag it), and then we get the support mail. So bugs that affect multiple users are much more costly when discovered after the release. > If you have a staff of programmers sitting in an office everyday [...] I have no expectation that everyone participate actively in the release. But I also don't think it's true that everyone ignores the release either.

1 0

Re: [petsc-dev] plans for PETSc release
by Barry Smith 24 Apr '16

24 Apr '16

Lisandro, Please send me what to add in dev.html regarding your new APIs for 2nd order systems in TS or add it yourself directly in the maint branch. Thanks Barry > On Apr 24, 2016, at 3:08 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > > On 24 April 2016 at 22:48, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> On the other hand, a feature freeze for a week is a perfect opportunity for improving documentation and other work that will not break the code. And even if one prefers to keep coding, new features can still be put in next for testing. Particularly if communicated in advance ("Hey, we will have a feature freeze in 7 days and plan to release in 10 days"), people can easily arrange things. I think Lisandro's feature would have been ready on time if he had known 7 days in advance. >> >> Nonsense, he knew weeks ahead. > > Dear Barry, your comments are a bit unfair. I've been working hard the > last two months to get my features in. However, in the process of > developing them, I found many issues in the TS interface as well as > implementation that needed proper fixing before adding my stuff. I > could be an irresponsible guy, and just put my crap in, add hacks here > and there as needed to get my stuff work as I want, and do not care at > all about all the cleanup and refactoring that was required. It was > not easy, I had to touch many pieces that interact each other, in the > process I broke the builds more than once as you likely remember. And > still there are things that should be fixed or done better. > > At this point, if PETSc 3.7.0 has to be released because of political > issues, and we will accept adding new forward-compatible APIs in > 3.7.1, then that's fine with me. It is not my intention to delay the > release or put you in the pressure of accepting my new features if > that does not convince you. If you guys decide that adding APIs in > 3.7.1 is not acceptable, then I'll be a little bit disappointed, but > is not the end of the world, I should be able to live with that :-) > > > > > > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459

1 0

Re: [petsc-dev] plans for PETSc release
by Barry Smith 24 Apr '16

24 Apr '16

> On Apr 24, 2016, at 3:08 PM, Lisandro Dalcin <dalcinl(a)gmail.com> wrote: > > On 24 April 2016 at 22:48, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >>> On the other hand, a feature freeze for a week is a perfect opportunity for improving documentation and other work that will not break the code. And even if one prefers to keep coding, new features can still be put in next for testing. Particularly if communicated in advance ("Hey, we will have a feature freeze in 7 days and plan to release in 10 days"), people can easily arrange things. I think Lisandro's feature would have been ready on time if he had known 7 days in advance. >> >> Nonsense, he knew weeks ahead. Lisandro, My only point in stating this was that you were aware a release was coming up, not that you had done anything wrong at all. I know you have been devoting a great deal of time to adding TS features and fixing many things in TS and we all appreciate it. There was a lot of cleanup in TS needed and there was not enough time for you to get it all done plus add features before the (artificial) release date. I apologize for any offense, I should have stated it more clearly that what I meant to say was "knowing when a release is coming doesn't magically make more time for someone to finish their work". I am fine with putting your features in the release 3.7; the advantage of them getting in 3.7 is that they can be mentioned in the changes file (dev.html) and in the release announcement. If we put them in 3.7.1 then there is a much bigger chance people will not notice them. You have two branches 2016-04-24 16:06:35 +0300 (origin/dalcinl/tsbdf) TS: Add BDF implementation 2016-04-24 16:28:04 +0300 (origin/dalcinl/ts-2nd-order-systems) TSALPHA2: Add tutorial example ex44 I will put them in maint now and they can be tested in the daily before Satish makes the tarball on Monday. If anything breaks we can deal with it, no big deal. Barry > > Dear Barry, your comments are a bit unfair. I've been working hard the > last two months to get my features in. However, in the process of > developing them, I found many issues in the TS interface as well as > implementation that needed proper fixing before adding my stuff. I > could be an irresponsible guy, and just put my crap in, add hacks here > and there as needed to get my stuff work as I want, and do not care at > all about all the cleanup and refactoring that was required. It was > not easy, I had to touch many pieces that interact each other, in the > process I broke the builds more than once as you likely remember. And > still there are things that should be fixed or done better. > > At this point, if PETSc 3.7.0 has to be released because of political > issues, and we will accept adding new forward-compatible APIs in > 3.7.1, then that's fine with me. It is not my intention to delay the > release or put you in the pressure of accepting my new features if > that does not convince you. If you guys decide that adding APIs in > 3.7.1 is not acceptable, then I'll be a little bit disappointed, but > is not the end of the world, I should be able to live with that :-) > > > > > > > -- > Lisandro Dalcin > ============ > Research Scientist > Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > Extreme Computing Research Center (ECRC) > King Abdullah University of Science and Technology (KAUST) > http://ecrc.kaust.edu.sa/ > > 4700 King Abdullah University of Science and Technology > al-Khawarizmi Bldg (Bldg 1), Office # 0109 > Thuwal 23955-6900, Kingdom of Saudi Arabia > http://www.kaust.edu.sa > > Office Phone: +966 12 808-0459

1 0

Re: [petsc-dev] plans for PETSc release
by Barry Smith 24 Apr '16

24 Apr '16

> On Apr 24, 2016, at 3:26 PM, Jed Brown <jed(a)jedbrown.org> wrote: > > Barry Smith <bsmith(a)mcs.anl.gov> writes: > >> No PETSc developer/user has ever done or is likely ever to do all >> the cool testing you think will happen during a feature freeze; >> your fantasies are quite bizarre :-). > > Well, I've done this at least with PFLOTRAN and libMesh/MOOSE for > various previous releases. I'm pretty sure Lisandro has done similar > with some of his packages and Matt with PyLith. I've tested on NERSC > and ALCF early access machines. I thought others had been doing similar > as time and access allowed. > >> Plus if there are testing environments not in the nightly builds >> that one "could" try during the freeze week the priority should be >> getting those environments into the nightly builds, not just >> testing on them one week a year. > > The machines I'm thinking of are managed by other people and typically > not convenient to have running automated tests (sometimes due to > security like cryptocard logins). When we test a version there, we're > testing both that version of PETSc and the environment on the machine, > which are often being "upgraded" periodically. There is still no excuse for not during regular testing there. Maybe not daily but at least once a week and automating it as much as possible. For example I can envision each Wednesday morning at 9 am Satish receiving an email that says: Log into mira.alcf.anl.gov and run the script testmira to start the test build. Then that script runs all the compiles and ships off to the dashboard all results without requiring further actions by a human. Each Thursday Jason gets a similar email for hopper etc. It is not perfect but it is better than someone manually doing tests when they are inspired. The timing of these pseudo automated testing could be increased around releases. > >> This is also the first time that I was aware that we ever did a >> week long feature freeze in the past! We've always done the move it >> quickly into a release approach. > > We've usually had precise deadlines announced in advance, with the > release being tagged some days later. It was a literal code freeze back > in v3.3, but you're right that it has been more variable and ad-hoc than > I remembered. If you look at 3.3 log merges I suspect you'll see that the "code freeze" was more "in spirit" and plenty of stuff that was not "fixes" got pushed during that long time frame. > > > v3.6: > > 2015-05-28: bsmith on petsc-dev "Satish and I are planning a PETSc > release by Friday June 5" > > 2015-06-09 Release tagged > > v3.5: > > 2014-06-12: bsmith on petsc-dev "Please get all changes into master > before 9pm central time USA on Sunday June 15th that you would like in > the release. At that time Satish will make a release branch and the > only changes allowed in that release branch will be bug fixes that do > not change the API." > > 2014-06-30: Release tagged > > v3.4 seems to have been relatively stealth, though there was plenty of > off-list discussion. This was shortly after we switched to Git. > > v3.3: > > 2012-05-17: bsmith on petsc-dev "Recall that we are in a phase of a > code freeze that means do not push random new code to petsc-dev, push > only bug fixes and clean up to petsc-dev. By all means keep on with > code development, just don't push the new stuff to petsc-dev. > > We hope to make the release repository (and then tarball) next > Wednesday May 23. So please do be checking the nightly builds and > making fixes to petsc-dev." > > 2012-06-05: Release tagged

1 0

Re: [petsc-dev] plans for PETSc release
by Jed Brown 24 Apr '16

24 Apr '16

Barry Smith <bsmith(a)mcs.anl.gov> writes: > No PETSc developer/user has ever done or is likely ever to do all > the cool testing you think will happen during a feature freeze; > your fantasies are quite bizarre :-). Well, I've done this at least with PFLOTRAN and libMesh/MOOSE for various previous releases. I'm pretty sure Lisandro has done similar with some of his packages and Matt with PyLith. I've tested on NERSC and ALCF early access machines. I thought others had been doing similar as time and access allowed. > Plus if there are testing environments not in the nightly builds > that one "could" try during the freeze week the priority should be > getting those environments into the nightly builds, not just > testing on them one week a year. The machines I'm thinking of are managed by other people and typically not convenient to have running automated tests (sometimes due to security like cryptocard logins). When we test a version there, we're testing both that version of PETSc and the environment on the machine, which are often being "upgraded" periodically. > This is also the first time that I was aware that we ever did a > week long feature freeze in the past! We've always done the move it > quickly into a release approach. We've usually had precise deadlines announced in advance, with the release being tagged some days later. It was a literal code freeze back in v3.3, but you're right that it has been more variable and ad-hoc than I remembered. v3.6: 2015-05-28: bsmith on petsc-dev "Satish and I are planning a PETSc release by Friday June 5" 2015-06-09 Release tagged v3.5: 2014-06-12: bsmith on petsc-dev "Please get all changes into master before 9pm central time USA on Sunday June 15th that you would like in the release. At that time Satish will make a release branch and the only changes allowed in that release branch will be bug fixes that do not change the API." 2014-06-30: Release tagged v3.4 seems to have been relatively stealth, though there was plenty of off-list discussion. This was shortly after we switched to Git. v3.3: 2012-05-17: bsmith on petsc-dev "Recall that we are in a phase of a code freeze that means do not push random new code to petsc-dev, push only bug fixes and clean up to petsc-dev. By all means keep on with code development, just don't push the new stuff to petsc-dev. We hope to make the release repository (and then tarball) next Wednesday May 23. So please do be checking the nightly builds and making fixes to petsc-dev." 2012-06-05: Release tagged

1 0