February 2013 - petsc-dev - mailman3.cels.anl.gov

Re: [petsc-dev] Example cleanup
by Jed Brown 10 Feb '13

10 Feb '13

On Sun, Feb 10, 2013 at 8:40 AM, Matthew Knepley <knepley(a)gmail.com> wrote: > The phrase "may as well" is very slippery. Python arrays are just [x, y, > z], whereas JSON needs a library. > However, the format is is now way crucial to the scheme. I want to get the > workflow up an running with > the simplest format. We will inevitably tweak it was we add tons of crap > on top if it works. > The syntax for plain arrays is identical. I was suggesting JSON to manage recursive data structures (nested solvers, perhaps monitoring different things). So it becomes a dict of arrays or a dict of dicts of arrays. Also, we could spit out -snes_view in machine-readable form, we could have nicer diagnostics on the GUI side. It is because of these extensions that I think it's worth starting with something that will extend gracefully rather than a hack that is only intended for arrays of numbers.

1 0

Re: [petsc-dev] Example cleanup
by Matthew Knepley 10 Feb '13

10 Feb '13

On Sun, Feb 10, 2013 at 1:05 AM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Sat, Feb 9, 2013 at 8:04 PM, Matthew Knepley <knepley(a)gmail.com>wrote: >> >> What's your plan for implementing this? >>> >> >> My plan was to try the easiest thing I could think of first: >> >> a) Make a replacement monitor for SNES and KSP >> >> It would have a big honking sentinel before each solve and >> iterations in Python array format >> >> b) Run a braindead Python parser over this which snips out the monitors >> >> c) Diff the monitor sections with a numerical tolerance >> >> d) Diff the whole thing, and throw out differences in the monitor >> sections, since we know the line numbers. >> Maybe it would be easier to just remove those lines, but we need to >> experiment a little. >> > > Okay, so this would all be implemented in C? > Does the Bear(y) shit in the woods? > If you want Python array format, you may as well use JSON since it's more > of an exchange standard and it's the most convenient format for monitoring > in a web-based GUI. There are a bunch of good JSON libraries, ranging from > the ultra-compact http://zserge.bitbucket.org/jsmn.html and > http://ccodearchive.net/info/json.html, to the object-oriented > http://www.digip.org/jansson/doc/dev/tutorial.html. > The phrase "may as well" is very slippery. Python arrays are just [x, y, z], whereas JSON needs a library. However, the format is is now way crucial to the scheme. I want to get the workflow up an running with the simplest format. We will inevitably tweak it was we add tons of crap on top if it works. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] Example cleanup
by Jed Brown 10 Feb '13

10 Feb '13

On Sat, Feb 9, 2013 at 8:04 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > What's your plan for implementing this? >> > > My plan was to try the easiest thing I could think of first: > > a) Make a replacement monitor for SNES and KSP > > It would have a big honking sentinel before each solve and > iterations in Python array format > > b) Run a braindead Python parser over this which snips out the monitors > > c) Diff the monitor sections with a numerical tolerance > > d) Diff the whole thing, and throw out differences in the monitor > sections, since we know the line numbers. > Maybe it would be easier to just remove those lines, but we need to > experiment a little. > Okay, so this would all be implemented in C? If you want Python array format, you may as well use JSON since it's more of an exchange standard and it's the most convenient format for monitoring in a web-based GUI. There are a bunch of good JSON libraries, ranging from the ultra-compact http://zserge.bitbucket.org/jsmn.html and http://ccodearchive.net/info/json.html, to the object-oriented http://www.digip.org/jansson/doc/dev/tutorial.html.

1 0

Re: [petsc-dev] Example cleanup
by Matthew Knepley 09 Feb '13

09 Feb '13

On Sat, Feb 9, 2013 at 8:23 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Sat, Feb 9, 2013 at 7:15 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > >> On Sat, Feb 9, 2013 at 6:52 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: >> >>> >>> On Feb 8, 2013, at 11:04 PM, Matthew Knepley <knepley(a)gmail.com> wrote: >>> >>> > There are some broken examples in SNES that I would like to clean up >>> in preparation for real output checking there. They are >>> > >>> > ex5: I think we just overwrite the solutions with the current stuff >>> > >>> > ex22 and ex58: Barry this look like you, and the results are really >>> different >>> >>> The new values are no less valid than the old so just overwrite the >>> old outputs. >> >> >> Okay, pushed. >> > > The ex5 output changed here, for a justifiable reason, though the code got > much messier to implement this change. > > > https://bitbucket.org/petsc/petsc-dev/commits/4c6428f5d4f1623dc2ea08f9e0be1… > > Tests output needs to be updated when an intentional change like this is > made. > I completely agree. That is why I want to get to a believable state soon. Then we have nightly tests behaving like nightly builds. Breakage is fixed immediately. > I didn't bisect for ex22 or ex58, but perhaps it's a similar issue. > > >> Next step, insert Python validator for output, possibly with more >> structure. >> > > What's your plan for implementing this? > My plan was to try the easiest thing I could think of first: a) Make a replacement monitor for SNES and KSP It would have a big honking sentinel before each solve and iterations in Python array format b) Run a braindead Python parser over this which snips out the monitors c) Diff the monitor sections with a numerical tolerance d) Diff the whole thing, and throw out differences in the monitor sections, since we know the line numbers. Maybe it would be easier to just remove those lines, but we need to experiment a little. Matt -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] private branch of petsc-dev (fwd)
by Jed Brown 09 Feb '13

09 Feb '13

On Sat, Feb 9, 2013 at 6:56 PM, Dmitry Karpeev <karpeev(a)mcs.anl.gov> wrote: > So, this resolution is applied to jed/test and also saved [I'm guessing > the detached HEAD is this stashed resolution?) Whoops, I copied the log message from a different context rather than setting up a full example. There's no detached HEAD in the setup I explained. (Junio's post suggests doing the throw-away merge in detached HEAD only because it saves the step of deleting the throw-away branch.) Implementation-wise, the resolution is stored in .git/rr-cache. If interacting with other people on jed/feature2, I might publish my "throw-away" merge so that other people could access the merged state in their own throw-away merges. (This is the purpose of 'pu' in kerne/git workflow.) > to be applied again later to master when jed/feature2 is finally merged > with the upstream. Meanwhile you work on jed/feature2 without this > resolution applied, so you don't actually see the code as it will end up. I'm assuming here that the conflict is incidental, not core to the way feature2 functions. Common examples are adjacent lines in headers or registration routines. If the merge is not helpful for the implementation and proper functioning of feature2, they shouldn't exist in the jed/feature2 branch. > Wouldn't that be confusing? I imagine this workflow works and "preserves > the ... testing" because the testing is done on jed/test, which contains > the resolutions of all of the conflicts with the upstream. I feel like I > would find it disorienting working on one branch, testing in another, then > fixing whatever problems in the first, testing again in the second, and so > on. What am I missing? You test in the feature branch, the purpose of which is to tell a complete, but uncluttered story of the implementation of that feature. The purpose of the throw-away branch is (a) to confirm that we don't have real conflicts that will be messy to sort out later and (b) to run with the latest experimental everything, e.g., if an application is prepared to leverage many new features at once and we don't have a suitable test. This blog post has a good discussion of the purpose of branches and merges. http://gitster.livejournal.com/42247.html

1 0

Re: [petsc-dev] Example cleanup
by Jed Brown 09 Feb '13

09 Feb '13

On Sat, Feb 9, 2013 at 7:15 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > On Sat, Feb 9, 2013 at 6:52 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > >> >> On Feb 8, 2013, at 11:04 PM, Matthew Knepley <knepley(a)gmail.com> wrote: >> >> > There are some broken examples in SNES that I would like to clean up in >> preparation for real output checking there. They are >> > >> > ex5: I think we just overwrite the solutions with the current stuff >> > >> > ex22 and ex58: Barry this look like you, and the results are really >> different >> >> The new values are no less valid than the old so just overwrite the >> old outputs. > > > Okay, pushed. > The ex5 output changed here, for a justifiable reason, though the code got much messier to implement this change. https://bitbucket.org/petsc/petsc-dev/commits/4c6428f5d4f1623dc2ea08f9e0be1… Tests output needs to be updated when an intentional change like this is made. I didn't bisect for ex22 or ex58, but perhaps it's a similar issue. > Next step, insert Python validator for output, possibly with more > structure. > What's your plan for implementing this?

1 0

Re: [petsc-dev] Example cleanup
by Matthew Knepley 09 Feb '13

09 Feb '13

On Sat, Feb 9, 2013 at 6:52 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > > On Feb 8, 2013, at 11:04 PM, Matthew Knepley <knepley(a)gmail.com> wrote: > > > There are some broken examples in SNES that I would like to clean up in > preparation for real output checking there. They are > > > > ex5: I think we just overwrite the solutions with the current stuff > > > > ex22 and ex58: Barry this look like you, and the results are really > different > > The new values are no less valid than the old so just overwrite the old > outputs. Okay, pushed. Next step, insert Python validator for output, possibly with more structure. Matt > > > > Once regression passes, I am going to mess around with the output > > a little so I can get it easily interpreted by Python, and then checked > > automatically with nice tolerances. > > > > Matt > > > > -- > > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > > -- Norbert Wiener > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener

1 0

Re: [petsc-dev] private branch of petsc-dev (fwd)
by Dmitry Karpeev 09 Feb '13

09 Feb '13

On Sat, Feb 9, 2013 at 12:39 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > On Fri, Feb 8, 2013 at 7:48 PM, Satish Balay <balay(a)mcs.anl.gov> wrote: > >> >> > > - *never* sync with petsc-dev until this work is done. >> > > >> > > - merge with petsc-dev when its done. [either merge - or rebase it >> over] >> > > >> > > The *never* sync with petsc-dev is a drawback for this. >> > >> > You could just `hg pull --rebase`? Unless you mean don't push to >> > petsc-dev, then yes, don't push to petsc-dev until the feature is done. >> >> 'hg pull --rebase' would be great for a single developer of the >> feature. But with 2 developers commiting/communicating commits - one >> would have to 'strip out the old changes' - as you put it - either >> manually or with mercurial tools - and then propogate this to remote >> repo aswell. So its easy to make mistakes in the workflow? >> > > A feature like MINRESQLP should be able to go a very long time before > conflicting. It's usually considered bad practice to constantly merge > (because most merge commits don't convey useful content), but it's also > useful to know when something will automerge successfully and sometimes we > want to test with all known patches applied. In git, I use a personal > testing branch for this (jed/test in the discussion below). I explain my > git workflow below. It's mildly advanced, but hopefully explains how to > achieve > > *1.* Many people can work independently with minimal interruptions. > *2.* Every commit that eventually appears in a release (to be tagged from > 'master') is semantically significant and preserves the integrity and > testing performed the first time it is committed with intent to publish. > (You can rework locally as much as you want, but the SHA1 is indelible once > you decide that a commit is complete.) This implies zero content-free > merges and zero rebases for topic branches. > *3.* Every commit can be reviewed before it graduates to 'master'. > *4.* The process can be applied recursively and scales naturally to an > arbitrary number of contributors working on an arbitrary number of projects. > > $ cd the-repo && git checkout master # 'master' is the default branch > (master) $ git pull --ff-only > > Comments: > 1. The --ff-only option tells 'pull' to abort if I have any changes in > 'master' that are not already upstream (not fast-forward). I use it here > only for explicitness. > 2. Since I put __git_ps1 in my bash prompt, I see the current branch (and > sometimes other useful info). > > I always keep my 'master' branch clean. I use it to test the upstream > version of the code and to apply isolated bug fixes. Those look like the > following (though I usually commit from emacs using magit) > > (master) $ emacs broken-file.c makefile > (master) $ make runex42_3 DIFF=diffupdate # my script to refresh output > (master) $ git add output/ex42_3.out # new file > (master) $ git commit -am'Fix bug X introduced in <SHA1>, add runex42_3' > (master) $ git push > > If there is a conflict here, I just > > (master) $ git pull --rebase > (master) $ git push > > because this spot bug-fix doesn't have any useful history that I can > maintain. Now let's consider a more interesting feature > > (master) $ git checkout -b jed/feature1 # create a branch and check it out > (jed/feature1) $ emacs file-to-modify.c && git commit -am'implement first > part of feature1' > > and another > > (jed/feature1) $ git checkout -b jed/fetaure2 master # base this on > 'master' because it is independent of feature1 > (jed/feature2) $ emacs file-to-modify.c # oops, same file, and some > changes conflict > (jed/feature2) $ git commit -am'first part of feature2, I might not know > that it conflicts with feature1' > > Now I want to see what's happening upstream: > > $ git checkout master > (master) $ git pull > > and to find out how my patches will appear if merged into upstream: > > (master) $ git checkout -b jed/test > (jed/test) $ git merge jed/feature1 # applies cleanly, cool > (jed/test) $ git merge jed/feature2 # conflicts > Auto-merging file-to-modify.c > CONFLICT (content): Merge conflict in file-to-modify.c > *Recorded preimage for 'file-to-modify.c'* > Automatic merge failed; fix conflicts and then commit the result. > > (I'll explain the bold part later.) I will resolve this merge and commit > the result: > > (jed/test) $ git commit -a -m 'test resolution' > *Recorded resolution for 'file-to-modify.c'.* > [detached HEAD 2fe010b] test resolution > So, this resolution is applied to jed/test and also saved [I'm guessing the detached HEAD is this stashed resolution?) to be applied again later to master when jed/feature2 is finally merged with the upstream. Meanwhile you work on jed/feature2 without this resolution applied, so you don't actually see the code as it will end up. Wouldn't that be confusing? I imagine this workflow works and "preserves the ... testing" because the testing is done on jed/test, which contains the resolutions of all of the conflicts with the upstream. I feel like I would find it disorienting working on one branch, testing in another, then fixing whatever problems in the first, testing again in the second, and so on. What am I missing? Dmitry. > > Then test in the 'jed/test' branch. Maybe I build other applications that > will be users of feature1 and feature2. Now I go back to working on > feature1: > > $ git checkout jed/feature1 > (jed/feature1) $ emacs && make all alltests > (jed/feature1) $ git commit -am'finish implementing feature1, add tests, > etc' > > Now I'm ready to publish feature1, so I > > (master) $ git pull > (master) $ git merge jed/feature1 && make alltests > (master) $ git push > > Now I work on feature2 a bit more and integrate all the latest > > (jed/test) $ git merge master # get upstream stuff > (jed/test) $ git merge jed/feature2 # get new feature2 stuff > > then go back to feature2 and finish up > > (jed/feature2) $ emacs && make all alltests > (jed/feature2) $ git commit -am'finish implementing feature2, add tests, > etc' > > and now it's time to publish > > (master) $ git merge jed/feature2 > Auto-merging file-to-modify.c > CONFLICT (content): Merge conflict in file-to-modify.c > *Resolved 'file-to-modify.c' using previous resolution.* > Automatic merge failed; fix conflicts and then commit the result. > > *Explanation of the bold:* > So what happened here? We resolved that conflict in jed/test, but jed/test > is messy and may have lots of incomplete stuff in it. We never merge > jed/test anywhere and we might throw it away at any time. However, since we > resolved this conflict before (even though it was in a different branch > that we won't use now), git can REuse the REcorded REsolution. Git tells me > there was a merge conflict and doesn't auto-commit the merge (I need to run > 'git commit' to complete the process), but I don't have to manually resolve > this time. (The fact that Hg has no equivalent functionality perplexes me, > but explains why Hg projects with many contributors are always cluttered > with meaningless merges.) > > I find this mechanism super convenient, but it's disabled by default, > presumably because it can feel like magic. Enable it with > > $ git config rerere.enabled true > > or globally with > > $ git config --global rerere.enabled true > > As a user, the effect is that I only ever need to resolve conflicts once. > Git remembers that resolution and will reuse it anywhere that I merge > similar content. There is technical documentation for the feature, but I > used it for years before reading these: > > http://git-scm.com/2010/03/08/rerere.html > http://www.kernel.org/pub/software/scm/git/docs/git-rerere.html > http://gitster.livejournal.com/41795.html > > > This "topic branch" workflow lets me have long-lived feature branches that > have linear history (no merges from upstream into the feature branch), but > in which the combined state is frequently tested and the eventual merge to > 'master' is painless. > > If something shows up in 'master' that I need to complete my work on > feature3, then I make an explicit merge > > (jed/feature3) $ git merge master # explain what functionality I'm > obtaining in commit message > > This merge is semantically important and unavoidable so there is no reason > to try to avoid it. We're only trying to avoid the meaningless merges "some > other stuff happened, don't know if it affects me so I'll merge it". > > Note that it's fine to have many people working on the same topic branch. > When you push to a topic branch, you are "publishing" to _those_ people. If > that group is sizable, you'll still have a personal branch that you use to > manage and test your patches before publishing to that topic branch. (Just > apply the workflow above recursively, with 'master' replaced by > 'topic-branch-with-several-people'.) > > *Throw-away integration branches* > If many people are working concurrently, it's sometimes useful to have a > snapshot of what a merge would look like with everyone's latest (still > incomplete) work. In the kernel/git workflow, that is provided by a branch > they call 'pu' (Proposed Updates). It is blown away and rebuilt > periodically, but then we can easily check whether our new stuff conflicts > with anything anyone else is working on by running > > (jed/feature4) $ git checkout -b jed/test-pu # create temporary branch at > same state as jed/feature4 > (jed/test-pu) $ git merge origin/pu > test things out, go back to work, eventually discard jed/test-pu > $ git branch -D jed/test-pu > > If we discover some bad conflict, this experiment might prompt a design > discussion with the people working on a component that we thought was > independent of our work. > > >> > Erase the command `hg branch` from your vocabulary. >> > > I wish the Hg developers would admit they made a mistake and erase 'hg > branch' from Hg's vocabulary. Users that really want that ridiculous thing > can add it to their hgrc. >

1 0

Re: [petsc-dev] Example cleanup
by Barry Smith 09 Feb '13

09 Feb '13

On Feb 9, 2013, at 6:03 PM, Jed Brown <jedbrown(a)mcs.anl.gov> wrote: > > On Sat, Feb 9, 2013 at 5:52 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > The new values are no less valid than the old so just overwrite the old outputs. > > Do we have an explanation for why they changed? Switching to a very different way of handling DMs and functions in DMs and many other things. Yes one could do bi-section and see exactly when the change happened but look, the old code was not validated, so it is not like we are any less (or more) certain with the new code. When the error bars are infinite any answer is equally valid. Barry > The change is not small (though it's better, mostly).

1 0

Re: [petsc-dev] Example cleanup
by Jed Brown 09 Feb '13

09 Feb '13

On Sat, Feb 9, 2013 at 5:52 PM, Barry Smith <bsmith(a)mcs.anl.gov> wrote: > The new values are no less valid than the old so just overwrite the old > outputs. Do we have an explanation for why they changed? The change is not small (though it's better, mostly).

1 0