May 2014 - Acme-se - mailman3.cels.anl.gov

ACME Git setup
by Douglas Jacobsen 05 Jun '14

05 Jun '14

Hey Guys, Reposting my last message, just so it's archived. Something else to note, to start this has a count of 17 repos. 2-3 of these at least will be removed for ACME (all MPAS components). And anything that we don't need (do we need dead_share) can also be removed. So, I've been trying to setup a "release" version of CESM in git (as a starting place for ACME). And I just thought I'd share some of my experiences so far. Currently I've been trying to preserve all of the history from CESM's repo. One of the problems I've run into however, is that CESM is currently a mess of svn externals that point all over the root of the repository. It could be possible to convert all of the history to git and have it work properly, but it would either take a robust (and / or hacky) script or a lot of time. Also, several of these repositories are nested, and should probably be joined into a repo (i.e. all of the scripts and tools should be joined together). What might be a workable solution would be to pick the latest public release of CESM, and start the repos from there. Then we can test setting things up in a correct manner. After that, we should be able to easily incorporate history modifications from the svn repo (assuming we remember the revision number we want). It seems like from here: http://www.cesm.ucar.edu/models/cesm1.2/ we can safely start from 1.2. And then we can setup branches as needed. Here is a rough example of what I'm planning to setup for the multiple repo case, at least to begin with. https://gist.github.com/douglasjacobsen/89322ebcc746dfe51d40 Any comments or questions would be more than welcome. Thanks, Doug

7 61

Re: [ACME-SE] Setting up v0
by Taylor, Mark A 02 Jun '14

02 Jun '14

Hi Doug, Rob, I just sent you a link to our google drive document which describes what is in v0. on Titan, you can find this code in my directory: ~taylorm/codes/acme_v0 I think one thing missing from that document are the changes Pat added to automatically track performance data. Most code is changed directly in the source tree so would show up with svn diff. But there are a few things that are changed via CESM's SourceMod's feature. Some of those should not be part of the source code (as they are used to change parameter values), but the aice bug fix should I think be part of the v0 source code. It would be great to have v0 as something we can check out from the acme repository. This is the code for the CAM5/POP high-res simulations. Mat is also running CAM4/POP high-res simulations, but I'm not sure if that is with what we call ACMEv0 - it might be with the code base with the tuned CAM4-SE atmosphere. Mark -- Mark Taylor, mataylo(a)sandia.gov, 505-284-1874 1442 Numerical Analysis & Applications Sandia National Labs, Albuquerque NM 87185-1320 ________________________________________ From: Douglas Jacobsen [douglasj(a)lanl.gov] Sent: Thursday, May 29, 2014 3:40 PM To: jacob(a)mcs.anl.gov; acme-se(a)lists.mcs.anl.gov; Taylor, Mark A Subject: [EXTERNAL] Re: [ACME-SE] Setting up v0 Hey Rob, I agree. I don't think we want to start development from that point. But people are still in the process or running simulations (i.e. Mat). I'm mostly just meaning we setup a place for them to easily share the code that is v0 (the people who need it at least). Longer term, we can probably setup v0 as a branch of whatever development repo we end up with. But at least as a starting place they wouldn't have to ship around tar files. In addition to that, it would work to get some people who we expect to be using these models more familiar with git in the short term. Thanks! Doug On 05/29/2014 03:38 PM, Robert Jacob wrote: > > Hey Doug, > > I believe v0 is so far behind that we don't really want to start > *development* from that point. But we will still measure its > performance. The set of branch/trunk tags we'll start with is still > being identified. Mark can say more. > > Rob > > > On 5/29/14 4:33 PM, Douglas Jacobsen wrote: >> Hey guys, >> >> Since v0 is needed more immediately than v0.1 or anything else, I was >> considering burning one of our repos with a v0 setup. I'd probably >> consolidate some of our other repos, or remove some completely to allow >> this, but some people are trying to use it and it seems like currently >> they are using tar files to move this around. >> >> Does anyone have any objections to this? If not I might work with Mat >> Maltrud here to try and get a copy of the current v0 and throw it up >> into a repo. >> >> In addition to this, I'm thinking we'll need to setup a team on the >> github account (maybe Jim can do this?) that I can make the repo in, and >> we can add the v0 users to this team. >> >> Thanks! >> Doug >> _______________________________________________ >> ACME-SE mailing list >> ACME-SE(a)lists.mcs.anl.gov >> https://lists.mcs.anl.gov/mailman/listinfo/acme-se

3 2

Setting up v0
by Douglas Jacobsen 02 Jun '14

02 Jun '14

Hey guys, Since v0 is needed more immediately than v0.1 or anything else, I was considering burning one of our repos with a v0 setup. I'd probably consolidate some of our other repos, or remove some completely to allow this, but some people are trying to use it and it seems like currently they are using tar files to move this around. Does anyone have any objections to this? If not I might work with Mat Maltrud here to try and get a copy of the current v0 and throw it up into a repo. In addition to this, I'm thinking we'll need to setup a team on the github account (maybe Jim can do this?) that I can make the repo in, and we can add the v0 users to this team. Thanks! Doug

3 5

Second private repo to share with CSEG
by Robert Jacob 28 May '14

28 May '14

Hi everyone, Can we set up a second private repo? I mean something besides github.org/ACME-Climate? I'm wondering if that requires another account. The reason is that ACME-Climate will eventually be the official repo for code with only ACME people on it, etc. But we should get another private area to play around with git layouts with the CSEG group. We need to do that with them (possibly re-hashing some of what we've discussed) to help them see the value of moving to git. I run this idea by them and they're up for it! It has to be private since we'll be using unreleased code. I also asked what history they want to preserve in a move to git. Just the trunks. No need to remember already merged or dead branches. Some in-development branches may need to be transferred over as well. Rob

3 4

Fwd: bug reporting
by Robert Jacob 27 May '14

27 May '14

See below. Provided CESM clears up their bug reporting procedure (they have a Bugzilla at least) and we of course have a well defined one, do we have to watch two sites? Rob -------- Original Message -------- Subject: bug reporting Date: Sun, 25 May 2014 18:10:29 +0000 From: Ghan, Steven J <Steve.Ghan(a)pnnl.gov> To: ssc(a)cgd.ucar.edu <ssc(a)cgd.ucar.edu>, cseg(a)cgd.ucar.edu <cseg(a)cgd.ucar.edu>, wg-cochairs(a)cgd.ucar.edu <wg-cochairs(a)cgd.ucar.edu>, ccsm-liaisons(a)cgd.ucar.edu <ccsm-liaisons(a)cgd.ucar.edu> I recently found a bug in cam that could have been serious but luckily did not have a major impact on the simulated climate. I told people who I thought should know about it (J-F, co-chairs of Atmosphere Model and Climate-Chemistry wG, and key users who care about the process involved), and have since learned from one key user that his team had identified the bug in 2012 and had reported it to someone, but it had never been communicated to someone who would make sure the bug is corrected in the released version. So the bug persisted. This tells me that the most effective and preferred bug-reporting procedure is not sufficiently well-known or easy to find. I looked on the cesm web site, and could not easily find the procedure. I looked on the bulletin board and found lots of interesting forum topics, but nothing related to reporting bugs. A bub-reporting link briefly appeared, but then vanished. Unless I am just looking in the wrong place, the bug-reporting procedure needs to be made easier to find. Steve Ghan WG Co-Chair, Climate-Chemistry

2 2

single-repo progress
by Foucar, James G 22 May '14

22 May '14

Hi guys, I decided to revisit the single-repo solution for ACME after giving up on git subtree. I came up with a hybrid solution that uses filter-branch to bring in commits from the sub-repo and subtree split to go back the other way. This is as close as I've gotten to the best-of-both-worlds. The history of the sub-repo is preserved and pushes can be made to the sub-repo from the super-repo. I've done some preliminary testing and the attached scripts seem to work for simple cases. There appear to be at least two drawbacks: 1) Some of the commands involve a lot of commit rewriting and can therefore take a while 2) The independent repo for the sub-repo needs to be locked during a push operation from the super-repo (I think) One thing to note, the svn-conversions are generating a lot of empty commits that were causing problems, so I had to rewrite history in the repositories that Doug created with the following command: % git filter-branch --commit-filter 'git_commit_non_empty_tree "$@"' HEAD I've been doing most of my testing as follows: # Set up base of single-repo % git init acme_single_repo % cd !$ % touch .gitignore % git add !$ % git commit -m "Initial commit" # Add scripts and cice as sub-repos % setup_sub_repo scripts ../scripts.noempty.bare master scripts % setup_sub_repo cice ../cice.noempty.bare master models/ice/cice # And test % *make commits to scripts from super-repo* % *make commits to scripts from sub-repo, push sub-repo* % push_to_subrepo scripts master scripts # Run from super-repo Š I wanted to let people know that single-repo may be viable with some work and I also wanted to get feedback on the implementation provided by the attached scripts. Is there any hope this could work? Thanks, -Jim

3 9

Re: [ACME-SE] single-repo progress
by Salinger, Andrew 20 May '14

20 May '14

I am going on vacation without internet access Thursday 5/22 – 6/1. I added Jim as "Owner" of ACME-Climate in case that access is needed. Do others need Owner access? It is only the issue of non-Sandians obligating the Sandia Credit card that makes me pause, not in reality but in process. I noticed we are at 19 of 20 Private Repos. We will have to quickly drop the many-multiple-repo exploration, or pay for more private repos (upgrade from Silver to Gold). Andy

4 8