Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim
Hi James, Is there a "v0" tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we've implemented through soucecode modes, but that should be committed directly to the v0 tag). I personally think the CESM tags is enough history for the ACME repository - we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible? Mark From: Foucar, James G Sent: Monday, June 09, 2014 2:59 PM To: [email protected] Cc: Taylor, Mark A Subject: last remaining repo items Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim
Hi Mark, Two things. First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there). So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet. I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that. Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME. There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure. In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention. All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate. This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk. Hope that helps! I'll update you guys when I get a v0 up on github. Thanks, Doug On 06/09/2014 03:41 PM, Taylor, Mark A wrote:
Hi James,
Is there a "v0" tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we've implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository -- we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Hi guys, It looks like we still have several things to do, so I'm going to go ahead and make a task list. 1. Get v0 code and put it on github (Jim) 2. Diff v0 with ACME-Climate/CESM_versions repo and update our repo accordingly (Doug?) (We decided to go with CESM_versions over single-repo-example, right?) 3. Figure out how to host large binary files (Doug?) Missing anything? -Jim From: Douglas Jacobsen <[email protected]<mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items Hi Mark, Two things. First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there). So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet. I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that. Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME. There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure. In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention. All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate. This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk. Hope that helps! I'll update you guys when I get a v0 up on github. Thanks, Doug On 06/09/2014 03:41 PM, Taylor, Mark A wrote: Hi James, Is there a “v0” tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we’ve implemented through soucecode modes, but that should be committed directly to the v0 tag). I personally think the CESM tags is enough history for the ACME repository – we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible? Mark From: Foucar, James G Sent: Monday, June 09, 2014 2:59 PM To: [email protected]<mailto:[email protected]> Cc: Taylor, Mark A Subject: last remaining repo items Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim _______________________________________________ ACME-SE mailing list [email protected]<mailto:[email protected]>https://lists.mcs.anl.gov/mailman/listinfo/acme-se
I think that looks good. I don't know how much help I can be in figuring out how to host large binary data files. I'm trying to figure out if we could host any subset of them here, but it would be good to know what sort of hosting capabilities there are at other labs. There are two setups for hosting the files that we can choose from. The first is to have a single repo that contains all of the binary data files. The second is to have each component maintain their own repo with only the binary files for that component. The first might be a bit easier to maintain, but the second would be more flexible for components that want standalone capability. I'm not sure we have enough resources to support the second option though, so starting with the first is probably our best bet. --- Off topic --- So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) 3) Atlassian (JIRA + Confluence (Could also use hipchat for group chat rooms)) 4) Google? (Mailing lists? IM?) This isn't counting the 3 for OLCF, ALCF, and NERSC though. Doug On 06/10/2014 09:39 AM, Foucar, James G wrote:
Hi guys,
It looks like we still have several things to do, so I'm going to go ahead and make a task list.
1. Get v0 code and put it on github (Jim) 2. Diff v0 with ACME-Climate/CESM_versions repo and update our repo accordingly (Doug?) (We decided to go with CESM_versions over single-repo-example, right?) 3. Figure out how to host large binary files (Doug?)
Missing anything?
-Jim
From: Douglas Jacobsen <[email protected] <mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items
Hi Mark,
Two things.
First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there).
So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet.
I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that.
Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME.
There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure.
In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention.
All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate.
This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk.
Hope that helps! I'll update you guys when I get a v0 up on github.
Thanks, Doug
On 06/09/2014 03:41 PM, Taylor, Mark A wrote:
Hi James,
Is there a "v0" tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we've implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository -- we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected]https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Which binary files are we talking about? The huge input data set or the binary files that were in the source code? Putting the input data under version control never seemed like a good idea. They don't change often and when they do they always get a new filename too so why bother? I don't know what to do about the data files that were in the (HOMME?) source code. Are they needed to build or were they there for the convenience of running test cases? If the latter, ACME should just make them part of the standard input data set. Rob On 6/10/14 10:50 AM, Douglas Jacobsen wrote:
I think that looks good.
I don't know how much help I can be in figuring out how to host large binary data files. I'm trying to figure out if we could host any subset of them here, but it would be good to know what sort of hosting capabilities there are at other labs.
There are two setups for hosting the files that we can choose from. The first is to have a single repo that contains all of the binary data files. The second is to have each component maintain their own repo with only the binary files for that component.
The first might be a bit easier to maintain, but the second would be more flexible for components that want standalone capability. I'm not sure we have enough resources to support the second option though, so starting with the first is probably our best bet.
--- Off topic ---
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) 3) Atlassian (JIRA + Confluence (Could also use hipchat for group chat rooms)) 4) Google? (Mailing lists? IM?)
This isn't counting the 3 for OLCF, ALCF, and NERSC though.
Doug
On 06/10/2014 09:39 AM, Foucar, James G wrote:
Hi guys,
It looks like we still have several things to do, so I'm going to go ahead and make a task list.
1. Get v0 code and put it on github (Jim) 2. Diff v0 with ACME-Climate/CESM_versions repo and update our repo accordingly (Doug?) (We decided to go with CESM_versions over single-repo-example, right?) 3. Figure out how to host large binary files (Doug?)
Missing anything?
-Jim
From: Douglas Jacobsen <[email protected] <mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items
Hi Mark,
Two things.
First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there).
So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet.
I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that.
Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME.
There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure.
In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention.
All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate.
This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk.
Hope that helps! I'll update you guys when I get a v0 up on github.
Thanks, Doug
On 06/09/2014 03:41 PM, Taylor, Mark A wrote:
Hi James,
Is there a �v0� tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we�ve implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository � we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected]https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
We're talking about input files for each version of ACME/CESM. CESM currently has an svn repo with all of the binary files needed to run the model, but it's a gigantic repo that we don't plan on migrating or maintaining. Instead, we're trying to figure out a way to "version" the binary files and easily use them within the repo. To do this, the current plan is to use git-annex which is more like a "plugin" for git that doesn't actually version the binary files, but allows a commit to be linked to a specific version of the binary file. Right now, I just deleted the data files that were part of model source code (CICE, HOMME, etc). Doug On 06/10/2014 10:09 AM, Robert Jacob wrote:
Which binary files are we talking about? The huge input data set or the binary files that were in the source code?
Putting the input data under version control never seemed like a good idea. They don't change often and when they do they always get a new filename too so why bother?
I don't know what to do about the data files that were in the (HOMME?) source code. Are they needed to build or were they there for the convenience of running test cases? If the latter, ACME should just make them part of the standard input data set.
Rob
On 6/10/14 10:50 AM, Douglas Jacobsen wrote:
I think that looks good.
I don't know how much help I can be in figuring out how to host large binary data files. I'm trying to figure out if we could host any subset of them here, but it would be good to know what sort of hosting capabilities there are at other labs.
There are two setups for hosting the files that we can choose from. The first is to have a single repo that contains all of the binary data files. The second is to have each component maintain their own repo with only the binary files for that component.
The first might be a bit easier to maintain, but the second would be more flexible for components that want standalone capability. I'm not sure we have enough resources to support the second option though, so starting with the first is probably our best bet.
--- Off topic ---
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) 3) Atlassian (JIRA + Confluence (Could also use hipchat for group chat rooms)) 4) Google? (Mailing lists? IM?)
This isn't counting the 3 for OLCF, ALCF, and NERSC though.
Doug
On 06/10/2014 09:39 AM, Foucar, James G wrote:
Hi guys,
It looks like we still have several things to do, so I'm going to go ahead and make a task list.
1. Get v0 code and put it on github (Jim) 2. Diff v0 with ACME-Climate/CESM_versions repo and update our repo accordingly (Doug?) (We decided to go with CESM_versions over single-repo-example, right?) 3. Figure out how to host large binary files (Doug?)
Missing anything?
-Jim
From: Douglas Jacobsen <[email protected] <mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items
Hi Mark,
Two things.
First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there).
So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet.
I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that.
Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME.
There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure.
In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention.
All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate.
This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk.
Hope that helps! I'll update you guys when I get a v0 up on github.
Thanks, Doug
On 06/09/2014 03:41 PM, Taylor, Mark A wrote:
Hi James,
Is there a �v0� tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we�ve implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository � we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected]https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Doug, Since you have the task of merging-in v0, I'll take over the large binary file issue: 1. Get v0 code and put it on github (Jim) 2. Diff v0 with ACME-Climate/CESM_versions repo and update our repo accordingly (Doug) (We decided to go with CESM_versions over single-repo-example, right?) 3. Figure out how to host large binary files (Jim) -Jim From: Douglas Jacobsen <[email protected]<mailto:[email protected]>> Date: Tuesday, June 10, 2014 9:50 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [ACME-SE] [EXTERNAL] Re: last remaining repo items I think that looks good. I don't know how much help I can be in figuring out how to host large binary data files. I'm trying to figure out if we could host any subset of them here, but it would be good to know what sort of hosting capabilities there are at other labs. There are two setups for hosting the files that we can choose from. The first is to have a single repo that contains all of the binary data files. The second is to have each component maintain their own repo with only the binary files for that component. The first might be a bit easier to maintain, but the second would be more flexible for components that want standalone capability. I'm not sure we have enough resources to support the second option though, so starting with the first is probably our best bet. --- Off topic --- So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) 3) Atlassian (JIRA + Confluence (Could also use hipchat for group chat rooms)) 4) Google? (Mailing lists? IM?) This isn't counting the 3 for OLCF, ALCF, and NERSC though. Doug On 06/10/2014 09:39 AM, Foucar, James G wrote: Hi guys, It looks like we still have several things to do, so I'm going to go ahead and make a task list. 1. Get v0 code and put it on github (Jim) 2. Diff v0 with ACME-Climate/CESM_versions repo and update our repo accordingly (Doug?) (We decided to go with CESM_versions over single-repo-example, right?) 3. Figure out how to host large binary files (Doug?) Missing anything? -Jim From: Douglas Jacobsen <[email protected]<mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items Hi Mark, Two things. First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there). So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet. I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that. Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME. There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure. In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention. All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate. This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk. Hope that helps! I'll update you guys when I get a v0 up on github. Thanks, Doug On 06/09/2014 03:41 PM, Taylor, Mark A wrote: Hi James, Is there a “v0” tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we’ve implemented through soucecode modes, but that should be committed directly to the v0 tag). I personally think the CESM tags is enough history for the ACME repository – we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible? Mark From: Foucar, James G Sent: Monday, June 09, 2014 2:59 PM To: [email protected]<mailto:[email protected]> Cc: Taylor, Mark A Subject: last remaining repo items Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim _______________________________________________ ACME-SE mailing list [email protected]<mailto:[email protected]>https://lists.mcs.anl.gov/mailman/listinfo/acme-se _______________________________________________ ACME-SE mailing list [email protected]<mailto:[email protected]>https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts)
Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
I went over this a long time ago in the google doc (https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wuHs2S97...) but each of the possible options for storing binary files has pros and cons. The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files. The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them. Doug On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
Hi Doug, Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team. I'm going to set up a test repo and see how git-fat works for me. -Jim On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc (https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wuHs2 S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Hey Jim, There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind. The first was that you couldn't have multiple remotes if you wanted to. The second was that you had to sync all files, and not choose one by one. The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over. I would say, neither of them are that difficult to use. Git-fat is easier to use, and that was another thing I liked about it, but git-annex is not that complicated in addition. Doug On 06/11/2014 10:05 AM, Foucar, James G wrote:
Hi Doug,
Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team.
I'm going to set up a test repo and see how git-fat works for me.
-Jim
On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc (https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wuHs2 S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Hi Doug,
Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes.
Wow, that¹s much bigger than I expected. So if the user did a sync with git-fat, they would fill their drive? Is the size of the data repo due to inefficient versioning of binary files or are the files really that huge? -Jim On 6/11/14 11:46 AM, "Douglas Jacobsen" <[email protected]> wrote:
Hey Jim,
There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind.
The first was that you couldn't have multiple remotes if you wanted to. The second was that you had to sync all files, and not choose one by one.
The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over.
I would say, neither of them are that difficult to use. Git-fat is easier to use, and that was another thing I liked about it, but git-annex is not that complicated in addition.
Doug
On 06/11/2014 10:05 AM, Foucar, James G wrote:
Hi Doug,
Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team.
I'm going to set up a test repo and see how git-fat works for me.
-Jim
On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc
(https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wuH s2 S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
On 6/11/14 1:00 PM, Foucar, James G wrote:
Hi Doug,
Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes.
Wow, that�s much bigger than I expected. So if the user did a sync with git-fat, they would fill their drive? Is the size of the data repo due to inefficient versioning of binary files or are the files really that huge? They are really that huge. Some of them are "master" high resolution data sets used to derive boundary and initial conditions for lower resolution runs.
Rob
-Jim
On 6/11/14 11:46 AM, "Douglas Jacobsen" <[email protected]> wrote:
Hey Jim,
There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind.
The first was that you couldn't have multiple remotes if you wanted to. The second was that you had to sync all files, and not choose one by one.
The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over.
I would say, neither of them are that difficult to use. Git-fat is easier to use, and that was another thing I liked about it, but git-annex is not that complicated in addition.
Doug
On 06/11/2014 10:05 AM, Foucar, James G wrote:
Hi Doug,
Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team.
I'm going to set up a test repo and see how git-fat works for me.
-Jim
On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc
(https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wuH s2 S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Because its so large, typically this stash of input files gets copied to one common location on each site and everyone who's running CESM reads from that common location (even if they are different INCITE projects). And they download everything because different cases need different subsets of the files and its easier to just grab it all. Those stashes are already present on the LCF's and NERSC (linked to the NCAR svn repo). If we set up our own master, "sync all" will probably still be the standard mode for those platforms. When you build a CESM case, it will check the data directory and automatically svn co any files missing for that case. I assume that's how laptop developers get just the files they need. Rob On 6/11/14 12:46 PM, Douglas Jacobsen wrote:
Hey Jim,
There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind.
The first was that you couldn't have multiple remotes if you wanted to. The second was that you had to sync all files, and not choose one by one.
The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over.
I would say, neither of them are that difficult to use. Git-fat is easier to use, and that was another thing I liked about it, but git-annex is not that complicated in addition.
Doug
On 06/11/2014 10:05 AM, Foucar, James G wrote:
Hi Doug,
Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team.
I'm going to set up a test repo and see how git-fat works for me.
-Jim
On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc (https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wuHs2
S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
When you build a CESM case, it will check the data directory and automatically svn co any files missing for that case. I assume that's how laptop developers get just the files they need.
So, if we want to support laptop development, then we have to use git annex? -Jim On 6/11/14 12:02 PM, "Robert Jacob" <[email protected]> wrote:
Because its so large, typically this stash of input files gets copied to one common location on each site and everyone who's running CESM reads from that common location (even if they are different INCITE projects). And they download everything because different cases need different subsets of the files and its easier to just grab it all.
Those stashes are already present on the LCF's and NERSC (linked to the NCAR svn repo).
If we set up our own master, "sync all" will probably still be the standard mode for those platforms.
When you build a CESM case, it will check the data directory and automatically svn co any files missing for that case. I assume that's how laptop developers get just the files they need.
Rob
On 6/11/14 12:46 PM, Douglas Jacobsen wrote:
Hey Jim,
There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind.
The first was that you couldn't have multiple remotes if you wanted to. The second was that you had to sync all files, and not choose one by one.
The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over.
I would say, neither of them are that difficult to use. Git-fat is easier to use, and that was another thing I liked about it, but git-annex is not that complicated in addition.
Doug
On 06/11/2014 10:05 AM, Foucar, James G wrote:
Hi Doug,
Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team.
I'm going to set up a test repo and see how git-fat works for me.
-Jim
On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc
(https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wu Hs2
S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
"Foucar, James G" <[email protected]> writes:
When you build a CESM case, it will check the data directory and automatically svn co any files missing for that case. I assume that's how laptop developers get just the files they need.
So, if we want to support laptop development, then we have to use git annex?
The normal usage mode for git-fat is out because it places the full file in the repository, rather than a symlink. For laptops, do we want something as automatic as git-annex? Attempting to download terabytes to a laptop isn't likely to work, so we certainly don't want it attempting that without an explicit user request.
So, if we want to support laptop development, then we have to use git annex?
The normal usage mode for git-fat is out because it places the full file in the repository, rather than a symlink.
For laptops, do we want something as automatic as git-annex? Attempting to download terabytes to a laptop isn't likely to work, so we certainly don't want it attempting that without an explicit user request.
I wouldn't really call git-annex automatic. It requires people to request a particular file to transfer it. As per my last email, I wouldn't expect a user trying to download terabytes. Mostly that we need to have access to terabyes of data through whatever tool we choose to use. (git-annex or git-fat). Doug
Douglas Jacobsen <[email protected]> writes:
I wouldn't really call git-annex automatic. It requires people to request a particular file to transfer it. As per my last email, I wouldn't expect a user trying to download terabytes. Mostly that we need to have access to terabyes of data through whatever tool we choose to use. (git-annex or git-fat).
When a user wants to run the test suite (or some part), will they manually check file sizes and download piecemeal or will they run a script that downloads as needed and runs the suite? It seems like it will be very easy to accidentally get a lot more than you bargained for. git-fat is not appropriate because it duplicates files into the working tree (for safety and because that is how clean/smudge filters work). Some file systems deduplicate, but the bytes still move around.
I'd probably assume there was a script that would pull in just the files needed for the test cases requested. Since git-annex can pull in individual files this should be fairly trivial if the remotes are setup properly. Doug ________________________________________ From: Jed Brown [[email protected]] Sent: Saturday, June 14, 2014 6:38 AM To: Jacobsen, Douglas W; Foucar, James G; [email protected]; [email protected] Subject: Re: [ACME-SE] [EXTERNAL] Re: last remaining repo items Douglas Jacobsen <[email protected]> writes:
I wouldn't really call git-annex automatic. It requires people to request a particular file to transfer it. As per my last email, I wouldn't expect a user trying to download terabytes. Mostly that we need to have access to terabyes of data through whatever tool we choose to use. (git-annex or git-fat).
When a user wants to run the test suite (or some part), will they manually check file sizes and download piecemeal or will they run a script that downloads as needed and runs the suite? It seems like it will be very easy to accidentally get a lot more than you bargained for. git-fat is not appropriate because it duplicates files into the working tree (for safety and because that is how clean/smudge filters work). Some file systems deduplicate, but the bytes still move around.
"Jacobsen, Douglas W" <[email protected]> writes:
I'd probably assume there was a script that would pull in just the files needed for the test cases requested.
So how does the user predict the storage/bandwidth requirement implied by a given test case and how do we share those files with other users (whenever possible)? As part of the testing task, can we be quantitative about effectiveness of reduced test cases that would run fast on small machines using only small data files? I would like to understand under which conditions the large test cases actually need to be run. It would certainly improve productivity to replace them with faster-running tests whenever possible.
I don't think I have answers to your questions as they are not things I've been thinking about. I agree it would be good to have faster running tests where ever possible. Perhaps if you've been thinking about these, you might have suggestions to offer up? Doug ________________________________________ From: Jed Brown [[email protected]] Sent: Saturday, June 14, 2014 9:47 AM To: Jacobsen, Douglas W; Foucar, James G; [email protected]; [email protected] Subject: RE: [ACME-SE] [EXTERNAL] Re: last remaining repo items "Jacobsen, Douglas W" <[email protected]> writes:
I'd probably assume there was a script that would pull in just the files needed for the test cases requested.
So how does the user predict the storage/bandwidth requirement implied by a given test case and how do we share those files with other users (whenever possible)? As part of the testing task, can we be quantitative about effectiveness of reduced test cases that would run fast on small machines using only small data files? I would like to understand under which conditions the large test cases actually need to be run. It would certainly improve productivity to replace them with faster-running tests whenever possible.
"Jacobsen, Douglas W" <[email protected]> writes:
I don't think I have answers to your questions as they are not things I've been thinking about. I agree it would be good to have faster running tests where ever possible.
Perhaps if you've been thinking about these, you might have suggestions to offer up?
It would be interesting to log a sufficient test case for every regression discovered in the next year. Then we could retroactively set up some simplified tests and check for accuracy (false negatives and positives). There are many possible variations.
I should probably also mention, the binary data files are not only test cases. Some are test cases, some are full on simulation input files. For example, the input data set for a 0.1 degree pop CORE2 forcing run, which is not a test case, but people want to be able to run. It also includes all the grids for all compsets, and their associated mapping files. On top of all of that, it includes some testing files. Doug ________________________________________ From: Jed Brown [[email protected]] Sent: Saturday, June 14, 2014 9:47 AM To: Jacobsen, Douglas W; Foucar, James G; [email protected]; [email protected] Subject: RE: [ACME-SE] [EXTERNAL] Re: last remaining repo items "Jacobsen, Douglas W" <[email protected]> writes:
I'd probably assume there was a script that would pull in just the files needed for the test cases requested.
So how does the user predict the storage/bandwidth requirement implied by a given test case and how do we share those files with other users (whenever possible)? As part of the testing task, can we be quantitative about effectiveness of reduced test cases that would run fast on small machines using only small data files? I would like to understand under which conditions the large test cases actually need to be run. It would certainly improve productivity to replace them with faster-running tests whenever possible.
To what extent is everything to do with data files the domain of Workflow and not SE? I am unsure where the line is drawn. Andy On 6/11/14 12:02 PM, "Robert Jacob" <[email protected]> wrote:
Because its so large, typically this stash of input files gets copied to one common location on each site and everyone who's running CESM reads from that common location (even if they are different INCITE projects). And they download everything because different cases need different subsets of the files and its easier to just grab it all.
Those stashes are already present on the LCF's and NERSC (linked to the NCAR svn repo).
If we set up our own master, "sync all" will probably still be the standard mode for those platforms.
When you build a CESM case, it will check the data directory and automatically svn co any files missing for that case. I assume that's how laptop developers get just the files they need.
Rob
On 6/11/14 12:46 PM, Douglas Jacobsen wrote:
Hey Jim,
There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind.
The first was that you couldn't have multiple remotes if you wanted to. The second was that you had to sync all files, and not choose one by one.
The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over.
I would say, neither of them are that difficult to use. Git-fat is easier to use, and that was another thing I liked about it, but git-annex is not that complicated in addition.
Doug
On 06/11/2014 10:05 AM, Foucar, James G wrote:
Hi Doug,
Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team.
I'm going to set up a test repo and see how git-fat works for me.
-Jim
On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc
(https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wu Hs2
S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
I would say ensuring every developer has access to the files they need is a SE requirement. Ensuring people have the correct tools to use the files they needed and have access to is a workflow requirement. The main reason I'm saying we need to think about how and where to host these is because the LCF's currently point to the NCAR svn repo. If we come up with "custom" input files for ACME, we'll want those to be placed somewhere that's not the NCAR repo as well, and provide an easy way for a developer to get them. Doug On 06/11/2014 02:21 PM, Salinger, Andrew wrote:
To what extent is everything to do with data files the domain of Workflow and not SE? I am unsure where the line is drawn.
Andy
On 6/11/14 12:02 PM, "Robert Jacob" <[email protected]> wrote:
Because its so large, typically this stash of input files gets copied to one common location on each site and everyone who's running CESM reads from that common location (even if they are different INCITE projects). And they download everything because different cases need different subsets of the files and its easier to just grab it all.
Those stashes are already present on the LCF's and NERSC (linked to the NCAR svn repo).
If we set up our own master, "sync all" will probably still be the standard mode for those platforms.
When you build a CESM case, it will check the data directory and automatically svn co any files missing for that case. I assume that's how laptop developers get just the files they need.
Rob
On 6/11/14 12:46 PM, Douglas Jacobsen wrote:
Hey Jim,
There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind.
The first was that you couldn't have multiple remotes if you wanted to. The second was that you had to sync all files, and not choose one by one.
The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over.
I would say, neither of them are that difficult to use. Git-fat is easier to use, and that was another thing I liked about it, but git-annex is not that complicated in addition.
Doug
On 06/11/2014 10:05 AM, Foucar, James G wrote:
Hi Doug,
Just looking at your google doc, it looks like the balance of pros/cons favors git-fat. An additional pro for git-fat is that the author is on our team.
I'm going to set up a test repo and see how git-fat works for me.
-Jim
On 6/11/14 8:09 AM, "Douglas Jacobsen" <[email protected]> wrote:
I went over this a long time ago in the google doc
(https://docs.google.com/document/d/1EP9g8XQ0R25yyC9I-5fwOn-x4_Q60uO8wu Hs2
S97RRQ/edit?usp=sharing) but each of the possible options for storing binary files has pros and cons.
The main pros for git-fat are that it stores the configuration file in the repo. The main cons for git-fat are that you can only have a single remote, and that you can't "sync" individual files, you either get all files or none of the files.
The main pros for git-annex are that you can have as many remotes as you want, and it's more flexible in that you can sync individual files to independent remotes. The main cons are it's a bit more complicated it use, and you can't store config files in the repo. So we'd either need a list of command people would use to configure git-annex, or we'd need a script to set it up for them.
Doug
On 06/11/2014 02:43 AM, Jed Brown wrote:
Douglas Jacobsen <[email protected]> writes:
So far, it seems like we're likely to require a minimum of 2-4 user accounts for each ACME person. 1) Github 2) "Hosted" input files (unless we don't mind them being world readable, in which case only people who will add new files or update files need access and accounts) Note that if those binary files are named by their sha1, for example, and directory listings are blocked, they can be world readable (because nobody will guess the file name). This is in fact one use case for git-fat: rsync pushes to a public http(s) directory and anyone that knows the sha1 can fetch over unauthenticated http(s).
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
[Back from France, so I have internet and a more practical time zone.] Douglas Jacobsen <[email protected]> writes:
Hey Jim,
There are definitely things I like about git-fat. I really liked how the configuration was a permanent fixture in the repo. But there were two things that were big down sides in my mind.
The first was that you couldn't have multiple remotes if you wanted to.
Some other people have requested this and it's not difficult to add.
The second was that you had to sync all files, and not choose one by one.
Selective retrieval has been supported since April 2013, though I see this is only documented in the test script.
The second of these was the main reason I chose to favor git-annex over git-fat. Mostly because the input data repo for CESM is currently something on the order of tera to peta bytes. It seemed to me like we would require the capability of selecting the input files we wanted to transfer over.
I'm skeptical of whether either of these are a good solution for multi-TB data files. Both should be safe for large files, but simply touching that much disk once (e.g., to compute a SHA1 hash) is extremely expensive and you want to guarantee that it is not done more than once ever.
Some other people have requested this and it's not difficult to add.
I'm not sure how necessary this feature is for ACME in it's present state, but it might be useful in the future.
The second was that you had to sync all files, and not choose one by one.
It might be nice to explore how well it works. As I think this is something that is necessary.
I'm skeptical of whether either of these are a good solution for multi-TB data files. Both should be safe for large files, but simply touching that much disk once (e.g., to compute a SHA1 hash) is extremely expensive and you want to guarantee that it is not done more than once ever.
I might be mis-reading what you're saying, but I don't think there are files that are actually multiple-TBs in size. I'd say the largest files *might* be order 1TB, but most of them are going to be multi-GB in size. I was referring to the repo as a whole. It's a gigantic repo that will fill up anyones disk space if they try to get the entire thing. Doug
Hi all, I've added Mark's v0 to our github account: https://github.com/ACME-Climate/acme_v0 Doug has volunteered to merge the necessary changes into CESM_versions. Doug, I've attached a script that might be useful for you; it's for comparing repositories. -Jim From: Douglas Jacobsen <[email protected]<mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items Hi Mark, Two things. First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there). So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet. I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that. Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME. There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure. In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention. All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate. This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk. Hope that helps! I'll update you guys when I get a v0 up on github. Thanks, Doug On 06/09/2014 03:41 PM, Taylor, Mark A wrote: Hi James, Is there a “v0” tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we’ve implemented through soucecode modes, but that should be committed directly to the v0 tag). I personally think the CESM tags is enough history for the ACME repository – we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible? Mark From: Foucar, James G Sent: Monday, June 09, 2014 2:59 PM To: [email protected]<mailto:[email protected]> Cc: Taylor, Mark A Subject: last remaining repo items Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim _______________________________________________ ACME-SE mailing list [email protected]<mailto:[email protected]>https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Do the diff's in the root directory need to be applied still? Thanks, Doug On 06/10/2014 01:21 PM, Foucar, James G wrote:
Hi all,
I've added Mark's v0 to our github account: https://github.com/ACME-Climate/acme_v0
Doug has volunteered to merge the necessary changes into CESM_versions.
Doug, I've attached a script that might be useful for you; it's for comparing repositories.
-Jim
From: Douglas Jacobsen <[email protected] <mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items
Hi Mark,
Two things.
First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there).
So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet.
I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that.
Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME.
There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure.
In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention.
All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate.
This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk.
Hope that helps! I'll update you guys when I get a v0 up on github.
Thanks, Doug
On 06/09/2014 03:41 PM, Taylor, Mark A wrote:
Hi James,
Is there a �v0� tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we�ve implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository � we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected]https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Hopefully Mark knows the answer to that. -Jim From: Douglas Jacobsen <[email protected]<mailto:[email protected]>> Date: Tuesday, June 10, 2014 1:25 PM To: "Foucar, James G" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: [EXTERNAL] Re: [ACME-SE] last remaining repo items Do the diff's in the root directory need to be applied still? Thanks, Doug On 06/10/2014 01:21 PM, Foucar, James G wrote: Hi all, I've added Mark's v0 to our github account: https://github.com/ACME-Climate/acme_v0 Doug has volunteered to merge the necessary changes into CESM_versions. Doug, I've attached a script that might be useful for you; it's for comparing repositories. -Jim From: Douglas Jacobsen <[email protected]<mailto:[email protected]>> Date: Monday, June 9, 2014 3:55 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items Hi Mark, Two things. First, I'm waiting for the v0 code still (Mat is going to get it from titan for me, but if someone who already has access wants to push it to a temporary repo on github, I could "merge" it into our history from there). So, the short answer to your first question, is no. We don't have a "v0" we can checkout yet. I've been thinking the CESM tags are enough as well, for our purposes (and most other peoples purposes). Also, it would likely improve our software practices, rather than having people going back to a CAM or CLM tag / commit and working off of that. Incorporating the remaining history is not an easy task. At least, incorporating it in a way that makes it useful to keep. It's easy to export the trunk history of CAM and CLM individually, but not in a way that ties them directly to CESM / ACME. There are two main problems here. The first is that CAM and CLM have "mini-CESMs" that represent their stand-alone drivers. The second is that CESM has a completely different "full-CESM" that makes use of only a portion of the CAM and CLM directory structure. In order to migrate the history, first you need to export the svn history to git (pretty easy using git-svn). The second step is stripping out the mini-CESM (also pretty easy using filter branch, or subtree split). The third step may or may not be needed, but large data files need to be removed (fairly easy using filter branch, but can cause some issues). And the fourth and final step is the most complicated, you somehow need to "tie" each commit in the CAM / CLM trunk to a CESM tag, and use that as it's base. This last step is not easy to do at all, and requires a lot of manual intervention. All steps except the final step can be performed to preserve the history of each trunk, but the trunk will not be usable. Alternatively, the second and fourth steps can be skipped, but the histories need to remain separate. This is one of the things that I've been giving a lot of thought to, since NCAR specifically said they wanted to maintain the histories of each trunk. Hope that helps! I'll update you guys when I get a v0 up on github. Thanks, Doug On 06/09/2014 03:41 PM, Taylor, Mark A wrote: Hi James, Is there a “v0” tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we’ve implemented through soucecode modes, but that should be committed directly to the v0 tag). I personally think the CESM tags is enough history for the ACME repository – we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible? Mark From: Foucar, James G Sent: Monday, June 09, 2014 2:59 PM To: [email protected]<mailto:[email protected]> Cc: Taylor, Mark A Subject: last remaining repo items Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim _______________________________________________ ACME-SE mailing list [email protected]<mailto:[email protected]>https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Hi Mark,
But if it is easy, why not have as much history as possible? I should mention that ACME-Climate/single-repo-example does not have any major release information/tags. What it does have are commit histories for the source files. So, I guess the question is which one do we want more?
I don't have a titan account. Would it be possible to archive that directory and put it somewhere more accessible? -Jim From: <Taylor>, Mark A <[email protected]<mailto:[email protected]>> Date: Monday, June 9, 2014 3:41 PM To: "Foucar, James G" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: RE: last remaining repo items Hi James, Is there a “v0” tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we’ve implemented through soucecode modes, but that should be committed directly to the v0 tag). I personally think the CESM tags is enough history for the ACME repository – we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible? Mark From: Foucar, James G Sent: Monday, June 09, 2014 2:59 PM To: [email protected]<mailto:[email protected]> Cc: Taylor, Mark A Subject: last remaining repo items Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim
One last repo thing we haven't dealt with yet is how to store large binary data files. Based on the exploration I did earlier, I would suggest using git-annex, but regardless of what we use we need some place to host them (we can't do that on github, or any hosting site really). Doug On 06/09/2014 03:56 PM, Foucar, James G wrote:
Hi Mark,
But if it is easy, why not have as much history as possible? I should mention that ACME-Climate/single-repo-example does not have any major release information/tags. What it does have are commit histories for the source files. So, I guess the question is which one do we want more?
I don't have a titan account. Would it be possible to archive that directory and put it somewhere more accessible?
-Jim
From: <Taylor>, Mark A <[email protected] <mailto:[email protected]>> Date: Monday, June 9, 2014 3:41 PM To: "Foucar, James G" <[email protected] <mailto:[email protected]>>, "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: RE: last remaining repo items
Hi James,
Is there a "v0" tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we've implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository -- we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] <mailto:[email protected]> *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected] https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Douglas Jacobsen <[email protected]> writes:
Based on the exploration I did earlier, I would suggest using git-annex, but regardless of what we use we need some place to host them (we can't do that on github, or any hosting site really).
It's a good product, but installation and usage complexity are concerns.
Oh yeah, thanks for reminding us, Doug. We could host them ourselves, but I'd hate to have to maintain access control in two spots. Is there any way we could use github as an intermediary for access control to a self-hosted repository? -Jim From: Douglas Jacobsen <[email protected]<mailto:[email protected]>> Date: Tuesday, June 10, 2014 7:46 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items One last repo thing we haven't dealt with yet is how to store large binary data files. Based on the exploration I did earlier, I would suggest using git-annex, but regardless of what we use we need some place to host them (we can't do that on github, or any hosting site really). Doug On 06/09/2014 03:56 PM, Foucar, James G wrote: Hi Mark,
But if it is easy, why not have as much history as possible? I should mention that ACME-Climate/single-repo-example does not have any major release information/tags. What it does have are commit histories for the source files. So, I guess the question is which one do we want more?
I don't have a titan account. Would it be possible to archive that directory and put it somewhere more accessible? -Jim From: <Taylor>, Mark A <[email protected]<mailto:[email protected]>> Date: Monday, June 9, 2014 3:41 PM To: "Foucar, James G" <[email protected]<mailto:[email protected]>>, "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: RE: last remaining repo items Hi James, Is there a “v0” tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we’ve implemented through soucecode modes, but that should be committed directly to the v0 tag). I personally think the CESM tags is enough history for the ACME repository – we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible? Mark From: Foucar, James G Sent: Monday, June 09, 2014 2:59 PM To: [email protected]<mailto:[email protected]> Cc: Taylor, Mark A Subject: last remaining repo items Hello repo team, I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so: 1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots). Thanks, -Jim _______________________________________________ ACME-SE mailing list [email protected]<mailto:[email protected]>https://lists.mcs.anl.gov/mailman/listinfo/acme-se
I don't believe so, but I'll look into it. Currently I think the only option for large data files is hosting them ourselves. Doug On 06/10/2014 09:33 AM, Foucar, James G wrote:
Oh yeah, thanks for reminding us, Doug.
We could host them ourselves, but I'd hate to have to maintain access control in two spots. Is there any way we could use github as an intermediary for access control to a self-hosted repository?
-Jim
From: Douglas Jacobsen <[email protected] <mailto:[email protected]>> Date: Tuesday, June 10, 2014 7:46 AM To: "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items
One last repo thing we haven't dealt with yet is how to store large binary data files.
Based on the exploration I did earlier, I would suggest using git-annex, but regardless of what we use we need some place to host them (we can't do that on github, or any hosting site really).
Doug
On 06/09/2014 03:56 PM, Foucar, James G wrote:
Hi Mark,
But if it is easy, why not have as much history as possible? I should mention that ACME-Climate/single-repo-example does not have any major release information/tags. What it does have are commit histories for the source files. So, I guess the question is which one do we want more?
I don't have a titan account. Would it be possible to archive that directory and put it somewhere more accessible?
-Jim
From: <Taylor>, Mark A <[email protected] <mailto:[email protected]>> Date: Monday, June 9, 2014 3:41 PM To: "Foucar, James G" <[email protected] <mailto:[email protected]>>, "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: RE: last remaining repo items
Hi James,
Is there a �v0� tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we�ve implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository � we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] <mailto:[email protected]> *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected]https://lists.mcs.anl.gov/mailman/listinfo/acme-se
It looks like the answer is no. But you can get public ssh keys from a user's github account. So it would be really easy to script something that would synchronize ssh keys between our github organization and what ever self hosted git we had. Assuming we did authentication through ssh keys at least. Doug On 06/10/2014 09:36 AM, Douglas Jacobsen wrote:
I don't believe so, but I'll look into it.
Currently I think the only option for large data files is hosting them ourselves.
Doug
On 06/10/2014 09:33 AM, Foucar, James G wrote:
Oh yeah, thanks for reminding us, Doug.
We could host them ourselves, but I'd hate to have to maintain access control in two spots. Is there any way we could use github as an intermediary for access control to a self-hosted repository?
-Jim
From: Douglas Jacobsen <[email protected] <mailto:[email protected]>> Date: Tuesday, June 10, 2014 7:46 AM To: "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: [EXTERNAL] Re: [ACME-SE] last remaining repo items
One last repo thing we haven't dealt with yet is how to store large binary data files.
Based on the exploration I did earlier, I would suggest using git-annex, but regardless of what we use we need some place to host them (we can't do that on github, or any hosting site really).
Doug
On 06/09/2014 03:56 PM, Foucar, James G wrote:
Hi Mark,
But if it is easy, why not have as much history as possible? I should mention that ACME-Climate/single-repo-example does not have any major release information/tags. What it does have are commit histories for the source files. So, I guess the question is which one do we want more?
I don't have a titan account. Would it be possible to archive that directory and put it somewhere more accessible?
-Jim
From: <Taylor>, Mark A <[email protected] <mailto:[email protected]>> Date: Monday, June 9, 2014 3:41 PM To: "Foucar, James G" <[email protected] <mailto:[email protected]>>, "[email protected] <mailto:[email protected]>" <[email protected] <mailto:[email protected]>> Subject: RE: last remaining repo items
Hi James,
Is there a �v0� tag in github that we can checkout? Can someone with a titan account automate a file by file diff with ~taylorm/codes/acme_v0 ? (and there is one more change in CICE, that we�ve implemented through soucecode modes, but that should be committed directly to the v0 tag).
I personally think the CESM tags is enough history for the ACME repository � we can always get the detailed history from the CESM repository. But if it is easy, why not have as much history as possible?
Mark
*From:*Foucar, James G *Sent:* Monday, June 09, 2014 2:59 PM *To:* [email protected] <mailto:[email protected]> *Cc:* Taylor, Mark A *Subject:* last remaining repo items
Hello repo team,
I'd like to try to set up a candidate repository by this Friday. There are a couple items that are preventing me from doing so:
1. A confirmation from someone (Doug, Mark?) that the codes we've put on github match the versions that we want for v0. 2. A decision on what we want our history to look like. The choices are what you see in ACME-Climate/single-repo-example (full history) vs. ACME-Climate/CESM_Version (just version snapshots).
Thanks,
-Jim
_______________________________________________ ACME-SE mailing list [email protected]https://lists.mcs.anl.gov/mailman/listinfo/acme-se
Douglas Jacobsen <[email protected]> writes:
It looks like the answer is no. But you can get public ssh keys from a user's github account. So it would be really easy to script something that would synchronize ssh keys between our github organization and what ever self hosted git we had.
Assuming we did authentication through ssh keys at least.
That's the usual mode of operation for gitolite, and it could be set up with basic ssh (e.g., for rsync).
participants (7)
-
Douglas Jacobsen -
Foucar, James G -
Jacobsen, Douglas W
-
Jed Brown -
Robert Jacob -
Salinger, Andrew -
Taylor, Mark A