Hi Doug,
Here maybe a better way to phrase this question is, could the script be implementation agnostic, i.e. could some of the subrepos be submodules, while others are actually included?
Yes, I think that would be pretty easy since submodules are so much easier than what I'm trying to do with the "inlined" repositories. I have no idea how many resources we can allocate towards maintainers. I think it could be a pretty easy job as long as there wasn't heavy development going on simultaneously in the single and independent repos.
Sure, so filter branch works fine for pulling in history. How do it work for pushing history back out?
I found that subtree spilt worked better than filter-branch for pushing history back out. However, even subtree split is not able to reproduce exactly what's in the independent repository. For example, say you have independent repository foo. We fetch foo/master, filter it, and merge it into our single repository. We then run subtree split on foo to pull it back out. I've found that I don't get the exact history of foo/master out that was put in. This is what forces me to rebase work that happened in the single repo which doesn't appear to be an acceptable solution. I may need to switch to some other filtering mechanism. The repo setup you've described sounds reasonable to me. -Jim On 5/27/14 1:33 PM, "Douglas Jacobsen" <[email protected]> wrote:
Hey Jim,
I envisioned a scenario where the single-repo-helper script was run daily by a code maintainer. If commits brought in from the independent repo conflicted with commits that were made in the single repo, then the maintainer would potentially have to talk to the author of the commit to get help, otherwise no communication would be needed. Hopefully, most development of a product would occur in one place or the other, so conflicts should be rare.
Who are the maintainers? And how would you handle the case where an independent repo doesn't want to be "brought in" (i.e. allow development on the component within the single repo)?
Here maybe a better way to phrase this question is, could the script be implementation agnostic, i.e. could some of the subrepos be submodules, while others are actually included?
The maintainers issue isn't one you necessarily need to answer, but it is one we need to think about. Mostly how many there are, what their roles are, who they are, what they primarily do, and how they interact with the coupler and component teams.
This answer could guide what we can and cannot do with repos in the future.
I'm actually not using subtree merge in the single-repo-helper script because it mangled the history too much. I think I outlined my problems with subtree merge in a previous email. What I'm doing is fetching a remote branch, rewriting the remote branch with filter-branch, and merging the remote branch into the current branch. You can see the details if you look at the update_subrepo_impl function in single-repo-helper. To answer your question, yes, this technique will work for what you describe. Basically, anything that filter-branch can support, which includes just grabbing commits from a subdirectory, we can support.
Sure, so filter branch works fine for pulling in history. How do it work for pushing history back out?
I've been thinking a lot more about how CLM and CAM don't have any stand alone capability, and don't necessarily plan to implement such a thing.
Something I was thinking about in order to solve this, was to setup a "single repo" that actually contains all of the "tagged" versions of CESM, up to whatever are considered publicly available versions. These can be converted into git repos pretty easily I think. This could be the public "upstream". Then we could setup orgs for both CAM and CLM that have public forks of this upstream, and can have their own access lists. These lists could be co-manged by DOE and NCAR, but the repo look the same, in that CLM and CAM both have access to every model that was in the tag they branched from.
Their development can happen in this fork in branches where they only modify the relevant component. i.e. CAM developers shouldn't mess with CLM, and vice versa. They can then contribute back to the upstream via pull requests from the CAM / CLM repos to the "CESM" repo.
Then for ACME, we can "fork" (i.e. duplicated) the "CESM" repo as a private development repo called ACME or something. And our internal developers could work in forks that have branches that only touch CAM or CLM, the same way they would under the CAM / CLM development repos.
The only thing that's not clear is how to deal with independent repos in this case. Though the only real way I can see doing this is via submodules. Which might not be a big deal in this case, since the components are independent.
This might be identical to what some people were thinking earlier, so I apologize if this is where some people wanted to be to begin with. But, let me know if anyone has any comments on this setup. It wouldn't require any additional scripts, but we might be able to clean up the submodule issues with hooks.
Thanks, Doug
-Jim
On 5/23/14 12:42 PM, "Jacobsen, Douglas W" <[email protected]> wrote:
Hey Jim,
I actually really like the setup you have. I like how the model is more than just a version of external repos, it's a complete package itself.
Things I'd like to explore at some point which you might already have workflow descriptions for is the communication issues related to bringing an external change into this and pushing a change from this back out to the external.
One other concern I have is that I'm not 100% sure this layout will work for what NCAR wants. Since their model is to have more flexibility with components. Though, I guess I don't see a big reason why we can't have more flexibility with components, just making sure they each how their own directories in models.
A final question is, is it possible to so the sub-tree merge with a subdirectory of the external repo. For example with the pio repo, we don't really want the whole repo in models/utils/pio, we want the directory that is currently models/utils/pio/pio to be models/utils/pio (if that makes any sense). It would be nice to be able to pull in just the subdirectory (preserving the relevant history) and push back out to the subdirectory. Then external projects can maintain whatever higher level interfaces they need that are not relevant for the coupled project. This is a detail and we can get around it without the subtree merges, but I just thought this would make it nicer too.
Thanks, Doug ________________________________________ From: Foucar, James G [[email protected]] Sent: Friday, May 23, 2014 11:44 AM To: Jed Brown; Jacobsen, Douglas W; [email protected] Subject: Re: [EXTERNAL] Re: [ACME-SE] ACME Git setup
Hi Jed,
That's a good point. I'm going to test what happens when there's a merge conflict and see how the tool works.
I've also noticed that the rebase on line 244 is causing duplication of commits in the super-repo.
-Jim
On 5/23/14 11:33 AM, "Jed Brown" <[email protected]> wrote:
"Foucar, James G" <[email protected]> writes:
Hi Jed,
The merge would be done by whoever did the "single-repo-helper update-subrepo" command, presumably a code maintainer. That's assuming the conflict was caused by work that happened on the independent repo with work that happened in the super-repo. The conflict could be "caused" by either side. In these cases, usually nobody is at fault, just a lack of out-of-band communication or an attempt to integrate features developed at different times. Committing the merge for an incompatible state is wrong. (Well, not wrong if you fix it before pushing.) But even if one side is "at fault", the best way to fix it could be on the other side.
If the integrator notices the problem before pushing and if they understand all the semantics handled by the script in the friendly case, they can manually repair. But now they don't have the support/consistency of the tool and it's easy to forget things. If they don't notice the problem, someone else needs to be able to diagnose what went wrong and either fix it or report and wait for the SE/integrator to fix it.