RFC/RFH submodule handling in big setups

RFC/RFH submodule handling in big setups

[Finn Arne Gangstad]

We're trying to get git to work well with our vision of submodules,
and have to figure out how to make submodule fetching and pushing work
in a smooth way.

This is our situation (simplified):

          [product1]          [product2]  ...         (supermodules)
           /      \             /     \
    ... [foo]  [os-abstraction-lib] [log-merger] ...  (submodules)


A developer does a modification to the os-abstraction-lib, and a
modification to the log-merger that depends on the change in the
os-abstraction-lib. He wants this into product2, and doesn't know or
care about product1.  He doesn't know whether his modification is
acceptable or not, or whether his modification will go in before some
other modification.

He needs some way of pushing his modifications to a branch in the
supermodule (e.g. "change-131345"), without interfering with anyone
else.  The problem is where to push the sub-modules, they need to be
available for anyone who wants to get the "change-131345" branch of
the product2, but the modifications shouldn't show up anywhere else
(yet).  Here are solutions we have thought of so far:

1. push and fetch sha1s directly - this was sort of vetoed on this list.

2. each time you push a submodule, push it to a auto-generated
   tag (something like submodule-autogen-<sha1>), and use fetch -t
   when fetching the submodules

   Issue: Need to clean up these tags at some point, or there will be too
   many of them. There will be many ugly tags no matter what.

3. each time you push a submodule, do a merge ours to a
   "internal-submodule-tracking" branch, and push to that. Something
   like this in other words:

     git fetch origin internal-submodule-tracking
     git merge -s ours origin/internal-submodule-tracking
     git push origin internal-submodule-tracking
     git reset --hard HEAD^1

   Issue: feels wrong somehow?

4. Manually push all sub-modules to some new branch before pushing the
   super-module. This is what we'd rather avoid, but the developer
   should obviously have the option of doing this to some sub-modules
   if he wants to.

5. Secret option 5 - something we didn't think about

- Finn Arne

Re: RFC/RFH submodule handling in big setups

[Junio C Hamano]

Finn Arne Gangstad <finnag@pvv.org> writes:

> We're trying to get git to work well with our vision of submodules,
> and have to figure out how to make submodule fetching and pushing work
> in a smooth way.
>
> This is our situation (simplified):
>
>           [product1]          [product2]  ...         (supermodules)
>            /      \             /     \
>     ... [foo]  [os-abstraction-lib] [log-merger] ...  (submodules)
>
>
> A developer does a modification to the os-abstraction-lib, and a
> modification to the log-merger that depends on the change in the
> os-abstraction-lib. He wants this into product2, and doesn't know or
> care about product1.  He doesn't know whether his modification is
> acceptable or not, or whether his modification will go in before some
> other modification.
>
> He needs some way of pushing his modifications to a branch in the
> supermodule (e.g. "change-131345"), without interfering with anyone
> else.  The problem is where to push the sub-modules, they need to be
> available for anyone who wants to get the "change-131345" branch of
> the product2, but the modifications shouldn't show up anywhere else
> (yet).  Here are solutions we have thought of so far:

The phrase "without interfering with anyone else" feels somewhat
wrong and I sense there is something missing in the description
of the workflow.  When a developer pushes a change to somewhere
else, it must be because the change needs to be shared with
_somebody else_ (but not necessarily with _everybody_).

Without knowing exactly what the desired workflow is, I'd hazard
a guess and would recommend something else, which is  hopefully
a more useful way.

When the developer makes that 131345 change in os-lib submodule,
he is acting as a developer of that library and advancing its
state.  He wants to add an enhancement so that a particular way
to use it elsewhere (his use in log-merger program perhaps being
the first one of them) can be supported, but the change is not
accepted as the project wide one from os-lib project's point of
view.  The change needs to be shared with people in product2
group that want to further work on top of the change he makes in
log-merger project, but not necessarily with everybody in
product2 group, let alone people in product1 group.

That's exactly what a topic branch is about, isn't it?

He has a topic to enhance something in os-lib project, so
whatever the mysterious 131345 can be made into one branch in
os-lib project and given a more meaningful topic name.  He can
push that out to os-lib project, and bind its tip in his tree of
product2 superproject at os-lib directory, and push it to
produc2 repository (maybe to its 'master', or maybe also to its
one of the topic branches that houses his work-in-progress).

That way, when somebody from product2 group needs to work on top
of the change he made and pushed out thusly updates, his copy of
os-lib will get the updates to (or perhaps creation of) the
topic branch that contains 131345 change, and he can start
working from where the original developer left off.  He can even
fix bugs in the change in that 131345 topic in os-lib project
the original developer made, push that back, and the original
developer can build on top of that change.

Re: RFC/RFH submodule handling in big setups

[Jan Hudec]

On Mon, Jan 07, 2008 at 11:23:27 +0100, Finn Arne Gangstad wrote:
> 3. each time you push a submodule, do a merge ours to a
>    "internal-submodule-tracking" branch, and push to that. Something
>    like this in other words:
> 
>      git fetch origin internal-submodule-tracking
>      git merge -s ours origin/internal-submodule-tracking
>      git push origin internal-submodule-tracking
>      git reset --hard HEAD^1
> 
>    Issue: feels wrong somehow?

Only one thing feels wrong here -- the merge -s ours. For one thing, the
commit is probably descendant of what was in the internal-submodule-tracking
branch (the branch name should actually have product2 in it's name).
If it is not, you should not artificially make it so, but rewind the branch.

But as Junio already told you, you should really manage the subproject like
a project (because it is a project), so the change should get to a topic
branch where the other superprojects can look at the change and decide
whether they want it or not.

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>

Re: RFC/RFH submodule handling in big setups

[Nigel Magnay]

Do you currently use svn?
This feels like the common technique of using svn:externals, where
product1 has
foo repo/foo/trunk
os-abstraction-lib repo/os-abstraction-lib/trunk

product2 has
os-abstraction-lib repo/os-abstraction-lib/trunk
log-merger repos/log-merger/trunk

Where git (if I understand submodules correctly) can't do the above,
because the links are to SHA1s rather than tags or names, so in svn
terms it'd be something like
os-abstraction-lib repo/os-abstraction-lib/trunk@xxx
log-merger repos/log-merger/trunk@yyy

At first I thought the option (4) you suggest (Manually push all
sub-modules to some new branch before pushing the  super-module) was
going to be a pain - but actually I came to the conclusion it was
actually better. In our svn world, commits to shared libraries (can)
cause all hell to break loose - it'd actually be an advantage to have
to promote changes into the supermodules (the products in our case).




On Jan 7, 2008 7:46 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Finn Arne Gangstad <finnag@pvv.org> writes:
>
> > We're trying to get git to work well with our vision of submodules,
> > and have to figure out how to make submodule fetching and pushing work
> > in a smooth way.
> >
> > This is our situation (simplified):
> >
> >           [product1]          [product2]  ...         (supermodules)
> >            /      \             /     \
> >     ... [foo]  [os-abstraction-lib] [log-merger] ...  (submodules)
> >
> >
> > A developer does a modification to the os-abstraction-lib, and a
> > modification to the log-merger that depends on the change in the
> > os-abstraction-lib. He wants this into product2, and doesn't know or
> > care about product1.  He doesn't know whether his modification is
> > acceptable or not, or whether his modification will go in before some
> > other modification.
> >
> > He needs some way of pushing his modifications to a branch in the
> > supermodule (e.g. "change-131345"), without interfering with anyone
> > else.  The problem is where to push the sub-modules, they need to be
> > available for anyone who wants to get the "change-131345" branch of
> > the product2, but the modifications shouldn't show up anywhere else
> > (yet).  Here are solutions we have thought of so far:
>
> The phrase "without interfering with anyone else" feels somewhat
> wrong and I sense there is something missing in the description
> of the workflow.  When a developer pushes a change to somewhere
> else, it must be because the change needs to be shared with
> _somebody else_ (but not necessarily with _everybody_).
>
> Without knowing exactly what the desired workflow is, I'd hazard
> a guess and would recommend something else, which is  hopefully
> a more useful way.
>
> When the developer makes that 131345 change in os-lib submodule,
> he is acting as a developer of that library and advancing its
> state.  He wants to add an enhancement so that a particular way
> to use it elsewhere (his use in log-merger program perhaps being
> the first one of them) can be supported, but the change is not
> accepted as the project wide one from os-lib project's point of
> view.  The change needs to be shared with people in product2
> group that want to further work on top of the change he makes in
> log-merger project, but not necessarily with everybody in
> product2 group, let alone people in product1 group.
>
> That's exactly what a topic branch is about, isn't it?
>
> He has a topic to enhance something in os-lib project, so
> whatever the mysterious 131345 can be made into one branch in
> os-lib project and given a more meaningful topic name.  He can
> push that out to os-lib project, and bind its tip in his tree of
> product2 superproject at os-lib directory, and push it to
> produc2 repository (maybe to its 'master', or maybe also to its
> one of the topic branches that houses his work-in-progress).
>
> That way, when somebody from product2 group needs to work on top
> of the change he made and pushed out thusly updates, his copy of
> os-lib will get the updates to (or perhaps creation of) the
> topic branch that contains 131345 change, and he can start
> working from where the original developer left off.  He can even
> fix bugs in the change in that 131345 topic in os-lib project
> the original developer made, push that back, and the original
> developer can build on top of that change.
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: RFC/RFH submodule handling in big setups

[Finn Arne Gangstad]

On Mon, Jan 07, 2008 at 11:46:21AM -0800, Junio C Hamano wrote:
> Finn Arne Gangstad <finnag@pvv.org> writes:
> 
> > We're trying to get git to work well with our vision of submodules,
> > and have to figure out how to make submodule fetching and pushing work
> > in a smooth way.
> >
> > This is our situation (simplified):
> >
> >           [product1]          [product2]  ...         (supermodules)
> >            /      \             /     \
> >     ... [foo]  [os-abstraction-lib] [log-merger] ...  (submodules)
> >
> > [...]
> 
> The phrase "without interfering with anyone else" feels somewhat
> wrong and I sense there is something missing in the description
> of the workflow.  When a developer pushes a change to somewhere
> else, it must be because the change needs to be shared with
> _somebody else_ (but not necessarily with _everybody_).
> 
> Without knowing exactly what the desired workflow is, I'd hazard
> a guess and would recommend something else, which is  hopefully
> a more useful way.

I'll try to explain a bit better, in different ways:

Way 1: Think of product2 as a big repository

I want to use git as if product2 was a single repository containing
all its submodules as directories.

I want to be able to send email patches agains product2 that affect
any number of the submodules. I want to apply these in any order, and
to let them live as topic branches in product2.

Way 2: Changes to several submodules must be "globally atomic"

If I change submodule 1 and submodule 2, I want to be able to commit
both those changes as one atomic change in the supermodule - in a
topic branch. When merging that topic branch in the supermodule, both
changes in the submodules are merged. But NOT BEFORE. The submodules
cannot be pre-merged (hence the need to push them "somewhere")

Way 3: The submodules are not released on their own

Only the products are released, each product has different release
criteria, code freezes, whatever. Syncing of submodules between
products happens rarely (a few times a year maybe). Modifications to
submodules must fit each product's release criteria.

There is usually no one who is responsible for each submodule, they
live their life as part of the supermodule. Anyone can modify a
submodule, but each product has a maintainer who decides what
modifications to the submodules are allowed in that product.


Does this make things clearer in any way?

- Finn Arne

Re: RFC/RFH submodule handling in big setups

[Finn Arne Gangstad]

On Mon, Jan 07, 2008 at 09:07:56PM +0100, Jan Hudec wrote:
> On Mon, Jan 07, 2008 at 11:23:27 +0100, Finn Arne Gangstad wrote:
> > 3. each time you push a submodule, do a merge ours to a
> >    "internal-submodule-tracking" branch, and push to that. Something
> >    like this in other words:
> > 
> >      git fetch origin internal-submodule-tracking
> >      git merge -s ours origin/internal-submodule-tracking
> >      git push origin internal-submodule-tracking
> >      git reset --hard HEAD^1
> > 
> >    Issue: feels wrong somehow?
> 
> Only one thing feels wrong here -- the merge -s ours. For one thing, the
> commit is probably descendant of what was in the internal-submodule-tracking
> branch (the branch name should actually have product2 in it's name).
> If it is not, you should not artificially make it so, but rewind the branch.

The idea of the branch is to not have to make thousands of tags as
time passes, just to have a branch that refers to all the commits that
the supermodule refers to in the submodule.

The internal-submodule-tracking branch should look like this:

        / ... etc
       m
      / \
     m   o-commit 3  
    / \
   m   o-commit 2
  / \
 .   o-commit 1

Where all the "m" commits are just dummy merge commits, the contents
are of no consequence (hence the "merge -s ours" that should always
silently go through). commit 1, 2 and 3, ... are the interesting ones,
they are the sha1s the supermodule refers to in the submodule. They
may or may not have any relationship to eachother.  If they _do_ have
a realtionship to eachother, this will be tracked in some other branch
in the submodule.

So the purpose of the internal-submodule-tracking branch is: Have a
branch to push to (you cannot just push a sha1 to nowhere), have
something that refers to the sha1 so that it will not be pruned away,
and have something that refers to the sha1 so it will actually be
_fetched_ by git fetch.

> But as Junio already told you, you should really manage the subproject like
> a project (because it is a project), so the change should get to a topic
> branch where the other superprojects can look at the change and decide
> whether they want it or not.

See another mail I wote, I don't want to manage the subproject like a
separate project (it isn't).

- Finn Arne

Re: RFC/RFH submodule handling in big setups

[Finn Arne Gangstad]

On Mon, Jan 07, 2008 at 08:08:38PM +0000, Nigel Magnay wrote:
> Do you currently use svn?

Currently we use CVS. It hurts a bit.

> This feels like the common technique of using svn:externals, where
> product1 has
> foo repo/foo/trunk
> os-abstraction-lib repo/os-abstraction-lib/trunk
> 
> product2 has
> os-abstraction-lib repo/os-abstraction-lib/trunk
> log-merger repos/log-merger/trunk
> 
> Where git (if I understand submodules correctly) can't do the above,
> because the links are to SHA1s rather than tags or names, so in svn
> terms it'd be something like
> os-abstraction-lib repo/os-abstraction-lib/trunk@xxx
> log-merger repos/log-merger/trunk@yyy
> 
> At first I thought the option (4) you suggest (Manually push all
> sub-modules to some new branch before pushing the  super-module) was
> going to be a pain - but actually I came to the conclusion it was
> actually better. In our svn world, commits to shared libraries (can)
> cause all hell to break loose - it'd actually be an advantage to have
> to promote changes into the supermodules (the products in our case).

With the way we envision using git, I don't think this is a
problem. Each supermodule decides what version of the shared library
it wants to use, and since git does not point to tags/branches in
submodules but to sha1s, there is nothing you can do to a submodule
that will cause any problems in a supermodule unless the supermodule
decides to use a new version.

I think we'd be perfectly happy to use detached HEADs in all
submodules, but there is no way to get that to work currently with
push and fetch, so I'm looking for some alternative.

- Finn Arne

Re: RFC/RFH submodule handling in big setups

[Junio C Hamano]

Finn Arne Gangstad <finnag@pvv.org> writes:

> I'll try to explain a bit better, in different ways:
>
> Way 1: Think of product2 as a big repository
>
> I want to use git as if product2 was a single repository containing
> all its submodules as directories.
>
> I want to be able to send email patches agains product2 that affect
> any number of the submodules. I want to apply these in any order, and
> to let them live as topic branches in product2.
>
> Way 2: Changes to several submodules must be "globally atomic"
>
> If I change submodule 1 and submodule 2, I want to be able to commit
> both those changes as one atomic change in the supermodule - in a
> topic branch. When merging that topic branch in the supermodule, both
> changes in the submodules are merged. But NOT BEFORE. The submodules
> cannot be pre-merged (hence the need to push them "somewhere")
>
> Way 3: The submodules are not released on their own
>
> Only the products are released, each product has different release
> criteria, code freezes, whatever. Syncing of submodules between
> products happens rarely (a few times a year maybe). Modifications to
> submodules must fit each product's release criteria.
>
> There is usually no one who is responsible for each submodule, they
> live their life as part of the supermodule. Anyone can modify a
> submodule, but each product has a maintainer who decides what
> modifications to the submodules are allowed in that product.
>
> Does this make things clearer in any way?

Yes, I think my guess was not too far off the mark.

To achieve #1, when you create a topic for product2, you can
create a corresponding topic in its submodules.  Because os-lib
is shared between product1 and product2 groups, you may want to
have a namespace rule agreed upon between the groups involved.
It could be something as simple as "in os-lib, topics from
product2 group are named with product2/ prefix".

Then a topic to add a fancy coloring to log-merger program in
product2 can be called "fancy-color".  When you create that
topic branch, you also create product2/fancy-color in os-lib
submodule and switch both to it (currently there is no
single-step Porcelain for this, so you would do something like
"git checkout -b fancy-color; cd os-lib; git checkout -b
product2/fancy-color; cd ..".  There is a beginning of
discussion regarding recursive git-submodule behaviour, which
would be a good place to polich the idea further).  The topic
branch product2/fancy-color in os-lib project would be the one
I discussed in my previous message.  That can be pushed out
first if it needs further futzing by people other than the
original developer.

I am not sure if you are stating what you really want correctly
about #2.  git submodule is designed to be usable as a
standalone project.  A commit in a submodule is an atomic event
in that submodule.  Another commit that updates the containing
project to point at that commit in the submodule will be the
atomic event that updates both the containing project and the
contained submodule at the same time from the containing
project's point of view, in the sense that before that commit
the containing project does not see that change in the
submodule, and after that commit it does.

So a possible workflow would be:

 - The original developer creates fancy-color topic in product2
   and product2/fancy-color topic in os-lib and log-merge;

 - He hacks away, creates commits to advance the topic branches
   in the submodules involved and at the toplevel project as he
   sees fit;

 - He pushes out product2/fancy-color topic of os-lib project.
   This can be done even before pushing out the toplevel or
   log-merge project if your process requires for managers of
   product2 group to approve and update the change to os-lib
   project made by the group (that is, topic branches under
   product2/ namespace).

 - If the manager updated product2/fancy-color topic in os-lib,
   the original developer can pull to update his copy of os-lib,
   update his copy of the toplevel product2 project and commit
   the integration results.  In his repository, that commit in
   the toplevel is the single atomic event that finishes the
   fancy-color topic (eh, the initial round of it, anyway). 

 - That toplevel commit to fancy-color topic is pushed out.  The
   appearance of that commit could be the single atomic event
   that makes the fancy-color topic global.  Alternatively,
   merging that whole topic (both "fancy-color" in the toplevel
   and "product2/fancy-color" in submodules) to the mainline
   branch of the product2 group (perhaps that branch is called
   'master' in the product2 project and 'product2/master' in
   os-lib project) could be the single atomic event that makes
   the changes "official", from product2 group's point of view.

I do not think #3 matters in the discussion at all.  If os-lib
does not get released, that's your code and that's your choice.
It does not change the project and history management aspect of
anything.

Although it is unclear how you plan to avoid reinventing
potentially conflicting wheels across groups without _any_
coordination, once your product groups start to have diverging
needs, but if your product groups never talk with each other and
never share their improvements and enhancements to the shared
part of the codebase, that's also your choice.  It does not
change the history management aspect of anything.