Publishing "filtered branch repositories" - workflow / recommendations?

Publishing "filtered branch repositories" - workflow / recommendations?

[Martin Langhoff]

Hi folks.

currently working on a project based on Moodle (the LMS that got me
into git in the first place). This is a highly modular software, and I
would like to maintain a bunch of "out of tree" modules in a single
repository, and be able to publish them in "per-module" repositories.

So I would like to maintain a tree with looking like

  auth/foomatic/{code}
  mod/foomatic/{code}

where I can develop, branch and tag all the foomatic code together.
Yet, at release time I want to _also_ publish two repos

  auth-foomatic.git
  mod-foomatic.git

each of them with matching tags and code at the "root" of the git
tree, and ideally with a truthful history (i.e.: similar to having run
git filter-branch --subdirectory-filter, but able to update that
filtered history incrementally).

Is there a reasonable approach to scripting this?

Alternatively, has git submodule been improved so that it's usable by
mere mortals (i.e.: my team), or are there strong alternatives to git
submodule?

cheers,



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Martin Langhoff]

On Wed, Dec 4, 2013 at 6:01 PM, Martin Langhoff
<martin.langhoff@gmail.com> wrote:
> Is there a reasonable approach to scripting this?

Found my answers.

The 'subtree' merge strategy is smart enough to "mostly" help here.
However, it does not handle new files created in the subdirectory.

My workflow is this one. It is similar to the recipes for the subtree
merge strategies, but invoking git mv to pull files out of the

git merge -s ours --no-commit upstream/branch
git read-tree --prefix= -u upstream/branch
git mv mysubdir/* ./ ### read-tree can't do this
git commit

... time passes

git merge -s subtree --no-commit upstream/branch
if [ -d mysubdir ]; then
    # handle new files
    git mv mysubdir/* ./
fi
git commit

glad that I ended up reading a lot about subtree.

cheers,



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Jens Lehmann]

Am 05.12.2013 00:01, schrieb Martin Langhoff:
> Hi folks.
> 
> currently working on a project based on Moodle (the LMS that got me
> into git in the first place). This is a highly modular software, and I
> would like to maintain a bunch of "out of tree" modules in a single
> repository, and be able to publish them in "per-module" repositories.
> 
> So I would like to maintain a tree with looking like
> 
>   auth/foomatic/{code}
>   mod/foomatic/{code}
> 
> where I can develop, branch and tag all the foomatic code together.
> Yet, at release time I want to _also_ publish two repos
> 
>   auth-foomatic.git
>   mod-foomatic.git
> 
> each of them with matching tags and code at the "root" of the git
> tree, and ideally with a truthful history (i.e.: similar to having run
> git filter-branch --subdirectory-filter, but able to update that
> filtered history incrementally).
> 
> Is there a reasonable approach to scripting this?
> 
> Alternatively, has git submodule been improved so that it's usable by
> mere mortals (i.e.: my team), or are there strong alternatives to git
> submodule?

Without knowing more I can't think of a reason why submodules should
not suit your use case (but you'd have to script branching and tagging
yourself until these commands learn to recurse into submodules too).

But maybe you'll be happier with git subtree, dunno.

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Martin Langhoff]

On Thu, Dec 5, 2013 at 2:18 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Without knowing more I can't think of a reason why submodules should
> not suit your use case (but you'd have to script branching and tagging
> yourself until these commands learn to recurse into submodules too).

The submodules feature is way too fiddly and has abundant gotchas.

I am diving into subtrees, and finding it a lot more workable.

cheers,



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Jens Lehmann]

Am 05.12.2013 20:27, schrieb Martin Langhoff:
> On Thu, Dec 5, 2013 at 2:18 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Without knowing more I can't think of a reason why submodules should
>> not suit your use case (but you'd have to script branching and tagging
>> yourself until these commands learn to recurse into submodules too).
> 
> The submodules feature is way too fiddly and has abundant gotchas.

Care to explain what bothers you the most? Being one of the people
improving submodules I'm really interested in hearing more about that.

> I am diving into subtrees, and finding it a lot more workable.

Yep, for exporting parts of your repo as separate repos one way
subtree might be just what you want.

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Martin Langhoff]

On Thu, Dec 5, 2013 at 2:54 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Am 05.12.2013 20:27, schrieb Martin Langhoff:
>> On Thu, Dec 5, 2013 at 2:18 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>> Without knowing more I can't think of a reason why submodules should
>>> not suit your use case (but you'd have to script branching and tagging
>>> yourself until these commands learn to recurse into submodules too).
>>
>> The submodules feature is way too fiddly and has abundant gotchas.
>
> Care to explain what bothers you the most? Being one of the people
> improving submodules I'm really interested in hearing more about that.

Very glad to hear submodules is getting TLC! I have other scenarios at
$dayjob where I may need submodules, so happy happy.

I may be unaware of recent improvements, here's my (perhaps outdated) list

 - git clone does not clone existing submodules by default. An ideal
workflow assumes that the user wants a fully usable checkout.

 - git pull does not fetch&update all submodules (assuming a trivial
"tracking repos" scenario)

 - git push does not warn if you forgot to push commits in the submodule

there's possibly a few others that I've forgotten. The main issue is
that things that are conceptually simple (clone, git pull with no
local changes) are very fiddly. Our new developers, testers and
support folks hurt themselves with it plenty.

I don't mind complex scenarios being complex to handle. If you hit a
nasty merge conflict in your submodule, and that's gnarly to resolve,
that's not a showstopper.


While writing this email, I reviewed Documentation/git-submodule.txt
in git master, and it does seem to have grown some new options. I
wonder if there is a tutorial with an example workflow anywhere
showing the current level of usability. My hope is actually for some
bits of automagic default behaviors to help things along (rather than
new options)...

Early git was very pedantic, and over time it learned some DWIMery.
You're giving me hope that similar smarts might have grown in around
submodule support ...

cheers,



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Jens Lehmann]

Am 05.12.2013 23:06, schrieb Martin Langhoff:
> On Thu, Dec 5, 2013 at 2:54 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> Am 05.12.2013 20:27, schrieb Martin Langhoff:
>>> On Thu, Dec 5, 2013 at 2:18 PM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>>> Without knowing more I can't think of a reason why submodules should
>>>> not suit your use case (but you'd have to script branching and tagging
>>>> yourself until these commands learn to recurse into submodules too).
>>>
>>> The submodules feature is way too fiddly and has abundant gotchas.
>>
>> Care to explain what bothers you the most? Being one of the people
>> improving submodules I'm really interested in hearing more about that.
> 
> Very glad to hear submodules is getting TLC! I have other scenarios at
> $dayjob where I may need submodules, so happy happy.
>
> I may be unaware of recent improvements, here's my (perhaps outdated) list

Thanks a lot for your feedback!

>  - git clone does not clone existing submodules by default. An ideal
> workflow assumes that the user wants a fully usable checkout.

You get that when using "git clone --recurse-submodules", but there
is no configuration option yet to switch that on by default (but we
are planning to add one).

>  - git pull does not fetch&update all submodules (assuming a trivial
> "tracking repos" scenario)

Current pull does fetch them (when changes to them are found in the
fetched commits of the superproject), but it does not yet update
them (there is the "recursive update" work in progress to do that).

>  - git push does not warn if you forgot to push commits in the submodule

We do have a command line option ("--recurse-submodules=check" is
what you want here), but no configuration option yet.

> there's possibly a few others that I've forgotten. The main issue is
> that things that are conceptually simple (clone, git pull with no
> local changes) are very fiddly. Our new developers, testers and
> support folks hurt themselves with it plenty.

Seems like we already solved some of those, and your feedback shows
me that we are moving in the right direction. I keep a list of open
issues we are aware of at:

  https://github.com/jlehmann/git-submod-enhancements/wiki

Feel free to point out missing topics.

> I don't mind complex scenarios being complex to handle. If you hit a
> nasty merge conflict in your submodule, and that's gnarly to resolve,
> that's not a showstopper.

Good to hear that! Solving them automatically is hard, only fast
forwards are currently resolved without user intervention.

> While writing this email, I reviewed Documentation/git-submodule.txt
> in git master, and it does seem to have grown some new options. I
> wonder if there is a tutorial with an example workflow anywhere
> showing the current level of usability. My hope is actually for some
> bits of automagic default behaviors to help things along (rather than
> new options)...

Right you are, we need tutorials for the most prominent use cases.

> Early git was very pedantic, and over time it learned some DWIMery.
> You're giving me hope that similar smarts might have grown in around
> submodule support ...

That's what we are aiming at :-)

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Martin Langhoff]

On Fri, Dec 6, 2013 at 3:48 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> Right you are, we need tutorials for the most prominent use cases.

In the meantime, are there any hints? Emails on this list showing a
current "smart" workflow? Blog posts? Notes on a wiki?

>> Early git was very pedantic, and over time it learned some DWIMery.
>> You're giving me hope that similar smarts might have grown in around
>> submodule support ...
>
> That's what we are aiming at :-)

That is fantastic! Thank you.



m
-- 
 martin.langhoff@gmail.com
 -  ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 ~ http://docs.moodle.org/en/User:Martin_Langhoff

Re: Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Heiko Voigt]

On Fri, Dec 06, 2013 at 02:40:15PM -0500, Martin Langhoff wrote:
> On Fri, Dec 6, 2013 at 3:48 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
> > Right you are, we need tutorials for the most prominent use cases.
> 
> In the meantime, are there any hints? Emails on this list showing a
> current "smart" workflow? Blog posts? Notes on a wiki?

None that I know of mainly because we have not yet reached the goal we
are aiming at. Maybe we should write something, A few points from
$dayjob that come to my mind:

  * A submodule commit is only allowed to be merged into master in a
    superproject commit if it is merged into master (or a stable branch)
    in the submodule. That way you ensure that any submodules commits
    that are tracked in a superproject are contained in each other and
    can be cleanly merged. (no rewinds, one commit contains the other)

  * Submodule should be selfcontained (i.e. if its a library have tests
    that use the code you implement). That way changes in the submodule
    can be made independent from the superproject

  * If a submodule needs another submodule have them side by side
    instead of one inside another. See the next point for explanation.

  * Only one depth of recursion for submodules. Even though intuition
    tell you that if some submodule needs another it should contain the
    other its IMO not wise to do so. There will be times when some other
    submodule needs the same submodule that is contained in the other
    and then you end up with two copies of the same code. My suggestion:
    Let the superproject bundle all the dependencies between modules.

  * Submodules are a good solution for shared code where the dependency
    goes superproject needs submodule. If you divide code into
    submodules because of access control and the dependency is actually
    that the submodule needs the superproject it works but is less than
    optimal.

Thats what I can quickly suggest and probably far from complete.

Cheers Heiko

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Junio C Hamano]

Heiko Voigt <hvoigt@hvoigt.net> writes:

> On Fri, Dec 06, 2013 at 02:40:15PM -0500, Martin Langhoff wrote:
>> On Fri, Dec 6, 2013 at 3:48 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>> > Right you are, we need tutorials for the most prominent use cases.
>> 
>> In the meantime, are there any hints? Emails on this list showing a
>> current "smart" workflow? Blog posts? Notes on a wiki?
>
> None that I know of mainly because we have not yet reached the goal we
> are aiming at. Maybe we should write something, A few points from
> $dayjob that come to my mind:
>
>   * A submodule commit is only allowed to be merged into master in a
>     superproject commit if it is merged into master (or a stable branch)
>     in the submodule. That way you ensure that any submodules commits
>     that are tracked in a superproject are contained in each other and
>     can be cleanly merged. (no rewinds, one commit contains the other)

I think this is closely related to Martin's list of wishes we
earlier saw in the thread: remind the user to push necessary
submodule tip before the top-level commit that needs that commit in
the submodule is pushed out.  Giving projects a way to implement
such a policy decision would be good, and having a default policy,
if we can find one that would be reasonable for any submodule users,
would be even better.  Would adding a generic pre-push hook at the
top-level sufficient for that kind of thing, I have to wonder.

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Jens Lehmann]

Am 10.12.2013 00:56, schrieb Junio C Hamano:
> Heiko Voigt <hvoigt@hvoigt.net> writes:
> 
>> On Fri, Dec 06, 2013 at 02:40:15PM -0500, Martin Langhoff wrote:
>>> On Fri, Dec 6, 2013 at 3:48 AM, Jens Lehmann <Jens.Lehmann@web.de> wrote:
>>>> Right you are, we need tutorials for the most prominent use cases.
>>>
>>> In the meantime, are there any hints? Emails on this list showing a
>>> current "smart" workflow? Blog posts? Notes on a wiki?
>>
>> None that I know of mainly because we have not yet reached the goal we
>> are aiming at. Maybe we should write something, A few points from
>> $dayjob that come to my mind:
>>
>>   * A submodule commit is only allowed to be merged into master in a
>>     superproject commit if it is merged into master (or a stable branch)
>>     in the submodule. That way you ensure that any submodules commits
>>     that are tracked in a superproject are contained in each other and
>>     can be cleanly merged. (no rewinds, one commit contains the other)
> 
> I think this is closely related to Martin's list of wishes we
> earlier saw in the thread: remind the user to push necessary
> submodule tip before the top-level commit that needs that commit in
> the submodule is pushed out.  Giving projects a way to implement
> such a policy decision would be good, and having a default policy,
> if we can find one that would be reasonable for any submodule users,
> would be even better.  Would adding a generic pre-push hook at the
> top-level sufficient for that kind of thing, I have to wonder.

That could call "git push --dry-run --recurse-submodules=check" to
deny the push if the submodule commit isn't on a remote branch. But
that would only work for a single hardcoded remote, as the remote
itself does not seem to be passed to the pre-push hook.

So me thinks adding a configuration option for the --recurse-submodule
option of push is the best way to achieve that. This could be set to
"check" for those who want to push the submodule manually and to
"on-demand" for those who would like to push automatically. And with
that option the user could configure push just like he already can do
for fetch and pull.

Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Junio C Hamano]

Jens Lehmann <Jens.Lehmann@web.de> writes:

>> I think this is closely related to Martin's list of wishes we
>> earlier saw in the thread: remind the user to push necessary
>> submodule tip before the top-level commit that needs that commit in
>> the submodule is pushed out.  Giving projects a way to implement
>> such a policy decision would be good, and having a default policy,
>> if we can find one that would be reasonable for any submodule users,
>> would be even better.  Would adding a generic pre-push hook at the
>> top-level sufficient for that kind of thing, I have to wonder.
>
> That could call "git push --dry-run --recurse-submodules=check" to
> deny the push if the submodule commit isn't on a remote branch.
> that would only work for a single hardcoded remote, as the remote
> itself does not seem to be passed to the pre-push hook.
>
> So me thinks adding a configuration option for the --recurse-submodule
> option of push is the best way to achieve that. This could be set to
> "check" ...

Yes, that uses only a single hard-coded decision, and making the
branch name or remote name customizable is not enough, as you are
still hardcoding "if ... isn't on" part. It is not far-fetched to
imagine a project wants to add more restrictions to what commit in
the submodule history can be bound to a tree of a published commit
in the top-level project (e.g. "must be a tagged release point",
"must be older at least by more than two weeks", "must be signed by
one of these developers' keys", etc.).

So I am not yet convinced that a simple "option" that supplies a few
parameters to a single hard-coded policy is sufficient in the long
run.

Re: Re: Publishing "filtered branch repositories" - workflow / recommendations?

[Heiko Voigt]

On Wed, Dec 11, 2013 at 03:16:24PM -0800, Junio C Hamano wrote:
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
> >> I think this is closely related to Martin's list of wishes we
> >> earlier saw in the thread: remind the user to push necessary
> >> submodule tip before the top-level commit that needs that commit in
> >> the submodule is pushed out.  Giving projects a way to implement
> >> such a policy decision would be good, and having a default policy,
> >> if we can find one that would be reasonable for any submodule users,
> >> would be even better.  Would adding a generic pre-push hook at the
> >> top-level sufficient for that kind of thing, I have to wonder.
> >
> > That could call "git push --dry-run --recurse-submodules=check" to
> > deny the push if the submodule commit isn't on a remote branch.
> > that would only work for a single hardcoded remote, as the remote
> > itself does not seem to be passed to the pre-push hook.
> >
> > So me thinks adding a configuration option for the --recurse-submodule
> > option of push is the best way to achieve that. This could be set to
> > "check" ...
> 
> Yes, that uses only a single hard-coded decision, and making the
> branch name or remote name customizable is not enough, as you are
> still hardcoding "if ... isn't on" part. It is not far-fetched to
> imagine a project wants to add more restrictions to what commit in
> the submodule history can be bound to a tree of a published commit
> in the top-level project (e.g. "must be a tagged release point",
> "must be older at least by more than two weeks", "must be signed by
> one of these developers' keys", etc.).
> 
> So I am not yet convinced that a simple "option" that supplies a few
> parameters to a single hard-coded policy is sufficient in the long
> run.

Well, for the implementation of --recurse-submodules=check on push we
tried to be as dumb as possible and just try to find out whether the
commit is on any remote. The reason is that when someone works on a
branch and pushes it for demonstration/backup/before holiday the most
common mistake is that he forgets to push the submodule when pushing the
superproject. So "some remote" seems to be the most common denominator
for push here. At least its better than no check at all.

For extended rules that answer questions like: "When am I allowed to
merge in master?" we need some kind of workflow definition from which we
can deduct such rules. A pre-push hook would allow to define rules but
how about an approach that is easier for the user and can maybe guide
him to the correct workflow.

I find workflow guidance is a general problem in git and even though we
use hooks at $dayjob they are not always sufficient. E.g. we enforce an
"all commits have to be reviewed first before they enter master" policy
which is correct in most situations. But for more lightweight projects I
would like to loosen this rule. The problem relies in the automatic
distribution of hooks or options for them by project instead of by
installation.

One idea: We could distribute a default set of enabled hooks that
implement several typical workflow options which can then be enabled by
simply choosing one option. The configuration should be distributeable
via the project e.g. a .githookconfig or .gitworkflowconfig ?

The git tools could then adapt their defaults depending on such a
workflow definition. But it also means we would first have to collect
and define some typical workflows.

What do others think? There probably have been other ideas about
workflow definitions already, no?

Cheers Heiko