Submodules with feature branches

Submodules with feature branches

[Robert Dailey]

I have a question regarding submodules and their applicability given
our workflow at the place I work.

When I work on a feature, I normally create a feature branch. If I
happen to make changes to the submodule that only work with the
changes introduced in my feature branch, that seems to complicate
things. For the purposes of the feature branch, do I need to create a
corresponding feature branch in the submodule and temporarily update
the submodule URL to point to it? When I merge my feature branch, I'd
have to swap it back?

What is a recommended workflow for this? Thanks in advance.

Re: Submodules with feature branches

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 3208 bytes --]

On Thu, Jun 05, 2014 at 09:03:25AM -0500, Robert Dailey wrote:
> When I work on a feature, I normally create a feature branch. If I
> happen to make changes to the submodule that only work with the
> changes introduced in my feature branch, that seems to complicate
> things. For the purposes of the feature branch, do I need to create
> a corresponding feature branch in the submodule and temporarily
> update the submodule URL to point to it? When I merge my feature
> branch, I'd have to swap it back?

So you have:

  On the trunk host:   On your public host:   Locally:
  superproject         superproject           superproject
  submodule            submodule              `-- submodule

In that case, a corresponding feature branch to the submodule, and an
update to submodule.<name>.url (and possibly submodule.<name>.branch)
would be the way I'd go (at A in the figure below).  Once the trunk
maintainers were happy with things, they could merge the submodule
branch into trunk's submodule (at B in the figure below), and you
could add a capping commit to your superproject branch that reverted
the gitmodule changes (at C in the figure below):

  -o---o---o---o-------o  trunk's superproject/master
    \                 /
     A---o---o---o---C    your superproject/feature

  -o---o-----------B  trunk's submodule/master
    \             /
     o---o---o----    your submodule/feature

An alternative is to use relative URLs in the trunk:

  superproject$ cat .gitmodules
  [submodule "bpl-subset"]
    path = submod
    url = ../submodule

which makes it easier for folks who mirror/fork both the superproject
and submodule (no need to change submodule.<name>.url).  However, it
makes it harder for folks who just mirror/fork the superproject (and
don't need to tweak the submodule), because they have to mirror/fork
the submodule as well to support the relative URL (or edit
submodule.<name>.url, which turns attempted mirrors into forks).
Personally, I prefer relative URLs [1,2], but both external projects
I've approached on this front have ended up with absolute URLs [3,4]
;).

This is less of an issue for loosely-coupled submodules, since you'll
can motivate your submodule changes to the submodule maintainers
independent of the superproject (i.e. you can just say things like
“I'm extending the API so I can iterate over widgets.  This lets you
do things like frobbling whatsits in superproject” without having to
present the associated superproject code).  Once you land the
submodule changes upstream, your superproject branch will work without
the need to tweak the URL (for absolute URLs) or publish a sibling
mirror (for relative URLs).

Cheers,
Trevor

[1]: https://github.com/inducer/pycuda/pull/21
[2]: http://thread.gmane.org/gmane.comp.python.ipython.devel/10287/focus=10299 
[3]: https://github.com/wking/pycuda/commit/5218bd449d6aae0bce3a3d1bf54a91377445e2f9
[4]: https://github.com/minrk/ipython/commit/4fe230e96e357b3612b6fadaeec9d8de71d6fca9

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Submodules with feature branches

[Robert Dailey]

On Thu, Jun 5, 2014 at 10:15 AM, W. Trevor King <wking@tremily.us> wrote:
> So you have:
>
>   On the trunk host:   On your public host:   Locally:
>   superproject         superproject           superproject
>   submodule            submodule              `-- submodule
>
> In that case, a corresponding feature branch to the submodule, and an
> update to submodule.<name>.url (and possibly submodule.<name>.branch)
> would be the way I'd go (at A in the figure below).  Once the trunk
> maintainers were happy with things, they could merge the submodule
> branch into trunk's submodule (at B in the figure below), and you
> could add a capping commit to your superproject branch that reverted
> the gitmodule changes (at C in the figure below):
>
>   -o---o---o---o-------o  trunk's superproject/master
>     \                 /
>      A---o---o---o---C    your superproject/feature
>
>   -o---o-----------B  trunk's submodule/master
>     \             /
>      o---o---o----    your submodule/feature
>
> An alternative is to use relative URLs in the trunk:
>
>   superproject$ cat .gitmodules
>   [submodule "bpl-subset"]
>     path = submod
>     url = ../submodule
>
> which makes it easier for folks who mirror/fork both the superproject
> and submodule (no need to change submodule.<name>.url).  However, it
> makes it harder for folks who just mirror/fork the superproject (and
> don't need to tweak the submodule), because they have to mirror/fork
> the submodule as well to support the relative URL (or edit
> submodule.<name>.url, which turns attempted mirrors into forks).
> Personally, I prefer relative URLs [1,2], but both external projects
> I've approached on this front have ended up with absolute URLs [3,4]
> ;).
>
> This is less of an issue for loosely-coupled submodules, since you'll
> can motivate your submodule changes to the submodule maintainers
> independent of the superproject (i.e. you can just say things like
> “I'm extending the API so I can iterate over widgets.  This lets you
> do things like frobbling whatsits in superproject” without having to
> present the associated superproject code).  Once you land the
> submodule changes upstream, your superproject branch will work without
> the need to tweak the URL (for absolute URLs) or publish a sibling
> mirror (for relative URLs).

Thanks, this is excellent information. Perhaps I should provide a
little more detail into what I'm doing. I know that having such
dependencies between superproject & submodule is bad and creates
complications like this, so maybe there is a different approach.

Right now our build system does not download third party dependencies.
We build on Windows & Android, so we maintain our own package binaries
& source code. Right now these are stored in 7zip files and checked
into the superproject. I was planning on creating a submodule for our
third party libs and store them extracted in there. That way, when we
switch branches, we don't have to delete & re-extract the third party
archives again (since between branches, libraries may change, be added
or removed). The submodule buys us an important thing, which is that
we won't have big binary blobs in our history. If we ever want to
remove them, all we're removing is a weak link to the submodule. The
binaries live with us forever since they're in history.

The only other thing I can think to do is incorporate logic into our
makefiles to copy down the 7zips from a permanent server, and we'd
adopt a naming format for the archives and download them based on that
information. This way, when we go back to an earlier tag, it will
always pull the correct version of the dependencies. We'd have to make
sure to never delete old libraries from the remote server or edit
existing ones.

I was exploring submodules to see if they would solve this problem.
However, because of the feature branch workflow, they do not seem
practical. I'm open to any other suggestions. Thanks!!

Re: Submodules with feature branches

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 711 bytes --]

On Thu, Jun 05, 2014 at 10:57:17AM -0500, Robert Dailey wrote:
> I was planning on creating a submodule for our third party libs and
> store them extracted in there.

3rd party libraries sound loosely-coupled to me ;).  In one of my more
mature projects I did a similar thing, and just used relative URLs [1]
and sibling mirrors/forks [2,3,4].

Cheers,
Trevor

[1]: https://github.com/wking/pygrader/blob/master/.gitmodules
[2]: https://github.com/wking/pgp-mime
[3]: https://github.com/wking/pyassuan
[4]: https://github.com/wking/jinja2

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Submodules with feature branches

[Robert Dailey]

On Thu, Jun 5, 2014 at 11:23 AM, W. Trevor King <wking@tremily.us> wrote:
> 3rd party libraries sound loosely-coupled to me ;).  In one of my more
> mature projects I did a similar thing, and just used relative URLs [1]
> and sibling mirrors/forks [2,3,4].
>
> Cheers,
> Trevor
>
> [1]: https://github.com/wking/pygrader/blob/master/.gitmodules
> [2]: https://github.com/wking/pgp-mime
> [3]: https://github.com/wking/pyassuan
> [4]: https://github.com/wking/jinja2

I guess I'm still confused on how relative URLs help here. Won't the
capping commits (A and C in your first email) still be needed? Or is
there a way I can modify the local "../third-party.git" submodule repo
instead? Can you explain?

Unfortunately, the reason why I feel third party in a submodule
creates tight coupling is because:

* You can't make changes to third party libs for your feature branch
without breaking the trunk
* Merge conflicts are insane to resolve and involve two clones if
trunk maintainers modify third party binaries and you do as well.
* Feature branching requires those capping / meta commits to simply
setup your branch to be a feature branch.

Instead of just creating my branch and starting to make commits, I now
have to setup my submodule branch first. Also pull requests won't show
the changes to the third party libraries unless I do a second pull
request for the third party repo.

It just seems like a mess :-(

Re: Submodules with feature branches

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 3579 bytes --]

On Thu, Jun 05, 2014 at 01:31:39PM -0500, Robert Dailey wrote:
> On Thu, Jun 5, 2014 at 11:23 AM, W. Trevor King wrote:
> > 3rd party libraries sound loosely-coupled to me ;).  In one of my more
> > mature projects I did a similar thing, and just used relative URLs [1]
> > and sibling mirrors/forks [2,3,4].
> >
> > Cheers,
> > Trevor
> >
> > [1]: https://github.com/wking/pygrader/blob/master/.gitmodules
> > [2]: https://github.com/wking/pgp-mime
> > [3]: https://github.com/wking/pyassuan
> > [4]: https://github.com/wking/jinja2
> 
> I guess I'm still confused on how relative URLs help here.

If you want to add a feature to pygrader that needs tweaks to
pgp-mime, you can put your public repositories somewhere as siblings,
and:

  $ git clone --recursive git://you.net/pygrader.git

will work fine (drawing from git://you.net/pgp-mime.git, etc.).

> Won't the capping commits (A and C in your first email) still be
> needed? Or is there a way I can modify the local
> "../third-party.git" submodule repo instead? Can you explain?

Anyone reviewing your changes locally will need a way to get your
submodule commits as well as your superproject commits.  In both
cases, they can use the usual:

  $ git add remote you git://you.net/….git
  $ git fetch

or other tweaks like GitHub's refs/pull/*/head namespace [1].  Even a
shared central repository, if that's how your team rolls.

> Unfortunately, the reason why I feel third party in a submodule
> creates tight coupling is because:
> 
> * You can't make changes to third party libs for your feature branch
> without breaking the trunk

You can in a branch.  Maybe I'm missing something here.  In any case,
edits to third party libs are best upstreamed ;).

> * Merge conflicts are insane to resolve and involve two clones if
> trunk maintainers modify third party binaries and you do as well.

You resolve the merge conflicts in the submodule, and then amend the
superproject merge commit to point to the resolved submodule commit.
That is one --amend away from what you're doing without submodules.

> * Feature branching requires those capping / meta commits to simply
> setup your branch to be a feature branch.

With relative URLs (or shared centralized repository, or a
refs/pull/*/head namespace) it's easy to share the commits themselves.
Unless you're using 'git submodule update --remote …', you don't need
to care where the gitlinked commits live, you just need to get them
into the submodule repository somehow.  That seems fairly orthogonal
to feature branching to me.

> Instead of just creating my branch and starting to make commits, I
> now have to setup my submodule branch first. Also pull requests
> won't show the changes to the third party libraries unless I do a
> second pull request for the third party repo.

That I agree with ;).  However, if you're treating the third-party
library as a separate repo, I think it makes sense that you need to be
making branches and pull requests in the submodule independently from
your branches and pull requests in the superproject.  If you feel that
the minimal (branch + PR for changes) overhead of managing the
projects independently is too high, you're probably better off with
the single repository or subtree approach.

Cheers,
Trevor

[1]: https://help.github.com/articles/checking-out-pull-requests-locally

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Submodules with feature branches

[W. Trevor King]

[-- Attachment #1: Type: text/plain, Size: 1534 bytes --]

On Thu, Jun 05, 2014 at 12:00:33PM -0700, W. Trevor King wrote:
> On Thu, Jun 05, 2014 at 01:31:39PM -0500, Robert Dailey wrote:
> > Instead of just creating my branch and starting to make commits, I
> > now have to setup my submodule branch first. Also pull requests
> > won't show the changes to the third party libraries unless I do a
> > second pull request for the third party repo.
> 
> That I agree with ;).  However, if you're treating the third-party
> library as a separate repo, I think it makes sense that you need to
> be making branches and pull requests in the submodule independently
> from your branches and pull requests in the superproject.

To make this more concrete, I think you'll rarely have tight
one-to-one binding between third-party library changes and your
superproject.  More likely, you'll have some high-level feature branch
in the superproject (“accept comments via email”) and an unrelated
number of prerequisite feature branches for your libraries (“add
support for MIME documents,” “parse RFC 2822 dates,” …).  You only
have synchronized branches when you mess with the API tying components
together (updating the submodule API and updating the superproject to
use it).  With good library design, that type of API migration should
happen more and more rarely as the library stabilizes.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]