Query on git submodules

Query on git submodules

[Frawley, Sarah]

Hi there
I am a design automation engineer supporting 200+ designers who use git for hardware design.  We also use the submodule feature where we can have quite complex hierarchy's with 10+ layers.  We have experience issues with re-use of design projects was we move from one derivative to another due to the complexity of the hierarchy along with lack of discipline (using absolute paths in .gitmodule files). To enforce more discipline I am currently working on a pre-commit hook to check the integrity of .gitmodule files.  I would be interested in seeing how other users in the community find submodules for large scale projects and if there are any best known methods for using them.


Operating system (specifically which version)  - Suse Linux (SLES11SP2)
Git version (git --version) - 1.7.12.2


Thanks,
Sarah 

-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: Query on git submodules

[Heiko Voigt]

Hi,

On Fri, May 22, 2015 at 01:46:24PM +0000, Frawley, Sarah wrote:
> I am a design automation engineer supporting 200+ designers who use
> git for hardware design.  We also use the submodule feature where we
> can have quite complex hierarchy's with 10+ layers.

What does this 10+ layers mean? Nested submodule repositories 10
recursion steps deep?

> We have experience issues with re-use of design projects was we move
>from one derivative to another due to the complexity of the hierarchy
>along with lack of discipline (using absolute paths in .gitmodule
>files). To enforce more discipline I am currently working on a
>pre-commit hook to check the integrity of .gitmodule files.  I would be
>interested in seeing how other users in the community find submodules
>for large scale projects and if there are any best known methods for
>using them.

I do not have anything to share here since our projects did not have
such problems (not that large scale). It would be interesting to see
what you come up with. Maybe we can add some of that into core git to
support such large scale projects better. So maybe you can share exactly
what problems you have (only absolute paths?) or the pre-commit hooks
you will use.

Cheers Heiko

FW: Query on git submodules

[Frawley, Sarah]

Thanks Heiko for getting back to me.

Correct when I referred to 10+ layers I meant nested repositories which make up a large hierarchy.  Some repositories are repeated across the hierarchy.  We check-out submodules to tag versions (as opposed to master branch).  If we need to roll out a particular change across a hierarchy (e.g. 60 repos) then every level in the hierarchy needs to pick up new submodule tags.

The main 2 issues that we see are:

1. Enforcement of absolute paths in git submodules - I am currently trialing using a pre-commit hook to highlight this issue to users so that they can fix their submodule urls to relative paths.

2. Slowness Integrating updates to submodule hierarchy -I have been looking at ways of automating such a roll out - this can be useful for new project setups or for rolling out an update to all repos and quickly integrating it into the submodule hierarchy.  The link below shows something similar however it checks out a master branch of each submodule in its hierarchy. https://chrisjean.com/recursively-updating-git-submodules/




Thanks,
Sarah 
-----Original Message-----
From: Heiko Voigt [mailto:hvoigt@hvoigt.net] 
Sent: Tuesday, May 26, 2015 6:01 PM
To: Frawley, Sarah
Cc: git@vger.kernel.org
Subject: Re: Query on git submodules

Hi,

On Fri, May 22, 2015 at 01:46:24PM +0000, Frawley, Sarah wrote:
> I am a design automation engineer supporting 200+ designers who use 
> git for hardware design.  We also use the submodule feature where we 
> can have quite complex hierarchy's with 10+ layers.

What does this 10+ layers mean? Nested submodule repositories 10 recursion steps deep?

> We have experience issues with re-use of design projects was we move 
>from one derivative to another due to the complexity of the hierarchy 
>along with lack of discipline (using absolute paths in .gitmodule 
>files). To enforce more discipline I am currently working on a 
>pre-commit hook to check the integrity of .gitmodule files.  I would be 
>interested in seeing how other users in the community find submodules 
>for large scale projects and if there are any best known methods for 
>using them.

I do not have anything to share here since our projects did not have such problems (not that large scale). It would be interesting to see what you come up with. Maybe we can add some of that into core git to support such large scale projects better. So maybe you can share exactly what problems you have (only absolute paths?) or the pre-commit hooks you will use.

Cheers Heiko
-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

Re: FW: Query on git submodules

[Mattias Vannergård]

Frawley, Sarah <sarah.frawley <at> intel.com> writes:
> Correct when I referred to 10+ layers I meant nested repositories 
which make up a large hierarchy.  Some
> repositories are repeated across the hierarchy.  We check-out 
submodules to tag versions (as opposed
> to master branch).  If we need to roll out a particular change across 
a hierarchy (e.g. 60 repos) then
> every level in the hierarchy needs to pick up new submodule tags.
> 
> The main 2 issues that we see are:
> 
> 1. Enforcement of absolute paths in git submodules - I am currently 
trialing using a pre-commit hook to
> highlight this issue to users so that they can fix their submodule 
urls to relative paths.
> 

We enforced the config file instead (containing the host, port and 
username) because we use Gerrit. So, our urls are just
ssh:\\<server>:<repo>

> 2. Slowness Integrating updates to submodule hierarchy -I have been 
looking at ways of automating such a
> roll out - this can be useful for new project setups or for rolling 
out an update to all repos and quickly
> integrating it into the submodule hierarchy.  The link below shows 
something similar however it checks
> out a master branch of each submodule in its hierarchy. 
https://chrisjean.com/recursively-updating-git-submodules/
> 

I haven't been able to check performance at our site yet.

> Thanks,
> Sarah 

I am planning on using submodules in a way, where I can specify a 
configuration of repos with tags, and rebuild the submodule tree, when 
selecting a configuration on the top level.

But, yet I have found no easy way to select the tag in a subrepo. I 
would like the .gitmodules to accept tag instead of branch. I think 
there is very little difference between specifying a branch or a tag, 
but it is more convenient to have the release version as a tag.

So, if a product A, consist of subcomponents a1, a2 and a3, the version 
v1.1 of product A will have a configuration saying: a1, tag=v0.4; a2, 
tag=v1.0; a3, tag=v2.0

Subcomponent can be both a pre-built binary or a shared/common source 
base, or a complete 3rd party repository.

Is this comparable to your situation?

I am currently working on editing the git-submodules to add -t (as in 
tag). I need to find an easy way to switch between tracking a branch and 
locking to tag, though.

BR
/Mattias

Re: FW: Query on git submodules

[Heiko Voigt]

On Tue, Jun 02, 2015 at 09:45:39AM +0000, Mattias Vannergård wrote:
> I am planning on using submodules in a way, where I can specify a 
> configuration of repos with tags, and rebuild the submodule tree, when 
> selecting a configuration on the top level.
> 
> But, yet I have found no easy way to select the tag in a subrepo. I 
> would like the .gitmodules to accept tag instead of branch. I think 
> there is very little difference between specifying a branch or a tag, 
> but it is more convenient to have the release version as a tag.
> 
> So, if a product A, consist of subcomponents a1, a2 and a3, the version 
> v1.1 of product A will have a configuration saying: a1, tag=v0.4; a2, 
> tag=v1.0; a3, tag=v2.0
> 
> Subcomponent can be both a pre-built binary or a shared/common source 
> base, or a complete 3rd party repository.
> 
> Is this comparable to your situation?
> 
> I am currently working on editing the git-submodules to add -t (as in 
> tag). I need to find an easy way to switch between tracking a branch and 
> locking to tag, though.

What exactly is the feature of using a tag instead of a commit id like it
is currently the default? A tag should be nailed to a commit anyway or
does it change in your situation? In that case I would say it is a
branch.

Cheers Heiko

Re: FW: Query on git submodules

[Mattias Vannergård]

Heiko Voigt <hvoigt <at> hvoigt.net> writes:

> 
> What exactly is the feature of using a tag instead of a commit id like 
it
> is currently the default? A tag should be nailed to a commit anyway or
> does it change in your situation? In that case I would say it is a
> branch.
> 
> Cheers Heiko
> 


Essentially, it is a commit, but an annotated tag is easier to read, 
than the SHA-1.

And, I don't want gerrit to track a tag, but I want gerrit to track a 
branch.

BR
/Mattias

Re: FW: Query on git submodules

[Heiko Voigt]

On Wed, May 27, 2015 at 01:20:11PM +0000, Frawley, Sarah wrote:
> Thanks Heiko for getting back to me.
> 
> Correct when I referred to 10+ layers I meant nested repositories
> which make up a large hierarchy.  Some repositories are repeated
> across the hierarchy.  We check-out submodules to tag versions (as
> opposed to master branch).  If we need to roll out a particular change
> across a hierarchy (e.g. 60 repos) then every level in the hierarchy
> needs to pick up new submodule tags.
> 
> The main 2 issues that we see are:
> 
> 1. Enforcement of absolute paths in git submodules - I am currently
> trialing using a pre-commit hook to highlight this issue to users so
> that they can fix their submodule urls to relative paths.

So you mean, enforcement of relative paths. Right? It might be a good
idea to put some code in 'git submodule add' that would automatically
use a relative path instead of absolute path if it can calculate one
from the superproject. I think that would help your users even more as
they do not get an error from the pre-commit hook, but just having
everything right after doing the add.

If you want to go that route it should not be that much different from
the hook, since the submodule subcommand code is all bash.

> 2. Slowness Integrating updates to submodule hierarchy -I have been
> looking at ways of automating such a roll out - this can be useful for
> new project setups or for rolling out an update to all repos and
> quickly integrating it into the submodule hierarchy.  The link below
> shows something similar however it checks out a master branch of each
> submodule in its hierarchy.
> https://chrisjean.com/recursively-updating-git-submodules/

How about

	git submodule update --init --recursive

? That should do the same as the script written in that blog post. Or am
I missing something?

Cheers Heiko

RE: FW: Query on git submodules

[Frawley, Sarah]

Thanks for getting back to me Heiko

To comment on the below.  Adding some automated way of setting a relative path via git submodule add would be really useful.
Is this something that is possible in a new git release?

As regards using the command below.
git submodule update --init --recursive
We use this already however it just refreshes any submodule changes that have been pushed to git.  
What I am looking for is a way to re-align submodule tags to new tags across a large hierarchy.

See example below.  It show a simple submodule hierarchy.  You can see that the 'c_block' occurs underneath both the 'e' and the 'd' blocks and they are both set to the same tag (tag_1).  If I decide to create a new tag for 'c_block' called tag_2 in order for both d_block and e_block to pick it up I will need to make a update to both blocks where I check out the tag_2 for 'd/e_blocks' and commit, push and tag and make another change to 'a_block' so that it checks out the new tags for 'c_block','d_block' and 'e_block'.  This all happens manually in my team currently.  Sometimes there is a need to roll out the same change to all repos and for the hierarchy to be up rev'd to the latest tag for every submodule so a script for this case would be useful.  It would be nice to be able to have a script read in a list of tags for update across the hierarchy.

a_block   ->  c_block (tag_1)    ->   d_block(tag_4)     

                     d_block (tag_4)     ->  c_block(tag_1)    

	       e_block (tag_8)      -> d_block(tag_4)     
                                                          c_block(tag_1)    

Commandline use to update e_block to pick up new tag for c_block:
cd e_block/submodules/c_block
git checkout tag_2
cd ../..
git commit
git push
git tag



git submodule update --init --recursive


Thanks,
Sarah Frawley
Quark DA

To raise a DA help ticket please file using the url below: https://hsdes.intel.com/home/default.html#article?&subject=support&tenant=quark

-----Original Message-----
From: Heiko Voigt [mailto:hvoigt@hvoigt.net] 
Sent: Tuesday, June 2, 2015 12:12 PM
To: Frawley, Sarah
Cc: git@vger.kernel.org
Subject: Re: FW: Query on git submodules

On Wed, May 27, 2015 at 01:20:11PM +0000, Frawley, Sarah wrote:
> Thanks Heiko for getting back to me.
> 
> Correct when I referred to 10+ layers I meant nested repositories 
> which make up a large hierarchy.  Some repositories are repeated 
> across the hierarchy.  We check-out submodules to tag versions (as 
> opposed to master branch).  If we need to roll out a particular change 
> across a hierarchy (e.g. 60 repos) then every level in the hierarchy 
> needs to pick up new submodule tags.
> 
> The main 2 issues that we see are:
> 
> 1. Enforcement of absolute paths in git submodules - I am currently 
> trialing using a pre-commit hook to highlight this issue to users so 
> that they can fix their submodule urls to relative paths.

So you mean, enforcement of relative paths. Right? It might be a good idea to put some code in 'git submodule add' that would automatically use a relative path instead of absolute path if it can calculate one from the superproject. I think that would help your users even more as they do not get an error from the pre-commit hook, but just having everything right after doing the add.

If you want to go that route it should not be that much different from the hook, since the submodule subcommand code is all bash.

> 2. Slowness Integrating updates to submodule hierarchy -I have been 
> looking at ways of automating such a roll out - this can be useful for 
> new project setups or for rolling out an update to all repos and 
> quickly integrating it into the submodule hierarchy.  The link below 
> shows something similar however it checks out a master branch of each 
> submodule in its hierarchy.
> https://chrisjean.com/recursively-updating-git-submodules/

How about

	git submodule update --init --recursive

? That should do the same as the script written in that blog post. Or am I missing something?

Cheers Heiko
-------------------------------------------------------------
Intel Ireland Limited (Branch)
Collinstown Industrial Park, Leixlip, County Kildare, Ireland
Registered Number: E902934

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.