[Buildroot] Submodule granularity instead of LIB_GIT_SUBMODULES

[Buildroot] Submodule granularity instead of LIB_GIT_SUBMODULES

[lpdev at cordier.org]

Hi there,



I have a repository that have few submodules:

- myLib

? -> submodule1 (~10MB)

? -> submodule2 (8GB)



Actually I don't need to initialize submodule2 because myLib does not request it while compiling for my buildroot distro.



The option xxx_GIT_SUBMODULES is doing a recursive init on all submodules of the project. As I am using Buildroot 2018.02, there is no support for the latest git patches that improve the overall space overhead and avoid compressing the repo.



Each time I am updating the git hash of my package, buildroot is cloning the repo, getting the subodules (from scratch), compress tar it, and then start the compilation. I was wondering if there is a way to specify _GIT_SUBMODULES to no, but still get the submodule1, without breaking buildroot's download philosophy.



Thank you for your help,

Louis-Paul


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20181128/12b28101/attachment.html>

[Buildroot] Submodule granularity instead of LIB_GIT_SUBMODULES

[Arnout Vandecappelle]

On 28/11/2018 16:28, lpdev at cordier.org wrote:
> Hi there,
> 
> I have a repository that have few submodules:
> - myLib
> ? -> submodule1 (~10MB)
> ? -> submodule2 (8GB)
> 
> Actually I don't need to initialize submodule2 because myLib does not request it
> while compiling for my buildroot distro.
> 
> The option xxx_GIT_SUBMODULES is doing a recursive init on all submodules of the
> project. As I am using Buildroot 2018.02, there is no support for the latest git
> patches that improve the overall space overhead and avoid compressing the repo.
> 
> Each time I am updating the git hash of my package, buildroot is cloning the
> repo, getting the subodules (from scratch), compress tar it, and then start the
> compilation. I was wondering if there is a way to specify _GIT_SUBMODULES to no,
> but still get the submodule1, without breaking buildroot's download philosophy.

 No, there's no simple way to do that. And adding the infrastructure to do that
would lead us way too far.

 However, what you can do is to leave _GIT_SUBMODULES to NO, add submodule1 as
<PKG>_EXTRA_DOWNLOADS and extract it in a <PKG>_POST_EXTRACT_HOOK.  This does
mean you have to manually

 Unfortunately, I'm not sure if it is possible to use the git download method
for extra downloads. You can add git+ in front of the URL, but I don't think
there's a way to specify the ref to check out... So perhaps you'll instead have
to define a post-download hook that calls dl-wrapper directly...

 Regards,
 Arnout

[Buildroot] Submodule granularity instead of LIB_GIT_SUBMODULES

[LP C]

Thank you for your reply, I will try this!

On Dec 3 2018, at 5:40 pm, Arnout Vandecappelle <arnout@mind.be> wrote:
>
>
>
> On 28/11/2018 16:28, lpdev at cordier.org wrote:
> > Hi there,
> >
> > I have a repository that have few submodules:
> > - myLib
> > -> submodule1 (~10MB)
> > -> submodule2 (8GB)
> >
> > Actually I don't need to initialize submodule2 because myLib does not request it
> > while compiling for my buildroot distro.
> >
> > The option xxx_GIT_SUBMODULES is doing a recursive init on all submodules of the
> > project. As I am using Buildroot 2018.02, there is no support for the latest git
> > patches that improve the overall space overhead and avoid compressing the repo.
> >
> > Each time I am updating the git hash of my package, buildroot is cloning the
> > repo, getting the subodules (from scratch), compress tar it, and then start the
> > compilation. I was wondering if there is a way to specify _GIT_SUBMODULES to no,
> > but still get the submodule1, without breaking buildroot's download philosophy.
>
>
> No, there's no simple way to do that. And adding the infrastructure to do that
> would lead us way too far.
>
> However, what you can do is to leave _GIT_SUBMODULES to NO, add submodule1 as
> <PKG>_EXTRA_DOWNLOADS and extract it in a <PKG>_POST_EXTRACT_HOOK. This does
> mean you have to manually
>
> Unfortunately, I'm not sure if it is possible to use the git download method
> for extra downloads. You can add git+ in front of the URL, but I don't think
> there's a way to specify the ref to check out... So perhaps you'll instead have
> to define a post-download hook that calls dl-wrapper directly...
>
> Regards,
> Arnout
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20181204/b9ca9683/attachment.html>

[Buildroot] Submodule granularity instead of LIB_GIT_SUBMODULES

[Yann E. MORIN]

Arnout, All,

On 2018-12-03 17:40 +0100, Arnout Vandecappelle spake thusly:
> On 28/11/2018 16:28, lpdev at cordier.org wrote:
> > I have a repository that have few submodules:
> > - myLib
> > ? -> submodule1 (~10MB)
> > ? -> submodule2 (8GB)
> > 
> > Actually I don't need to initialize submodule2 because myLib does not request it
> > while compiling for my buildroot distro.
> > 
> > The option xxx_GIT_SUBMODULES is doing a recursive init on all submodules of the
> > project. As I am using Buildroot 2018.02, there is no support for the latest git
> > patches that improve the overall space overhead and avoid compressing the repo.
> > 
> > Each time I am updating the git hash of my package, buildroot is cloning the
> > repo, getting the subodules (from scratch), compress tar it, and then start the
> > compilation. I was wondering if there is a way to specify _GIT_SUBMODULES to no,
> > but still get the submodule1, without breaking buildroot's download philosophy.
> 
>  No, there's no simple way to do that. And adding the infrastructure to do that
> would lead us way too far.

Actually, it would be rather easy, as 'git submodule init' accepts a
list of paths to initialise (it has accepted that since submodules were
introduced in 2007).

See (totally untested):
    https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/git-submodules&id=6a4f7c504e04dfe8dd6d5f8540a6cbc81e3758ae

Arnout, like you, I am not sure we would want such a feature. However,
I don't have a strong opinion against it either, as it's pretty easy to
do...

If you really believe we should not have it, then I'll drop my patch.
Otherwise, if you think it's worth pursuing, I'll refine it before
properly submitting it.

Regards,
Yann E. MORIN.

>  However, what you can do is to leave _GIT_SUBMODULES to NO, add submodule1 as
> <PKG>_EXTRA_DOWNLOADS and extract it in a <PKG>_POST_EXTRACT_HOOK.  This does
> mean you have to manually
> 
>  Unfortunately, I'm not sure if it is possible to use the git download method
> for extra downloads. You can add git+ in front of the URL, but I don't think
> there's a way to specify the ref to check out... So perhaps you'll instead have
> to define a post-download hook that calls dl-wrapper directly...
> 
>  Regards,
>  Arnout
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] Submodule granularity instead of LIB_GIT_SUBMODULES

[Yann E. MORIN]

Louis-Paul, Arnout, All,

On 2018-12-06 09:32 +0100, Yann E. MORIN spake thusly:
> On 2018-12-03 17:40 +0100, Arnout Vandecappelle spake thusly:
> > On 28/11/2018 16:28, lpdev at cordier.org wrote:
> > > compilation. I was wondering if there is a way to specify _GIT_SUBMODULES to no,
> > > but still get the submodule1, without breaking buildroot's download philosophy.
> >  No, there's no simple way to do that. And adding the infrastructure to do that
> > would lead us way too far.
> Actually, it would be rather easy, as 'git submodule init' accepts a
> list of paths to initialise (it has accepted that since submodules were
> introduced in 2007).
> See (totally untested):
>     https://git.buildroot.org/~ymorin/git/buildroot/commit/?h=yem/git-submodules&id=6a4f7c504e04dfe8dd6d5f8540a6cbc81e3758ae
> Arnout, like you, I am not sure we would want such a feature. However,
> I don't have a strong opinion against it either, as it's pretty easy to
> do...

Well, I take that back. It is not easy, in fact.

My proposal only accounts for first-level submodules, and doers not
works for sub-submodules and below. Handling them would not be trivial.

So, I side with Arnout: we can't support that, as that would lead us way
too far, especially since our git wrapper is already complex...

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'