[PATCH] bitbake: gitsm: download submodules

[PATCH] bitbake: gitsm: download submodules

[Freihofer, Adrian]

The unpack function failed because the submodules were not downloaded.
Calling download before unpack for each submodule solves this issue.

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 bitbake/lib/bb/fetch2/gitsm.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/bb/fetch2/gitsm.py
b/bitbake/lib/bb/fetch2/gitsm.py
index c622771d21..3715e9824f 100644
--- a/bitbake/lib/bb/fetch2/gitsm.py
+++ b/bitbake/lib/bb/fetch2/gitsm.py
@@ -184,6 +184,7 @@ class GitSM(Git):
 
             try:
                 newfetch = Fetch([url], d, cache=False)
+                newfetch.download()
                 newfetch.unpack(root=os.path.dirname(os.path.join(repo
_conf, 'modules', module)))
             except Exception as e:
                 logger.error('gitsm: submodule unpack failed: %s %s' %
(type(e).__name__, str(e)))

✗ patchtest: failure for bitbake: gitsm: download submodules

[Patchwork]

== Series Details ==

Series: bitbake: gitsm: download submodules
Revision: 1
URL   : https://patchwork.openembedded.org/series/23079/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series cannot be parsed correctly due to malformed diff lines [test_mbox_format] 
  Suggested fix    Create the series again using git-format-patch and ensure it can be applied using git am
  Diff line        _conf, 'modules', module)))


* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at d6b1809e8c)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH] bitbake: gitsm: download submodules

[Paul Barker]

On Wed, 4 Mar 2020 08:12:27 +0000
"Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:

> The unpack function failed because the submodules were not downloaded.
> Calling download before unpack for each submodule solves this issue.
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>  bitbake/lib/bb/fetch2/gitsm.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> b/bitbake/lib/bb/fetch2/gitsm.py
> index c622771d21..3715e9824f 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -184,6 +184,7 @@ class GitSM(Git):
>  
>              try:
>                  newfetch = Fetch([url], d, cache=False)
> +                newfetch.download()
>                  newfetch.unpack(root=os.path.dirname(os.path.join(repo
> _conf, 'modules', module)))
>              except Exception as e:
>                  logger.error('gitsm: submodule unpack failed: %s %s' %
> (type(e).__name__, str(e)))

You shouldn't be trying to download submodules in the do_unpack step. If
they're missing at this stage it probably indicates a fetch issue.

What's the exact error that you're seeing?

This could be related to the issue I saw when the fetcher uses git shallow
tarballs from a mirror - if that's the case I'm planning to get that fixed
this weekend.

Also, your email client seems to have chewed the patch up. It's best to send
patches using `git send-email` only.

-- 
Paul Barker
Konsulko Group

Re: [PATCH] bitbake: gitsm: download submodules

[Freihofer, Adrian]

-----Original Message-----
From: Paul Barker <pbarker@konsulko.com>
To: "Freihofer, Adrian" <adrian.freihofer@siemens.com>
Cc: openembedded-core@lists.openembedded.org <
openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
Date: Wed, 04 Mar 2020 09:59:44 +0000

On Wed, 4 Mar 2020 08:12:27 +0000
"Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:

> The unpack function failed because the submodules were not
> downloaded.
> Calling download before unpack for each submodule solves this issue.
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>  bitbake/lib/bb/fetch2/gitsm.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> b/bitbake/lib/bb/fetch2/gitsm.py
> index c622771d21..3715e9824f 100644
> --- a/bitbake/lib/bb/fetch2/gitsm.py
> +++ b/bitbake/lib/bb/fetch2/gitsm.py
> @@ -184,6 +184,7 @@ class GitSM(Git):
>  
>              try:
>                  newfetch = Fetch([url], d, cache=False)
> +                newfetch.download()
>                  newfetch.unpack(root=os.path.dirname(os.path.join(re
> po
> _conf, 'modules', module)))
>              except Exception as e:
>                  logger.error('gitsm: submodule unpack failed: %s %s'
> %
> (type(e).__name__, str(e)))

You shouldn't be trying to download submodules in the do_unpack step.
If
they're missing at this stage it probably indicates a fetch issue.

Basically true, but the information about which submodules need to be
downloaded is not available before the top level repo has been
unpacked. I guess the solution must be something like:

fetch top-level
unpack top-level
foreach submodule:
  fetch submodule
  unpack submodule

What's the exact error that you're seeing?

Don't remeber exactly. But when I debugged the flow, it became obvious,
that the submodules are not downloaded.

This could be related to the issue I saw when the fetcher uses git
shallow
tarballs from a mirror - if that's the case I'm planning to get that
fixed
this weekend.

Yes, the problem occurs with git shallows. The patch solves it with and
without shallows for us.

Also, your email client seems to have chewed the patch up. It's best to
send
patches using `git send-email` only.

Sorry, we switched to a new e-mail solution. I have to adapt my setup.

Re: [PATCH] bitbake: gitsm: download submodules

[Paul Barker]

On Wed, 4 Mar 2020 12:30:10 +0000
"Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:

> -----Original Message-----
> From: Paul Barker <pbarker@konsulko.com>
> To: "Freihofer, Adrian" <adrian.freihofer@siemens.com>
> Cc: openembedded-core@lists.openembedded.org <
> openembedded-core@lists.openembedded.org>
> Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> Date: Wed, 04 Mar 2020 09:59:44 +0000
> 
> On Wed, 4 Mar 2020 08:12:27 +0000
> "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> 
> > The unpack function failed because the submodules were not
> > downloaded.
> > Calling download before unpack for each submodule solves this issue.
> > 
> > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> > ---
> >  bitbake/lib/bb/fetch2/gitsm.py | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > b/bitbake/lib/bb/fetch2/gitsm.py
> > index c622771d21..3715e9824f 100644
> > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > @@ -184,6 +184,7 @@ class GitSM(Git):
> >  
> >              try:
> >                  newfetch = Fetch([url], d, cache=False)
> > +                newfetch.download()
> >                  newfetch.unpack(root=os.path.dirname(os.path.join(re
> > po
> > _conf, 'modules', module)))
> >              except Exception as e:
> >                  logger.error('gitsm: submodule unpack failed: %s %s'
> > %
> > (type(e).__name__, str(e)))  
> 
> You shouldn't be trying to download submodules in the do_unpack step.
> If
> they're missing at this stage it probably indicates a fetch issue.
> 
> Basically true, but the information about which submodules need to be
> downloaded is not available before the top level repo has been
> unpacked. I guess the solution must be something like:
> 
> fetch top-level
> unpack top-level
> foreach submodule:
>   fetch submodule
>   unpack submodule
> 
> What's the exact error that you're seeing?
> 
> Don't remeber exactly. But when I debugged the flow, it became obvious,
> that the submodules are not downloaded.
> 
> This could be related to the issue I saw when the fetcher uses git
> shallow
> tarballs from a mirror - if that's the case I'm planning to get that
> fixed
> this weekend.
> 
> Yes, the problem occurs with git shallows. The patch solves it with and
> without shallows for us.
> 
> Also, your email client seems to have chewed the patch up. It's best to
> send
> patches using `git send-email` only.
> 
> Sorry, we switched to a new e-mail solution. I have to adapt my setup.

The problem with your patch is that it would require network access during
do_unpack which then breaks our ability to do offline builds.

Do you only see this issue when fetching from git shallow mirror tarballs? If
so, the mirror tarballs need to be extracted during do_fetch but not placed
in the usual 'git2' directory (as future fetches can't cope with shallow
clones in that directory).

Fixing this also has some impact on archiver.bbclass as that also needs to be
able to walk the tree of submodules correctly.

-- 
Paul Barker
Konsulko Group

Re: [PATCH] bitbake: gitsm: download submodules

[Freihofer, Adrian]

On Thu, 2020-03-05 at 09:42 +0000, Paul Barker wrote:
> On Wed, 4 Mar 2020 12:30:10 +0000
> "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> 
> > -----Original Message-----
> > From: Paul Barker <pbarker@konsulko.com>
> > To: "Freihofer, Adrian" <adrian.freihofer@siemens.com>
> > Cc: openembedded-core@lists.openembedded.org <
> > openembedded-core@lists.openembedded.org>
> > Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> > Date: Wed, 04 Mar 2020 09:59:44 +0000
> > 
> > On Wed, 4 Mar 2020 08:12:27 +0000
> > "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> > 
> > > The unpack function failed because the submodules were not
> > > downloaded.
> > > Calling download before unpack for each submodule solves this
> > > issue.
> > > 
> > > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> > > ---
> > >  bitbake/lib/bb/fetch2/gitsm.py | 1 +
> > >  1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > > b/bitbake/lib/bb/fetch2/gitsm.py
> > > index c622771d21..3715e9824f 100644
> > > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > > @@ -184,6 +184,7 @@ class GitSM(Git):
> > >  
> > >              try:
> > >                  newfetch = Fetch([url], d, cache=False)
> > > +                newfetch.download()
> > >                  newfetch.unpack(root=os.path.dirname(os.path.joi
> > > n(re
> > > po
> > > _conf, 'modules', module)))
> > >              except Exception as e:
> > >                  logger.error('gitsm: submodule unpack failed: %s
> > > %s'
> > > %
> > > (type(e).__name__, str(e)))  
> > 
> > You shouldn't be trying to download submodules in the do_unpack
> > step.
> > If
> > they're missing at this stage it probably indicates a fetch issue.
> > 
> > Basically true, but the information about which submodules need to
> > be
> > downloaded is not available before the top level repo has been
> > unpacked. I guess the solution must be something like:
> > 
> > fetch top-level
> > unpack top-level
> > foreach submodule:
> >   fetch submodule
> >   unpack submodule
> > 
> > What's the exact error that you're seeing?
> > 
> > Don't remeber exactly. But when I debugged the flow, it became
> > obvious,
> > that the submodules are not downloaded.
> > 
> > This could be related to the issue I saw when the fetcher uses git
> > shallow
> > tarballs from a mirror - if that's the case I'm planning to get
> > that
> > fixed
> > this weekend.
> > 
> > Yes, the problem occurs with git shallows. The patch solves it with
> > and
> > without shallows for us.
> > 
> > Also, your email client seems to have chewed the patch up. It's
> > best to
> > send
> > patches using `git send-email` only.
> > 
> > Sorry, we switched to a new e-mail solution. I have to adapt my
> > setup.
> 
> The problem with your patch is that it would require network access
> during
> do_unpack which then breaks our ability to do offline builds.
Fetching in the unpack task is probably really wrong, I agree.
However, it also works on our CI infrastructure which runs without
Internet connectivity.
> 
> Do you only see this issue when fetching from git shallow mirror
> tarballs?

On our CI which works with gitshallow archives, fetching did not work
at all. On machines with Internet access we also had problems, but I did not completely analyze what happened. The ovmf package from poky was causing the troubles in our case.

> If so, the mirror tarballs need to be extracted during do_fetch but
> not placed in the usual 'git2' directory (as future fetches can't
> cope with shallow clones in that directory).

Do you already have an idea how this could be implemented?

> 
> Fixing this also has some impact on archiver.bbclass as that also
> needs to be able to walk the tree of submodules correctly.
Yes.
> 

Re: [OE-core] [PATCH] bitbake: gitsm: download submodules

[Paul Barker]

On Thu, 5 Mar 2020 22:56:15 +0000
"Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:

> On Thu, 2020-03-05 at 09:42 +0000, Paul Barker wrote:
> > On Wed, 4 Mar 2020 12:30:10 +0000
> > "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> >   
> > > -----Original Message-----
> > > From: Paul Barker <pbarker@konsulko.com>
> > > To: "Freihofer, Adrian" <adrian.freihofer@siemens.com>
> > > Cc: openembedded-core@lists.openembedded.org <
> > > openembedded-core@lists.openembedded.org>
> > > Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> > > Date: Wed, 04 Mar 2020 09:59:44 +0000
> > > 
> > > On Wed, 4 Mar 2020 08:12:27 +0000
> > > "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> > >   
> > > > The unpack function failed because the submodules were not
> > > > downloaded.
> > > > Calling download before unpack for each submodule solves this
> > > > issue.
> > > > 
> > > > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> > > > ---
> > > >  bitbake/lib/bb/fetch2/gitsm.py | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > > 
> > > > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > > > b/bitbake/lib/bb/fetch2/gitsm.py
> > > > index c622771d21..3715e9824f 100644
> > > > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > > > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > > > @@ -184,6 +184,7 @@ class GitSM(Git):
> > > >  
> > > >              try:
> > > >                  newfetch = Fetch([url], d, cache=False)
> > > > +                newfetch.download()
> > > >                  newfetch.unpack(root=os.path.dirname(os.path.joi
> > > > n(re
> > > > po
> > > > _conf, 'modules', module)))
> > > >              except Exception as e:
> > > >                  logger.error('gitsm: submodule unpack failed: %s
> > > > %s'
> > > > %
> > > > (type(e).__name__, str(e)))    
> > > 
> > > You shouldn't be trying to download submodules in the do_unpack
> > > step.
> > > If
> > > they're missing at this stage it probably indicates a fetch issue.
> > > 
> > > Basically true, but the information about which submodules need to
> > > be
> > > downloaded is not available before the top level repo has been
> > > unpacked. I guess the solution must be something like:
> > > 
> > > fetch top-level
> > > unpack top-level
> > > foreach submodule:
> > >   fetch submodule
> > >   unpack submodule
> > > 
> > > What's the exact error that you're seeing?
> > > 
> > > Don't remeber exactly. But when I debugged the flow, it became
> > > obvious,
> > > that the submodules are not downloaded.
> > > 
> > > This could be related to the issue I saw when the fetcher uses git
> > > shallow
> > > tarballs from a mirror - if that's the case I'm planning to get
> > > that
> > > fixed
> > > this weekend.
> > > 
> > > Yes, the problem occurs with git shallows. The patch solves it with
> > > and
> > > without shallows for us.
> > > 
> > > Also, your email client seems to have chewed the patch up. It's
> > > best to
> > > send
> > > patches using `git send-email` only.
> > > 
> > > Sorry, we switched to a new e-mail solution. I have to adapt my
> > > setup.  
> > 
> > The problem with your patch is that it would require network access
> > during
> > do_unpack which then breaks our ability to do offline builds.  
> Fetching in the unpack task is probably really wrong, I agree.
> However, it also works on our CI infrastructure which runs without
> Internet connectivity.
> > 
> > Do you only see this issue when fetching from git shallow mirror
> > tarballs?  
> 
> On our CI which works with gitshallow archives, fetching did not work
> at all. On machines with Internet access we also had problems, but I did not completely analyze what happened. The ovmf package from poky was causing the troubles in our case.
> 
> > If so, the mirror tarballs need to be extracted during do_fetch but
> > not placed in the usual 'git2' directory (as future fetches can't
> > cope with shallow clones in that directory).  
> 
> Do you already have an idea how this could be implemented?

Sorry for the late response, I meant to let you know when my fix for this had
gone in to bitbake but I ended up away due to illness for a couple of weeks.
Commit 3987db95 "fetch2/gitsm: Unpack shallow mirror tarballs" should resolve
your issue, please let me know if it doesn't.

-- 
Paul Barker
Konsulko Group

Re: [OE-core] [PATCH] bitbake: gitsm: download submodules

[Adrian Freihofer]

On Mon, 2020-03-23 at 17:30 +0000, Paul Barker wrote:
> On Thu, 5 Mar 2020 22:56:15 +0000
> "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> 
> > On Thu, 2020-03-05 at 09:42 +0000, Paul Barker wrote:
> > > On Wed, 4 Mar 2020 12:30:10 +0000
> > > "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> > >   
> > > > -----Original Message-----
> > > > From: Paul Barker <pbarker@konsulko.com>
> > > > To: "Freihofer, Adrian" <adrian.freihofer@siemens.com>
> > > > Cc: openembedded-core@lists.openembedded.org <
> > > > openembedded-core@lists.openembedded.org>
> > > > Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> > > > Date: Wed, 04 Mar 2020 09:59:44 +0000
> > > > 
> > > > On Wed, 4 Mar 2020 08:12:27 +0000
> > > > "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> > > >   
> > > > > The unpack function failed because the submodules were not
> > > > > downloaded.
> > > > > Calling download before unpack for each submodule solves this
> > > > > issue.
> > > > > 
> > > > > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> > > > > ---
> > > > >  bitbake/lib/bb/fetch2/gitsm.py | 1 +
> > > > >  1 file changed, 1 insertion(+)
> > > > > 
> > > > > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > > > > b/bitbake/lib/bb/fetch2/gitsm.py
> > > > > index c622771d21..3715e9824f 100644
> > > > > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > > > > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > > > > @@ -184,6 +184,7 @@ class GitSM(Git):
> > > > >  
> > > > >              try:
> > > > >                  newfetch = Fetch([url], d, cache=False)
> > > > > +                newfetch.download()
> > > > >                  newfetch.unpack(root=os.path.dirname(os.path.joi
> > > > > n(re
> > > > > po
> > > > > _conf, 'modules', module)))
> > > > >              except Exception as e:
> > > > >                  logger.error('gitsm: submodule unpack failed: %s
> > > > > %s'
> > > > > %
> > > > > (type(e).__name__, str(e)))    
> > > > 
> > > > You shouldn't be trying to download submodules in the do_unpack
> > > > step.
> > > > If
> > > > they're missing at this stage it probably indicates a fetch issue.
> > > > 
> > > > Basically true, but the information about which submodules need to
> > > > be
> > > > downloaded is not available before the top level repo has been
> > > > unpacked. I guess the solution must be something like:
> > > > 
> > > > fetch top-level
> > > > unpack top-level
> > > > foreach submodule:
> > > >   fetch submodule
> > > >   unpack submodule
> > > > 
> > > > What's the exact error that you're seeing?
> > > > 
> > > > Don't remeber exactly. But when I debugged the flow, it became
> > > > obvious,
> > > > that the submodules are not downloaded.
> > > > 
> > > > This could be related to the issue I saw when the fetcher uses git
> > > > shallow
> > > > tarballs from a mirror - if that's the case I'm planning to get
> > > > that
> > > > fixed
> > > > this weekend.
> > > > 
> > > > Yes, the problem occurs with git shallows. The patch solves it with
> > > > and
> > > > without shallows for us.
> > > > 
> > > > Also, your email client seems to have chewed the patch up. It's
> > > > best to
> > > > send
> > > > patches using `git send-email` only.
> > > > 
> > > > Sorry, we switched to a new e-mail solution. I have to adapt my
> > > > setup.  
> > > 
> > > The problem with your patch is that it would require network access
> > > during
> > > do_unpack which then breaks our ability to do offline builds.  
> > Fetching in the unpack task is probably really wrong, I agree.
> > However, it also works on our CI infrastructure which runs without
> > Internet connectivity.
> > > Do you only see this issue when fetching from git shallow mirror
> > > tarballs?  
> > 
> > On our CI which works with gitshallow archives, fetching did not work
> > at all. On machines with Internet access we also had problems, but I did not completely analyze what happened. The ovmf package from poky was causing the troubles in our case.
> > 
> > > If so, the mirror tarballs need to be extracted during do_fetch but
> > > not placed in the usual 'git2' directory (as future fetches can't
> > > cope with shallow clones in that directory).  
> > 
> > Do you already have an idea how this could be implemented?
> 
> Sorry for the late response, I meant to let you know when my fix for this had
> gone in to bitbake but I ended up away due to illness for a couple of weeks.
> Commit 3987db95 "fetch2/gitsm: Unpack shallow mirror tarballs" should resolve
> your issue, please let me know if it doesn't.

Thank you Paul. We picked your commit during the last days. It looks
much better than mine and it works perfect.

Regards,
Adrian
> 
>