Re: [OE-core] [PATCH] bitbake: gitsm: download submodules

["Paul Barker" <pbarker@konsulko.com>]

On Thu, 5 Mar 2020 22:56:15 +0000
"Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:

> On Thu, 2020-03-05 at 09:42 +0000, Paul Barker wrote:
> > On Wed, 4 Mar 2020 12:30:10 +0000
> > "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> >   
> > > -----Original Message-----
> > > From: Paul Barker <pbarker@konsulko.com>
> > > To: "Freihofer, Adrian" <adrian.freihofer@siemens.com>
> > > Cc: openembedded-core@lists.openembedded.org <
> > > openembedded-core@lists.openembedded.org>
> > > Subject: Re: [OE-core] [PATCH] bitbake: gitsm: download submodules
> > > Date: Wed, 04 Mar 2020 09:59:44 +0000
> > > 
> > > On Wed, 4 Mar 2020 08:12:27 +0000
> > > "Freihofer, Adrian" <adrian.freihofer@siemens.com> wrote:
> > >   
> > > > The unpack function failed because the submodules were not
> > > > downloaded.
> > > > Calling download before unpack for each submodule solves this
> > > > issue.
> > > > 
> > > > Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> > > > ---
> > > >  bitbake/lib/bb/fetch2/gitsm.py | 1 +
> > > >  1 file changed, 1 insertion(+)
> > > > 
> > > > diff --git a/bitbake/lib/bb/fetch2/gitsm.py
> > > > b/bitbake/lib/bb/fetch2/gitsm.py
> > > > index c622771d21..3715e9824f 100644
> > > > --- a/bitbake/lib/bb/fetch2/gitsm.py
> > > > +++ b/bitbake/lib/bb/fetch2/gitsm.py
> > > > @@ -184,6 +184,7 @@ class GitSM(Git):
> > > >  
> > > >              try:
> > > >                  newfetch = Fetch([url], d, cache=False)
> > > > +                newfetch.download()
> > > >                  newfetch.unpack(root=os.path.dirname(os.path.joi
> > > > n(re
> > > > po
> > > > _conf, 'modules', module)))
> > > >              except Exception as e:
> > > >                  logger.error('gitsm: submodule unpack failed: %s
> > > > %s'
> > > > %
> > > > (type(e).__name__, str(e)))    
> > > 
> > > You shouldn't be trying to download submodules in the do_unpack
> > > step.
> > > If
> > > they're missing at this stage it probably indicates a fetch issue.
> > > 
> > > Basically true, but the information about which submodules need to
> > > be
> > > downloaded is not available before the top level repo has been
> > > unpacked. I guess the solution must be something like:
> > > 
> > > fetch top-level
> > > unpack top-level
> > > foreach submodule:
> > >   fetch submodule
> > >   unpack submodule
> > > 
> > > What's the exact error that you're seeing?
> > > 
> > > Don't remeber exactly. But when I debugged the flow, it became
> > > obvious,
> > > that the submodules are not downloaded.
> > > 
> > > This could be related to the issue I saw when the fetcher uses git
> > > shallow
> > > tarballs from a mirror - if that's the case I'm planning to get
> > > that
> > > fixed
> > > this weekend.
> > > 
> > > Yes, the problem occurs with git shallows. The patch solves it with
> > > and
> > > without shallows for us.
> > > 
> > > Also, your email client seems to have chewed the patch up. It's
> > > best to
> > > send
> > > patches using `git send-email` only.
> > > 
> > > Sorry, we switched to a new e-mail solution. I have to adapt my
> > > setup.  
> > 
> > The problem with your patch is that it would require network access
> > during
> > do_unpack which then breaks our ability to do offline builds.  
> Fetching in the unpack task is probably really wrong, I agree.
> However, it also works on our CI infrastructure which runs without
> Internet connectivity.
> > 
> > Do you only see this issue when fetching from git shallow mirror
> > tarballs?  
> 
> On our CI which works with gitshallow archives, fetching did not work
> at all. On machines with Internet access we also had problems, but I did not completely analyze what happened. The ovmf package from poky was causing the troubles in our case.
> 
> > If so, the mirror tarballs need to be extracted during do_fetch but
> > not placed in the usual 'git2' directory (as future fetches can't
> > cope with shallow clones in that directory).  
> 
> Do you already have an idea how this could be implemented?

Sorry for the late response, I meant to let you know when my fix for this had
gone in to bitbake but I ended up away due to illness for a couple of weeks.
Commit 3987db95 "fetch2/gitsm: Unpack shallow mirror tarballs" should resolve
your issue, please let me know if it doesn't.

-- 
Paul Barker
Konsulko Group