Sumo to Thud migration: Fetch failure from gitsm.py

Sumo to Thud migration: Fetch failure from gitsm.py

[da Silva Filho, Josias Inacio]

[-- Attachment #1: Type: text/plain, Size: 1353 bytes --]

Hi folks,

I'm working on migrating my environment from Sumo to Thud. After fixing a few quirks, it seems the fetch is failing for my foo.bb which uses gitsm.

.gitmodules content in foo repo:
[submodule "common-lib"]
                path = common-lib
                url = ssh://git@<url_to_common-lib>/common-lib.git

foo.bb:
SRC_URI="gitsm://git@b<url_to_foo>.git;protocol=ssh;branch=master"
SRCREV="f55fae9dc9b476f8ed193f15e2d5cd6747777451"

Digging inside gitsm.py it looks like module_hash is not being set correctly (empty) and that causes the issue:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/gitsm.py#n87

I added the following right after line 87, so I could dump the variables content. This is how I confirmed module_hash is empty:
raise bb.fetch2.FetchError('module: %s, submodules: %s, module_hash: %s, basecmd: %s, revisions: %s, paths: %s, clonedir: %s ' % ( module, submodules, module_hash, ud.basecmd, ud.revisions[name], paths[module], ud.clonedir))

And the output:
ERROR: Fetcher failure: module: common-lib, submodules: ['common-lib'], module_hash: , basecmd: git -c core.fsyncobjectfiles=0, revisions: f55fae9dc9b476f8ed193f15e2d5cd6747777451, paths: common-lib, clonedir: /localdata/yocto/downloads/git2/<foo's path>.git

Any ideas why module_hash is not being set correctly?

[-- Attachment #2: Type: text/html, Size: 4279 bytes --]

Sumo to Thud migration: Fetch failure from gitsm.py

[da Silva Filho, Josias Inacio]

[-- Attachment #1: Type: text/plain, Size: 1355 bytes --]

Hi folks,

I'm working on migrating my environment from Sumo to Thud. After fixing a few quirks, it seems the fetch is failing for my foo.bb which uses gitsm.

.gitmodules content in foo repo:
[submodule "common-lib"]
                path = common-lib
                url = ssh://git@<url_to_common-lib>/common-lib.git

foo.bb:
SRC_URI="gitsm://git@b<url_to_foo>.git;protocol=ssh;branch=master"
SRCREV="f55fae9dc9b476f8ed193f15e2d5cd6747777451"

Digging inside gitsm.py it looks like module_hash is not being set correctly (empty) and that causes the issue:
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/gitsm.py#n87

I added the following right after line 87, so I could dump the variables content. This is how I confirmed module_hash is empty:
raise bb.fetch2.FetchError('module: %s, submodules: %s, module_hash: %s, basecmd: %s, revisions: %s, paths: %s, clonedir: %s ' % ( module, submodules, module_hash, ud.basecmd, ud.revisions[name], paths[module], ud.clonedir))

And the output:
ERROR: Fetcher failure: module: common-lib, submodules: ['common-lib'], module_hash: , basecmd: git -c core.fsyncobjectfiles=0, revisions: f55fae9dc9b476f8ed193f15e2d5cd6747777451, paths: common-lib, clonedir: /localdata/yocto/downloads/git2/<foo's path>.git

Any ideas why module_hash is not being set correctly?


[-- Attachment #2: Type: text/html, Size: 4323 bytes --]

Re: Sumo to Thud migration: Fetch failure from gitsm.py

[Robert Yang]

Hi,

Please refer to the following 2 commits in bitbake which may help:

commit a042179a784bab6a6f4bd5d63db80c1c02624b1c
Author: Mark Hatle <mark.hatle@windriver.com>
Date:   Wed Oct 31 15:21:44 2018 -0400

     bitbake: fetch2/gitsm.py: Fix the references when the module and path are 
different

commit a28637abbceb05b52a6e38ab2340d3da9d0b1b37
Author: Mark Hatle <mark.hatle@windriver.com>
Date:   Wed Oct 31 15:21:43 2018 -0400

     bitbake: fetch2/gitsm.py: Disable branch checking on submodules


// Robert

On 11/22/18 11:31 PM, da Silva Filho, Josias Inacio wrote:
> Hi folks,
> 
> I’m working on migrating my environment from Sumo to Thud. After fixing a few 
> quirks, it seems the fetch is failing for my foo.bb which uses gitsm.
> 
> *.gitmodules content in foo repo:*
> 
> [submodule "common-lib"]
> 
>                  path = common-lib
> 
>                  url = ssh://git@<url_to_common-lib>/common-lib.git
> 
> *foo.bb:*
> 
> SRC_URI="gitsm://git@b<url_to_foo>.git;protocol=ssh;branch=master"
> 
> SRCREV="f55fae9dc9b476f8ed193f15e2d5cd6747777451"
> 
> Digging inside gitsm.py it looks like module_hash is not being set correctly 
> (empty) and that causes the issue:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/gitsm.py#n87
> 
> I added the following right after line 87, so I could dump the variables 
> content. This is how I confirmed module_hash is empty:
> 
> *raise bb.fetch2.FetchError('module: %s, submodules: %s, module_hash: %s, 
> basecmd: %s, revisions: %s, paths: %s, clonedir: %s ' % ( module, submodules, 
> module_hash, ud.basecmd, ud.revisions[name], paths[module], ud.clonedir))*
> 
> And the output:
> 
> ERROR: Fetcher failure: module: common-lib, submodules: ['common-lib'], 
> *module_hash*: , basecmd: git -c core.fsyncobjectfiles=0, revisions: 
> f55fae9dc9b476f8ed193f15e2d5cd6747777451, paths: common-lib, clonedir: 
> /localdata/yocto/downloads/git2/<foo's path>.git
> 
> Any ideas why module_hash is not being set correctly?
> 
> 

Re: [**EXTERNAL**] Re: Sumo to Thud migration: Fetch failure from gitsm.py

[da Silva Filho, Josias Inacio]

Hi Robert,

I had a look at those two commits before, also tried reverting them but got the same error.

I reverted a few other changes in gitsm.py for testing, but the error only went away once I reverted this one:

commit 5cd00e3e53819f3c10d1733480077ce9f3cc2ca8
Author: Mark Hatle <mark.hatle@windriver.com>
Date:   Tue Sep 25 13:15:25 2018 -0400

    bitbake: fetch2/gitsm.py: Rework the git submodule fetcher




-----Original Message-----
From: Robert Yang <liezhi.yang@windriver.com> 
Sent: Friday, November 23, 2018 5:31 AM
To: da Silva Filho, Josias Inacio <jdasilva@ciena.com>
Cc: bitbake-devel@lists.openembedded.org; Mark Hatle <mark.hatle@windriver.com>
Subject: [**EXTERNAL**] Re: [bitbake-devel] Sumo to Thud migration: Fetch failure from gitsm.py

Hi,

Please refer to the following 2 commits in bitbake which may help:

commit a042179a784bab6a6f4bd5d63db80c1c02624b1c
Author: Mark Hatle <mark.hatle@windriver.com>
Date:   Wed Oct 31 15:21:44 2018 -0400

     bitbake: fetch2/gitsm.py: Fix the references when the module and path are different

commit a28637abbceb05b52a6e38ab2340d3da9d0b1b37
Author: Mark Hatle <mark.hatle@windriver.com>
Date:   Wed Oct 31 15:21:43 2018 -0400

     bitbake: fetch2/gitsm.py: Disable branch checking on submodules


// Robert

On 11/22/18 11:31 PM, da Silva Filho, Josias Inacio wrote:
> Hi folks,
> 
> I'm working on migrating my environment from Sumo to Thud. After 
> fixing a few quirks, it seems the fetch is failing for my foo.bb which uses gitsm.
> 
> *.gitmodules content in foo repo:*
> 
> [submodule "common-lib"]
> 
>                  path = common-lib
> 
>                  url = ssh://git@<url_to_common-lib>/common-lib.git
> 
> *foo.bb:*
> 
> SRC_URI="gitsm://git@b<url_to_foo>.git;protocol=ssh;branch=master"
> 
> SRCREV="f55fae9dc9b476f8ed193f15e2d5cd6747777451"
> 
> Digging inside gitsm.py it looks like module_hash is not being set 
> correctly
> (empty) and that causes the issue:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fet
> ch2/gitsm.py#n87
> 
> I added the following right after line 87, so I could dump the 
> variables content. This is how I confirmed module_hash is empty:
> 
> *raise bb.fetch2.FetchError('module: %s, submodules: %s, module_hash: 
> %s,
> basecmd: %s, revisions: %s, paths: %s, clonedir: %s ' % ( module, 
> submodules, module_hash, ud.basecmd, ud.revisions[name], 
> paths[module], ud.clonedir))*
> 
> And the output:
> 
> ERROR: Fetcher failure: module: common-lib, submodules: 
> ['common-lib'],
> *module_hash*: , basecmd: git -c core.fsyncobjectfiles=0, revisions: 
> f55fae9dc9b476f8ed193f15e2d5cd6747777451, paths: common-lib, clonedir: 
> /localdata/yocto/downloads/git2/<foo's path>.git
> 
> Any ideas why module_hash is not being set correctly?
> 
> 

Re: Sumo to Thud migration: Fetch failure from gitsm.py

[Mark Hatle]

On 11/22/18 9:31 AM, da Silva Filho, Josias Inacio wrote:
> Hi folks,
> 
>  
> 
> I’m working on migrating my environment from Sumo to Thud. After fixing a few
> quirks, it seems the fetch is failing for my foo.bb which uses gitsm.
> 
>  
> 
> *.gitmodules content in foo repo:*
> 
> [submodule "common-lib"]
> 
>                 path = common-lib
> 
>                 url = ssh://git@<url_to_common-lib>/common-lib.git
> 
>                
> 
> *foo.bb:*
> 
> SRC_URI="gitsm://git@b<url_to_foo>.git;protocol=ssh;branch=master"
> 
> SRCREV="f55fae9dc9b476f8ed193f15e2d5cd6747777451"
> 
>  
> 
> Digging inside gitsm.py it looks like module_hash is not being set correctly
> (empty) and that causes the issue:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/gitsm.py#n87
> 
>  
> 
> I added the following right after line 87, so I could dump the variables
> content. This is how I confirmed module_hash is empty:
> 
> *raise bb.fetch2.FetchError('module: %s, submodules: %s, module_hash: %s,
> basecmd: %s, revisions: %s, paths: %s, clonedir: %s ' % ( module, submodules,
> module_hash, ud.basecmd, ud.revisions[name], paths[module], ud.clonedir))*

Run the cloning steps manually.  And run the same git instructions as what the
gitsm fetcher does to find the hash for the submodule.


git clone --bare <url> <destination>
cd <destination>
git show <commit>:.gitmodules

for each of the modules listed:
  git ls-tree -z -d <commit> <module path>

It's the bit in the for loop that doesn't seem to be returning anything.  The
only time I've ever seen this not work is when a module is defined in the
.gitmodules file, but was never actually added to the repository.

>  
> 
> And the output:
> 
> ERROR: Fetcher failure: module: common-lib, submodules: ['common-lib'],
> *module_hash*: , basecmd: git -c core.fsyncobjectfiles=0, revisions:
> f55fae9dc9b476f8ed193f15e2d5cd6747777451, paths: common-lib, clonedir:
> /localdata/yocto/downloads/git2/<foo's path>.git

git clone --bare ssh://git@b<url_to_foo>.git my_repo
cd my_repo
git show f55fae9dc9b476f8ed193f15e2d5cd6747777451:.gitmodules

for each of the modules listed: (focusing on common-lib)
  git ls-tree -z -d f55fae9dc9b476f8ed193f15e2d5cd6747777451 common-lib

BTW my suggestion is to throw an exception or at least do a warning in the for
loop if module_hash is empty... since this is clearly an invalid configuration.

(after the split line)

if not module_hash:
   logger.warning("Submodule %s of %s does not appear to have a commit defined."
% ( ud.url, module ))

--Mark

> Any ideas why module_hash is not being set correctly?
> 
>  
> 
> 
> 

Re: [**EXTERNAL**] Re: Sumo to Thud migration: Fetch failure from gitsm.py

[da Silva Filho, Josias Inacio]

Bingo! Indeed the module was added into .gitmodules but not added in the repo itself.

I will get someone to add the repo then I will re-test, but that looks to be the issue.

Thanks Mark!


-----Original Message-----
From: Mark Hatle <mark.hatle@windriver.com> 
Sent: Monday, November 26, 2018 9:09 AM
To: da Silva Filho, Josias Inacio <jdasilva@ciena.com>; bitbake-devel@lists.openembedded.org
Subject: [**EXTERNAL**] Re: [bitbake-devel] Sumo to Thud migration: Fetch failure from gitsm.py

On 11/22/18 9:31 AM, da Silva Filho, Josias Inacio wrote:
> Hi folks,
> 
>  
> 
> I'm working on migrating my environment from Sumo to Thud. After 
> fixing a few quirks, it seems the fetch is failing for my foo.bb which uses gitsm.
> 
>  
> 
> *.gitmodules content in foo repo:*
> 
> [submodule "common-lib"]
> 
>                 path = common-lib
> 
>                 url = ssh://git@<url_to_common-lib>/common-lib.git
> 
>                
> 
> *foo.bb:*
> 
> SRC_URI="gitsm://git@b<url_to_foo>.git;protocol=ssh;branch=master"
> 
> SRCREV="f55fae9dc9b476f8ed193f15e2d5cd6747777451"
> 
>  
> 
> Digging inside gitsm.py it looks like module_hash is not being set 
> correctly
> (empty) and that causes the issue:
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fet
> ch2/gitsm.py#n87
> 
>  
> 
> I added the following right after line 87, so I could dump the 
> variables content. This is how I confirmed module_hash is empty:
> 
> *raise bb.fetch2.FetchError('module: %s, submodules: %s, module_hash: 
> %s,
> basecmd: %s, revisions: %s, paths: %s, clonedir: %s ' % ( module, 
> submodules, module_hash, ud.basecmd, ud.revisions[name], 
> paths[module], ud.clonedir))*

Run the cloning steps manually.  And run the same git instructions as what the gitsm fetcher does to find the hash for the submodule.


git clone --bare <url> <destination>
cd <destination>
git show <commit>:.gitmodules

for each of the modules listed:
  git ls-tree -z -d <commit> <module path>

It's the bit in the for loop that doesn't seem to be returning anything.  The only time I've ever seen this not work is when a module is defined in the .gitmodules file, but was never actually added to the repository.

>  
> 
> And the output:
> 
> ERROR: Fetcher failure: module: common-lib, submodules: 
> ['common-lib'],
> *module_hash*: , basecmd: git -c core.fsyncobjectfiles=0, revisions:
> f55fae9dc9b476f8ed193f15e2d5cd6747777451, paths: common-lib, clonedir:
> /localdata/yocto/downloads/git2/<foo's path>.git

git clone --bare ssh://git@b<url_to_foo>.git my_repo cd my_repo git show f55fae9dc9b476f8ed193f15e2d5cd6747777451:.gitmodules

for each of the modules listed: (focusing on common-lib)
  git ls-tree -z -d f55fae9dc9b476f8ed193f15e2d5cd6747777451 common-lib

BTW my suggestion is to throw an exception or at least do a warning in the for loop if module_hash is empty... since this is clearly an invalid configuration.

(after the split line)

if not module_hash:
   logger.warning("Submodule %s of %s does not appear to have a commit defined."
% ( ud.url, module ))

--Mark

> Any ideas why module_hash is not being set correctly?
> 
>  
> 
> 
>