Re: [PATCH 1/7] gitsm.py: Fix when a submodule is defined, but not initialized

[Mark Hatle <mark.hatle@windriver.com>]

Sorry for the delay in getting a fix back to you..

See:

http://git.openembedded.org/bitbake-contrib/log/?h=mgh/gitsm

top two commits should resolve both this and the CLI11 issue.  (I am assuming
the third one is the same problem.)

Can you please give this a try.. if it's working now, I'll send a formal pull
request.

--Mark

On 1/19/19 12:44 AM, Khem Raj wrote:
> Mark
> 
> another regression
> 
> http://errors.yoctoproject.org/Errors/Details/217455/
> 
> http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-core/dbus/dbus-broker_git.bb?h=master
> 
> 
> 
> On Fri, Jan 18, 2019 at 5:27 PM Khem Raj <raj.khem@gmail.com> wrote:
>>
>> I cherry picked this commit on top of master and I still see failure to unpack
>>
>> ERROR: cli11-1.6.2+gitAUTOINC+bd4dc91184-r0 do_unpack: Fetcher
>> failure: Fetch command export PSEUDO_DISABLED=1; export
>> DBUS_SESSION_BUS_ADDRESS="unix:path=/run/user/1000/bus"; export
>> SSH_AGENT_PID="1235"; export SSH_AUTH_SOCK="/tmp/ssh-agent.sock.1000";
>> export PATH="/mnt/a/yoe/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/mnt/a/yoe/sources/openembedded-core/scripts:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/bin/arm-yoe-linux-musleabi:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot/usr/bin/crossscripts:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/sbin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/usr/bin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/sbin:/mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/recipe-sysroot-native/bin:/mnt/a/yoe/sources/bitbake/bin:/mnt/a/yoe/build/tmp/hosttools";
>> export HOME="/home/kraj"; git -c core.fsyncobjectfiles=0 config
>> core.bare false
>> ERROR: cli11-1.6.2+gitAUTOINC+bd4dc91184-r0 do_unpack: Function
>> failed: base_do_unpack
>> ERROR: Logfile of failure stored in:
>> /mnt/a/yoe/build/tmp/work/cortexa5t2hf-neon-vfpv4-yoe-linux-musleabi/cli11/1.6.2+gitAUTOINC+bd4dc91184-r0/temp/log.do_unpack.10171
>> ERROR: Task (/mnt/a/yoe/sources/meta-openembedded/meta-oe/recipes-support/cli11/cli11_1.6.2.bb:do_unpack)
>> failed with exit code '1'
>>
>> My bitbake tree is here
>>
>> https://github.com/YoeDistro/bitbake/commits/yoe/mut
>>
>> On Fri, Jan 18, 2019 at 3:06 PM Mark Hatle <mark.hatle@windriver.com> wrote:
>>>
>>> I think I have a fix for this.  Can you try the bitbake available from:
>>>
>>> git://git.openembedded.org/bitbake-contrib
>>>
>>> branch mgh/gitsm
>>>
>>> The commit that resolves this issue is:
>>>
>>> 0a7d8bd99a5ba394b00ed35593101b4cd324a58d
>>>
>>> --Mark
>>>
>>> On 1/18/19 3:20 PM, Mark Hatle wrote:
>>>> On 1/18/19 12:31 PM, Khem Raj wrote:
>>>>> On Fri, Jan 18, 2019 at 10:08 AM Mark Hatle <mark.hatle@windriver.com> wrote:
>>>>>>
>>>>>> On 1/18/19 9:07 AM, Khem Raj wrote:
>>>>>>>  I am seeing
>>>>>>>
>>>>>>> http://errors.yoctoproject.org/Errors/Details/217408/
>>>>>>> http://errors.yoctoproject.org/Errors/Details/217412/
>>>>>>
>>>>>> Do you know where these failing recipes are so that I can reproduce it locally?
>>>>>>
>>>>>> rwmem looks like it might be meta-ti?
>>>>>> cli11 looks like something from meta-oe.
>>>>>>
>>>>>
>>>>> yes thats right.
>>>>> http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-support/cli11/cli11_1.6.2.bb?h=master
>>>>>
>>>>> gitsm version of rwmem is not yes applied in meta-ti so you might have
>>>>> to get it from my staging branch
>>>>>
>>>>> https://github.com/YoeDistro/meta-ti/commit/3418beda1869882d5f94dc405d2727f09ffe20c4
>>>>
>>>> The reproducer appears to be that these repositories are 'git' based, but use
>>>> relative submodule locations from the main repository.
>>>>
>>>> So now the gitsm code needs to resolve relative URLs, which is something not
>>>> previously implemented.
>>>>
>>>> --Mark
>>>>
>>>>>> --Mark
>>>>>>
>>>>>>> On Tue, Jan 15, 2019 at 1:33 PM Mark Hatle <mark.hatle@windriver.com> wrote:
>>>>>>>>
>>>>>>>> It is possible for a submodule to be defined in the .gitmodules file, but
>>>>>>>> never initialized in the repository itself.  This shows itself when searching
>>>>>>>> for the defined module hash you will get back a empty value.
>>>>>>>>
>>>>>>>> Similarly we need to identify and skip defined but not initialized submodules
>>>>>>>> during the unpack stages as well.
>>>>>>>>
>>>>>>>> Thanks to raphael.lisicki@siemens.com for their help is figuring out how
>>>>>>>> to resolve this issue.
>>>>>>>>
>>>>>>>> Additionally a problem was found where, while unlikely, it may be possible
>>>>>>>> for the wrong revision to have been searched using ls-tree.  This has been
>>>>>>>> resolved in the update_submodules function by keeping the correct revision
>>>>>>>> along with the submodule path.
>>>>>>>>
>>>>>>>> Signed-off-by: Mark Hatle <mark.hatle@windriver.com>
>>>>>>>> ---
>>>>>>>>  lib/bb/fetch2/gitsm.py | 32 ++++++++++++++++++++++++++++----
>>>>>>>>  1 file changed, 28 insertions(+), 4 deletions(-)
>>>>>>>>
>>>>>>>> diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
>>>>>>>> index 35729db..b7959ff 100644
>>>>>>>> --- a/lib/bb/fetch2/gitsm.py
>>>>>>>> +++ b/lib/bb/fetch2/gitsm.py
>>>>>>>> @@ -64,6 +64,7 @@ class GitSM(Git):
>>>>>>>>      def update_submodules(self, ud, d):
>>>>>>>>          submodules = []
>>>>>>>>          paths = {}
>>>>>>>> +        revision = {}
>>>>>>>>          uris = {}
>>>>>>>>          local_paths = {}
>>>>>>>>
>>>>>>>> @@ -77,6 +78,7 @@ class GitSM(Git):
>>>>>>>>              for m, md in self.parse_gitmodules(gitmodules).items():
>>>>>>>>                  submodules.append(m)
>>>>>>>>                  paths[m] = md['path']
>>>>>>>> +                revision[m] = ud.revisions[name]
>>>>>>>>                  uris[m] = md['url']
>>>>>>>>                  if uris[m].startswith('..'):
>>>>>>>>                      newud = copy.copy(ud)
>>>>>>>> @@ -84,7 +86,17 @@ class GitSM(Git):
>>>>>>>>                      uris[m] = Git._get_repo_url(self, newud)
>>>>>>>>
>>>>>>>>          for module in submodules:
>>>>>>>> -            module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], paths[module]), d, quiet=True, workdir=ud.clonedir)
>>>>>>>> +            try:
>>>>>>>> +                module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, revision[module], paths[module]), d, quiet=True, workdir=ud.clonedir)
>>>>>>>> +            except:
>>>>>>>> +                # If the command fails, we don't have a valid file to check.  If it doesn't
>>>>>>>> +                # fail -- it still might be a failure, see next check...
>>>>>>>> +                module_hash = ""
>>>>>>>> +
>>>>>>>> +            if not module_hash:
>>>>>>>> +                logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
>>>>>>>> +                continue
>>>>>>>> +
>>>>>>>>              module_hash = module_hash.split()[2]
>>>>>>>>
>>>>>>>>              # Build new SRC_URI
>>>>>>>> @@ -143,7 +155,7 @@ class GitSM(Git):
>>>>>>>>          if not ud.shallow or ud.localpath != ud.fullshallow:
>>>>>>>>              self.update_submodules(ud, d)
>>>>>>>>
>>>>>>>> -    def copy_submodules(self, submodules, ud, destdir, d):
>>>>>>>> +    def copy_submodules(self, submodules, ud, name, destdir, d):
>>>>>>>>          if ud.bareclone:
>>>>>>>>              repo_conf = destdir
>>>>>>>>          else:
>>>>>>>> @@ -156,6 +168,18 @@ class GitSM(Git):
>>>>>>>>              srcpath = os.path.join(ud.clonedir, 'modules', md['path'])
>>>>>>>>              modpath = os.path.join(repo_conf, 'modules', md['path'])
>>>>>>>>
>>>>>>>> +            # Check if the module is initialized
>>>>>>>> +            try:
>>>>>>>> +                module_hash = runfetchcmd("%s ls-tree -z -d %s %s" % (ud.basecmd, ud.revisions[name], md['path']), d, quiet=True, workdir=ud.clonedir)
>>>>>>>> +            except:
>>>>>>>> +                # If the command fails, we don't have a valid file to check.  If it doesn't
>>>>>>>> +                # fail -- it still might be a failure, see next check...
>>>>>>>> +                module_hash = ""
>>>>>>>> +
>>>>>>>> +            if not module_hash:
>>>>>>>> +                logger.debug(1, "submodule %s is defined, but is not initialized in the repository. Skipping", module)
>>>>>>>> +                continue
>>>>>>>> +
>>>>>>>>              if os.path.exists(srcpath):
>>>>>>>>                  if os.path.exists(os.path.join(srcpath, '.git')):
>>>>>>>>                      srcpath = os.path.join(srcpath, '.git')
>>>>>>>> @@ -188,7 +212,7 @@ class GitSM(Git):
>>>>>>>>                  continue
>>>>>>>>
>>>>>>>>              submodules = self.parse_gitmodules(gitmodules)
>>>>>>>> -            self.copy_submodules(submodules, ud, dest, d)
>>>>>>>> +            self.copy_submodules(submodules, ud, name, dest, d)
>>>>>>>>
>>>>>>>>      def unpack(self, ud, destdir, d):
>>>>>>>>          Git.unpack(self, ud, destdir, d)
>>>>>>>> @@ -211,7 +235,7 @@ class GitSM(Git):
>>>>>>>>                  continue
>>>>>>>>
>>>>>>>>              submodules = self.parse_gitmodules(gitmodules)
>>>>>>>> -            self.copy_submodules(submodules, ud, ud.destdir, d)
>>>>>>>> +            self.copy_submodules(submodules, ud, name, ud.destdir, d)
>>>>>>>>
>>>>>>>>              submodules_queue = [(module, os.path.join(repo_conf, 'modules', md['path'])) for module, md in submodules.items()]
>>>>>>>>              while len(submodules_queue) != 0:
>>>>>>>> --
>>>>>>>> 1.8.3.1
>>>>>>>>
>>>>>>>> --
>>>>>>>> _______________________________________________
>>>>>>>> bitbake-devel mailing list
>>>>>>>> bitbake-devel@lists.openembedded.org
>>>>>>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
>>>>>>
>>>>
>>>