Re: [PATCH 4/5] fetch2/gitsm: Add clean function

[Robert Yang <liezhi.yang@windriver.com>]

Hi Mark,

On 3/15/19 11:44 PM, Mark Hatle wrote:
> On 3/14/19 9:50 PM, Robert Yang wrote:
>> Hi,
>>
>> On 3/14/19 8:34 PM, Lisicki, Raphael wrote:
>>> I have not fully understand the logic but I am wondering what happens here is a submodule is located in two different repositories? Wouldn't a clean of one module delete submodules of the other one as well?
>>
>> Here are more explanations about the logic, for example, repo_A has two
>> submodules repo_B and repo_C, then after do_fetch,
>> they will be"
>> DL_DIR/repo_A
>> DL_DIR/repo_B
>> DL_DIR/repo_C
>>
>> Without this patch, do_cleanall only removes DL_DIR/repo_A, but leaves
>> repo_B and repo_C there, this patch makes it remove repo_B and C.
> 
> The issue Raphael is mentioning is when you have multiple projects that use the
> same submodules.  I.e.:
> 
> recipeA:
>   DL_DIR/repo_A
>   DL_DIR/repo_B
>   DL_DIR/repo_C
> 
> recipeB:
>   DL_DIR/repo_D
>   DL_DIR/repo_C
> 
> If you do a clean of A, you will end up removing repo_C which both items end up
> using.
> 
> I don't know any way to resolve this, since the dependency is implicit, not
> explicit -- so the system won't have any idea that the same thing is used in
> multiple places.
> 
>>
>> For your question, if repo_B is submodule of both repo_A and repo_D, it
>> is like foo and foo-native use the same git source, when you run
>> "bitbake foo -cccleanall", both foo and foo-native's source will be
>> removed since they use the same source, the similar to gitsm, so it is not
>> a problem, the source will be re-downloaded automatically when needed.
> 
> I'm not sure if this situation will work, since the recipeB will have a stamp
> that says the download is complete, even though part of it is now missing.


It's a generic issue when the same sources are use across recipes, e.g.:

# Cleanup everything
$ bitbake gnu-config gnu-config-native -ccleanall

# Fetch
$ bitbake gnu-config gnu-config-native -cfetch

# Cleanup the git repo
$ bitbake gnu-config -ccleanall

# Unpack gnu-config-native
$ bitbake gnu-config-native -cunpack

ERROR: gnu-config-native-20181128+gitAUTOINC+058639be22-r0 do_unpack: Unpack 
failure for URL: 'git://git.savannah.gnu.org/config.git'. No up to date source 
found: clone directory not available or not up to date: 
/buildarea1/lyang1/rebase-work/test_fetch/downloads/git2/git.savannah.gnu.org.config.git; 
shallow clone not enabled
ERROR: gnu-config-native-20181128+gitAUTOINC+058639be22-r0 do_unpack:
ERROR: gnu-config-native-20181128+gitAUTOINC+058639be22-r0 do_unpack: Function 
failed: base_do_unpack

So the users should handle it themselves in such a case.

// Robert

> 
> --Mark
> 
>> // Robert
>>
>>>
>>> Best regards
>>> Raphael
>>>
>>>> -----Original Message-----
>>>> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
>>>> bounces@lists.openembedded.org> On Behalf Of Robert Yang
>>>> Sent: Thursday, March 14, 2019 10:28 AM
>>>> To: bitbake-devel@lists.openembedded.org
>>>> Subject: [bitbake-devel] [PATCH 4/5] fetch2/gitsm: Add clean function
>>>>
>>>> The git's clean can only remove parent git repo from DL_DIR, but doesn't
>>>> remove
>>>> submodules, this patch fixes the problem.
>>>>
>>>> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
>>>> ---
>>>>    bitbake/lib/bb/fetch2/gitsm.py | 15 +++++++++++++--
>>>>    1 file changed, 13 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/bitbake/lib/bb/fetch2/gitsm.py b/bitbake/lib/bb/fetch2/gitsm.py
>>>> index b21fed2..bb61093 100644
>>>> --- a/bitbake/lib/bb/fetch2/gitsm.py
>>>> +++ b/bitbake/lib/bb/fetch2/gitsm.py
>>>> @@ -76,8 +76,8 @@ class GitSM(Git):
>>>>            for name in ud.names:
>>>>                try:
>>>>                    gitmodules = runfetchcmd("%s show %s:.gitmodules" %
>>>> (ud.basecmd, ud.revisions[name]), d, quiet=True, workdir=workdir)
>>>> -            except:
>>>> -                # No submodules to update
>>>> +            except Exception as esc:
>>>> +                logger.info("No submodules found: %s" % workdir)
>>>>                    continue
>>>>
>>>>                for m, md in parse_gitmodules(gitmodules).items():
>>>> @@ -164,6 +164,17 @@ class GitSM(Git):
>>>>            Git.download(self, ud, d)
>>>>            self.process_submodules(ud, ud.clonedir, download_submodule, d)
>>>>
>>>> +    def clean(self, ud, d):
>>>> +        def clean_submodule(ud, url, module, modpath, d):
>>>> +            try:
>>>> +                newfetch = Fetch([url], d, cache=False)
>>>> +                newfetch.clean()
>>>> +            except Exception as e:
>>>> +                logger.warn('gitsm: submodule clean failed: %s %s' %
>>>> (type(e).__name__, str(e)))
>>>> +
>>>> +        self.process_submodules(ud, ud.clonedir, clean_submodule, d)
>>>> +        Git.clean(self, ud, d)
>>>> +
>>>>        def unpack(self, ud, destdir, d):
>>>>            def unpack_submodules(ud, url, module, modpath, d):
>>>>                url += ";bareclone=1;nobranch=1"
>>>> --
>>>> 2.7.4
>>>>
>>>> --
>>>> _______________________________________________
>>>> bitbake-devel mailing list
>>>> bitbake-devel@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/bitbake-devel
> 
>