Re: [OE-core] file checksums versus SRC_URI globs

["Rasmus Villemoes" <rasmus.villemoes@prevas.dk>]

On 02/07/2020 08.42, Rasmus Villemoes via lists.openembedded.org wrote:
> On 01/07/2020 16.03, Quentin Schulz wrote:
>> Hi Rasmus,
>>
>> On Wed, Jul 01, 2020 at 03:51:19PM +0200, Rasmus Villemoes wrote:
>>> Hi,
>>>
>>> We have a recipe that uses
>>>
>>>   SRC_URI += "file://somedir/*"
>>>
>>
>> Glob aren't supported. Use "file://somedir/" instead.
> 
> Thanks, that actually works for one of the cases we have (there are
> others that use globs which cannot be solved quite that simply, but for
> now I'm just listing files explicitly instead).
> 
> However, I'm not sure that "globs aren't supported". The commit I
> referenced clearly tried to make that work (better), it also "works" in
> the sense of unpacking the expected things when building from scratch -
> there's even
> 
>     def test_local_wildcard(self):
>         tree = self.fetchUnpack(['file://a', 'file://dir/*'])
>         self.assertEqual(tree, ['a',  'dir/c', 'dir/d', 'dir/subdir/e'])
> 
> in bitbake/lib/bb/tests/fetch.py. And the two upstream recipes
> connman-gnome_0.7.bb and matchbox-desktop_2.2.bb both use that exact
> pattern.
> 
> So either
> 
> - this is a plain bug in the signature computation,

I'm guessing the culprit is

commit 6c0706a28d72c591f1b75b6e3f3b645859387c7e
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Mon Dec 8 21:25:23 2014 +0000

    cache/fetch2/siggen: Ensure we track include history for file checksums

which was the one that introduced the ":True: or ":False" suffixing via

+                filelist.append(f + ":" + str(os.path.exists(f)))

and then also did

@@ -981,6 +980,10 @@ def get_file_checksums(filelist, pn):

     checksums = []
     for pth in filelist.split():
+        exist = pth.split(":")[1]
+        if exist == "False":
+            continue
+        pth = pth.split(":")[0]
         if '*' in pth:
             # Handle globs
             for f in glob.glob(pth):

which practically guaranteed that the "if '*' in pth" would be dead code.

Richard, do you agree that this is a bug in the signature computation?

As I wrote previously, there's no warning anywhere that using globs in
SRC_URI will fail to take the contents of the referenced file into
account in hashes, but there's obviously still a some code that makes
file://*.c work wrt. unpacking (and a test case for that).

Rasmus