Re: [OE-core] file checksums versus SRC_URI globs

["Richard Purdie" <richard.purdie@linuxfoundation.org>]

On Thu, 2020-07-09 at 21:31 +0200, Rasmus Villemoes wrote:
> On 02/07/2020 08.42, Rasmus Villemoes via lists.openembedded.org wrote:
> > On 01/07/2020 16.03, Quentin Schulz wrote:
> > > Hi Rasmus,
> > > 
> > > On Wed, Jul 01, 2020 at 03:51:19PM +0200, Rasmus Villemoes wrote:
> > > > Hi,
> > > > 
> > > > We have a recipe that uses
> > > > 
> > > >   SRC_URI += "file://somedir/*"
> > > > 
> > > 
> > > Glob aren't supported. Use "file://somedir/" instead.
> > 
> > Thanks, that actually works for one of the cases we have (there are
> > others that use globs which cannot be solved quite that simply, but for
> > now I'm just listing files explicitly instead).
> > 
> > However, I'm not sure that "globs aren't supported". The commit I
> > referenced clearly tried to make that work (better), it also "works" in
> > the sense of unpacking the expected things when building from scratch -
> > there's even
> > 
> >     def test_local_wildcard(self):
> >         tree = self.fetchUnpack(['file://a', 'file://dir/*'])
> >         self.assertEqual(tree, ['a',  'dir/c', 'dir/d', 'dir/subdir/e'])
> > 
> > in bitbake/lib/bb/tests/fetch.py. And the two upstream recipes
> > connman-gnome_0.7.bb and matchbox-desktop_2.2.bb both use that exact
> > pattern.
> > 
> > So either
> > 
> > - this is a plain bug in the signature computation,
> 
> I'm guessing the culprit is
> 
> commit 6c0706a28d72c591f1b75b6e3f3b645859387c7e
> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
> Date:   Mon Dec 8 21:25:23 2014 +0000
> 
>     cache/fetch2/siggen: Ensure we track include history for file checksums
> 
> which was the one that introduced the ":True: or ":False" suffixing via
> 
> +                filelist.append(f + ":" + str(os.path.exists(f)))
> 
> and then also did
> 
> @@ -981,6 +980,10 @@ def get_file_checksums(filelist, pn):
> 
>      checksums = []
>      for pth in filelist.split():
> +        exist = pth.split(":")[1]
> +        if exist == "False":
> +            continue
> +        pth = pth.split(":")[0]
>          if '*' in pth:
>              # Handle globs
>              for f in glob.glob(pth):
> 
> which practically guaranteed that the "if '*' in pth" would be dead code.
> 
> Richard, do you agree that this is a bug in the signature computation?
> 
> As I wrote previously, there's no warning anywhere that using globs in
> SRC_URI will fail to take the contents of the referenced file into
> account in hashes, but there's obviously still a some code that makes
> file://*.c work wrt. unpacking (and a test case for that).

Sorry about the delay in replying to this, it was flagged in my inbox,
I've just lacked time to investigate and resolve it.

I agree its a bug but my patch above didn't break globing support for
file checksums, its simply never worked even before that change. After
a lot of thought, I've also concluded that I don't think it can work in
a sane way (which is why it was never implemented).

The checksum code needs to know about the files that were searched that
didn't exist as well as those that existed and were checksumed. For the
simple file://xxx/* url that is possible but for more imaginative
globs, its hard. In the file://xxx/* case it may as well be file://xxx/
 in the url which works and is already checksumed correctly.

I've therefore sent out patches to remove the globbing support and
clean up the couple of existing references in OE-Core which are
trivially adapted.

I appreciate this isn't the answer you want but I can't realistically
see how to make this work properly in a sane way that scales. I do
agree the pretence it works is bad though so want to fix that.

Cheers,

Richard