[PATCH]] checksum: Support uri formatted file list

[PATCH]] checksum: Support uri formatted file list

[Paulo Neves]

Before this commit, if the file list for checksumming had
files names with spaces there would be a crash. This happened
due to filelist.split breaking on the file names instead of
on k:v boundaries. Now we validate this case and emit a fatal
error if such case is found. This needs to be fatal as the split
will generate broken k:v from then on.

Instead of putting literal spaces in the file list the user
should urlencode the file names and if they contain coded spaces
they will be decoded. This is consistent with the current
practice where file names are urlencoded. A reproducer of the
issue this commit fixes, was to pass a do_compile[file-checksums]
list with files containing spaces in their names, urlencoded or
literal.

Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992
Signed-off-by: Paulo Neves <ptsneves@gmail.com>
---
 bitbake/lib/bb/checksum.py | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
index fb8a77f6ab..97cf10825e 100644
--- a/bitbake/lib/bb/checksum.py
+++ b/bitbake/lib/bb/checksum.py
@@ -8,6 +8,7 @@
 import glob
 import operator
 import os
+import urllib.parse
 import stat
 import bb.utils
 import logging
@@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache):
 
         checksums = []
         for pth in filelist.split():
-            exist = pth.split(":")[1]
+            spl = pth.split(':')
+            if len(spl) != 2:
+                bb.fatal("found unformatted path in filelist " + pth)
+
+            exist = spl[1]
             if exist == "False":
                 continue
-            pth = pth.split(":")[0]
+            pth = urllib.parse.unquote(spl[0])
             if '*' in pth:
                 # Handle globs
                 for f in glob.glob(pth):
-- 
2.25.1

Re: [bitbake-devel] [PATCH]] checksum: Support uri formatted file list

[Richard Purdie]

On Fri, 2022-01-21 at 16:39 +0100, Paulo Neves wrote:
> Before this commit, if the file list for checksumming had
> files names with spaces there would be a crash. This happened
> due to filelist.split breaking on the file names instead of
> on k:v boundaries. Now we validate this case and emit a fatal
> error if such case is found. This needs to be fatal as the split
> will generate broken k:v from then on.
> 
> Instead of putting literal spaces in the file list the user
> should urlencode the file names and if they contain coded spaces
> they will be decoded. This is consistent with the current
> practice where file names are urlencoded. A reproducer of the
> issue this commit fixes, was to pass a do_compile[file-checksums]
> list with files containing spaces in their names, urlencoded or
> literal.
> 
> Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992
> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
> ---
>  bitbake/lib/bb/checksum.py | 9 +++++++--
>  1 file changed, 7 insertions(+), 2 deletions(-)
> 
> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
> index fb8a77f6ab..97cf10825e 100644
> --- a/bitbake/lib/bb/checksum.py
> +++ b/bitbake/lib/bb/checksum.py
> @@ -8,6 +8,7 @@
>  import glob
>  import operator
>  import os
> +import urllib.parse
>  import stat
>  import bb.utils
>  import logging
> @@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache):
>  
>          checksums = []
>          for pth in filelist.split():
> -            exist = pth.split(":")[1]
> +            spl = pth.split(':')
> +            if len(spl) != 2:
> +                bb.fatal("found unformatted path in filelist " + pth)
> +
> +            exist = spl[1]
>              if exist == "False":
>                  continue
> -            pth = pth.split(":")[0]
> +            pth = urllib.parse.unquote(spl[0])
>              if '*' in pth:
>                  # Handle globs
>                  for f in glob.glob(pth):

Does any code currently urlencode the filepaths into this flag?

I think we may also need to add some tests around this to bitbake-selftest...

Cheers,

Richard

Re: [bitbake-devel] [PATCH]] checksum: Support uri formatted file list

[Paulo Neves]

I am not aware of it directly being done anywhere but files declared in 
the recipe are in URI form as far as i know. Where in the self test 
should i add such tests?

Paulo Neves

On 1/25/22 1:20 PM, Richard Purdie wrote:
> On Fri, 2022-01-21 at 16:39 +0100, Paulo Neves wrote:
>> Before this commit, if the file list for checksumming had
>> files names with spaces there would be a crash. This happened
>> due to filelist.split breaking on the file names instead of
>> on k:v boundaries. Now we validate this case and emit a fatal
>> error if such case is found. This needs to be fatal as the split
>> will generate broken k:v from then on.
>>
>> Instead of putting literal spaces in the file list the user
>> should urlencode the file names and if they contain coded spaces
>> they will be decoded. This is consistent with the current
>> practice where file names are urlencoded. A reproducer of the
>> issue this commit fixes, was to pass a do_compile[file-checksums]
>> list with files containing spaces in their names, urlencoded or
>> literal.
>>
>> Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992
>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>> ---
>>   bitbake/lib/bb/checksum.py | 9 +++++++--
>>   1 file changed, 7 insertions(+), 2 deletions(-)
>>
>> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
>> index fb8a77f6ab..97cf10825e 100644
>> --- a/bitbake/lib/bb/checksum.py
>> +++ b/bitbake/lib/bb/checksum.py
>> @@ -8,6 +8,7 @@
>>   import glob
>>   import operator
>>   import os
>> +import urllib.parse
>>   import stat
>>   import bb.utils
>>   import logging
>> @@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache):
>>   
>>           checksums = []
>>           for pth in filelist.split():
>> -            exist = pth.split(":")[1]
>> +            spl = pth.split(':')
>> +            if len(spl) != 2:
>> +                bb.fatal("found unformatted path in filelist " + pth)
>> +
>> +            exist = spl[1]
>>               if exist == "False":
>>                   continue
>> -            pth = pth.split(":")[0]
>> +            pth = urllib.parse.unquote(spl[0])
>>               if '*' in pth:
>>                   # Handle globs
>>                   for f in glob.glob(pth):
> Does any code currently urlencode the filepaths into this flag?
>
> I think we may also need to add some tests around this to bitbake-selftest...
>
> Cheers,
>
> Richard
>

Re: [bitbake-devel] [PATCH]] checksum: Support uri formatted file list

[Quentin Schulz]

Hi all,

On 1/25/22 13:55, Paulo Neves wrote:
> I am not aware of it directly being done anywhere but files declared in 
> the recipe are in URI form as far as i know. Where in the self test 
> should i add such tests?
> 
> Paulo Neves
> 
> On 1/25/22 1:20 PM, Richard Purdie wrote:
>> On Fri, 2022-01-21 at 16:39 +0100, Paulo Neves wrote:
>>> Before this commit, if the file list for checksumming had
>>> files names with spaces there would be a crash. This happened
>>> due to filelist.split breaking on the file names instead of
>>> on k:v boundaries. Now we validate this case and emit a fatal
>>> error if such case is found. This needs to be fatal as the split
>>> will generate broken k:v from then on.
>>>
>>> Instead of putting literal spaces in the file list the user
>>> should urlencode the file names and if they contain coded spaces
>>> they will be decoded. This is consistent with the current
>>> practice where file names are urlencoded. A reproducer of the
>>> issue this commit fixes, was to pass a do_compile[file-checksums]
>>> list with files containing spaces in their names, urlencoded or
>>> literal.
>>>
>>> Change-Id: I6ac4f1cffbb86e913883491d46e8cc69a028e992
>>> Signed-off-by: Paulo Neves <ptsneves@gmail.com>
>>> ---
>>>   bitbake/lib/bb/checksum.py | 9 +++++++--
>>>   1 file changed, 7 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/bitbake/lib/bb/checksum.py b/bitbake/lib/bb/checksum.py
>>> index fb8a77f6ab..97cf10825e 100644
>>> --- a/bitbake/lib/bb/checksum.py
>>> +++ b/bitbake/lib/bb/checksum.py
>>> @@ -8,6 +8,7 @@
>>>   import glob
>>>   import operator
>>>   import os
>>> +import urllib.parse
>>>   import stat
>>>   import bb.utils
>>>   import logging
>>> @@ -110,10 +111,14 @@ class FileChecksumCache(MultiProcessCache):
>>>           checksums = []
>>>           for pth in filelist.split():
>>> -            exist = pth.split(":")[1]
>>> +            spl = pth.split(':')
>>> +            if len(spl) != 2:
>>> +                bb.fatal("found unformatted path in filelist " + pth)
>>> +
>>> +            exist = spl[1]
>>>               if exist == "False":
>>>                   continue
>>> -            pth = pth.split(":")[0]
>>> +            pth = urllib.parse.unquote(spl[0])
>>>               if '*' in pth:
>>>                   # Handle globs
>>>                   for f in glob.glob(pth):
>> Does any code currently urlencode the filepaths into this flag?
>>

Not sure it's related in any way but I've had issues with spaces in 
filenames in externalsrc recipes in thud.

Wanted to fix it instead of opening a bug and I forgot. Been a few years 
already :/

May be a way to provide a reproducer for this and check this patch fixes it.

Cheers,
Quentin

>> I think we may also need to add some tests around this to 
>> bitbake-selftest...
>>
>> Cheers,
>>
>> Richard
>>
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13280): https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_message_13280&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=pncegqWmjAxc9HxxMkfqtewvKngB2W_pcdgejWkzwLKvGFKmfr6uVcaphu-SiuwN&s=mLKHnmnxJkR7swGcar1oHE6xSVtMIr693Gcm7foIKTo&e=
> Mute This Topic: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_88585130_6293953&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=pncegqWmjAxc9HxxMkfqtewvKngB2W_pcdgejWkzwLKvGFKmfr6uVcaphu-SiuwN&s=h2HL5U0BZSx3nzTr6VH_57MEWt3w5l1rMfSOQem3A1Q&e=
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_bitbake-2Ddevel_unsub&d=DwIFaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq8yBP6m6qZZ4njZguQhZhkI_-172IIy1t&m=pncegqWmjAxc9HxxMkfqtewvKngB2W_pcdgejWkzwLKvGFKmfr6uVcaphu-SiuwN&s=uFEWtQO2c-QNbOVQsu7jvNWbudDEpbbuWtZHwy_ZFJ8&e=  [quentin.schulz@theobroma-systems.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>