Re: [PATCH] bitbake: cooker: Ensure bbappends are found in stable order

Re: [PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
> core-bounces@lists.openembedded.org> On Behalf Of Wes Lindauer
> Sent: den 28 augusti 2019 23:38
> To: openembedded-core@lists.openembedded.org
> Subject: [OE-core] [PATCH] bitbake: cooker: Ensure bbappends are found
> in stable order
> 
> Thanks to wildcards in bbappend filenames, it's possible to have
> multiple bbappends that apply to the same recipe in the same directory.
> In order to get sstate hits between different workspaces, we want to
> apply those bbappend files in a consistent order.  Since readdir()
> returns files in a non-deterministic order between workspaces (based on
> inode number and/or time of creation), we'll need to sort its result in
> order to have any consistency.
> 
> Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
> ---
>  bitbake/lib/bb/cooker.py | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 0607fcc708..a9ee142aa5 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
>              (bbappend, filename) = b
>              if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
>                  filelist.append(filename)
> +        filelist.sort()
>          return filelist
> 
>      def collection_priorities(self, pkgfns, d):
> --
> 2.14.5

I don't think this was a good idea at all. Before, bbappends were applied 
in the priority order of the layers, but this change throws the sorting 
based on priority out the window and might cause bbappend files for higher 
layers to be applied before lower layers' bbappend files.

Please revert this change, and rethink how to maintain a stable order for 
bbappend files from the same directory without affecting bbappend files 
from different layers.

//Peter

Re: [PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[Mark Hatle]

On 9/11/19 12:07 PM, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: openembedded-core-bounces@lists.openembedded.org <openembedded-
>> core-bounces@lists.openembedded.org> On Behalf Of Wes Lindauer
>> Sent: den 28 augusti 2019 23:38
>> To: openembedded-core@lists.openembedded.org
>> Subject: [OE-core] [PATCH] bitbake: cooker: Ensure bbappends are found
>> in stable order
>>
>> Thanks to wildcards in bbappend filenames, it's possible to have
>> multiple bbappends that apply to the same recipe in the same directory.
>> In order to get sstate hits between different workspaces, we want to
>> apply those bbappend files in a consistent order.  Since readdir()
>> returns files in a non-deterministic order between workspaces (based on
>> inode number and/or time of creation), we'll need to sort its result in
>> order to have any consistency.
>>
>> Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
>> ---
>>  bitbake/lib/bb/cooker.py | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index 0607fcc708..a9ee142aa5 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
>>              (bbappend, filename) = b
>>              if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
>>                  filelist.append(filename)
>> +        filelist.sort()
>>          return filelist
>>
>>      def collection_priorities(self, pkgfns, d):
>> --
>> 2.14.5
> 
> I don't think this was a good idea at all. Before, bbappends were applied 
> in the priority order of the layers, but this change throws the sorting 
> based on priority out the window and might cause bbappend files for higher 
> layers to be applied before lower layers' bbappend files.
> 
> Please revert this change, and rethink how to maintain a stable order for 
> bbappend files from the same directory without affecting bbappend files 
> from different layers.

I had assumed that the sorting was only within each layer's bbappends.  If it's
global, you are absolutely right that priority order matters.

So maybe:

Within a layer (multiple .bbappends) 'sort' (alphabetic)

Between layers first by priority, then by order in the bblayers (for two layers
with the same priority)?

--Mark

> //Peter
> 

[PATCH] bitbake: cooker: Ensure bbappends are found in stable order

[Wes Lindauer]

Thanks to wildcards in bbappend filenames, it's possible to have
multiple bbappends that apply to the same recipe in the same directory.
In order to get sstate hits between different workspaces, we want to
apply those bbappend files in a consistent order.  Since readdir()
returns files in a non-deterministic order between workspaces (based on
inode number and/or time of creation), we'll need to sort its result in
order to have any consistency.

Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
---
 bitbake/lib/bb/cooker.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 0607fcc708..a9ee142aa5 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1869,6 +1869,7 @@ class CookerCollectFiles(object):
             (bbappend, filename) = b
             if (bbappend == f) or ('%' in bbappend and bbappend.startswith(f[:bbappend.index('%')])):
                 filelist.append(filename)
+        filelist.sort()
         return filelist
 
     def collection_priorities(self, pkgfns, d):
-- 
2.14.5