[PATCH] bitbake warning fix: No bb files matched BBFILE_PATTERN_packageName

[PATCH] bitbake warning fix: No bb files matched BBFILE_PATTERN_packageName

[Mounesh Sutar]

From: Mounesh Sutar <mounesh_sutar@mentor.com>

bitbake is parsing all recipe files, against layers BBFILE_PATTERN for warning message display.
In case of sublayer with lower/equal priority, then all sublayer's patterns are not being
considered for pattern match. While checking recipe file against matching pattern,
the first matching pattern is accepted and added to matched list, while remaning PATTERNs are
not being considered. With this, the recipes are being neglected from PATTERN match.
This fix let's parser run through all the layers PATTERNs and on matching added to match list.

Upstream-Status: Submitted [bitbake-devel@lists.openembedded.org]

Signed-off-by: Mounesh Sutar <mounesh_sutar@mentor.com>
---
 bitbake/lib/bb/cooker.py |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 4df8881..5fde995 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1705,7 +1705,7 @@ class CookerCollectFiles(object):
                 if matched != None:
                     if not regex in matched:
                         matched.add(regex)
-                return pri
+                        return pri
         return 0
 
     def get_bbfiles(self):
-- 
1.7.9.5

Re: [PATCH] bitbake warning fix: No bb files matched BBFILE_PATTERN_packageName

[Mounesh Sutar]

[-- Attachment #1: Type: text/plain, Size: 3255 bytes --]

Hi,
Adding more information about this issue.
For example, consider below recipes with respective Pattern Name and
priority.

Package Name           Recipe file path
 BBFILE_PATTERN                    Priority

alsa                           /Projects/abc/alsa
/Projects/abc/alsa                            160
layer-man                 /Projects/abc/layer-man
/Projects/abc/layer-man                  150
xyz                            /Projects/abc/xyz
   /Projects/abc/xyz                             130
gpu-viv                      /Projects/abc/xyz/gpu-viv
/Projects/abc/xyz/gpu-viv                 130
update-1                   /Projects/abc/xyz/update-1
/Projects/abc/xyz/update-1              130
update-2                   /Projects/abc/xyz/update-2
/Projects/abc/xyz/update-2              130


If we consider the case of parsing alsa recipe, with its file name parsing
against all patterns, it would find matching pattern("/Projects/abc/alsa ")
and parse successfully.
Similarly the case with layer-man recipes.
When it comes to case of xyz, it does parses like other alsa recipe, as it
would find matching pattern("/Projects/abc/xyz") to its file name.
When it picks up next gpu-viv recipe file("/Projects/abc/xyz/gpu-viv/*/*.bb"),
and starts parsing against all PATTERNs,
on finding pattern "/Projects/abc/xyz", the condition :
"if regex.match(filename):"

will succeed, as partial file name "/Projects/abc/xyz/gpu-viv" matches to
it.
So, it is not matching against correct pattern ("Projects/abc/xyz/gpu-viv").
This is causing remaining recipes of folder "xyz" are not being parsed
against all patterns.
By current change, it will force to parse all recipe files against all
PATTERN's and store matching patterns.

Regards,
Mounesh

On Tue, Sep 27, 2016 at 11:54 AM, Mounesh Sutar <mounesh.sutar@gmail.com>
wrote:

> From: Mounesh Sutar <mounesh_sutar@mentor.com>
>
> bitbake is parsing all recipe files, against layers BBFILE_PATTERN for
> warning message display.
> In case of sublayer with lower/equal priority, then all sublayer's
> patterns are not being
> considered for pattern match. While checking recipe file against matching
> pattern,
> the first matching pattern is accepted and added to matched list, while
> remaning PATTERNs are
> not being considered. With this, the recipes are being neglected from
> PATTERN match.
> This fix let's parser run through all the layers PATTERNs and on matching
> added to match list.
>
> Upstream-Status: Submitted [bitbake-devel@lists.openembedded.org]
>
> Signed-off-by: Mounesh Sutar <mounesh_sutar@mentor.com>
> ---
>  bitbake/lib/bb/cooker.py |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> index 4df8881..5fde995 100644
> --- a/bitbake/lib/bb/cooker.py
> +++ b/bitbake/lib/bb/cooker.py
> @@ -1705,7 +1705,7 @@ class CookerCollectFiles(object):
>                  if matched != None:
>                      if not regex in matched:
>                          matched.add(regex)
> -                return pri
> +                        return pri
>          return 0
>
>      def get_bbfiles(self):
> --
> 1.7.9.5
>
>

[-- Attachment #2: Type: text/html, Size: 4448 bytes --]

Re: [PATCH] bitbake warning fix: No bb files matched BBFILE_PATTERN_packageName

[Andre McCurdy]

> On Tue, Sep 27, 2016 at 11:54 AM, Mounesh Sutar <mounesh.sutar@gmail.com>
>>
>> bitbake is parsing all recipe files, against layers BBFILE_PATTERN for
>> warning message display.
>> In case of sublayer with lower/equal priority, then all sublayer's
>> patterns are not being
>> considered for pattern match. While checking recipe file against matching
>> pattern,
>> the first matching pattern is accepted and added to matched list, while
>> remaning PATTERNs are
>> not being considered. With this, the recipes are being neglected from
>> PATTERN match.
>> This fix let's parser run through all the layers PATTERNs and on matching
>> added to match list.
>>
>> Upstream-Status: Submitted [bitbake-devel@lists.openembedded.org]
>>
>> Signed-off-by: Mounesh Sutar <mounesh_sutar@mentor.com>
>> ---
>>  bitbake/lib/bb/cooker.py |    2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
>> index 4df8881..5fde995 100644
>> --- a/bitbake/lib/bb/cooker.py
>> +++ b/bitbake/lib/bb/cooker.py
>> @@ -1705,7 +1705,7 @@ class CookerCollectFiles(object):
>>                  if matched != None:
>>                      if not regex in matched:
>>                          matched.add(regex)
>> -                return pri
>> +                        return pri
>>          return 0
>>
>>      def get_bbfiles(self):
>> --

Resurrecting an old thread as it's been pointed out to me that this
patch fixes the issue I raised a while ago on the oe-core list:

  http://lists.openembedded.org/pipermail/openembedded-core/2016-July/123466.html

Are there any problems or concerns with this change?

Re: [PATCH] bitbake warning fix: No bb files matched BBFILE_PATTERN_packageName

[Richard Purdie]

On Fri, 2017-10-06 at 13:37 -0700, Andre McCurdy wrote:
> > 
> > On Tue, Sep 27, 2016 at 11:54 AM, Mounesh Sutar <mounesh.sutar@gmai
> > l.com>
> > > 
> > > 
> > > bitbake is parsing all recipe files, against layers
> > > BBFILE_PATTERN for
> > > warning message display.
> > > In case of sublayer with lower/equal priority, then all
> > > sublayer's
> > > patterns are not being
> > > considered for pattern match. While checking recipe file against
> > > matching
> > > pattern,
> > > the first matching pattern is accepted and added to matched list,
> > > while
> > > remaning PATTERNs are
> > > not being considered. With this, the recipes are being neglected
> > > from
> > > PATTERN match.
> > > This fix let's parser run through all the layers PATTERNs and on
> > > matching
> > > added to match list.
> > > 
> > > Upstream-Status: Submitted [bitbake-devel@lists.openembedded.org]
> > > 
> > > Signed-off-by: Mounesh Sutar <mounesh_sutar@mentor.com>
> > > ---
> > >  bitbake/lib/bb/cooker.py |    2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > > index 4df8881..5fde995 100644
> > > --- a/bitbake/lib/bb/cooker.py
> > > +++ b/bitbake/lib/bb/cooker.py
> > > @@ -1705,7 +1705,7 @@ class CookerCollectFiles(object):
> > >                  if matched != None:
> > >                      if not regex in matched:
> > >                          matched.add(regex)
> > > -                return pri
> > > +                        return pri
> > >          return 0
> > > 
> > >      def get_bbfiles(self):
> > > --
> Resurrecting an old thread as it's been pointed out to me that this
> patch fixes the issue I raised a while ago on the oe-core list:
> 
>   http://lists.openembedded.org/pipermail/openembedded-core/2016-July
> /123466.html
> 
> Are there any problems or concerns with this change?

I had to go and look at this again but yes, there are problems with the
patch:

a) If matched = None, the function never returns a priority after this 
   change.
b) Currently the first match is returned and also used in parallel for 
   the priority calculation. After the patch, the priority used would 
   be the last match found. The patch therefore reverses priority 
   assignments for layers.

There is a definite problem here but the patch simply moves the
goalposts, it doesn't actually fix it. It breaks the current behaviour
too.

Cheers,

Richard

Re: [PATCH] bitbake warning fix: No bb files matched BBFILE_PATTERN_packageName

[Mounesh Sutar]

[-- Attachment #1: Type: text/plain, Size: 3345 bytes --]

Hi,
The intention of this patch was to, let parse recipe files against all
PATTERNS.
So that, the correct priority of recipe file with respect to pattern is
returned.
As you agree, currently the first match is being returned, I believe this
is not expected behaviour.
As per the function name, it should check the bbfile priority and return
the corrosponding value.
I don't know much about priority assignments to layers and this patches
implications.

I have modified the patch and submitted now.
Request you to provide your feedback and suggestions for possible
improvements.


On Sat, Oct 7, 2017 at 3:31 PM, Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Fri, 2017-10-06 at 13:37 -0700, Andre McCurdy wrote:
> > >
> > > On Tue, Sep 27, 2016 at 11:54 AM, Mounesh Sutar <mounesh.sutar@gmai
> > > l.com>
> > > >
> > > >
> > > > bitbake is parsing all recipe files, against layers
> > > > BBFILE_PATTERN for
> > > > warning message display.
> > > > In case of sublayer with lower/equal priority, then all
> > > > sublayer's
> > > > patterns are not being
> > > > considered for pattern match. While checking recipe file against
> > > > matching
> > > > pattern,
> > > > the first matching pattern is accepted and added to matched list,
> > > > while
> > > > remaning PATTERNs are
> > > > not being considered. With this, the recipes are being neglected
> > > > from
> > > > PATTERN match.
> > > > This fix let's parser run through all the layers PATTERNs and on
> > > > matching
> > > > added to match list.
> > > >
> > > > Upstream-Status: Submitted [bitbake-devel@lists.openembedded.org]
> > > >
> > > > Signed-off-by: Mounesh Sutar <mounesh_sutar@mentor.com>
> > > > ---
> > > >  bitbake/lib/bb/cooker.py |    2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
> > > > index 4df8881..5fde995 100644
> > > > --- a/bitbake/lib/bb/cooker.py
> > > > +++ b/bitbake/lib/bb/cooker.py
> > > > @@ -1705,7 +1705,7 @@ class CookerCollectFiles(object):
> > > >                  if matched != None:
> > > >                      if not regex in matched:
> > > >                          matched.add(regex)
> > > > -                return pri
> > > > +                        return pri
> > > >          return 0
> > > >
> > > >      def get_bbfiles(self):
> > > > --
> > Resurrecting an old thread as it's been pointed out to me that this
> > patch fixes the issue I raised a while ago on the oe-core list:
> >
> >   http://lists.openembedded.org/pipermail/openembedded-core/2016-July
> > /123466.html
> >
> > Are there any problems or concerns with this change?
>
> I had to go and look at this again but yes, there are problems with the
> patch:
>
> a) If matched = None, the function never returns a priority after this
>    change.
> b) Currently the first match is returned and also used in parallel for
>    the priority calculation. After the patch, the priority used would
>    be the last match found. The patch therefore reverses priority
>    assignments for layers.
>
> There is a definite problem here but the patch simply moves the
> goalposts, it doesn't actually fix it. It breaks the current behaviour
> too.
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 4830 bytes --]

[PATCH] bitbake warning fix: No bb files matched BBFILE_PATTERN_packageName

[Mounesh Sutar]

From: Mounesh Sutar <mounesh_sutar@mentor.com>

bitbake is parsing all recipe files, against layers BBFILE_PATTERN for warning message display.
In case of sublayer with lower/equal priority, then all sublayer's patterns are not being
considered for pattern match. While checking recipe file against matching pattern,
the first matching pattern is accepted and added to matched list, while remaning PATTERNs are
not being considered. With this, the recipes are being neglected from PATTERN match.
This fix let's parser run through all the layers PATTERNs and on matching added to match list.

Signed-off-by: Mounesh Sutar <mounesh_sutar@mentor.com>
---
 bitbake/lib/bb/cooker.py | 22 ++++++++++++++++------
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index f44a088..4cf00ee 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1412,13 +1412,23 @@ class CookerCollectFiles(object):
         self.bbfile_config_priorities = priorities
 
     def calc_bbfile_priority( self, filename, matched = None ):
+        objectInitialised = False
+        tmppri = 0
+        tmpregex = {}
         for _, _, regex, pri in self.bbfile_config_priorities:
-            if regex.match(filename):
-                if matched != None:
-                    if not regex in matched:
-                        matched.add(regex)
-                return pri
-        return 0
+            if regex.match(filename) :
+                if objectInitialised == False:
+                   tmpregex = regex
+                   tmppri = pri
+                   objectInitialised = True
+                else:
+                    if len(regex.pattern) > len(tmpregex.pattern):
+                        tmpregex = regex
+                        tmppri = pri
+        if matched != None:
+            if not tmpregex in matched:
+                matched.add(tmpregex)
+        return tmppri
 
     def get_bbfiles(self):
         """Get list of default .bb files by reading out the current directory"""
-- 
1.9.1