[PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Cuero Bugot]

When uninative is activated (poky's default) internal datastore variables are modified (NATIVELSBSTRING and SSTATEPOSTUNPACKFUNCS) to enable uninative
support. This is happening after parsing is done at the beginning of the build. On the next bitbake call the recipe would be parsed if the two
variables above were not added to the parsing whitelist BB_HASHCONFIG_WHITELIST.

The fix is to add these two variables to the recipe parsing whitelist BB_HASHCONFIG_WHITELIST, this is done at recipe parsing time, only when
uninative.bbclass is used.

Signed-off-by: Cuero Bugot <cbugot@sierrawireless.com>
---
 meta/classes/uninative.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/uninative.bbclass b/meta/classes/uninative.bbclass
index 8f34483..d1fdbc8 100644
--- a/meta/classes/uninative.bbclass
+++ b/meta/classes/uninative.bbclass
@@ -8,6 +8,9 @@ UNINATIVE_TARBALL ?= "${BUILD_ARCH}-nativesdk-libc.tar.bz2"
 #UNINATIVE_CHECKSUM[x86_64] = "dead"
 UNINATIVE_DLDIR ?= "${DL_DIR}/uninative/"
 
+# Enabling uninative will change the following variables so they need to go the parsing white list to prevent multiple recipe parsing
+BB_HASHCONFIG_WHITELIST += "NATIVELSBSTRING SSTATEPOSTUNPACKFUNCS"
+
 addhandler uninative_event_fetchloader
 uninative_event_fetchloader[eventmask] = "bb.event.BuildStarted"
 
-- 
2.7.4

Re: [PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1911 bytes --]

On Fri, Mar 16, 2018 at 10:31 AM Cuero Bugot <cbugot@sierrawireless.com>
wrote:

> When uninative is activated (poky's default) internal datastore variables
> are modified (NATIVELSBSTRING and SSTATEPOSTUNPACKFUNCS) to enable uninative
> support. This is happening after parsing is done at the beginning of the
> build. On the next bitbake call the recipe would be parsed if the two
> variables above were not added to the parsing whitelist
> BB_HASHCONFIG_WHITELIST.
>
> The fix is to add these two variables to the recipe parsing whitelist
> BB_HASHCONFIG_WHITELIST, this is done at recipe parsing time, only when
> uninative.bbclass is used.


It seems you have a case where data is already parsed and then uninstive is
enabled after this the reparse is happening. Or is it always happening when
uninative is enabled

>
>
> Signed-off-by: Cuero Bugot <cbugot@sierrawireless.com>
> ---
>  meta/classes/uninative.bbclass | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/meta/classes/uninative.bbclass
> b/meta/classes/uninative.bbclass
> index 8f34483..d1fdbc8 100644
> --- a/meta/classes/uninative.bbclass
> +++ b/meta/classes/uninative.bbclass
> @@ -8,6 +8,9 @@ UNINATIVE_TARBALL ?= "${BUILD_ARCH}-nativesdk-libc.tar.bz2"
>  #UNINATIVE_CHECKSUM[x86_64] = "dead"
>  UNINATIVE_DLDIR ?= "${DL_DIR}/uninative/"
>
> +# Enabling uninative will change the following variables so they need to
> go the parsing white list to prevent multiple recipe parsing
> +BB_HASHCONFIG_WHITELIST += "NATIVELSBSTRING SSTATEPOSTUNPACKFUNCS"
> +
>  addhandler uninative_event_fetchloader
>  uninative_event_fetchloader[eventmask] = "bb.event.BuildStarted"
>
> --
> 2.7.4
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2669 bytes --]

Re: [PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Cuero Bugot]

> > On Fri, Mar 16, 2018 at 10:31 AM Cuero Bugot <mailto:cbugot@sierrawireless.com> wrote:
> > When uninative is activated (poky's default) internal datastore variables are modified (NATIVELSBSTRING and SSTATEPOSTUNPACKFUNCS) to enable uninative
> > support. This is happening after parsing is done at the beginning of the build. On the next bitbake call the recipe would be parsed if the two
> > variables above were not added to the parsing whitelist BB_HASHCONFIG_WHITELIST.
> >
> > The fix is to add these two variables to the recipe parsing whitelist BB_HASHCONFIG_WHITELIST, this is done at recipe parsing time, only when
> > uninative.bbclass is used.

> It seems you have a case where data is already parsed and then uninstive is enabled after this the reparse is happening. Or is it always happening when uninative is enabled 

It is always happening when uninative is enabled (which is poky's default). The 2 first times you build you will have a full recipe parsing.
The reason is that the data is effectively modified on reception of BuildStarted event that happens after the parsing is done. Next time you run bitbake, the datastore signature is different and thus retrigger a recipe aprsing.

Re: [PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Cuero Bugot]

> > On Fri, Mar 16, 2018 at 10:31 AM Cuero Bugot <mailto:cbugot@sierrawireless.com> wrote:
> > When uninative is activated (poky's default) internal datastore variables are modified (NATIVELSBSTRING and SSTATEPOSTUNPACKFUNCS) to enable uninative
> > support. This is happening after parsing is done at the beginning of the build. On the next bitbake call the recipe would be parsed if the two
> > variables above were not added to the parsing whitelist BB_HASHCONFIG_WHITELIST.
> >
> > The fix is to add these two variables to the recipe parsing whitelist BB_HASHCONFIG_WHITELIST, this is done at recipe parsing time, only when
> > uninative.bbclass is used.

> It seems you have a case where data is already parsed and then uninstive is enabled after this the reparse is happening. Or is it always happening when uninative is enabled 

It is always happening when uninative is enabled (

Re: [PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Cuero Bugot]

> > > On Fri, Mar 16, 2018 at 10:31 AM Cuero Bugot <mailto:cbugot@sierrawireless.com> wrote:
> > > When uninative is activated (poky's default) internal datastore 
> > > variables are modified (NATIVELSBSTRING and SSTATEPOSTUNPACKFUNCS) 
> > > to enable uninative support. This is happening after parsing is done at the beginning of the build. On the next bitbake call the recipe would be parsed if the two variables above were not added to the parsing whitelist BB_HASHCONFIG_WHITELIST.
> > >
> > > The fix is to add these two variables to the recipe parsing 
> > > whitelist BB_HASHCONFIG_WHITELIST, this is done at recipe parsing time, only when uninative.bbclass is used.
>
> > It seems you have a case where data is already parsed and then 
> > uninstive is enabled after this the reparse is happening. Or is it 
> > always happening when uninative is enabled
>
> It is always happening when uninative is enabled (which is poky's default). The 2 first times you build you will have a full recipe parsing.
> The reason is that the data is effectively modified on reception of BuildStarted event that happens after the parsing is done. Next time you run bitbake, the datastore signature is different and thus retrigger a recipe aprsing.

Anything I could do help make merge-in this proposal?

Thx,
Cuero

Re: [PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Richard Purdie]

On Fri, 2018-03-30 at 12:45 +0000, Cuero Bugot wrote:
> > 
> > > 
> > > > 
> > > > On Fri, Mar 16, 2018 at 10:31 AM Cuero Bugot <mailto:cbugot@sie
> > > > rrawireless.com> wrote:
> > > > When uninative is activated (poky's default) internal
> > > > datastore 
> > > > variables are modified (NATIVELSBSTRING and
> > > > SSTATEPOSTUNPACKFUNCS) 
> > > > to enable uninative support. This is happening after parsing is
> > > > done at the beginning of the build. On the next bitbake call
> > > > the recipe would be parsed if the two variables above were not
> > > > added to the parsing whitelist BB_HASHCONFIG_WHITELIST.
> > > > 
> > > > The fix is to add these two variables to the recipe parsing 
> > > > whitelist BB_HASHCONFIG_WHITELIST, this is done at recipe
> > > > parsing time, only when uninative.bbclass is used.
> > > 
> > > It seems you have a case where data is already parsed and then 
> > > uninstive is enabled after this the reparse is happening. Or is
> > > it 
> > > always happening when uninative is enabled
> > It is always happening when uninative is enabled (which is poky's
> > default). The 2 first times you build you will have a full recipe
> > parsing.
> > The reason is that the data is effectively modified on reception of
> > BuildStarted event that happens after the parsing is done. Next
> > time you run bitbake, the datastore signature is different and thus
> > retrigger a recipe aprsing.
> Anything I could do help make merge-in this proposal?

Sorry about the delay, I wanted to check that we shouldn't be teaching
get_hash() in data_smart.py in bitbake something about excludevardeps
to make it work better.

In short the answer is that no, we shouldn't and your patch is the
better option. I've queued it in sumo-next. Thanks for figuring it out
as it is an annoying problem.

Cheers,

Richard

Re: [PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Cuero Bugot]

> In short the answer is that no, we shouldn't and your patch is the better option. I've queued it in sumo-next. Thanks for figuring it out as it is an annoying problem.

Hey no prb ! Given he traffic on this list I understand everyone is busy!
Would it be possible to add it to rocko-next  as well ? (the patch applies as is) (I ask in my self interest as we are using rocko as of now); or does it contradict specific policies for released branches ?
Thx,
Cuero

Re: [PATCH] uninative: add variables to the whitelist so that it does not re-triger recipe parsing

[Randy MacLeod]

On 2018-04-03 10:15 AM, Cuero Bugot wrote:
>> In short the answer is that no, we shouldn't and your patch is the better option. I've queued it in sumo-next. Thanks for figuring it out as it is an annoying problem.
> 
> Hey no prb ! Given he traffic on this list I understand everyone is busy!
> Would it be possible to add it to rocko-next  as well ? (the patch applies as is) (I ask in my self interest as we are using rocko as of now); or does it contradict specific policies for released branches ?

I'm not sure if there's any objection to backporting this patch but
Armin is the guy to ask about that. I've added him to the thread in case
he missed your rocko reference.

Usually, a patch has to be in master and not just master-next
before being backported IIRC, so give it a few days.

../Randy

> Thx,
> Cuero
> 


-- 
# Randy MacLeod.  WR Linux
# Wind River