[PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

[PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

[Peter Kjellerstedt]

Differences in COMMON_LICENSE_DIR should not affect the task hashes.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index fba99e8f0c..47c8cb39f9 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
-    OMP_NUM_THREADS BB_CURRENTTASK"
+    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
 BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
     SSTATE_DIR SOURCE_DATE_EPOCH"
 BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \

Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

[Richard Purdie]

On Sun, 2022-02-13 at 21:34 +0100, Peter Kjellerstedt wrote:
> Differences in COMMON_LICENSE_DIR should not affect the task hashes.
> 
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
>  meta/conf/bitbake.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index fba99e8f0c..47c8cb39f9 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
>      BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
>      SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
>      SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
> -    OMP_NUM_THREADS BB_CURRENTTASK"
> +    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
>  BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
>      SSTATE_DIR SOURCE_DATE_EPOCH"
>  BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \

I think this has been discussed before and I'm very uneasy at the idea. Some
users would expect that if they add "their" version of a license in a layer with
higher priority, they'd expect the hashes to change.

Where is this causing an issue?

The whole idea of the "common" licenses probably needs revisiting anyway too :/

Cheers,

Richard

RE: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 13 februari 2022 22:44
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON
> 
> On Sun, 2022-02-13 at 21:34 +0100, Peter Kjellerstedt wrote:
> > Differences in COMMON_LICENSE_DIR should not affect the task hashes.
> >
> > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > ---
> >  meta/conf/bitbake.conf | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > index fba99e8f0c..47c8cb39f9 100644
> > --- a/meta/conf/bitbake.conf
> > +++ b/meta/conf/bitbake.conf
> > @@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
> >      BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
> >      SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
> >      SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
> > -    OMP_NUM_THREADS BB_CURRENTTASK"
> > +    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
> >  BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
> >      SSTATE_DIR SOURCE_DATE_EPOCH"
> >  BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
> 
> I think this has been discussed before and I'm very uneasy at the idea. Some
> users would expect that if they add "their" version of a license in a layer with
> higher priority, they'd expect the hashes to change.

If the value was a relative path I could buy that, but not for an 
absolute path.

> Where is this causing an issue?

Due to the huge number of licenses in meta/files/common-licenses after 
all SPDX licenses were added, we cannot use that directory anymore as 
it triples the recipe parsing time for us (since we define 
INCOMPATIBLE LICENSE as AVAILABLE_LICENSES minus COMPATIBLE_LICENSES, 
and the license code doesn't really handle having many hundreds of 
licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to 
create a common-licenses directory in one of our layers which only has 
symbolic links to the licenses in meta that we need. Then we set:

COMMON_LICENSE_DIR := "${LAYERDIR}/files/common-licenses"

in that layer's layer.conf, but since ${LAYERDIR} varies from build host 
to build host, our whole global sstate was now only useable on the build 
servers and not for local developer builds.

> 
> The whole idea of the "common" licenses probably needs revisiting anyway
> too :/
> 
> Cheers,
> 
> Richard

//Peter

Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

[Richard Purdie]

On Sun, 2022-02-13 at 22:36 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > Sent: den 13 februari 2022 22:44
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>;
> > openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to
> > BB_HASHEXCLUDE_COMMON
> > 
> > On Sun, 2022-02-13 at 21:34 +0100, Peter Kjellerstedt wrote:
> > > Differences in COMMON_LICENSE_DIR should not affect the task hashes.
> > > 
> > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > > ---
> > >  meta/conf/bitbake.conf | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > > index fba99e8f0c..47c8cb39f9 100644
> > > --- a/meta/conf/bitbake.conf
> > > +++ b/meta/conf/bitbake.conf
> > > @@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
> > > BB_TASKHASH BBPATH BBSERVER DL_DI
> > >      BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot
> > > DEPLOY_DIR \
> > >      SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
> > >      SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE
> > > GIT_CEILING_DIRECTORIES \
> > > -    OMP_NUM_THREADS BB_CURRENTTASK"
> > > +    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
> > >  BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS
> > > BUILDHISTORY_DIR \
> > >      SSTATE_DIR SOURCE_DATE_EPOCH"
> > >  BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME
> > > SSH_AGENT_PID \
> > 
> > I think this has been discussed before and I'm very uneasy at the idea. Some
> > users would expect that if they add "their" version of a license in a layer
> > with
> > higher priority, they'd expect the hashes to change.
> 
> If the value was a relative path I could buy that, but not for an 
> absolute path.

True, however my concern is more that if you're changing this you are changing
the configuration and you'd expect the hashes to change as a result. Adding it
to the exclusion list hides that.

> > Where is this causing an issue?
> 
> Due to the huge number of licenses in meta/files/common-licenses after 
> all SPDX licenses were added, we cannot use that directory anymore as 
> it triples the recipe parsing time for us (since we define 
> INCOMPATIBLE LICENSE as AVAILABLE_LICENSES minus COMPATIBLE_LICENSES, 
> and the license code doesn't really handle having many hundreds of 
> licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to 
> create a common-licenses directory in one of our layers which only has 
> symbolic links to the licenses in meta that we need. Then we set:
> 
> COMMON_LICENSE_DIR := "${LAYERDIR}/files/common-licenses"
> 
> in that layer's layer.conf, but since ${LAYERDIR} varies from build host 
> to build host, our whole global sstate was now only useable on the build 
> servers and not for local developer builds.

So shouldn't you just add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON in your
config?

Cheers,

Richard

RE: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON

[Peter Kjellerstedt]

> -----Original Message-----
> From: Richard Purdie <richard.purdie@linuxfoundation.org>
> Sent: den 14 februari 2022 11:32
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; openembedded-
> core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to
> BB_HASHEXCLUDE_COMMON
> 
> On Sun, 2022-02-13 at 22:36 +0000, Peter Kjellerstedt wrote:
> > > -----Original Message-----
> > > From: Richard Purdie <richard.purdie@linuxfoundation.org>
> > > Sent: den 13 februari 2022 22:44
> > > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>;
> > > openembedded-core@lists.openembedded.org
> > > Subject: Re: [OE-core] [PATCH] bitbake.conf: Add COMMON_LICENSE_DIR to
> > > BB_HASHEXCLUDE_COMMON
> > >
> > > On Sun, 2022-02-13 at 21:34 +0100, Peter Kjellerstedt wrote:
> > > > Differences in COMMON_LICENSE_DIR should not affect the task hashes.
> > > >
> > > > Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> > > > ---
> > > >  meta/conf/bitbake.conf | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> > > > index fba99e8f0c..47c8cb39f9 100644
> > > > --- a/meta/conf/bitbake.conf
> > > > +++ b/meta/conf/bitbake.conf
> > > > @@ -922,7 +922,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
> > > > BB_TASKHASH BBPATH BBSERVER DL_DI
> > > >      BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH
> extend_recipe_sysroot
> > > > DEPLOY_DIR \
> > > >      SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
> > > >      SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE
> > > > GIT_CEILING_DIRECTORIES \
> > > > -    OMP_NUM_THREADS BB_CURRENTTASK"
> > > > +    OMP_NUM_THREADS BB_CURRENTTASK COMMON_LICENSE_DIR"
> > > >  BB_HASHBASE_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON}
> PSEUDO_IGNORE_PATHS
> > > > BUILDHISTORY_DIR \
> > > >      SSTATE_DIR SOURCE_DATE_EPOCH"
> > > >  BB_HASHCONFIG_WHITELIST ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME
> > > > SSH_AGENT_PID \
> > >
> > > I think this has been discussed before and I'm very uneasy at the
> idea. Some
> > > users would expect that if they add "their" version of a license in a
> layer
> > > with
> > > higher priority, they'd expect the hashes to change.
> >
> > If the value was a relative path I could buy that, but not for an
> > absolute path.
> 
> True, however my concern is more that if you're changing this you are changing
> the configuration and you'd expect the hashes to change as a result. Adding it
> to the exclusion list hides that.

On the other hand, most (all?) other variables that take absolute paths are 
also present in BB_HASHEXCLUDE_COMMON so that differences to where the build 
directory happens to be do not affect the sstate.

> > > Where is this causing an issue?
> >
> > Due to the huge number of licenses in meta/files/common-licenses after
> > all SPDX licenses were added, we cannot use that directory anymore as
> > it triples the recipe parsing time for us (since we define
> > INCOMPATIBLE LICENSE as AVAILABLE_LICENSES minus COMPATIBLE_LICENSES,
> > and the license code doesn't really handle having many hundreds of
> > licenses in INCOMPATIBLE LICENSE). Thus as a workaround I have had to
> > create a common-licenses directory in one of our layers which only has
> > symbolic links to the licenses in meta that we need. Then we set:
> >
> > COMMON_LICENSE_DIR := "${LAYERDIR}/files/common-licenses"
> >
> > in that layer's layer.conf, but since ${LAYERDIR} varies from build host
> > to build host, our whole global sstate was now only useable on the build
> > servers and not for local developer builds.
> 
> So shouldn't you just add COMMON_LICENSE_DIR to BB_HASHEXCLUDE_COMMON in
> your config?

Well, based on your reluctance to add it to bitbake.conf, I have now instead 
added COMMON_LICENSE_DIR[vardepvalue] = "" together with where we redefine 
COMMON_LICENSE_DIR. 

However, the reason I suggested adding it to bitbake.conf is that if you 
change this variable, it is easy to miss that it can cause your sstate to 
be different for each build host. I know I did, and we have had it modified 
for half a year. It wasn't until yesterday, when I made a local build that 
I expected to fully build from our global sstate and it instead rebuilt a 
lot of recipes, that I found out there was something wrong. It took me a 
couple of hours to dig into the siginfo files to finally end up at the 
absolute path in COMMON_LICENSE_DIR as the source of the differing sstate.

> Cheers,
> 
> Richard

//Peter