[OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[Wes Lindauer]

It should not matter what xz/zstd parallelism and resource usage is set to
when determining hash signatures. This allows sharing more sstate across
host machines with different CPU configurations.

Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
---
 meta/conf/bitbake.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index dd2df8a552..d9eb1831ea 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -944,7 +944,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DI
     BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot DEPLOY_DIR \
     SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
     SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE GIT_CEILING_DIRECTORIES \
-    OMP_NUM_THREADS BB_CURRENTTASK"
+    XZ_MEMLIMIT XZ_THREADS XZ_DEFAULTS ZSTD_THREADS OMP_NUM_THREADS BB_CURRENTTASK"
 BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS BUILDHISTORY_DIR \
     SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS RUST_TARGET_SYS"
 BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME SSH_AGENT_PID \
-- 
2.25.1

Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[Ernst Sjöstrand]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

If you have a compressed initrd for example this could give you a surprise,
so I'm not 100% sure about this.

/Regards

Den mån 22 aug. 2022 kl 07:00 skrev Wes Lindauer <wesley.lindauer@gmail.com
>:

> It should not matter what xz/zstd parallelism and resource usage is set to
> when determining hash signatures. This allows sharing more sstate across
> host machines with different CPU configurations.
>
> Signed-off-by: Wes Lindauer <wesley.lindauer@gmail.com>
> ---
>  meta/conf/bitbake.conf | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
> index dd2df8a552..d9eb1831ea 100644
> --- a/meta/conf/bitbake.conf
> +++ b/meta/conf/bitbake.conf
> @@ -944,7 +944,7 @@ BB_HASHEXCLUDE_COMMON ?= "TMPDIR FILE PATH PWD
> BB_TASKHASH BBPATH BBSERVER DL_DI
>      BB_WORKERCONTEXT BB_LIMITEDDEPS BB_UNIHASH extend_recipe_sysroot
> DEPLOY_DIR \
>      SSTATE_HASHEQUIV_METHOD SSTATE_HASHEQUIV_REPORT_TASKDATA \
>      SSTATE_HASHEQUIV_OWNER CCACHE_TOP_DIR BB_HASHSERVE
> GIT_CEILING_DIRECTORIES \
> -    OMP_NUM_THREADS BB_CURRENTTASK"
> +    XZ_MEMLIMIT XZ_THREADS XZ_DEFAULTS ZSTD_THREADS OMP_NUM_THREADS
> BB_CURRENTTASK"
>  BB_BASEHASH_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} PSEUDO_IGNORE_PATHS
> BUILDHISTORY_DIR \
>      SSTATE_DIR SOURCE_DATE_EPOCH RUST_BUILD_SYS RUST_HOST_SYS
> RUST_TARGET_SYS"
>  BB_HASHCONFIG_IGNORE_VARS ?= "${BB_HASHEXCLUDE_COMMON} DATE TIME
> SSH_AGENT_PID \
> --
> 2.25.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#169658):
> https://lists.openembedded.org/g/openembedded-core/message/169658
> Mute This Topic: https://lists.openembedded.org/mt/93175519/4947266
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> ernstp@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 3108 bytes --]

Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[Quentin Schulz]

Hi Wes,

On 8/22/22 06:59, Wes Lindauer wrote:
> It should not matter what xz/zstd parallelism and resource usage is set to
> when determining hash signatures. This allows sharing more sstate across
> host machines with different CPU configurations.
> 

It seems like it does matter in some ways, c.f.:
https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
and
https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d

Cheers,
Quentin

Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[Richard Purdie]

On Mon, 2022-08-22 at 11:28 +0200, Quentin Schulz wrote:
> Hi Wes,
> 
> On 8/22/22 06:59, Wes Lindauer wrote:
> > It should not matter what xz/zstd parallelism and resource usage is set to
> > when determining hash signatures. This allows sharing more sstate across
> > host machines with different CPU configurations.
> > 
> 
> It seems like it does matter in some ways, c.f.:
> https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
> and
> https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d

As Quentin mentions, the value does matter. In particular, those
compiled with a single thread differ to those with multiple threads. As
such the proposed patch can't be accepted.

Cheers,

Richard

RE: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 22 augusti 2022 13:19
> To: Quentin Schulz <quentin.schulz@theobroma-systems.com>; Wes Lindauer
> <wesley.lindauer@gmail.com>; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash
> ignore list
> 
> On Mon, 2022-08-22 at 11:28 +0200, Quentin Schulz wrote:
> > Hi Wes,
> >
> > On 8/22/22 06:59, Wes Lindauer wrote:
> > > It should not matter what xz/zstd parallelism and resource usage is set to
> > > when determining hash signatures. This allows sharing more sstate across
> > > host machines with different CPU configurations.
> > >
> >
> > It seems like it does matter in some ways, c.f.:
> > https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
> > and
> > https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d
> 
> As Quentin mentions, the value does matter. In particular, those
> compiled with a single thread differ to those with multiple threads. As
> such the proposed patch can't be accepted.
> 
> Cheers,
> 
> Richard

Isn't the problem of varying XZ_THREAD/ZSTD_THREAD vs taskhashes 
already taken care of by having:

XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" 
XZ_THREADS[vardepvalue] = "1"
XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}" 
XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" 
ZSTD_THREADS[vardepvalue] = "1"

in bitbake.conf.

//Peter

Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[Wes Lindauer]

[-- Attachment #1: Type: text/plain, Size: 2411 bytes --]

I think I understand. I am sorry for the FAD. I was experiencing this
sstate mismatch on the dunfell branch that does not contain the
'XZ_THREADS[vardepvalue] = "1"' change. This is probably why I attempted my
change in the hash exclude list and then tried to "port it forward to
master". I appreciate everyone taking the time to explain. I am still a
little confused why calculating every rpm task signature with a value of 1
is acceptable, but ignoring the variable in the signature is not. Are these
not different implementations of the same thing?

Thanks,
Wes L

On Mon, Aug 22, 2022 at 9:17 AM Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:

> > -----Original Message-----
> > From: openembedded-core@lists.openembedded.org <openembedded-
> > core@lists.openembedded.org> On Behalf Of Richard Purdie
> > Sent: den 22 augusti 2022 13:19
> > To: Quentin Schulz <quentin.schulz@theobroma-systems.com>; Wes Lindauer
> > <wesley.lindauer@gmail.com>; openembedded-core@lists.openembedded.org
> > Subject: Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash
> > ignore list
> >
> > On Mon, 2022-08-22 at 11:28 +0200, Quentin Schulz wrote:
> > > Hi Wes,
> > >
> > > On 8/22/22 06:59, Wes Lindauer wrote:
> > > > It should not matter what xz/zstd parallelism and resource usage is
> set to
> > > > when determining hash signatures. This allows sharing more sstate
> across
> > > > host machines with different CPU configurations.
> > > >
> > >
> > > It seems like it does matter in some ways, c.f.:
> > >
> https://git.openembedded.org/openembedded-core/commit/?id=0296dc71c01d1b7953d74ef37d56429e2f4fcfab
> > > and
> > >
> https://git.openembedded.org/openembedded-core/commit/?id=e72c0b94554a9bc293844ec2bddb0c04ea19791d
> >
> > As Quentin mentions, the value does matter. In particular, those
> > compiled with a single thread differ to those with multiple threads. As
> > such the proposed patch can't be accepted.
> >
> > Cheers,
> >
> > Richard
>
> Isn't the problem of varying XZ_THREAD/ZSTD_THREAD vs taskhashes
> already taken care of by having:
>
> XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> XZ_THREADS[vardepvalue] = "1"
> XZ_DEFAULTS ?= "--memlimit=${XZ_MEMLIMIT} --threads=${XZ_THREADS}"
> XZ_DEFAULTS[vardepsexclude] += "XZ_MEMLIMIT XZ_THREADS"
> ZSTD_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}"
> ZSTD_THREADS[vardepvalue] = "1"
>
> in bitbake.conf.
>
> //Peter
>
>

[-- Attachment #2: Type: text/html, Size: 3786 bytes --]

Re: [OE-core][PATCH] bitbake.conf: Add resource vars to hash ignore list

[Richard Purdie]

On Tue, 2022-08-30 at 15:27 -0400, Wes Lindauer wrote:
> I think I understand. I am sorry for the FAD. I was experiencing this
> sstate mismatch on the dunfell branch that does not contain the
> 'XZ_THREADS[vardepvalue] = "1"' change. This is probably why I
> attempted my change in the hash exclude list and then tried to "port
> it forward to master". I appreciate everyone taking the time to
> explain. I am still a little confused why calculating every rpm task
> signature with a value of 1 is acceptable, but ignoring the variable
> in the signature is not. Are these not different implementations of
> the same thing?

> > XZ_THREADS ?= "${@oe.utils.cpu_count(at_least=2)}" 
> > XZ_THREADS[vardepvalue] = "1"

Note we do *two* things, we set it to at least a count of 2 and give it
a known consistent value. It is the two things together than work out
ok.

If the user changes XZ_THREADS, as long as it is set to a value or 2 or
more it is still ok. If the user sets it to a value of 1, builds are
not reproducible.

You're right that we could have just ignored it, but this at least
gives a hint that a value of 2 or more should be used. Ideally we'd
have put a warning around a value of 1 but there are a lot of things
we'd do ideally and limited time...

Cheers,

Richard