oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Joakim Tjernlund]

In poky/meta/lib/oeqa/selftest/context.py there is:
        if "buildhistory.bbclass" in self.tc.td["BBINCLUDED"]:
            self.tc.logger.error("You have buildhistory enabled already and this isn't recommended for selftest, please disable it first.")
            raise OEQAPreRun

        if "rm_work.bbclass" in self.tc.td["BBINCLUDED"]:
            self.tc.logger.error("You have rm_work enabled which isn't recommended while running oe-selftest. Please disable it before continuing.")
            raise OEQAPreRun

        if "PRSERV_HOST" in self.tc.td:
            self.tc.logger.error("Please unset PRSERV_HOST in order to run oe-selftest")
            raise OEQAPreRun

        if "SANITY_TESTED_DISTROS" in self.tc.td:
            self.tc.logger.error("Please unset SANITY_TESTED_DISTROS in order to run oe-selftest")
            raise OEQAPreRun

I see not way to disable/unset any of these vars short of modify the src for the original machine, make the src
tree "dirty" before running the test.

In meta/lib/oeqa/selftest/cases/reproducible.py there is:
     config = textwrap.dedent('''\
            PACKAGE_CLASSES = "{package_classes}"
            INHIBIT_PACKAGE_STRIP = "1"
            TMPDIR = "{tmpdir}"
            LICENSE_FLAGS_ACCEPTED = "commercial"
            DISTRO_FEATURES:append = ' systemd pam'
            USERADDEXTENSION = "useradd-staticids"
            USERADD_ERROR_DYNAMIC = "skip"
            USERADD_UID_TABLES += "files/static-passwd"
            USERADD_GID_TABLES += "files/static-group"

These are hardcoded settings too of which some could need need tewaking, DISTRO_FEATURES for sure.

So reproducible builds does not seem usable outside poky or am I missing something here ?

 Jocke

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Alexander Kanavin]

On Fri, 26 Jan 2024 at 12:09, Joakim Tjernlund via
lists.openembedded.org
<Joakim.Tjernlund=infinera.com@lists.openembedded.org> wrote:
> These are hardcoded settings too of which some could need need tewaking, DISTRO_FEATURES for sure.
>
> So reproducible builds does not seem usable outside poky or am I missing something here ?

That is not the intention. If there are obstacles to reusability, we'd
like to eliminate them.

To comply with selftest/context.py things you likely need to create a
special configuration with these extra settings, and perhaps save it
as a config template, but that doesn't mean making a special distro.
There are various reasons for all these restrictions for running
selftests and some of it probably don't apply or can be solved better,
but as of now they're enforced.

systemd has been dropped from DISTRO_FEATURES in master, so you can
backport that change if that's what is problematic. You can also copy
and tweak the whole repro selftest if it's truly unsuitable.

Alex

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Joakim Tjernlund]

On Fri, 2024-01-26 at 13:18 +0100, Alexander Kanavin wrote:
> On Fri, 26 Jan 2024 at 12:09, Joakim Tjernlund via
> lists.openembedded.org
> <Joakim.Tjernlund=infinera.com@lists.openembedded.org> wrote:
> > These are hardcoded settings too of which some could need need tewaking, DISTRO_FEATURES for sure.
> > 
> > So reproducible builds does not seem usable outside poky or am I missing something here ?
> 
> That is not the intention. If there are obstacles to reusability, we'd
> like to eliminate them.
> 
> To comply with selftest/context.py things you likely need to create a
> special configuration with these extra settings, and perhaps save it
> as a config template, but that doesn't mean making a special distro.
> There are various reasons for all these restrictions for running
> selftests and some of it probably don't apply or can be solved better,
> but as of now they're enforced.
> 
> systemd has been dropped from DISTRO_FEATURES in master, so you can
> backport that change if that's what is problematic. You can also copy
> and tweak the whole repro selftest if it's truly unsuitable.

I will try workaround these somehow but will not go so far as creating a new MACHINE just to
run these tests.

But now you know what the main issue is, hopefully you can fix these somehow.
Why not force offending config off instead of error out(in kirkstone too please)?

 Jocke 

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Alexander Kanavin]

On Fri, 26 Jan 2024 at 14:21, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
> I will try workaround these somehow but will not go so far as creating a new MACHINE just to
> run these tests.

Why would you need a new machine? You only need a new configuration
template in addition to existing one (e.g.
conf/templates/repro-test/local.conf.sample and bblayer.conf.sample).
Then initialize the build with TEMPLATECONF pointing to that.

> But now you know what the main issue is, hopefully you can fix these somehow.
> Why not force offending config off instead of error out(in kirkstone too please)?

Patches from you would be most welcome. I think there's a already a
mechanism where build-st/conf/ contains a custom local.conf, so all
the needed settings could be added at the point where that is created?

Alex

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Joakim Tjernlund]

On Fri, 2024-01-26 at 14:33 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> On Fri, 26 Jan 2024 at 14:21, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > I will try workaround these somehow but will not go so far as creating a new MACHINE just to
> > run these tests.
> 
> Why would you need a new machine? You only need a new configuration
> template in addition to existing one (e.g.
> conf/templates/repro-test/local.conf.sample and bblayer.conf.sample).
> Then initialize the build with TEMPLATECONF pointing to that.

Ah, there is an idea. But this will still change the original machine so one cannot switch
between normal builds and repro builds in he same tree.

> 
> > But now you know what the main issue is, hopefully you can fix these somehow.
> > Why not force offending config off instead of error out(in kirkstone too please)?
> 
> Patches from you would be most welcome. I think there's a already a
> mechanism where build-st/conf/ contains a custom local.conf, so all
> the needed settings could be added at the point where that is created?

Yes, I think so. My python skills are pretty non existent though.

> 
> Alex
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#194360): https://lists.openembedded.org/g/openembedded-core/message/194360
> Mute This Topic: https://lists.openembedded.org/mt/103973969/7472049
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Joakim.Tjernlund@infinera.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Mikko Rapeli]

Hi,

On Fri, Jan 26, 2024 at 02:04:18PM +0000, Joakim Tjernlund via lists.openembedded.org wrote:
> On Fri, 2024-01-26 at 14:33 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> > On Fri, 26 Jan 2024 at 14:21, Joakim Tjernlund
> > <Joakim.Tjernlund@infinera.com> wrote:
> > > I will try workaround these somehow but will not go so far as creating a new MACHINE just to
> > > run these tests.
> > 
> > Why would you need a new machine? You only need a new configuration
> > template in addition to existing one (e.g.
> > conf/templates/repro-test/local.conf.sample and bblayer.conf.sample).
> > Then initialize the build with TEMPLATECONF pointing to that.
> 
> Ah, there is an idea. But this will still change the original machine so one cannot switch
> between normal builds and repro builds in he same tree.

FWIW, I ran into the same issues. Many selftests also have dependencies which only work
on pure poky distro, machine and build configs. The dependencies to various features
which break tests are not clearly marked inside the tests.

At least with repro test it is now possible to define different image and recipe
build targets instead of world builds now but sadly these tests can not be run on
real product machine configurations and need a vanilla poky machine and/or distro
configs. I guess patches welcome but not sure into which direction...

Cheers,

-Mikko

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Joakim Tjernlund]

On Fri, 2024-01-26 at 16:18 +0200, Mikko Rapeli wrote:
> Hi,
> 
> On Fri, Jan 26, 2024 at 02:04:18PM +0000, Joakim Tjernlund via lists.openembedded.org wrote:
> > On Fri, 2024-01-26 at 14:33 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> > > On Fri, 26 Jan 2024 at 14:21, Joakim Tjernlund
> > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > I will try workaround these somehow but will not go so far as creating a new MACHINE just to
> > > > run these tests.
> > > 
> > > Why would you need a new machine? You only need a new configuration
> > > template in addition to existing one (e.g.
> > > conf/templates/repro-test/local.conf.sample and bblayer.conf.sample).
> > > Then initialize the build with TEMPLATECONF pointing to that.
> > 
> > Ah, there is an idea. But this will still change the original machine so one cannot switch
> > between normal builds and repro builds in he same tree.
> 
> FWIW, I ran into the same issues. Many selftests also have dependencies which only work
> on pure poky distro, machine and build configs. The dependencies to various features
> which break tests are not clearly marked inside the tests.
> 
> At least with repro test it is now possible to define different image and recipe
> build targets instead of world builds now but sadly these tests can not be run on
> real product machine configurations and need a vanilla poky machine and/or distro
> configs. I guess patches welcome but not sure into which direction...

Testing this ATM:
INHERIT += "${@bb.utils.contains('BBLAYERS', 'meta-selftest', 'buildhistory', '', d)}"
SANITY_TESTED_DISTROS = "${@bb.utils.contains('BBLAYERS', 'meta-selftest', 'gentoo-* \n', '', d)}"

Initial tests seems OK. Need to make SANITY_TESTED_DISTROS more generic though.

 Jocke

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Joakim Tjernlund]

On Fri, 2024-01-26 at 14:33 +0100, Alexander Kanavin via lists.openembedded.org wrote:
> On Fri, 26 Jan 2024 at 14:21, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > I will try workaround these somehow but will not go so far as creating a new MACHINE just to
> > run these tests.
> 
> Why would you need a new machine? You only need a new configuration
> template in addition to existing one (e.g.
> conf/templates/repro-test/local.conf.sample and bblayer.conf.sample).
> Then initialize the build with TEMPLATECONF pointing to that.
> 
> > But now you know what the main issue is, hopefully you can fix these somehow.
> > Why not force offending config off instead of error out(in kirkstone too please)?
> 
A thought just occurred to me, these vars(SANITY_TESTED_DISTROS, buildhistory etc.),
do they need to be off in the original machine when building up the SSTATE ?
I figured they just needed to be off for the repro build(in xxxx-st dir) but
now I am not sure any more.

 Jocke

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Alexander Kanavin]

On Fri, 26 Jan 2024 at 15:52, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
> A thought just occurred to me, these vars(SANITY_TESTED_DISTROS, buildhistory etc.),
> do they need to be off in the original machine when building up the SSTATE ?
> I figured they just needed to be off for the repro build(in xxxx-st dir) but
> now I am not sure any more.

Machine is typically understood as MACHINE setting, please do not
confuse it with a build from a specific configuration in build/conf/.

SANITY_TESTED_DISTROS and buildhistory do not factor in sstate
signature calculation, so turning them off and one will not cause
cache invalidation and a rebuild.

Alex

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Joakim Tjernlund]

On Fri, 2024-01-26 at 15:55 +0100, Alexander Kanavin wrote:
> On Fri, 26 Jan 2024 at 15:52, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > A thought just occurred to me, these vars(SANITY_TESTED_DISTROS, buildhistory etc.),
> > do they need to be off in the original machine when building up the SSTATE ?
> > I figured they just needed to be off for the repro build(in xxxx-st dir) but
> > now I am not sure any more.
> 
> Machine is typically understood as MACHINE setting, please do not
> confuse it with a build from a specific configuration in build/conf/.
> 
> SANITY_TESTED_DISTROS and buildhistory do not factor in sstate
> signature calculation, so turning them off and one will not cause
> cache invalidation and a rebuild.
> 
> Alex

A bit odd that repro. builds forbids them if they don't factor in sstate

 Jocke

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Alexander Kanavin]

They're forbidden in selftests as a whole, for other reasons. You can
probably find the reasons out via git blame.

Alex

On Fri, 26 Jan 2024 at 16:14, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> On Fri, 2024-01-26 at 15:55 +0100, Alexander Kanavin wrote:
> > On Fri, 26 Jan 2024 at 15:52, Joakim Tjernlund
> > <Joakim.Tjernlund@infinera.com> wrote:
> > > A thought just occurred to me, these vars(SANITY_TESTED_DISTROS, buildhistory etc.),
> > > do they need to be off in the original machine when building up the SSTATE ?
> > > I figured they just needed to be off for the repro build(in xxxx-st dir) but
> > > now I am not sure any more.
> >
> > Machine is typically understood as MACHINE setting, please do not
> > confuse it with a build from a specific configuration in build/conf/.
> >
> > SANITY_TESTED_DISTROS and buildhistory do not factor in sstate
> > signature calculation, so turning them off and one will not cause
> > cache invalidation and a rebuild.
> >
> > Alex
>
> A bit odd that repro. builds forbids them if they don't factor in sstate
>
>  Jocke

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Joakim Tjernlund]

I may have found a couple of odd things:
1) It appears oe-selftest does a fresh checkout from git when building the B branch. We got a few pkgs
   that builds from tip of master branch and that branch may change between A and B builds

2) There are also some extra variants of pkgs; -dbg, -src etc.
   Are these significant or are they just included when an error is found in the binary pkg?

 Jocke

On Fri, 2024-01-26 at 16:21 +0100, Alexander Kanavin wrote:
> They're forbidden in selftests as a whole, for other reasons. You can
> probably find the reasons out via git blame.
> 
> Alex
> 
> On Fri, 26 Jan 2024 at 16:14, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > 
> > On Fri, 2024-01-26 at 15:55 +0100, Alexander Kanavin wrote:
> > > On Fri, 26 Jan 2024 at 15:52, Joakim Tjernlund
> > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > A thought just occurred to me, these vars(SANITY_TESTED_DISTROS, buildhistory etc.),
> > > > do they need to be off in the original machine when building up the SSTATE ?
> > > > I figured they just needed to be off for the repro build(in xxxx-st dir) but
> > > > now I am not sure any more.
> > > 
> > > Machine is typically understood as MACHINE setting, please do not
> > > confuse it with a build from a specific configuration in build/conf/.
> > > 
> > > SANITY_TESTED_DISTROS and buildhistory do not factor in sstate
> > > signature calculation, so turning them off and one will not cause
> > > cache invalidation and a rebuild.
> > > 
> > > Alex
> > 
> > A bit odd that repro. builds forbids them if they don't factor in sstate
> > 
> >  Jocke

Re: [OE-core] oe-selftest reproducible.ReproducibleTests.test_reproducible_builds appears to hardcode important tunable params

[Alexander Kanavin]

On Mon, 29 Jan 2024 at 17:19, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> I may have found a couple of odd things:
> 1) It appears oe-selftest does a fresh checkout from git when building the B branch. We got a few pkgs
>    that builds from tip of master branch and that branch may change between A and B builds

That behaviour is not reproducible by yocto project's definition of
reproducible. You need to exclude them, or pin the recipes to specific
revisions.

> 2) There are also some extra variants of pkgs; -dbg, -src etc.
>    Are these significant or are they just included when an error is found in the binary pkg?

Yes, the reproducibility test includes the full set of packages that a
given recipe produces. In some circumstances they may be less
important, but the test checks everything.

Alex