[PATCH 1/2] lib/oe/sstatesig.py: dump locked.sigs.inc only when explicitly asked via -S lockedsigs

[PATCH 1/2] lib/oe/sstatesig.py: dump locked.sigs.inc only when explicitly asked via -S lockedsigs

[Alexander Kanavin]

This was writing out locked-sigs.inc into cwd with every
'bitbake -S' invocation. When the intent is only to to get task
stamps (-S none), or print the difference between them (-S printdiff),
the file is unnecessary clutter.

A couple of selftests/scripts were however relying on this, so they're
adjusted to explicitly request the file.

eSDK code calls dump_lockedsigs() separately via
oe.copy_buildsystem.generate_locked_sigs() and so isn't affected.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oe/sstatesig.py                 | 7 ++++---
 meta/lib/oeqa/selftest/cases/archiver.py | 2 +-
 meta/lib/oeqa/selftest/cases/signing.py  | 2 +-
 scripts/lib/checklayer/__init__.py       | 2 +-
 4 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 42e13a8c800..e250f51c124 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -142,9 +142,10 @@ class SignatureGeneratorOEBasicHashMixIn(object):
         super().set_taskdata(data[3:])
 
     def dump_sigs(self, dataCache, options):
-        sigfile = os.getcwd() + "/locked-sigs.inc"
-        bb.plain("Writing locked sigs to %s" % sigfile)
-        self.dump_lockedsigs(sigfile)
+        if 'lockedsigs' in options:
+            sigfile = os.getcwd() + "/locked-sigs.inc"
+            bb.plain("Writing locked sigs to %s" % sigfile)
+            self.dump_lockedsigs(sigfile)
         return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options)
 
 
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 3fa59fff510..3cb888c5067 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -141,7 +141,7 @@ class Archiver(OESelftestTestCase):
         pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV']
 
         # Generate the tasks signatures
-        bitbake('mc:mc1:%s mc:mc2:%s -c %s -S none' % (pn, pn, task))
+        bitbake('mc:mc1:%s mc:mc2:%s -c %s -S lockedsigs' % (pn, pn, task))
 
         # Check the tasks signatures
         # To be machine agnostic the tasks needs to generate the same signature for each machine
diff --git a/meta/lib/oeqa/selftest/cases/signing.py b/meta/lib/oeqa/selftest/cases/signing.py
index 322e753ed3b..18cce0ba258 100644
--- a/meta/lib/oeqa/selftest/cases/signing.py
+++ b/meta/lib/oeqa/selftest/cases/signing.py
@@ -191,7 +191,7 @@ class LockedSignatures(OESelftestTestCase):
 
         bitbake(test_recipe)
         # Generate locked sigs include file
-        bitbake('-S none %s' % test_recipe)
+        bitbake('-S lockedsigs %s' % test_recipe)
 
         feature = 'require %s\n' % locked_sigs_file
         feature += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n'
diff --git a/scripts/lib/checklayer/__init__.py b/scripts/lib/checklayer/__init__.py
index 0a0db2f02ac..8271ed7fe3b 100644
--- a/scripts/lib/checklayer/__init__.py
+++ b/scripts/lib/checklayer/__init__.py
@@ -307,7 +307,7 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
     cmd += 'bitbake '
     if failsafe:
         cmd += '-k '
-    cmd += '-S none world'
+    cmd += '-S lockedsigs world'
     sigs_file = os.path.join(builddir, 'locked-sigs.inc')
     if os.path.exists(sigs_file):
         os.unlink(sigs_file)
-- 
2.39.2

[PATCH 2/2] selftest/sstatetests: add a test for CDN sstate cache

[Alexander Kanavin]

Specifically, the test checks that everything needed for building
standard oe-core images for x86_64 and arm64 is available from
the cache (with minor exceptions). Going forward, a complete
world check could be enabled and additional configurations,
but that requires improvements to performance of hash equivalence
server in particular.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 meta/lib/oeqa/selftest/cases/sstatetests.py | 47 +++++++++++++++++++++
 1 file changed, 47 insertions(+)

diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
index 6ef339897bf..e392d6a1a63 100644
--- a/meta/lib/oeqa/selftest/cases/sstatetests.py
+++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
@@ -14,6 +14,7 @@ import re
 
 from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, get_bb_vars
 from oeqa.selftest.case import OESelftestTestCase
+from oeqa.core.decorator import OETestTag
 
 import oe
 import bb.siggen
@@ -879,3 +880,49 @@ expected_sametmp_output, expected_difftmp_output)
 INHERIT += "base-do-configure-modified"
 """,
 expected_sametmp_output, expected_difftmp_output)
+
+@OETestTag("yocto-mirrors")
+class SStateMirrors(SStateBase):
+    def check_bb_output(self, output, exceptions):
+        in_tasks = False
+        missing_objects = []
+        for l in output.splitlines():
+            if "The differences between the current build and any cached tasks start at the following tasks" in l:
+                in_tasks = True
+                continue
+            if "Writing task signature files" in l:
+                in_tasks = False
+                continue
+            if in_tasks:
+                recipe_task = l.split("/")[-1]
+                recipe, task = recipe_task.split(":")
+                for e in exceptions:
+                    if e[0] in recipe and task == e[1]:
+                        break
+                else:
+                    missing_objects.append(recipe_task)
+        self.assertTrue(len(missing_objects) == 0, "Missing objects in the cache:\n{}".format("\n".join(missing_objects)))
+
+
+    def run_test_cdn_mirror(self, machine, targets, exceptions):
+        exceptions = exceptions + [[t, "do_deploy_source_date_epoch"] for t in targets.split()]
+        exceptions = exceptions + [[t, "do_image_qa"] for t in targets.split()]
+        self.config_sstate(True)
+        self.append_config("""
+MACHINE = "{}"
+BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687"
+SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
+""".format(machine))
+        result = bitbake("-S printdiff {}".format(targets))
+        self.check_bb_output(result.output, exceptions)
+
+    def test_cdn_mirror_qemux86_64(self):
+        # Example:
+        # exceptions = [ ["packagegroup-core-sdk","do_package"] ]
+        exceptions = []
+        self.run_test_cdn_mirror("qemux86-64", "core-image-minimal core-image-full-cmdline core-image-weston core-image-sato-sdk", exceptions)
+
+    def test_cdn_mirror_qemuarm64(self):
+        exceptions = []
+        # core-image-weston isn't produced for arm64 currently
+        self.run_test_cdn_mirror("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions)
-- 
2.39.2

Re: [OE-core] [PATCH 2/2] selftest/sstatetests: add a test for CDN sstate cache

[Alexandre Belloni]

Hello Alex,

This doesn't apply cleanly, it seems the context is from another series.

On 20/10/2023 19:12:32+0200, Alexander Kanavin wrote:
> Specifically, the test checks that everything needed for building
> standard oe-core images for x86_64 and arm64 is available from
> the cache (with minor exceptions). Going forward, a complete
> world check could be enabled and additional configurations,
> but that requires improvements to performance of hash equivalence
> server in particular.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  meta/lib/oeqa/selftest/cases/sstatetests.py | 47 +++++++++++++++++++++
>  1 file changed, 47 insertions(+)
> 
> diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
> index 6ef339897bf..e392d6a1a63 100644
> --- a/meta/lib/oeqa/selftest/cases/sstatetests.py
> +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
> @@ -14,6 +14,7 @@ import re
>  
>  from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, get_bb_vars
>  from oeqa.selftest.case import OESelftestTestCase
> +from oeqa.core.decorator import OETestTag
>  
>  import oe
>  import bb.siggen
> @@ -879,3 +880,49 @@ expected_sametmp_output, expected_difftmp_output)
>  INHERIT += "base-do-configure-modified"
>  """,
>  expected_sametmp_output, expected_difftmp_output)
> +
> +@OETestTag("yocto-mirrors")
> +class SStateMirrors(SStateBase):
> +    def check_bb_output(self, output, exceptions):
> +        in_tasks = False
> +        missing_objects = []
> +        for l in output.splitlines():
> +            if "The differences between the current build and any cached tasks start at the following tasks" in l:
> +                in_tasks = True
> +                continue
> +            if "Writing task signature files" in l:
> +                in_tasks = False
> +                continue
> +            if in_tasks:
> +                recipe_task = l.split("/")[-1]
> +                recipe, task = recipe_task.split(":")
> +                for e in exceptions:
> +                    if e[0] in recipe and task == e[1]:
> +                        break
> +                else:
> +                    missing_objects.append(recipe_task)
> +        self.assertTrue(len(missing_objects) == 0, "Missing objects in the cache:\n{}".format("\n".join(missing_objects)))
> +
> +
> +    def run_test_cdn_mirror(self, machine, targets, exceptions):
> +        exceptions = exceptions + [[t, "do_deploy_source_date_epoch"] for t in targets.split()]
> +        exceptions = exceptions + [[t, "do_image_qa"] for t in targets.split()]
> +        self.config_sstate(True)
> +        self.append_config("""
> +MACHINE = "{}"
> +BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687"
> +SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
> +""".format(machine))
> +        result = bitbake("-S printdiff {}".format(targets))
> +        self.check_bb_output(result.output, exceptions)
> +
> +    def test_cdn_mirror_qemux86_64(self):
> +        # Example:
> +        # exceptions = [ ["packagegroup-core-sdk","do_package"] ]
> +        exceptions = []
> +        self.run_test_cdn_mirror("qemux86-64", "core-image-minimal core-image-full-cmdline core-image-weston core-image-sato-sdk", exceptions)
> +
> +    def test_cdn_mirror_qemuarm64(self):
> +        exceptions = []
> +        # core-image-weston isn't produced for arm64 currently
> +        self.run_test_cdn_mirror("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions)
> -- 
> 2.39.2
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189575): https://lists.openembedded.org/g/openembedded-core/message/189575
> Mute This Topic: https://lists.openembedded.org/mt/102085736/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH 2/2] selftest/sstatetests: add a test for CDN sstate cache

[Alexander Kanavin]

Right, I omitted the printdiff tests and that created a conflict with
master. I resent a version that's rebased directly on that.

Alex

On Sat, 21 Oct 2023 at 00:58, Alexandre Belloni
<alexandre.belloni@bootlin.com> wrote:
>
> Hello Alex,
>
> This doesn't apply cleanly, it seems the context is from another series.
>
> On 20/10/2023 19:12:32+0200, Alexander Kanavin wrote:
> > Specifically, the test checks that everything needed for building
> > standard oe-core images for x86_64 and arm64 is available from
> > the cache (with minor exceptions). Going forward, a complete
> > world check could be enabled and additional configurations,
> > but that requires improvements to performance of hash equivalence
> > server in particular.
> >
> > Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> > ---
> >  meta/lib/oeqa/selftest/cases/sstatetests.py | 47 +++++++++++++++++++++
> >  1 file changed, 47 insertions(+)
> >
> > diff --git a/meta/lib/oeqa/selftest/cases/sstatetests.py b/meta/lib/oeqa/selftest/cases/sstatetests.py
> > index 6ef339897bf..e392d6a1a63 100644
> > --- a/meta/lib/oeqa/selftest/cases/sstatetests.py
> > +++ b/meta/lib/oeqa/selftest/cases/sstatetests.py
> > @@ -14,6 +14,7 @@ import re
> >
> >  from oeqa.utils.commands import runCmd, bitbake, get_bb_var, create_temp_layer, get_bb_vars
> >  from oeqa.selftest.case import OESelftestTestCase
> > +from oeqa.core.decorator import OETestTag
> >
> >  import oe
> >  import bb.siggen
> > @@ -879,3 +880,49 @@ expected_sametmp_output, expected_difftmp_output)
> >  INHERIT += "base-do-configure-modified"
> >  """,
> >  expected_sametmp_output, expected_difftmp_output)
> > +
> > +@OETestTag("yocto-mirrors")
> > +class SStateMirrors(SStateBase):
> > +    def check_bb_output(self, output, exceptions):
> > +        in_tasks = False
> > +        missing_objects = []
> > +        for l in output.splitlines():
> > +            if "The differences between the current build and any cached tasks start at the following tasks" in l:
> > +                in_tasks = True
> > +                continue
> > +            if "Writing task signature files" in l:
> > +                in_tasks = False
> > +                continue
> > +            if in_tasks:
> > +                recipe_task = l.split("/")[-1]
> > +                recipe, task = recipe_task.split(":")
> > +                for e in exceptions:
> > +                    if e[0] in recipe and task == e[1]:
> > +                        break
> > +                else:
> > +                    missing_objects.append(recipe_task)
> > +        self.assertTrue(len(missing_objects) == 0, "Missing objects in the cache:\n{}".format("\n".join(missing_objects)))
> > +
> > +
> > +    def run_test_cdn_mirror(self, machine, targets, exceptions):
> > +        exceptions = exceptions + [[t, "do_deploy_source_date_epoch"] for t in targets.split()]
> > +        exceptions = exceptions + [[t, "do_image_qa"] for t in targets.split()]
> > +        self.config_sstate(True)
> > +        self.append_config("""
> > +MACHINE = "{}"
> > +BB_HASHSERVE_UPSTREAM = "hashserv.yocto.io:8687"
> > +SSTATE_MIRRORS ?= "file://.* http://cdn.jsdelivr.net/yocto/sstate/all/PATH;downloadfilename=PATH"
> > +""".format(machine))
> > +        result = bitbake("-S printdiff {}".format(targets))
> > +        self.check_bb_output(result.output, exceptions)
> > +
> > +    def test_cdn_mirror_qemux86_64(self):
> > +        # Example:
> > +        # exceptions = [ ["packagegroup-core-sdk","do_package"] ]
> > +        exceptions = []
> > +        self.run_test_cdn_mirror("qemux86-64", "core-image-minimal core-image-full-cmdline core-image-weston core-image-sato-sdk", exceptions)
> > +
> > +    def test_cdn_mirror_qemuarm64(self):
> > +        exceptions = []
> > +        # core-image-weston isn't produced for arm64 currently
> > +        self.run_test_cdn_mirror("qemuarm64", "core-image-minimal core-image-full-cmdline core-image-sato-sdk", exceptions)
> > --
> > 2.39.2
> >
>
> >
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#189575): https://lists.openembedded.org/g/openembedded-core/message/189575
> > Mute This Topic: https://lists.openembedded.org/mt/102085736/3617179
> > Group Owner: openembedded-core+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>
> --
> Alexandre Belloni, co-owner and COO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Re: [OE-core] [PATCH 1/2] lib/oe/sstatesig.py: dump locked.sigs.inc only when explicitly asked via -S lockedsigs

[Alexandre Belloni]

https://autobuilder.yoctoproject.org/typhoon/#/builders/69/builds/7980/steps/24/logs/stdio

On 20/10/2023 19:12:31+0200, Alexander Kanavin wrote:
> This was writing out locked-sigs.inc into cwd with every
> 'bitbake -S' invocation. When the intent is only to to get task
> stamps (-S none), or print the difference between them (-S printdiff),
> the file is unnecessary clutter.
> 
> A couple of selftests/scripts were however relying on this, so they're
> adjusted to explicitly request the file.
> 
> eSDK code calls dump_lockedsigs() separately via
> oe.copy_buildsystem.generate_locked_sigs() and so isn't affected.
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  meta/lib/oe/sstatesig.py                 | 7 ++++---
>  meta/lib/oeqa/selftest/cases/archiver.py | 2 +-
>  meta/lib/oeqa/selftest/cases/signing.py  | 2 +-
>  scripts/lib/checklayer/__init__.py       | 2 +-
>  4 files changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 42e13a8c800..e250f51c124 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -142,9 +142,10 @@ class SignatureGeneratorOEBasicHashMixIn(object):
>          super().set_taskdata(data[3:])
>  
>      def dump_sigs(self, dataCache, options):
> -        sigfile = os.getcwd() + "/locked-sigs.inc"
> -        bb.plain("Writing locked sigs to %s" % sigfile)
> -        self.dump_lockedsigs(sigfile)
> +        if 'lockedsigs' in options:
> +            sigfile = os.getcwd() + "/locked-sigs.inc"
> +            bb.plain("Writing locked sigs to %s" % sigfile)
> +            self.dump_lockedsigs(sigfile)
>          return super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigs(dataCache, options)
>  
>  
> diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
> index 3fa59fff510..3cb888c5067 100644
> --- a/meta/lib/oeqa/selftest/cases/archiver.py
> +++ b/meta/lib/oeqa/selftest/cases/archiver.py
> @@ -141,7 +141,7 @@ class Archiver(OESelftestTestCase):
>          pn = 'gcc-source-%s' % get_bb_vars(['PV'], 'gcc')['PV']
>  
>          # Generate the tasks signatures
> -        bitbake('mc:mc1:%s mc:mc2:%s -c %s -S none' % (pn, pn, task))
> +        bitbake('mc:mc1:%s mc:mc2:%s -c %s -S lockedsigs' % (pn, pn, task))
>  
>          # Check the tasks signatures
>          # To be machine agnostic the tasks needs to generate the same signature for each machine
> diff --git a/meta/lib/oeqa/selftest/cases/signing.py b/meta/lib/oeqa/selftest/cases/signing.py
> index 322e753ed3b..18cce0ba258 100644
> --- a/meta/lib/oeqa/selftest/cases/signing.py
> +++ b/meta/lib/oeqa/selftest/cases/signing.py
> @@ -191,7 +191,7 @@ class LockedSignatures(OESelftestTestCase):
>  
>          bitbake(test_recipe)
>          # Generate locked sigs include file
> -        bitbake('-S none %s' % test_recipe)
> +        bitbake('-S lockedsigs %s' % test_recipe)
>  
>          feature = 'require %s\n' % locked_sigs_file
>          feature += 'SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n'
> diff --git a/scripts/lib/checklayer/__init__.py b/scripts/lib/checklayer/__init__.py
> index 0a0db2f02ac..8271ed7fe3b 100644
> --- a/scripts/lib/checklayer/__init__.py
> +++ b/scripts/lib/checklayer/__init__.py
> @@ -307,7 +307,7 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
>      cmd += 'bitbake '
>      if failsafe:
>          cmd += '-k '
> -    cmd += '-S none world'
> +    cmd += '-S lockedsigs world'
>      sigs_file = os.path.join(builddir, 'locked-sigs.inc')
>      if os.path.exists(sigs_file):
>          os.unlink(sigs_file)
> -- 
> 2.39.2
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#189574): https://lists.openembedded.org/g/openembedded-core/message/189574
> Mute This Topic: https://lists.openembedded.org/mt/102085735/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH 1/2] lib/oe/sstatesig.py: dump locked.sigs.inc only when explicitly asked via -S lockedsigs

[Richard Purdie]

On Sat, 2023-10-21 at 13:44 +0200, Alexandre Belloni via
lists.openembedded.org wrote:
> https://autobuilder.yoctoproject.org/typhoon/#/builders/69/builds/7980/steps/24/logs/stdio

It depends on one of the autobuilder-helper patches which is master-
next there. The hard part is these patches are "flag day" ones :/.

Cheers,

Richard